Task intent intermediate representation system

Abstract

The application relates to the technical field of artificial intelligence task representation and compilation, and provides a task intention intermediate representation system. The application adopts a differential update structure, only updates a changed part, does not need to regenerate an entire intention graph, greatly improves intention update efficiency, and reduces update cost. A traceability field of an intention graph node records a channel source, solves the problem that a task node cannot trace input modes in the prior art, and facilitates task modification and maintenance. The application realizes platform-independent unified expression of a task intention, breaks the barrier of private formats of various AI arrangement platforms through predetermined node type, edge type rules and formal data structure specifications, makes the task intention representation free from specific platform limitations, can be used as a standard bridging layer between multi-modal intention expression and heterogeneous execution, supports cross-platform reuse and compilation output, and greatly improves the interoperability of the task intention.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence task representation and compilation technology, and in particular to a task intent intermediate representation system. Background Technology

[0002] In the field of traditional programming language compilers, intermediate representation (IR) is a key abstraction layer connecting high-level language source code and target machine code. Typical IRs, such as LLVM IR and JVM bytecode, have four core characteristics: platform independence, optimizability, analyzability, and serializability. They can achieve cross-platform compilation, unified optimization, standardized analysis, and persistent transmission, providing core support for the efficient operation of compilers.

[0003] However, in the field of AI task execution, there is currently no equivalent intermediate representation standard for task intent. Existing AI orchestration platforms all use proprietary task description formats, such as LangChain's Chain / Graph JSON, Zapier's Zap configuration, and n8n's workflow JSON. These formats are strongly bound to specific platforms, cannot be reused across platforms, lack formal type systems and complete task semantic expression capabilities, do not carry semantic source information, and have extremely poor interoperability.

[0004] Meanwhile, existing research on task representations for multimodal interaction, such as visual-language task graphs and robot task planning graphs, has limitations such as domain-specificity, lack of modal tracing, and lack of differential update support. Each domain has its own custom representation format, which cannot be universally applied, and modifying a task requires resubmitting a complete description, resulting in low efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a task intent intermediate representation system to address the aforementioned technical problems.

[0006] A task intent intermediate representation system, the system comprising a multimodal input module, an alignment and fusion module, and an intent graph generation module; The multimodal input module collects the user's task intent from multiple independent channels and records the input timestamps; The alignment and fusion module establishes the association between different channel inputs in the time, space and semantic dimensions based on the time window, and generates a unified and fused task intent expression; The intent graph generation module converts the unified and integrated task intent expression into a directed intent graph according to predetermined node type and edge type rules. Each node in the intent graph contains a source field and an ambiguity flag field. The source field is used to record the source of the node channel, and the ambiguity flag field is used to mark unresolved referential ambiguities. The node types include object nodes for representing task objects, action nodes for representing actions to be performed, condition nodes for representing conditional decision branches, and target nodes for representing task objectives. The edge types include sequential edges, trigger edges, parallel edges, and conditional edges, and each edge type carries a source modality label. The intent graph adopts a differential update structure.

[0007] In one embodiment, the intent graph adopts a differential update structure, describes the update operation of the intent graph through the RFC 6901 JSONPointer path, and records the set of change nodes and trigger source of each differential update through a version history array.

[0008] In one embodiment, the intent graph has a built-in version management module that supports version rollback, version comparison, and selective recompilation.

[0009] In one embodiment, each node of the intent graph contains a constraint array for attaching typed constraints, including timeout constraints, retry constraints, priority constraints, and output constraints.

[0010] In one embodiment, the intent graph supports semantic equivalence interchange with JSON, binary compression, and human-readable simplified formats.

[0011] In one embodiment, the intent graph generation module includes an integrity verifier for verifying the integrity of the intent graph based on uniqueness rules, reference integrity rules, acyclicity rules, ambiguity clearing rules, and SHA256-based checksum integrity rules.

[0012] In one embodiment, the uniqueness rules include: all node IDs are unique within the document; all edge IDs are unique within the document; and there is one and only one target node. The referential integrity rules include: the source node ID and destination node ID of all edges must correspond to node IDs existing in the document node; the true branch ID and false branch ID of the condition node must be valid; and all IDs in the terminal node ID list of the target node must be valid. The acyclicity rule includes: except for branch paths guided by conditional edges, sequential edges and triggering edges in the intent graph do not form directed loops; The ambiguity clearing rule includes: maintaining the ambiguity count at 0; The SHA256-based checksum and integrity rules include: ensuring that the SHA256 checksum of the document is consistent with the document content.

[0013] In one embodiment, the update operation includes adding, removing, replacing, and moving.

[0014] In one embodiment, the system further includes a scene form template containing an array of synonyms to eliminate semantic equivalence ambiguity.

[0015] In one embodiment, the intent graph declares compilation targets through an array of compilation targets, each compilation target containing a target type and a target-specific configuration.

[0016] The aforementioned task intent intermediate representation system, by employing a differential update structure, updates only the changed parts without regenerating the entire intent graph, significantly improving intent update efficiency and reducing update costs. By recording the channel source through the traceability field of intent graph nodes, it solves the problem in existing technologies where task nodes cannot trace the input method, facilitating task modification and maintenance. This invention achieves platform-independent unified expression of task intents. Through predetermined node types, edge type rules, and formal data structure specifications, it breaks down the barriers of proprietary formats of various AI orchestration platforms, allowing task intent representation to be independent of specific platforms. It can serve as a standard bridging layer between multimodal intent expression and heterogeneous execution, supporting cross-platform reuse and compiled output, significantly improving the interoperability of task intents. Attached Figure Description

[0017] Figure 1 This is a block diagram of a task intent intermediate representation system in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In one embodiment, a multimodal intent editing and multi-target execution compilation system based on an intent protocol is provided. The system includes a multimodal input module, an alignment and fusion module, an intent protocol generation module, an execution compilation module, and an execution invocation module.

[0020] The multimodal input module collects the user's task intent from multiple independent channels and records the input timestamps.

[0021] The alignment and fusion module establishes the association between different channel inputs in the time, space and semantic dimensions based on the time window, and generates a unified and fused task intent expression.

[0022] The intent protocol generation module transforms the unified and integrated task intent expression into an intent protocol instance that conforms to the unified intent protocol specification. The intent protocol instance contains at least one intent unit. Each intent unit contains semantic fields of at least two of the following dimensions: object dimension, action dimension, sequence dimension, condition dimension, constraint dimension, spatial dimension, and time dimension. Each semantic field of the intent protocol instance contains a modality tracing field that records the modality source of the semantic field.

[0023] The execution compilation module converts the generated intent protocol instance into executable instructions according to the corresponding compilation template through a target-aware compilation mechanism, based on the target execution environment type.

[0024] The execution invocation module sends executable instructions to the corresponding execution entity for execution.

[0025] In the aforementioned multimodal intent editing and multi-target execution compilation system based on intent protocols, the multimodal input module collects task intents from multiple independent channels and records timestamps, breaking through the linear limitations of traditional one-dimensional text input. Furthermore, the alignment and fusion module achieves three-dimensional temporal, spatial, and semantic association and fusion of multi-channel information, significantly improving intent expression bandwidth. It can fully convey the logical relationships, execution constraints, and spatial layout of complex tasks without cumbersome text descriptions, significantly improving intent expression efficiency. The intent protocol generation module converts the unified and fused task intent expression into intent protocol instances conforming to the unified intent protocol specification, forming an auditable structured intermediate representation, solving the execution deviation problem caused by ambiguity in natural language reference. Finally, the execution compilation module adapts the same intent protocol into executable instructions for multiple heterogeneous execution targets, fundamentally solving the expression bandwidth bottleneck problem in the current AI interaction paradigm.

[0026] Understandably, the intent protocol instance can fully cover the seven dimensions of intent semantics and is independent of the execution target type, filling a fundamental gap in the field of AI task orchestration by lacking an underlying standard protocol. Furthermore, the same intent protocol instance can be compiled and output to any type of execution target through target-aware compilation, eliminating the redundant work of repeatedly writing task descriptions for different execution targets in existing systems. Task modifications only require updating the intent protocol instance (UIP instance), and the compilation output of all execution targets is automatically updated. Moreover, each semantic field includes a modality tracing field that records the source of the semantic field modality, allowing users to correct only erroneous input modalities without affecting the intent expression of other dimensions.

[0027] In one embodiment, the independent channels include point selection, drawing, trajectory, voice, blocks, multimedia upload, or scene form. Specifically, point selection input means that the user clicks on a target object, area, or function node on the two-dimensional operation interface, and the system records the click coordinates (x, y), timestamp, and semantic label of the clicked object (if an associated object library exists). Drawing input means that the user freely draws graphics (circular selection, arrow pointing, path sketch, etc.), and the system parses the geometric type of the graphic (closed area / directed line segment / polyline path), the coverage coordinate range, and the drawing time sequence. Trajectory input means that the user draws a continuous path in a map or spatial view, and the system converts the coordinate sequence into an ordered list of path nodes by time sampling, with each node containing coordinates and a timestamp. Voice input means that the user provides supplementary explanations through voice, and the system performs voice recognition and extracts the following from a predefined dictionary: entity words (object names), pronouns ("here" / "that" / "previous step"), action words ("execute" / "send" / "wait"), and constraint words ("timeout" / "retry" / "priority"). Block input involves users dragging and dropping predefined functional blocks (task type blocks, conditional blocks, loop blocks, data conversion blocks, etc.) and connecting them to assemble task logic. The system parses the block type identifiers and connection topology. Scene form input involves the system dynamically loading the corresponding scene form template (SceneTemplate) based on the user's selected task scene type. The scene form contains several fields, each corresponding to a control type (single-select, multiple-select, dropdown, slider, numeric input box, switch). Users directly set field values ​​by manipulating the controls, eliminating the need for text input. Each field value is directly written to the corresponding attribute of the corresponding node in the intent map through predefined mapping rules (FieldMapping), with the source_modality marked as form. Scene form input achieves zero ambiguity at the input source.

[0028] In one embodiment, when the intent protocol instance is implemented using a directed graph data structure, the unified and integrated task intent expression is converted into a directed intent graph according to predetermined node type and edge type rules. The nodes of the intent graph correspond to intent units, and the edges correspond to dependencies.

[0029] It should be noted that the intent graph data structure is a graph structure implementation of the Unified Intent Protocol (UIP). UIP also supports tree structures, linear sequences, and other implementations. The intent graph is a lower-level implementation of the intent protocol.

[0030] In one embodiment, a two-dimensional interface is provided for expressing multi-dimensional task intents. This method is applied to the two-dimensional operation interface of a multimodal intent editing and multi-target execution compilation system based on an intent protocol. The two-dimensional operation interface adopts a multi-layered canvas structure, including a logic layer, a spatial layer, a constraint layer, and a temporal layer stacked from bottom to top. The method includes the following steps: The intent of the user, expressed through graphical operations on the two-dimensional interface, includes the object dimension, action dimension, sequence dimension, condition dimension, constraint dimension, spatial dimension, and time dimension.

[0031] The intent of the object dimension can be expressed at the logical level through point placement, box selection association, and drag-and-drop import.

[0032] Specifically, point-and-click placement means that the user clicks on a blank area in the logic layer and selects the object type (data object / spatial object / Agent object) from the pop-up object type menu. The system then generates a corresponding Object node at the clicked location. Box selection association means that the user draws a closed area in the spatial layer using the drawing tool. The system aggregates all spatial coordinate points within the area into an Object node, and the node's attributes record the boundary coordinates of the selected area (spatial_range). Drag and drop import means that the user drags an external data source (file, URL, database table) into the logic layer, and the system automatically generates an Object node carrying a data source identifier.

[0033] The intention of the action dimension is expressed at the logical layer by dragging and dropping building blocks.

[0034] Specifically, users select the function block type (query / processing / send / control / AI call) from the left-hand block panel, drag it to the specified position in the logic layer, and the system generates the corresponding Action node. The visual appearance (color / icon) of the block is determined by action_type, enabling users to quickly identify the node function visually.

[0035] The intention of the sequential dimension is expressed at the logical level through directed connections.

[0036] Specifically, when a user hovers the mouse over the edge of the source node, a connection anchor point appears, and the user drags the mouse towards the target node. Releasing the mouse generates a directed connection (sequence edge). The direction of the connection is automatically resolved to the execution order: the direction the arrow points to indicates the subsequent execution node. Condition labels can be added to the midpoint of the connection. By right-clicking the midpoint of the connection, the user can upgrade the sequence edge to a trigger edge and fill in the trigger condition expression.

[0037] The intent of the condition dimension is expressed at the logical layer through diamond-shaped condition nodes.

[0038] Specifically, the user drags a diamond-shaped condition block from the block panel and places it at the location where a branch is needed. The diamond block automatically generates two exit anchor points (true branch / false branch). The user drags lines from the true anchor point and the false anchor point to the corresponding successor node, generating two condition edges. The system automatically labels the branches. The user clicks on the expression area within the diamond block to input the condition judgment logic (supports simple comparison expressions and voice input).

[0039] The intent of the constraint dimension is expressed on the constraint layer through a node badge mechanism.

[0040] The intent of spatial dimension is expressed through a dual-layer linkage mechanism of spatial layer and logical layer.

[0041] The intent of the time dimension is expressed through the time axis and parallel swimlane mechanism of the time layer.

[0042] The logic layer, spatial layer, constraint layer, and temporal layer are overlaid and displayed in a unified two-dimensional view, and converted into an intent protocol that conforms to the unified intent protocol specification. The spatial and temporal dimensions corresponding to the operations of each layer are recorded.

[0043] It should be noted that the object dimension refers to which objects, data, and locations are involved in the task; the action dimension refers to which operations are performed on the objects; the sequence dimension refers to the order of execution and dependencies of the actions; the condition dimension refers to the preconditions and branching logic of the action execution; the constraint dimension refers to the parameter restrictions and quality requirements of the action execution; the spatial dimension refers to the location of the object or action in the physical / logical space; and the time dimension refers to the triggering time and timing of the task.

[0044] Understandably, the logical layer (located in Layer 0) contains the logical structure of nodes, connections, and conditional branches; the spatial layer (located in Layer 1) contains the map background, geographic trajectory, and area drawing; the constraint layer (located in Layer 2) contains the constraint badges and parameter labels of each node; and the temporal layer (located in Layer 3) contains the timeline, timed trigger markers, and parallel swimlane lines.

[0045] By incorporating the intents of the architecture's object dimension, action dimension, sequence dimension, condition dimension, constraint dimension, spatial dimension, and temporal dimension, these intents are expressed at the logical, spatial, constraint, and temporal layers and then stacked and mapped onto a two-dimensional interface. This allows the two-dimensional interface to carry multi-dimensional task semantics without requiring interface switching or text input, thus fully expressing the intents of multi-dimensional and complex tasks.

[0046] In one embodiment, the graphics operations include point selection, drawing, trajectory drawing, and node connection.

[0047] In one embodiment, the two-dimensional operation interface further includes a separate scene form panel for loading parameters related to the scene selected by the user.

[0048] Understandably, the scene form panel is a floating panel independent of the four layers, does not occupy layer space, and can be called up and collapsed at any time. After the user selects a scene category, the panel dynamically loads the field controls of the corresponding SceneTemplate, displaying only parameters related to the current scene. After the scene form is filled out, the system batch injects the field values ​​into the corresponding nodes in the logical layer using FieldMapping rules. The logical layer nodes are automatically updated, and constraint badges are generated synchronously. Scene form operations can be used in combination with other layer operations: first fill out the form to set basic parameters, then switch to the spatial layer to draw the path, and then use voice to supplement constraints.

[0049] In one embodiment, in the two-dimensional operation interface, when any layer is active, the opacity of the inactive layer is reduced to keep the background visible, while the active layer is displayed with full opacity.

[0050] Specifically, all layers are always displayed in an overlay. When a user operates on any layer, the content of other layers remains visible with reduced opacity (default 40%) to avoid losing contextual information during operations. Users activate the target layer using the layer selector in the sidebar of the 2D interface. Activated layers regain 100% opacity, while inactive layers are reduced to 40% opacity.

[0051] In one embodiment, when there is a binding relationship between logical layer nodes and spatial layer coordinate points, the two-dimensional operation interface connects the corresponding elements of the two layers with dashed lines.

[0052] Specifically, when there is a binding relationship between logical layer nodes and spatial layer coordinate points, the interface connects the corresponding elements of the two layers with dashed lines to help users perceive cross-dimensional relationships.

[0053] In one embodiment, the intent of the constraint dimension is expressed on the constraint layer through a node badge mechanism, including: Badges can be added to target nodes through quick addition, automatic voice binding, automatic binding of scene forms, and batch application; the badges include timeout badges, retry badges, priority badges, and output constraint badges.

[0054] Specifically, the timeout badge is located in the upper right corner of the node, an orange circle displaying the timeout value (e.g., 5s). The retry badge is located in the lower right corner of the node, a blue circle displaying the number of retries (e.g., 3x). The priority badge is located in the upper left corner of the node, a red / yellow / green diamond corresponding to high / medium / low priority. The output constraint badge is located in the lower left corner of the node, a gray rectangle displaying the output limit parameter (e.g., 200 characters).

[0055] Quick Addition: Users can drag and drop constraint templates from the constraint template panel in the constraint layer to the target node. The node automatically generates a corresponding badge, and the badge parameter values ​​can be modified by clicking. Automatic Voice Binding: When voice input contains constraint words, the system automatically adds badges to the target node according to the constraint mounting rules of the sub-description, without manual operation. Automatic Scenario Form Binding: When constraint fields (such as timeout, retries, and priority) are filled in the scenario form, the system automatically generates corresponding constraint badges on the corresponding nodes, achieving the same effect as manually dragging and dropping constraint templates. A source tooltip is displayed when the badge from the scenario form is hovered over it. Batch Application: After selecting multiple nodes, users can drag constraint templates to the selected area to apply the same constraints to all selected nodes simultaneously.

[0056] It should be noted that, by default, the node badge is always visible, and users can intuitively perceive the global constraint distribution without entering the node attribute panel; when a node carries more than 4 constraints, the node displays a "+N" badge (N is the number of constraints exceeding the limit), and clicking it expands to show all constraints.

[0057] In one embodiment, the intent of the spatial dimension is expressed through a dual-layer linkage mechanism of the spatial layer and the logical layer, including: When a user adds or moves a path anchor point in the spatial layer, the coordinates of the corresponding object node in the logical layer are updated synchronously; when an object node in the logical layer carries a coordinate attribute, the spatial layer automatically displays the corresponding coordinate point. After a closed region in the spatial layer is bound to a specified node in the logical layer, the constraint layer of the node displays a spatial region badge, the content of which is the region area or region name; When the order of the logical layer sequential edges changes, the order of the path anchor points in the spatial layer is automatically adjusted synchronously, and the path connections are redrawn.

[0058] It's important to note that user operations in the spatial layer include path drawing, area painting, and coordinate annotation. Specifically, path drawing involves the user continuously clicking or dragging on the spatial layer (map background) to create a continuous path. The system segments the path, automatically generating spatial anchor points (Waypoints) at each key inflection point, and creating associated Object nodes at the corresponding locations in the logical layer. Area painting involves the user drawing closed areas (polygons or free curves) in the spatial layer. The system converts these closed areas into spatial constraints, binding them to target nodes in the logical layer as the execution area constraints for those nodes. Coordinate annotation involves the user clicking on a point on the map in the spatial layer to generate a coordinate label (Pin). The system creates a corresponding Object node in the logical layer, with the coordinate values ​​serving as the node's `coordinates` attribute.

[0059] In this embodiment, a dual-layer linkage mechanism is used to achieve precise binding and synchronous linkage between spatial intent and logical intent, effectively solving the problem of expression deviation caused by the disconnect between spatial dimension intent and logical intent.

[0060] In one embodiment, the intent of the time dimension is expressed through a timeline and parallel swimlane mechanism of the time layer, including: Drag a logic layer node to a specified moment on the timeline, and the node will generate a timed trigger attribute, which is mapped from the timeline position to a specific time value or relative delay. When a user places multiple nodes in the same time column in different lanes, the system automatically converts the sequential edges between these nodes into parallel edges, indicating parallel execution. When node A is earlier than node B on the timeline, and there is a sequence edge between them, the time layer displays a delay marker between the two nodes, prompting the user that node A needs to complete execution first.

[0061] In one embodiment, the association between different channel inputs in time, space, and semantic dimensions is established based on a time window to generate a unified and fused task intent expression, including: Based on the time window, a unified and integrated task intent expression is generated through time alignment rules, spatial anchor binding rules, action-node binding rules, constraint word mounting rules, and ambiguity resolution rules. The temporal alignment rule is that when a pronoun appears in the voice input, input events from other independent channels are searched within the time window, and the most recent event is used as a candidate spatial anchor point for the pronoun. The spatial anchor point binding rule is that the pronouns refer to the nearest point are bound to the most recent selected coordinates or the starting point of the trajectory within the time window, and the pronouns refer to the far point are bound to the center of the most recent painted area or the end point of the trajectory. The action-node binding rule is that action words in speech are bound to graphic nodes or blocks within the same time window. If there are multiple candidate nodes, the unique binding is determined by the principle of closest spatial distance. The constraint word mounting rule is that the constraint words extracted from the speech (timeout time, number of retries, priority, etc.) are mounted onto the last node operated on within the time window; The ambiguity resolution rule is as follows: if there is no unique spatial anchor point candidate within the time window, an ambiguity flag (ambiguity_flag=true) is set on the corresponding node in the intent map, and the user is prompted to manually confirm the binding relationship by highlighting it on the interface.

[0062] It should be noted that pronouns refer to the nearest point in time, such as: here, this place, current, this; while pronouns refer to the far point in time, such as: there, target location, that node, end point.

[0063] In one embodiment, the node type includes object nodes, action nodes, condition nodes, and target nodes; The edge types include sequential edges, triggered edges, parallel edges, and conditional edges.

[0064] Specifically, in the intent graph, object nodes represent objects or data involved in the task, with fields including: id, label (semantic name), coordinates (spatial coordinates, optional), and source_modality (source modality). Action nodes represent actions to be performed, with fields including: id, action_type, target_object (target object id), parameters (action parameter key-value pairs), and source_modality. Condition nodes represent conditional judgments, with fields including: id, condition_expression, true_branch, and false_branch. Goal nodes represent task goals, with fields including: id, success_condition, timeout (seconds), and priority. Sequential edges represent sequential execution relationships, pointing from the predecessor node to the successor node. Triggered edges represent event-triggered relationships, activating the target node when the source node's state meets the condition. Parallel edges represent parallel execution relationships. Conditional edges represent conditional branches, carrying branch condition expressions.

[0065] In one embodiment, a method for acquiring multimodal intent signals and generating structured intent descriptions is provided, applied to a multimodal intent editing and multi-target execution compilation system based on an intent protocol. The method includes the following steps: The system collects the user's task intent from multiple independent channels, records the input timestamps, and generates a multimodal input event stream, whereby the independent channels include a voice channel.

[0066] A time window scan is performed on each speech pronoun in the input event stream. Candidate spatial anchors are extracted according to a preset priority rule, and the anchor type is marked. The time window is constructed with the occurrence time of the pronoun as the center.

[0067] Based on the semantic type of pronouns and combined with the anchor type, spatial anchors are bound to each pronoun; constraint words are extracted from the speech input and attached to the target node according to the preset mounting rules; a unified and integrated task intent expression is generated.

[0068] The unified and integrated task intent expression is transformed into an intent protocol instance that conforms to the unified intent protocol specification. The intent protocol instance contains at least one intent unit. Each intent unit contains semantic fields of at least two of the following dimensions: object dimension, action dimension, sequence dimension, condition dimension, constraint dimension, spatial dimension, and time dimension. Each semantic field of the intent protocol instance contains a modality tracing field that records the modality source of the semantic field.

[0069] Understandably, the multimodal intent signal acquisition and structured intent description generation method is a specific generation method of intent protocol in a multimodal intent editing and multi-target execution compilation system based on intent protocols.

[0070] The aforementioned method for acquiring multimodal intent signals and generating structured intent descriptions binds each pronoun to a spatial anchor point based on the semantic type of the pronoun and the anchor point type. It also attaches constraint words to the target node according to preset mounting rules. This allows the multimodal user input to be fused to generate a structured intent protocol carrying source tracing fields, thereby achieving accurate and auditable expression of complex task intents.

[0071] It should be noted that modal names include select, draw, path, voice, block, and form. If a node is generated by multimodal fusion, the source field is assigned the value of fused (source_modality=fused), and the source_modality array records the names of all participating modalities and their contributions. The time window is constructed centered on the time of the reference token's appearance. Specifically, the time window W = [t_ref - T_before, t_ref + T_after] is defined centered on the time point t_ref when the reference token appears in the speech, where T_before defaults to 3000 milliseconds and T_after defaults to 1000 milliseconds (the usage habit of speech preceding the pointing action determines that the front window is larger).

[0072] In one embodiment, the preset priority rule is: Priority 1: For point selection events within the time window, extract the point selection coordinates and associated labels as candidate anchor points, and mark the candidate anchor point type as SelectAnchor; Priority 2: For drawing events within the time window, take the geometric center of the closed drawing area as the candidate anchor point and the end coordinates of the open path as the candidate anchor point. The candidate anchor point type is marked as DrawAnchor. Priority 3: For path trajectory nodes within the time window, candidate anchor points are selected based on the type of pronoun. For pronouns that are close to the target, the starting point is selected as the candidate anchor point, and for pronouns that are far from the target, the ending point is selected as the candidate anchor point. The candidate anchor point type is marked as PathAnchor. Priority level four: For block nodes within the time window, select the center coordinates and type identifier of the block as candidate anchor points, and mark the candidate anchor point type as BlockAnchor.

[0073] Understandably, priorities are executed in ascending order.

[0074] In one embodiment, a time window scan is performed on each speech pronoun in the input event stream, candidate spatial anchors are extracted according to a preset priority rule, and the anchor type is marked. The method also includes: If multiple candidate anchor points of the same priority exist within the time window, the one that is closest to the user's current focus will be the only candidate anchor point, and the remaining candidate anchor points will be kept as an alternative list.

[0075] In one embodiment, based on the semantic type of the pronoun and combined with the anchor type, spatial anchor binding is performed on each pronoun, including: Determine the semantic type of the pronoun. If the pronoun is a proximate pronoun, prioritize binding it to the candidate anchor with the highest priority within the same time window. Determine the type of the candidate anchor. If the candidate anchor is a SelectAnchor, the binding relationship is recorded as ReferenceToken → SelectAnchor.coordinates (coordinates) + SelectAnchor.label (label). If the candidate anchor is a DrawAnchor, the binding relationship is: ReferenceToken → DrawAnchor.region_center (DrawAnchor region center), and the geometric range of the drawing area is used as the spatial constraint for this binding. If the candidate anchor is a PathAnchor, the binding relationship is: ReferenceToken → PathAnchor.start_point (starting point coordinates). If the pronoun is a distant pronoun, it will be bound to DrawAnchor.region_center within the same time window first; if there is no DrawAnchor, it will be bound to PathAnchor.end_point (end point coordinates); if there is no path anchor, it will be bound to the SelectAnchor that is furthest from the current focus within the time window. If the pronoun is a sequence pronoun, then based on the node sequence of the currently generated intent map, the node ID corresponding to the sequence reference is located in chronological order.

[0076] It should be noted that sequence referents, such as "previous step," "next step," "after," and "previous node," are used for spatial anchor binding. Specifically, "previous step" is bound to the most recently generated Action node; "next step" triggers the operation of inserting a new node after the current node.

[0077] In one embodiment, the mounting rules include form field mapping rules, semantic matching priority rules, time-proximity mounting rules, and global constraint rules; The form field mapping rule is as follows: after the scene form is submitted, the system directly writes the value of each field into the corresponding intent map node attribute according to the scene form (FieldMappingTable); It should be noted that the mapping is a deterministic rule, without semantic inference and unaffected by the time window; the scene form value takes precedence over the voice constraint word; the user can explicitly cancel the scene form setting via voice, and the cancellation command overwrites the form value after the target option is identified by matching the synonym array.

[0078] The semantic matching priority rule is as follows: if the constraint word contains a keyword of a specific node type, it will be directly attached to the corresponding node through semantic matching, without being restricted by the time window. The time-based proximity mounting rule is as follows: if there is no semantically matching node, the constraint word is mounted to the last node operated on within the time window; The global constraint rule is as follows: if the semantics of the constraint word are global attributes, they are attached to the Goal node of the intent graph.

[0079] In one embodiment, the node type rules are as follows: each block generates an Action node, and the node's action_type is mapped by the block type identifier; each successfully bound pronoun generates or updates an Object node; action words in speech generate new Action nodes when there is no corresponding block; constraint words extracted from speech are attached as attributes to the target node; if a Goal semantic description exists, a Goal node is generated and connected to all terminal Action nodes; The edge type rules are as follows: the connection relationship between blocks is converted into a sequence edge; if the target node carries a ParallelFlag, it is converted into a parallel edge; if the sequence pronoun in the speech describes the node order, the corresponding sequence edge is generated; the true_branch and false_branch of the Condition node generate a condition edge.

[0080] In one embodiment, converting the unified and fused task intent expression into a directed intent graph according to predetermined node type rules and edge type rules further includes: If the ambiguity_flag of the pronoun is true, set the ambiguity_flag of the corresponding node to true and represent the node in the intent graph as pending confirmation. In the user interface, all nodes with ambiguity_flag=true are highlighted, and the user manually confirms all ambiguous bindings.

[0081] Understandably, for ambiguous nodes, the ambiguous nodes are highlighted in the user interface, and intent graphs containing ambiguous nodes are prevented from entering the compilation process until the user manually confirms all ambiguous bindings.

[0082] In one embodiment, based on the target execution environment type, the generated intent protocol instance is converted into executable instructions according to the corresponding compilation template using a target-aware compilation mechanism, including: When the target execution environment is an LLM (Large Model), the intent graph topology is traversed to sort the node sequence, the Prompt template is populated according to the action type and parameters, context constraints and output format requirements are automatically inserted, and an LLM Prompt is generated. When the target execution environment is an Agent, the action node sequence is converted into the Tool call sequence of the Agent framework, the condition node is converted into conditional routing logic, and a complete Agent execution plan JSON is generated. When the target execution environment is a drone, the trajectory nodes are converted into a geographic coordinate sequence, the action nodes are converted into drone action commands, the constraint nodes are converted into flight limitation parameters, and the task file that conforms to the target control protocol is output. When the target execution environment is a robot, path nodes are converted into robot motion commands, action nodes are converted into end effector actions, and robot control scripts are output. When the target execution environment is an automation script, the intent map is converted into a browser automation or system operation script, the drawn and circled areas are converted into selectors, and the arrow connections are converted into data flow directions.

[0083] In one embodiment, a unified intent protocol-driven multi-target heterogeneous execution compilation system is provided, the system including an intent protocol receiver, a target-aware compiler, an execution scheduler, and an execution subject adaptation layer.

[0084] The intent protocol receiver receives the intent protocol and performs a pre-translation check on the intent protocol.

[0085] The target-aware compiler adopts a plug-in architecture and has independent compiler plug-ins corresponding to the execution target. The target-aware compiler receives the intent protocol after pre-translation inspection. The compiler plug-in compiles the intent protocol into specific instructions according to the execution target type and maps the node ID in the intent protocol to the index position in the output instruction through the tracing field. The tracing field supports the precise tracing of instruction-level exceptions to the corresponding node in the intent protocol during execution.

[0086] The execution scheduler is used to register and maintain the execution subject registry, and to route instructions to the corresponding execution subject according to the edge type in the intent protocol; at the same time, it maintains the execution status object and records the execution progress of each node.

[0087] The execution entity adaptation layer is configured with an execution entity adapter, which is used to convert the unified instruction format into the native API calls of the execution entity.

[0088] It should be noted that the unified intent protocol-driven multi-target heterogeneous execution compilation system is used as the execution compilation module of the multimodal intent editing and multi-target execution compilation system based on the intent protocol.

[0089] By using the Intent Protocol (UIP) as a platform-independent Task Intermediate Representation (Task IR), a target-aware compiler converts the same Intent Protocol into corresponding executable instructions according to the semantic requirements of the target execution environment, and an execution scheduler manages the task execution of various heterogeneous execution entities, thereby decoupling the task intent from the execution implementation.

[0090] In one embodiment, the pre-translation check includes checking for ambiguity markers and integrity checks.

[0091] In one embodiment, the execution target includes at least one of LLM, Agent, drone, robot, and software automation; The independent compiler plugins corresponding to the execution targets include the LLM Prompt compiler plugin, the Agent task chain compiler plugin, the UAV flight path compiler plugin, the robot control sequence compiler plugin, and the software automation script compiler plugin. The execution entity adapters include LLM adapters, Agent adapters, drone adapters, robot adapters, and automation script adapters.

[0092] Understandably, the execution entity adaptation layer is configured with corresponding execution entity adapters for various execution entities, thereby converting the unified CompiledInstruction format into the native API calls of that execution entity. The adapters shield the underlying API differences between execution entities, so that the scheduler does not need to be aware of the specific access details of each execution entity.

[0093] In one embodiment, the compilation rules of the LLM Prompt compiler plugin include: pre-context generation, node sequence transformation, constraint embedding, condition node transformation, and output format declaration. Specifically, pre-context generation includes: traversing the Goal nodes of the intent graph, using goal.success_condition as the task target paragraph of the Prompt, and goal.priority as the priority description. Node sequence transformation includes: performing topological sorting on the intent graph, converting the action_type and parameters of each Action node into corresponding natural language instruction paragraphs according to the sorting order, using the compiler's built-in action_type to natural language description mapping dictionary. Constraint embedding includes: converting the constraints array of each node into the corresponding Prompt constraint description, embedding it after the corresponding instruction paragraph (e.g., TimeoutConstraint is converted into "Please complete within N seconds", OutputConstraint is converted into "Output length does not exceed N words"). Condition node transformation includes: converting the condition_expression of the Condition node into condition judgment instructions (e.g., "If [condition], then [true branch action]; otherwise [false branch action]"). The output format declaration includes: appending the output format requirements to the end of the Prompt, with the format determined by the llm_output_format parameter in compile_targets (the default is structured JSON).

[0094] The compilation rules of the Agent task chain compiler plugin include: tool mapping, parameter serialization, retry policy injection, conditional routing transformation, and parallel task transformation. Specifically, tool mapping involves mapping the `action_type` of each Action node to the registered tool name (`tool_name`) in the Agent framework via the tool registry; the tool registry is a configurable JSON file that supports extensions. Parameter serialization involves serializing the `parameters` dictionary of the Action node into JSON according to the tool call parameter format of the target framework. Retry policy injection involves injecting the `retry_config` (retry) field (`max_retries`, `backoff_strategy`) into the tool call configuration if the node carries a `RetryConstraint`. Conditional routing transformation involves converting Condition nodes into conditional routing nodes (Routers) of the Agent framework, with `true` / `false` branches corresponding to two tool call paths. Parallel task transformation involves converting node groups connected by parallel edges into parallel call configurations (if the target framework supports them) or sequential call configurations of the Agent framework.

[0095] The compilation rules of the UAV flight path compiler plugin include: coordinate extraction and transformation, action enumeration mapping, flight parameter injection, sequence order preservation, and automatic return-to-home point generation. Specifically, coordinate extraction and transformation involves: traversing the `coordinates` attribute of all Object nodes in the intent graph and converting planar or map coordinates into the latitude, longitude, and altitude 3D coordinate format (WGS84 coordinate system) required by the UAV control protocol. Action enumeration mapping involves: converting the `action_type` of the Action node into the action enumeration value of the target UAV SDK through the UAV action mapping table. Flight parameter injection involves: injecting the flight altitude (`altitude_constraint`), flight speed (`speed_constraint`), and hover time (`hover_duration`) from the constraint nodes into the flight parameter fields of the corresponding waypoints. Sequence order preservation involves: directly mapping the topological order of the sequence edges in the intent graph to the waypoint sequence order in the flight path task file. Automatic return-to-home point generation involves: automatically appending a return-to-home instruction to the end of the flight path if the Goal node in the intent graph has a `return_to_origin` attribute.

[0096] The compilation rules of the robot control sequence compiler plugin include: path node transformation, motion instruction mapping, velocity and acceleration injection, and collision detection marking. Specifically, path node transformation involves converting a sequence of Object nodes carrying coordinate attributes in the intent map into a target pose sequence (position + attitude quaternion) for the robot's end effector. Motion instruction mapping involves converting the `action_type` of Action nodes into robot control instructions using the robot motion mapping table. Velocity and acceleration injection involves extracting velocity constraints (`speed_constraint`) and acceleration constraints from the node's constraint attributes and injecting them into the velocity / acceleration parameter fields of the motion instructions. Collision detection marking involves inserting a collision detection call before the corresponding motion instruction if the node carries a `collision_check` constraint.

[0097] The compilation rules of the software automation script compiler plugin include: region selector generation, data flow transformation, filter constraint transformation, and exception handling injection. Specifically, region selector generation involves converting object nodes associated with painted regions in the intent map into CSS selectors or XPath expressions. Data flow transformation involves converting data flow relationships represented by arrows in the logic layer into variable passing code in the script. Filter constraint transformation involves converting filtering rules in FilterConstraint (ignoring null values / deduplication, etc.) into corresponding data cleaning code segments in the script. Exception handling injection involves converting RetryConstraint and TimeoutConstraint into try-catch-retry wrapper logic in the script.

[0098] In one embodiment, routing instructions to the corresponding execution entity based on the edge type in the intent graph includes: Sequential edges correspond to serial scheduling, and instructions are distributed sequentially according to the topology order. After the previous instruction is completed and returns a success status, the subsequent instruction is distributed. Parallel edges correspond to parallel scheduling, which simultaneously distributes all instructions within a parallel group to multiple execution entities, and concurrently waits for all parallel instructions to complete before continuing to distribute subsequent serial instructions. The triggering edge corresponds to event-driven scheduling. After the source node completes execution, the scheduler checks whether the triggering condition is met. If it is met, the scheduler distributes the instruction to the target node; otherwise, it enters a waiting state. Conditional edges correspond to conditional routes. After the conditional node is executed, the scheduler selects the correct branch or the incorrect branch to distribute subsequent instructions based on the execution result.

[0099] In one embodiment, when the execution scheduler registers multiple execution entities for the same target type, it selects the execution entity according to a load balancing strategy.

[0100] In one embodiment, the execution scheduler has a built-in exception handling mechanism: If the execution time of a node exceeds the timeout constraint setting value, the execution scheduler sends a cancellation signal to the execution subject, sets the node status to failure, and triggers a retry constraint or a false branch of the execution conditional route; After a node carrying retry constraints fails to execute, the execution scheduler automatically redistributes the instruction according to the maximum number of retries, and the retry interval adopts an exponential backoff strategy. If the current execution entity is unavailable, the execution scheduler checks whether there is a backup execution entity with matching capabilities. If so, it automatically switches to the backup execution entity for execution. When any node in a parallel edge group fails, the execution scheduler does not terminate the execution of the remaining parallel nodes by default. After the parallel group is completed, it enters the successor node in a partially completed state.

[0101] Understandably, the above exception handling mechanisms are timeout handling, retry mechanism, degraded execution, and local failure isolation, respectively.

[0102] In one embodiment, the execution scheduler supports resuming execution from any failed node while the state of the successful node remains unchanged.

[0103] In one embodiment, the execution scheduler supports automatically injecting the output fields of the predecessor node into the input parameters of the successor node's instructions to achieve data transfer between nodes.

[0104] In one embodiment, the intent protocol instance further includes a multimodal payload binding field; When the independent channel is used for multimedia upload, the input task intent data is bound to the semantic field as a multimodal payload (PayloadRef); the multimedia upload includes images, audio, video, and point clouds.

[0105] In one embodiment, when the intent protocol instance carries a multimodal payload (PayloadRef), the target-aware compiler handles cross-target payload transfer according to the following rules: When the target execution environment is LLM, the image payload is converted to base64 encoding and embedded in the multimodal Prompt; the audio payload is converted to speech recognition text and then embedded; the video payload is extracted from keyframes and then processed. When the target execution environment is Agent, the payload is passed as an input parameter for tool invocation, and format adaptation is performed using a predefined media processing tool; When the target execution environment is a robot, the point cloud payload is directly transmitted as perceptual data; the image payload is transmitted as visual perception input. When the target execution environment is an enterprise system, the payload is passed via a pre-signed URL reference.

[0106] For execution targets that do not support a certain type of payload, the target-aware compiler marks PayloadRef.unsupported (does not support a certain type of payload) = true in the compilation result and lists all downgraded or ignored payloads in the compilation summary so that the caller is aware of the information loss.

[0107] In this embodiment, media payloads are adapted to different execution targets through cross-target delivery rules, eliminating the need for users to repeatedly upload or describe them.

[0108] Reference Figure 1 In one embodiment, a task intent intermediate representation system is provided, the system including a multimodal input module, an alignment and fusion module, and an intent graph generation module; The multimodal input module collects the user's task intent from multiple independent channels and records the input timestamps; The alignment and fusion module establishes the association between different channel inputs in the time, space and semantic dimensions based on the time window, and generates a unified and fused task intent expression; The intent graph generation module converts the unified and integrated task intent expression into a directed intent graph according to predetermined node type and edge type rules. Each node in the intent graph contains a source field and an ambiguity flag field. The source field is used to record the source of the node channel, and the ambiguity flag field is used to mark unresolved referential ambiguities. The node types include object nodes for representing task objects, action nodes for representing actions to be performed, condition nodes for representing conditional decision branches, and target nodes for representing task objectives. The edge types include sequential edges, trigger edges, parallel edges, and conditional edges, and each edge type carries a source modality label. The intent graph adopts a differential update structure.

[0109] By employing a differential update structure, only the changed parts are updated, eliminating the need to regenerate the entire intent graph, significantly improving intent update efficiency and reducing update costs. By recording the source channel through the traceability field of intent graph nodes, the problem of task nodes being unable to trace the input method in existing technologies is solved, facilitating task modification and maintenance. This invention achieves a platform-independent unified expression of task intents. Through predetermined node types, edge type rules, and formal data structure specifications, it breaks down the barriers of proprietary formats of various AI orchestration platforms, allowing task intent representation to be independent of specific platforms. It can serve as a standard bridging layer between multimodal intent expression and heterogeneous execution, supporting cross-platform reuse and compiled output, significantly improving the interoperability of task intents.

[0110] In one embodiment, the intent graph adopts a differential update structure, describes the update operation of the intent graph through the RFC 6901 JSONPointer path, and records the set of change nodes and trigger source of each differential update through a version history array.

[0111] It should be noted that after the intent map is updated only by differential update, the system automatically updates the document's timestamp, recalculates the checksum, and appends this update to the version history (version_history).

[0112] In one embodiment, the intent graph has a built-in version management module that supports version rollback, version comparison, and selective recompilation.

[0113] Specifically, the version management module records each differential update through the version history (version_history) array. Version rollback involves specifying a target version_id, and the system reverse-engineers the IRPatch (differential update) operation from the version history, restoring the document to the target version state. Version comparison compares the changed_nodes and changed_edges between two version_ids, generating a node-level difference report. Selective recompilation means that when the set of changed_nodes for a given version is known, the compiler only recompiles changed nodes and their downstream nodes; the compilation results for unchanged nodes are read from the cache.

[0114] In this embodiment, through version management of intents, different versions of intent graphs can be retained, traced, and compared, and missing links in intent expression can be discovered in a timely manner, ensuring the integrity and consistency of task intent expression, and providing a reliable structured intent foundation for subsequent execution and compilation.

[0115] In one embodiment, each node of the intent graph contains a constraint array for attaching typed constraints, including timeout constraints, retry constraints, priority constraints, and output constraints.

[0116] Specifically, it also includes filter constraints, spatial constraints, speed constraints, altitude constraints, parallel flags, and custom constraints.

[0117] In one embodiment, the intent graph supports semantic equivalence interchange with JSON, binary compression, and human-readable simplified formats.

[0118] In one embodiment, the intent graph generation module includes an integrity verifier for verifying the integrity of the intent graph based on uniqueness rules, reference integrity rules, acyclicity rules, ambiguity clearing rules, and SHA256-based checksum integrity rules.

[0119] In one embodiment, the uniqueness rules include: all node IDs are unique within the document; all edge IDs are unique within the document; and there is one and only one target node. The referential integrity rules include: the source node ID and destination node ID of all edges must correspond to node IDs existing in the document node; the true branch ID and false branch ID of the condition node must be valid; and all IDs in the terminal node ID list of the target node must be valid. The acyclicity rule includes: except for branch paths guided by conditional edges, sequential edges and triggering edges in the intent graph do not form directed loops; The ambiguity clearing rule includes: maintaining the ambiguity count at 0; The SHA256-based checksum and integrity rules include: ensuring that the SHA256 checksum of the document is consistent with the document content.

[0120] In one embodiment, the update operation includes adding, removing, replacing, and moving.

[0121] In one embodiment, the system further includes a scene form template containing an array of synonyms to eliminate semantic equivalence ambiguity.

[0122] In one embodiment, the intent graph declares compile targets via an array of compile targets, each compile target containing a target type and target-specific configuration.

[0123] In one embodiment, the execution invocation module supports two scheduling modes: single-subject serial execution and multi-subject parallel execution based on parallel edges.

[0124] In one embodiment, the execution entity accesses the multimodal intent editing and execution compilation system through a unified interface.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A task intent intermediate representation system, characterized in that, The system includes a multimodal input module, an alignment and fusion module, and an intent map generation module; The multimodal input module collects the user's task intent from multiple independent channels and records the input timestamps; The alignment and fusion module establishes the association between different channel inputs in the time, space and semantic dimensions based on the time window, and generates a unified and fused task intent expression; The intent graph generation module converts the unified and integrated task intent expression into a directed intent graph according to predetermined node type and edge type rules. Each node in the intent graph contains a source field and an ambiguity flag field. The source field is used to record the source of the node channel, and the ambiguity flag field is used to mark unresolved referential ambiguities. The node types include object nodes for representing task objects, action nodes for representing actions to be performed, condition nodes for representing conditional decision branches, and target nodes for representing task objectives. The edge types include sequential edges, trigger edges, parallel edges, and conditional edges, and each edge type carries a source modality label. The intent graph adopts a differential update structure.

2. The task intent intermediate representation system according to claim 1, characterized in that, The intent graph adopts a differential update structure, and the update operation of the intent graph is described by the RFC 6901 JSONPointer path. The version history array records the set of change nodes and the trigger source for each differential update.

3. The task intent intermediate representation system according to claim 2, characterized in that, The intent map has a built-in version management module that supports version rollback, version comparison, and selective recompilation.

4. The task intent intermediate representation system according to claim 1, characterized in that, Each node in the intent graph contains a constraint array for attaching typed constraints, which include timeout constraints, retry constraints, priority constraints, and output constraints.

5. The task intent intermediate representation system according to claim 1, characterized in that, The intent graph supports semantic equivalence interchange with JSON, binary compression, and human-readable simplified formats.

6. The task intent intermediate representation system according to claim 1, characterized in that, The intent graph generation module includes an integrity verifier, which is used to verify the integrity of the intent graph according to the uniqueness rule, the reference integrity rule, the acyclicity rule, the ambiguity clearing rule, and the SHA256-based checksum integrity rule.

7. The task intent intermediate representation system according to claim 6, characterized in that, The uniqueness rules include: all node IDs are unique within the document; all edge IDs are unique within the document; and there is one and only one target node. The referential integrity rules include: the source node ID and destination node ID of all edges must correspond to node IDs existing in the document node; the true branch ID and false branch ID of the condition node must be valid; and all IDs in the terminal node ID list of the target node must be valid. The acyclicity rule includes: except for branch paths guided by conditional edges, sequential edges and triggering edges in the intent graph do not form directed loops; The ambiguity clearing rule includes: maintaining the ambiguity count at 0; The SHA256-based checksum and integrity rules include: ensuring that the SHA256 checksum of the document is consistent with the document content.

8. The task intent intermediate representation system according to claim 2, characterized in that, The update operations include adding, removing, replacing, and moving.

9. The task intent intermediate representation system according to claim 1, characterized in that, The system also includes a scene form template, which contains an array of synonyms to eliminate semantic equivalence ambiguity.

10. The task intent intermediate representation system according to claim 1, characterized in that, The intent graph declares compilation targets through an array of compilation targets, each containing a target type and a target-specific configuration.

Images