Complex instruction task planning execution interaction method based on large model

Abstract

The invention discloses a complex instruction task planning execution interaction method based on a large model, and relates to the field of artificial intelligence and human-computer interaction. The method comprises the following steps: performing wake-up detection and instruction receiving, and obtaining a user text / voice instruction; analyzing the instruction based on a large language model and managing a dialogue state; recognizing UI elements to construct visual scene information; mapping a system and an application software API; analyzing instruction dependence to generate an optimization execution plan; and executing across application software and feeding back a result. According to the method, multiple technologies are fused to realize end-to-end automation, instruction understanding and environment adaptability are improved, multiple operating systems are supported, the method is suitable for scenes such as intelligent desktop assistants, and operation efficiency is improved.

Description

Technical Field

[0002] This invention relates to the fields of artificial intelligence and natural human-computer interaction technology, specifically to a method for planning and automating complex instruction tasks based on a large language model, which is particularly suitable for driving cross-platform and cross-application software system-level automated task execution through natural language instructions. Background Technology

[0004] With the rapid development of artificial intelligence technology, users' demand for intelligent assistants has expanded from simple information queries to automated systems capable of performing complex, multi-step tasks. Traditional human-computer interaction methods (such as command lines and graphical interface selection) suffer from pain points such as cumbersome operation, high learning costs, and low automation when faced with complex cross-application, multi-step tasks such as "creating a PPT of last month's sales data and sending it to Mr. Wang via email".

[0005] In existing technologies, rule-based automation scripts (such as RPA robots) can automate some processes, but their flexibility and generalization capabilities are insufficient, making it difficult to understand the diversity and contextual relationships of natural language instructions. While current dialogue systems based on large language models have made significant progress in semantic understanding, they are usually limited to text generation and simple API calls, lacking the ability to visually perceive graphical interfaces (UIs), dynamically map system and software APIs, and plan and optimize the execution of multi-step tasks. Consequently, they cannot achieve end-to-end closed-loop execution from natural language instructions to actual device operation.

[0006] Therefore, there is a technical problem in the existing technology: how to accurately understand, decompose, plan and transform a user's complex natural language command into a series of specific cross-platform and cross-application operations that can be reliably executed in a real operating system environment, so as to achieve true end-to-end task automation. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of existing technologies and provide a complex instruction task planning and execution interaction method based on a large model, realizing end-to-end conversion from user natural language instructions to automated execution across multiple platforms and applications.

[0009] The technical solution adopted in this invention is summarized as follows:

[0010] A method for planning and executing complex instruction tasks based on a large model includes the following steps:

[0011] Step S1: Perform wake-up detection and instruction reception operations to obtain task instructions input by the user in text or voice form. Step S2: Use a large language model to semantically parse the task instructions and match them with a set of evolvable predefined task intent templates to obtain a structured task instruction description. When the matched intent template contains a verified instruction sequence, directly use that instruction sequence as the execution plan. Otherwise, proceed to steps S3 to S6. Step S3: Obtain the system screen image, identify UI elements using a visual recognition model, and construct an accessibility tree. Based on the accessibility tree, use a recursive dependency inference algorithm to map the structured task instruction description into a sequence of UI operation actions. Step S4: Map system-level operations involved in the structured task instruction description to a preset system API instruction set stored in vectorized form to generate system API call instructions. Step S5: Map application software-level operations involved in the structured task instruction description to a preset application software API instruction set stored in vectorized form to generate application software API call instructions. Step S6: Integrate the UI operation sequence, system API call instructions, and application software API call instructions to perform instruction dependency analysis and optimization, and generate a structured execution plan. Step S7: Execute the instructions according to the structured execution plan, and provide feedback on the execution results through the cross-platform device driver module.

[0012] Further, step S2 specifically includes: Step S21: Perform semantic parsing and word sense disambiguation on the task instructions, extract core actions, objects, parameters, and modifiers, and construct a semantic parsing tree. Step S22: Match the semantic parsing tree with a predefined task intent template, use Sentence-BERT encoding and cosine similarity calculation to identify the task intent and output a structured task instruction description. The predefined task intent template allows users to confirm and update successfully executed instruction sequences. Step S23: Extract task parameters based on the semantic parsing tree, perform parameter normalization and integrity verification, and output parameterized JSON instructions. Step S24: Construct a memory graph through a graph neural network to manage multi-turn dialogue states, achieving referential resolution and contextual connection. Step S25: Based on parameter integrity scoring and confidence, trigger interactive intent clarification to generate the final structured task instruction description.

[0013] Further, the feature is that step S3 specifically includes: Step S31: Semantic segmentation of the system screen image, identification of interface layout and functional components, and establishment of relative position encoding of UI elements. Step S32: Identification of the boundaries, categories, and text content of interactive UI elements through a combination of visual feature detection and screen text extraction, and extraction of element attributes including position, state, and functional description, to construct the accessibility tree. Step S33: Matching task instructions with visual elements in the accessibility tree and parsing the spatial reference relationships in the instructions. Step S34: Based on a recursive dependency reasoning algorithm, converting instructions into a cross-application, cross-level UI operation action sequence, and configuring abnormal state handling logic for this sequence.

[0014] Preferably, the recursive dependency reasoning algorithm in step S34 specifically involves: traversing the accessibility tree and recursively searching for operable UI element paths from the current state to the target state. When the current node is invisible or inoperable, the operation of its dependent parent node is inferred, and corresponding operation actions are supplemented. A complete sequence of UI operation actions is generated, and during execution, the expected state is checked and compared with the actual interface state. If the expected state is not met, backtracking, retrying, or strategy adjustment is triggered.

[0015] Furthermore, the system API instruction set and application software API instruction set in steps S4 and S5 are hierarchical, dynamically registerable instruction sets that support instruction mapping through semantic classification and vector retrieval, and are encapsulated into callable API functions.

[0016] Preferably, both the system API instruction set and the application software API instruction set are stored in a vector database in vectorized form, supporting retrieval based on semantic similarity. The instruction mapping process includes: determining the category of the user's intent using a semantic classifier, encoding the user instruction into a vector, retrieving the matching instruction definition from the vector database, and completing parameter adaptation and call encapsulation through an instruction wrapper.

[0017] Further, step S6 specifically includes: Step S61: Analyze the dependencies between instructions, construct an instruction dependency graph, and detect circular dependencies through topological sorting. Step S62: Identify the critical path and calculate the earliest and latest start times of each instruction node. Step S63: Merge redundant instructions, identify a subset of instructions that can be executed in parallel, and formulate a parallel execution strategy. Step S64: Conduct sandbox simulation testing, assess execution risks, and generate a risk assessment report. Step S65: Generate a structured execution plan that includes execution order, parameter configuration, dependency conditions, and verification nodes.

[0018] Further, step S7 specifically includes: Step S71: Parsing and executing UI operations, system API calls, and application software API calls in the structured execution plan through the cross-platform device driver module. Step S72: Monitoring the execution results of each step in real time and comparing them with the expected state; if they do not meet expectations, triggering exception handling or replanning. Step S73: Collecting the final execution results and converting them into a user-friendly format for feedback.

[0019] Furthermore, the cross-platform device driver module supports Windows, macOS, Linux, iOS, and Android operating systems, provides a unified action interface including mouse movement, clicking, dragging, keyboard input, and touch screen operation, and has a visual calibration function to convert logical coordinates into the physical coordinates of the current device.

[0020] Meanwhile, a complex instruction task planning and execution interaction system based on a large model is provided for implementing the above method, characterized by comprising:

[0021] The voice processing module is used for voice acquisition, conversion, and wake-up detection.

[0022] The semantic understanding module is used for instruction parsing, intent matching, parameter extraction, and dialogue management.

[0023] The visual recognition module is used for screen parsing, UI element recognition, and accessibility tree construction.

[0024] The API integration module is used for the registration, vectorized storage, mapping, and call management of system and application software APIs.

[0025] The interaction planning module is used for UI action planning, instruction dependency analysis, parallel optimization, and execution plan generation.

[0026] The operation driver module is used to drive peripheral devices to perform operations and monitor and provide feedback results across platforms.

[0027] Beneficial effects

[0028] Compared with the prior art, the present invention has the following significant advantages:

[0029] 1. Achieves true end-to-end automation of complex tasks: This invention deeply integrates natural language understanding, computer vision, software automation and system integration technologies, and can automatically transform a user's complex natural language command (such as "save last month's sales summary table in Excel into a PDF and then send it to Mr. Wang via DingTalk") into a series of precise cross-application UI operations and API calls, and execute them reliably, greatly improving the level and efficiency of automation in office and system operations.

[0030] 2. Possesses powerful capabilities in understanding complex instructions and managing context: By combining deep semantic parsing of large language models, evolvable intent template matching, and dialogue state management based on graph neural networks, the system can not only accurately understand the current instruction, but also effectively handle referential resolution, multi-turn dialogue clarification, and long-range contextual dependencies, making the interaction more natural and intelligent.

[0031] 3. Innovative integration of visual perception and planning reasoning: By constructing an accessibility tree (VDOM) to perform structured modeling of the graphical interface and adopting a recursive dependency reasoning algorithm, the system can "understand" the interface layout, intelligently plan the operation path from the current state to the target state, and handle non-intuitive interaction scenarios such as interface occlusion and dynamic pop-ups. It has strong environmental adaptability and self-healing ability.

[0032] 4. A flexible and scalable API integration architecture has been built: By establishing a layered (system-level, application-level), registerable, and vectorized API instruction set, the system can dynamically adapt to different operating system versions and software environments, and abstract complex multi-step UI operations into concise API calls, which greatly improves task execution efficiency and cross-platform compatibility.

[0033] 5. Provides global optimization and risk-controllable execution assurance: By performing instruction dependency analysis, critical path identification, sandbox simulation testing and risk assessment before execution, the system can optimize the execution order, maximize parallelism, predict and avoid potential failure points, thereby generating a robust, efficient and reliable structured execution plan, which significantly improves the probability of successful execution of complex tasks on the first attempt.

[0034] In summary, this invention provides a new paradigm of human-computer interaction that is highly intelligent, automated, robust, and reliable. It is particularly suitable for a wide range of scenarios such as intelligent desktop assistants, cross-software process automation, and barrier-free assisted operation, and has significant practical value and broad application prospects. Attached Figure Description

[0036] Figure 1 The present invention provides a flowchart illustrating a complex instruction task planning and execution interaction method based on a large model. Detailed implementation method:

[0037] This invention provides a method for fully implementing software or system interactions based on UI visual recognition, driven by voice or text, to complete complex tasks. The method achieves end-to-end conversion of task commands into device interaction actions, ensuring accuracy of command parsing, efficiency of interaction execution, and compatibility across application scenarios. Specific implementation details are as follows:

[0038] System Architecture Overview:

[0039] The system of this invention consists of six core modules. Each module achieves data interoperability and collaborative work through standardized interfaces, ensuring traceability and reproducibility throughout the entire process. Specifically, it includes:

[0040] 1. Voice Processing Module: Responsible for voice acquisition, STT conversion, and wake word detection.

[0041] 2. Semantic Understanding Module: Based on a large model, it enables complex instruction parsing, intent recognition, parameter extraction, and cross-turn dialogue management.

[0042] 3. Visual Recognition Module: Responsible for screen scene segmentation, UI element detection, interactive area recognition, and interaction mode determination.

[0043] 4. API Integration Module: Provides registration, adaptation, and call management for operating system APIs and application software APIs.

[0044] 5. Interaction Planning Module: Transforms semantic instructions into UI operation sequences and optimizes the execution path.

[0045] 6. Operation driver module: Drives peripheral devices such as keyboard and mouse to perform interactive actions and implement commands.

[0046] The data flow relationship between the modules is as follows: speech processing module → semantic understanding module → API integration module / visual recognition module → interaction planning module → operation driving module.

[0047] 1. Hardware requirements: CPU (Intel i7 or above or AMD Ryzen 7 or above), GPU (NVIDIA GTX3090 or above, VRAM ≥24GB), RAM ≥32GB, Storage ≥100GB, Microphone array (≥4 channels, sampling rate ≥16kHz).

[0048] 2. Software Environment Requirements: Operating System (Windows 10 21H2 and above, MacOS 12 and above, iOS 13 and above, Android 7.0 and above), Office Software (Office 2019 and above or 365), Python Version (3.8-3.10), Dependency Library List (PyTorch 1.12.0+, TensorFlow 2.8.0+, YOLO 11+, GOT-OCR 2.0+, Sentence-BERT 2.2.2+, etc.).

[0049] A complex instruction task planning and execution interaction method based on a large model, such as... Figure 1 As shown, the specific steps include the following:

[0050] Step S1: Wake-up Detection and Command Reception

[0051] This invention employs two methods to deliver specific task objectives: one is to provide a text input box to receive text task commands from the user; the other is to provide voice input task commands. For voice input, the technology used in this invention is as follows:

[0052] The sound recording system acquires human voices from the surrounding environment. It supports effective sound pickup within 3 meters, with a sampling rate of 16kHz and a bit depth of 16bit. The acquired human voices are converted into text using a speech-to-text (STT) model. The STT model can be flexibly selected based on the deployment environment and business scenario. For deployment on small-scale devices, the Vosk model is recommended, with an approximate 80MB of data suitable for Mandarin Chinese, achieving an accuracy of ≥96.7%, a recognition time of ≤300ms, and no grammatical errors. If considering scenarios such as dialect recognition, FunSAR is selected, achieving an accuracy of ≥97.8%, a recognition latency of ≤300ms, and no grammatical errors.

[0053] A Bayesian classifier is used to determine whether the text contains a preset wake-up keyword. The keyword detection probability model is as follows:

[0054]

[0055] Where W is the wake word and T is the text. This represents the probability of observing text T when the wake word W is present. This represents the prior probability that the wake word W will occur naturally in any text. This represents the total probability that text T appears among all possible texts.

[0056] The prior probability of the preset wake word can be adjusted by the user. The default prior probability of the wake word "helper" is P(W)=0.001. Users can add / delete wake words, with a maximum of 5 wake words that can be set, and their priority can be sorted. When P(T|W) reaches 98% or more and P(W|T) exceeds the wake-up threshold (the wake-up threshold is configurable, with a default of 0.95), the system is woken up and activated, and the complete text is forwarded to the large language model (using the DeepSeek-R1 model, with a context window length of 128k tokens) for further processing. If the text does not contain a wake-up keyword, the system continues to wait for wake-up.

[0057] Voice acquisition anomaly handling: When the microphone malfunctions (no audio input detected), a pop-up window will display "No microphone signal detected, please check the device connection," and the system will retry after 3 seconds, with a maximum of 2 retries; when STT conversion fails (recognition accuracy < 80%), the system will display "The voice signal is unclear, please speak again," and voice acquisition will be restarted.

[0058] Step S2: Complex Instruction Understanding and Intent Parsing

[0059] After receiving the instruction provided in step S1, the system uses a large model to understand the user's natural language instruction intent, and performs instruction recognition and classification. It also supports robust understanding of multi-turn interactions.

[0060] Step S21: Semantic Deep Parsing and Disambiguation

[0061] 1. In this step, a dataset of 5000 data points (including 700 negative samples) is used, consisting of three parts: Query (raw input), Context (environmental context, including the currently active app, time, address, currently selected content, connected hardware (Bluetooth, WiFi), system parameters (brightness, volume), etc.), and Standard Output (parsed results). This dataset covers various task types, including daily office work, life tasks, mobile app operations, and system configurations. Then, a BERT model based on the Transformer architecture is used (specifically, the RoBERTa-wwm-ext pre-trained model is used, with fine-tuning configured as batch size=32, epoch=10, learning rate 2e-5, optimizer AdamW, and loss function [missing information]. First, by analyzing the sentence structure, a complete semantic parse tree is constructed (the node types of the semantic parse tree include: core action nodes, object nodes, parameter nodes, and modifier nodes; the hierarchical relationship determination rule is: the core action node is the root node, the object node is the first-level child node, and the parameter node and modifier node are the child nodes of the object node). The core action (main_action) and key argument roles (preliminary actor, target, etc.) are identified, and the semantic completeness reaches 95.8%. Word Sense Disambiguation (WSD) is performed in combination with domain knowledge, and the disambiguation recall rate reaches 93.5%. BERT-NER is used to extract key parameters and objects (NER annotation accuracy ≥95.8%, supported entity types include: department, time, file type, personnel, topic, remarks, etc.), and a standardized structured (JSON) representation of the core action, parameters, and modifiers is output, providing a parsing framework and preliminary semantic elements for S22 intent classification and S23 parameter extraction.

[0062] When constructing the semantic parsing tree, cosine similarity is used to address the problem of polysemy.

[0063]

[0064] The word whose meaning is closest to the context vector, i.e. the word with the highest similarity, is selected as the parsing result; The word vector representing the target polysemous word (output from the last hidden layer of the BERT model, dimension 768). The vector representing the current context (the average of the word vectors of the five words before and after the polysemous word, dimension 768). Represents the length of the vector. Indicates the length of the context vector.

[0065] Step S22: Intent recognition and classification

[0066] By using an evolutionary predefined task intent template, we achieve efficient parsing of complex instructions, multi-turn dialogue clarification, and preliminary operation planning, significantly improving the accuracy and automation of task parsing. In this step, we establish predefined task intent templates (which can be updated by user confirmation of successfully executed sequences or by self-evolution using algorithms such as success rate) for more efficient and accurate intent recognition and execution planning. If, after intent recognition and classification, an intent template is matched and contains predefined execution instructions, then after step S23 or S25, the process proceeds to step S7 for execution of the optimal template instructions. If no predefined execution instructions are provided, then after generating preliminary task instructions in step S23 or S25, the process proceeds to steps S3, S4, S5, etc.

[0067] A typical task intent template includes template ID, intent name, intent category, domain, priority level, intent semantic description, parameter slot, instruction sequence type, predefined instruction sequence (the verified optimal instruction sequence for this task type), and trigger words.

[0068] Typical template examples are as follows:

[0069] {

[0070] "template_id": "tpl_office_001",

[0071] "intent_name": "create_ppt_from_topic",

[0072] "category": "SOFTWARE_APP",

[0073] "domain": "OFFICE_AUTOMATION",

[0074] "priority": 10,

[0075] "description": "Automatically generate a multi-page PPT presentation based on the theme provided by the user".

[0076] "slots": { / / Slot (parameter) definition

[0077] "topic": { "type": "string", "required": true, "prompt": "What is the theme of this PPT?"},

[0078] "synopsis": { "type": "string", "required": true, "prompt": "What is the outline of the PPT?"},

[0079] "pages": { "type": "integer", "default": 5},

[0080] "style": { "type": "string", "enum": ["business", "minimalist", "creative"], "default": "business"}

[0081] },

[0082] "logic_type": "STATIC_SEQUENCE", / / Logical type: whether it is a static sequence or dynamically generated sequence.

[0083] "predefined_sequence": [ / / Predefined sequence of instructions]

[0084] { "step": 1, "action": "open_software", "params": { "app": "PowerPoint"}},

[0085] { "step": 2, "action": "apply_template", "params": { "style": "{{style}}"}},

[0086] { "step": 3, "action": "generate_content_via_llm", "params": { "context": "{{topic},{synopsis}}"}},

[0087] { "step": 4, "action": "render_slides", "params": { "data": "$PREV_RESULT"}}

[0088] ],

[0089] "trigger_phrases": [ / / Example sentences used for vector retrieval or training]

[0090] "Help me create a slideshow about the development of artificial intelligence."

[0091] "Create a PowerPoint presentation with the theme of last year's sales summary."

[0092] Create a presentation ]

[0094] }

[0095] The semantic tree output from S21 is matched with predefined intent templates (encoded into semantic vectors using the Sentence-BERT algorithm, and then cosine similarity is used for judgment), and composite intents are identified and separated (composite intent splitting rules: based on the independence of subtrees in the semantic tree, when the core actions corresponding to the subtrees have no dependency relationship, they are split into independent simple intents; when there is a dependency relationship, the composite intent structure is retained and the dependency order is marked). The probability distribution is calculated using the Softmax algorithm + threshold segmentation (the threshold is 0.8 by default, but can be configured) to obtain the intent confidence distribution. High-confidence simple intents are executed, while composite / low-confidence intents are verified. Finally, the intent classification results and confidence are output. Intents with the highest probability below the threshold (e.g., 0.8) are triggered for S25 calibration. At the same time, the identified intents are also used to guide the parameter extraction range of S23.

[0096] In the Sentence-BERT algorithm encoding of the S21 semantic tree, to improve noise resistance and similarity calculation, SBERT is used to obtain more stable sentence vectors through mean pooling, compared to the original BERT directly taking the [CLS] vector. Furthermore, a triple loss function is used to optimize the semantic space during training (using the pre-trained model bge-small-zh-v1.5 as the base model for Sentence-BERT).

[0097] in Anchor sentence, Positive example sentence, Negative example sentence The L2 norm (Euclidean distance) ε of the vector represents the boundary margin, which is generally set to 0.5. By minimizing this loss, the model can learn a more discriminative semantic vector representation, thereby improving the accuracy of subsequent intent matching.

[0098] The similarity calculation formula is as follows:

[0099]

[0100] Where A is the Sentence-BERT encoded sentence vector of the user input command, and B is the Sentence-BERT encoded sentence vector of the predefined intent template.

[0101] Step S23: Parameter Structure Extraction and Filling

[0102] Based on a semantic framework, Conditional Random Fields (CRF) are used to accurately extract parameters (time, location, value, amount, etc.). First, a BIO (beginning, middle, or end of an entity, non-entity part) tagging system is constructed for different scenarios, including entity categories (e.g., file, person / nickname, application name, time / period, quantity / quantifier, operation target / object), tag names (e.g., FILE, Pepole, APP, TIME, VAL, OBJ, etc.), and examples (e.g., sales report.xls, Mr. Wang, DingTalk, tomorrow at 8 AM, 200 yuan, brightness, etc.). Then, a dataset of 8000 entries is collected and labeled. A BERT-CRF model (batch size=32, epoch=15, learning rate 3e-3, Max Seq Length 63, optimizer AdamW, accuracy 97.2%, F1-Score) is fine-tuned and trained. (94.1%, slot matching rate 92.4%) Finally, the corresponding parameters are extracted and standardized (e.g., time parameters are uniformly formatted as "year-quarter-month", personnel parameters are uniformly formatted as "name + position", and department parameters are uniformly matched with the enterprise organizational structure library). Then, the completeness of the parameters is verified by using a rule engine and slot filling detection (the rule engine includes 5 types of rule algorithms: mandatory parameter validation rules, parameter format legality validation rules, parameter logic feasibility validation rules, boundary value detection rules, and domain rationality rules), and missing items are marked. Finally, parameterized JSON instructions are output, which include structured parameters (with type, value, missing status flag, and confidence level). If a parameter is marked as missing, the intent clarification process in step S25 will be triggered.

[0103] Step S24: Cross-round dialogue state management

[0104] By tracking and establishing multi-turn dialogue history (using a Graph Neural Network (GNN) to construct a memory graph, the node structure of the GNN includes: dialogue turn ID, entity ID, entity type, action description, timestamp, and associated node ID; the edge structure includes: entity-action association edges, action-action dependency edges, and entity-entity association edges), and performing referential resolution (using an attention-based referential resolution model, with inputs being the dialogue history text and the current instruction text, and outputting a referential relationship matrix, achieving a resolution accuracy of ≥90%), the system identifies topic transitions and contextual connections (through semantic vector distance, entity coverage and the emergence of new entities, intent domain jumps, graph attention decay, and interactive explicit judgment, etc.). The logic gate determines the topic shift, thereby maintaining the entity reference, historical behavior, and topic continuity of the current dialogue, providing dialogue context information (such as pronoun referents) for S21-S23, assisting in understanding long-range dependencies, and ensuring the coherence of intentions across rounds.

[0105] The GNN memory graph update mechanism is as follows: When a new dialogue round is added, nodes and associated edges are automatically added; after the dialogue ends, core entities and action nodes are retained, and unassociated nodes are eliminated (elimination rule: nodes without associated edges have not been visited for more than 5 dialogue rounds).

[0106] Step S25: Interactive Intent Refinement

[0107] By receiving the parameter missing flag from S23 and the confidence result from S22, the completeness / clarity of the instruction is evaluated (completeness score = number of filled parameters / total number of parameters × 100%; clarity score = mean confidence score of each parameter × 100%). This can trigger re-parsing (S21) or parameter supplementation (S23), forming a self-correcting loop. Based on the confidence score, it is determined whether to trigger clarification interaction (triggered when parameter confidence score < 0.9 or normalization status is "pending clarification"). If interactive clarification is required, the clarification priority of the fuzzy parameters is calculated based on information entropy (the higher the information entropy, the higher the priority), generating a high-information clarification question.

[0108] For interactive clarification strategies generated for missing / fuzzy parameters, the context of S24 is used to ensure consistency of the issue, and a user-friendly confirmation prompt (clarification of the issue or confirmation instruction) is generated for the user through the large model.

[0109] The intention is to clarify the termination rules: If the user fails to provide the required parameters, a prompt will be made every 10 seconds, up to a maximum of 2 times. If the user still fails to provide the parameters, the task will be terminated and the message "Necessary parameters not obtained, the task has been terminated, and you can re-initiate the command" will be displayed.

[0110] The algorithm for calculating fuzzy parameters based on information entropy is as follows:

[0111] Where H represents the degree of uncertainty or ambiguity of a certain parameter (such as time, location, etc.) in the instruction (H≥1.0 is considered high ambiguity, 0.5≤H<1.0 is considered medium ambiguity, and H<0.5 is considered low ambiguity), P(x) represents the probability of a certain specific value (x) occurring, and x represents the specific parameter value (based on the possible parameter values ​​statistically determined from the company's historical data).

[0112] Finally, this step outputs a complete description of the final execution manifest instructions in JSON format, as shown below:

[0113] {

[0114] "task_metadata": {

[0115] "session_id": "job_hybrid_2025_001",

[0116] "description": "Cross-application sales analysis and reporting",

[0117] "priority": "high"

[0118] },

[0119] "instruction_sequence": [

[0120] {

[0121] "step": 1,

[0122] "module": "SOFTWARE_APP",

[0123] "tool": "excel_api",

[0124] "action": "extract_and_summarize",

[0125] "params": {

[0126] "file": "Q3_Sales.xlsx",

[0127] "sheet": "Monthly_Data",

[0128] "range": "A1:E500"

[0129] },

[0130] "output_var": "sales_summary

[0131] },

[0132] {

[0133] "step": 2,

[0134] "module": "UI_VISUAL",

[0135] "tool": "screen_controller",

[0136] "action": "launch_and_locate",

[0137] "params": {

[0138] "app_name": "PowerPoint",

[0139] "wait_for": "window_visible",

[0140] "anchor_text": "Blank presentation"

[0141] }

[0142] },

[0143] {

[0144] "step": 3,

[0145] "module": "SOFTWARE_APP",

[0146] "tool": "ppt_api",

[0147] "action": "insert_slides_with_data",

[0148] "params": {

[0149] "data ": "$sales_summary",

[0150] "template": "Standard_Analysis.potx"

[0151] }

[0152] },

[0153] {

[0154] "step": 4,

[0155] "module": "UI_VISUAL",

[0156] "tool": "visual_click",

[0157] Comment: "Ensure aesthetically pleasing typography by applying enterprise-specific design styles through UI applications."

[0158] "params": {

[0159] "target_element": "Design tabs",

[0160] "offset": [0, 0],

[0161] "action_type": "click"

[0162] },

[0163] "verification": {

[0164] "type": "pixel_match",

[0165] "target_area": ​​"top_ribbon",

[0166] "expected_state": "design_tab_active"

[0167] }

[0168] },

[0169] {

[0170] "step": 5,

[0171] "module": "UI_VISUAL",

[0172] "tool": "visual_type",

[0173] "comment": "Simulating actions in the email attachment bar",

[0174] "params": {

[0175] "find_element": {

[0176] "image_ref": "outlook_attachment_icon.png",

[0177] "text_ref": "Attachment"

[0178] },

[0179] "action": "drag_and_drop",

[0180] "source_path": "$ppt_file_path"

[0181] }

[0182] },

[0183] {

[0184] "step": 6,

[0185] "module": "UI_VISUAL",

[0186] "tool": "visual_click",

[0187] Comment: Final security check: Simulate clicking the send button.

[0188] "params": {

[0189] "target_element": "Send",

[0190] "color_hint": "blue",

[0191] "action_type": "double_click"

[0192] },

[0193] "post_condition": {

[0194] "check": "window_closed",

[0195] "app": "Outlook"

[0196] }

[0197] }

[0198] ],

[0199] "error_handling": {

[0200] "on_ui_failure": "capture_screenshot_and_replan",

[0201] "on_api_failure": "rollback_to_last_safe_state"

[0202] }

[0203] }

[0204] Step S3: UI Element Recognition and Interaction Pattern Determination

[0205] The system acquires and parses the current system screen image, uses the model to identify the operable elements of the system, and generates a UI operation sequence.

[0206] Step S31: Perform scene segmentation and semantic understanding

[0207] Identify key visual elements, construct a hierarchical structure of scene elements including the relationship between containers and content, analyze the spatial relationship between interface layout and functional components, perform semantic segmentation on screen images (using a fine-tuned SegFormer semantic segmentation model, with a training set of approximately 4000 labeled samples from various software), uniformly scale the input image resolution to 512×512, and achieve a semantic segmentation mIoU ≥ 0.94), and establish relative position encoding for UI elements: Used to parse spatial descriptions such as "top right corner". Where w and h are the total width and total height of the screen resolution. This represents the horizontal coordinate of the top-left corner of element A. This represents the vertical coordinate of the top-left corner of element A. Finally, the model outputs scene element labels (such as navigation, desktop, window, menu bar, toolbar, taskbar, pop-up, workspace, table, sidebar, etc.) and a list of coordinates.

[0208] UI element recognition exception handling: When the recognition of interactive elements fails due to interface occlusion (confidence < 0.75), the screenshot area is automatically adjusted (expanded by 10%) and re-recognized, with a maximum of 3 retries; if it still fails, the message "The interface is occluded and the interactive element cannot be recognized. Please clear the interface and try again" will be displayed.

[0209] Step S32: UI Interaction Element Recognition and Structured Modeling

[0210] It identifies and categorizes UI interaction elements (buttons, text boxes, menus, etc., supporting 20 common UI element categories with a categorization accuracy of ≥94%), extracts element attribute information (position, state, function description, size, color, text content), constructs an element accessibility tree, supports the mapping relationship between instructions and operations, records element state changes, and supports dynamic scene understanding.

[0211] Step S321: Interactive element detection and semantic parsing

[0212] A fine-tuned visual detection model (based on the YOLO 11 architecture, with a fine-tuned dataset containing 50,000 UI screenshots from system interfaces, office software (Excel, PowerPoint, Outlook, etc.), browsers, various common web pages, and mobile apps, covering different versions, resolutions, and operating states; annotation information includes element bounding boxes, category labels, and interaction modes; training parameters: imgsz=1280, mosaic enabled, close_mosaic=10, rect=True, IOU 0.7, initial learning rate 1e-3, epoch=150; accuracy 98.8%, F1-Score 0.97) works in conjunction with an OCR module (using GOT-OCR, supporting Chinese and English recognition, with text recognition accuracy ≥98%) to extract interactive elements from screenshots. The specific process is as follows:

[0213] 1. The visual inspection model outputs the bounding box coordinates. (in This represents the x-coordinate of the left side of the bounding box. The y-coordinate of the top of the bounding box. This represents the x-coordinate of the right side of the bounding box. The detection results with a confidence score ≥ 0.8 are retained, including the y-coordinate of the bottom of the bounding box and the category label (such as button, text box).

[0214] 2. Use the OCR module to extract the text content within the bounding box, filtering out non-interactive static text that has no text and no obvious interactive features (such as no border or color difference);

[0215] 3. Use IoU (Intersection over Union) overlap determination (threshold set to 0.9) to remove duplicates from the detection results and retain the element with the highest confidence.

[0216] 4. Interactive Area Determination: Visual features (whether there is a border, whether it is a high-contrast color, whether there is a state change indicator), text features (whether it contains operation-related words, such as "open", "save", "send"), and state features (whether it is clickable, can be entered, and can be dragged) are combined and weighted (visual feature weight 0.4, text feature weight 0.3, state feature weight 0.3) to calculate the interactive confidence score. Areas with a confidence score ≥ 0.75 are determined to be interactive areas.

[0217] 5. Interaction Pattern Recognition: An interaction pattern rule base is established based on element category, text content, and visual features (the rule base contains 3 core interaction patterns: click type (buttons, menu options, icons, rule: category is button / menu / icon + text contains operation words or no text), input type (text box, rich text area, rule: category is text box + presence of input cursor or blank area), drag type (scroll bar, slider, file icon, rule: category is scroll bar / slider + presence of draggable icon or size ratio conforms to draggable component characteristics)). The interaction pattern (interaction action) of the element is determined by matching the rules, with a matching accuracy of ≥92%.

[0218] 6. The semantic description of each element is generated by CLIP-ViT (a contrastive language-image pre-trained model, using the vit-base-patch16-224 pre-trained model, with the first 10 layers frozen during fine-tuning; the training dataset uses our own pair of "UI element screenshot-semantic description" samples (approximately 30,000) labeled for complex UI scenarios for domain adaptation training, with a learning rate of 5e-6, WeightDecay of 0.1, BatchSize=256, epoch=15, Zero-shot Accuracy of 91.2, and Image-TextRetrieval=87%). For example, the icon is parsed as a "search function button" or a "file download menu item", and its interaction state (such as disabled / activated, determined by the grayscale value: grayscale value ≤128 indicates disabled, otherwise it is activated) and operation type (click / drag / input) are inferred. Finally, a list of the icon's coordinates, category, function, interaction state, operation method, etc., is returned.

[0219] Step S322: Interface Structure Modeling

[0220] An accessibility tree V-Dom, which includes hierarchical relationships and spatial features, is constructed using a heuristic algorithm based on spatial inclusion relationships. Nodes store the following information:

[0221] 1) Normalize coordinates, converting absolute coordinates into relative values. Supports cross-resolution adaptation. Indicates the horizontal position of the element's center on the screen. Indicates the position in the vertical direction;

[0222] 2) Spatial relationship encoding: Calculate the containment relationship (e.g., "icons in the toolbar", determined by the parent node's bounding box containing the child node's bounding box) and the adjacency relationship (e.g., "checkboxes on the left", determined by the horizontal / vertical distance being less than 1.5 times the element's width / height) based on the element's position.

[0223] 3) Dynamic state tracking records the history of element state changes (such as the content update after text box input). Through continuous frames of SSIM (structural similarity index, threshold set to 0.95, SSIM < 0.95 is judged as local interface update), structural similarity is used to detect the local update area of ​​the interface; at the same time, the triggering conditions of element state changes are recorded (such as button color change after clicking, text box content change after input).

[0224] Step S33: Matching instructions with the visual scene

[0225] Integrate task instructions and visual scene information, parse spatial references in the instructions (such as "click the button in the upper right corner"), and establish a correspondence between linguistic descriptions and visual elements. This includes calculating the joint probability of candidate elements.

[0226]

[0227] By maximizing the joint probability, the system can integrate linguistic description and visual context to select the optimal solution, thereby handling multi-candidate target disambiguation under fuzzy instructions. Wherein:

[0228] P(C|E) represents the degree of matching between instruction C and element E based on text matching: the similarity between instruction keywords and element semantic description is calculated using edit distance, similarity = 1 - (edit distance / max(keyword length, semantic description length)).

[0229] P(E|I) represents the probability that element E itself will be noticed or pointed to by the user in visual scene I based on visual saliency: the visual saliency score is calculated by the contrast, position (the center area of ​​the screen has a higher weight than the edge area), and size (larger size has a higher weight than smaller size), and the score is normalized as P(E|I).

[0230] Step S34: Interaction planning

[0231] By employing dependency reasoning and chained programming algorithms, semantically understood instructions are mapped into precise UI operation sequences across applications and levels, enabling automated execution planning for complex tasks. Based on the VDOM output by S322 and the instructions by S33, user instructions are ultimately converted into specific UI operation sequences through dependency reasoning and chained programming.

[0232] In this step, we will traverse the VDOM tree based on the instructions generated by S33, recursively searching for operable UI elements from the root to the target, and generating and arranging the corresponding operation instructions. During this process, it is necessary to check whether the UI element in each operation is visible and operable, and predict and plan the filling action through dependency reasoning (such as if the current node is not visible or operable, it means that it needs to rely on operating its parent node to wake it up). Finally, the system completes the traversal and completes the planning of the entire interaction action chain.

[0233] Here's an example of a complex business scenario: a user instruction that reads, "Save last month's sales summary from Excel to PDF, and then send it to Mr. Wang via the Chrome web version of DingTalk." This task spans two apps (Excel and Chrome) and involves multiple hidden layers and dynamic interactions.

[0234] The following sections S1-S4 explain how to generate an action sequence in a linked manner.

[0235] Phase 1:

[0236] At this point, (S1 / S21) the screen is captured and semantic segmentation is performed, and S22 constructs the current V-DOM tree:

[0237] • Window 1: Excel

[0238] o Region: Ribbon -> Tab: File, Tab: Home...

[0239] o Region: Workspace -> Table: Sales_Data

[0240] • Windows 2: Chrome (minimized in the taskbar)

[0241] Phase Two: Intent Recognition

[0242] S3 received the first-stage instruction: "Save Excel as PDF".

[0243] • Knowledge retrieval: The internal knowledge base matches the Save as PDF option, which is usually located under the File menu.

[0244] • V-DOM retrieval: The File menu is visible in the current V-DOM, but the Export or Save as buttons are not visible.

[0245] • Target Anchoring: Lock the File button coordinates [x_1, y_1] as the first physical touch point.

[0246] Phase 3: Dependency Reasoning and Chain Programming

[0247] S4 begins building the action chain. Since Save as PDF is a deeply nested operation, S4 initiates a "recursive unroll":

[0248] 1. Action 1: Click(File_Tab).

[0249] 2. Prediction: After clicking Prediction, a blue "BackstageView" will appear on the left.

[0250] 3. Dependency Filling: The next step can only be performed after the Export button is detected.

[0251] o Sequence increment: Wait_For(Export_Button)->lick(Export_Button).

[0252] 4. Prediction, Filling, and Non-Intuitive Interactive Processing

[0253] This leads to the most complex **"Save As" dialog box** stage, which is a typical example of non-intuitive interaction.

[0254] • Handling non-intuitive interactions (modal dialog boxes):

[0255] Problem: After clicking "Publish", Windows will pop up a dialog box for saving the original file.

[0256] o S4 strategy: It recognizes that this is a Modal Dialog.

[0257] o Sequence generation:

[0258] 1) Type("Last_Month_Sales.pdf") (Predicts focus in the filename input box).

[0259] 2) Key_Combo("Enter").

[0260] 3) Self-healing verification: S4 will monitor the "progress bar" or "save successful" prompt to confirm that the file has been written to the disk.

[0261] Phase 4: Cross-App Collaboration and Context Alignment

[0262] The task now moves to its second part: "Send it to General Manager Wang."

[0263] 1. App Switching

[0264] Action: Click(Taskbar_Chrome_Icon).

[0265] Verification: S1 rescanned to confirm that the browser window was on top.

[0266] 2. Complex Search and Matching (Search Alignment)

[0267] Instruction: Send to "General Manager Wang".

[0268] S4 Action Generation:

[0269] 1) Click(Search_Box) (located by S21 in the sidebar of the DingTalk web version).

[0270] 2) Type("General Manager Wang")->Wait(Dropdown_List).

[0271] 3) Prediction and Fill: Identify the avatar of "General Manager Wang" from the drop-down list and execute Click(User_Card).

[0272] 3. Handling non-intuitive "drag and drop" interactions

[0273] Scenario: Uploading a file.

[0274] S4 Optimized Path: Instead of clicking "Attachment icon -> Select File", the Agent can use a more efficient drag & drop.

[0275] Action: Drag from the file manager at coordinates [x_a, y_a] to the web chat box [x_b, y_b].

[0276] Phase 5: Summarize the complete output of the action sequence

[0277] The final execution sequence list (Action Manifest) generated by S4 is as follows:

[0278] Step sequence, action type, goal description, expected result (Verification), processing strategy

[0279] 1. Click, Excel - File tab, sidebar menu expands, navigation triggered.

[0280] 2. Click Export -> Create PDF, a save dialog box will pop up, indicating deep nesting processing.

[0281] 3. Type, enter the filename and press Enter. The file will be saved to C:\Temp. An interactive dialog box will appear.

[0282] 4. Activate, switch to Chrome, the browser window will be on top, and the focus will switch.

[0283] 5. Type, "Enter "General Manager Wang" in the search box", the contact list updates, dynamic search.

[0284] 6. Drop: Files are dragged from the disk to the webpage, and the chat box displays the file to be sent; this is not a straightforward drag-and-drop feature.

[0285] 7. Key, Enter, Message sent successfully, final confirmation.

[0286] The main algorithms in this process include the following parts:

[0287] 1. Definition of State Space and Objective Function

[0288] Define UI interactions as a quintuple:

[0289] Wherein, the state space S is a hierarchical structure defined by the VDOM tree. A state... It includes not only the currently visible pixels, but also the set of visibility of the current node. .

[0290] Action space A: Set of atomic operations .

[0291] Transition probability : Execute action Changes in the post-context environment. In UI, this is usually deterministic (e.g., clicking a menu to bring up a submenu), but it becomes random due to network latency.

[0292] Objective function: to find a sequence of actions So that from the initial state Reaching the target state node Minimize the cost:

[0293] 2. Path-based recursive search algorithm: For dependency reasoning (i.e., "clicking the parent node to reveal the child node"), we can use a recursive function Plan(u,v) to describe it, where u is the current focus and v is the target.

[0294] Suppose Path(root,v) is a sequence of semantic path nodes from the root to the target: Nodes=( , ,in Action generation logic description: For any node in the sequence Its activation function Defined as:

[0295]

[0296]

[0297] Final action sequence

[0298] 3. Probabilistic Filling

[0299] For non-intuitive interactions (such as pop-ups), CoA introduces an expected state check. Let E(s|a) be the expected state description after action a (provided by S3).

[0300] Self-healing determination formula:

[0301] If in After a certain period of time, the actual observed state The similarity to the expected state E(s|a) is less than the threshold. Fallback is the backtracking / degradation strategy, and Rescan is the strategy to wait for a rescan of the screen image and update the VDOM tree.

[0302]

[0303] Examples of business scenarios

[0304] Scenario: Click the "Save As" button.

[0305] • E(s|a): A system dialog box titled "Save As" is expected to appear.

[0306] · The result was a warning pop-up window that said "File already exists, do you want to overwrite?"

[0307] • Calculate the similarity Sim: The feature vector of the warning pop-up has extremely low similarity to the expected save dialog box (Sim < 0.05). ).

[0308] Self-healing actions:

[0309] 1. Identify the type of warning pop-up.

[0310] 2. Generate patch action: Click (Yes).

[0311] 3. Reset E(s|a) to its current state and restore the main sequence.

[0312] 4. Heuristic patching for handling "non-intuitive interactions"

[0313] For tasks like the "Save As" dialog box that cannot be modeled in the current VDOM, we introduce heuristic jumps:

[0314] when At that time, calculate the predefined action sequence with the highest transition probability.

[0315] Goal represents the intended outcome. For example, if a user says "Save as PDF," the Goal isn't "Click the Save As button," but rather the environment state satisfies File_Exists(path / *.pdf) == True.

[0316] Knowledge_Base represents a knowledge base / prior experience. Examples: Common sense A: Most "Save / Export" functions are hidden in the "File" menu in the top left corner. Common sense B: If "Upload" is clicked, the next action usually appears in a new pop-up native dialog box. Its purpose is to provide "intuition" when the current VDOM tree cannot offer guidance.

[0317] Current_App represents the current application context and environment constraints. The same goal (saved as PDF) will have completely different paths in Word and Chrome.

[0318] This means that if the system knows it has an Excel file and wants a PDF, it will initiate the sequence based on the operation with the highest prior probability (clicking "File"), even if it doesn't see the menu.

[0319] Step S4: System API Interface Integration and Management

[0320] By building system and application-level API instruction sets, multi-step UI operations are abstracted into one-click API calls, greatly simplifying complex system-level tasks and improving execution efficiency and cross-version compatibility.

[0321] In this step, the system queries the pre-established key-value custom operating system instruction and API interface mapping, meaning that one operating system instruction corresponds to one callable API function. This step mainly includes the following steps:

[0322] Step S41: System API command set preset and registration management

[0323] We pre-establish a list of instruction categories to define the system's capabilities. This list mainly includes instruction categories (such as OS_UI, OS_Interna_File, OS_Setting, etc.), instruction names, and instruction semantic descriptions.

[0324] We then built corresponding instruction sets based on the instruction classification list. This set includes a list of APIs for common UI elements, system settings, and tasks, compiled from official documentation for the corresponding system types (Windows, macOS, iOS, Android). This data provides information such as API name, API category, functional semantic description, parameter list (parameter name, type, required fields, default value), return value type, call permission requirements (if current permissions are not met during execution, the user is prompted for authorization, authentication information is provided, or execution is abandoned), execution mapping (local API path, built-in script, code snippet), and compatible system version. It also provides an API registration interface, allowing users to add APIs themselves or upgrade the interface layer remotely via the cloud. A simple example of an API instruction set is as follows:

[0325] For example, adjust_system_volume:

[0326] {

[0327] "id": "os_sys_001",

[0328] "name": "adjust_system_volume",

[0329] "description": "Used to adjust the operating system's master volume. Invoked when the user mentions 'turn up / down the volume,' 'mute,' or 'set the volume to 50%'."

[0330] "category": "OS_Setting",

[0331] "parameters": {

[0332] "type": "object",

[0333] "properties": {

[0334] "level": {

[0335] "type": "integer",

[0336] "description": "Target volume percentage, range 0-100",

[0337] "minimum": 0,

[0338] "maximum": 100

[0339] },

[0340] "relative": {

[0341] "type": "boolean",

[0342] "description": "Whether it is a relative adjustment. True if 'increase a little', false if 'set to 50'"

[0343] }

[0344] },

[0345] "required": ["level"]

[0346] },

[0347] "permission": {

[0348] "requires_confirmation": "false"

[0349] "requires_administrator": "false"

[0350] "requires_login": "false"

[0351] "requires_pay": "false"

[0352] }

[0353] "returns": {

[0354] "name": "Info"

[0355] "type": "text",

[0356] Values: "Null"

[0357] }

[0358] "execution": {

[0359] "type": "shell_script",

[0360] "command": "powershell.exe -Command (Install-Module -NameAudioDeviceCmdlets -Force; Set-AudioDevice -ID (Get-AudioDevice -Playback).ID; Set-AudioDevice -PlaybackVolume ({level}"

[0361] }

[0362] }

[0363] Finally, the instruction classification list and instruction set are converted and stored in the vector database.

[0364] Step S42: Dynamic instruction parsing and query mapping

[0365] This step uses a semantic classifier to determine which category of the aforementioned instruction classification list the user's intent belongs to. Then, the user's instruction is converted into a vector and stored in a vector database to retrieve the relevant instruction capability definition. Next, through an instruction wrapper, the API corresponding to the system version in the current environment is matched, and the structured parameters extracted in S23 are converted into the parameter format required by the API. This supports parameter type conversion (such as string to path, number to date) and parameter completion (completing optional parameters based on default values ​​or historical configurations).

[0366] Step S5: Application Software Instruction Management

[0367] Query pre-defined custom instruction sets for various application software. The instruction sets are based on the official APIs or automation interfaces of the application software (such as Excel's VBA interface and PowerPoint's COM interface) and interface with S4's API integration framework.

[0368] In this step, the system queries the pre-established key-value custom operating system instruction and API interface mapping, meaning that one operating system instruction corresponds to one callable API function. The main steps in this step are the same as in S4:

[0369] Step S51: Application software instruction set preset and registration management

[0370] Similarly, we pre-establish an APP category list to define the system's APP capabilities. This list mainly includes APP categories (such as browser_chrome, browser_firefox, navigation_Gaode, navigation_Baidu, office_EXECL, office_Outlook, etc.), APP names, and semantic descriptions of APP capabilities.

[0371] Next, we created corresponding app instruction sets based on the app category list. This set includes a list of APIs for common tasks, UI operations, etc., compiled from the official documentation of the corresponding app types. This data provides information such as the app name, API name, functional semantic description, parameter list (parameter name, type, required fields, default value), return value type, call permission requirements (if current permissions are not met during execution, the user will be prompted for authorization, provided with authentication information, or the execution will be abandoned), execution mapping (local API path or remote API URL, built-in script, code snippet), and compatible system version. It also provides an API registration interface, allowing users to add APIs themselves or upgrade the interface layer remotely via the cloud. A simple example of the API instruction set is as follows:

[0372] {

[0373] "id": "Office_EXECL_012",

[0374] "name": "execl_data_remove_duplicates",

[0375] "description": "Used for data cleaning and deduplication in the Excel suite. Invoked when a user mentions 'using Excel,' 'data cleaning,' or 'removing duplicate rows.'"

[0376] "category": "Office_EXECL",

[0377] "parameters": [

[0378] {

[0379] "name": "Columns",

[0380] "data_type": "Variant",

[0381] "description": "An array of column indexes containing repeating information. This array can specify a single column or multiple columns."

[0382] "usage_notes": [

[0383] For single-column data, please use integers (e.g., use 3 for the third column).

[0384] For multi-column data, use the Array function (for example, Array(1, 3) represents the first and third columns). The system will remove rows with duplicate values ​​in these column combinations. ]

[0386] },

[0387] {

[0388] "name": "Header",

[0389] "data_type": "XlYesNoGuess",

[0390] "default_value": "xlNo",

[0391] "description": Specifies whether the first line is a header line. It has three possible values.

[0392] "valid_values": [

[0393] {

[0394] "name": "xlYes",

[0395] "value": 1,

[0396] "description": "The first line is the title and is not included in the comparison and deduplication."

[0397] },

[0398] {

[0399] "name": "xlNo",

[0400] "value": 2,

[0401] "description": "The first line contains ordinary data, used for comparison and deduplication."

[0402] },

[0403] {

[0404] "name": "xlGuess",

[0405] "value": 0,

[0406] "description": "Let Excel automatically determine if there are headings."

[0407] } ]

[0409] }

[0410] ],

[0411] "required": ["Columns"]

[0412] },

[0413] “permission”: {

[0414] "requires_confirmation": "false"

[0415] "requires_administrator": "false"

[0416] "requires_login": "false"

[0417] "requires_pay": "false"

[0418] }

[0419] "returns": "Null"

[0420] "execution": {

[0421] "type": "vba_script",

[0422] "Sub RemoveDups()

[0423] ThisWorkbook.ActiveSheet.UsedRange.Select.RemoveDuplicatesColumns:= $Columns, Header:=$xlYes

[0424] End Sub

[0425] }

[0426] }

[0427] Finally, the instruction classification list and instruction set are converted and stored in the vector database.

[0428] Step S52: Dynamic instruction parsing and query mapping

[0429] This step uses a semantic classifier to determine which category of the aforementioned instruction classification list the user's intent belongs to, converts the user's instruction into a vector, and stores it in the Vector Database (RAG) to retrieve the relevant instruction capability definition. Then, through an instruction wrapper, it matches the API of the corresponding system version in the current environment, converts the structured parameters extracted in S23 into the parameter format required by the API, and supports parameter type conversion (such as string to path, number to date) and parameter completion (completing optional parameters based on default values ​​or historical configurations).

[0430] Step S6: Instruction optimization and execution plan generation

[0431] This step involves using a large model to evaluate the original instructions, the mapped and generated actual executed instructions, and the execution results of each step, and then rearranging and optimizing the instruction execution sequence.

[0432] Step S61: Instruction Standardization and Dependency Analysis

[0433] In this step, we employ an on-demand loading strategy, integrating intent classification, instruction classification, original task requirements, instruction vectorization retrieval, and instruction execution lists as the context of the large model through steps S2, S3, S4, and S5. We then fine-tune the model using Few-Shot instructions, analyze and standardize the collected multi-step instructions through the large model, eliminate redundant instructions (such as repeated save operations), merge semantically equivalent operations (e.g., merging "open file" and "load file" into "open file"), and identify implicit dependencies between instructions (dependency criteria: time dependency (operation B can only be executed after operation A is completed), data dependency (operation B requires the output data of operation A), resource dependency (operations A and B cannot simultaneously occupy the same resource)). We use an adjacency matrix to represent instruction dependencies: i and j represent two instructions in a multi-instruction set.

[0434]

[0435] Construct a complete dependency graph, use topological sorting to check for circular dependencies (topological sorting algorithm: Kahn algorithm, dequeuing nodes with an in-degree of 0 in sequence; if the final number of dequeued nodes is less than the total number of nodes, then a circular dependency exists), remove or modify dependency edges or merge instructions, thereby detecting and resolving instruction conflicts and contradictions (contradictory instruction determination: mutually exclusive operations on the same resource, such as "editing the same file simultaneously", resolution strategy: sort by instruction priority, priority = intent confidence × operation urgency), and use semantic analysis to detect contradictory instructions and set instruction priorities.

[0436] Step S62: Dependency Graph Verification and Critical Path Identification

[0437] Construct an instruction dependency matrix to clarify the relationships between preceding and succeeding instructions, identify and mark instruction nodes on the critical path, and use topological sorting and depth-first search algorithms to assess the risk of dependency cycles and loops, ensuring that the dependency graph is directed acyclic. Generate a visual dependency graph (described using DOT language, which can be generated using Graphviz) to support execution plan review. Critical path identification method: Calculate the earliest start time (ES) and latest start time (LS) of each instruction node. ES = the maximum EF (earliest completion time) of all preceding instructions, EF = ES + operation time, LS = LF (latest completion time) - operation time, LF = the minimum LS of all succeeding instructions. Nodes with ES = LS constitute the critical path.

[0438] Step S63: Instruction Optimization and Parallelism Strategies

[0439] The algorithm combines instruction merging and simplification to reduce operation steps, identifies a subset of instructions that can be executed in parallel (instructions that have no dependencies and do not occupy the same resources can be executed in parallel), formulates parallel strategies (using multi-threaded execution, with the main thread managing dependencies and child threads executing parallel instructions), optimizes data flow paths, reduces intermediate result transmission (e.g., Excel data is directly transferred to PowerPoint via memory sharing instead of file export and import), and applies heuristic algorithms to find the optimal solution for resource efficiency.

[0440] Step S64: Simulation Testing and Risk Assessment

[0441] The execution of the instruction set is simulated in a sandbox to predict performance bottlenecks (based on the operation time of the critical path, the operation with the longest time consumption is identified as the performance bottleneck, such as the current bottleneck being step 2: Excel data analysis and chart creation, which takes 22 seconds). The correctness and completeness of the optimized plan are verified (by simulating the execution of each instruction, it is checked whether the parameter passing is correct, whether the dependency relationship is satisfied, and whether the operation result meets expectations). Boundary condition and abnormal situation tests are performed (boundary conditions include: extremely large data volume (more than 100,000 rows), abnormal screen resolution (800×600), and network interruption; abnormal situations include: file corruption, application unresponsiveness, and API call failure). An execution risk assessment report is generated, which includes possible failure points, risk levels (high / medium / low), and coping strategies.

[0442] Examples of risks and corresponding response strategies are as follows:

[0443] Risk 1: The Excel file may not be found or can be opened.

[0444] Risk 2: The data analysis process may require human intervention to determine trends.

[0445] Risk 3: Charts may be distorted during the PowerPoint import process.

[0446] Risk 4: The recipient's email address may need to be looked up in the company's address book.

[0447] Risk 5: Large attachments may cause emails to fail to send.

[0448] The system generates risk response strategies:

[0449] 1. Provide a file search dialog interface when the file cannot be found.

[0450] 2. Use preset analysis templates to ensure key indicators are calculated automatically.

[0451] 3. Use native format links instead of image imports to maintain data editability.

[0452] 4. Access the enterprise address book API to automatically fill in email addresses.

[0453] 5. Detect attachment size; if it exceeds the threshold, share via cloud link.

[0454] Step S65: Generation of Structured Execution Plan

[0455] Integrate the optimization results of the above steps to generate a new structured execution plan, which includes detailed execution order, parameter configuration and dependency conditions, adds key checkpoints and verification conditions during the execution process (checkpoints are set after critical path nodes, and verification conditions are that the operation results meet expectations), generates a human-understandable execution plan overview, and a detailed instruction execution sequence format.

[0456] Step S7: Instruction Execution and Result Feedback

[0457] Execute instructions in the optimized order.

[0458] Step S71: Instruction dispatch and execution

[0459] Through a cross-platform device driver module and a unified action interface, it precisely drives peripherals such as keyboards and mice to execute planned sequences of interactive actions, achieving a closed loop of "planning-execution". It parses the instruction sequence and parameters in the execution plan, dispatches instructions to the corresponding execution components (UI operations, API calls, software operation instructions, etc.), and establishes cross-component communication channels (using a message queue mechanism, with components communicating via JSON format messages containing instruction IDs, parameters, status, and timestamps) to ensure smooth data flow.

[0460] The technical solutions for implementing UI command operations are as follows:

[0461] The device driver module is implemented as follows:

[0462] 1. Driver Adaptation Layer: Supports cross-platform adaptation for Windows, Linux, macOS, iOS, and Android. For Windows, keyboard and mouse drivers are implemented based on the SendInput function (user32.dll) and PyAutoGUI; for Linux, keyboard and mouse drivers are implemented using Python-UIAutomator2; for macOS, they are implemented based on Quartz Event Services and the Pynput framework; for iOS, UI automation is implemented using XCUITest; and for Android, ADB and AccessibilityService are used to implement click, swipe, long press, zoom, and other operations.

[0463] 2. Unified Action Interface: This interface layer mainly abstracts standard action sets such as mouse actions in different system environments.

[0464] Movement: mouse.move(x,y,duration=0.2) achieves smooth cursor movement through an interpolation algorithm (linear interpolation, updating coordinates every 10ms) to avoid jumping;

[0465] Click: mouse.click(button="left",clickType="single"), supports left / right click, single / double click, click duration is controlled between 50-100ms to simulate human clicking habits;

[0466] Drag and drop: mouse.drag(startX,startY,endX,endY,duration=0.5) achieves dragging by continuously moving coordinates without stopping;

[0467] Deviation correction: The actual cursor position is obtained in real time (via the GetCursorPos function) and compared with the target position. Correction is triggered when the deviation is greater than 5 pixels. Correction algorithm: Corrected coordinates = current coordinates + (target coordinates - current coordinates) × 0.3;

[0468] Keyboard actions:

[0469] Character input: keyboard.type(text, delay=50), supports Chinese and English characters, numbers, and symbols. Each character is entered 50ms apart, simulating human input speed;

[0470] Function keys / combination keys: keyboard.press(key, combination=[]), supports single function keys (such as Enter, Esc) and combination keys (such as Ctrl+S, Ctrl+C), with a 100ms interval between key press and release;

[0471] Input validation: After input is completed, the content of the input box is recognized by OCR and compared with the target text. If they do not match, re-input is triggered (maximum of 2 retries).

[0472] Touch operation:

[0473] Specify coordinates: TAP(x,y)

[0474] Long press: LONG_PRESS(x,y,duration) Press and hold at the specified position for duration milliseconds.

[0475] Drag: DRAG_AND_DROP(start_x,start_y,end_x,end_y) drags the object to the target position.

[0476] Simulated system buttons: KEY_EVENT(key_code), such as BACK, HOME, POWER, VOLUME_UP, VOLUME_Down).

[0477] Slide: SWIPE(direction, distance): (UP, DOWN, LEFT, RIGHT), a sliding operation encapsulated in the form of a variable. For example, "Slide up one screen".

[0478] 3. Visual Calibration Module: This module acquires Resolution, Orientation, DPI, and OS Version in real time, accurately converting the intent seen by the large model at the "logical level" into pixel coordinates at the current device's "physical level." This ensures accurate UI operation even on different brands (Huawei, Apple, Samsung) and different screen ratios (16:9, 21:9, foldable screens). The system does not directly transmit pixel values ​​(e.g., 1080x1920), but uses normalized proportional coordinates. This module mainly consists of the following three layers.

[0479] Semantic / Relative layer: Visual model output (0.5, 0.5), representing the center of the screen.

[0480] Calibration layer (VCM Transformer): Performs scaling, offsetting, and safe zone processing based on device metadata, as well as screen rotation (orientation).

[0481] Physical / Pixel: The execution layer is ultimately clicked on the device (540, 960).

[0482] 4. Execution monitoring: Monitor the execution status of instructions in real time, and record the operation result of the UI once every 200ms, including: operation type, target element, execution status (success / failure), time consumption, and error information (if any).

[0483] Among them, the technical solutions for implementing system API commands mainly adopt common scripts for the corresponding systems, such as Python, PowerShell, VBA, and C# under Windows, and bash and Python under Linux, macOS, iOS, and Android.

[0484] Among them, the technical solutions for implementing application software instruction operations mainly adopt Python + software COM interface or open API.

[0485] Step S72: Execution in steps and evaluation of execution results

[0486] Collect the execution results of each instruction step, and distribute the execution results to different evaluation stages and further instruction execution judgments for different types of instructions. For example, for UI operation instructions, compare the changes in the UI before and after instruction execution to see if they meet the execution expectations; for system API instruction execution, check if the return value matches the API instruction definition; for application software instruction execution results, check if they meet the instruction expectations. If they do not meet the expectations, trigger the corresponding exception risk handling branch, either to initiate manual interaction or to enter S6 for instruction optimization and rearrangement.

[0487] Step S73: Execution Result Collection and Feedback Output

[0488] Collect the final task execution results, perform consistency verification, transform the technical results into user-friendly expressions, generate multi-level execution result summaries based on different task types, and format several outputs using different media (voice, text).

[0489] The hardware environment configuration, software dependency list, specific algorithms for each step (such as Bayesian wake word detection, Sentence-BERT intent matching, and recursive dependency reasoning) and exception handling strategies provided in this embodiment not only ensure the reproducibility of the method, but also verify the significant value of the technical solution in improving operational efficiency, enhancing environmental adaptability, and reducing manual intervention costs for practical scenarios such as office automation and cross-software process collaboration.

[0490] In the future, based on this embodiment, the scope of the instruction set can be further expanded (such as adding API adaptation for professional design software and industrial control software), the accuracy of the visual recognition model under complex interfaces (such as multiple window overlaps and dynamic pop-ups) can be optimized, and the instruction synchronization mechanism in multi-device collaborative scenarios can be strengthened to continuously improve the universality and robustness of the method and provide more comprehensive technical support for the intelligent upgrade of the natural human-computer interaction field.

[0491] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for interactive planning and execution of complex instruction tasks based on a large model, characterized in that, The steps include: Step S1: Perform wake-up detection and instruction reception operations to obtain task instructions input by the user in the form of text or voice; Step S2: Use a large language model to perform semantic parsing on the task instructions and match them with a set of evolvable predefined task intent templates to obtain a structured task instruction description; When the matched intent template contains a verified instruction sequence, the instruction sequence is directly used as the execution plan. Otherwise, proceed to steps S3 to S6; Step S3: Obtain the system screen image, identify UI elements through a visual recognition model, and construct an accessibility tree; based on the accessibility tree, use a recursive dependency reasoning algorithm to map the structured task instruction description into a UI operation action sequence; Step S4: Map the system-level operations involved in the structured task instruction description to a preset system API instruction set stored in vector form, and generate system API call instructions; Step S5: Map the application software-level operations involved in the structured task instruction description to a preset application software API instruction set stored in vector form, and generate application software API call instructions; Step S6: Combine the UI operation action sequence, system API call instructions, and application software API call instructions, perform instruction dependency analysis and optimization, and generate a structured execution plan; Step S7: Execute the instructions according to the structured execution plan, and provide feedback on the execution results through the cross-platform device driver module.

2. The method according to claim 1, characterized in that, Step S2 specifically includes: Step S21: Semantic parsing and word sense disambiguation of task instructions, extracting core actions, objects, parameters, and modifiers, and constructing a semantic parsing tree; Step S22: Matching the semantic parsing tree with a predefined task intent template, using Sentence-BERT encoding and cosine similarity calculation to identify the task intent and output a structured task instruction description; the predefined task intent template allows users to confirm and update successfully executed instruction sequences; Step S23: Extracting task parameters based on the semantic parsing tree, performing parameter normalization and integrity verification, and outputting parameterized JSON instructions; Step S24: Constructing a memory graph through a graph neural network to manage multi-turn dialogue states, achieving referential resolution and contextual connection; Step S25: Triggering interactive intent clarification based on parameter integrity score and confidence level, generating the final structured task instruction description.

3. The method according to claim 1, characterized in that, Step S3 specifically includes: Step S31: Semantic segmentation of the system screen image, identification of interface layout and functional components, and establishment of relative position encoding of UI elements; Step S32: Identification of the boundaries, categories and text content of interactive UI elements by combining visual feature detection and screen text extraction, and extraction of element attributes including position, state and function description, and construction of the accessibility tree; Step S33: Matching task instructions with visual elements in the accessibility tree and parsing the spatial reference relationship in the instructions; Step S34: Based on the recursive dependency reasoning algorithm, converting the instructions into a cross-application, cross-level UI operation action sequence, and configuring abnormal state handling logic for the sequence.

4. The method according to claim 3, characterized in that, The recursive dependency reasoning algorithm in step S34 is as follows: traverse the accessibility tree and recursively find the operable UI element path from the current state to the target state; when the current node is not visible or operable, infer the operation of its dependent parent node and supplement the corresponding operation action; generate a complete UI operation action sequence, and compare the expected state with the actual interface state during the execution process. If it does not meet the expectations, trigger backtracking, retry or strategy adjustment.

5. The method according to claim 1, characterized in that, The system API instruction set and application software API instruction set in steps S4 and S5 are hierarchical, dynamically registerable instruction sets that support instruction mapping through semantic classification and vector retrieval, and are encapsulated into callable API functions.

6. The method according to claim 5, characterized in that, Both the system API instruction set and the application software API instruction set are stored in a vector database in vector form, supporting retrieval based on semantic similarity; The instruction mapping process includes: determining the category of the user's intent through a semantic classifier, encoding the user's instruction into a vector, retrieving the matching instruction definition from the vector database, and completing parameter adaptation and call encapsulation through an instruction wrapper.

7. The method according to claim 1, characterized in that, Step S6 specifically includes: Step S61: Analyze the dependencies between instructions, construct an instruction dependency graph, and detect cyclic dependencies through topological sorting; Step S62: Identify critical paths and calculate the earliest and latest start times of each instruction node; Step S63: Merge redundant instructions, identify a subset of instructions that can be executed in parallel, and formulate a parallel execution strategy; Step S64: Conduct sandbox simulation tests, assess execution risks, and generate a risk assessment report; Step S65: Generate a structured execution plan that includes execution order, parameter configuration, dependency conditions, and verification nodes.

8. The method according to claim 1, characterized in that, Step S7 specifically includes: Step S71: Parse and execute UI operations, system API calls and application software API calls in the structured execution plan through the cross-platform device driver module; Step S72: Monitor the execution results of each step in real time, compare them with the expected state, and trigger exception handling or replanning when they do not meet expectations; Step S73: Collect the final execution results and convert them into a user-friendly form for feedback.

9. The method according to claim 8, characterized in that, The cross-platform device driver module supports Windows, macOS, Linux, iOS, and Android operating systems, provides a unified action interface including mouse movement, clicking, dragging, keyboard input, and touch screen operation, and has a visual calibration function to convert logical coordinates into the physical coordinates of the current device.

10. A complex instruction task planning and execution interactive system based on a large model, characterized in that, include: The voice processing module is used for voice acquisition, conversion, and wake-up detection; The semantic understanding module is used for instruction parsing, intent matching, and dialogue management; The visual recognition module is used for screen parsing, UI element recognition, and interaction action planning. The API integration module is used for the registration, mapping, and invocation of system and application software APIs; An interactive planning module is used for instruction dependency analysis and execution plan generation; an operation driving module is used to drive peripheral devices to perform operations and provide feedback results; wherein, the system executes the method as described in any one of claims 1-9.

Images