Multitask structure based on semantic intention and interface state fusion and generation method

Abstract

The invention discloses a multi-step task generation method based on semantic intention and interface state fusion, which is characterized in that a big language model is guided to generate a structured operation sequence capable of being executed across applications through joint coding of a user natural language instruction and a real-time UI control hierarchical structure. The method comprises an abnormal state feedback and increment regeneration mechanism, can dynamically adapt to a changeable interface environment, and is suitable for task automation requirements of scenes such as an intelligent terminal and a vehicle-mounted system. The method for generating the multi-step operation instruction through the large language model is suitable for automatic task execution scenes such as a mobile terminal, wearable equipment and a vehicle-mounted system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, information processing and human-computer interaction technology, and in particular relates to a multi-task structure and generation method based on the fusion of semantic intent and interface state. Background Art

[0002] Artificial intelligence, information processing, and human-computer interaction are AI technologies that combine big data with hardware. However, AI technologies operate based on a collection of instruction sets. Currently, Xiaoai, Siri, DeepSeek, and other applications are essentially based on the 2.0 version of Baidu's first-generation search engine, all of which are single-instruction-set search engines. Consequently, automated task generation technologies based on voice or text suffer from the following flaws: 1. Disconnection between intent and interface: Traditional methods rely on pre-set rules to match user commands and fail to dynamically adjust the action path based on real-time UI state. For example, a user command like "Share to WeChat" requires launching an app on the desktop, while a contact can be directly selected in the chat interface. 2. Weak task generalization: Fixed scripts cannot cover the varying control hierarchies (such as UI differences between different Android versions or third-party apps). 3. Lack of logical continuity: Generated single-step actions lack context and are unable to recover from interruptions. Although large language models (LLMs) possess natural language understanding and logical reasoning capabilities, existing technologies fail to effectively encode UI state structure into an understandable input format for the model, resulting in a mismatch between generated action commands and the actual controls on the interface. Furthermore, the free text output of LLMs is difficult to directly convert into structured action commands. Therefore, how to combine natural language instructions with dynamic UI states to model and guide the large language model to generate a structured operation sequence that matches the current interface has become a key technical issue in improving the efficiency and robustness of intelligent task execution. Summary of the Invention

[0003] The purpose of this invention is to propose a multi-task structure and generation method based on the fusion of semantic intent and interface state. The core solution is to input a large language model with a unified data format to generate an operation instruction sequence that matches the current interface state; guide the LLM to generate an executable operation instruction sequence through structured output constraints; and design an abnormal state detection and task dynamic adjustment mechanism to improve the robustness of the generated results.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A multi-task structure based on the fusion of semantic intent and interface state, the multi-task structure includes a semantic intent structuring module, an interface state modeling module, a multimodal joint reasoning module, and an abnormal feedback and dynamic correction module; the semantic intent structuring module inputs user voice / text instructions (such as "book a plane ticket and share the itinerary with colleagues"); and outputs structured data containing atomic operation intentions.

[0006] The interface state modeling module inputs a real-time UI control tree (obtained through AccessibilityService or UIAutomator); after processing, the control hierarchy is converted into a standardized tree representation, including: node attributes: control ID, text content, coordinate range, executable actions (click / input / scroll). The multimodal joint reasoning module inputs structured intent data and standardized UI state descriptions (LLM prompt engineering: output format is constrained by template. (For example: input format: [user instruction] + [current interface control tree]);

[0007] Output requirements:

[0008] 1. The operation steps must match the current interface controls;

[0009] 2. Each step format: {"Action type":"click","Target control":"Control ID or text","Parameter":""}

[0010] The exception feedback and dynamic correction module includes state verification, incremental regeneration, and a fallback mechanism. The state verification compares the LLM output instructions with the executableness of the current UI control. If the target control does not exist, it is marked as an exception; the incremental regeneration re-inputs the exception mark and the updated UI state into the LLM, triggering local path correction; the fallback mechanism deletes invalid steps and replans subsequent operations when continuous correction failures exceed a threshold.

[0011] Preferably, the output structured data containing atomic operation intentions includes but is not limited to the following examples:

[0012] {"Action":"Book a flight","Parameters":{"Date":"2024-07-20","Destination":"Beijing"},"Related Operation":"Share itinerary"};

[0013] Preferably, the standardized tree includes but is not limited to the example fragment: {"id":"com.app:id / search_box","text":"","actions":["click","input"]};

[0014] Preferably, the format of each step output by the multimodal joint reasoning module includes but is not limited to the example: [

[0015]

[0016] A multi-task generation method based on the fusion of semantic intent and interface state includes the following steps: (This step is one type of multi-task generation method, used to automatically execute the user's natural language instruction "Share this song to Weibo and bookmark it.")

[0017] Step 1: User command input and semantic analysis

[0018] Users input natural language commands through voice or text: "Share this song to Weibo and collect it."

[0019] The system analyzes the instruction through the semantic intent parsing module and identifies two atomic tasks:

[0020] [{"action":"Share","target":"Weibo"},{"action":"Collect"}]

[0021] Each atomic task contains an action type and corresponding target parameters, forming structured intent data.

[0022] Step 2: Interface state collection and modeling

[0023] The system uses accessibility interfaces (such as Android AccessibilityService or UIAutomator)

[0024] Get the interface control tree information of the current application.

[0025] The example interface is a music app playback page, and the control structure is as follows:

[0026] {"screen":"Play Page","controls":[{"id":"player_title","text":"Sunny Day",

[0027] "actions":[]},{"id":"player_play_btn","text":"","actions":

[0028] ["click"]},{"id":"share_btn","text":"Share","actions":["click"]},

[0029] {"id":"favorite_btn","text":"Favorite","actions":["click"]}]}

[0030] The system converts the control tree into a standardized interface state description as one of the inputs of the large language model.

[0031] Step 3: Joint reasoning to generate operation sequences

[0032] The structured intent data and the standardized interface state description are jointly input into the large language model (LLM).

[0033] Prompt the output format of the engineering constraint model and generate an operation instruction sequence, for example:

[0034] [{"action":"click","target":"share_btn","parameters":""},{"action":"click",

[0035] "Target": "Weibo Icon", "Parameter": ""}, {"Action": "click", "Target":

[0036] "favorite_btn","parameters":""}]

[0037] Each instruction contains action type, target control, and parameter information, with a clear structure and executable.

[0038] Step 4: Task execution and status feedback

[0039] The system executes the above instructions in sequence:

[0040] 1. Click the "Share" button;

[0041] 2. Find and click the "Weibo icon" in the pop-up sharing interface;

[0042] 3. Return to the main interface and click the "Favorites" button.

[0043] During the execution process, the system verifies the execution status of each step of the operation, such as whether the control exists and whether it is clickable.

[0044] Step 5: Exception handling and dynamic correction

[0045] If a target control is missing during execution (e.g. the "Weibo icon" is not displayed), the system triggers the abnormal feedback mechanism:

[0046] 1. Re-collect the current interface status;

[0047] 2. Re-enter the exception mark and the updated interface state into the large language model;

[0048] 3. Generate new alternative paths.

[0049] For example, the model can be modified as follows:

[0050] [{"action":"click","target":"share_btn","parameter":"parameter"},{"action":"click","target":"copy link","parameter":"parameter"},{"action":"launch app","target":"Weibo","parameter":"parameter"},{"action":"paste","target":"publish box","parameter":"parameter"}]

[0051] If the number of consecutive correction failures exceeds the set number, the system will execute the task rollback mechanism, delete the invalid steps and prompt the user to intervene.

[0052] The multi-task structure and generation method based on the fusion of semantic intent and interface state of the present invention have cross-platform compatibility, generation executableness, and dynamic error correction capabilities. Through standardized UI state modeling, it adapts to different devices and operating systems. At the same time, structured output constraints ensure that instructions can be directly mapped to system APIs or ADB commands. The abnormal state feedback closed loop improves the task completion rate in complex scenarios. It can be widely deployed in smartphones, vehicle systems, smart assistants, wearable devices, etc., and supports multiple application scenarios such as voice assistants, barrier-free assistance, automated testing, and intelligent customer service. It has good commercial implementation and industrial prospects.

[0053] This invention fully combines natural language understanding capabilities with dynamic user interface structure perception capabilities, breaking through bottlenecks in existing automated task generation technologies, such as "language understanding out of interface context," "operation instructions cannot be executed," and "abnormal tasks cannot be self-healed." It has the following four characteristics:

[0054] 1. Joint modeling mechanism of intent and interface state

[0055] This invention is the first to unify the modeling of user natural language input and real-time UI control hierarchical structure, and structures "semantic intent" and "control state" as joint input, which is input into the large language model for reasoning; this fusion mechanism supports the model to generate different operation paths under different interface states, realizes context-aware task planning, and significantly improves the adaptability and intelligence level of instructions.

[0056] 2. Interface modeling method for multiple applications and platforms

[0057] By standardizing the interface state modeling method (such as describing the control properties and hierarchical relationships in a tree structure), the present invention realizes the abstract representation of UI state and has good cross-platform adaptability;

[0058] Regardless of whether the terminal is a mobile phone, tablet, car system or wearable device, a unified interface state description can be used as model input, thereby improving the portability and versatility of the system.

[0059] 3. Abnormal state detection and dynamic path correction mechanism

[0060] The present invention introduces a status verification mechanism after the operation is executed, which automatically triggers abnormal feedback when it detects missing controls, abnormal jumps, etc.

[0061] Through the "local incremental regeneration + path rollback" mechanism, the system can dynamically correct and restore the task path without interrupting the task, significantly improving the robustness and continuity of task completion.

[0062] 4. Support multi-step, cross-interface, and cross-application task flow generation capabilities;

[0063] The present invention is not limited to a single page or operation, but supports the generation of a complete task flow including multiple operation steps, cross-application jumps, and multiple rounds of interaction based on complex user intentions (such as "book a ticket and send it to a colleague"); it realizes a complete closed loop from intention analysis → operation planning → control matching → dynamic execution → exception handling → completion feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Generate a flow chart for the task of the present invention;

[0065] Figure 2 A code structure diagram for modeling the UI state of the present invention;

[0066] Figure 3 Schematic diagram of the abnormal feedback and fallback mechanism of the present invention. DETAILED DESCRIPTION

[0067] In order to more clearly illustrate the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are only used to illustrate the present invention and do not constitute a limitation on the scope of protection of the present invention. This section will describe the specific embodiments of the present invention in detail. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it should not be understood as a limitation on the scope of protection of the present invention.

[0068] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0069] In the description of this invention, terms such as "greater than," "less than," and "exceed" are understood to exclude the number itself, while terms such as "above," "below," and "within" are understood to include the number itself. The use of terms such as "first" and "second" is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0070] Reference Figure 1 The multi-task structure based on the fusion of semantic intent and interface state applied in the present invention includes a semantic intent structuring module, an interface state modeling module, a multimodal joint reasoning module, and an abnormal feedback and dynamic correction module;

[0071] The semantic intention structuring module inputs user voice / text instructions (such as "book a plane ticket and share the itinerary with colleagues"); and outputs structured data containing atomic operation intentions.

[0072] The interface state modeling module inputs a real-time UI control tree (obtained through AccessibilityService or UIAutomator); after processing, the control hierarchy is converted into a standardized tree representation, including: node attributes: control ID, text content, coordinate range, executable actions (click / input / scroll). The multimodal joint reasoning module inputs structured intent data and standardized UI state descriptions (LLM prompt engineering: output format is constrained by template. (For example: input format: [user instruction] + [current interface control tree]);

[0073] Output requirements:

[0074] 1. The operation steps must match the current interface controls;

[0075] 2. Each step format: {"Action type":"click","Target control":"Control ID or text","Parameter":""}

[0076] The exception feedback and dynamic correction module includes state verification, incremental regeneration, and a fallback mechanism. The state verification compares the LLM output instructions with the executableness of the current UI control. If the target control does not exist, it is marked as an exception; the incremental regeneration re-inputs the exception mark and the updated UI state into the LLM, triggering local path correction; the fallback mechanism deletes invalid steps and replans subsequent operations when continuous correction failures exceed a threshold.

[0077] Furthermore, the output structured data containing the atomic operation intention includes but is not limited to the following examples: {"action":"book a flight","parameters":{"date":"2024-07-20","destination":"Beijing"},"associated operation":"share itinerary"};

[0078] Furthermore, the standardized tree includes but is not limited to example fragments:

[0079] {"id":"com.app:id / search_box","text":"","actions":["click","input"]};

[0080] Furthermore, the format of each step output by the multimodal joint reasoning module includes but is not limited to the example: [

[0081]

[0082] This application provides Example 1: A multi-task generation method based on the fusion of semantic intent and interface state, comprising the following steps: (This step is one type of multi-task generation method, used to automatically execute a user's natural language instruction "Share this song to Weibo and bookmark it.")

[0083] Step 1: User command input and semantic analysis

[0084] Users input natural language commands through voice or text: "Share this song to Weibo and collect it."

[0085] The system analyzes the instruction through the semantic intent parsing module and identifies two atomic tasks:

[0086] [{"action":"Share","target":"Weibo"},{"action":"Collect"}]

[0087] Each atomic task contains an action type and corresponding target parameters, forming structured intent data.

[0088] Step 2: Interface state collection and modeling

[0089] The system uses accessibility interfaces (such as Android AccessibilityService or UIAutomator)

[0090] Get the interface control tree information of the current application.

[0091] The example interface is a music app playback page, and the control structure is as follows:

[0092] {"screen":"Play Page","controls":[{"id":"player_title","text":"Sunny Day",

[0093] "actions":[]},{"id":"player_play_btn","text":"","actions":

[0094] ["click"]},{"id":"share_btn","text":"Share","actions":["click"]},

[0095] {"id":"favorite_btn","text":"Favorite","actions":["click"]}]}

[0096] The system converts the control tree into a standardized interface state description as one of the inputs of the large language model.

[0097] Step 3: Joint reasoning to generate operation sequences

[0098] The structured intent data and the standardized interface state description are jointly input into the large language model (LLM).

[0099] Prompt the output format of the engineering constraint model and generate an operation instruction sequence, for example:

[0100] [{"action":"click","target":"share_btn","parameters":""},{"action":"click",

[0101] "Target": "Weibo Icon", "Parameter": ""}, {"Action": "click", "Target":

[0102] "favorite_btn","parameters":""}]

[0103] Each instruction contains action type, target control, and parameter information, with a clear structure and executable.

[0104] Step 4: Task execution and status feedback

[0105] The system executes the above instructions in sequence:

[0106] 1. Click the "Share" button;

[0107] 2. Find and click the "Weibo icon" in the pop-up sharing interface;

[0108] 3. Return to the main interface and click the "Favorites" button.

[0109] During the execution process, the system verifies the execution status of each step of the operation, such as whether the control exists and whether it is clickable.

[0110] Step 5: Exception handling and dynamic correction

[0111] If a target control is missing during execution (e.g. the "Weibo icon" is not displayed), the system triggers the abnormal feedback mechanism:

[0112] 1. Re-collect the current interface status;

[0113] 2. Re-enter the exception mark and the updated interface state into the large language model;

[0114] 3. Generate new alternative paths.

[0115] For example, the model can be modified as follows:

[0116] [{"action":"click","target":"share_btn","parameter":"parameter"},{"action":"click","target":"copy link","parameter":"parameter"},{"action":"launch app","target":"Weibo","parameter":"parameter"},{"action":"paste","target":"publish box","parameter":"parameter"}]

[0117] If the number of consecutive correction failures exceeds the set number, the system will execute the task rollback mechanism, delete the invalid steps and prompt the user to intervene.

[0118] This application provides Example 2: A multi-task generation method based on the fusion of semantic intent and interface state, comprising the following steps:

[0119] The task of creating a schedule and notifying others includes an example instruction: "Please arrange a meeting with Zhang San at 4 pm tomorrow and notify him."

[0120] Step 1: Semantic parsing output two-step task:

[0121] Create a new meeting event;

[0122] Notify contact Zhang San (via WeChat or SMS).

[0123] Step 2: The system recognizes that the current interface is in desktop state and automatically starts the calendar application;

[0124] Step 3: The control tree identifies the "Add Event" button, "Time Selector", and "Contact Input Box";

[0125] Step 4: LLM generates the following operation instructions:

[0126] [{"action":"click","target":"add_event_btn"},{"action":"input","target":"time_field","parameter":"Tomorrow at 4 pm"},{"action":"input","target":"title_field","parameter":"Meeting with Zhang San"},{"action":"click","target":"Save button"},{"action":"Launch application","target":"WeChat"},{"action":"Send message","target":"Zhang San","parameter":"Meeting at 4 pm tomorrow"}];

[0127] Step 5: The system executes in sequence. If WeChat is not logged in, it triggers a rollback and prompts the user to operate.

[0128] The above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, but these descriptions should not be understood as limiting the scope of the present invention. The scope of protection of the present invention is defined by the accompanying claims, and any changes based on the claims of the present invention are within the scope of protection of the present invention.

Claims

1. A multi-task structure based on the fusion of semantic intent and interface state, comprising a semantic intent structuring module, an interface state modeling module, a multimodal joint reasoning module, and an exception feedback and dynamic correction module; the semantic intent structuring module inputs user voice / text instructions (such as "book a flight and share the itinerary with colleagues"); the semantic intent structuring module outputs structured data derived from the operational intent. The interface state modeling module inputs the real-time UI control tree (through AccessibilityService or UIAutomator , or other interfaces provided by the operating system After processing, the control hierarchy is converted into a standardized tree representation, including: Node attributes: control ID, text content, coordinate range, executable actions (click / input / scroll). The multimodal joint reasoning module inputs structured intent data and standardized UI state descriptions (LLM prompt engineering: output format is constrained by template. (For example: input format: [user instruction] + [current interface control tree]); The multimodal joint reasoning module output requirements:

1. The operation steps must match the current interface controls; 2. Each step format: {"Action Type":"click","Target Control":"Control ID or Text","Parameters":""} The exception feedback and dynamic correction module includes state verification, incremental regeneration, and fallback mechanisms. The state verification compares the LLM output instructions with the executableness of the current UI control. If the target control does not exist, it is marked as an exception; The incremental regeneration re-enters the exception mark and the updated UI state into the LLM, triggering local path correction; the fallback mechanism deletes invalid steps and replans subsequent operations when continuous correction failures exceed a threshold.

2. According to the multi-task structure based on the fusion of semantic intent and interface state in claim 1, the output structured data containing atomic operation intent includes but is not limited to the following examples: {"Action":"Book a flight","Parameters":{"Date":"2024-07-20","Destination":"Beijing"},"Related operations":"Share itinerary"}.

3. The multi-task structure based on the fusion of semantic intent and interface state according to claim 1, wherein the standardized tree includes but is not limited to the example fragment: {"id":"com.app:id / search_box","text":"", "actions":["click","input"]}.

4. According to the multi-task structure based on the fusion of semantic intent and interface state in claim 1, the format of each step output by the multimodal joint reasoning module includes but is not limited to the following example:

5. A multi-task generation method based on the fusion of semantic intent and interface state includes the following steps: (This step is a type of multi-task generation method used to automatically execute the user's natural language instruction "Share this song to Weibo and save it.") Step 1: User command input and semantic analysis The user enters a natural language command via voice or text: "Share this song on Weibo and save it." The system analyzes the command using the semantic intent parsing module and identifies two atomic tasks: [{"action":"share","target":"Weibo"},{"action":"save"}] Each atomic task contains an action type and corresponding target parameters, forming structured intent data. Step 2: Interface state collection and modeling The system obtains the interface control tree information of the current application through the auxiliary function interface (such as Android AccessibilityService or UIAutomator, or interfaces provided by other operating systems). The example interface is a music app playback page, and the control structure is as follows: {"screen":"Play Page","controls":[{"id":"player_title","text":"Sunny Day","actions":[]},{"id":"player_play_btn","text":"","actions":["click"]},{"id":"share_btn","text":"Share","actions":["click"]},{"id":"favorite_btn","text":"Favorite","actions":["click"]}]} The system converts the control tree into a standardized interface state description as one of the inputs of the large language model. Step 3: Joint reasoning to generate operation sequences The structured intent data and the standardized interface state description are jointly input into the Large Language Model (LLM). By prompting the engineering constraints model output format, an operation instruction sequence is generated, for example: [{"action":"click","target":"share_btn","parameters":""},{"action":"click","target":"Weibo icon","parameters":""},{"action":"click","target":"favorite_btn","parameters":""}] Each instruction contains action type, target control, and parameter information, with a clear structure and executable. Step 4: Task execution and status feedback The system executes the above instructions in sequence:

1. Click the "Share" button; 2. Find and tap the "Weibo icon" on the pop-up sharing interface; 3. Return to the main interface and click the "Favorites" button. During the execution process, the system verifies the execution status of each step of the operation, such as whether the control exists and whether it is clickable. Step 5: Exception handling and dynamic correction If a target control is missing during execution (e.g. the "Weibo icon" is not displayed), the system triggers the abnormal feedback mechanism:

1. Re-collect the current interface status; 2. Re-enter the exception mark and the updated interface state into the large language model; 3. Generate new alternative paths. For example, the model can be modified as follows: [{"action":"click","target":"share_btn","parameter":"parameter"},{"action":"click","target":"copy link","parameter":"parameter"},{"action":"launch app","target":"Weibo","parameter":"parameter"},{"action":"paste","target":"publish box","parameter":"parameter"}] If the number of consecutive correction failures exceeds the set number, the system will execute the task rollback mechanism, delete the invalid steps and prompt the user to intervene.

6. A multi-task generation method based on the fusion of semantic intent and interface state includes the following steps: (For example, the schedule creation and notification tasks include the following instructions: "Please arrange a meeting with Zhang San at 4 pm tomorrow and notify him.") Step 1: Semantic parsing output two-step task: Create a new meeting event; Notify contact Zhang San (via WeChat or SMS). Step 2: The system recognizes that the current interface is in desktop state and automatically starts the calendar application; Step 3: The control tree identifies the "Add Event" button, "Time Selector", and "Contact Input Box"; Step 4: LLM generates the following operation instructions: [{"action":"click","target":"add_event_btn"},{"action":"input","target":"time_field","parameter":"Tomorrow at 4 pm"},{"action":"input","target":"title_field","parameter":"Meeting with Zhang San"},{"action":"click","target":"Save button"},{"action":"Launch application","target":"WeChat"},{"action":"Send message","target":"Zhang San","parameter":"Meeting at 4 pm tomorrow"}]; Step 5: The system executes in sequence. If WeChat is not logged in, it triggers a rollback and prompts the user to operate.

Images