Agent-oriented interface information leakage reduction method based on size model and cloud collaboration

Abstract

The application discloses an intelligent agent interface information leakage reduction size model end cloud cooperation method, in the preprocessing stage of interface structure perception, the user interface is divided into semantic related structured blocks by analyzing the XML hierarchical structure; in the local-cloud cooperation planning stage, the local LLM proposes multiple subtask candidates according to the divided blocks, the cloud LLM selects or generates more accurate current subtasks based on the global understanding of all candidates; in the local-cloud cooperation decision-making stage, the local LLM performs preliminary screening and sorting on the interface blocks according to the subtasks, and the cloud LLM further performs fine-grained decision-making in the high correlation blocks; the application combines the strong reasoning ability of the cloud LLM and the privacy protection advantage of the local LLM, and realizes local-cloud cooperation mobile terminal task automatic processing, so that the intelligent agent only interacts with the key area of the user interface information, while guaranteeing the task execution efficiency and success rate, the unnecessary user interface information exposure is greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and particularly relates to a large model end-cloud collaborative method for interface information leakage reduction of intelligent agent. BACKGROUND

[0002] With the popularity of smart phones, users have an increasing demand for automatically completing operation tasks on mobile terminals. Task automation technology aims to enable intelligent agents to autonomously complete a series of operation tasks according to natural language instructions of users, thereby improving user experience and reducing the operation burden of users. In recent years, with the development of large language model (LLM) technology, various mobile terminal operation agents based on LLM in the prior art can simulate human operation processes, gradually perceive interface states, plan subtasks, and interact with a graphical user interface (GUI) until the task is completed. In actual application, it is usually necessary to upload complete interface information (such as an XML structure or a screenshot) to a cloud model for decision-making in each operation. This process causes unnecessary leakage of user privacy information. SUMMARY

[0003] The present application proposes a large model end-cloud collaborative method for interface information leakage reduction of intelligent agent, which combines the strong reasoning ability of cloud LLM and the privacy protection advantage of local LLM to realize local-cloud collaborative mobile terminal task automation processing, so that the intelligent agent only interacts with the key area on the user interface information, greatly reduces unnecessary exposure of user interface information while ensuring the efficiency and success rate of task execution. Experiments show that the user interface information exposure amount is reduced by 55.60%, the task success rate is close (the difference is controlled within 5%), the cloud side reasoning time is reduced by 19.16%, and the task success rate is increased by 36.36%.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a large model end-cloud collaborative method for interface information leakage reduction of intelligent agent, comprising:

[0006] 1) A preprocessing stage of interface structure perception, which divides the user interface into semantically related structured blocks by analyzing the XML hierarchical structure;

[0007] 2) A local-cloud collaborative planning stage, which proposes multiple subtask candidates by the local LLM according to the divided blocks, and selects or generates a more accurate current subtask based on the global understanding of all candidates by the cloud LLM;

[0008] 3) Local-Cloud Collaborative Decision Phase, Local LLM performs initial screening and sorting on interface blocks according to sub-tasks, and Cloud LLM further performs fine-grained decision within high-relevance blocks.

[0009] When the Cloud LLM determines that the current information is insufficient to support reliable decision, it requests the Local LLM for more context content, achieving a trade-off between guaranteeing task success rate and reducing information exposure.

[0010] The present application relates to a system for implementing the above-mentioned collaborative method, comprising: a user interface block unit, a local sub-task generation unit, a cloud sub-task screening and correction unit, a local user interface block sorting and screening unit, a multi-round information accumulation control unit, a cloud decision unit, and an execution unit, wherein: the user interface block unit obtains the XML tree structure of the current interface, extracts important interactive elements with semantic information from it, and groups them by analyzing their common ancestor nodes in the tree structure, obtaining at least three semantically coherent UI blocks; the local sub-task generation unit generates candidate sub-tasks that may be executed within the current block without involving specific interface elements, independently for each UI block, by the local LLM in combination with user task description and historical operation; the cloud sub-task screening and correction unit receives all candidate sub-tasks generated by the local LLM through the cloud LLM, and judges the most suitable sub-task in combination with global task description and historical operation; the local user interface block sorting and screening unit scores and sorts all UI blocks according to their relevance to the sub-task in importance according to the current most suitable sub-task by the local LLM; the multi-round information accumulation control unit judges whether the information of the first round upload and the first importance score and sorting of the UI block is sufficient to complete the current sub-task by the cloud LLM; if the information is insufficient, the next round of upload mechanism is triggered to select the sub-optimal block from the remaining blocks to continue uploading, until the cloud LLM determines that there is enough information to support reliable decision; the cloud decision unit performs fine-grained analysis and decision on the received UI block set by the cloud LLM to determine the specific target UI element and its interaction action; the execution unit receives the final decision and automatically executes it on the user's mobile device to complete the current step; after execution, the system records the operation and prepares for the next round of interaction until all sub-tasks are completed.

[0011] Technical effects

[0012] The application generalizes the block information into subtask description which can be used to guide the operation, without exposing the specific content to the cloud; a local-cloud collaborative decision-making mechanism: the local LLM condenses the UI block information into subtasks and uploads it, and the cloud LLM confirms the optimal subtask, and the local LLM sorts each block according to its importance to complete the subtask, which is provided for the cloud LLM to perform fine-grained decision-making only in the most relevant blocks. Compared with the prior art, the interface division method based on XML helps the LLM to deeply understand the partition structure of the UI interface, discards the traditional visual perception, and realizes block division according to the tree-like hierarchical logic of the UI design itself, laying a foundation for the LLM to accurately perceive the key blocks and eliminate irrelevant information later; the UI block semantic condensing method based on the local LLM effectively protects the specific content privacy of the user interface, avoids direct uploading of sensitive information, but transmits key information to the cloud-side LLM through the condensed subtask description, so that it has a better perception of the current page state and makes better planning, making up for the insufficient planning ability of the local model; the local-cloud collaborative task execution mechanism takes into account the privacy protection ability of the local LLM and the reasoning advantage of the cloud LLM, and through block screening, only uploads the interface information most relevant to the current task, so that the cloud LLM can focus on the key area and carry out high-quality reasoning. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a flowchart of the present application;

[0014] Figure 2 is a specific example diagram in an actual application scenario. DETAILED DESCRIPTION

[0015] As shown in Figure 1 , a local-cloud collaborative method for reducing interface information leakage of an intelligent agent is involved, which includes:

[0016] Step 1, preprocessing stage of interface structure perception, the user interface is divided into structured blocks with semantic relevance by analyzing the XML hierarchical structure, specifically including:

[0017] 1.1. Use the uiautomator2 tool to extract the XML file containing the attributes and hierarchical structure of each UI element in the current user interface from the mobile device;

[0018] 1.2. Parse the XML file to obtain the corresponding XML tree structure, and extract the important nodes that meet the conditions through depth-first traversal, specifically for meeting any of the following conditions: a. clickable and containing description information b. editable c. containing text, description, hint semantic information field;

[0019] 1.3. In the traversal process of step 1.2, all nodes are assigned unique numbers, and the ancestor paths of important nodes are recorded in the form of , where: is the root node number, is the direct parent node number of the important node ;

[0020] 1.4. The ancestor path set of all important nodes is traversed from depth to the maximum path depth ;

[0021] 1.5. At each depth , the nodes are divided according to their layer ancestor numbers , and important nodes with the same ancestor are divided into the same block, and important nodes with different ancestors at this depth are divided into different blocks.

[0022] 1.6. If the number of blocks divided at a certain depth is not less than three, the division result at this depth is selected as the final block division result;

[0023] 1.7. The important nodes in each block are arranged in depth-first order according to their positions in the XML tree, and their properties in the original XML tree structure are preserved, including text content, description, class name, resource identifier, and bounding box position coordinate information;

[0024] 1.8. All the divided user interface blocks are sequentially numbered to obtain , where each represents a block composed of corresponding important nodes , i.e. .

[0025] Step 2, local-cloud collaborative planning phase, the local LLM proposes multiple sub-task candidates for each block divided by the division method, and the cloud LLM selects or generates more accurate current sub-tasks based on the global understanding of all candidates, including:

[0026] 2.1. For each UI block obtained by the block division method, combine it with the user-provided task description and the completed historical operation sequence to form the input and send it to the local LLM;

[0027] 2.2. The local LLM determines task progress and outputs a logical subtask that may be executed within the current block. Specifically, it must simultaneously meet the following conditions: a. Based only on the current block information. The conclusions are: a) It does not rely on information from other blocks; b) It does not disclose any specific UI element information; c) It expresses logical intent in natural language, such as "Open settings" or "Enter query content".

[0028] 2.3. Repeat steps 2.1-2.2 to obtain the set of candidate subtasks corresponding to all blocks. ;

[0029] 2.4. Set up the candidate subtasks Along with user task description and historical operation sequence Combined as input Sending the data to the cloud-based LLM system for planning includes:

[0030] i) The cloud-based LLM infers the functional distribution of the overall user interface based on the candidate subtasks in each block, ensuring strong contextual consistency and actual executability of task planning even when specific UI information cannot be obtained.

[0031] ii) Cloud-based LLM leverages its strong planning capabilities to comprehensively analyze the set of candidate subtasks. Choose the most reasonable subtask from them. As the subtask that should be executed at present;

[0032] iii) If set The candidate subtasks in the model are all unreasonable; the cloud-based LLM can generate a new subtask based on its own reasoning and analysis. .

[0033] Step 3: Local-Cloud Collaborative Decision-Making Phase. The local LLM performs initial screening and sorting of interface blocks based on sub-tasks, while the cloud LLM further performs fine-grained decisions within highly relevant blocks, specifically including:

[0034] 3.1. The local LLM is based on the previously partitioned set of user interface blocks. and the current sub-task derived from collaborative planning For each block Perform an importance assessment and generate a corresponding importance score. Specifically: That is, the total score generated by the large model is 1.

[0035] 3.2. Based on the scores, rank all blocks by importance, resulting in a block sequence. ,in: The block with the highest score is most likely to contain the key information needed to solve the subtask;

[0036] 3.3. The local LLM uploads the block with the highest score to the cloud LLM , which combines the complete task description , historical interaction records , and the current block content to assess whether there is enough information to make a decision. If the cloud LLM determines that the current block information is sufficient, it makes a decision and directly jumps to step 3.6.

[0037] 3.4. If the cloud LLM determines that the current block information is insufficient to support a reliable decision, it requests the next highest-scoring block from the local LLM that has not yet been uploaded and continues to upload it to the cloud and merge it with the received block information.

[0038] 3.5. After each new block is uploaded, the cloud LLM re-evaluates the decision based on the current accumulated block information. This process uses a progressive information accumulation mechanism until the cloud LLM is confident that the current information is sufficient to make an accurate and executable decision, or all blocks have been uploaded. This mechanism effectively balances the reduction of uploaded information and the accuracy of the decision. Even if the local LLM initially sorts incorrectly or the task is complex, the cloud LLM can still obtain sufficient and necessary user interface information through multiple iterations to make the correct decision.

[0039] 3.6. The final decision result of the cloud LLM is , where: is the action type (click, long press, or text input) for interacting with the user interface, is the specific UI element number to interact with, is the input content (valid when is text input, and for the rest of the cases);

[0040] If the cloud LLM cannot make an effective decision after all blocks are uploaded, it indicates that it believes the information contained in the current user interface is insufficient to complete the task. At this time, the intelligent agent will automatically perform a downward swipe operation to obtain a new user interface state and upload it for the cloud LLM to continue decision-making attempts.

[0041] The sliding, uploading, and decision-making process can be repeated up to 5 times to avoid infinite sliding. If an effective decision result is still not obtained after the sliding operation reaches the upper limit, it is determined that the cloud LLM cannot complete the current task, and the system terminates the task flow.

[0042] As Figure 1 shown, the information security protection system for implementing the above method in the embodiment includes a user interface block division unit, a local subtask generation unit, a cloud subtask screening and correction unit, a local user interface block sorting screening unit, a multi-round information accumulation control unit, a cloud decision unit, and an execution unit.

[0043] The user interface block division unit includes a structure extraction module and a block division module, wherein: the structure extraction module parses the hierarchical structure and attributes of the user interface elements according to the XML data of the current page to obtain the processed UI tree structure; and the block division module traverses the UI tree obtained by the structure extraction module, identifies important interactive nodes, and groups them based on their ancestor paths in the tree to output at least three semantically coherent UI blocks.

[0044] The local subtask generation unit includes a context fusion module and a semantic condensation module, wherein: the context fusion module takes the user's current task description and historical operation sequence as input, and takes the content of each block as the input context of the local model; and the semantic condensation module generates executable subtask descriptions within the corresponding block through the local LLM, without involving specific UI element information, to ensure that the user interface privacy is not uploaded.

[0045] The cloud subtask screening and correction unit includes a context fusion module and a subtask screening and correction module, wherein: the context fusion module is responsible for receiving all subtask candidate descriptions generated by the local LLM, and combining the complete task description and historical operation record as the input context of the cloud LLM; and the subtask screening and correction module evaluates the rationality and feasibility of each subtask candidate based on the input context, selects the most suitable one as the final subtask, and if there is no candidate that meets the task requirements, the cloud LLM corrects or regenerates the subtask to ensure that the subtask has clear operation guidance and meets the current task promotion goal.

[0046] The local user interface block sorting screening unit includes a task association analysis module and a block sorting module, wherein: the task association analysis module analyzes the importance of each block content for completing the current subtask according to the confirmation of the cloud LLM, and scores in turn; and the block sorting module sorts all UI blocks according to the scoring of the local LLM to determine the importance ranking and upload priority of each block.

[0047] The multi-round information accumulation control unit comprises an information sufficiency judgment module and a block progressive upload module, wherein the information sufficiency judgment module analyzes the first-ranked high-priority UI block that has been uploaded by the cloud LLM, judges whether the block content is sufficient to support the cloud LLM to complete the sub-task decision, and if the information is insufficient, the block progressive upload module selects the second-best block from the remaining unuploaded blocks in the order of the locally generated priority order for uploading until the information sufficiency judgment module considers that the current content has met the needs of cloud decision.

[0048] The cloud decision unit comprises an information aggregation module and an action decision module, wherein the information aggregation module integrates the UI block content uploaded to the cloud and the current sub-task to construct context information for decision-making, and the action decision module completes fine-grained interactive decision-making through the cloud LLM, that is, determines the target UI element and the type of interaction to be performed.

[0049] The execution unit comprises an operation execution module and a state updating module, wherein the operation execution module receives the cloud decision output, calls ADB and other tools to control the mobile device to perform corresponding operations (click, input, long press), and the state updating module records the current operation to the interaction history and automatically enters the next round of task processing process until the task is completed.

[0050] The embodiment is specifically verified in the following environment configuration: the local end uses a consumer personal computer with NVIDIA GeForce RTX 4090D graphics card, deploys the quantized local large language model (Gemma2-9B-Instruct, Qwen 2.5-7B-Instruct, LLaMA 3.1-8B-Instruct), and runs through the Ollama tool. The cloud model uses GPT-4o (2024-11-20 version) and is called through the official API interface of OpenAI. The test terminal includes a real mobile phone (Honor Play3) running Android 9 (suitable for DroidTask dataset) and a Pixel 7 Pro emulator running Android 13 (suitable for AndroidLab dataset). The user interface state is extracted in the form of an XML file, obtained using the uiautomator2 tool, and the interactive operation is realized through the Android Debug Bridge (ADB).

[0051] Under the above settings, comparative experiments are carried out by different methods, and the task success rate and UI information upload reduction rate are counted. The experimental data are shown in Table 1.

[0052] Table 1

[0053]

[0054] The complete workflow of the mobile terminal intelligent agent system of the present embodiment is as follows:

[0055] Step 1. First, connect the mobile smart phone to the local personal computer with basic computing capability. Deploy a local LLM on the computer, such as running the Gemma2-9B model through the Ollama tool. If the video memory resource is insufficient, a quantized version can be selected to reduce resource requirements. For closed-source cloud LLM deployed on a cloud server, select the corresponding paid interface for calling, such as the GPT-4o API interface provided by OpenAI;

[0056] Step 2. The user inputs a specific task described in natural language on the computer and specifies the target application program. The mobile intelligent agent system uses the Android Debug Bridge (ADB) tool to start the target application program and execute the command adb shell am start -n com.example.app / .MainActivity <mainactivity>Enter the application main interface;

[0057] Step 3. After the application is started, the mobile intelligent agent extracts the XML file of the current user interface and performs UI block division, which serves as the input for subsequent collaborative processing. The specific steps have been described in detail in the foregoing;

[0058] Step 4. Local-cloud collaborative planning and decision-making. The local LLM is responsible for performing block sorting and preliminary screening, and uploading the block with the highest score to the cloud LLM. The cloud LLM makes fine-grained decisions based on task descriptions, interaction history and block content. The specific steps have been described in the foregoing;

[0059] Step 5. After receiving the decision of the cloud LLM, the mobile intelligent agent uses the ADB tool to perform the corresponding interaction operation: a. If the decision is a click operation, execute: adb shell input tap <x> <y>; b. If it is a long press operation, lengthen the time of clicking the target element bounding box center coordinate: adb shell input touchscreen swipe <x> <y> <x> <y> <duration>; c. If it is a text input operation, it is necessary to first empty the input box content and then input, which can be completed by using the following command combination: adb shell am broadcast -a ADB_CLEAR_TEXT, adb shell am broadcast -a ADB_INPUT_TEXT -es msg <input_text>, which requires pre-installing ADBKeyBoard on the mobile smart phone, which supports Chinese input.

[0060] Step 6. After each round of interaction of the mobile intelligent agent, the interaction operation is recorded and the interaction history is updated, and the user interface state is also updated due to the interaction. The next round continues to perform steps 3, 4 and 5, and after multiple rounds of interaction, it is judged that there is no sub-task to be completed after local-cloud LLM collaborative planning in a certain round, the user task is completed, and the process is terminated.

[0061] Compared with the prior art, the present application divides the UI block according to the UI XML tree structure in the preprocessing stage, that is, the UI block with high semantic aggregation degree is divided according to the UI structure, so that the subsequent processing focuses on the block-level semantic information, avoiding complex screening at the element level, while ensuring the task completion rate, significantly reducing the interface information upload volume. The actual measurement shows that compared with the pure cloud LLM method, the present application can reduce the interface upload content by 55.60%, significantly enhancing the user data privacy protection ability; secondly, the present application condenses the UI block semantic based on the local large language model, and completes the condensation of the content of each block on the local, and summarizes it into a sub-task description with operation guidance significance, and replaces the specific UI content to send to the cloud, ensuring privacy while realizing information transmission. This mechanism enables the cloud LLM to determine the task progress reflected by the current UI without directly accessing the original UI content, and to determine the reasonable sub-task. Thirdly, the present application uses the local-cloud collaborative decision mechanism, and the local model sorts and filters each block according to the sub-task confirmed by the cloud, and only uploads the most relevant block to the cloud, reducing the interference of redundant information and significantly improving the reasoning efficiency of the cloud model. The actual measurement data shows that the cloud decision time is reduced by 19.16%.

[0062] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraints of the present application.< / duration> < / y> < / x> < / y> < / x> < / y> < / x> < / mainactivity>

Claims

1. An agent-oriented interface information leakage reduction and size model end-cloud collaboration method, characterized in that, In the preprocessing stage of interface structure perception, the user interface is divided into semantic-related structured blocks by analyzing the XML hierarchy; in the local-cloud collaborative planning stage, the local LLM proposes multiple sub-task candidates based on the divided blocks, and the cloud LLM selects or generates more accurate current sub-tasks based on the global understanding of all candidates; in the local-cloud collaborative decision-making stage, the local LLM performs preliminary screening and sorting of the interface blocks, and the cloud LLM further makes fine-grained decisions within the high-relevance blocks; When the cloud LLM judges that the current information is insufficient to support reliable decision-making, it requests the local LLM for more context content, achieving a trade-off between ensuring task success rate and reducing information exposure.

2. The method of claim 1, wherein the method is characterized in that, The preprocessing stage includes: 1.

1. Use the uiautomator2 tool to extract the XML file containing the attributes and hierarchy of each UI element in the current user interface from the mobile device; 1.

2. Parse the XML file to obtain the corresponding XML tree structure, and extract the important nodes that meet the following conditions through depth-first traversal: a. clickable and containing description information b. editable c. containing text, description, and hint semantic information fields; 1.

3. In the traversal process of step 1.2, all nodes are assigned unique numbers, and the ancestor path of important nodes is recorded in the form of where: is the root node number, is the direct parent node number of the important node . 1.

4. Set of ancestor paths to all important nodes , in turn, from depth to maximum path depth ; 1.

5. At each depth , divide the nodes by their first layer ancestor number , and put the nodes with the same ancestor into the same block, and the nodes with different ancestors into different blocks. 1.

6. If the number of blocks divided at a certain depth is not less than three, select the division result at that depth as the final block division result; 1.

7. Arrange the important nodes in each block according to their depth-first order in the XML tree, and retain their attributes in the original XML tree structure, including text content, description, class name, resource identifier, and bounding box position coordinate information; 1.

8. Number all the divided user interface blocks in order, obtaining , each representing a block, which is composed of the corresponding important node , that is .

3. The method of claim 1, wherein the method further comprises: The local-cloud collaborative planning stage specifically includes: 2.

1. For each UI block resulting from the block division method combining it with the user-provided task description the completed historical operation sequence as input to the local LLM; 2.

2. Local LLM judges task progress, outputs a logical sub-task that can be executed in the current block , Specifically, it meets the following conditions: a. Based only on current block information , not dependent on other block information b. Does not leak any specific UI element information c. Express logical intent in natural language; 2.

3. Repeat step 2.1-step 2.2 to obtain the candidate subtask set corresponding to all the blocks ; 2.

4. Set up the candidate subtasks Along with user task description and historical operation sequence Combined as input Sending the data to the cloud-based LLM system for planning includes: i) The cloud LLM infers the functional distribution of the overall user interface based on the candidate sub-tasks in each block, ensuring that the task planning has strong context consistency and actual executability even in the absence of specific UI information; ii) the cloud LLM uses its stronger planning capabilities to analyze the candidate subtask set and select the most reasonable subtask from among them as the current subtask to be executed; iii) If none of the candidate sub-tasks in the set is reasonable, the cloud LLM can generate a new sub-task based on its own reasoning and analysis .

4. The method of claim 1, wherein the method further comprises: The local-cloud collaborative decision-making stage specifically includes: 3.

1. The local LLM is based on the previously divided user interface block set And the current subtask derived by collaborative planning For each block Importance evaluation is carried out to generate its corresponding importance score Specifically: That is, the score generated by the large model is 1; 3.

2. Forming a ranking of the importance of all the tiles according to the score results, the ranked sequence of tiles wherein: is the tile with the highest score, most likely containing the key information needed to solve the subtask; 3.

3. The local LLM will select the block with the highest score. Upload to the cloud-based LLM library, and combine the cloud-based LLM library with the complete task description. Historical interaction records And the current block content, first assess whether there is enough information to complete this decision. If the cloud LLM determines that the current block information is sufficient, then make a decision and jump directly to step 3.

6. 3.

4. If the cloud LLM determines that the current block information is insufficient to support a reliable decision, request the next highest scoring but not yet uploaded block from the local , continue uploading to the cloud and merging with the received block information; 3.

5. After uploading a new block, the cloud LLM re-decides based on the accumulated information of multiple blocks, using a progressive information accumulation mechanism until the cloud LLM is confident that the information is sufficient, making an accurate and executable decision, or all blocks have been uploaded. This mechanism effectively balances the reduction of uploaded information and the accuracy of decision-making, even in cases where the local LLM's initial sorting and screening is incorrect or the task itself is complex. The cloud LLM can still obtain sufficient and necessary user interface information through multiple iterations to make correct decisions; 3.

6. Cloud LLM final decision result is where: is the action type (click, long press, or text input) to interact with the user interface, is the specific UI element number to interact with, is the input content (when is valid, and the rest is ).

5. A system for implementing the synergic method of any of claims 1-4, characterized in that, It includes: The user interface partitioning unit, the local subtask generation unit, the cloud subtask screening and correction unit, the local user interface block sorting screening unit, the multi-round information accumulation control unit, the cloud decision unit, and the execution unit, wherein: the user interface partitioning unit obtains an XML tree structure of a current interface, extracts important interactive elements with semantic information from the XML tree structure, and groups the important interactive elements by analyzing common ancestor nodes of the important interactive elements in the tree structure to obtain at least three semantically coherent UI blocks; the local subtask generation unit independently generates candidate subtasks that can be executed in a current block without involving specific interface elements, by a local LLM in combination with a user task description and historical operations, for each UI block; the cloud subtask screening and correction unit receives all candidate subtasks generated by the local LLM through a cloud LLM, and judges a current most suitable subtask in combination with a global task description and historical operations; the local user interface block sorting screening unit performs importance scoring and sorting on all UI blocks according to a correlation with the subtask by the local LLM according to the current most suitable subtask; the multi-round information accumulation control unit judges whether information of a UI block that is uploaded in a first round and has a first importance score is sufficient to complete the current subtask by the cloud LLM; if the information is insufficient, a next round uploading mechanism is triggered to select a suboptimal block from remaining blocks to continue uploading until the cloud LLM judges that there is sufficient information to support a reliable decision; the cloud decision unit performs fine-grained analysis and decision on the received set of UI blocks by the cloud LLM to determine specific target UI elements and interaction actions; and the execution unit receives a final decision and automatically executes on a user mobile device to complete a current step; after the execution, the system records the operation and prepares to enter a next round of interaction until all subtasks are completed.

6. The system of claim 5, wherein, The user interface partitioning unit comprises a structure extraction module and a block division module, wherein: the structure extraction module parses a hierarchical structure and attributes of user interface elements according to XML data of a current page to obtain a processed UI tree structure; and the block division module traverses the UI tree obtained by the structure extraction module, identifies important interactive nodes, and groups the important interactive nodes based on ancestor paths of the important interactive nodes in the tree to output at least three semantically coherent UI blocks.

7. The system of claim 5, wherein, The local subtask generation unit comprises a context fusion module and a semantic condensation module, wherein: the context fusion module takes a user current task description and a historical operation sequence as input, and combines content of each block as an input context of a local model; and the semantic condensation module generates a subtask description that can be executed in a corresponding block by the local LLM without involving specific UI element information, to ensure that user interface privacy is not uploaded. The cloud subtask screening and correction unit comprises a context fusion module and a subtask screening and correction module. The context fusion module is responsible for receiving all subtask candidate descriptions generated by the local LLM, and combining the complete task description and the historical operation record as the input context of the cloud LLM. The subtask screening and correction module evaluates the rationality and feasibility of each subtask candidate on the basis of the input context, selects the most suitable one as the final subtask, and if no candidate meets the task requirements, the cloud LLM corrects or regenerates the subtask to ensure that the subtask has clear operation guidance and meets the current task promotion target.

8. The system of claim 5, wherein, The local user interface block sorting and screening unit comprises a task association analysis module and a block sorting module. The task association analysis module analyzes the importance of each block content for completing the subtask according to the current subtask confirmed by the cloud LLM, and scores in turn. The block sorting module sorts according to the score of the local LLM on all UI blocks, determines the importance order and upload priority of each block. The multi-round information accumulation control unit comprises an information sufficiency judgment module and a block progressive upload module. The information sufficiency judgment module analyzes the first-ranked high-priority UI block uploaded by the cloud LLM, and judges whether the block content is sufficient to support the cloud LLM to complete the subtask decision. If the information is insufficient, the block progressive upload module selects the second-best block from the remaining unuploaded blocks in order according to the priority order generated locally, and uploads until the information sufficiency judgment module considers that the current content meets the needs of cloud decision.

9. The system of claim 5, wherein, The cloud decision unit comprises an information aggregation module and an action decision module. The information aggregation module integrates the UI block content uploaded to the cloud and the current subtask to build the context information for decision. The action decision module completes the fine-grained interactive decision through the cloud LLM, that is, it determines the target UI element and the type of interaction to be performed.

10. The system of claim 5, wherein, The execution unit comprises an operation execution module and a state updating module. The operation execution module receives the cloud decision output, calls the ADB tool to control the mobile device to perform the corresponding operation, such as clicking, inputting and long pressing. The state updating module records the current operation to the interaction history, automatically enters the next round of task processing process, and continues until the task is completed.

Images