Android side task automation method and computer readable storage medium

Abstract

The application relates to the technical field of mobile terminal human-computer interaction and task automation, and provides an Android terminal side task automation method and a computer readable storage medium. The method acquires a natural language task input by a user and performs semantic embedding, and searches a cache task according to the task semantic embedding; when cache hits, a difference patch is generated based on a cache abstract syntax tree and task differences, and a target executable script is obtained by merging; when cache misses, structural semantics, visual semantics and time sequence transition information of an Android interface are fused, a large language model is used to generate a target executable script with a control flow; the target executable script is executed on an Android terminal device side, and when an execution exception occurs, the interface state is re-sensed and the script is corrected. Through cache reuse, incremental script generation and interface information compression, the number of complete model reasoning times and the amount of input data are reduced, the terminal side calculation cost and power consumption are reduced, the task response time is shortened, and the task execution efficiency of the Android terminal is improved.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and mobile computing, specifically relating to human-computer interaction and task automation technology on mobile terminals, and particularly to a task automation method on the Android side and a computer-readable storage medium. Background Technology

[0002] As mobile applications increasingly support more functions, users frequently need to perform repetitive operations across pages, multiple steps, and even across applications on Android devices. In recent years, large language models have demonstrated strong natural language understanding and code generation capabilities, leading researchers to begin converting natural language tasks into mobile interface operations. Existing solutions typically obtain the Scalable Markup Language (SML) hierarchy tree of the Android interface, screenshots, or optical character recognition results, and then the model progressively predicts actions such as clicks, inputs, and swipes; some solutions use visual language models to directly predict click coordinates.

[0003] For example, authorized invention patent CN119342136B discloses an intelligent control method and system for Android phones based on a large-scale intelligent model. This method can acquire natural language tasks, Android page hierarchy information, and screenshots. It compresses the page information and completes it using optical character recognition (OCR), then the model outputs the operation type and location, which is executed through accessibility services. AutoDroid generates interface transition information through offline exploration and uses HTML-like text to assist the large language model in decision-making. The related LLM mobile interface dialogue framework primarily outputs action commands, and the visual language model-based solution can directly predict click coordinates. These technologies can reduce the configuration cost of fixed-rule scripts, providing a foundation for the automation of natural language-driven mobile tasks.

[0004] However, existing technologies still have the following shortcomings: First, the method of progressively predicting discrete actions usually lacks program control capabilities such as conditional branches, loops, and exception handling. When the task involves batch processing, conditional judgment, or the interface state deviates from expectations, it is easy to cause repeated operations, long waiting times, or task interruptions. Second, extensible markup language information and screenshot and optical character recognition information are often processed separately, lacking a mechanism to actively adjust the perception method based on the interface's historical state. When non-standard rendering interfaces such as Flutter, Unity, and Canvas appear suddenly, or when pop-ups suddenly appear, the interface semantics may be incomplete. Third, similar tasks usually still require a relatively complete large model inference to be performed again. Historically successful scripts are difficult to reuse incrementally at the code structure level, which increases inference latency, computational load, and model input overhead when deployed on the edge.

[0005] Furthermore, directly executing clicks based on visual coordinates is easily affected by differences in terminal resolution and dynamic interface reordering; shifts in coordinate position reduce operational stability. Therefore, there is a need for an Android-side task automation method that can reuse historical executable script structures, integrate structural semantics, visual semantics, and interface timing information when the cache is missed, and generate control flow with support for exception recovery, in order to adapt to complex dynamic interfaces and long-sequence tasks. Summary of the Invention

[0006] The technical problem to be solved by this invention is that existing Android client-side task automation methods have insufficient perception information under complex dynamic interfaces, can only output discrete actions step by step and lack control flow and exception recovery, and cannot perform code-level incremental reuse of similar tasks, resulting in high inference overhead.

[0007] In a first aspect, the present invention provides an Android-side task automation method, the method comprising the following steps: Obtain the natural language task input by the user, perform semantic embedding on the natural language task, and retrieve the cached task based on the obtained task semantic embedding; When a cached task is found that meets the preset semantic similarity threshold between the natural language task and the cached task, the cached abstract syntax tree corresponding to the cached task is obtained. Based on the task differences between the natural language task and the cached task, the nodes to be edited in the cached abstract syntax tree and the corresponding abstract syntax tree editing operations are determined to form a difference skeleton. Based on the difference skeleton, the difference patch is obtained and the difference patch is merged with the cached abstract syntax tree to generate a target executable script for driving the Android application to perform natural language tasks. When a cached task with semantic similarity to a natural language task that does not meet a preset semantic similarity threshold is not hit, the Extensible Markup Language (XML) hierarchy tree of the current display interface of the Android terminal device is obtained through Android Accessibility Service or Android Debug Interface, along with a screenshot of the current display interface and historical interface states. The XML hierarchy tree is compressed to obtain a structural semantic representation, and interface temporal change information is determined based on historical interface states. When the number of interactive nodes in the structural semantic representation is lower than a preset node threshold, or when a sudden change in the current display interface is determined based on the interface temporal change information, visual semantic backtracking is performed on the screenshot to obtain visual semantic anchors, and the structural semantic representation is enhanced using these visual semantic anchors. A fused interface state is formed based on the structural semantic representation, interface temporal change information, and the visual semantic anchors obtained when visual semantic backtracking is triggered. The fused interface state, natural language task, and interface description of Android operation primitives are input into a large language model with code generation capabilities. The large language model generates a target executable script that calls Android operation primitives and has control flow, and caches the abstract syntax tree of the target executable script. On the Android terminal device side, predefined Android operation primitives are called to execute the target executable script, thereby driving the target application on the Android terminal device to perform interactive operations corresponding to the natural language task. When an exception occurs during the execution of the target executable script, the current interface state of the Android terminal device is retrieved again, and the target executable script is regenerated or corrected based on the exception information and the re-retrieved current interface state before continuing execution. The exception information and the corresponding regeneration or correction information are recorded as recovery information until the natural language task is completed or the preset number of replanning attempts is reached.

[0008] Optionally, compressing the Extensible Markup Language (Extreme Markup Language) hierarchical tree yields a structural semantic representation, including: Traverse the Extensible Markup Language hierarchy tree, removing nodes with the invisible attribute set to true, nodes with zero opacity, nodes with zero width, nodes with zero height, and pure layout container nodes. Merge consecutive non-interactive leaf nodes that share the same clickable ancestor into a single entry, while preserving the node identifier, resource identifier, text, content description, bounding box, clickable attribute, and scrollable attribute; Based on the merged entries and the retained attributes, a structured text in HTML format is generated, and the structured text is used as a structural semantic representation.

[0009] Optionally, visual semantic backtracking is performed on the screenshot to obtain visual semantic anchors, and the structural semantic representation is enhanced using the visual semantic anchors, including: Optical character recognition is performed on the screenshot to obtain text blocks and corresponding bounding boxes. The icons in the screenshot are classified and recognized using a visual model. Based on the text block and icon classification and recognition results, visual semantic anchors containing bounding boxes, semantic labels and confidence scores are generated. Associate visual semantic anchors with corresponding nodes in the Extensible Markup Language (XML) hierarchy tree, and write visual saliency attributes and visual semantic labels to the corresponding nodes; if no node corresponding to a visual semantic anchor exists in the XML hierarchy tree, create a virtual node and write visual saliency attributes and visual semantic labels to the virtual node. The structural semantic representation is updated based on the node information written with the visual saliency attribute and visual semantic label, so that the structural semantic representation contains the visual semantic information corresponding to the visual semantic anchor point, resulting in an enhanced structural semantic representation. The enhanced structural semantic representation is then used as the structural semantic representation for forming the fusion interface state.

[0010] Optionally, interface temporal change information is determined based on historical interface states, and a fused interface state is formed based on structural semantic representation, interface temporal change information, and visual semantic anchor points when visual semantic backtracking is triggered, including: Record the interface state fingerprint of a preset number of frames and the corresponding previous action sequence, and generate the interface state fingerprint based on the set of visible text and node topology of the currently displayed interface. When the number of times a fingerprint appears repeatedly in the same interface state exceeds the preset number of loops, the execution process of the natural language task is determined to have entered a loop state. When the increase in the number of nodes in the current display interface relative to the number of nodes in the previous display interface exceeds a preset mutation threshold, a mutation is determined to have occurred in the current display interface. When a mutation is determined, the scanning depth of the extensible markup language hierarchy tree is increased, and visual semantic backtracking is triggered. The interface state fingerprint, the previous action sequence, the loop state determination result, and the interface mutation determination result are determined as interface temporal change information. When visual semantic backtracking is triggered, structural semantic quality features are determined based on node coverage and semantic density, visual semantic quality features are determined based on optical character recognition confidence and icon classification confidence, and temporal transition quality features are determined based on the degree of interface state change. Based on the structural semantic quality features, visual semantic quality features, and temporal change quality features, determine the structural semantic fusion weight, visual semantic fusion weight, and temporal change fusion weight. Then, based on the structural semantic fusion weight, visual semantic fusion weight, and temporal change fusion weight, fuse the corresponding structural semantic representation, visual semantic anchor point, and interface temporal change information to obtain the fused interface state.

[0011] Optionally, the caching system used by the Android-side task automation method includes application-level template caching, session-level adaptation caching, cross-session migration caching, and abstract syntax tree differential caching; cached tasks are retrieved from application-level template caching and / or session-level adaptation caching based on task semantic embedding. The application-level template cache uses the application package name and task semantic cluster tag as the retrieval key and stores the abstract syntax tree template containing variable placeholders. The session-level adaptation cache uses the nearest neighbor task corresponding to the task semantic embedding of the current natural language task as the retrieval basis, and stores the instantiated executable script and its corresponding abstract syntax tree; Cross-session migration cache uses cross-application intent patterns as retrieval keys and stores the corresponding data pipeline descriptors; Abstract Syntax Tree Differential Cache uses the hash value of the Abstract Syntax Tree edit operation mode as the retrieval key and stores the corresponding difference template; Difference patches are obtained based on the difference skeleton, including: When the abstract syntax tree editing operation mode corresponding to the difference skeleton hits the abstract syntax tree difference cache, the corresponding difference template is obtained from the abstract syntax tree difference cache as a difference patch; wherein, the abstract syntax tree editing operation includes at least one of leaf node replacement, subtree insertion, and subtree deletion; When the abstract syntax tree difference cache is not hit, the difference skeleton is input into a large language model with code generation capabilities to obtain a difference patch, and the difference template corresponding to the difference patch is stored in the abstract syntax tree difference cache.

[0012] Optionally, Android operation primitives include at least one of the following: click operation, text input operation, scroll operation, wait operation, swipe operation, system button operation, screenshot operation, region optical character recognition operation, and element existence assertion operation. The target executable script has at least one control flow of conditional branching, looping, and exception handling, and supports variable definition and state memory.

[0013] Optionally, the method may also include the following before executing the target executable script: The target executable script is parsed using an abstract syntax tree. The function call nodes in the abstract syntax tree of the target executable script are traversed to obtain the text, resource identifier, and content description of the target control corresponding to the interactive operation, and then matched with a preset risk keyword dictionary. For function call nodes that hit risk keywords, the text and content descriptions of a preset number of controls surrounding the target control are obtained to form a control context. The control context is then input into the semantic understanding model to determine the risk level of the corresponding interactive operation. The risk levels include high risk, medium risk, and low risk. For high-risk interactive operations, a user confirmation message is output before execution, and the target executable script is stopped upon receiving a rejection confirmation message. For medium-risk interactive operations, an undo entry is provided after execution. For low-risk interactive operations, execution is performed directly and an execution log is recorded. The rollback method is determined based on the type of interactive operation. For data modification operations, the undo interface is queried through the content provider. If the undo interface exists, it is called. If the undo interface does not exist, a message indicating that automatic recovery is not possible is output. For page navigation operations, the Android system's back button is used to navigate back step by step. For setting modification operations, a snapshot of the setting value is saved before the setting modification operation is performed, and the setting value is restored based on the snapshot when rollback is triggered. In the same application, the number of times the same risk keyword is continuously judged as low-risk by the semantic understanding model in the same control context is counted. When the number exceeds the preset safe limit, the correspondence between the risk keyword and the control context is added to the safety whitelist, so that subsequent interactive operations that match the safety whitelist are processed as low-risk.

[0014] Optionally, when the natural language task involves cross-application data flow, the method includes: The Android package manager is used to query the target application's support information for Android intents and the export status of the target activity components. Based on the support information and export status, it is determined whether the Android intent path meets the preset routing conditions. When the Android intent path does not meet the preset routing conditions, cross-application operations are performed through the accessibility service path. When the Android intent path meets the preset routing conditions, the Android intent path and the accessibility service path are constructed respectively, and the Android intent path and the accessibility service path are started with a preset time difference. The cross-application data flow is completed by the path that succeeds first, and the other path is terminated. When cross-application data transfer fails to be completed successfully through both the Android intent path and the accessibility service path, cross-application data transfer can be completed through the Android system sharing panel.

[0015] Optionally, after the natural language task is successfully executed, the method also includes: The semantics of the natural language task, the abstract syntax tree of the successfully executed target executable script, and the exception information and corresponding recovery information generated during the execution process are encoded into self-evolving samples categorized by application. When the number of self-evolving samples corresponding to the same application reaches a preset number, the script abstract syntax tree is extracted from multiple self-evolving samples. The common structure in multiple script abstract syntax trees is abstracted by a large language model with script structure abstraction capabilities, and the nodes corresponding to specific task parameters are generalized into condition variables to generate a general abstract syntax tree template with condition variables. The number of visual semantic backtracking triggers for the corresponding application is counted. When the number of visual semantic backtracking triggers exceeds the preset number, the application's package name is added to the visual priority whitelist, so that subsequent natural language tasks corresponding to the application can directly trigger visual semantic backtracking.

[0016] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0017] The present invention has at least the following beneficial effects: This invention prioritizes historical task retrieval through task semantic embedding. When a similar task is found, it determines the nodes to be edited and the editing operations based on the cached abstract syntax tree, generating only the difference patch and merging it with the cached abstract syntax tree, thereby reducing the inference overhead of repeatedly generating complete scripts for similar tasks. When a task is not found, it combines the interface hierarchy tree compression results, historical interface temporal change information, and conditionally triggered visual semantic backtracking to form a fused interface state, enabling it to provide relatively complete interface semantics to the large language model even in cases of non-standard rendering or interface mutations. The large language model generates executable scripts that call Android operation primitives and have control flow, which can express the logical control in the execution of complex tasks. After an execution exception, it re-perceives the current interface and regenerates or corrects the script based on the exception information to continue execution, thereby improving the continuous execution capability and exception recovery capability of complex Android tasks. Attached Figure Description

[0018] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0019] Figure 1 This is a flowchart of an Android-side task automation method in one embodiment of this application. Detailed Implementation

[0020] The technical solution of the present invention will now be described in detail and completely with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, it should be noted that the terms "upper", "lower", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0022] It should be noted that the natural language task in this invention refers to a task instruction expressed in natural language text form, used to instruct an Android terminal to perform corresponding terminal operations on one or more terminal operation objects. The terminal object includes at least one of applications, files, contact data, and interface controls, and the terminal operation includes at least one of file search, file filtering, application switching, interface control operation, cross-application data transfer, and file deletion.

[0023] In a specific application scenario, an Android terminal has a chat application and an email application installed. The user inputs a natural language task: send all PDF files received today in the chat application to the project team contact in the email application; close any pop-up ads that appear; delete local temporary copies after successful sending. This task involves traversing the file list, conditional judgment, cross-application data transfer, and data deletion. Furthermore, the chat application may use a non-standard rendering framework, and pop-ups may appear temporarily on the page during execution. If a method of predicting individual click actions step by step is used, the actions may lose context when the number of files changes, pop-ups appear, or the target control is not exposed in the interface hierarchy tree. If a large language model is called again to generate a complete action sequence each time, similar tasks will result in repetitive reasoning. Therefore, this scenario simultaneously suffers from incomplete interface awareness, difficulty in expressing complex logic, difficulty in reusing similar tasks, and difficulty in recovering from abnormal states.

[0024] To address this technical problem, this invention provides an Android-side task automation method. For example... Figure 1 As shown, the method includes steps 11 to 14. The specific implementation of each step is explained below for both the first execution of the task and subsequent executions of similar tasks. Step 11 is responsible for semantic embedding and cache retrieval for the natural language task; Step 12 corresponds to the incremental generation path for cache hits; Step 13 corresponds to the complete perception and code generation path for cache misses; Step 14 is responsible for script execution, risk control, cross-application routing, exception recovery, and success trajectory updates. Both code generation paths proceed to step 14 after obtaining the target executable script.

[0025] Step 11: Obtain the natural language task input by the user, perform semantic embedding on the natural language task, and retrieve the cached task based on the obtained task semantic embedding.

[0026] Specifically, user input can first be normalized, retaining the application name, time range, file type, target contact, conditional statements, and post-task processing actions. For example, the aforementioned task can be parsed as the intent to send a file, with the source application being a chat application, the target application being an email application, the filtering conditions being "today" and "PDF," the conditional branch being to close the file if an ad pop-up appears, and the post-processing being to delete a local temporary copy after successful sending. The normalized text is then input into a semantic embedding model to obtain a d-dimensional task vector e_new. The semantic embedding model can be Instructor-XL, or it can be Sentence-BERT, OpenAI Embedding, or other models that can map tasks to a semantic vector space.

[0027] This embodiment uses cosine similarity to measure the semantic similarity between the current task and the cached task. The calculation formula is as follows:

[0028] Where T_new represents the current natural language task, T_i represents the i-th cached task, and e_new and e_i represent the corresponding task semantic embeddings, respectively. A preset semantic similarity threshold is denoted as γ, which is set to 0.85 in this embodiment. In actual deployment, γ can be set between 0.80 and 0.95 depending on the task type and fault tolerance requirements. If the highest similarity is not lower than γ, the cached task is considered to have been hit; if the highest similarity is lower than γ, the cached task is considered to have been missed.

[0029] The cache employs a four-level organization. The application-level template cache uses the application package name and task semantic cluster tag as the search key, storing abstract syntax tree templates with variable placeholders, such as saving file type, date, and target contact as replaceable nodes. The session-level adaptation cache uses the nearest neighbor task of the current task's semantic vector as the search criterion, storing instantiated executable scripts and their corresponding abstract syntax trees. The cross-session migration cache uses cross-application intent patterns as the search key, storing data pipeline descriptors from the source application to the target application, such as file sharing, text transfer, or Uniform Resource Identifier (URI) transfer methods. The abstract syntax tree differential cache uses the hash value of the abstract syntax tree editing operation mode as the search key, storing frequently differing templates. Candidates for determining whether the current task is a cached task are primarily obtained from the application-level template cache and session-level adaptation cache; the cross-session migration cache and abstract syntax tree differential cache are invoked as needed during subsequent generation and execution phases.

[0030] For example, when the terminal receives the aforementioned task for the first time, there are not enough similar historical tasks; the highest similarity is 0.63, which is less than 0.85, so it proceeds to step 13. If the user later inputs to send all Word files received in the chat application today to the same contact, the semantic vector of this task has a similarity of 0.92 with the successfully completed task, which is greater than 0.85, so it proceeds to step 12. Through this determination method, tasks within the same application that only have parameter changes or local logic changes can enter the incremental reuse path, while tasks with large semantic differences still use the complete generation path.

[0031] Step 12: When a cached task is found whose semantic similarity with the natural language task meets the preset semantic similarity threshold, the cached abstract syntax tree corresponding to the cached task is obtained. Based on the task differences between the natural language task and the cached task, the nodes to be edited in the cached abstract syntax tree and the corresponding abstract syntax tree editing operations are determined to form a difference skeleton. Based on the difference skeleton, the difference patch is obtained and the difference patch is merged with the cached abstract syntax tree to generate a target executable script for driving the Android application to perform natural language tasks.

[0032] In practice, if the caching task comes from the application-level template cache, the abstract syntax tree template within it is read as the cached abstract syntax tree; if the caching task comes from the session-level adaptation cache, the abstract syntax tree corresponding to the instantiated executable script is read as the cached abstract syntax tree. Each abstract syntax tree node records the node type, function name, parameter values, conditional expressions, and child node relationships. Task parameters in the application-level template are stored as variable placeholders, such as FILE_TYPE, DATE_RANGE, and DESTINATION. The current task and the cached task are compared in task slots to identify parameter and logical differences, which are then used to locate the nodes that need modification based on the abstract syntax tree edit distance. Abstract syntax tree editing operations include at least leaf node replacement, subtree insertion, and subtree deletion.

[0033] For example, if the caching task is to send all PDF files received today, and the current task only changes the file type to Word, the string constant of the suffix judgment node in the script can be replaced from .pdf to .doc or .docx, and the corresponding leaf node can be replaced accordingly. If the current task adds a condition to close the window when an ad pop-up appears, an if subtree containing an element existence assertion and a click operation can be inserted before the list traversal logic, and the corresponding subtree can be inserted accordingly. If the current task cancels the preview step before sending, the subtree containing the waiting preview page and return operation can be deleted, and the corresponding subtree can be deleted accordingly. By temporarily extracting the specific parameter values ​​in the editing operation and retaining only the operation type, node path, parent-child relationship, and semantic slots that need to be filled, the difference skeleton is obtained.

[0034] An example of an implementable difference skeleton is as follows: ; ; ; The difference skeleton is further standardized into an editing operation mode, for example, only retaining REPLACE_LEAF, INSERT_SUBTREE, DELETE_SUBTREE and node categories, and then calculating a hash value on the standardized result. If the hash value hits the abstract syntax tree difference cache, the difference template is directly read and instantiated with the current task parameters to obtain the difference patch; if it does not hit, only the difference skeleton, the current task difference, and the Android operation primitive interface description are input into a large language model with code generation capabilities. The large language model fills in the semantic gaps in the difference and writes the newly generated difference template into the abstract syntax tree difference cache.

[0035] After obtaining the difference patch, the cached abstract syntax tree is replaced, inserted, or deleted according to the node path. Then, a syntax tree integrity check is performed, including parent-child node type matching, variable definition, branch closure, loop exit conditions, and whether function calls belong to allowed Android operation interfaces. If the check passes, the merged abstract syntax tree is reverse-engineered to generate the target executable script. If the check fails, the failed nodes and parsing errors can be used as local feedback to regenerate the difference patch without rewriting the entire script.

[0036] For example, when the complete generation script requires about 2,500 model input and output tokens, passing only the difference skeleton can compress the incremental generation scale to about 300 tokens; when the difference pattern already exists in the abstract syntax tree difference cache, the difference template can be instantiated directly without calling the large language model.

[0037] This step leverages code structure-level differences rather than regenerating all actions, reducing repetitive reasoning for similar tasks and preserving the original conditional branches, loops, and exception handling structures of the cached script.

[0038] Step 13: When a cached task with semantic similarity to the natural language task that does not meet a preset semantic similarity threshold is not hit, the Extensible Markup Language (XML) hierarchy tree of the current display interface of the Android terminal device is obtained through Android Accessibility Service or Android Debug Interface, and a screenshot of the current display interface and historical interface states are obtained; the XML hierarchy tree is compressed to obtain a structural semantic representation, and interface temporal change information is determined based on the historical interface states; when the number of interactive nodes in the structural semantic representation is lower than a preset node threshold, or when the current display interface is determined to have undergone a sudden change based on the interface temporal change information, visual semantic backtracking is performed on the screenshot to obtain visual semantic anchors, and the structural semantic representation is enhanced using the visual semantic anchors; a fused interface state is formed based on the structural semantic representation, interface temporal change information, and the visual semantic anchors obtained when visual semantic backtracking is triggered; the fused interface state, the natural language task, and the interface description of the Android operation primitives are input into a large language model with code generation capabilities, and the large language model generates a target executable script that calls the Android operation primitives and has control flow, and caches the abstract syntax tree of the target executable script.

[0039] Specifically, the first step is to obtain the structural information of the current interface. The preferred method is to read the accessible node tree through Android AccessibilityService, or to obtain the Extensible Markup Language (Extreme Markup Language) hierarchy tree by calling uiautomator dump via ADB, or to use LayoutInspector, which can export equivalent hierarchy structures. Simultaneously, screenshots are captured, and the interface state of the most recent k frames and the actions performed before each frame are maintained. In this embodiment, k=5.

[0040] When compressing the Extensible Markup Language hierarchical tree, all nodes are traversed to eliminate container nodes whose invisible attribute is true, transparency is zero, width is zero, height is zero, and which only serve a layout function without containing valid interaction semantics. For consecutive non-interactive leaf nodes that share the same clickable ancestor, these leaf nodes are merged into a single entry, and the original text order is preserved using line break markers. Each retained entry only records the node identifier, resource identifier, text, content description, bounding box, clickable attribute, and scrollable attribute. The compression result is serialized into HTML-like structured text as the structural semantic representation.

[0041] For example, a clickable entry in the file list of a chat application may originally consist of four leaf nodes for the file name, date, size, and thumbnail, and after compression, it can be expressed as: <item id="n38" resource-id="file_row" text="project.pdf Today 09:31 2.4MB" bounds="[48,420][1032,592]" clickable="true" scrollable="false" / >; By deleting redundant layout attributes and merging the same interaction entries, the length of the structural semantic representation can be significantly smaller than that of the original hierarchical tree. In a specific data example, the original hierarchical information is about 40,000 Tokens, and the average HTML-like text after compression is less than 500 Tokens. The compression does not change the control identifiers, texts and position boundaries that are useful for automation, so the main interaction semantics can be retained while reducing the model input scale.

[0042] Subsequently, it is determined whether visual semantic backtracking is required. In this embodiment, the number of valid interactive leaf nodes in the structural semantic representation is compared with the node threshold for comparison, preferably , and can also be adjusted within the range of 2 to 5. When the chat application is rendered using methods such as Flutter, Unity or Canvas, the hierarchical tree may only expose one SurfaceView or Canvas container, and the number of valid interactive leaf nodes is 0 or 1, at which point visual semantic backtracking is directly triggered. Even if the number of interactive nodes is not less than , as long as the interface temporal change information indicates that a sudden change has occurred in the current interface, visual semantic backtracking is also forcibly triggered.

[0043] Visual semantic backtracking performs optical character recognition (OCR) on screenshots to obtain text blocks and corresponding bounding boxes, and uses a lightweight vision model to recognize common icons such as share, menu, back, camera, shopping cart, and close. Each recognition result forms a visual semantic anchor point containing a bounding box, semantic label, and confidence score. The OCR engine can be Tesseract, PaddleOCR, or other engines capable of outputting text and bounding boxes. For icons, a lightweight classification network or visual language model can be used.

[0044] The association between visual semantic anchors and hierarchical tree nodes can be achieved by combining bounding box intersection-union ratio (IU), text similarity, and centroid distance. For example, when the IU of a visual semantic anchor and a node's bounding box is not less than 0.5 and the text or content description is consistent, an association is established, and a visual saliency attribute and a visual semantic label are written to that node. When there is no matching node, a virtual node is created under the closest container node, and the same attributes are written. In the aforementioned scenario, if the OCR recognizes the filename project.pdf, the icon classifier recognizes a share icon in the right area with a confidence score of 0.94, but there is no corresponding icon node in the hierarchical tree, then a virtual node can be created. <virtual-node bounds="[910,468][982,540]" visual_saliency="0.94" visual_label="share" / > ; After the visual semantic information is written back, the structural semantic representation is regenerated, enabling the large language model to simultaneously see the control structure and visual labels during subsequent code generation. For cases where corresponding structural nodes already exist for visual labels, there is no need to create duplicate virtual nodes; only the visual saliency and visual semantic fields of the original nodes are updated.

[0045] Historical interface states are used to generate information on the temporal changes of the interface. For each frame, the set of visible text is normalized in a fixed order, and the node types and parent-child topology are serialized. Then, an MD5 hash is calculated as the interface state fingerprint, and the sequence of actions preceding the generation of that frame is saved. If the same interface state fingerprint appears consecutively or repeatedly within the last k frames more than a preset number of loops, for example, more than 3 times, it can be determined that the execution process has entered a loop state, and the repetition of the same action sequence will be prohibited in subsequent generation or restoration.

[0046] Abrupt changes in the interface can be identified by variations in the number of nodes. Let the number of visible nodes in the current frame be n_t, and the number of visible nodes in the previous frame be n_{t-1}. The increase in the number of nodes, r_t, can be calculated using the following formula:

[0047] When r_t is greater than the preset mutation threshold When a sudden change occurs in the current interface, this embodiment can determine that a change has taken place. Set to 0.5. For example, when the number of nodes increases from 24 to 41, the increase is approximately 70.8%, exceeding 50%, which suggests the appearance of a pop-up window or a new overlay. At this point, the hierarchical tree scanning depth is increased from the default 3 layers to 5 layers, and a visual semantic backtracking is forced to complete, thereby supplementing information such as the close button and confirmation button of the pop-up window that may not have been captured by the shallow scan. The interface state fingerprint, the previous action sequence, the loop state determination result, and the interface abrupt change determination result together constitute the interface temporal transition information.

[0048] When visual semantic backtracking is not triggered, the fused interface state is formed based on the structural semantic representation and interface temporal change information; when visual semantic backtracking is triggered, the visual semantic anchor points are used together to form the fused interface state.

[0049] To dynamically integrate structural, visual, and temporal information based on different page qualities, three types of quality features can be constructed first. The structural semantic quality feature, `feat_xml`, can be determined by node coverage and semantic density. Node coverage represents the proportion of visible nodes that retain effective semantics after compression out of the total number of visible nodes, while semantic density represents the proportion of entries with text, content descriptions, or resource identifiers out of the retained entries. The visual semantic quality feature, `feat_vis`, can be determined by the average confidence score of OCR and the average confidence score of icon classification. The temporal change quality feature, `feat_time`, can be determined based on the rate of change of the visible text set, the rate of change of the number of nodes, and the cycle state. For ease of calculation, each sub-feature can be normalized to 0 to 1 and then a weighted average can be taken.

[0050] Before performing the fusion computation, the structural semantic representation, visual semantic anchor set, and interface temporal transition information are each mapped into d-dimensional vectors using encoders of the same dimension. The encoded results are still denoted as X_t, V_t, and H_t, respectively. Simultaneously, the original HTML-like text, anchor fields, and temporal fields are retained for interpretable input in subsequent prompts. The fusion state S_t of the current interface is formed according to the following formula:

[0051] The fusion weights are calculated using the following formula:

[0052] Where S_t represents the fusion state of the interface at time t; X_t represents the structured semantic encoding output by the structural semantic layer; V_t represents the semantic anchor encoding obtained from visual semantic backtracking; and H_t represents the encoding of interface temporal change information. , α_time and α_time represent the fusion weights of structural semantics, visual semantics, and temporal change, respectively, and the sum of the three is 1; W is a learnable weight matrix, which can be trained on offline labeled interface quality samples and fixed when deployed on the terminal; feat_xml, feat_vis, and feat_time represent the three types of quality features, respectively.

[0053] For example, in a non-standard rendered page, if feat_xml=0.45, feat_vis=0.92, feat_time=0.65, and using the identity matrix as a simplified example of W, then Softmax calculates... Approximately 0.262 Approximately 0.419 The value is approximately 0.320, indicating a relatively higher proportion of visual semantics in the fusion state. In standard view pages with complete hierarchical tree information, `feat_xml` is typically higher, allowing for a corresponding increase in structural semantic weight. Driving weight changes with quality features avoids applying the same proportion of perceptual information to all pages.

[0054] When providing the fused interface state to the large language model, different adaptation methods can be adopted according to the input interface of the large language model. When the large language model only supports text input, the original HTML-like text, interface timing fields, and visual semantic anchor points obtained when triggering visual semantic backtracking are weighted or prioritized according to the fusion weights corresponding to the semantic information involved in the current fusion, and the fused interface state text is formed according to a preset serialization format. The fused interface state text is then used as the input to the large language model. When the large language model supports external vector input, the fused state S_t is mapped to the embedding space of the large language model through the adaptation layer and then input into the large language model.

[0055] The interface descriptions of the fused interface state, natural language task, and Android operation primitives, obtained according to the aforementioned adaptation method, are input into a large language model with code generation capabilities. When using text input, prompts can be organized in the following order: DSL interface documentation, 2 to 3 few-shot examples, fused interface state text, natural language task, and recent history context. Android operation primitives can include at least tap(selector) click operation, input_text(selector,text) text input operation, scroll(direction,amount) scroll operation, wait_for(selector,timeout) wait operation, swipe(from,to) swipe operation, press(key) system key operation, screenshot() screen capture operation, ocr_region(bbox) region optical character recognition operation, and assert_exists(selector) element existence assertion operation. The underlying driver can use uiautomator2, or Appium, adb-shell, or other automation interfaces that can achieve equivalent Android interactions.

[0056] When calling the cloud-based large language model, local desensitization of personal information can be performed before sending prompts. Fields such as mobile phone number, email address, ID card number, bank card number, and name are first replaced with placeholders such as NAME_i, PHONE_i, and EMAIL_i. After the model returns the script, the reverse mapping is completed locally on the terminal. If a local code-generated model is used, this desensitization step can still be retained to unify the script template, or it can be disabled according to the deployment strategy. This processing does not change the task logic, but only reduces the chance of real personal information leaving the terminal.

[0057] The aforementioned task can generate scripts with conditional branches, loops, exception handling, variable definitions, and state memory. Below is a pseudocode example where device operations are performed using predefined Android operation primitives: sent_count = 0 page_index = 0 while page_index<8: if assert_exists(text="close") or assert_exists(content_desc="close"): tap(text="close") screenshot() blocks = ocr_region(bbox=[0, 220, 1080, 2060]) for block in blocks: if block.text.lower().endswith(".pdf") and block.context_contains("Today"): try: tap(bounds=block.bounds) wait_for(text="Share", timeout=3) tap(text="Share") sent_count = sent_count + 1 except TimeoutError: press(key="BACK") raise scroll(direction="down", amount=0.8) page_index = page_index + 1; This example uses `while` and `for` loops to express list traversal, `if` statements to handle pop-up conditions, `try-except` to handle timeout exceptions, and `sent_count` and `page_index` to record execution status. The actual script can directly select controls based on structural semantic nodes, or select controls using visual semantic labels or bounding boxes after triggering visual backtracking. After generation, the script undergoes abstract syntax tree parsing, and the script and its corresponding abstract syntax tree are written to a session-level adaptation cache, providing a reuse basis for subsequent similar tasks. The general abstract syntax tree template in the application-level template cache is generated and updated according to the self-evolving sample accumulation and template abstraction process described later.

[0058] The key role of step 13 is that when the cache cannot provide a directly reusable script, it not only uses the current static interface but also considers the historical interface state; visual backtracking is conditionally invoked only when there are insufficient hierarchical tree nodes or sudden interface changes, and the visual recognition results are used to supplement the structural semantic representation. This can fill the semantic gaps caused by non-standard rendering and sudden pop-ups while controlling the scale of model input, thereby enabling the code generation stage to obtain a more complete task context and generate a target executable script with control flow.

[0059] Step 14: On the Android terminal device side, call the predefined Android operation primitives to execute the target executable script, so as to drive the target application on the Android terminal device to perform interactive operations corresponding to the natural language task.

[0060] Before actual execution, an abstract syntax tree (AST) security check is performed on the target executable script. All function call nodes are traversed to obtain the text, resource identifiers, and content descriptions of the interactive target, and then matched against a pre-defined risk keyword dictionary. Risk keywords can include delete, kill, uninstall, send money, transfer, clear all, delete account, format, clear, etc. For any system command calls appearing in the script, further restrictions can be imposed using function whitelists and blacklists. For example, unauthorized calls to os.system, subprocess, etc., can be rejected, allowing only encapsulated Android operation primitives and necessary data processing functions.

[0061] For function calls that hit risk keywords, instead of directly providing a final risk conclusion based on the keywords, the text and content descriptions of the three controls before and after the target control are extracted to form the control context. This context is then input into a lightweight BERT or other semantic understanding model for secondary judgment. Risk Level It can be determined by the following formula: Keyword matching and semantic context confirm high risk; The keywords match but the semantic context is ambiguous; If no keyword match or semantic context is confirmed, it is considered safe. High-risk interactions output a confirmation message before execution, and the script is immediately stopped if the user refuses. Medium-risk interactions can be executed first, and an undo option is provided after execution. Low-risk interactions are executed directly and logged. In the aforementioned scenarios, closing ad pop-ups usually does not hit risky keywords and can be executed directly. Deleting local temporary copies will hit deletion-related keywords. If the context also contains information such as "sent," "temporary copy," and "can be downloaded again," it can be judged as medium-risk based on the model, and an undo option is provided after execution. If the task becomes clearing all chat history, and the semantic context clearly corresponds to large-scale irreversible data deletion, it is judged as high-risk and confirmation is required before execution.

[0062] The rollback method is selected based on the operation type. For data modification operations, the ContentProvider is first used to check if the target application provides an undo interface. If an undo interface exists, it is called directly; otherwise, the reason for the inability to automatically restore is recorded, and the user is prompted to use the application's own restoration capabilities. For page navigation operations, the user is chronologically returned to the checkpoint page via the Android back button. For setting modification operations, a snapshot of the setting value is read and saved before modification, and the original value is written back when a rollback is triggered. For high-risk operations, an Activity stack snapshot can also be recorded before execution, and timeout watchpoints and rollback checkpoints can be set so that in the event of a crash or prolonged unresponsiveness, the user can be restored to a page that can be continued.

[0063] Security policies can be adaptively updated based on usage history. The system counts the number of times the same risky keyword is consecutively classified as low-risk by the semantic understanding model within the same control context in the same application. When this number exceeds a preset safety threshold, the mapping between the risky keyword and the control context is added to a security whitelist. Subsequent occurrences of a completely matching context are treated as low-risk, but if the context changes significantly, the system re-enters a secondary semantic assessment to prevent misuse of the whitelist across different scenarios.

[0064] When a natural language task involves cross-application data transfer, before executing the corresponding sharing action, the Android PackageManager is used to query the target email application's support information for the target Intent and the exported status of the target Activity. Based on support status, component accessibility, and historical success records, it is determined whether the Intent path meets the preset routing conditions. One implementation normalizes the routing success rate to 0 to 1, with a preset routing threshold of 0.7. If the evaluation value is below 0.7, application switching, target control location, and data input are directly completed through the accessibility service path, avoiding the execution of high-probability-of-failure Intents first.

[0065] If the Intent path meets the routing conditions, both the Intent path and the accessibility service path are constructed and started with a preset time difference. For example, the Intent path can be started first, followed by the accessibility path 50 milliseconds later. Success events and cancellation flags are set for each path. Once either path confirms that the target email editing interface has appeared and the file mounting is complete, it is marked as successful, and the other path is terminated. If neither the Intent path nor the accessibility service path successfully completes the cross-application data transfer, an available target is selected via the Android system's sharing panel to complete the cross-application data transfer. Intent path execution failures include situations where the target application version does not support the corresponding Action, the target component is unreachable, or the application is not installed. The data pipeline descriptor stored in the cross-session migration cache records the Action, MIME type, URI transmission method, and target package name of the last successfully used action, providing routing priors for subsequent similar tasks.

[0066] During script execution, each Android operation primitive returns its execution status, execution time, and a summary of necessary UI changes. Clicks, inputs, scrolling, waiting, swiping, system buttons, screenshots, region optical character recognition, and element existence assertions can be executed in combination according to the script's control flow. If wait_for times out, the target element does not exist, the application jump fails, a pop-up window obscures the view, or the script throws other exceptions, the current action sequence is immediately stopped, and the hierarchy tree, screenshot, and most recent historical state are re-acquired. A new current UI state is then formed according to the perception method in step 13.

[0067] When an exception occurs during the execution of the target executable script, the current interface state of the Android terminal device is retrieved again. Based on the exception information and the retrieved current interface state, the target executable script is regenerated or corrected before execution continues. The exception information and the corresponding regeneration or correction information are recorded as recovery information until the natural language task is completed or the preset number of replanning attempts is reached.

[0068] During exception recovery, the exception type, the abstract syntax tree node where the exception occurred, the most recently successful action, the reacquired UI state, and the conditions for incomplete tasks are input into the large language model. If the exception only involves changes to control text or insufficient waiting time, local difference patches are generated first, such as replacing selectors, inserting wait_for, or adding a return operation; if the UI flow has changed significantly, subsequent script subtrees are regenerated. After correction, execution continues from the most recent safety checkpoint, rather than completely replaying from the task starting point. Exception information, along with the differences in the abstract syntax tree before and after modification, the re-perception results, and the recovery action sequence, are recorded as recovery information.

[0069] The preset number of replanning attempts is 3. If the fingerprint of the same interface state appears repeatedly beyond the loop limit, or if the task completion condition is not met after 3 replanning attempts, automatic execution will stop and the failure reason will be returned; otherwise, it will continue to run until all target PDFs have been sent across applications and local temporary copies have been processed according to the risk policy. By setting an upper limit for replanning, infinite loops under abnormal interface conditions can be avoided.

[0070] After a natural language task is successfully executed, the task semantics, the abstract syntax tree of the successfully executed script, the exception information generated during execution, and the corresponding recovery information are encoded into self-evolving samples categorized by application. As additional statistical fields, execution time, model token consumption, number of visual backtracking triggers, and number of risk confirmations can also be recorded for subsequent evaluation of template quality, but these additional fields do not change the core structure of the samples.

[0071] When the number of self-evolving samples accumulated by the same application reaches a preset number N, the script abstract syntax tree is extracted from multiple samples. A large language model with script structure abstraction capabilities is used to identify common structures, and leaf nodes corresponding to task parameters such as specific dates, file types, contacts, and scroll counts are generalized into condition variables, forming a general abstract syntax tree template with condition variables. In this embodiment, N=50 is preferred, but it can also be set between 30 and 100 depending on the application complexity. Taking a file sending task as an example, specific file extensions such as PDF, Word, and image can be uniformly abstracted as FILE_TYPE, and the target contact can be abstracted as DESTINATION, making it easier for subsequent tasks to hit the application-level template cache.

[0072] Simultaneously, the number of times each application triggers visual semantic backtracking is counted. If an application continuously triggers visual backtracking in multiple successful tasks due to insufficient valid structural nodes, and the number of triggers exceeds the preset trigger count, the application's package name is added to the visual priority whitelist. Subsequent processing of natural language tasks for this application can directly initiate visual semantic backtracking without waiting for the number of interactive nodes to fall below a certain threshold. Only then is it triggered. The security whitelist is updated synchronously according to the aforementioned low-risk continuous judgment logic. In this way, the successful execution result is not only used to cache the script, but also used to update the awareness and security handling strategies for subsequent tasks.

[0073] The key role of step 14 is to transform the code generation result into a continuously executable Android interaction process, and to provide clear handling paths for risky operations, cross-application propagation, and abnormal states. After an exception occurs, the script is re-detected and locally modified, forming a closed loop of perception, generation, execution, and feedback in the execution process; the success trajectory further forms a generalizable template and application-level strategy, reducing the number of times subsequent similar tasks need to be generated from scratch.

[0074] In other feasible implementations, the large language model can be a cloud-based model or a locally or edge-deployed model such as CodeLlama, Qwen-Coder, or Vicuna; UI hierarchy information can be obtained through AccessibilityService, uiautomator dump, or LayoutInspector; the optical character recognition engine and the underlying device driver library can be replaced with implementations with equivalent functionality. These replacements do not alter the logical relationships between task semantic retrieval, abstract syntax tree differencing, dynamic perception fusion, Android operation primitive execution, and exception closed-loop recovery.

[0075] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0076] This application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the steps described in the various method embodiments above.

[0077] The above embodiments are used to illustrate the technical principles and executable implementation of the present invention, and do not limit the parameters to only the example values. Those skilled in the art can adjust the semantic similarity threshold, visual backtracking threshold, number of historical frames, interface mutation threshold, number of replanning attempts, and number of accumulated samples according to the terminal computing power, application type, and task complexity, as long as the same functional relationship is maintained between each step.

Claims

1. An Android client-side task automation method, characterized in that, include: Obtain the natural language task input by the user, perform semantic embedding on the natural language task, and retrieve the cached task based on the obtained task semantic embedding; When a cached task is found whose semantic similarity with the natural language task meets a preset semantic similarity threshold, the cached abstract syntax tree corresponding to the cached task is obtained. Based on the task differences between the natural language task and the cached task, the nodes to be edited in the cached abstract syntax tree and the corresponding abstract syntax tree editing operations are determined to form a difference skeleton. Based on the difference skeleton, a difference patch is obtained, and the difference patch is merged with the cached abstract syntax tree to generate a target executable script for driving the Android application to execute the natural language task. When no cached task is found whose semantic similarity with the natural language task meets the preset semantic similarity threshold, the Extensible Markup Language (XML) hierarchy tree of the current display interface of the Android terminal device is obtained through Android Accessibility Service or Android Debug Interface, and a screenshot of the current display interface and historical interface states are obtained; the XML hierarchy tree is compressed to obtain a structural semantic representation, and interface temporal change information is determined based on the historical interface states; When the number of interactive nodes in the structural semantic representation is lower than a preset node threshold, or when the current display interface is determined to have undergone a sudden change based on the interface temporal change information, visual semantic backtracking is performed on the screenshot to obtain visual semantic anchors, and the visual semantic anchors are used to enhance the structural semantic representation; a fused interface state is formed based on the structural semantic representation, the interface temporal change information, and the visual semantic anchors obtained when the visual semantic backtracking is triggered; the fused interface state, the natural language task, and the interface description of the predefined Android operation primitives are input into a large language model with code generation capabilities; the large language model generates the target executable script that calls the Android operation primitives and has control flow, and caches the abstract syntax tree of the target executable script; On the Android terminal device side, predefined Android operation primitives are invoked to execute the target executable script, thereby driving the target application on the Android terminal device to perform interactive operations corresponding to the natural language task. When an exception occurs during the execution of the target executable script, the current interface state of the Android terminal device is reacquired, and the target executable script is regenerated or corrected based on the exception information and the reacquired current interface state before continuing execution. The exception information and the corresponding regeneration or correction information are recorded as recovery information until the natural language task is completed or a preset number of replanning attempts are reached.

2. The Android client-side task automation method according to claim 1, characterized in that, The compression of the Extensible Markup Language (Extreme Markup Language) hierarchical tree yields a structural semantic representation, including: Traverse the Extensible Markup Language hierarchy tree, removing nodes with the invisible attribute set to true, nodes with zero opacity, nodes with zero width, nodes with zero height, and pure layout container nodes. Merge consecutive non-interactive leaf nodes that share the same clickable ancestor into a single entry, while preserving the node identifier, resource identifier, text, content description, bounding box, clickable attribute, and scrollable attribute; Based on the merged entries and the retained attributes, a structured text in HTML format is generated, and the structured text is used as the structural semantic representation.

3. The Android client-side task automation method according to claim 2, characterized in that, Visual semantic backtracking is performed on the screenshot to obtain visual semantic anchors, and the structural semantic representation is enhanced using the visual semantic anchors, including: Optical character recognition is performed on the screenshot to obtain text blocks and bounding boxes corresponding to the text blocks. The icons in the screenshot are classified and recognized using a visual model. Based on the classification and recognition results of the text blocks and icons, visual semantic anchors containing bounding boxes, semantic labels and confidence scores are generated. The visual semantic anchor is associated with the corresponding node in the Extensible Markup Language (XML) hierarchy tree, and the visual saliency attribute and visual semantic label are written to the corresponding node; when there is no node in the XML hierarchy tree corresponding to the visual semantic anchor, a virtual node is created, and the visual saliency attribute and visual semantic label are written to the virtual node. The structural semantic representation is updated based on the node information written with the visual saliency attribute and visual semantic label, so that the structural semantic representation contains visual semantic information corresponding to the visual semantic anchor point, thereby obtaining an enhanced structural semantic representation, and the enhanced structural semantic representation is used as the structural semantic representation for forming the fusion interface state.

4. The Android client-side task automation method according to claim 3, characterized in that, The step of determining interface temporal change information based on the historical interface state, and forming a fused interface state based on the structural semantic representation, the interface temporal change information, and the visual semantic anchor point when the visual semantic backtracking is triggered, includes: Record the interface state fingerprint of a preset number of frames and the corresponding previous action sequence, and generate the interface state fingerprint based on the visible text set and node topology of the currently displayed interface. When the number of times a fingerprint appears repeatedly in the same interface state exceeds a preset number of loops, it is determined that the execution process of the natural language task has entered a loop state. When the increase in the number of nodes in the current display interface relative to the number of nodes in the previous display interface exceeds a preset mutation threshold, it is determined that a mutation has occurred in the current display interface. When the mutation is determined to have occurred, the scanning depth of the Extensible Markup Language hierarchy tree is increased, and the visual semantic backtracking is triggered. The interface state fingerprint, the previous action sequence, the loop state determination result, and the interface mutation determination result are determined as the interface temporal transition information. When the visual semantic backtracking is triggered, the structural semantic quality features are determined based on the node coverage and semantic density, the visual semantic quality features are determined based on the confidence level of optical character recognition and icon classification, and the temporal transition quality features are determined based on the degree of interface state change. Based on the structural semantic quality features, visual semantic quality features, and temporal change quality features, structural semantic fusion weights, visual semantic fusion weights, and temporal change fusion weights are determined. Then, based on the structural semantic fusion weights, visual semantic fusion weights, and temporal change fusion weights, the corresponding structural semantic representations, visual semantic anchors, and interface temporal change information are fused to obtain the fused interface state.

5. The Android client-side task automation method according to claim 1, characterized in that, The caching system used in the Android client-side task automation method includes application-level template caching, session-level adaptation caching, cross-session migration caching, and abstract syntax tree differential caching. The cache task is retrieved from the application-level template cache and / or the session-level adaptation cache based on the task semantic embedding; The application-level template cache uses the application package name and task semantic cluster tag as the retrieval key, and stores an abstract syntax tree template containing variable placeholders. The session-level adaptation cache uses the nearest neighbor task corresponding to the task semantic embedding of the current natural language task as the retrieval basis, and stores the instantiated executable script and its corresponding abstract syntax tree. The cross-session migration cache uses cross-application intent patterns as the retrieval key and stores the corresponding data pipeline descriptors; The abstract syntax tree differential cache uses the hash value of the abstract syntax tree editing operation mode as the retrieval key and stores the corresponding difference template; The process of obtaining the difference patch based on the difference skeleton includes: When the abstract syntax tree editing operation mode corresponding to the difference skeleton hits the abstract syntax tree difference cache, the corresponding difference template is obtained from the abstract syntax tree difference cache as the difference patch; wherein, the abstract syntax tree editing operation includes at least one of leaf node replacement, subtree insertion, and subtree deletion; When the abstract syntax tree difference cache is not hit, the difference skeleton is input into the large language model with code generation capability to obtain the difference patch, and the difference template corresponding to the difference patch is stored in the abstract syntax tree difference cache.

6. The Android client-side task automation method according to claim 1, characterized in that, The Android operation primitives include at least one of the following: click operation, text input operation, scroll operation, wait operation, swipe operation, system button operation, screenshot operation, region optical character recognition operation, and element existence assertion operation. The target executable script has at least one control flow among conditional branching, looping, and exception handling, and supports variable definition and state memory.

7. The Android client-side task automation method according to claim 1, characterized in that, Before executing the target executable script, the method further includes: The target executable script is parsed using an abstract syntax tree. The function call nodes in the abstract syntax tree of the target executable script are traversed to obtain the text, resource identifier, and content description of the target control corresponding to the interactive operation, and then matched with a preset risk keyword dictionary. For function call nodes that hit risk keywords, the text and content descriptions of a preset number of controls surrounding the target control are obtained to form a control context. This control context is then input into a semantic understanding model to determine the risk level of the corresponding interactive operation. The risk levels include high risk, medium risk, and low risk. For high-risk interactive operations, user confirmation information is output before execution, and the target executable script is terminated upon receiving a rejection confirmation message. For medium-risk interactive operations, an undo entry is provided after execution. For low-risk interactive operations, execution is performed directly, and an execution log is recorded. The corresponding rollback method is determined based on the type of interactive operation. For data modification operations, the undo interface is queried through the content provider. If the undo interface exists, it is called. If the undo interface does not exist, a prompt message indicating that automatic recovery is not possible is output. For page navigation operations, the Android system's back button is used to navigate back level by level. For setting modification operations, a snapshot of the setting value is saved before the setting modification operation is performed, and the setting value is restored based on the snapshot of the setting value when rollback is triggered. In the same application, the semantic understanding model continuously determines the same risk keyword as low-risk in the same control context. When the number of times exceeds the preset number of safe entries, the correspondence between the risk keyword and the control context is added to the security whitelist, so that subsequent interactive operations that match the security whitelist are processed as low-risk.

8. The Android client-side task automation method according to claim 1, characterized in that, When the natural language task involves cross-application data flow, the method includes: The Android package manager is used to query the target application's support information for Android intents and the export status of the target activity components. Based on the support information and the export status, it is determined whether the Android intent path meets the preset routing conditions. When the Android intent path does not meet the preset routing conditions, cross-application operations are performed through the accessibility service path. When the Android intent path meets the preset routing conditions, an Android intent path and an accessibility service path are constructed respectively, and the Android intent path and the accessibility service path are started with a preset time difference. The cross-application data flow is completed by the path that is executed first, and the other path is terminated. If the cross-application data transfer fails to be completed successfully in both the Android intent path and the accessibility service path, the cross-application data transfer is completed through the Android system sharing panel.

9. The Android client-side task automation method according to claim 1, characterized in that, After the natural language task is successfully executed, the method further includes: The semantics of the natural language task, the abstract syntax tree of the successfully executed target executable script, and the exception information and corresponding recovery information generated during execution are encoded into self-evolving samples categorized by application. When the number of self-evolving samples corresponding to the same application reaches a preset number, script abstract syntax trees are extracted from multiple self-evolving samples. The common structure in multiple script abstract syntax trees is abstracted through a large language model with script structure abstraction capabilities. The nodes corresponding to specific task parameters are generalized into condition variables to generate a general abstract syntax tree template with condition variables. The number of visual semantic backtracking triggers corresponding to the application is counted. When the number of visual semantic backtracking triggers exceeds the preset number of triggers, the package name of the application is added to the visual priority whitelist, so that the subsequent natural language tasks corresponding to the application directly trigger the visual semantic backtracking.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.

Images