Android system user interface interaction method and device based on large language model

Abstract

The invention discloses an Android system user interface interaction method and device based on a large language model, and relates to the technical field of intelligent man-machine interaction, and the method comprises the steps: receiving a natural language instruction of a user, and synchronously capturing a real-time interface state of a current Android screen to generate a structured context containing semantic information of interactive elements; inputting the instruction and the context into a large language model, and generating a hierarchical task plan formed by a plurality of atomic operations with a logic dependency relationship; based on a dynamic adaptation mechanism, converting the atomic operation sequence into an operation instruction which can be executed by the Android system; receiving interface change information after instruction execution, and feeding back the interface change information as a new round of context to the large language model to determine a task execution state; and finally generating natural language feedback information to the user according to the execution state. By means of the mode, complex cross-application tasks can be understood and executed, and interaction intelligence of the Android system is remarkably improved through real-time interface perception and dynamic planning adjustment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent human-computer interaction, and in particular to an Android system user interface interaction method and device based on a large language model. BACKGROUND

[0002] In the existing Android system interaction mode, the user needs to interact with the application interface through manual operations such as touch and click. With the increasing complexity of application functions, it often takes a series of tedious operations for the user to complete a task among multiple applications and multiple hierarchical interfaces. For example, if the user wants to "send the last photo taken last night to Zhang San through WeChat", he needs to complete a series of steps such as unlocking the phone, finding and opening the album, filtering photos, selecting sharing, finding and clicking WeChat, and selecting contacts. This way is not only inefficient, but also has a use threshold for users with limited mobility or special needs. Although there are some automated scripts or voice assistants in the prior art, they can usually only perform pre-set and fixed processes, lack deep semantic understanding of complex natural language instructions and flexible adaptation to dynamic changes in interface state, and cannot truly realize intelligent interaction centered on user intent.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide an Android system user interface interaction method and device based on a large language model, aiming to solve the technical problem that the prior art cannot perform reliable task planning and execution in a dynamically changing Android interface environment.

[0005] To achieve the above purpose, the present application provides an Android system user interface interaction method based on a large language model, the method comprising: receiving a natural language instruction of a user and synchronously capturing a real-time interface state of a current Android screen, generating a structured context containing semantic information of an interactive element according to the real-time interface state; inputting the natural language instruction and the structured context into a large language model together, generating a hierarchical task planning containing at least one atomic operation and having a logical dependency relationship between operations; based on a dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state, sequentially converting the atomic operations in the hierarchical task planning into operation instructions executable by the Android system; receiving changes in the interface state after execution based on the operation instructions in the Android system, obtaining change information, feeding back the change information as a new round of context to the large language model, and determining a task execution state; According to the task execution state, natural language feedback information is generated and fed back to the user.

[0006] In an embodiment, the step of generating a structured context containing semantic information of the interactive elements according to the real-time interface state of the current Android screen includes: Obtaining a complete view hierarchy of the real-time interface state of the current Android screen through an accessibility service interface of the Android system; Parsing all visible and interactive user interface elements from the view hierarchy and extracting attribute information of each user interface element; Based on the hierarchical relationship of the user interface elements and the attribute information, a structured context describing the semantic association of the interface is constructed.

[0007] In an embodiment, the step of constructing a structured context describing the semantic association of the interface based on the hierarchical relationship of the user interface elements and the attribute information includes: Taking each of the user interface elements as a node, and taking the attribute information corresponding to the node as a node attribute; According to the parent-child relationship in the view hierarchy, a vertical connection edge between nodes is established; According to the spatial adjacent relationship of the elements in the screen display area, a horizontal connection edge between nodes is established; Adding type labels to the vertical connection edge and the horizontal connection edge, and generating a structured context describing the semantic association of the interface according to the node attribute and the type label.

[0008] In an embodiment, the step of inputting the natural language instruction and the structured context into a large language model to generate a hierarchical task planning containing at least one atomic operation and having a logical dependency relationship between operations includes: Inputting the natural language instruction and the structured context into a large language model to obtain the top-level intention of the user and the atomic tasks for realizing the top-level intention; Based on the structured context, analyzing the interface elements and the pre-operation conditions on which each of the atomic tasks depends, and based on the interface elements and the pre-operation conditions, determining the execution order and logical constraints of each of the atomic operations; Based on the execution order and the logical constraints, a hierarchical task planning is generated.

[0009] In an embodiment, the step of analyzing the interface elements and the pre-operation conditions on which each of the atomic tasks depends based on the structured context, and determining the execution order and logical constraints of each of the atomic operations based on the interface elements and the pre-operation conditions includes: traversing each atomic operation in the hierarchical task planning, identifying a dependent interface element or a dependent system interface state of the atomic operation; determining whether the dependent interface element or the dependent system interface state exists in the current structured context and whether it is in a triggerable state; when the dependent interface element or the dependent system interface state exists in the structured context and is in a triggerable state, determining the relationship between the dependent interface element or the dependent system interface state of the atomic operation and a previous atomic operation; when the dependent interface element or the dependent system interface state of the atomic operation is generated by the execution of the previous atomic operation, determining the sequential dependency and logical constraints of the atomic operation and the previous atomic operation; comprehensively integrating the dependency relationships between all atomic operations to obtain a global execution order.

[0010] In an embodiment, the step of converting the atomic operations in the hierarchical task planning into executable operation instructions of the Android system based on the dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state comprises: extracting each atomic operation from the hierarchical task planning; before executing each atomic operation, obtaining the latest captured real-time interface state, and determining whether the target element of the atomic operation exists and is in an operable state according to the latest captured real-time interface state to obtain a verification result; when the verification result is a target verification result, determining the action type of the atomic operation, mapping the action type to a corresponding Android system underlying input event interface, and generating executable operation instructions.

[0011] In an embodiment, the step of determining whether the target element of the atomic operation exists and is in an operable state according to the latest captured real-time interface state to obtain a verification result comprises: extracting the positioning features of the target element from the atomic operation; matching the positioning features with the element attributes in the structured context corresponding to the latest captured real-time interface state; if the matching is successful, determining whether the target element is visible, not disabled, and its interaction area is not occluded in the latest captured real-time interface state to obtain a verification result.

[0012] In an embodiment, the step of receiving the change of the interface state after the execution of the operation instructions in the Android system to obtain change information, feeding the change information back to the large language model as a new round of context, and determining the task execution state comprises: receive a feedback interface state based on the operation instruction in the Android system based on a preset time interval, determine the interface state change of the feedback interface state and the initial interface state, obtain change information; analyze the change information to obtain change key elements and state change content; the change key elements, the state change content and the feedback interface state as a new round of context; feed back the new round of context to the large language model to obtain a task execution state.

[0013] In an embodiment, the step of generating natural language feedback information according to the task execution state and feeding back the feedback information to the user comprises: generate summary text based on the execution history of the entire task chain and the task execution state; readability optimization of the summary text to obtain natural language feedback information; feed back the natural language feedback information to the user through text-to-speech or text box in the graphical user interface.

[0014] In addition, in order to achieve the above-mentioned purpose, the present application also provides an Android system user interface interaction device based on a large language model, which comprises: The structured context module is used for receiving natural language instructions of the user and synchronously capturing real-time interface states of the current Android screen, and generating a structured context containing interactive element semantic information according to the real-time interface states; The hierarchical task planning module is used for inputting the natural language instructions and the structured context into a large language model to generate a hierarchical task planning containing at least one atomic operation and having a logical dependency relationship between operations; The operation instruction conversion module is used for converting the atomic operations in the hierarchical task planning into operation instructions executable by the Android system based on a dynamic adaptation mechanism of the hierarchical task planning and the real-time interface states; The task state determination module is used for receiving changes of interface states based on the operation instructions in the Android system to obtain change information, feeding back the change information as a new round of context to the large language model, and determining a task execution state; The user feedback generation module is used for generating natural language feedback information according to the task execution state and feeding back the feedback information to the user.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides an Android system user interface interaction device based on a large language model, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the above-mentioned Android system user interface interaction method based on a large language model.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned Android system user interface interaction method based on a large language model.

[0017] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned Android system user interface interaction method based on a large language model.

[0018] The present application provides an Android system user interface interaction method based on a large language model, which receives a natural language instruction of a user, synchronously captures a real-time interface state of a current Android screen to generate a structured context containing semantic information of an interactive element, inputs the instruction and the context into a large language model to generate a hierarchical task planning composed of a plurality of atomic operations with logical dependency relationship, converts the atomic operation sequence into an operation instruction executable by an Android system based on a dynamic adaptation mechanism, receives interface change information after execution of the instruction and feeds back to the large language model as a new round of context to determine a task execution state, and finally generates natural language feedback information to the user according to the execution state. In the above manner, the present application can understand and execute complex cross-application tasks, and significantly improve the intelligence of Android system interaction through real-time interface perception and dynamic planning adjustment. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0021] Figure 1 Flowchart of the Android system user interface interaction method based on a large language model according to Embodiment One of the present application; Figure 2Fig. 1 is a schematic diagram of a login page in an Android system according to an embodiment of the Android system user interface interaction method based on a large language model of the present application; Figure 3 Fig. 2 is a schematic diagram of a module structure of an Android system user interface interaction device based on a large language model according to an embodiment of the present application; Figure 4 Fig. 3 is a schematic diagram of a device structure of a hardware running environment involved in an Android system user interface interaction method based on a large language model according to an embodiment of the present application.

[0022] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0024] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the accompanying drawings and specific embodiments.

[0025] The main solution of the embodiment of the present application is: receiving a natural language instruction of a user, and synchronously capturing a real-time interface state of a current Android screen, generating a structured context containing semantic information of an interactive element according to the real-time interface state; inputting the natural language instruction and the structured context into a large language model together, generating a hierarchical task planning containing at least one atomic operation and having a logical dependency relationship between operations; based on a dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state, sequentially converting the atomic operation in the hierarchical task planning into an operation instruction executable by an Android system; receiving a change of an interface state after execution of the operation instruction in the Android system, obtaining change information, feeding back the change information as a new round of context to the large language model, and determining a task execution state; generating feedback information in natural language according to the task execution state, and feeding back the feedback information to the user.

[0026] Currently, in the existing Android system interaction mode, the user needs to interact with the application interface through manual operations such as touch and click. With the increasing complexity of application functions, completing a task often requires the user to perform a series of tedious operations among multiple applications and multiple hierarchical interfaces. For example, if the user wants to "send the last photo taken last night to Zhang San through WeChat", he needs to complete a series of steps such as unlocking the phone, finding and opening the album, filtering photos, selecting sharing, finding and clicking WeChat, and selecting contacts. This way not only is inefficient, but also has a use threshold for users with limited mobility or special needs. Although there are some automated scripts or voice assistants in the prior art, they can usually only perform pre-set and fixed processes, lack deep semantic understanding of complex natural language instructions and flexible adaptation to dynamically changing interface states, and cannot truly realize user-intent-centered intelligent interaction.

[0027] The present application provides a solution that receives the user's natural language instruction and synchronously captures the real-time interface state of the current Android screen to generate a structured context containing the semantic information of the interactive elements; inputs the instruction and the context into a large language model to generate a hierarchical task plan composed of multiple atomic operations with logical dependencies; based on a dynamic adaptation mechanism, converts the atomic operation sequence into operation instructions executable by the Android system; receives interface change information after instruction execution and feeds it back to the large language model as a new round of context to determine the task execution state; and finally generates natural language feedback information to the user according to the execution state. In this way, the present application can understand and execute complex cross-application tasks, significantly improving the intelligence of Android system interaction through real-time interface perception and dynamic planning adjustment.

[0028] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an Android system user interface interaction device based on a large language model, etc., and the present embodiment does not make specific limitations thereon. The following will take the Android system user interface interaction device based on a large language model as an example to describe the present embodiment and the following embodiments.

[0029] All actions of obtaining signals, information or data in the present application are carried out in compliance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the corresponding device owner.

[0030] The present application provides an Android system user interface interaction method based on a large language model, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the Android system user interface interaction method based on a large language model of the present application is shown in the figure.

[0031] In this embodiment, the Android system user interface interaction method based on a large language model comprises steps S10-S50: Step S10, receiving a natural language instruction of a user, and synchronously capturing a real-time interface state of a current Android screen, and generating a structured context containing semantic information of an interactive element according to the real-time interface state.

[0032] It should be noted that the natural language instruction refers to a task request issued by the user in the form of daily spoken language or written language, for example, “send this picture to Zhang San through WeChat”, which is characterized by complexity and abstraction and needs to be decomposed and understood. The real-time interface state refers to a complete and dynamic snapshot of the user interface (UI) displayed on the screen of the Android device at the moment when the user instruction is received. The interactive element refers to a component on the screen interface that can respond to operations such as clicking, long pressing, and inputting, for example, buttons, text boxes, check boxes, etc. The structured context refers to formatted data with clear semantics and associated relationships that is easy for a large language model to understand and reason after the original and disorganized interface view hierarchy data is parsed and reconstructed.

[0033] It can be understood that the complete view hierarchy of the current screen is obtained in a programmatic manner through the accessibility service (Accessibility Service) interface provided by the Android system, which is an XML structure containing all UI elements and their attributes such as text, coordinates, and whether they can be clicked. Subsequently, instead of directly throwing the original data with a large amount of redundant information to the large language model, the system performs in-depth cleaning and semantic enhancement, for example, filtering out invisible elements, identifying core interactive components, and constructing connections between elements according to spatial positions and hierarchical relationships. Finally, the structured context generated in this process (which may be presented in JSON or a specific format of text description) converts the screen state into a map rich in semantic information, providing a key basis for the large language model to understand where and what can be done, thereby anchoring the abstract instruction and the specific operating system environment together.

[0034] In a feasible implementation, the step of synchronously capturing a real-time interface state of a current Android screen and generating a structured context containing semantic information of an interactive element according to the real-time interface state comprises: obtaining a complete view hierarchy structure of the real-time interface state of the current Android screen through an accessibility service interface of the Android system; parsing all visible and interactive user interface elements from the view hierarchy structure and extracting attribute information of each user interface element; Construct a structured context describing the semantic association of the interface based on the hierarchical relationship of the user interface elements and the attribute information.

[0035] It should be noted that the accessibility service interface is a special framework provided by the Android system, which allows authorized applications to listen to system events, retrieve screen focus, and programmatically obtain the complete view hierarchy structure of the current active window. The view hierarchy structure refers to a data model in which all user interface components that make up the current screen interface are organized in a tree structure. This tree describes the parent-child nesting relationship, relative position, and attributes (such as ID, text content, coordinates, and whether it can be clicked) between various components (such as buttons and text boxes).

[0036] In a specific implementation, obtaining the original view hierarchy structure in XML format through the accessibility service interface is a complex tree structure containing a large amount of redundancy and internal nodes. First, node filtering is performed through predefined rules, retaining only elements that users may interact with. The rules are set as follows:

[0037] Subsequently, the key attribute information of each retained element is extracted. Then, the bounds attribute in the format is processed and converted into the center point coordinates of the element, providing a location basis for subsequent simulated clicks. This process can be formalized as a filtering function , where each is a set of its key attributes.

[0038] Then, based on the original hierarchical relationship and spatial position, the semantic association between elements is reconstructed in the structured context. For example, a text input box and a button labeled "Search" are adjacent on the interface and will be organized together in the structured context, possibly with a logical grouping tag such as "search box combination". This reconstruction promotes low-level UI coordinate data to high-level semantic maps that can be understood by large language models for task planning. The objective function can be represented as , with the goal of maximizing the auxiliary utility of the structured context for large language models to understand interface functionality and execute operation planning.

[0039] In a feasible implementation, the step of constructing a structured context describing the semantic association of the interface based on the hierarchical relationship of the user interface elements and the attribute information includes: Taking each user interface element as a node, the attribute information corresponding to the node is taken as a node attribute; According to the parent-child relationship in the view hierarchy structure, longitudinal connection edges between nodes are established. According to the spatial adjacent relationship of the elements in the screen display area, a horizontal connection edge between nodes is established; A type label is added to the vertical connection edge and the horizontal connection edge, and a structured context describing the semantic association of the interface is generated according to the node attribute and the type label.

[0040] It should be noted that the user interface elements screened out are abstracted as nodes, the nodes carry all the attribute information of the elements, and the node attribute is a set of key-value pairs attached to each node, which is used to describe all the features of the user interface, for example: The vertical connection edge refers to an edge in the graph structure, which is established according to the parent-child subordinate relationship of the original view hierarchy. For example, a login button node can be a child node of a vertical layout node. This edge reflects the assembly logic and grouping relationship of the UI. The horizontal connection edge refers to an edge in the graph structure, which is established according to the spatial adjacent relationship (such as left and right adjacent, top and bottom adjacent, or within a certain pixel threshold) of the elements on the screen. This edge is independent of the code hierarchy, but is based on the visual layout, and reflects the functional parallel or operation flow sequence relationship between the elements.

[0041] In a specific implementation, the system models the UI as a graph , where the node set is composed of all UI elements, and the edge set is composed of the union of vertical edges and horizontal edges . The vertical edge is established directly by parsing the nested relationship of the view tree. If element A is the direct parent node of element B, an edge is established and marked as . The establishment of the horizontal edge depends on the spatial geometry calculation, and the bounding box (bounds) of each element is usually used. For example, the following rules can be used: if the bounding boxes of two non-overlapping elements B and C have a large overlap in the vertical direction and are horizontally adjacent, a horizontal edge is established and marked as . This process can be formalized as a function .

[0042] The generated graph structure The complex object in memory itself needs to be serialized into a text format (such as a structured JSON or plain text description) that the large language model can understand. The serialization process traverses the nodes and edges in the graph, integrating node attributes and edge relationship type labels into a coherent natural language description or specific data schema. For example, the final output might be: "A 'login' button (type: Button) is located within the 'user credential input box group' (type: LinearLayout) [vertical relationship]. Next to it on the right is a 'forgot password' link [horizontal relationship]." Referring to Figure 2 , Figure 2 is a schematic diagram of a login page in the Android system. This structured context rich in relationship semantics can help large language models better understand the logic of interface layout, making more accurate task planning. The goal is to convert the graph into a text description that maximizes information , .

[0043] Step S20, input the natural language instructions and the structured context into a large language model to generate a hierarchical task plan containing at least one atomic operation and logical dependencies between operations.

[0044] It should be noted that the hierarchical task plan refers to the plan generated by the large language model, which may not be a flat linear list, but a hierarchical structure. High-level abstract tasks (such as "login application") are broken down into a series of low-level atomic operation sequences, forming an organized plan tree. Atomic operation: refers to a basic action that can be directly executed at the Android automation framework (such as UIAutomator) level and cannot be further divided. For example: click (ui_element) (click), set_text (ui_element, input_string) (input text), scroll (direction) (scroll) and so on. They are the basic instruction set that constitutes complex tasks.

[0045] It can be understood that by designing a specific prompt template, the user instruction (such as "help me book a high-speed rail ticket from Beijing to Shanghai tomorrow") is spliced with the structured context representing the current screen state as input provided to the large language model, and guided by technologies such as thinking chain to perform task decomposition: first understand the final goal, and then deduce the prerequisites and operation sequence required to achieve the goal, and finally output a structured JSON or specific format text, which explicitly lists the atomic operations in sequence, the target UI element identifier (such as resource-id) of each operation, and the parameters required for the operation (such as the text to be input), thereby accurately mapping the user's natural language command to a set of executable automation scripts in the current interface environment.

[0046] In a feasible implementation, the step of inputting the natural language instruction and the structured context into the large language model to generate a hierarchical task plan containing at least one atomic operation and having a logical dependency relationship between operations includes: inputting the natural language instruction and the structured context into the large language model to obtain a user's top-level intent and atomic tasks for achieving the top-level intent; analyzing the interface elements and pre-operation conditions on which each atomic task depends based on the structured context, and determining the execution order and logical constraints of each atomic operation based on the interface elements and the pre-operation conditions; generating a hierarchical task plan based on the execution order and the logical constraints.

[0047] It should be noted that the top-level intent refers to the ultimate goal or high-level goal that the user's natural language instruction hopes to achieve. It is the driving force of atomic tasks. For example, the top-level intent of the instruction "help me clear the shopping cart" is clear_shopping_cart, and the top-level intent of "log in to my account" is user_login. The large language model first needs to accurately abstract this intent to guide subsequent task decomposition. Atomic tasks refer to sub-goals or steps that are one level higher than "atomic operations" and are abstracted by the large language model based on the top-level intent. It is still a description of "what to do", but it has not been specified to the UI operation level. For example, to achieve the top-level intent clear_shopping_cart, the atomic tasks that are decomposed may include: navigate_to_cart_page (navigate to the shopping cart page), select_all_items (select all items), confirm_deletion (confirm deletion), or submit_all_orders (submit all orders). Atomic tasks are intermediate abstraction layers in the planning process.

[0048] It should be understood that pre-conditions refer to the state that the interface must satisfy before performing a certain atomic operation or atomic task. This is a key concept of logical constraints. For example, the pre-conditions of the atomic operation click("Confirm Delete Button") are: a) "Confirm Delete Button" must be visible and clickable in the current interface; b) At least one item has been selected. These conditions determine the feasibility and order of operations. Logical constraints refer to rules or dependencies that must be followed in the task flow, which determine the execution order between atomic operations / tasks. They mainly include sequential constraints (A must be executed before B), conditional constraints (Y is executed only when condition X is met), and loop constraints (operation Z is performed on each item in the list).

[0049] In a specific implementation, the input of the large language model is a carefully designed prompt word, whose template can be: "You are an Android automation assistant. The current interface information is as follows: [structured context]. The user's instruction is: [natural language instruction]. Please first analyze the user's top-level intent, and then decompose it into a series of atomic tasks." The model will reason and output a structured intermediate representation. This process can be formally represented as: Here, SubTask is an atomic task, which together constitutes the outline of the plan to achieve the intent.

[0050] The large language model needs to map each atomic task to a specific atomic operation, and analyze the pre-conditions for executing these operations. For example, for the atomic task navigate_to_cart_page, the model needs to determine, based on the structured context, that if the current interface already has a "cart" icon, the atomic operation is click(cart_icon); if not, it may need to first execute click(more_options_menu) and click(cart_item). This context-based conditional judgment generates logical constraints between operations. Finally, the model organizes all atomic operations into a linear or conditional branching sequence according to the execution order and logical constraints, i.e., the final executable hierarchical task plan. This process can be seen as a function: where is the final task planning sequence , and satisfies all constraint conditions .

[0051] In a feasible implementation, the step of analyzing the interface elements and pre-conditions that each atomic task depends on based on the structured context, determining the execution order and logical constraints of each atomic operation based on the interface elements and pre-conditions comprises: traversing each atomic operation in the hierarchical task plan, identifying a dependent interface element or a dependent system interface state of the atomic operation; determining whether the dependent interface element or the dependent system interface state exists in the current structured context and whether it is in a triggerable state; when the dependent interface element or the dependent system interface state exists in the structured context and is in the triggerable state, determining a relationship of the dependent interface element or the dependent system interface state of the atomic operation with a previous atomic operation; when the dependent interface element or the dependent system interface state of the atomic operation is generated by execution of the previous atomic operation, determining a sequential dependency relationship and a logical constraint of the atomic operation and the previous atomic operation; comprehensively determining a global execution order among all atomic operations.

[0052] It should be noted that the dependent interface element / dependent system interface state refers to a specific embodiment of a prerequisite condition on an interface for executing an atomic operation. The dependent interface element refers to a specific UI component, such as the atomic operation: the dependent interface element of clicking the login button is the “login button” itself. The dependent system interface state can be more abstract, referring to global states such as “currently on the application home page”, “keyboard has been popped up”, or “a certain permission has been granted”. The two together constitute the precondition of the atomic operation.

[0053] The triggerable state refers to whether the dependent interface element or system state can currently be interacted with or meet the operation condition. For an interface element, this means that it not only exists, but is also visible, not disabled, and can be clicked / focused. For a system state, it means that the state is true (such as the keyboard has indeed been popped up). This is the key to determining whether an operation can be executed immediately.

[0054] The sequential dependency relationship refers to a mandatory order constraint, that is, operation B must be executed after operation A because the condition on which B depends is created by the execution result of A. The logical constraint includes the sequential constraint and can also include conditional judgments (for example, if A fails, execute B, otherwise execute C).

[0055] In a specific implementation, first, the system traverses the atomic operation sequence generated by the large language model . For each operation , the system extracts its precondition set from its description, which includes the interface elements and system states it depends on. Then, the system matches with the current structured context to check the condition satisfaction: More importantly, it needs to determine whether the condition in is "fresh", i.e. whether it is produced by the execution of some operation earlier in the sequence. This establishes a directed edge from to , forming a dependency .

[0056] Secondly, by traversing all operations and establishing such dependency edges, the system constructs a directed acyclic graph (DAG), namely , where nodes are operations and the edge set is composed of all dependency relations. This DAG explicitly encodes the "precedence dependencies and logical constraints" among all operations. The final "global execution order" is not simply the original sequence, but a topological sort of this DAG. This sort guarantees that any operation will not be executed until all its predecessors have completed, thus ensuring the logical correctness of the task plan and its executability in the current interface context. This process transforms the latent linear plan generated by LLM into a robust, dependency-based execution flowchart.

[0057] Step S30, based on the dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state, sequentially converts the atomic operations in the hierarchical task planning into operation instructions executable by the Android system.

[0058] It should be noted that the real-time interface state refers to the latest information of the current screen obtained in real time through tools such as Android accessibility services or UI Automator during the execution of the task planning. It is different from the initial structured context used when generating the plan, but is dynamically changing, used to verify the preset conditions and respond to unexpected situations during execution. The dynamic adaptation mechanism refers to a feedback loop that the system has when executing the sequence of atomic operations, rather than mechanically executing them in the pre-generated order. It checks the real-time interface state before and after executing each operation to ensure that the preconditions are met, and can retry, back up or dynamically adjust the subsequent operation path according to the state changes. Operation instructions executable by the Android system: refers to the underlying API calls that can be directly recognized and executed by the Android automated testing framework (such as UI Automator, Appium). For example, the atomic operation "click the login button" is converted into specific code similar to mDevice.findObject(By.res("com.example.app:id / login_btn")).click().

[0059] It can be understood that first, the preconditions of the operation (such as the existence and clickability of the target control) are verified by querying the real-time interface state, and if the preconditions are met, the atomic operation is converted into specific instructions by calling the API of the Android automation framework and executed, and the interface state is obtained again after execution to confirm the operation effect and determine whether to continue to execute the next operation; if the preconditions are not met or the expected state is not reached after execution, a dynamic adaptation mechanism is triggered, such as waiting for the element to appear, retrying the operation, or throwing an exception and re-planning, so as to ensure that the entire task flow can flexibly and robustly cope with dynamic changes such as interface loading delay and pop-up window interference in the actual application environment.

[0060] In a feasible implementation, the step of converting the atomic operations in the hierarchical task planning into operation instructions executable by the Android system in sequence based on the dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state comprises: extracting each atomic operation from the hierarchical task planning; before executing each atomic operation, obtaining the latest captured real-time interface state, and determining whether the target element of the atomic operation exists and is in an operable state according to the latest captured real-time interface state to obtain a verification result; when the verification result is a target verification result, determining the action type of the atomic operation, mapping the action type to a corresponding Android system underlying input event interface, and generating an executable operation instruction.

[0061] In a specific implementation, for each atomic operation in the planning sequence , the system does not directly execute, but first performs perception: calling the function capture_current_state() to obtain the real-time interface context . Then, decision-making is performed: the target element is checked by a verification function to determine whether it meets the operable condition , i.e. . This verification result is the key to deciding whether to continue execution.

[0062] Secondly, when is true (i.e., the target verification result), the system enters the execution phase. It parses the action type of such as CLICK, and then converts it to a specific Android underlying interface through a preset mapping function The call to Map(CLICK) may correspond to accessibilityNodeInfo.performAction(ACTION_CLICK). Ultimately, this interface call is encapsulated into a concrete, executable action instruction and executed by the system, thus completing the final conversion from abstract task planning to physical device interaction.

[0063] In one possible implementation, the step of determining, according to the latest captured real-time interface state, whether the target element of the atomic operation exists and is in an operable state to obtain a verification result includes: extracting a positioning feature of the target element from the atomic operation; matching the positioning feature with an element attribute in a structured context corresponding to the latest captured real-time interface state; if the matching is successful, determining whether the target element is visible, not disabled, and its interaction region is not occluded in the latest captured real-time interface state to obtain the verification result.

[0064] In a specific implementation, an abstract task target is anchored to a specific interface control instance through a precise positioning matching algorithm (such as a query based on a resource-id, text, XPath, or other positioning features in a real-time acquired UI layout XML tree); after the matching is successful, the system further parses multiple key UI attributes (such as visible-to-user, enabled, and bounds) of the control instance, judges the visibility, enabled state, and calculates the screen region and current window layering relationship to evaluate the occlusion situation, so as to comprehensively determine whether the element is truly in a stable, acceptable user input operable state, and finally generate a Boolean type verification result to determine the subsequent flow direction.

[0065] Step S40, receiving a change in an interface state in the Android system based on execution of the operation instruction, obtaining change information, feeding back the change information as a new round of context to the large language model, and determining a task execution state.

[0066] It should be noted that the change information refers to a key difference identified by comparing the real-time interface states before and after the execution of the operation instruction. This is not only a snapshot of the entire interface tree, but also an analyzed and meaningful signal, for example: whether the target page is successfully jumped, the appearance or disappearance of a specific UI element (such as a “login success” prompt), the loading of expected content (such as a search result list), or the pop-up of an error prompt box. It is a precise description of the operation result.

[0067] It can be understood that after the system drives the device to execute the operation instruction, it immediately obtains the new interface state through the way of watching the change or actively recapturing, and differentiates the state before the operation to analyze the key change information representing the operation result, and then feeds back the information as a new round of context rich in dynamic result to the large language model, so that the model judges the current task execution state (such as "completed", "need next step" or "stuck") based on its semantic understanding ability, thereby providing a decision basis for whether to continue to execute subsequent planning or need to re-plan.

[0068] In a feasible implementation, the step of receiving the change of the interface state in the Android system based on the execution of the operation instruction to obtain change information, and feeding back the change information as a new round of context to the large language model to determine the task execution state includes: receiving the feedback interface state in the Android system based on the execution of the operation instruction based on a preset time interval, determining the interface state change of the feedback interface state and the initial interface state, and obtaining change information; analyzing the change information to obtain change key elements and state change content; taking the change key elements, the state change content, and the feedback interface state as a new round of context; feeding back the new round of context to the large language model to obtain the task execution state.

[0069] In a specific implementation, instead of passively waiting for the interface to be stable, the system actively captures the feedback interface state after operation execution based on a preset time strategy such as polling or intelligent waiting, accurately extracts the interface state change information by comparing the difference between the initial state before operation and the feedback interface state; then, through semantic analysis of the change information, such as identifying new specific UI elements or text content, the change key elements and their state change content are extracted, and this structured information and the complete feedback interface state together form a new round of context rich in dynamic semantics; finally, the context is fed back to the large language model, which uses powerful situational reasoning ability to comprehensively judge the overall execution state of the current task, such as "successfully advancing to the next step", "stuck in a pop-up error" or "task completed", thereby realizing intelligent interpretation of environmental feedback and autonomous decision-making of task progress.

[0070] Step S50, generating feedback information in natural language according to the task execution state, and feeding back the feedback information to the user.

[0071] It can be understood that the system takes the internal decision of the structured task execution state as input, converts and polishes it into a natural language statement conforming to the dialogue habits by using the text generation capability of the large language model or the pre-defined templating rules, and finally presents the feedback information containing the task progress, success confirmation or problem description to the user through a graphical interface or voice, thereby completing a complete automatic closed loop from "perception-decision-execution-evaluation" to "communication", greatly improving the transparency of the task and the user experience.

[0072] In a feasible implementation, the step of generating the feedback information in natural language according to the task execution state and feeding back the feedback information to the user comprises: generating a summary text based on the execution history of the entire task chain and the task execution state; optimizing the readability of the summary text to obtain the feedback information in natural language; feeding back the feedback information in natural language to the user through text-to-speech or a text box in a graphical user interface.

[0073] In a specific implementation, first, a large language model is used to generate a preliminary summary text with coherent content based on the complete execution history of the entire task chain (including completed steps and encountered conditions) in combination with the current task execution state; then, the text is optimized for readability, such as adjusting the wording to make it more colloquial, adding logical conjunctions or simplifying technical terms, to obtain the final user-friendly natural language feedback information; finally, the system actively feeds back the clear task execution result to the user through multi-modal output channels (such as converting the text into voice broadcast or dynamically displaying it in the text box of the dialogue bubble, notification bar, etc. in the application interface), thereby forming a transparent interactive closed loop and enabling the user to intuitively understand the execution process and actual state of the system.

[0074] The embodiment provides an Android system user interface interaction method based on a large language model, which comprises the following steps: receiving a natural language instruction of a user, and synchronously capturing a real-time interface state of a current Android screen to generate a structured context containing semantic information of an interactive element; inputting the instruction and the context into a large language model to generate a hierarchical task plan composed of a plurality of atomic operations with logical dependency relationships; converting the sequence of atomic operations into operation instructions executable by an Android system based on a dynamic adaptation mechanism; receiving interface change information after execution of the instruction and feeding it back to the large language model as a new round of context to determine a task execution state; and finally generating natural language feedback information for the user according to the execution state. In the foregoing manner, the present application can understand and execute complex cross-application tasks, and significantly improve the intelligence of Android system interaction through real-time interface perception and dynamic planning adjustment.

[0075] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the Android system user interface interaction method based on a large language model of the present application. Further simple transformations in more forms based on this technical concept are within the protection scope of the present application.

[0076] The present application also provides an Android system user interface interaction device based on a large language model, which refers to Figure 3 The Android system user interface interaction device based on a large language model comprises: A structured context module 10 is configured to receive a natural language instruction of a user, synchronously capture a real-time interface state of a current Android screen, and generate a structured context containing semantic information of an interactive element according to the real-time interface state; A hierarchical task planning module 20 is configured to input the natural language instruction and the structured context into a large language model together, and generate a hierarchical task planning containing at least one atomic operation and a logical dependency relationship between operations; An operation instruction conversion module 30 is configured to convert the atomic operation in the hierarchical task planning into an operation instruction executable by an Android system in sequence based on a dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state; A task state determination module 40 is configured to receive a change of an interface state after execution of the operation instruction in an Android system, obtain change information, feed back the change information as a new round of context to the large language model, and determine a task execution state; A user feedback generation module 50 is configured to generate feedback information in a natural language according to the task execution state, and feed back the feedback information to the user.

[0077] In a feasible implementation, the structured context module 10 is further configured to obtain a complete view hierarchy structure of the real-time interface state of the current Android screen through an accessibility service interface of the Android system; All visible and interactive user interface elements are parsed from the view hierarchy structure, and attribute information of each user interface element is extracted; Based on the hierarchical relationship of the user interface elements and the attribute information, a structured context describing semantic association of an interface is constructed.

[0078] In a feasible implementation, the structured context module 10 is further configured to take each user interface element as a node, and take the attribute information corresponding to the node as a node attribute; According to a parent-child subordinate relationship in the view hierarchy structure, a vertical connection edge between nodes is established; According to a spatial adjacent relationship of elements in a screen display area, a horizontal connection edge between nodes is established; adding a type label to the longitudinal connection edge and the transverse connection edge, and generating a structured context describing the interface semantic association according to the node attribute and the type label.

[0079] In an implementable embodiment, the hierarchical task planning module 20 is further configured to input the natural language instruction and the structured context into a large language model to obtain a top-level intention of the user and atomic tasks for realizing the top-level intention. analyzing interface elements and pre-operation conditions on which each atomic task depends based on the structured context, and determining an execution order and a logical constraint of each atomic operation based on the interface elements and the pre-operation conditions; generating a hierarchical task planning based on the execution order and the logical constraint.

[0080] In an implementable embodiment, the hierarchical task planning module 20 is further configured to traverse each atomic operation in the hierarchical task planning, identify a dependent interface element or a dependent system interface state of the atomic operation. determining whether the dependent interface element or the dependent system interface state exists in the current structured context and whether it is in a triggerable state; when the dependent interface element or the dependent system interface state exists in the structured context and is in the triggerable state, determining a relationship between the dependent interface element or the dependent system interface state of the atomic operation and a previous atomic operation; when the dependent interface element or the dependent system interface state of the atomic operation is generated by execution of the previous atomic operation, determining a sequential dependency and a logical constraint between the atomic operation and the previous atomic operation; comprehensively determining a global execution order based on the dependency between all atomic operations.

[0081] In an implementable embodiment, the operation instruction conversion module 30 is further configured to extract each atomic operation from the hierarchical task planning. before executing each atomic operation, obtaining a latest captured real-time interface state, and determining whether a target element of the atomic operation exists and is in an operable state based on the latest captured real-time interface state to obtain a verification result; when the verification result is a target verification result, determining an action type of the atomic operation, mapping the action type to a corresponding Android system bottom-layer input event interface, and generating an executable operation instruction.

[0082] In an implementable embodiment, the operation instruction conversion module 30 is further configured to extract a positioning feature of a target element from the atomic operation. matching the positioning feature with an element attribute in a structured context corresponding to the latest captured real-time interface state; If the matching is successful, it is determined whether the target element is visible, not disabled, and its interaction region is not occluded in the latest captured real-time interface state, to obtain a verification result.

[0083] In a feasible implementation, the task state determination module 40 is further configured to receive a feedback interface state after execution of the operation instruction in the Android system based on a preset time interval, determine an interface state change between the feedback interface state and the initial interface state, and obtain change information; The change information is analyzed to obtain a change key element and state change content; The change key element, the state change content, and the feedback interface state are taken as a new round of context; The new round of context is fed back to the large language model to obtain a task execution state.

[0084] In a feasible implementation, the user feedback generation module 50 is further configured to generate summary text based on an execution history of the entire task chain and the task execution state; The summary text is readable optimized to obtain natural language feedback information; The natural language feedback information is fed back to the user through text-to-speech or a text box in a graphical user interface.

[0085] The Android system user interface interaction device based on the large language model provided in the present application adopts the Android system user interface interaction method based on the large language model in the above embodiments, which can solve the technical problem of being unable to perform reliable task planning and execution in a dynamically changing Android interface environment. Compared with the prior art, the Android system user interface interaction device based on the large language model provided in the present application has the same beneficial effects as the Android system user interface interaction method based on the large language model provided in the above embodiments, and other technical features in the Android system user interface interaction device based on the large language model are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0086] The present application provides an Android system user interface interaction device based on a large language model, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the Android system user interface interaction method based on the large language model in the above embodiment one.

[0087] The following is for reference. Figure 4 This document illustrates a structural schematic diagram of an Android system user interface interaction device suitable for implementing embodiments of this application based on a large language model. The Android system user interface interaction device based on a large language model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The Android system user interface interaction device based on the large language model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0088] like Figure 4 As shown, the Android system user interface interaction device based on a large language model may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the Android system user interface interaction device based on a large language model. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows Android user interface interaction devices based on a large language model to wirelessly or wiredly communicate with other devices to exchange data. Although Android user interface interaction devices based on a large language model with various systems are shown in the figures, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0089] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0090] The Android system user interface interaction device based on a large language model provided by the present application adopts the Android system user interface interaction method based on a large language model in the above-mentioned embodiments, and can solve the technical problem of Android system user interface interaction based on a large language model. Compared with the prior art, the Android system user interface interaction device based on a large language model provided by the present application has the same beneficial effects as the Android system user interface interaction method based on a large language model provided by the above-mentioned embodiments, and other technical features in the Android system user interface interaction device based on a large language model are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0091] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0092] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0093] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the Android system user interface interaction method based on a large language model in the above-mentioned embodiments.

[0094] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory or flash memory), an optical fiber, a CD-ROM (CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0095] The computer readable storage medium described above may be contained in a large language model-based Android system user interface interaction device; or may exist independently and not be assembled into a large language model-based Android system user interface interaction device.

[0096] The computer readable storage medium described above carries one or more programs, which, when executed by the large language model-based Android system user interface interaction device, cause the large language model-based Android system user interface interaction device to: receive a natural language instruction of a user, and synchronously capture a real-time interface state of a current Android screen, generate a structured context containing semantic information of an interactive element according to the real-time interface state; Input the natural language instruction and the structured context into a large language model together to generate a hierarchical task planning containing at least one atomic operation and having a logical dependency relationship between operations; Based on a dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state, sequentially convert the atomic operations in the hierarchical task planning into operation instructions executable by the Android system; Receive a change in the interface state of the Android system based on the execution of the operation instruction, obtain change information, feed back the change information as a new round of context to the large language model, and determine a task execution state; According to the task execution state, natural language feedback information is generated and fed back to the user.

[0097] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0098] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0099] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0100] The readable storage medium provided by the application is a computer readable storage medium, and the computer readable storage medium stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned large language model-based Android system user interface interaction method, and can solve the technical problem of large language model-based Android system user interface interaction. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the application are the same as those of the large language model-based Android system user interface interaction method provided by the above-mentioned embodiments, and are not described here.

[0101] The application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the large language model-based Android system user interface interaction method as described above.

[0102] The computer program product provided by the application can solve the technical problem of large language model-based Android system user interface interaction. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the large language model-based Android system user interface interaction method provided by the above-mentioned embodiments, and are not described here.

[0103] The above is only some embodiments of the application, and does not limit the patent scope of the application, and any equivalent structural transformation made by using the content of the application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the application.

Claims

1. A user interface interaction method for an Android system based on a large language model, characterized in that, The Android system user interface interaction method based on a large language model includes: Receive natural language commands from users and simultaneously capture the real-time interface state of the current Android screen, and generate a structured context containing semantic information of interactive elements based on the real-time interface state. The natural language instructions and the structured context are input into the large language model to generate a hierarchical task plan that includes at least one atomic operation and has logical dependencies between the operations. Based on the dynamic adaptation mechanism between the hierarchical task planning and the real-time interface state, the atomic operations in the hierarchical task planning are sequentially converted into operation instructions executable by the Android system. The system receives changes in the interface state after the operation command is executed in the Android system, obtains change information, and feeds the change information back to the large language model as a new round of context to determine the task execution state. The system generates natural language feedback information based on the task execution status and sends the feedback information back to the user.

2. The method as described in claim 1, characterized in that, The steps of synchronously capturing the real-time interface state of the current Android screen and generating a structured context containing semantic information of interactive elements based on the real-time interface state include: The complete view hierarchy of the current Android screen's real-time interface state is obtained through the Android system's accessibility service interface. All visible and interactive user interface elements are parsed from the view hierarchy, and the attribute information of each user interface element is extracted. Based on the hierarchical relationship of the user interface elements and the attribute information, a structured context describing the semantic associations of the interface is constructed.

3. The method as described in claim 2, characterized in that, The step of constructing a structured context describing the semantic relationships of the interface based on the hierarchical relationship of the user interface elements and the attribute information includes: Each of the user interface elements is treated as a node, and the attribute information corresponding to the node is used as a node attribute; Based on the parent-child hierarchy in the view hierarchy, establish vertical connection edges between nodes; Based on the spatial adjacency of elements within the screen display area, establish horizontal connection edges between nodes; Type labels are added to the vertical and horizontal connecting edges, and a structured context describing the semantic association of the interface is generated based on the node attributes and the type labels.

4. The method as described in claim 1, characterized in that, The step of inputting the natural language instructions and the structured context into a large language model to generate a hierarchical task plan containing at least one atomic operation and with logical dependencies between operations includes: The natural language instructions and the structured context are input into the large language model to obtain the user's top-level intent and the atomic tasks to achieve the top-level intent; Based on the structured context analysis, the interface elements and preconditions on which each atomic task depends are analyzed, and the execution order and logical constraints of each atomic operation are determined based on the interface elements and preconditions. A hierarchical task plan is generated based on the execution order and the logical constraints.

5. The method as described in claim 4, characterized in that, The step of analyzing the interface elements and preconditions upon which each atomic task depends based on the structured context, and determining the execution order and logical constraints of each atomic operation based on the interface elements and preconditions, includes: Traverse each atomic operation in the hierarchical task plan and identify the dependent interface elements or dependent system interface states of the atomic operation. Determine whether the dependent interface element or the dependent system interface state exists in the current structured context, and determine whether it is in a triggerable state. When the dependent interface element or the dependent system interface state exists in the structured context and is in a triggerable state, determine the relationship between the dependent interface element or the dependent system interface state of the atomic operation and the previous atomic operation. When the dependent interface element or dependent system interface state of the atomic operation is generated by the execution of the previous atomic operation, the sequential dependency relationship and logical constraints between the atomic operation and the previous atomic operation are determined. By combining the dependencies between all atomic operations, the global execution order is obtained.

6. The method as described in claim 1, characterized in that, The step of converting atomic operations in the hierarchical task planning into executable operation instructions of the Android system based on the dynamic adaptation mechanism of the hierarchical task planning and the real-time interface state includes: Extract each atomic operation from the hierarchical task plan; Before executing each atomic operation, the latest captured real-time interface state is obtained, and the existence and operability of the target element of the atomic operation are determined based on the latest captured real-time interface state to obtain the verification result. When the verification result is the target verification result, the action type of the atomic operation is determined, the action type is mapped to the corresponding Android system underlying input event interface, and an executable operation instruction is generated.

7. The method as described in claim 6, characterized in that, The steps for determining whether the target element of the atomic operation exists and is in an operable state based on the latest captured real-time interface state, and obtaining the verification result, include: Extract the location features of the target element from the atomic operations; The location features are matched with the element attributes in the structured context corresponding to the latest captured real-time interface state; If the match is successful, determine whether the target element is visible, not disabled, and whether its interactive area is not obscured in the latest captured real-time interface state, and obtain the verification result.

8. The method as described in claim 1, characterized in that, The steps of receiving changes in the interface state of the Android system based on the execution of the operation command, obtaining change information, and feeding this change information as a new round of context back to the large language model to determine the task execution state include: Based on a preset time interval, receive the feedback interface state in the Android system after the operation command is executed, determine the interface state changes between the feedback interface state and the initial interface state, and obtain change information. The change information is analyzed to obtain the key elements of the change and the content of the state change; The key elements of the change, the state change content, and the feedback interface state are used as the context for a new round; The new round of context is fed back to the large language model to obtain the task execution status.

9. The method as described in claim 1, characterized in that, The step of generating natural language feedback information based on the task execution status and sending the feedback information back to the user includes: Based on the execution history of the entire task chain and the execution status of the tasks, a summary text is generated; The summary text is optimized for readability to obtain natural language feedback information; The natural language feedback information is fed back to the user through text-to-speech or text boxes in a graphical user interface.

10. An Android system user interface interaction device based on a large language model, characterized in that, The Android system user interface interaction device based on a large language model includes: The structured context module is used to receive the user's natural language instructions and simultaneously capture the real-time interface state of the current Android screen, and generate a structured context containing semantic information of interactive elements based on the real-time interface state. The hierarchical task planning module is used to input the natural language instructions and the structured context into the large language model to generate a hierarchical task plan that includes at least one atomic operation and has logical dependencies between the operations. The operation instruction conversion module is used to convert the atomic operations in the hierarchical task planning into operation instructions executable by the Android system in sequence, based on the dynamic adaptation mechanism between the hierarchical task planning and the real-time interface state. The task status determination module is used to receive changes in the interface status in the Android system based on the execution of the operation command, obtain change information, and feed the change information as a new round of context back to the large language model to determine the task execution status. The user feedback generation module is used to generate natural language feedback information based on the task execution status and send the feedback information back to the user.

Images