Method, apparatus, and medium for data processing

Abstract

This paper provides a method, apparatus, and medium for data processing. The method includes obtaining a task sequence corresponding to natural language input, wherein the task sequence includes at least one task to be executed sequentially; and performing the following operations for each task in the execution order of the at least one task: obtaining a user interface node tree corresponding to each task via the accessibility service interface of the operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the user interface node tree; and operating the user interface corresponding to the user interface node based on the user interface node and the operation information. This method can obtain the user interface node tree via the accessibility service interface of the operating system, thereby enabling the execution of user instructions using the user interface node tree, and has advantages such as high accuracy and low performance consumption.

Description

Technical Field

[0001] One or more of the present invention relates to a method for data processing, an apparatus for data processing, and a corresponding electronic device and computer-readable storage medium. Background Technology

[0002] Currently, Artificial Intelligence (AI) technology has achieved significant development and widespread application in accurately understanding and efficiently executing user operation instructions, especially in human-computer interaction scenarios oriented towards Graphical User Interfaces (GUIs). In existing technical solutions, intelligent interaction and operation control for GUIs largely rely on Vision Large Models (VLMs) to analyze and understand device screen screenshots, thereby simulating the visual perception process and logical operation behavior of human users when using terminal devices. The implementation path of this technology typically involves continuously collecting screen image information from the terminal device, using a Vision Large Model to locate, identify, and semantically analyze various interface interaction elements (such as function buttons, text input boxes, menu options, icon controls, etc.) in the images, and further predicting and generating the next operation instruction (such as click operation, text input, swipe operation, long press selection, etc.) based on the recognition results, thereby achieving automated interaction and control of the GUI. Summary of the Invention

[0003] This summary section is provided to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed invention, nor is it intended to limit the scope of the claimed invention.

[0004] This paper provides a method for data processing in at least one scenario, comprising: obtaining a task sequence corresponding to the acquired natural language input based on the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; and performing the following operations for each task in the order of execution of the at least one task: obtaining a user interface node tree corresponding to each task via an accessibility service interface of the operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the acquired user interface node tree; and performing operations on the user interface corresponding to the user interface node based on the acquired user interface node and operation information.

[0005] At least one further embodiment of this document provides an apparatus for data processing, comprising: a first operation unit configured to obtain a task sequence corresponding to the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; and a second operation unit configured to perform the following operations sequentially for each task according to the execution order of the at least one task: obtaining a user interface node tree corresponding to each task via an accessibility service interface of an operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the obtained user interface node tree; and operating the user interface corresponding to the user interface node based on the obtained user interface node and operation information.

[0006] At least one embodiment of this document provides an electronic device comprising: at least one processor, and a memory storing one or more computer program instructions; wherein the one or more computer program instructions are executed by the at least one processor to perform the method provided in at least one embodiment of this document.

[0007] At least one further embodiment of this document provides a non-transitory computer-readable storage medium for storing computer-readable instructions, wherein the method provided in at least one embodiment of this document is implemented when the computer-readable instructions are executed by a processor.

[0008] At least one embodiment of this document provides a computer program product comprising one or more computer instructions, wherein the one or more computer instructions are executed by a processor to perform the method provided in at least one embodiment of this document.

[0009] This paper provides methods, corresponding devices, and media for data processing in one or more scenarios. The methods provided herein can obtain a user interface node tree via the operating system's accessibility service interface, thereby enabling the execution of user instructions using the user interface node tree. These methods offer advantages such as high accuracy, low performance consumption, and low risk of privacy breaches. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various scenarios described herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 The illustration schematically shows an application scenario of the data processing method and apparatus provided in at least one aspect of this disclosure;

[0012] Figure 2A flowchart illustrating a method for data processing provided in at least one scenario of this paper is shown schematically;

[0013] Figure 3 The diagram illustrates the core module structure and data flow relationships of the system provided in at least one scenario of this paper.

[0014] Figure 4 The illustration schematically depicts a complete end-to-end processing flow from user instruction to task completion provided for at least one scenario described herein;

[0015] Figure 5 The diagram illustrates the sequence of interactions between the main actors in a system for a specific example of “finding and clicking the ‘order’ button” provided in at least one scenario of this paper;

[0016] Figure 6 A schematic diagram illustrating the structure of a device for data processing provided in at least one of the scenarios described herein is shown.

[0017] Figure 7 A schematic block diagram of an electronic device in at least one of the scenarios described herein is shown. Detailed Implementation

[0018] The present invention will now be described in more detail with reference to the accompanying drawings. While some aspects of the present invention are illustrated in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and aspects are for illustrative purposes only and are not intended to limit the scope of this invention.

[0019] It should be understood that the steps described in the method embodiments herein may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this document is not limited in this respect.

[0020] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "according to" means "at least partially according to". The term "one situation" means "at least one situation"; the term "another situation" means "at least one additional situation"; the term "some situations" means "at least some situations". Definitions of other terms will be given in the following description.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this article are only used to distinguish different models, devices, modules or units, and are not used to limit the order of the functions performed by these models, devices, modules or units or their interdependencies.

[0022] It should be noted that the terms "one" and "more" used in this document are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices, modules or units in the embodiments herein are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0024] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0025] It is understandable that before using the technical solutions disclosed in various situations in this article, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this article and their authorization should be obtained through appropriate means in accordance with relevant laws and regulations. Relevant users may include any type of rights holder, such as individuals, enterprises, or groups.

[0026] For example, in the data processing process described later, when it is necessary to inform relevant users of accessibility service interfaces, operation information, and user interface nodes in accordance with relevant laws and regulations, a prompt message should be sent to the relevant users to clearly inform them. This allows the relevant users to choose whether to allow the use of the accessibility service interface, operation information, or user interface node based on the prompt message. This prompt can be in the form of a pop-up window, which can present the prompt message in text format. In addition, the pop-up window can also include a selection control for users to choose "agree" or "disagree".

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation method described in this article. Other methods that comply with relevant laws and regulations may also be applied to the implementation method described in this article.

[0028] Existing vision-based technical solutions have many inherent flaws in practical applications. These flaws not only affect the stability and reliability of the interaction, but may also bring potential risks in terms of performance and privacy, as detailed below:

[0029] 1. Accuracy Issues: The interface recognition and operation prediction performance of the Vision Large Model (VLM) heavily relies on the image quality of the screenshot (including sharpness and resolution), the lighting conditions of the acquisition scene, and the consistency of the graphical user interface (GUI) design style. In real-world scenarios, when the interface undergoes minor changes (such as style adjustments, color replacements, or positional shifts of function buttons, or modifications to text fonts and sizes), dynamic animation effects exist (such as loading animations or pop-up transitions), or non-standard controls (i.e., custom controls that do not follow general UI design specifications) appear, VLM is highly prone to errors in interface element recognition and positioning deviations. This can lead to the failure of subsequent simulated operation commands, affecting the normal progress of the entire interaction process and reducing the user experience.

[0030] 2. High Performance Consumption: To achieve real-time interface interaction simulation, this vision-based solution requires high-frequency execution of screen capture, image data transmission, and complex VLM model inference calculations. This series of operations puts enormous pressure on the core computing resources of the client device (including the Central Processing Unit (CPU) and Graphics Processing Unit (GPU),) memory, and network bandwidth. Especially on mobile devices such as smartphones and tablets, this high-load operation directly leads to problems such as overheating, lag, and response delays, while also significantly consuming battery power and shortening the battery life of mobile devices, severely limiting the promotion and application of this solution in mobile terminal scenarios.

[0031] 3. Privacy Leakage Risk: Because VLM requires complete interface context information for accurate identification and operation prediction, the system must continuously capture the screen content of the terminal device and upload the captured screen image data to the cloud or local model for analysis and processing. This process means that all content operated by the user on the terminal device, including some sensitive information, may be completely recorded, transmitted, and stored. Once a leak occurs during data transmission, or the cloud / local storage device is illegally accessed, this information will be leaked, failing to meet relevant data security and privacy protection requirements.

[0032] 4. Perception Limitations: VLM's core perception method is limited to "visual recognition," meaning it can only capture and analyze visual information rendered on the screen. It cannot acquire non-visual, structured information within the interface, nor can it identify obscured or undisplayed UI elements. For example, VLM cannot directly read and recognize content in scrolling views (such as long lists or documents) that is not displayed within the current screen area. Furthermore, it cannot understand the hierarchical relationships between UI elements, the interactive states of elements (such as whether buttons are disabled or clickable), or accessibility labels preset by application developers for UI elements. This results in an incomplete and shallow understanding of the interface, further affecting the accuracy of operation predictions.

[0033] The designers of this paper discovered that, in stark contrast to the inherent limitations of the aforementioned vision-based technical solutions, operating systems offer accessibility technology designed to ensure the normal use of terminal devices by people with disabilities. This accessibility technology, through a standardized Application Programming Interface (API), provides assistive applications (such as screen readers and voice control tools) with a programmatic channel to access and parse the application's UI structure, breaking the limitations of visual recognition. Specifically, during development, application developers can add complete content descriptions, clear role definitions (such as "confirm button" and "input box"), and real-time status information (such as "selected" and "not clickable") to each UI element (including function buttons, images, text boxes, menus, etc.). Assistive applications can then query the "Accessibility Node Tree" provided by the operating system to obtain a complete, structured interface "map." This "map" contains the precise location, functional description, role type, interaction state, and specific operations that each UI element can perform, thereby achieving accurate understanding and efficient interaction with the UI.

[0034] Based on the above findings, for at least some of the above-mentioned technical problems, this paper provides a method for data processing in at least one aspect, comprising: obtaining a task sequence corresponding to the acquired natural language input based on the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; and performing the following operations sequentially for each task according to the execution order of the at least one task: obtaining a user interface node tree corresponding to each task via an accessibility service interface of the operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the acquired user interface node tree; and operating the user interface corresponding to the user interface node based on the acquired user interface node and operation information.

[0035] In accordance with the above methods corresponding to at least one of the embodiments described herein, one or more embodiments of this invention also provide a corresponding device, computer-readable storage medium, and computer program product.

[0036] The following detailed description, with reference to the accompanying drawings, illustrates one or more scenarios and some examples thereof.

[0037] Figure 1 The illustration schematically depicts an application scenario of the data processing method and apparatus provided in at least one aspect of this disclosure.

[0038] like Figure 1 As shown, in this application scenario 100, a user 101, a terminal device 102, a communication network 103, and a server 104 may be included.

[0039] For example, user 101 may include users or other relevant parties using applications for data processing (such as intelligent agents, AI, assistants, etc.). Applications for data processing may include, but are not limited to, web applications (such as browser data processing applications), mobile terminal applications (Android applications, iOS applications), etc., which provide the client described in this disclosure; user 101 can initiate data processing requests on the client. This client can run on terminal device 102. Terminal device 102 can load and execute an executable program or instructions that run the client, thereby enabling user 101 to initiate data processing requests using the client; embodiments of this disclosure do not limit this.

[0040] For example, terminal device 102 may include any electronic device capable of running the aforementioned client. This arbitrary electronic device can provide users with a natural language input interface (such as text, voice, etc.) for operation and use. This arbitrary electronic device includes, but is not limited to, mobile terminals, mobile phones, tablets, portable computers, desktop computers, smart wearable devices, smart home appliances, or smart vehicle terminals.

[0041] For example, communication network 103 may include any suitable wired or wireless communication network. Wired communication may include, but is not limited to, twisted-pair communication, coaxial cable communication, and fiber optic communication. Wireless communication may include, but is not limited to, cellular mobile communication, Wi-Fi communication, satellite communication, wireless local area network, and wireless wide area network.

[0042] For example, server 104 runs a server program corresponding to the client and can communicate with terminal device 102 via wired or wireless communication networks. Server 104 may include, for example, a local area network server, a wide area network server, a cloud server, etc., and specifically may be an enterprise-level application server, a distributed server cluster, or a cloud server. Furthermore, in some cases, server 104 and terminal device 102 can be integrated; in this case, the aforementioned communication network 103 may not be necessary. For example, a locally deployed intelligent agent can acquire natural language input and perform corresponding user interface operations.

[0043] For example, the data processing method provided in at least one aspect of this disclosure can be implemented in any suitable manner, such as in software.

[0044] The terminal device 102 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), or other processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding storage units (such as shared cache, memory, etc.); for example, the CPU may adopt the x86 instruction set, the ARM instruction set, the RISC-V instruction set, etc.

[0045] The terminal device 102 can be equipped with an operating system (such as Android, iOS, etc.), application programming interfaces (APIs), etc., and can run the client by running executable programs or instructions.

[0046] For example, terminal device 102 may also include an output component, such as a display device, which may be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a quantum dot light-emitting diode (QLED) display, or a light-emitting diode display (e.g., a micro-LED display), etc. The embodiments disclosed herein are not limited to this. For example, the display component may display an interactive page, allowing a user to interact with information on the client on terminal device 102 through the interactive interface.

[0047] The following will combine Figure 2 The present disclosure provides a detailed description of at least one method for data processing.

[0048] Figure 2 A flowchart illustrating a method for data processing provided in at least one scenario of this document is shown schematically.

[0049] like Figure 2 As shown, the method 200 in this case may include steps S210 to S220.

[0050] As an example, the executing entity of method 200 may include an agent, which may run on a terminal or server (as described above). Figure 1The terminal device 102 or server 104 mentioned above). For example, an intelligent agent can refer to an intelligent functional entity that can achieve a closed-loop process of "perceiving the external environment - processing information - generating decisions - executing responses - receiving feedback - iterative optimization" through a preset or adaptive information interaction mechanism, decision-making logic, and action execution capability, in order to achieve specific task objectives or adapt to dynamic scene changes; this entity is not limited to hardware form, software form, or a combination of hardware and software, and its core components may include, but are not limited to, an information perception module (used to acquire external or internal information such as environmental data, user instructions, and system status, and the perception methods include, but are not limited to, sensors, data interfaces, signal receiving devices, text / voice / image recognition, etc.), information... The system includes an information processing and decision-making module (used to analyze, calculate, reason, or learn perceived information; decision logic can be constructed based on preset rules, algorithm models (such as machine learning, deep learning, reinforcement learning, logical reasoning algorithms, etc.) or a combination of both), an action execution module (used to transform decision results into actionable responses; execution methods include, but are not limited to, controlling hardware device operation, outputting data / text / voice / image information, adjusting system parameters, triggering process instructions, etc.), and an optional feedback iteration module (used to receive result information after action execution and optimize perception strategies, decision logic, or execution methods based on this information to improve task achievement accuracy or environmental adaptability).

[0051] As another example, the execution entity of method 200 may include a model obtained based on artificial intelligence (AI), such as a large language model (LLM) based on deep learning, which can run on a terminal or server (as described above). Figure 1 (The terminal device 102 or server 104 mentioned above). In order to perform the various operations of method 200, the AI ​​can be provided with appropriate prompts or the AI ​​can be pre-trained.

[0052] As another example, the execution entity of method 200 may include any suitable neural network that has been trained, which can run on a terminal or server (e.g., the combination mentioned above). Figure 1 The terminal device 102 or server 104 mentioned above. For example, a convolutional neural network (CNN), recurrent neural network (RNN), etc., trained through any suitable supervised or unsupervised learning.

[0053] Reference Figure 2 In step S210, a task sequence corresponding to the natural language input can be obtained based on the acquired natural language input.

[0054] As an example, a task sequence may include at least one task. At least one task may be executed sequentially.

[0055] As an example, natural language input can include text input (such as instructions manually entered by the user), voice input (such as instructions entered by the user via voice), multimodal input (such as images + text or images + voice), etc.

[0056] As an example, if the natural language input includes "Help me cancel the takeout I ordered yesterday on WM" (where WM refers to the platform for ordering takeout, such as an app), the task sequence could include:

[0057] Task 1: Open the "WM" application;

[0058] Task 2: Navigate to the "Orders" page;

[0059] Task 3: Find order items whose text contains the word "yesterday";

[0060] Task 4: Locate and click the "Request a Refund" button under this item.

[0061] According to at least one scenario described in this paper, obtaining a task sequence corresponding to the acquired natural language input may include: obtaining a task sequence using a large language model based on the acquired natural language input.

[0062] As an example, a Large Language Model (LLM) combined with cue word engineering can obtain task sequences based on natural language input. For example, the cue word could include: Please parse the user-provided natural language input into one or more structured task steps.

[0063] As can be seen, the methods presented in one or more scenarios in this paper can use large language models to parse natural language input to obtain task sequences. This can bring the following beneficial effects:

[0064] (1) Eliminate semantic ambiguity and accurately understand user intent: Natural language is often vague and colloquial. Large models can deeply understand the context and accurately break down, for example, “help me cancel the takeout I ordered from WM yesterday” into specific task nodes such as identifying WM application, order, yesterday, and refund, thus avoiding the comprehension bias of traditional rule matching.

[0065] (2) Reduce the cost of manual structuring: There is no need to pre-write complex parsing rules or label a large number of templates. The model can directly extract parameters (time, location, action) from free text and automatically generate standardized task flow, which greatly improves development efficiency.

[0066] (3) Supports complex multi-step reasoning: It can handle instructions with complex dependencies, such as "check the weather first, if it rains, change to indoor activities, otherwise book an outdoor venue". The model can generate a complete decision chain with conditional branches, rather than simple keyword matching.

[0067] (4) Enhance system flexibility and generalization: The model can still be robustly parsed when faced with new expressions (such as dialects, abbreviations, and colloquial descriptions), enabling the task scheduling system to have stronger adaptability to real-world scenarios and reducing reliance on hard-coded logic.

[0068] Continue to refer to Figure 2 In step S220, the following operations can be performed sequentially for each task, according to the execution order of at least one task:

[0069] Operation 1: Obtain the user interface node tree corresponding to each task through the accessibility service interface of the operating system;

[0070] Operation 2: Based on the acquired user interface node tree, obtain the user interface node corresponding to each task and the operation information related to the user interface node; and

[0071] Operation 3: Based on the acquired user interface nodes and operation information, perform operations on the user interface corresponding to the user interface nodes.

[0072] As an example, the task sequence could include: Task 1: Open the "WM" application; Task 2: Navigate to the "Orders" page; Task 3: Find order items whose text contains "yesterday"; Task 4: Find and click the "Request Refund" button under that item. In this case, operations 1, 2, and 3 are performed sequentially for each task, following the execution order of Task 1, Task 2, Task 3, and Task 4.

[0073] As an example, the accessibility service interface of an operating system may include the accessibility service interface of the operating system (such as Android, iOS, Windows, macOS, etc.).

[0074] As can be seen, the methods provided in one or more scenarios in this paper can obtain the user interface node tree through the accessibility service interface of the operating system, thereby enabling the execution of user instructions using the user interface node tree. This method has advantages such as high accuracy, low performance consumption, low risk of privacy leakage, and comprehensive input perception.

[0075] According to at least one of the scenarios described herein, regarding operation 1 above, obtaining the user interface node tree corresponding to each task via the accessibility service interface of the operating system may include: obtaining a first operating system user interface node tree corresponding to each task via the accessibility service interface; and preprocessing the obtained first operating system user interface node tree according to a preset user interface node tree format to obtain the user interface node tree corresponding to each task.

[0076] As an example, each task may include: Task 1: Open the "WM" application. In this case, the first operating system user interface node tree corresponding to Task 1 can be obtained via the operating system's accessibility service interface. This first operating system user interface node tree is a specific operating system user interface node tree directly from the operating system, and can be considered as the original operating system user interface node tree. Since the format of the native user interface node tree provided by each operating system may be different, to make the method provided in this paper widely adaptable, it is necessary to convert the original tree into a unified user interface node tree format. Therefore, it is necessary to preprocess the obtained first operating system user interface node tree according to the preset user interface node tree format to obtain the user interface node tree corresponding to each task; at this time, the obtained user interface node tree corresponding to each task is the converted unified user interface node tree format.

[0077] As an example, the preset user interface node tree format can be any suitable preset format, as long as it retains the attribute information of each user interface node. In this case, preprocessing methods can include one or more of the following: filtering irrelevant nodes, attribute pruning (to retain key fields), hierarchy flattening (to merge or remove deeply nested meaningless containers (such as FrameLayout used only for layout), highlighting the hierarchical relationship of functional units, and simplifying the tree structure), control type standardization (to map platform-specific control class names (such as android.widget.Button for Android, UIButton for iOS) to uniform functional labels (button, input, list) for cross-platform processing), spatial relationship encoding (to convert absolute coordinates (bounds) into relative position descriptions (such as "at the top of the screen", "to the right of the search box"), or generating region division labels (header, sidebar, main-content)), etc.

[0078] As an example, this attribute information may include one or more of the following: identification information, type information, displayed text content information, position and size information, executable operation information, and child node information. For example, identification information may include a unique identifier for the user interface element; type information may include the control type of the user interface (such as Button, TextView, ImageView); displayed text content information may include the displayed text content (such as "WM"); position and size information may include the position and size (boundary box) of the user interface element on the screen; executable operation information may include a list of executable operations (such as CLICK, SCROLL_FORWARD, SET_TEXT); and child node information may include a list of child nodes.

[0079] As can be seen, the method provided in one or more scenarios in this paper can transform the original first operating system user interface node tree into a unified user interface node tree. This can bring at least one of the following beneficial effects:

[0080] (1) Break down cross-platform barriers and achieve one-time development for multiple terminals: The UI frameworks of different operating systems (e.g., Android / iOS) are very different. The unified model abstracts native controls into standard types (Button / TextView / ImageView), so that the upper-level intent parsing engine does not need to adapt to the details of each platform, greatly reducing the deployment cost of multiple terminals.

[0081] (2) Provide a structured and semantic understanding interface for large models: Natural language instructions (such as "click the confirmation button") need to be mapped to specific screen elements. The unified model establishes a clear link of "user expression → control semantics → executable operation" through standardized fields (text / description / actions), so that the model can accurately locate the target element and select the correct interaction method.

[0082] (3) Supports complex spatial reasoning and multi-step operation planning: Position and size information provides accurate positional relationships, enabling the model to understand spatial descriptions such as "the back button in the upper left corner" and "the first result below the search box"; child node information can support the decomposition and execution of nested navigation tasks such as "entering the settings page → finding the privacy option → turning off the switch".

[0083] (4) Enhance system robustness and interpretability: When automated execution fails, the unified format of the user interface node tree can be used for backtracking and location (such as the inability to uniquely identify due to missing identification information). The executable operation information can explicitly declare the operable capabilities, avoid misoperation of non-interactive elements, and facilitate the generation of human-readable operation logs.

[0084] According to at least one scenario described herein, at least one task may include multiple tasks. Obtaining a first operating system user interface node tree corresponding to each task via an accessibility service interface may include: in response to each task not being the first of multiple tasks, obtaining a second operating system user interface node tree after the execution of the previous task for each task via the accessibility service interface, and comparing a third operating system user interface node tree corresponding to the previous task with the second operating system user interface node tree; in response to the third operating system user interface node tree being different from the second operating system user interface node tree, using the second operating system user interface node tree as the first operating system user interface node tree; and in response to the third operating system user interface node tree being the same as the second operating system user interface node tree, operating on the user interface corresponding to the user interface node corresponding to the previous task based on the user interface node corresponding to the previous task and operation information related to the user interface node corresponding to the previous task.

[0085] As an example, at least one task may include four tasks: Task 1: Open the "WM" application; Task 2: Navigate to the "Orders" page; Task 3: Find the order item whose text contains "yesterday"; Task 4: Find and click the "Request Refund" button under that item.

[0086] In this case, in response to each task being the first of the four tasks (i.e., task 1), the first operating system user interface node tree corresponding to task 1 is obtained via the accessibility service interface of the operating system; and the first operating system user interface node tree is preprocessed according to the preset user interface node tree format to obtain the user interface node tree corresponding to task 1, and then the aforementioned operations 2 and 3 are continued to be executed.

[0087] If each task is not the first of the four tasks, assuming it is the second (i.e., task 2: navigating to the "Orders" page), the system retrieves the second operating system user interface node tree after the execution of task 1 (before task 2) via the operating system's accessibility service interface. It then compares the third operating system user interface node tree corresponding to task 1 with the second operating system user interface node tree (i.e., comparing the operating system user interface node tree before task 1 execution with the operating system user interface node tree after task 1 execution). If the third operating system user interface node tree differs from the second operating system user interface node tree (indicating that task 1 was successfully executed and the "WM" application is now open), the second operating system user interface node tree is used as the first operating system user interface node tree corresponding to task 2 (i.e., the operating system user interface node tree corresponding to the page after successful execution of task 1 is used as the operating system user interface node tree for the upcoming task 2). In response to the fact that the third operating system user interface node tree is the same as the second operating system user interface node tree (this indicates that task 1 has not been successfully executed, and task 1 needs to be executed again until it is successfully executed or optionally rolled back or terminated), based on the user interface node corresponding to task 1 and the operation information related to the user interface node corresponding to task 1, the user interface corresponding to the user interface node corresponding to task 1 is operated (i.e., the aforementioned operation 3).

[0088] As can be seen, the method provided in one or more scenarios in this paper can obtain and analyze the updated user interface state after each task operation to verify whether the operation corresponding to the task has achieved the expected effect (i.e. whether it has been executed correctly), and decide to continue or retry based on the verification results, forming a complete "perception-decision-execution-feedback" closed loop. It has the advantages of reliability, fault tolerance, and support for the gradual advancement of long-term multi-step tasks.

[0089] According to at least one of the scenarios described herein, regarding step 2 above, obtaining the user interface node corresponding to each task and the operation information related to the user interface node based on the obtained user interface node tree may include: obtaining the user interface node corresponding to each task based on the object operated by each task and the attribute information of each user interface node in the user interface node tree; and obtaining the operation information related to the user interface node based on the obtained user interface node corresponding to each task and each task.

[0090] As an example, each task may include: Task 1: Open the "WM" application. In this case, the user interface node tree corresponding to Task 1 may include attribute information for each user interface node (corresponding to each application), such as the previously mentioned identification information, type information, and displayed text content information. The object operated on by Task 1 is the "WM" application, and the displayed text content information in the aforementioned attribute information includes "WM" information. In this case, based on the object operated on by Task 1 and the attribute information of each user interface node in the user interface node tree (e.g., through matching), the user interface node corresponding to Task 1 (e.g., the user interface node corresponding to the "WM" application) can be obtained. Finally, based on the obtained user interface node (e.g., the user interface node corresponding to the "WM" application) and Task 1 (e.g., opening the "WM" application), operation information related to the user interface node (e.g., opening the user interface node corresponding to the "WM" application) can be obtained.

[0091] As can be seen, the methods provided in one or more scenarios in this paper can map each specific task to a specific user interface node, thereby accurately performing relevant operations on that specific user interface node.

[0092] According to at least one of the scenarios described herein, regarding step 3 above, operating the user interface corresponding to the user interface node based on the acquired user interface node and operation information may include: determining the operating system user interface node corresponding to the acquired user interface node in the operating system user interface node tree; and operating the user interface corresponding to the operating system user interface node based on the determined operating system user interface node and operation information.

[0093] As an example, step 2 above has already identified which user interface node needs to be operated on. Since this user interface node is in the unified user interface node format as described above, when executing it in the corresponding operating system, it is necessary to convert the unified user interface node tree format back to the format corresponding to the operating system, that is, to convert the obtained user interface node back to the operating system user interface node corresponding to the operating system user interface node tree.

[0094] Specifically, the unified user interface node can be reversed back to the operating system user interface node by using the mapping relationship from the previous operating system user interface node to the unified user interface node, thereby determining the operating system user interface node corresponding to the obtained user interface node. Alternatively, the original operating system user interface node format obtained through the accessibility service interface can be compared with the obtained user interface node format to convert the obtained user interface node into the corresponding operating system user interface node. Finally, based on the converted operating system user interface node (such as the operating system's own user interface node corresponding to the "WM" application for Task 1) and operation information (such as opening the "WM" application), the user interface corresponding to the operating system user interface node is operated (such as opening the "WM" application on the operating system's desktop user interface).

[0095] As can be seen, the methods provided in one or more scenarios presented in this paper can translate user operation intentions back into user interface node objects of a specific operating system platform to perform corresponding operations. This can bring the following beneficial effects:

[0096] (1) Balancing abstract uniformity with native performance to achieve optimal execution efficiency: The unified format ensures that the upper layer understands the intent consistently across platforms, while the final implementation calls the system's native interface and directly utilizes the operating system's optimized event dispatch mechanism to avoid the delay and instability caused by the intermediate layer simulation.

[0097] (2) Ensure compliance and system-level stability: The native accessibility interface is a standard interface certified by the platform. Compared with external simulated clicks, it is less likely to trigger the application's anti-fraud mechanism or system security interception, reducing the risk of being judged as "malicious automation" and enjoying the same permission stability as the accessibility function.

[0098] (3) Precise control and state synchronization: The native interface can directly manipulate the logical behavior of user interface elements (such as SET_TEXT to trigger input method linkage, SCROLL_FORWARD to trigger inertial scrolling), rather than just simulating pixel-level clicks; after the operation, the accessibility service can listen in real time (such as AccessibilityEvent) to ensure that the state feedback is strictly synchronized with the actual rendering of the system.

[0099] (4) Reduce ecosystem adaptation costs and quickly cover mainstream platforms: Only the mapping layer of "unified format → interface of each platform" needs to be maintained (such as mapping the CLICK of the unified model to the ACTION_CLICK of Android and the tap() of iOS), and the same intent engine can be deployed to multiple terminals without the need to customize an access solution for each application or scenario.

[0100] To facilitate a better understanding of the above content, examples will be provided below.

[0101] As described above Figure 2 The paper describes one or more scenarios that enable a semantic bridging operation scheme based on accessibility and AI agents. The core idea is to use the AI ​​agent as the "brain," responsible for understanding the user's natural language intent; and to use the operating system's accessibility services as the "nervous system," responsible for providing standardized, structured user interface (UI) information for all applications. A "semantic bridging layer" connects the two, accurately translating abstract intents into specific operational instructions for accessibility nodes (UI elements).

[0102] Figure 3 The diagram illustrates the core module structure and data flow relationships of the system provided in at least one scenario of this paper.

[0103] Reference Figure 3 The entire system can include an external interface module and a semantic bridging operation module. The external interface module is used to receive natural language input from users (such as the user (natural language intent) shown in No. 310) and operations on the target UI interface (such as the target App (UI interface) shown in No. 320). The semantic bridging operation module is used to perform operations such as parsing user intent and updating state. It includes an Intent Parsing Engine 330, a Semantic Bridging Engine 340, an Action Executor 350, a Receipt & Strategy Engine 360, a Platform Adaptation Layer 370, and a Unified UI Tree Model 380.

[0104] Natural language input from the user is first fed into the intent parsing engine 330, which receives the user's natural language input (such as speech or text) and uses a large language model (LLM) to parse it into one or more structured task steps. For example, "Cancel my takeout order from WM yesterday" is parsed as: [Task step 1: Open the "WM" app, Task step 2: Navigate to the "Orders" page, Task step 3: Find the order item whose text contains "yesterday", Task step 4: Find and click the "Request Refund" button under that item].

[0105] The intent parsing engine 330 inputs the parsed structured task into the semantic bridging engine 340, which is responsible for mapping the structured task generated by the intent parsing engine to specific nodes and operations on the unified UI tree model.

[0106] The operation instructions generated by the semantic bridging engine 340 are input to the operation executor 350, which receives the operation instructions generated by the semantic bridging engine 340 and calls the platform adaptation layer to convert them back to the native accessibility operation API of the specific platform for execution.

[0107] The operation executor 350 performs operations on the target App (UI interface) shown in number 320 based on the operation to be performed (such as click, scroll, etc.) and the accessibility operation API converted back to the specific platform native.

[0108] The execution result of the operation executor 350 is input to the receipt and strategy engine 360. Its workload analyzes the new UI tree state after each operation execution, captures the operation receipt (e.g., success, failure (e.g., no UI change)), and determines the next strategy (continue, retry, or exception handling). The receipt and strategy engine 360 ​​sends the updated task status or triggers a retry to the semantic bridge engine 340, which decides whether to execute the next task, re-execute the current task, etc.

[0109] The platform adaptation layer 370 abstracts accessibility service interfaces for different operating systems (Android, iOS, Windows, macOS, etc.); it is responsible for calling the native APIs of each platform (such as Android's AccessibilityService and iOS's UIAccessibility) to obtain the original UI structure tree (as mentioned above). Figure 2 The aforementioned third operating system user interface node tree), and convert it into a unified format (as mentioned above regarding...). Figure 2 The user interface node tree corresponding to each task (as described above).

[0110] The Unified UI Tree Model 380 defines a cross-platform, standardized data structure to describe the user interface (UI) at any given time. Each node represents a UI element and contains the following attributes:

[0111] id: A unique identifier (if any).

[0112] class: The type of control (such as Button, TextView, ImageView).

[0113] text: The text content to be displayed.

[0114] description: Content description or accessibility tag.

[0115] rect: Position and size on the screen (boundary box).

[0116] actions: A list of executable operations (such as CLICK, SCROLL_FORWARD, SET_TEXT).

[0117] children: A list of child nodes.

[0118] As mentioned earlier, the unified UI tree model 380 can provide standardized node data for the semantic bridging engine 340 (as mentioned above regarding...). Figure 2 The data is converted into a unified user interface node tree format.

[0119] Figure 4 The illustration schematically depicts a complete end-to-end processing flow from user instruction to task completion, provided for at least one scenario described herein.

[0120] Reference Figure 4 In step 410: The user issues a command via voice or text (e.g., “Cancel yesterday’s takeout order in WM”).

[0121] The process proceeds to Intent Resolution 420: The Intent Resolution Engine receives the instruction and breaks it down into a structured task queue: [OpenApp('WM'), FindAndClick('Order'), FindOrder('Yesterday'), ClickButton('Request Refund')].

[0122] Step 430: The system triggers the platform adaptation layer to obtain the unified UI tree model for the current screen.

[0123] Step proceeds to task loop 440: The first task is retrieved from the task queue and handed over to the semantic bridge engine and operation executor for processing (i.e., Figure 4 Semantic bridging and execution (450).

[0124] The process proceeds to Receipt and Status Update 460: The receipt and strategy engine verify the result of the operation for the first task. If the task is successful, the process proceeds to the judgment step 470: Are all tasks completed? If the judgment is negative, the process proceeds to Enter Task Loop 440 to start executing the next task; if the judgment is positive, the entire process ends, and success feedback can be sent to the user as shown in step 480 (e.g., "Task completed"). If the task fails, the process proceeds to exception handling 490 (e.g., retry, rollback, prompt, etc., and the reason for failure can be provided as appropriate), and then the process proceeds to step 4100, where the process ends.

[0125] Figure 5 The diagram illustrates the sequence of interactions between key participants in a system for a specific example of “finding and clicking the ‘order’ button,” provided in at least one scenario of this paper.

[0126] See Figure 5 Based on the parsing of the user's natural language input, the AI ​​Agent (central control) executes step 510, which is to execute step to find and click the order (executeStep("FindAndClick",{"text":"order"})).

[0127] The semantic bridging engine sends a request to the operating system (OS) accessibility service to obtain the UI tree, specifically the getUlTree() function (shown in number 520). The OS accessibility service then returns the UI tree to the semantic bridging engine, specifically the returnuiTree function (shown in number 530).

[0128] In step 540, the semantic bridging engine performs a semantic bridging operation, which is to search for the user interface node corresponding to "order" (searchNode(text="order", isCjckable=true)).

[0129] Then, based on the found user interface nodes and operation information, a click action instruction is generated and sent to the AI ​​Agent, as shown in number 550: generateAction("CLICK","node_123"). The AI ​​Agent then sends a command to the OS accessibility service to execute the click action, as shown in number 560: performAction("node_123","CLICK"). The OS accessibility service sends a click event to the target app, as shown in number 570, triggering node_123. As shown in number 580, the target app processes the click logic and updates the UI. As shown in number 590, the target app sends a UI update notification to the OS accessibility service. As shown in number 5100, the OS accessibility service sends a notification event (notifyfvent(TYPE_WINDOW_CONTENT_CHANGED)) to the receipt and policy engine. Next, as shown in number 5110, the receipt and policy engine sends a request to the OS accessibility service to retrieve the UI tree (getUITree()). As shown in number 5120, the OS accessibility service returns the UI tree `newUiTree` to the receipt and policy engine. As shown in number 5130, the receipt and policy engine performs validation and feedback (`compare(oldUiTree, newUiTree)`). As shown in number 5140, if the task is executed successfully, the receipt and policy engine reports a success status to the AI ​​Agent using `reportStatus('SUCCESS')`. Finally, as shown in number 5150, the AI ​​Agent continues with any task operation (`proceedToNextStep()`).

[0130] The following is an example of a unified UI node model, presented in JSON format:

[0131] JSON

[0132] {

[0133] "id": "com.example.app:id / button_confirm",

[0134] "class": "android.widget.Button",

[0135] "text": "Confirm",

[0136] "description": "Click to confirm your selection",

[0137] "rect": [100, 200, 300, 250],

[0138] "actions": ["CLICK", "LONG_CLICK", "FOCUS"],

[0139] "children": []

[0140] }

[0141] In a semantic bridging engine, the Python representation of the pseudocode for the mapping from natural language to operations can be shown as follows:

[0142] def map_intent_to_action(task_step, ui_tree):

[0143] # task_step: e.g., "Find and click the 'Request a Refund' button"

[0144] #ui_tree: Unified UI Tree Model instance

[0145] # 1. Design a selector

[0146] selector = {

[0147] "text_contains": "Request a refund",

[0148] "class_is": "Button",

[0149] "is_clickable": True

[0150] }

[0151] # 2. Node Retrieval

[0152] candidate_nodes = ui_tree.find_nodes(selector)

[0153] # 3. Node matching and disambiguation (simple example: selecting the first node)

[0154] if not candidate_nodes:

[0155] Raise a NodeNotFoundError("No matching node found")

[0156] target_node = candidate_nodes[0]

[0157] # 4. Operation Generation

[0158] if "CLICK" in target_node.actions:

[0159] action_command = {

[0160] "node_id": target_node.id,

[0161] "action_type": "CLICK"

[0162] }

[0163] return action_command

[0164] else:

[0165] raise ActionNotSupportedError("Click operation is not supported for the node")

[0166] Based on the above combination Figures 3 to 5 The described examples further illustrate that the methods presented in this paper, in one or more scenarios, provide an automated operation method based on semantic bridging. This method can receive and parse natural language intents to generate structured task sequences; it can obtain the structured UI node tree of the target application through the accessibility interface provided by the operating system; it can map each step in the task sequence to one or more target nodes and corresponding operations in the UI node tree (semantic bridging); and it can execute operations through the accessibility interface and provide feedback and verification based on the changes in the UI node tree after the operation.

[0167] In addition, based on the above combination Figures 3 to 5 The described examples further illustrate that the methods presented in one or more scenarios in this paper provide a cross-platform unified UI tree model. This standardized data structure can be used to mask the differences in accessibility interfaces across different operating systems (such as Android, iOS, and Windows), providing a consistent and programmable interface view for upper-layer AI agents. This unified UI tree model includes the unique identifier, type, text, location, executable actions, and hierarchical relationships of nodes, as described above.

[0168] Furthermore, the combination described above Figures 3 to 5The described examples further illustrate that the methods presented in one or more scenarios in this paper also provide a dynamic mapping mechanism from intent to node operations. At the heart of this mechanism lies a semantic bridging engine, which uses a selector language containing attributes such as text, type, location, and hierarchical relationships to perform efficient and accurate node retrieval in the UI tree. This mechanism should also include a disambiguation algorithm when multiple candidate nodes are retrieved, as well as adaptive matching capabilities after node attributes (such as ID) change.

[0169] Furthermore, based on the above combination Figures 3 to 5 The described examples further demonstrate that the methods provided in one or more scenarios in this paper can also provide a closed-loop execution and feedback verification system. This system includes an operation executor and a receipt and strategy engine, which can proactively acquire and analyze the updated UI state (new UI tree) after each operation, verify whether the operation has achieved the expected effect, and decide to continue, retry, roll back or terminate the task based on the verification results, forming a complete "perception-decision-execution-feedback" closed loop.

[0170] Based on the above combination Figures 3 to 5 The described examples further demonstrate that the methods provided in one or more scenarios of this paper can achieve at least one of the following technical effects:

[0171] (1) Extremely high accuracy and stability: Because the method provided in this paper directly manipulates the structured UI nodes, it is completely unaffected by changes in the visual interface. No matter how the application updates its theme, color, layout, or resolution, as long as the structure and labels of its accessibility node tree remain logically consistent, the operation will be stable and successful. The accuracy is orders of magnitude higher than that of visual solutions.

[0172] (2) Extremely low performance consumption: The methods provided in one or more scenarios in this paper can avoid high-frequency screenshots and high-computational VLM image analysis. The parsing and transmission of UI tree data is extremely small (usually at the KB level), and the calculation is mainly completed on the edge side, resulting in extremely low consumption of device CPU, memory and power, enabling lightweight, background-resident automated services.

[0173] (3) Fundamental privacy protection: The entire operation of the method provided in one or more scenarios in this paper does not involve the collection of screen pixel data. The AI ​​Agent perceives abstract, non-visual UI structure information. When the user inputs sensitive information, the AI ​​Agent only knows that it is an "input box" and cannot obtain the specific content, thus fundamentally eliminating the risk of visual information leakage.

[0174] (4) Strong scalability and adaptability: Through a unified UI tree model, the methods provided in one or more scenarios in this paper can be easily extended to any platform that supports accessibility services (including desktop operating systems), realizing complex automated processes across applications and platforms. For dynamically changing applications, the strategy engine and adaptive matching algorithm can effectively deal with issues such as element ID changes, demonstrating strong robustness.

[0175] (5) Promoting the construction of an accessible ecosystem and empowering special groups: The methods provided in one or more scenarios in this paper can greatly enhance the application value of accessible technology. It can not only provide visually impaired users with unprecedented intelligent assistance and automatically complete complex operations, but also incentivize developers to better adapt to accessibility standards, because good adaptation means that its application can seamlessly connect to the next generation of AI operating ecosystem, forming a virtuous cycle.

[0176] (6) Opening up new scenarios for enterprise-level automation: For old enterprise software that lacks APIs, the methods provided in one or more scenarios in this paper can provide a low-cost "operational layer API" solution that can be used for automated testing, data migration and process automation (Robotic Process Automation, RPA), which greatly improves the efficiency and possibility of enterprise digital transformation.

[0177] In addition to the methods described above, this document also provides corresponding equipment, media, and products for one or more scenarios. These will be explained below with reference to the accompanying drawings.

[0178] Figure 6 A schematic block diagram of a device 600 for data processing, provided in at least one embodiment of this document, is illustrated. The foregoing description of the method 200 for data processing also applies to this device 600, unless otherwise explicitly stated. The device may include a terminal device or a server. Terminal devices may include, but are not limited to, mobile terminals, mobile phones, tablets, portable computers, desktop computers, smart wearable devices, smart home appliances, or smart vehicle terminals. Servers may include local area network servers, wide area network servers, cloud servers, and may specifically employ enterprise-level application servers, distributed server clusters, or cloud servers.

[0179] like Figure 6 As shown, the device 600 in this case may include a first operation unit 610 and a second operation unit 620.

[0180] According to at least one of the embodiments described herein, the first operation unit 610 may be configured to obtain a task sequence corresponding to the acquired natural language input based on the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially.

[0181] According to at least one of the embodiments described herein, the second operation unit 620 may be configured to perform the following operations sequentially for each task according to the execution order of the at least one task: obtaining a user interface node tree corresponding to each task via the accessibility service interface of the operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the obtained user interface node tree; and operating the user interface corresponding to the user interface node based on the obtained user interface node and operation information.

[0182] It should be noted that, for clarity and brevity, not all the constituent modules or units of the device 600 are shown in at least one scenario herein. To achieve the necessary functions of the information device, those skilled in the art may provide or configure other constituent units (not shown) according to specific needs, and this at least one scenario herein does not impose any limitations on this.

[0183] Figure 7 A schematic block diagram of an electronic device 700 in at least one of the embodiments described herein is shown.

[0184] Reference Figure 7 The electronic device 700 may include a processing device 701 and a storage device 708. Depending on the requirements, the electronic device 700 may also include a read-only memory (ROM) 702, a random access memory (RAM) 703, a bus 704, an (I / O) interface 705, an input device 706, an output device 707, and a communication device 709.

[0185] Figure 7 The electronic device shown is merely an example and should not impose any limitations on the functionality and scope of use in the context of this article.

[0186] For example, the processing device 701 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), or other forms of processing unit with data processing capabilities and / or instruction execution capabilities. It may be a general-purpose processor or a dedicated processor and may control other components in the electronic device to perform desired functions. The electronic device 700 may include one or more processing devices 701.

[0187] For example, storage device 708 may include one or more computer program instructions, which may be stored in various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. The processing device may execute the one or more computer program instructions to implement the functions described herein (implemented by the processing device) and / or other desired functions, such as methods for data processing. Various application programs and various data may also be stored in the computer-readable storage medium, such as the aforementioned user interface node tree, operating system user interface node tree, etc., which are not limited herein.

[0188] In addition, the processing device 701 can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage device 708 into the random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing device 701, ROM 702, and RAM 703 are interconnected via a bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0189] The following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0190] Specifically, according to the context of this document, the process described in the above-referenced flowchart can be implemented as a computer software program. For example, the context includes a computer program product comprising one or more computer instructions that can be used to perform the method shown in the flowchart. In such a scenario, the one or more computer instructions can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the one or more computer instructions are executed by processing device 701, the functions defined in the method of this context are performed.

[0191] At least one aspect of this document also provides a non-transitory computer-readable storage medium. This storage medium stores non-transitory computer-readable instructions. When executed by a processor, the computer-readable instructions can perform the method described with reference to the above figures according to this aspect. The storage medium in this aspect can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0192] It should be noted that the computer-readable medium described above in this document can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this document, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0193] The aforementioned computer-readable medium carries one or more computer program products, which include one or more computer instructions that, when executed by a processor, perform any of the aforementioned methods for data processing, etc.

[0194] Computer program code for performing the operations described herein may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0195] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to the various scenarios described herein. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of block numbers in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0196] The units or modules described in this document can be implemented in software or hardware. The names of the units or modules do not necessarily constitute a limitation on the unit or module itself.

[0197] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0198] In the context of this document, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0199] According to one or more scenarios described herein, Example 1 provides a method for data processing, comprising: obtaining a task sequence corresponding to the acquired natural language input based on the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; and performing the following operations sequentially for each task according to the execution order of the at least one task: obtaining a user interface node tree corresponding to each task via an accessibility service interface of the operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the acquired user interface node tree; and operating the user interface corresponding to the user interface node based on the acquired user interface node and operation information.

[0200] According to one or more scenarios in this paper, Example 2 provides the method of obtaining the user interface node tree corresponding to each task via the accessibility service interface of the operating system in Example 1, including: obtaining the first operating system user interface node tree corresponding to each task via the accessibility service interface; and preprocessing the obtained first operating system user interface node tree according to a preset user interface node tree format to obtain the user interface node tree corresponding to each task.

[0201] According to one or more scenarios in this paper, Example 3 provides that each user interface node in the user interface node tree corresponding to each task in Example 2 includes one or more of the following attribute information: identification information, type information, displayed text content information, position and size information, executable operation information, and child node information.

[0202] According to one or more scenarios in this document, Example 4 provides at least one task in Example 2 that includes multiple tasks. The step of obtaining a first operating system user interface node tree corresponding to each task via the accessibility service interface includes: in response to each task not being the first task among multiple tasks, obtaining a second operating system user interface node tree after the execution of the previous task for each task via the accessibility service interface, and comparing a third operating system user interface node tree corresponding to the previous task with the second operating system user interface node tree; in response to the third operating system user interface node tree being different from the second operating system user interface node tree, using the second operating system user interface node tree as the first operating system user interface node tree; and in response to the third operating system user interface node tree being the same as the second operating system user interface node tree, operating on the user interface corresponding to the user interface node corresponding to the previous task based on the user interface node corresponding to the previous task and operation information related to the user interface node corresponding to the previous task.

[0203] According to one or more scenarios in this document, Example 5 provides the method described in Example 1, which involves obtaining the user interface node corresponding to each task and operation information related to the user interface node based on the obtained user interface node tree. This includes: obtaining the user interface node corresponding to each task based on the object operated by each task and the attribute information of each user interface node in the user interface node tree; and obtaining operation information related to the user interface node based on the obtained user interface node corresponding to each task and each task.

[0204] According to one or more scenarios in this paper, Example 6 provides the method of obtaining a task sequence corresponding to the natural language input based on the acquired natural language input, as in Example 1, including: obtaining the task sequence based on the acquired natural language input using a large language model.

[0205] According to one or more scenarios in this paper, Example 7 provides the operation of the user interface corresponding to the user interface node based on the acquired user interface node and operation information in Example 1, including: determining the operating system user interface node corresponding to the acquired user interface node in the operating system user interface node tree; and operating the user interface corresponding to the operating system user interface node based on the determined operating system user interface node and operation information.

[0206] According to one or more embodiments of this document, Example 8 provides an apparatus for data processing, comprising: a first operation unit configured to obtain a task sequence corresponding to the acquired natural language input based on the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; and a second operation unit configured to perform the following operations sequentially for each task according to the execution order of the at least one task: obtaining a user interface node tree corresponding to each task via an accessibility service interface of an operating system; obtaining a user interface node corresponding to each task and operation information related to the user interface node based on the acquired user interface node tree; and operating the user interface corresponding to the user interface node based on the acquired user interface node and operation information.

[0207] According to one or more of the embodiments herein, Example 9 provides an electronic device including: at least one processor, and a memory storing one or more computer program instructions; wherein the one or more computer program instructions are executed by the at least one processor to perform the method provided in at least one of the embodiments herein.

[0208] According to one or more of the embodiments described herein, Example 10 provides a non-transitory computer-readable storage medium that non-transitory stores computer-readable instructions, wherein the methods provided in at least one of the embodiments described herein are implemented when the computer-readable instructions are executed by a processor.

[0209] According to one or more of the scenarios described herein, Example 11 provides a computer program product comprising one or more computer instructions, wherein the one or more computer instructions are executed by a processor to perform the method provided in at least one scenario described herein.

[0210] The above description is merely a preferred embodiment and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure herein is not limited to technical solutions formed by specific combinations of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed herein that have similar functions.

[0211] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be executed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain contexts. Similarly, while some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of this paper. Certain features described in the context of a single case can also be implemented in combination in a single case. Conversely, various features described in the context of a single case can also be implemented individually or in any suitable sub-combination in multiple cases.

[0212] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for data processing, comprising: Based on the acquired natural language input, a task sequence corresponding to the natural language input is obtained, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; and Perform the following operations sequentially for each of the at least one task, according to the execution order of the task: The user interface node tree corresponding to each task is obtained through the accessibility service interface of the operating system; Based on the obtained user interface node tree, obtain the user interface node corresponding to each task and the operation information related to the user interface node; as well as Based on the acquired user interface node and the operation information, the user interface corresponding to the user interface node is operated.

2. The method of claim 1, wherein, The step of obtaining the user interface node tree corresponding to each task via the operating system's accessibility service interface includes: The first operating system user interface node tree corresponding to each task is obtained via the accessibility service interface; and According to the preset user interface node tree format, the obtained first operating system user interface node tree is preprocessed to obtain the user interface node tree corresponding to each task.

3. The method of claim 2, wherein, Each user interface node in the user interface node tree corresponding to each task includes one or more of the following attribute information: Identification information Type information, The displayed text content information, Location and size information, Executable operation information, and Child node information.

4. The method according to claim 2, wherein, The at least one task includes multiple tasks. The step of obtaining the first operating system user interface node tree corresponding to each task via the accessibility service interface includes: In response that each task is not the first task among the plurality of tasks, the second operating system user interface node tree after the execution of the previous task of each task is obtained through the accessibility service interface, and the third operating system user interface node tree corresponding to the previous task is compared with the second operating system user interface node tree. In response to the fact that the third operating system user interface node tree is different from the second operating system user interface node tree, the second operating system user interface node tree is used as the first operating system user interface node tree; and In response to the fact that the third operating system user interface node tree is the same as the second operating system user interface node tree, operations are performed on the user interface corresponding to the user interface node corresponding to the previous task based on the user interface node corresponding to the previous task and the operation information related to the user interface node corresponding to the previous task.

5. The method according to claim 1, wherein, The step of obtaining the user interface node corresponding to each task and the operation information related to the user interface node based on the obtained user interface node tree includes: Based on the object operated by each task and the attribute information of each user interface node in the user interface node tree, obtain the user interface node corresponding to each task; and Based on the user interface node corresponding to each task and each task, obtain the operation information related to the user interface node.

6. The method according to claim 1, wherein, The step of obtaining a task sequence corresponding to the acquired natural language input includes: Based on the acquired natural language input, the task sequence is obtained using a large language model.

7. The method according to claim 1, wherein, The operation on the user interface corresponding to the user interface node based on the acquired user interface node and the operation information includes: Determine the operating system user interface node corresponding to the acquired user interface node in the operating system user interface node tree; and Based on the determined operating system user interface node and the operation information, the user interface corresponding to the operating system user interface node is operated.

8. An apparatus for data processing, comprising: The first operation unit is configured to obtain a task sequence corresponding to the acquired natural language input based on the acquired natural language input, wherein the task sequence includes at least one task, and the at least one task is executed sequentially; as well as The second operation unit is configured to perform the following operations sequentially for each task according to the execution order of the at least one task: obtain the user interface node tree corresponding to each task via the accessibility service interface of the operating system; and obtain the user interface node corresponding to each task and operation information related to the user interface node based on the obtained user interface node tree. And based on the acquired user interface node and the operation information, perform operations on the user interface corresponding to the user interface node.

9. An electronic device, comprising: At least one processor, and Memory, which stores one or more computer program instructions; The one or more computer program instructions are executed by the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable recording medium for non-transitory storage of computer-readable instructions, wherein, The method described in any one of claims 1 to 7 is implemented when the computer-readable instructions are executed by a processor.

Images