Barrier-free interaction method based on large model reinforcement and related equipment

Abstract

The invention discloses a barrier-free interaction method and related equipment based on large model enhancement, and the method comprises the steps: carrying out the traversing of an initial state node of an initial interface, and constructing a state transition diagram; labeling function information of the transfer nodes to generate a node function table; obtaining a current state node and a task purpose of the target task, querying the memorized task table, and when the current state node and the task purpose do not exist in the memorized task table, obtaining a simplified state node; according to the simplified state node, the node function table and the task purpose, generating a cue word; outputting an interaction element ID, an interaction event type and a task completion identifier of the target task through the large model; and updating the current state node, judging whether the task completion identifier is completed or not, and if the task completion identifier is not completed, returning to the step of acquiring the current state node until the identifier is completed. The method can improve the naturalness of barrier-free interaction and the accuracy of an interaction strategy, and can be widely applied to the technical field of artificial intelligence.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an accessible interaction method and related equipment based on large model reinforcement. Background Technology

[0002] The current Android accessibility service simulates user interaction with the device by monitoring changes in the UI state and providing corresponding event feedback. While this mechanism can meet specific user needs, it still has some shortcomings in addressing in-depth user usage. In the existing technical solutions, the Android accessibility service still has three main problems in terms of device accessibility: (1) Inability to understand user task intent: Existing accessibility services only focus on special scenarios with single interaction, such as screen reading and gesture control. These scenarios usually have single requirements and rely heavily on user input. On the other hand, this also makes it difficult for the accessibility service itself to understand the user's task intent, thus making it difficult to handle more complex interactive tasks, such as information retrieval and email reply. (2) Inability to perceive the functionality of the application UI: The interaction mechanism of the Android accessibility service simulates user interaction with the device by obtaining the UI state of the application and sending interaction events. Although this mechanism can parse the UI element state on the current interface, it cannot adjust the interaction strategy according to the user's task intent because it cannot understand the functionality of the application UI. All strategies need to be pre-configured by the developer during the development phase, which makes it impossible to fully generalize its application scope. (3) Risk of abuse of user privacy: Since accessibility services need to obtain the UI state of the application and perform interactive operations, this process may lead to user privacy leaks to some extent. For example, sometimes the application interface contains personal information, which may be found in user-entered forms. Summary of the Invention

[0003] In view of this, the main objective of the embodiments of the present invention is to provide an accessibility interaction method and related equipment based on large model enhancement, in order to solve at least one of the problems of the prior art. The present invention can improve the naturalness of accessibility interaction and the accuracy of interaction strategies.

[0004] To achieve the above objectives, one aspect of this invention provides an accessibility interaction method based on large model reinforcement, the method comprising: Based on the depth-first search algorithm, the initial state nodes of the starting interface are traversed to construct a state transition graph; The functional information of the transition nodes in the state transition diagram is labeled to generate a node function table; In response to the target task instruction, the task objective of the target task is obtained, and the current status node is obtained; Based on the current state node and the task objective, the memoized task table is queried. If the current state node and the task objective do not exist in the memoized task table, the current state node is optimized to obtain a simplified state node. Based on the simplified status node, the node function table, and the task objective, generate prompt words; Based on the prompt words, the large model outputs the target interaction element ID, the target interaction event type, and the task completion identifier of the target task. Based on the target interaction element ID and the target interaction event type, the current state node is updated, and it is determined whether the task completion flag is complete. If the task completion flag is not complete, the process returns to the step of obtaining the current state node until the task completion flag is complete, at which point the target task process is terminated.

[0005] In some embodiments, after obtaining the task objective of the target task and the current state node in response to the target task instruction, the method further includes the following steps: The memoized task table is queried based on the current state node and the task objective; When the current state node and the task objective exist in the memoized task table, the target interaction element ID and the target interaction event type are obtained through the memoized task table. Based on the target interaction element ID and the target interaction event type, the current state node is updated, and it is determined whether the task completion flag is complete. If the task completion flag is not complete, the process returns to the step of obtaining the current state node until the task completion flag is complete, at which point the target task process is terminated.

[0006] In some embodiments, optimizing the current state node to obtain a simplified state node includes the following steps: Identify the current state node; When personal information is detected in the current status node, the personal information is replaced with a general identifier; When a non-interactive state node is detected in the current state node, the non-interactive state node is removed. When redundant state nodes are detected in the current state nodes, the redundant state nodes are merged or removed.

[0007] In some embodiments, generating prompts based on the simplified status node, the node function table, and the task objective includes the following steps: Traverse the simplified state nodes to obtain the initial interaction element ID of the transition node in the simplified state nodes; Based on the initial interaction element ID, the node function table is retrieved to obtain the initial interaction event type and the function information that match the initial interaction element ID; The initial interaction event type and the functional information are concatenated to obtain the hint attribute value; The hint attribute value is inserted into the transfer node to obtain the hint enhancement status node; The prompt enhancement status node and the task objective are encapsulated to obtain the prompt word.

[0008] In some embodiments, the step of outputting the target interaction element ID, the target interaction event type, and the task completion identifier of the target task based on the prompt word and through a large model includes the following steps: Preset output style; The output style is inserted into the prompt word, and the target interactive element ID, the target interactive event type, and the task completion identifier are output through the large model.

[0009] In some embodiments, the accessibility interaction method based on large model reinforcement further includes the following steps: The task objective, the current state node, the target interaction element ID, and the target interaction event type are recorded as a single entry in the memoized task table.

[0010] In some embodiments, updating the current state node based on the target interaction element ID and the target interaction event type includes the following steps: Based on the target interaction element ID, find the corresponding target interaction node in the current state node; Based on the target interaction node and the target interaction event type, an accessibility interaction event is constructed; In response to the accessibility interaction event, the current state node is updated.

[0011] To achieve the above objectives, another aspect of the present invention proposes an accessibility interaction device based on large model enhancement, the device comprising: The first functional perception module is used to traverse the initial state nodes of the starting interface based on the depth-first algorithm and construct a state transition graph. The second function perception module is used to annotate the function information of the transition nodes in the state transition diagram and generate a node function table. The first prompt word generation module is used to respond to the target task instruction, obtain the task purpose of the target task, and obtain the current status node; The second prompt word generation module is used to query the memorized task table based on the current state node and the task objective. When the current state node and the task objective do not exist in the memorized task table, the current state node is optimized to obtain a simplified state node. The third prompt word generation module is used to generate prompt words based on the simplified status node, the node function table, and the task objective; The first decision output module is used to output the target interaction element ID, the target interaction event type, and the task completion identifier of the target task based on the prompt words and through the large model. The second decision output module is used to update the current state node according to the target interaction element ID and the target interaction event type, and to determine whether the task completion identifier is completed. If the task completion identifier is not completed, the module returns to the step of obtaining the current state node until the task completion identifier is completed, and then terminates the target task process.

[0012] To achieve the above objectives, another aspect of the present invention provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of the present invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0015] The embodiments of the present invention include at least the following beneficial effects: The present invention provides an accessibility interaction method and related devices based on large model reinforcement. This scheme is based on a depth-first search algorithm, traversing the initial state nodes of the starting interface to construct a state transition graph, providing a complete structured knowledge foundation for subsequent intelligent decision-making; labeling the transition nodes in the state transition graph with functional information to generate a node function table, helping the large model understand the operation intent; responding to the target task instruction, obtaining the task objective of the target task and obtaining the current state node; querying the memorized task table based on the current state node and task objective to avoid redundant calculations and improve efficiency; when the current state node and task objective do not exist in the memorized task table, optimizing the current state node to obtain a simplified state node, improving the processing speed and accuracy of the large model; based on the simplified state... The system generates prompts based on nodes, node function tables, and task objectives, laying the foundation for generating accurate and reliable operation instructions for the large model. Based on these prompts, the large model outputs the target interaction element ID, target interaction event type, and task completion flag, transforming complex interface understanding and task decomposition into clear and executable action instructions, thus achieving a leap from perception to decision-making. The system updates the current state node based on the target interaction element ID and target interaction event type, and checks if the task completion flag is complete. If the flag is incomplete, it returns to the step of retrieving the current state node until the flag is completed, terminating the target task process. This cyclical feedback mechanism ensures complete task execution, achieving end-to-end automated closed-loop processing of the task and improving the naturalness of accessible interaction and the accuracy of interaction strategies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the accessibility interaction method based on large model enhancement provided in an embodiment of the present invention; Figure 2 This is a technical framework diagram of barrier-free interaction based on large model enhancement provided in the embodiments of the present invention; Figure 3 This is a partial state transition diagram using a recording app as an example, provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the initial state node code of the starting interface provided in an embodiment of the present invention; Figure 5This is a simplified state node code diagram of the starting interface provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the initial state node code of the editing interface provided in this embodiment of the invention; Figure 7 This is a simplified state node code diagram of the editing interface provided in this embodiment of the invention; Figure 8 This is a schematic diagram of the enhanced status node code provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0019] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification, claims, and the foregoing drawings may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0020] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0022] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the normal operation of the embodiments of the present invention acquired. Before providing a detailed description of the embodiments of the present invention, some of the nouns and terms involved in the embodiments of the present invention will be explained first. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0023] A prompt, or cue word, is typically the input text or instruction provided by the user to the language model to guide the model in generating a corresponding response or output. It plays a crucial role in communicating and guiding the behavior of large language models.

[0024] UI (User Interface) refers to the interface through which users interact with devices or applications. UI design includes not only the visual design of the interface but also the interaction between the user and the interface.

[0025] Accessibility is a key feature of Android designed to help users with mobility impairments use their devices more easily. Accessibility provides developers with a range of methods to create apps that allow users with special needs to interact with them in different ways. For example, screen readers can help visually impaired users navigate the phone's interface and content, haptic feedback can provide interactive prompts, and gesture controls can be used to perform specific interactive tasks.

[0026] Accessibility services in the ColorOS system are mainly used in screen readers, voice input, and gesture control. They assist users with interaction difficulties by acquiring UI information and simulating event input, and typically perform well in scenarios requiring repetitive operations. However, existing accessibility services lack functional understanding of common UI elements and semantic understanding of user commands, thus failing to enable in-depth interactive operations (such as automatic editing / sending emails) for users with interaction difficulties.

[0027] In view of this, this invention provides an accessibility interaction method and related device based on a large model enhancement. The scheme first constructs a UI state transition graph using a depth-first search algorithm, and then injects functional information into the state transition nodes. During service runtime, based on the user's natural language, the scheme simplifies the UI state nodes through pruning algorithms and anonymization, and uses the simplified UI state nodes, user task intent, and application functional information to generate a prompt for the large model. Finally, based on the formatted output of the large model, the target UI interaction nodes and target interaction event types required to execute the task are determined, and the events are triggered to drive accessibility interaction, achieving accessibility interaction based on the user's natural language without requiring manual triggering.

[0028] Figure 1 This is an optional flowchart of an accessibility interaction method based on large model reinforcement provided in an embodiment of the present invention. Figure 1 The method may include, but is not limited to, steps S100 to S700: Step S100: Based on the depth-first search algorithm, traverse the initial state nodes of the starting interface to construct a state transition graph. Step S200: Label the functional information of the transition nodes in the state transition diagram and generate a node function table; Step S300: In response to the target task instruction, obtain the task objective of the target task and obtain the current status node; Step S400: Based on the current state node and task objective, query the memoized task table. If the current state node and task objective do not exist in the memoized task table, optimize the current state node to obtain a simplified state node. Step S500: Generate prompt words based on the simplified status node, node function table, and task objective; Step S600: Based on the prompt words, output the target interactive element ID, target interactive event type, and target task completion identifier through the large model; Step S700: Update the current status node according to the target interaction element ID and the target interaction event type, and determine whether the task completion flag is completed. If the task completion flag is not completed, return to the step of obtaining the current status node until the task completion flag is completed, and terminate the target task process.

[0029] In step S100 of some embodiments, starting from the main interface state of the application, a depth-first search algorithm is used to traverse all interactive nodes in the initial state nodes to construct a UI state transition graph and record the state transition paths between different states. This state transition path includes the initial state node before the transition, the initial state node after the transition, and the triggered transition node.

[0030] In step S200 of some embodiments, based on the state transition paths recorded in the state transition diagram, functional information is labeled for each transition node to generate a node function table. This node function table includes the initial interactive element ID, the type of interactive initial event, and the corresponding functional information for each transition node. Optionally, the data structure of the entries in the node function table is as follows: ; In the formula, Indicates the number of transfer nodes; Indicates the first The initial interaction element ID of each transfer node; Indicates the first The number of interactive events per transfer node; Indicates the first One initial interactive event type; Indicates triggering the first The functional information of each interaction event. This functional information represents the function of the initial UI state node after the interaction event is triggered within the application, and the function of the transition node is the function of the initial UI state node after the transition.

[0031] For example, taking the "Contacts" tab on the main interface (chat list) of a social media app as an example, the initial state node K1 before the transfer is the main interface of the social media app (displaying the chat list), the triggered transfer node is N1, which is the "Contacts" tab in the bottom navigation bar (interactive element ID is id=contacts_tab), and the initial state node K2 after the transfer is the Contacts interface (displaying the friend list). The function information F1 labeled for the transfer node N1 is the function of the initial state node K2, namely, viewing and managing contacts. Optionally, in the node function table, it can be recorded as {id=contacts_tab, event=CLICK,function="Enter Contacts Interface"}, where this function describes the function of the destination K2.

[0032] By labeling transition nodes with functional information, the large model can understand the role of each interactive node in the state nodes, enabling the large model to provide operational semantics when making decisions, rather than just text labels on the interface, thus making more accurate and reliable judgments.

[0033] In step S300 of some embodiments, the user sends a target task instruction by inputting natural language, and in response to the target task instruction, obtains the current state node of the current interface. and mission objectives Optionally, natural language input can be achieved by speaking task instructions aloud through a voice assistant (such as a built-in voice assistant or a dedicated voice interaction interface), with the system converting the speech into text instructions using Automatic Speech Recognition (ASR) technology; it can also be by directly typing task instructions into a text input box provided by the system; or it can be by using preset commonly used task templates (such as "scan") provided by the system, where the user clicks on these templates or speaks the corresponding shortcut phrases, which the system then converts into corresponding standard task instructions, but is not limited to these methods. After receiving the original natural language instructions, the system first performs basic semantic parsing, extracting core task verbs and key parameters to form a structured task objective. While parsing the instructions, the system uses the activated accessibility service to obtain the complete UI layout information of the app interface currently in the foreground on the phone screen in real time. This information is represented in the form of a UI node tree, which is the current state node. This node contains attributes of all UI elements on the current interface, such as ID, text, coordinates, and interactivity. Binding the parsed task objective to the captured current state node serves as the initial context for this round of task execution, ensuring the system's accurate perception of the task execution environment and providing correct initial conditions for subsequent decisions and operations.

[0034] In step S400 of some embodiments, the memoized task table is queried according to the current state node and the task objective. If the current state node and the task objective do not exist in the table, the current state node and the task objective are recorded. The initial UI state node is optimized based on anonymization and pruning algorithms, which reduces the leakage of user information during the UI state transition graph process, reduces the amount of information in the state node, and reduces the burden on the large model.

[0035] In some embodiments, the step of optimizing the current state node to obtain a simplified state node may include, but is not limited to, steps S410 to S440: Step S410: Identify the current state node; Step S420: When it is recognized that the current status node contains personal information, replace the personal information with a general identifier; Step S430: When it is detected that there are non-interactive state nodes in the current state nodes, the non-interactive state nodes are removed. Step S440: When redundant state nodes are detected in the current state nodes, the redundant state nodes are merged or removed.

[0036] In step S410 of some embodiments, since the initial state node may contain certain user personal information, causing privacy leakage, and redundant information on the state node will affect the response efficiency of the large model. By identifying the current state node, it is possible to identify whether there are personal information, non-interactive state nodes, redundant state nodes, etc.

[0037] In step S420 of some embodiments, when it is recognized that the current state node contains personal information entered by the user (e.g., the content of the account, address, or email input box), the UI state node is simplified based on anonymization, and the personal information is replaced with a general identifier. For example, it is checked whether the UI elements in the current state node contain personal information such as username, password, or contact information. If so, the personal information is replaced with an information type, such as "xxx@outlook.com" being replaced with "email address".

[0038] In step S430 of some embodiments, when a non-interactive node is detected in the current state node, the non-interactive node is removed from the current state node. For example, if the clickable, scrollable, and editable attributes of a UI element in the state node are all false, it means that the element cannot trigger interaction, and the element is removed from the current state node.

[0039] In step S440 of some embodiments, when redundant nodes such as layout nodes with empty content attributes are identified in the current state node, these layout nodes are removed. If there are nested layout nodes containing multiple layout nodes with empty content attributes, these layout nodes are merged. For example, if it is identified that there are nested layout nodes in the current state node containing multiple layout nodes with empty content attribute values ​​(text and content-desc attributes are both empty), they are merged into a single element.

[0040] In some embodiments, in response to a target task instruction, the task objective of the target task is obtained, and after obtaining the current state node, the memoized task table is queried based on the current state node and the task objective. If the current state node and the task objective exist in the memoized task table, the target interaction element ID and the target interaction event type to be interacted with are obtained from the memoized task table. Then, the interaction event is constructed and triggered through the accessibility service, and the current round of operation ends. That is, the current state node is updated based on the target interaction element ID and the target interaction event type, and it is determined whether the task completion flag is completed. If the task completion flag is not completed, the step of obtaining the current state node (i.e., obtaining the current state node for the next round of operation) is returned until the task completion flag is completed, and the target task process is terminated.

[0041] In step S500 of some embodiments, a prompt can be generated based on simplified state nodes, labeled functional information, and user intent. For example, functional information is obtained from the node function table according to the initial interaction element ID of the transition node and inserted into the functional attribute of the transition node. The task intent may include the user's task objective and the previous round of operation records.

[0042] In some embodiments, step S500 may include, but is not limited to, steps S510 to S550: Step S510: Traverse the simplified state nodes to obtain the initial interaction element ID of the transition node in the simplified state nodes; Step S520: Based on the initial interaction element ID, search the node function table to obtain the initial interaction event type and function information that match the initial interaction element ID; Step S530: Concatenate the initial interaction event type and function information to obtain the hint attribute value; Step S540: Insert the hint attribute value into the transfer node to obtain the hint enhancement status node; Step S550: Encapsulate the enhanced status node and task objective to obtain the prompt word.

[0043] In step S510 of some embodiments, the simplified state nodes are traversed to obtain the initial interactive element ID of the interactive UI state nodes (i.e., transition nodes) in the simplified state nodes.

[0044] In step S520 of some embodiments, the functional information and initial interaction event type of the transfer node are indexed by the initial interaction element ID of the transfer node.

[0045] In steps S530 to S540 of some embodiments, in order to enable the large model to perceive the function of the transfer node, a new field `hint` is inserted into the element attributes of the transfer node. The `hint` attribute value is composed of the initial interaction event type concatenated with functional information. After the corresponding `hint` attribute value is inserted, the transfer node can obtain a hint-enhanced state node.

[0046] In step S550 of some embodiments, the enhanced status node, task objective, and previous operation record are encapsulated to obtain the prompt word. For example, if the task objective is "delete two recording records in sequence" and the current operation is "delete the second recording record", then the previous operation record is "delete the first recording record".

[0047] In some embodiments, step S600 may include, but is not limited to, steps S610 to S620: Step S610: Preset the output style; Step S620: Insert prompt words into the output style and output the target interactive element ID, target interactive event type, and task completion identifier through the large model.

[0048] In step S610 of some embodiments, a fixed output style is preset. By "reducing" the open output of the large model to a closed, deterministic instruction set, the powerful understanding and reasoning capabilities of the large model are fully utilized, while effectively avoiding the risks brought about by its uncertainty, thereby ensuring the efficient, stable and reliable operation of the entire automation system.

[0049] In step S620 of some embodiments, to ensure the output of the large model meets expectations, an output pattern is pre-inserted into the prompt. The output pattern specifies the ID of the interactive element to be interacted with, the type of the interactive event, and the task completion flag. It is then sent to the large model's remote service and awaits the large model's output. Upon receiving the request, the large model responds according to the output pattern, outputting the target interactive element ID, the target interactive event type ACTION_TYPE, and the task completion flag FINISH_TAG in a fixed pattern.

[0050] In step S700 of some embodiments, based on the target interactive element ID and target interactive event type output by the large model, the accessibility service constructs a trigger event to drive accessibility interaction and update the application state. After triggering the accessibility interaction event, the initial UI state node of the current interface is updated. While waiting for the update to finish, it checks whether the task completion flag is complete. If it is, the user task is marked as complete; otherwise, it returns to the step of obtaining the initial state node and performs the next round of operations. The initial state node obtained at this time is the state node of the next interface, which is the updated initial state node. For example, after triggering the accessibility interaction event, the current state node of the current interface is updated. While waiting for the update to finish, it checks whether the task completion flag FINISH_TAG is true. If it is true, the user task is marked as complete; otherwise, it obtains the updated initial state node and performs the next round of operations until the task completion flag FINISH_TAG is true, the target task is completed, and the target task process is terminated. The target task is fully executed through a loop feedback mechanism, forming an end-to-end automated closed-loop processing of the target task.

[0051] In some embodiments, updating the current state node based on the target interaction element ID and the target interaction event type in step S700 may include, but is not limited to, steps S710 to S730: Step S710: Based on the target interaction element ID, find the corresponding target interaction node in the current state node; Step S720: Construct accessible interaction events based on the target interaction node and the target interaction event type; Step S730: In response to the accessibility interaction event, update the current state node.

[0052] In step S710 of some embodiments, the target interactive node, whose type is AccessibilityNodeInfo, is indexed from the current state node based on the target interactive element ID that needs to be interacted with output by the large model.

[0053] In step S720 of some embodiments, an accessible interaction event is constructed based on the target interaction node and the target interaction event type. The target interaction event type of the accessible interaction event includes click event type, long press event type, and swipe event type, but is not limited to these.

[0054] In step S730 of some embodiments, an interaction event of the accessibility node is triggered according to the target interaction event type. For example, if the event type is (CLICK_ACTION) (click event), the performAction(AccessibilityNodeInfo.ACTION_CLICK) method is called to trigger the click event; if the event type is (LONG_CLICK_ACTION) (long press event), the performAction(AccessibilityNodeInfo.ACTION_LONG_CLICK) method is called to trigger the long press event; if the event type is (SCROLL_ACTION) (swipe event), the performAction(AccessibilityNodeInfo.ACTION_SCROLL_FORWARD) method is called to trigger the swipe event. After the interaction event is triggered, the UI state node of the current interface will be updated.

[0055] In some embodiments, the task objective Current state node Target interactive element ID and target interactive event type Recorded as a single entry in the memoized task table. For example, the data structure in the memoized task table is as follows: ; in, Indicate the user's task objective; This represents the current state node of the current interface. Indicates the ID of the target interactive element that needs to be interacted with; This indicates the type of the target interaction event. Querying the memoized task table can reduce repetitive processes and lower the system load for large model services.

[0056] The accessibility interaction solution based on large model reinforcement proposed in this invention can help users process tasks through text or voice without manual operation, making it more convenient for users to operate application devices. For example, as... Figure 2 As shown, the solution of this embodiment consists of three modules: a UI function perception module for associating App scene functions and UI representations; a prompt generation module for enhancing the inference performance of the large model; and a large model decision output module for driving accessibility services. This solution extracts specific domain knowledge of the App as key features through the UI function perception module, integrates the App's UI function features into a designed prompt to enhance the inference performance of the large model in the specific domain, and finally, the decision output module strengthens accessibility services based on the feedback from the large model, thereby achieving automated UI interaction for users.

[0057] In some embodiments, taking a recording app as an example, the user's task objective is to delete the first recording, such as... Figure 2 As shown, the barrier-free interaction process based on large model enhancement in this invention is as follows: Step 1 (First Preprocessing Stage): The function awareness module starts from the main interface of the recording app and performs a depth-first traversal of the initial UI state nodes. During the traversal, it records the visited initial UI states and state transition relationships to construct a state transition graph, as shown below. Figure 3 As shown (only key parts are shown due to space limitations).

[0058] Step 2 (Second Preprocessing Stage): The function awareness module labels the transition nodes with functions based on the UI state transition diagram constructed in Step 1, and generates a node function table. The structure of each data entry in the table includes the element ID of the transition node, the type of interaction event, and the function after the transition. For example, the node function table for the state transition diagram displayed by the recording app is shown in Table 1.

[0059] Table 1

[0060] After the preprocessing stage, the actual operation stage begins. In this stage, the prompt generation module will obtain the current UI status nodes in real time. After removing sensitive information and redundant status nodes through anonymization and pruning algorithms, the functional information in the node function table will be inserted into the transfer node. Finally, the main prompt content will be generated after attaching the user's task intent and operation records.

[0061] Step 3: The prompt generation module first calls the getRootInActiveWindow() method to obtain the current UI state node and checks whether the content attributes of the UI elements in the node contain user personal information such as username, password, and contact information. Secondly, this module iterates through the current UI state nodes, removing non-interactive UI state nodes and eliminating or merging redundant UI state nodes.

[0062] For example, in the main interface of the recording app in this embodiment, the existence of user email address information (test_username@outlook.com) is detected, and the status does not contain sensitive information input elements (such as username, password input boxes, etc.). Therefore, the UI status node is anonymized, that is, the complete contact information test_username@outlook.com is replaced with the uniform identifier "email address".

[0063] Next, the prompt module iterates through the status nodes of the main interface, such as... Figure 4 As shown. This module checks for non-interactive UI state nodes (where the clickable, scrollable, and editable attributes are all false). The IDs of non-interactive elements in the main interface are `record_duration` and `record_date_modified`. Simultaneously, this module searches for nested layout nodes with no content (empty `text` and `content-desc` attributes), such as the layout node with the ID `mFootView`. The simplified state node is shown below. Figure 5 As shown.

[0064] Step 4: The prompt generation module inserts functional information and user intent into the transition nodes based on the simplified main interface status nodes to improve the large model's ability to perceive the context of the task.

[0065] For example, the module first indexes the element ID of the transfer node from the node function table and then retrieves the transfer node from the simplified status node. In this example, a transfer node `record_title` is found, so a `hint` attribute is added, and the function information is used as the value of this `hint` attribute. After insertion, the attributes of the transfer node are updated as follows: <node text="Standard Recording 4" resource-id="com.coloros.soundrecorder:id / record_title" class="android.widget.TextView" content-desc="" clickable="true" scrollable="false" bounds="[108,682][339,747]" hint="Click: Play audio; Long press: Change audio file" / > Next, the prompt generation module encapsulates the above state nodes with attached function information, task objectives, and the previous operation record into a prompt. In this example, the task objective is to "delete the first recording record". Since this is the first run of this round, the previous operation record is "launch the recording App".

[0066] Step 5: The decision output module inserts the specified output style based on the encapsulated prompt content, so that the output result of the large model only contains the element ID, interaction event type, and task completion identifier that need to be interacted with. The final complete prompt is: Given the current UI state of the App, task objectives, and operation records, your task is to determine the elements that need to be interacted with.

[0067] Simplified state nodes are omitted here.

[0068] Task: Delete the first recording record.

[0069] Previous operation record: Launch the recording App.

[0070] Determine the elements that should be interacted with currently. Your output only includes the element id, event type action, and task end symbol finish_tag and is output in the following style: id=<element id>,action= <click long-click scroll>,finish_tag= <true false> Step 6: The decision output module obtains the output results of the large model, including the IDs of the elements that need to be interacted with, the interaction events, and the task completion indicator. This module first determines whether the task has been completed based on the task completion indicator. If not, it finds the corresponding target interaction node from the status nodes and constructs the corresponding accessibility event.

[0071] For example, the large model returns the following result: id=com.coloros.soundrecorder:id / record_title,action=long-click,finish_tag=false; The target interaction node is indexed from the state node based on the element ID. Then, based on the interaction event type of long-click, the performAction(AccessibilityNodeInfo.ACTION_LONG_CLICK) method is called to trigger the long-press event and update the current App state.

[0072] The task completion tag finish_tag is false, meaning the App has not completed the task of deleting the recording. Therefore, it is necessary to proceed to the second round of operations and repeat steps 3-6.

[0073] In step 3 of the second round of this embodiment, the prompt generation module again calls the getRootInActiveWindow() method to obtain the current UI state node and checks whether the node contains user personal information. Since the user's email address information (test_username@outlook.com) still exists in the interface, the same anonymization scheme as in step 3 of the first round is still used. Next, the module traverses the state nodes of the editing interface, such as... Figure 6 As shown. In this state node, the ID of the non-interactive element is rl_content, therefore the simplified state node is as follows. Figure 7 As shown.

[0074] In step 4 of the second round of this embodiment, the prompt generation module inserts functional information and user intent into the transition nodes of the simplified editing interface state nodes. The current state node contains transition nodes item_rename, set_as, and item_delete. A hint attribute is added to the corresponding transition nodes, and the corresponding functional information is added. The state node after addition is as follows: Figure 8 As shown.

[0075] Next, the module inserts the status node, task objective, and previous operation into the prompt and passes it to the decision output module. In particular, since the previous round triggered a long press event to change the audio file, the "previous operation record" for the current step is "changed audio file".

[0076] In step 5 of the second round of this embodiment, the decision output module inserts an output style based on the input prompt, so that the output of the large model only contains the element IDs that need interaction, the type of interaction event, and the task completion identifier. The output style specified in the second round is the same as that in the first round.

[0077] In step 6 of the second round of this embodiment, the decision output module obtains the output result of the large model. For example, the decision output of the large model in this round is: id=com.coloros.soundrecorder:id / item_delete,action=click,finish_tag=true; The system indexes the state node based on the output element ID and calls the performAction(AccessibilityNodeInfo.ACTION_CLICK) method to trigger a click event.

[0078] Since the task finish tag is true, it means that the task of deleting the recording has been completed and the process has ended.

[0079] This invention also provides an accessibility interaction device based on large model reinforcement, which can implement the above-mentioned accessibility interaction method based on large model reinforcement. The device includes: The first functional perception module is used to traverse the initial state nodes of the starting interface based on the depth-first algorithm and construct a state transition graph. The second functional perception module is used to label the functional information of the transition nodes in the state transition diagram and generate a node function table. The first prompt word generation module is used to respond to the target task instruction, obtain the task purpose of the target task, and obtain the current status node; The second prompt word generation module is used to query the memorized task table based on the current state node and task purpose. When the current state node and task purpose do not exist in the memorized task table, the current state node is optimized to obtain a simplified state node. The third prompt word generation module is used to generate prompt words based on simplified status nodes, node function tables, and task objectives; The first decision output module is used to output the interactive element ID, interactive event type, and task completion identifier of the target task based on the prompt words and through the large model. The second decision output module is used to update the current status node based on the interaction element ID and the interaction event type, and to determine whether the task completion flag is complete. If the task completion flag is not complete, the module returns to the step of obtaining the current status node until the task completion flag is complete, at which point the target task process is terminated.

[0080] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0081] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including a tablet computer, an in-vehicle computer, or similar device.

[0082] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] refer to Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0084] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0085] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0086] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0087] In summary, the barrier-free interaction method and related device based on large model reinforcement according to embodiments of the present invention have the following advantages: 1. This invention implements an intelligent and natural interaction method. By leveraging the powerful natural language understanding capabilities of large models, users can directly interact with accessibility services through semantic commands. This eliminates the reliance on preset scenarios and manual triggering in traditional solutions, not only understanding more complex user task intentions but also significantly improving the naturalness and flexibility of the interaction.

[0088] 2. This invention endows service applications with functional awareness capabilities. By constructing a UI state transition graph using a depth-first search algorithm and incorporating application functional information, the large model can "understand" the functionality and logical relationships of the application interface. This enables accessibility services to intelligently plan interaction paths and dynamically adjust strategies based on user intent, thereby accurately completing multi-step automated tasks.

[0089] 3. This invention effectively protects user privacy and security by employing node pruning and anonymization techniques. During the construction and processing of the UI state transition graph, irrelevant UI state nodes are simplified and sensitive personal information is removed. This reduces the risk of user privacy leakage from the source and also alleviates the processing burden of large models.

[0090] 4. In this embodiment of the invention, the output results of the large model are recorded in a memoized task table. In actual execution, the relevant records in the task table will be queried first. By querying the memoized task, repetitive processes can be reduced, and the system burden of the large model service can also be reduced.

[0091] 5. This invention is built upon Android's native accessibility services, eliminating the need for complex customization or specific testing environment configurations. This low dependency makes it more widely applicable, easy to deploy and use on different devices and application scenarios, and expands the application potential of accessibility services in complex scenarios.

[0092] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0093] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0094] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0096] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0097] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0098] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0100] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.< / true> < / click>

Claims

1. A barrier-free interaction method based on large model reinforcement, characterized in that, Includes the following steps: Based on the depth-first search algorithm, the initial state nodes of the starting interface are traversed to construct a state transition graph; The functional information of the transition nodes in the state transition diagram is labeled to generate a node function table; In response to the target task instruction, the task objective of the target task is obtained, and the current status node is obtained; Based on the current state node and the task objective, the memoized task table is queried. If the current state node and the task objective do not exist in the memoized task table, the current state node is optimized to obtain a simplified state node. Based on the simplified status node, the node function table, and the task objective, generate prompt words; Based on the prompt words, the large model outputs the target interaction element ID, the target interaction event type, and the task completion identifier of the target task. The current state node is updated based on the target interaction element ID and the target interaction event type. It is then determined whether the task completion flag is complete. If the task completion flag is not complete, the process returns to the step of obtaining the current state node until the task completion flag is complete, at which point the target task process is terminated.

2. The method according to claim 1, characterized in that, After responding to the target task instruction, obtaining the task objective of the target task, and obtaining the current state node, the method further includes the following steps: The memoized task table is queried based on the current state node and the task objective; When the current state node and the task objective exist in the memoized task table, the target interaction element ID and the target interaction event type are obtained through the memoized task table. The current state node is updated based on the target interaction element ID and the target interaction event type. It is then determined whether the task completion flag is complete. If the task completion flag is not complete, the process returns to the step of obtaining the current state node until the task completion flag is complete, at which point the target task process is terminated.

3. The method according to claim 1, characterized in that, The optimization of the current state node to obtain a simplified state node includes the following steps: Identify the current state node; When personal information is detected in the current status node, the personal information is replaced with a general identifier; When a non-interactive state node is detected in the current state node, the non-interactive state node is removed. When redundant state nodes are detected in the current state nodes, the redundant state nodes are merged or removed.

4. The method according to claim 1, characterized in that, The step of generating prompts based on the simplified status nodes, the node function table, and the task objective includes the following steps: Traverse the simplified state nodes to obtain the initial interaction element ID of the transition node in the simplified state nodes; Based on the initial interaction element ID, the node function table is retrieved to obtain the initial interaction event type and the function information that match the initial interaction element ID; The initial interaction event type and the functional information are concatenated to obtain the hint attribute value; The hint attribute value is inserted into the transfer node to obtain the hint enhancement status node; The prompt enhancement status node and the task objective are encapsulated to obtain the prompt word.

5. The method according to claim 1, characterized in that, The step of outputting the target interaction element ID, the target interaction event type, and the task completion identifier of the target task based on the prompt word and through a large model includes the following steps: Preset output style; The output style is inserted into the prompt word, and the target interactive element ID, the target interactive event type, and the task completion identifier are output through the large model.

6. The method according to claim 1, characterized in that, The method further includes the following steps: The task objective, the current state node, the target interaction element ID, and the target interaction event type are recorded as a single entry in the memoized task table.

7. The method according to claim 1, characterized in that, The step of updating the current state node based on the target interaction element ID and the target interaction event type includes the following steps: Based on the target interaction element ID, find the corresponding target interaction node in the current state node; Based on the target interaction node and the target interaction event type, an accessibility interaction event is constructed; In response to the accessibility interaction event, the current state node is updated.

8. A barrier-free interactive device based on large model reinforcement, characterized in that, include: The first functional perception module is used to traverse the initial state nodes of the starting interface based on the depth-first algorithm and construct a state transition graph. The second function perception module is used to label the function information of the transition nodes in the state transition diagram and generate a node function table. The first prompt word generation module is used to respond to the target task instruction, obtain the task purpose of the target task, and obtain the current status node; The second prompt word generation module is used to query the memorized task table based on the current state node and the task objective. When the current state node and the task objective do not exist in the memorized task table, the current state node is optimized to obtain a simplified state node. The third prompt word generation module is used to generate prompt words based on the simplified status node, the node function table, and the task objective; The first decision output module is used to output the target interaction element ID, the target interaction event type, and the task completion identifier of the target task based on the prompt words and through the large model. The second decision output module is used to update the current state node according to the target interaction element ID and the target interaction event type, and to determine whether the task completion identifier is completed. If the task completion identifier is not completed, the module returns to the step of obtaining the current state node until the task completion identifier is completed, and then terminates the target task process.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images