Intelligent interaction method and device based on Android barrier-free service

Abstract

The invention provides an intelligent interaction method and device based on an Android barrier-free service, and the method comprises the steps: creating a barrier-free service when an Android application is started, and appointing an event type monitored by the barrier-free service; when the service is successfully started and the interface loading is completed, comprehensively scanning the interface, and constructing a node tree according to the hierarchical relationship based on the control information of all controls on the interface; continuously monitoring various events captured by the service, pausing the current interactive operation when an interface changes, and updating the node tree based on a result after interface re-scanning; traversing the updated node tree on the basis of a breadth-first search algorithm in combination with control features corresponding to controls needing to be operated by the intelligent interaction task to obtain a traversing result; and if the target control matched with the control feature exists, simulating the operation of the intelligent interaction task through an API provided by the service, and executing subsequent interaction according to the operation completion condition of the intelligent interaction task. According to the invention, the reliability and practicability of intelligent interaction are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an intelligent interaction method and device based on Android accessibility services. Background Art

[0002] In the current field of intelligent interaction, a common approach is to leverage the Android system's accessibility interface to achieve a certain degree of automated interaction by monitoring specific system events and operations. For example, when a specific app is opened, a series of preset actions are automatically executed, such as automatically entering an account and password or clicking a specific button.

[0003] However, this existing approach suffers from a significant drawback: a lack of adaptability to dynamic interface changes. In real-world scenarios, Android app interfaces can change due to a variety of factors, such as version updates, personalized settings, and network status. Existing intelligent interaction methods are often programmed based on fixed interface layouts and control positions. When the interface changes, the rules for locating target controls become invalid, making it impossible to accurately simulate touch and gesture operations. This, in turn, prevents the automation task from completing properly, significantly reducing the reliability and practicality of intelligent interaction. Summary of the Invention

[0004] The present invention provides an intelligent interaction method and device based on Android accessibility services, so as to improve the reliability and practicality of intelligent interaction.

[0005] In a first aspect, the present invention provides an intelligent interaction method based on Android accessibility services, comprising:

[0006] When the Android application starts, inherit the Accessibility Service class to create a custom accessibility service, and specify the event types that the accessibility service listens for by configuring the XML file;

[0007] When the accessibility service is successfully started and receives a signal indicating that the interface loading is complete, the current screen interface is fully scanned, control information of all controls on the current screen interface is obtained based on an API provided by the Android system, and a node tree is constructed according to a hierarchical relationship based on the control information;

[0008] Continuously monitor various events captured by the accessibility service, and when a change in the current screen interface is detected, suspend the current interactive operation and update the node tree based on the results of rescanning the interface;

[0009] Based on the breadth-first search algorithm and the control features corresponding to the controls required to be operated in the pre-set intelligent interaction tasks, the updated node tree is traversed to obtain the traversal results;

[0010] If the traversal result shows that there is a target control that matches the control feature, the operation of the intelligent interaction task is simulated through the API provided by the accessibility service, and subsequent interactions are performed according to the completion status of the operation of the intelligent interaction task.

[0011] In a second aspect, the present invention further provides an intelligent interaction device based on Android accessibility services, which is applied to the intelligent interaction method based on Android accessibility services as described in the first aspect; the intelligent interaction device based on Android accessibility services comprises:

[0012] The monitoring module is used to inherit the Accessibility Service class to create a custom accessibility service when the Android application is started, and specify the event types that the accessibility service monitors by configuring an XML file;

[0013] A construction module is configured to, after the accessibility service is successfully started and a signal indicating that the interface loading is complete is received, perform a comprehensive scan of the current screen interface, obtain control information of all controls on the current screen interface based on an API provided by the Android system, and construct a node tree according to a hierarchical relationship based on the control information;

[0014] An update module, configured to continuously monitor various events captured by the accessibility service, and when detecting a change in the current screen interface, suspend the current interactive operation and update the node tree based on the result of rescanning the interface;

[0015] The traversal module is used to traverse the updated node tree based on the breadth-first search algorithm and the control features corresponding to the controls that need to be operated in the pre-set intelligent interaction tasks to obtain the traversal results;

[0016] An intelligent interaction module is used to simulate the operation of the intelligent interaction task through the API provided by the accessibility service if the traversal result shows that there is a target control that matches the control characteristics, and perform subsequent interactions according to the completion status of the operation of the intelligent interaction task.

[0017] In a third aspect, the present invention also provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing any of the above-mentioned intelligent interaction methods based on Android accessibility services.

[0018] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium, in which a computer software program is stored. When the computer software program is executed by a processor, it implements any of the above-mentioned intelligent interaction methods based on Android accessibility services.

[0019] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned intelligent interaction methods based on Android accessibility services.

[0020] The intelligent interaction method based on Android accessibility service provided by the embodiment of the present invention can promptly detect changes in the interface through real-time event listening and interface update monitoring, suspend the current operation, and rebuild the node tree. In the dynamic positioning and verification of the target control, a breadth-first search algorithm is used in combination with predefined feature tags. Even if the interface layout changes, the target control can be accurately found in the updated node tree. Therefore, the problem that the existing method cannot work normally when the interface changes dynamically is effectively solved, and the reliability and practicality of intelligent interaction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 1 is a flow chart of an intelligent interaction method based on Android accessibility services provided by an embodiment of the present invention;

[0022] Figure 2 2 is a schematic diagram of the structure of an intelligent interactive device based on Android accessibility services provided by an embodiment of the present invention;

[0023] Figure 3 An embodiment diagram of an electronic device provided by an embodiment of the present invention;

[0024] Figure 4 An embodiment diagram of a computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0026] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0027] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0028] Optional, see Figure 1 , Figure 1 : is a flow chart of the intelligent interaction method based on Android accessibility services provided by the present invention. In the embodiment of the present invention, the execution subject of the intelligent interaction method based on Android accessibility services is an intelligent interaction device. Therefore, the intelligent interaction method based on Android accessibility services includes:

[0029] Step 10: When the Android application is started, a custom accessibility service is created by inheriting the Accessibility Service class, and the event type monitored by the accessibility service is specified by configuring an XML file.

[0030] Optionally, when an Android application is launched, the intelligent interactive device creates a class that inherits the Accessibility Service class provided by the system to implement a customized accessibility service. By inheriting this class, it overrides the methods within it to handle accessibility-related events. At the same time, the intelligent interactive device explicitly specifies in the configuration XML file the event types that the accessibility service needs to monitor, such as changes in the focus of interface elements, completion of interface loading, and button clicks. Therefore, when these specified events occur in the system, the accessibility service can perceive and handle them accordingly.

[0031] In one embodiment, the Android application corresponding to the smart interactive device is a video playback application. When the application is started, a class named "VideoApp Accessibility Service" is created, which inherits from the AccessibilityService class. A configuration file, such as "accessibility_config.xml", is created in the res / xml directory of the project. In this file, by setting corresponding tags and attributes, the accessibility service is specified to monitor the interface loading completion event and the video playback control click event, etc. For example, the configuration file may contain something like " <accessibility-serviceandroid:canRetrieveWindowContent="true"android:description="@string / accessibility_service_description"android:accessibilityEventTypes="typeWindowContentChanged|typeViewClicked" / > ", where "typeWindowContentChanged" is used to monitor changes in interface content (including completion of interface loading), and "typeViewClicked" is used to monitor control click events.

[0032] Step 20: After the accessibility service is successfully started and receives a signal indicating that the interface loading is complete, a comprehensive scan is performed on the current screen interface. Control information of all controls on the current screen interface is obtained based on the API provided by the Android system, and a node tree is constructed according to the hierarchical relationship based on the control information.

[0033] Furthermore, when the accessibility service is successfully started and a signal indicating that the interface loading is complete is detected, the accessibility service will call the relevant APIs provided by the Android system (such as the methods provided by the AccessibilityNodeInfo class) to scan the current screen interface. During the scanning process, detailed information about all controls on the screen will be obtained, such as the type of control (button, text box, image, etc.), position, size, text content, etc. Furthermore, the intelligent interactive device, based on this control information and in accordance with the hierarchical relationship of the controls in the interface, takes each control as a node and constructs a node tree, as described in steps 1 to 4. The root node of the node tree usually represents the entire screen interface, and the child nodes are the various controls, reflecting the inclusion and hierarchical relationship between the controls.

[0034] Continuing with the above embodiment, in a video playback application, when the "VideoApp Accessibility Service" accessibility service is started and receives a signal that the loading of the main interface of the video playback is complete, the interface is scanned by calling the relevant methods of the AccessibilityNodeInfo class. Information about controls such as the play button, pause button, progress bar, and video title text box is obtained. For example, the information about the play button may include that its type is a button, its position is in the lower middle left of the screen, and its text content is "Play". Then, the entire screen interface is taken as the root node, and each control is taken as a child node, and a node tree is constructed according to their hierarchical relationship in the interface. For example, the progress bar may be a child node, and the play button and pause button are the same-level child nodes of the progress bar, and are together at the next level of the root node of the screen interface.

[0035] Step 30: Continue to monitor various events captured by the accessibility service. When a change in the current screen interface is detected, suspend the current interactive operation and update the node tree based on the result of rescanning the interface.

[0036] Furthermore, the accessibility service will continuously monitor various events occurring in the system, including but not limited to the addition, deletion, and position change of interface elements. When a change in the current screen interface is detected, the current interactive operation needs to be paused to ensure the accuracy of subsequent operations.

[0037] Furthermore, the accessibility service calls the API provided by the Android system again to rescan the interface, obtain the changed control information, and updates the previously constructed node tree according to the new control information, so that the node tree can accurately reflect the status of the current screen interface, as shown in steps 301 to 303.

[0038] Continuing with the above embodiment, in the video playback application, when the user clicks the full-screen play button, the screen interface will change. The accessibility service "VideoApp Accessibility Service" continuously monitors events, and after detecting this interface change, it pauses any interactive operations that may be in progress, such as dragging the progress bar. Next, the method of the AccessibilityNodeInfo class is called again to scan the full-screen interface and obtain new control information, such as a new exit full-screen button. Then, based on this new information, the previously constructed node tree is updated, and the new control is added as a node at the appropriate hierarchical position, or the relevant properties of the existing node (such as position, size, etc.) are modified.

[0039] Step 40 , based on a breadth-first search algorithm and in combination with control features corresponding to controls required to be operated in a pre-set intelligent interactive task, traverse the updated node tree to obtain a traversal result.

[0040] After obtaining the updated node tree, a breadth-first search algorithm is used to traverse the tree. Starting from the root node, the breadth-first search algorithm sequentially visits nodes in hierarchical order. It also considers the characteristics of the controls required for the pre-defined intelligent interaction tasks, such as control type, text content, and location.

[0041] Furthermore, during the traversal process, it is checked whether the control represented by each node meets these characteristics, until the entire node tree is traversed, and finally a traversal result is obtained, which will show whether there are controls that meet the characteristics and the positions of these controls in the node tree.

[0042] In one embodiment, the intelligent interactive task is to click the "Pause" button in the video playback interface. The control features of the "Pause" button are preset to be a button type and a text content of "Pause". In the video playback application, when the interface changes and the node tree is updated, the "VideoApp Accessibility Service" barrier-free service uses a breadth-first search algorithm to traverse the updated node tree. Starting from the root node (screen interface), each child node (control) is visited in turn to check whether each control meets the feature of "the type is a button and the text content is 'Pause'". If a node that meets the feature is found (that is, a node representing the "Pause" button), the relevant information is recorded as the traversal result; if no node that meets the feature is found after traversing the entire node tree, the traversal result is that there is no control that meets the requirements.

[0043] Step 50: If the traversal result shows that there is a target control that matches the control characteristics, the operation of the intelligent interaction task is simulated through the API provided by the accessibility service, and subsequent interactions are performed according to the completion status of the operation of the intelligent interaction task.

[0044] Furthermore, when the traversal results indicate the presence of a target control that matches pre-defined control characteristics, the accessibility service uses its provided API to simulate the intelligent interaction task's operation on the target control. For example, if it is a click operation, the service simulates a click on the target control. After the simulated operation is completed, the service determines whether subsequent interactive operations are necessary based on the completion status of the intelligent interaction task. For example, after clicking the "Pause" button, subsequent operations such as displaying a pause prompt or updating the progress bar status may be necessary.

[0045] Continuing with the above embodiment, in a video playback application, when the "VideoApp Accessibility Service" barrier-free service determines through the traversal results that there is a target control that meets the characteristics of the "Pause" button, it uses the API provided by the barrier-free service to simulate a click operation and click the "Pause" button. After the click operation is completed, check whether the video is successfully paused (this can be done by checking the relevant flag bits of the video playback status). If the video is successfully paused, subsequent interactions are performed, such as displaying a prompt message "Video Paused" on the screen and updating the status of the progress bar so that it stays at the current playback position; if the video is not successfully paused, subsequent interactions for error handling may be performed, such as prompting the user that the operation failed and trying to re-operate.

[0046] The embodiment of the present invention can timely detect changes in the interface through real-time event listening and interface update monitoring, suspend the current operation, and rebuild the node tree. In the dynamic positioning and verification of the target control, the breadth-first search algorithm is used in combination with predefined feature tags. Even if the interface layout changes, the target control can be accurately found in the updated node tree. Therefore, it effectively solves the problem that the existing method cannot work normally when the interface changes dynamically, and improves the reliability and practicality of intelligent interaction.

[0047] In one embodiment, steps 1 to 4 are described as follows:

[0048] Step 1: Traverse all control information, determine the root control in the control information, and create the root control as the root node of the node tree. The root control is the outermost container of the entire screen layout, including the LinearLayout control or LinearLayout control in Android.

[0049] Optionally, after obtaining information about all controls on the screen, you need to traverse this information to find the root control. The root control, as the outermost container of the entire screen layout, is the starting point for building the node tree. In the Android system, common root control types include LinearLayout controls or LinearLayout controls.

[0050] Therefore, by checking the properties and hierarchical relationships of the controls, we can determine which control is at the outermost layer, which is the root control. This root control is then created as the root node in the node tree, and the nodes corresponding to other controls will be built based on this root node.

[0051] Continuing with the above embodiment, in a video playback application, for example, the layout of the current screen interface uses LinearLayout as the outermost container. After the intelligent interactive device obtains all control information, it traverses this information. By checking the layout properties of each control, it is found that a certain LinearLayout control is not contained by any other control, and its range covers the range of the entire screen interface. It is determined that this LinearLayout control is the root control. Therefore, when constructing the node tree, a root node representing this LinearLayout root control is created. This root node will serve as the starting point of the entire node tree, and the nodes corresponding to all subsequent controls will establish a hierarchical relationship with it.

[0052] Step 2: Starting from the root node, check the child controls contained in the root control, create a first new node for the child control of the root control, and establish a parent-child relationship between the first new node and the root node. In the Android system, its child control list is obtained through its layout parameters and related APIs.

[0053] After determining the root node, the system uses layout parameters and related APIs provided by the Android system to retrieve a list of child controls within the root control. For each child control in this list, a new node, the first new node, is created. A parent-child relationship is then established between these first new nodes and the root node in the node tree, indicating that these child controls are direct children of the root control in the screen layout. In this way, the node tree gradually expands, reflecting the hierarchical structure of the screen layout.

[0054] In the video playback application described above, the LinearLayout root control corresponding to the root node contains a video playback area (such as a custom View control) and a control bar (such as another LinearLayout control) containing a play button, a pause button, etc. as child controls. The intelligent interactive device obtains these two child controls through Android's layout parameters and related APIs. Next, a new node is created for the custom View control in the video playback area, and a new node is also created for the LinearLayout control containing the control buttons. Then, in the node tree, a parent-child relationship is established between these two new nodes and the root node, clarifying their hierarchical relationship in the layout, that is, these two controls are direct child controls of the root LinearLayout control.

[0055] Step 3: For each target child node created in step 2, check whether the control corresponding to the target child node has its own child control. If so, repeat step 2 to create a second new node for each child control and establish a parent-child relationship between the second new node and the target child node.

[0056] Furthermore, after completing step 2, the node tree already contains the root node and its direct child nodes. At this time, for each child node created in step 2 (i.e., the target child node), it is necessary to further check whether the corresponding control has its own child controls. If there are child controls, then just like step 2, create new nodes for these child controls, i.e., the second new nodes. And establish a parent-child relationship between these second new nodes and the corresponding target child nodes in the node tree. In this recursive way, the hierarchical structure of the screen layout is continuously explored in depth, and all controls are added to the node tree with a suitable hierarchical relationship.

[0057] Continuing with the above embodiment, in a video playback application, the previously created control bar LinearLayout control (which serves as the control corresponding to the target child node) containing a play button, a pause button, and the like, itself contains a play button (Button control), a pause button (Button control), and a progress bar (SeekBar control) as child controls. After the intelligent interactive device detects that this control bar LinearLayout control has child controls, it creates a new node for the play button, a new node for the pause button, and a new node for the progress bar. Then, in the node tree, a parent-child relationship is established between these three new nodes and the node corresponding to the control bar LinearLayout control, indicating that the play button, pause button, and progress bar are child controls of the control bar LinearLayout control, thereby further improving the hierarchical structure of the node tree.

[0058] Step 4: When creating nodes and establishing parent-child relationships, for each node, record its relative position information on the screen and build a node tree based on the relative position information. The relative position information includes the coordinate information and size information of the control.

[0059] Furthermore, in the process of building the node tree, each time a node is created, the relative position information of the control corresponding to the node on the screen must be recorded. This information includes the coordinate information of the control (such as the x and y coordinates of the upper left corner) and size information (width and height).

[0060] Furthermore, a node tree is constructed based on the relative position information, as specifically described in steps 41 to 44, wherein the node tree in the embodiment of the present invention not only reflects the hierarchical relationship between controls, but also reflects the actual position and size of each control on the screen.

[0061] Continuing with the above embodiment, when the video playback application constructs a node tree, when creating a node for the custom View control in the video playback area, the coordinate information of the control on the screen is obtained, such as the upper left corner coordinates are (x1, y1), the width is width1, and the height is height1, and this information is recorded in the corresponding node.

[0062] Similarly, when creating a node for the play button, its top-left corner coordinates (x2, y2), width width2, and height height2 are obtained and recorded in the node corresponding to the play button. Similarly, for each created node, the relative position information of its corresponding control is recorded, so that the constructed node tree can fully describe the hierarchical relationship of each control on the screen and their actual position and size on the screen.

[0063] The node tree constructed by the embodiment of the present invention accurately reflects the hierarchical relationship of all controls on the screen interface and the relative position information of each control on the screen. Therefore, the node tree can be used to clearly understand the screen layout structure and quickly locate the control that needs to be operated.

[0064] In one embodiment, steps 41 to 44 are described as follows:

[0065] Step 41: For each current node, check whether it has sibling nodes. Sibling nodes represent other nodes that share the same parent node as the current node. By traversing the parent node's child node list, find all other nodes that share the same parent node as the current node, establish a sibling relationship between the current node and its sibling nodes, and add references or lists pointing to the sibling nodes in the node.

[0066] Optionally, for each node in the established hierarchical node tree, it is necessary to determine whether it has sibling nodes. A sibling node is a node that shares the same parent node as the current node. The intelligent interaction device traverses the list of child nodes of the current node's parent node and finds all sibling nodes that meet the criteria. It then establishes sibling relationships between the current node and these sibling nodes, and adds references or lists to these sibling nodes in the current node.

[0067] Continuing with the above example, in the node tree of a video playback application, for example, the LinearLayout control in the control bar is the parent node, which has three child nodes: a play button, a pause button, and a progress bar. For the play button node, the intelligent interaction device traverses the child node list of the control bar LinearLayout node, finds the pause button node and the progress bar node, and determines that they are sibling nodes of the play button node. It then adds references to the pause button node and the progress bar node in the play button node, establishing a sibling relationship between them.

[0068] Step 42: After all controls have been created as nodes according to the hierarchical relationship and parent-child and sibling relationships have been established, a depth-first traversal of the node tree is performed. During the traversal, each node is checked for special attributes. If special attributes exist, the relevant information is recorded in the node. Controls with special attributes include button controls and input box controls.

[0069] Furthermore, once all controls have been converted to nodes and parent-child and sibling relationships have been established, the intelligent interactive device performs a depth-first traversal of the entire node tree. A depth-first traversal is a traversal method that first visits a node's children and then backtracks to visit other nodes. During the traversal, each control corresponding to a node is checked to see if it has special properties. Controls with special properties, such as buttons and input boxes, record the relevant information in the corresponding node if they do have special properties.

[0070] Continuing with the above example, during a depth-first traversal of the node tree of a video playback application, when a play button node is accessed, because the play button is a button control and has special properties, the special attribute information "button control" is recorded in the play button node. Similarly, if the interface contains a search input box control, the special attribute information "input box control" is recorded when the search input box node is traversed.

[0071] Step 43: For each leaf node in the node tree, check whether it has a logical association with other leaf nodes, and add a reference list pointing to the relevant leaf nodes in the node.

[0072] Furthermore, a leaf node is a node in the node tree that has no child nodes. For each leaf node in the node tree, the intelligent interaction device checks whether it has a logical association with other leaf nodes. A logical association can be a functional association, such as a logical association between a confirmation button and an input box for data confirmation. If a logical association exists, a reference list pointing to the relevant leaf node is added to the leaf node.

[0073] Continuing with the above example, in a video playback application, for example, the play button and the volume slider are leaf nodes. There's a logical connection between them, as the playback state may affect volume adjustment. The intelligent interaction device detects this logical connection and adds a reference list pointing to the volume slider node to the play button node. Simultaneously, it also adds a reference list pointing to the play button node to the volume slider node.

[0074] Step 44: Optimize the entire node tree based on the hierarchical relationships, location information, special attributes, and logical associations recorded in the node tree. This optimization includes optimizing the node order and the node storage structure. Node order optimization ensures that nodes with similar functions or logical relationships are adjacent. Node storage structure optimization ensures that nodes with similar functions or logical relationships provide the same access interface.

[0075] Furthermore, based on the hierarchical relationships, location information, special attributes, and logical connections recorded in the node tree, the intelligent interaction device optimizes and adjusts the entire node tree. This optimization and adjustment includes two aspects: first, node order optimization. By adjusting the order of nodes in the tree, nodes with similar functions or logical relationships are placed adjacent to each other, which improves node traversal and operation efficiency. Second, node storage structure optimization and adjustment are carried out, providing nodes with similar functions or logical relationships with the same access interface.

[0076] Continuing with the above example, in a video playback application, the play button, pause button, and fast-forward button have similar functions and are all used to control video playback. The intelligent interactive device optimizes the node order in the node tree, relocating the three button nodes to adjacent positions. Furthermore, the device provides a common access interface for these three button nodes. For example, a "play control button" access interface is defined. This interface allows for unified operations on these three button nodes, such as querying their status and triggering click events.

[0077] The embodiment of the present invention optimizes the node tree and establishes sibling relationships to make the horizontal connections between nodes clearer, facilitating horizontal queries and operations in the node tree. Recording special attributes provides a basis for subsequent targeted operations based on the control type. Establishing logical associations of leaf nodes allows the node tree to better reflect the logical relationships between controls, facilitating the implementation of complex interactive logic, thereby improving the traversal efficiency and maintainability of the node tree, enabling intelligent interactive devices to more efficiently analyze and operate the screen interface, and providing more powerful support for the implementation of intelligent interactive tasks.

[0078] In one embodiment, steps 301 to 303 are described as follows:

[0079] Step 301: Obtain a new control information list after rescanning the interface.

[0080] Optionally, when the intelligent interactive device detects a change in the screen interface, it will pause the current interactive operation and then call the relevant API provided by the Android system to rescan the current screen interface. During the scanning process, detailed information about all controls on the interface is collected, including the control type (such as button, text box, image, etc.), position, size, text content, etc. This collected control information is organized into a list, namely the new control information list, which reflects the latest status of the current screen interface.

[0081] Continuing with the above example, in a video playback application, when a user clicks the "Full Screen" button on the video playback interface, the screen interface changes. The intelligent interactive device pauses any ongoing operations, such as dragging a progress bar. It then calls the relevant methods of the AccessibilityNodeInfo class to rescan the full-screen interface, obtaining information about controls such as the exit full-screen button, the video playback area controls in full-screen mode, and the new progress bar style. This information is then organized into a new control information list.

[0082] Step 302 : Compare and check each control in the new control information list with each control in the original control information list based on the original node tree one by one, and determine whether each control is a newly added control, a removed control, or a persistent control.

[0083] Furthermore, each control in the new control information list obtained in step 301 is compared one by one with each control in the original control information list on which the original node tree is based. The status of each control is determined by comparing key control attributes, such as the control's unique identifier (if any), type, location, and text content. If a control exists in the new list but not in the original list, it is a newly added control; if a control exists in the original list but not in the new list, it is a removed control; and if a control exists in both lists, it is a persistent control. Continuing with the above embodiment, in a video playback application, the original node tree is constructed based on a non-full-screen interface. The original control information list contains control information such as the play button, pause button, and normal progress bar in non-full-screen mode. In the new control information list obtained after rescanning, an exit full-screen button is found, which does not exist in the original list. Therefore, the exit full-screen button is determined to be a newly added control. At the same time, a small icon control dedicated to non-full-screen in the original list does not appear in the new list; the small icon control is determined to be a removed control. The play button and pause button exist in both lists and are considered persistent controls.

[0084] Step 303: Update the nodes in the node tree based on the newly added controls, removed controls, and persistent controls.

[0085] Furthermore, the nodes in the node tree are updated according to the newly added controls, removed controls, and persistent controls, as specifically described in steps 3031 to 3033 .

[0086] The embodiment of the present invention can update the node tree in a timely and accurate manner according to changes in the screen interface. Therefore, the updated node tree can truly reflect the latest status of the current screen interface, including the addition, removal and property changes of controls. When performing subsequent intelligent interaction tasks, it is possible to quickly locate the controls that need to be operated based on accurate node tree information, thereby improving the accuracy and efficiency of the interaction.

[0087] In one embodiment, steps 3031 to 3033 are described as follows:

[0088] Step 3031: For newly added controls that exist in the new control information list but not in the original control information list, determine their hierarchical position in the interface. Starting from the root node, follow the node tree hierarchy and determine the corresponding parent node position based on the layout relationship between the newly added control and the controls corresponding to existing nodes. Create a new node to represent the newly added control, add it to the corresponding parent node, and establish the corresponding parent-child relationship. Record complete control information for the new node, including its unique properties and layout parameters.

[0089] Optionally, when a new control is found in the new control information list, the intelligent interactive device must first clarify the hierarchical position of the new control in the interface. This needs to start from the root node of the node tree and judge based on the layout relationship between the new control and the control corresponding to the existing node. The layout relationship can be determined by the position of the control, nesting method, etc. Once the appropriate parent node position is found, a new node is created for the new control. Then, this new node is added to the corresponding parent node, and a parent-child relationship is established. At the same time, complete control information is recorded for the new node. This information covers the unique properties of the new control (such as the text content of the button, the prompt information of the input box, etc.) and layout parameters (such as position, size, etc.).

[0090] Continuing with the above embodiment, in a video playback application, when the user switches to a specific playback mode, a small window control for the "picture-in-picture" function is added to the interface. The intelligent interactive device starts checking from the root node of the node tree and finds that this "picture-in-picture" window control is displayed in a layout container in the main video playback area. Therefore, the layout container node is determined to be the parent node of the newly added control. Next, a new node is created for the "picture-in-picture" window control and added to the layout container node to establish a parent-child relationship. At the same time, the unique properties of the "picture-in-picture" window control are recorded, such as the video source information it displays, and layout parameters, such as the size of the window and its position in the main video area.

[0091] Step 3032: For controls that exist in the original control information list but disappear in the new control information list, recursively remove all child nodes starting from the node in the node tree. Delete the node from the child node list of its parent node, and cut off all parent-child and sibling relationship references related to the node.

[0092] Furthermore, for controls that exist in the original control information list but disappear in the new control information list, the intelligent interactive device needs to remove them in the node tree. Starting from the node corresponding to the control, recursively remove all its child nodes to ensure that all child control information related to the control is cleared. Then, delete the node from the child node list of its parent node, and at the same time cut off all parent-child and sibling relationship references related to the node. Continuing with the above embodiment, in a video playback application, when the user switches from a video list interface to a video playback interface, the video list item control in the original interface disappears in the new interface. The intelligent interactive device finds the node corresponding to the video list item control in the node tree. Since the video list item may contain some child controls (such as video titles, thumbnails, etc.), it first recursively removes the nodes corresponding to these child controls. Then, delete the video list item node from the child node list of its parent node (such as a list container node), and cut off the relationship reference between the node and other sibling nodes.

[0093] Step 3033: For controls that exist in both the new control information list and the original control information list, check whether their properties have changed. Re-acquire the updated property information of the control through the Android system API and synchronize the updated property information to the record of the corresponding node in the node tree.

[0094] Furthermore, for controls that exist in both the new control information list and the original control information list, the intelligent interactive device needs to check whether their properties have changed. By calling the Android system API, the updated property information of these controls is retrieved, such as the text content, color, size, position, etc. of the controls. Then, these updated property information are synchronized to the records of the corresponding nodes in the node tree to ensure that the node information in the node tree is consistent with the control status on the actual interface. Continuing with the above embodiment, in the video playback application, the play button exists before and after the interface changes. When the user switches to full-screen mode, the style of the play button may have changed, such as the color becoming darker, the size becoming larger, etc. The intelligent interactive device retrieves the updated property information of the play button through the Android system API, including the new color, size, and position, etc. Then, these updated property information are synchronized to the records of the nodes corresponding to the play button in the node tree, so that the node tree can accurately reflect the current status of the play button.

[0095] The embodiment of the present invention can comprehensively and accurately update the node tree according to the changes in the interface. For new controls, they are reasonably added to the node tree so that the node tree can reflect the newly appeared elements of the interface; for removed controls, the relevant information is completely cleared from the node tree to ensure the simplicity and accuracy of the node tree; for persistent controls, their attribute information is updated in a timely manner to ensure that the node tree is synchronized with the actual state of the interface, so that the updated node tree can provide a reliable basis for subsequent operations, so that intelligent interaction tasks can be efficiently executed based on accurate interface information, thereby improving the accuracy and stability of the interaction.

[0096] In one embodiment, the intelligent interaction method based on Android accessibility service further includes:

[0097] Step 60: During the control comparison check process, if the hierarchical position of the persistent control in the interface changes from a child control of the first parent control to a child control of the second parent control, the node corresponding to the persistent control is determined in the node tree, removed from the child node list of the first parent node, and relocated to the second parent node to establish a new parent-child relationship and update the sibling node relationship of the node.

[0098] Optionally, when performing a comparative check on the controls, if it is found that the hierarchical position of a persistent control in the interface has changed, that is, it has changed from a child control of the first parent control to a child control of the second parent control. The intelligent interactive device needs to accurately locate the node corresponding to the persistent control in the node tree. First, remove this node from the child node list of its original parent node (the first parent node) and cut off the original parent-child relationship between them. Then, according to the new interface layout, relocate the node to the second parent node and establish a new parent-child relationship. At the same time, since the parent node of the node has changed, its sibling nodes have also changed accordingly, so the sibling node relationship of the node needs to be updated to ensure that the node tree can accurately reflect the actual hierarchy and sibling relationship of the control in the interface.

[0099] Continuing with the above embodiment, in the video playback application, there was originally a "play mode selection" button as a child control of a sidebar layout (first parent control). When the user switches to a specific settings page, the "play mode selection" button is moved to a new tab layout (second parent control) of the settings page. The intelligent interactive device discovered this change when comparing the control information, found the node corresponding to the "play mode selection" button in the node tree, and removed it from the child node list of the sidebar layout node (first parent node). Then, the node is added to the tab layout node (second parent node) to establish a new parent-child relationship. Afterwards, the sibling node relationship of the "play mode selection" button node is updated to establish a correct association with the new sibling node (other control nodes under the tab layout).

[0100] Step 70: If the node position in the node tree changes, the order of the child nodes under each parent node is re-determined, and the order of the reference relationship between sibling nodes in the node tree is adjusted according to the new interface layout. Node position changes include adding nodes, removing nodes, and adjusting node hierarchies.

[0101] Furthermore, when the position of a node in the node tree changes, such changes include adding a new node position, removing a node position, and adjusting the node hierarchy. The intelligent interactive device needs to redetermine the order of the child nodes under each parent node. This is because the change in node position may affect the relative order between child nodes. According to the new interface layout, the order of the reference relationship between sibling nodes in the node tree is adjusted to ensure that the reference order of sibling nodes in the node tree is consistent with the actual arrangement order of controls on the interface, so that when the node tree is traversed and operated subsequently, the actual situation of the interface can be more accurately reflected.

[0102] Continuing with the above embodiment, in the playlist interface of the video playback application, there are originally three video item controls arranged in order as video item 1, video item 2, and video item 3. After the user performs some operations, video item 2 is removed, and a new video item 4 is added to the original position of video item 2. The intelligent interactive device detects that the node position has changed and redetermines the order of the sub-nodes under the playlist layout node. The node corresponding to video item 2 is removed, and the node corresponding to video item 4 is added to the appropriate position. The order of the reference relationship between the sibling nodes is then adjusted so that the sibling node reference of the video item 1 node points to the video item 4 node, and the sibling node reference of the video item 4 node points to the video item 3 node, ensuring that the reference order of the sibling nodes in the node tree is consistent with the actual arrangement order of the video items on the interface.

[0103] By processing the persistent changes in the hierarchical positions of controls and various changes in node positions, the embodiment of the present invention ensures that the node tree always accurately reflects the true hierarchical relationships, parent-child relationships, and sibling relationships of controls on the interface. This enables the target control to be efficiently and precisely located based on accurate node tree information during subsequent intelligent interaction tasks, thus avoiding operational errors caused by changes in the interface layout.

[0104] In one embodiment, the intelligent interaction method based on Android accessibility service further includes:

[0105] Step 80 : During the control comparison and checking process, if the structure of the node tree changes, then the control nodes having a logical association relationship are determined, and whether the logical association relationship between the control nodes has changed.

[0106] Optionally, when performing a comparative check on the controls, when the structure of the node tree changes, it may be due to adjustments to the interface layout, addition or removal of controls, etc. The intelligent interactive device needs to first find the control nodes with logical associations. The logical association relationship can be a functional connection, such as a search input box and a search button, after entering the content, click the search button to search; it can also be a state association, such as a play button and a pause button, which control different playback states of the video. After determining these control nodes with logical associations, further check whether the logical associations between these control nodes have changed. This may require comparing whether the association rules, triggering conditions, etc. between the control nodes are different before and after the node tree structure changes.

[0107] Continuing with the above embodiment, in a video playback application, the original node tree structure is stable, and the play button and pause button have a clear logical association with the video playback status. Clicking the play button starts playing the video, and clicking the pause button pauses the video. When the user switches to a new playback mode, the node tree structure changes. For example, the interface layout is readjusted, and the positions of the play button and pause button change. At this point, the intelligent interactive device needs to determine whether the logical association relationship between the play button node, the pause button node, and the video playback status node has changed. It is possible that originally clicking the play button directly started playback, but now you need to select the playback source first and then click the play button to start playback, which means that the logical association relationship has changed.

[0108] Step 90: If a change occurs, the logical association relationship between the control nodes is repaired and updated according to the updated node tree structure and the actual state of the control, so that the association pointer or data structure in the updated logical association relationship is consistent with the association pointer or data structure in the original logical association relationship.

[0109] Furthermore, if changes are detected in step 80 regarding the logical associations between control nodes, the intelligent interactive device needs to repair and update these logical associations based on the updated node tree structure and the actual state of the controls. The goal is to ensure that the association pointers or data structures in the updated logical associations remain consistent with the original logical associations, thereby ensuring that the intelligent interactive task can be executed normally according to the original logical rules. This may involve resetting reference relationships between nodes and adjusting association conditions to ensure the correctness and consistency of the logical associations. Continuing with the above embodiment, in a video playback application, the logical associations change due to a new playback mode. The intelligent interactive device repairs and updates the logical associations between the play button, pause button, and video playback status nodes based on the updated node tree structure, namely the new interface layout and control positions, as well as the actual state of the controls, such as the playback source selection logic. For example, in the new logic, an association is added between the playback source selection node and the play button node. Once the playback source is selected, clicking the play button will trigger video playback. Simultaneously, the association pointers and data structures are adjusted so that the new logical associations are essentially consistent with the original logical associations, both of which are logically controlled around the video play and pause operations.

[0110] The embodiment of the present invention ensures that even if the structure of the node tree changes, the logical association relationship between the control nodes can be checked and repaired in a timely manner, maintaining the consistency of the logical association relationship. Therefore, even if the interface layout is constantly changing, the intelligent interaction tasks can still be accurately performed according to the original logical rules, avoiding the logical confusion caused by interface changes and improving the stability and reliability of intelligent interaction.

[0111] Furthermore, the intelligent interaction device based on Android accessibility service provided by the present invention is described below. The intelligent interaction device based on Android accessibility service described below and the intelligent interaction method based on Android accessibility service described above can refer to each other.

[0112] Optional, see Figure 2 , Figure 2 It is a structural diagram of the intelligent interactive device based on Android accessibility service provided by the present invention, and the intelligent interactive device based on Android accessibility service includes.

[0113] The monitoring module 210 is used to inherit the Accessibility Service class to create a custom accessibility service when the Android application is started, and specify the event type that the accessibility service monitors by configuring an XML file;

[0114] Construction module 220 is used to, after the accessibility service is successfully started and a signal indicating that the interface loading is complete is received, perform a comprehensive scan of the current screen interface, obtain control information of all controls on the current screen interface based on an API provided by the Android system, and construct a node tree based on the control information in a hierarchical relationship;

[0115] Update module 230 is used to continuously monitor various events captured by the accessibility service. When a change in the current screen interface is detected, the current interactive operation is suspended and the node tree is updated based on the results of the interface rescan;

[0116] The traversal module 240 is configured to traverse the updated node tree based on a breadth-first search algorithm and control features corresponding to the controls required to be operated in the pre-set intelligent interaction task to obtain a traversal result;

[0117] The intelligent interaction module 250 is used to simulate the operation of the intelligent interaction task through the API provided by the accessibility service if the traversal result shows that there is a target control that matches the control characteristics, and perform subsequent interactions based on the completion status of the operation of the intelligent interaction task.

[0118] The embodiment of the present invention can timely detect changes in the interface through real-time event listening and interface update monitoring, suspend the current operation, and rebuild the node tree. In the dynamic positioning and verification of the target control, the breadth-first search algorithm is used in combination with predefined feature tags. Even if the interface layout changes, the target control can be accurately found in the updated node tree. Therefore, it effectively solves the problem that the existing method cannot work normally when the interface changes dynamically, and improves the reliability and practicality of intelligent interaction.

[0119] See also Figure 3 , Figure 3 This is a diagram of an embodiment of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0120] When the Android app starts, inherit the Accessibility Service class to create a custom accessibility service, and specify the event types that the accessibility service listens for by configuring the XML file;

[0121] When the accessibility service is successfully started and receives the signal that the interface loading is complete, it performs a comprehensive scan of the current screen interface, obtains the control information of all controls on the current screen interface based on the API provided by the Android system, and constructs a node tree based on the hierarchical relationship based on the control information;

[0122] Continuously monitor various events captured by the accessibility service. When a change is detected in the current screen interface, the current interactive operation is paused and the node tree is updated based on the results of the interface rescan.

[0123] Based on the breadth-first search algorithm and the control features corresponding to the controls required to be operated in the pre-set intelligent interaction tasks, the updated node tree is traversed to obtain the traversal results;

[0124] If the traversal result shows that there is a target control that matches the control characteristics, the operation of the intelligent interaction task is simulated through the API provided by the accessibility service, and subsequent interactions are performed based on the completion status of the intelligent interaction task.

[0125] See also Figure 4 , Figure 4 Detailed description of an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 400 on which a computer program 311 is stored. When the computer program 311 is executed by a processor, the following steps are implemented:

[0126] When the Android app starts, inherit the Accessibility Service class to create a custom accessibility service, and specify the event types that the accessibility service listens for by configuring the XML file;

[0127] When the accessibility service is successfully started and receives the signal that the interface loading is complete, it performs a comprehensive scan of the current screen interface, obtains the control information of all controls on the current screen interface based on the API provided by the Android system, and constructs a node tree based on the hierarchical relationship based on the control information;

[0128] Continuously monitor various events captured by the accessibility service. When a change is detected in the current screen interface, the current interactive operation is paused and the node tree is updated based on the results of the interface rescan.

[0129] Based on the breadth-first search algorithm and the control features corresponding to the controls required to be operated in the pre-set intelligent interaction tasks, the updated node tree is traversed to obtain the traversal results;

[0130] If the traversal result shows that there is a target control that matches the control characteristics, the operation of the intelligent interaction task is simulated through the API provided by the accessibility service, and subsequent interactions are performed based on the completion status of the intelligent interaction task.

[0131] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the intelligent interaction method based on Android accessibility services provided by the above methods, which includes:

[0132] When the Android app starts, inherit the Accessibility Service class to create a custom accessibility service, and specify the event types that the accessibility service listens for by configuring the XML file;

[0133] When the accessibility service is successfully started and receives the signal that the interface loading is complete, it performs a comprehensive scan of the current screen interface, obtains the control information of all controls on the current screen interface based on the API provided by the Android system, and constructs a node tree based on the hierarchical relationship based on the control information;

[0134] Continuously monitor various events captured by the accessibility service. When a change is detected in the current screen interface, the current interactive operation is paused and the node tree is updated based on the results of the interface rescan.

[0135] Based on the breadth-first search algorithm and the control features corresponding to the controls required to be operated in the pre-set intelligent interaction tasks, the updated node tree is traversed to obtain the traversal results;

[0136] If the traversal result shows that there is a target control that matches the control characteristics, the operation of the intelligent interaction task is simulated through the API provided by the accessibility service, and subsequent interactions are performed based on the completion status of the intelligent interaction task.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0138] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An intelligent interaction method based on Android accessibility service, characterized in that: include: When the Android application starts, inherit the Accessibility Service class to create a custom accessibility service, and specify the event types that the accessibility service listens for by configuring the XML file; When the accessibility service is successfully started and receives a signal indicating that the interface loading is complete, the current screen interface is fully scanned, control information of all controls on the current screen interface is obtained based on an API provided by the Android system, and a node tree is constructed according to a hierarchical relationship based on the control information; Continuously monitor various events captured by the accessibility service, and when a change in the current screen interface is detected, suspend the current interactive operation and update the node tree based on the results of rescanning the interface; Based on the breadth-first search algorithm and the control features corresponding to the controls required to be operated in the pre-set intelligent interaction tasks, the updated node tree is traversed to obtain the traversal results; If the traversal result shows that there is a target control that matches the control feature, the operation of the intelligent interaction task is simulated through the API provided by the accessibility service, and subsequent interactions are performed according to the completion status of the operation of the intelligent interaction task.

2. The intelligent interaction method based on Android accessibility service according to claim 1, characterized in that: The node tree is constructed based on the control information according to the hierarchical relationship, including Step 1: Traverse all control information, determine the root control in the control information, and create the root control as the root node of the node tree; the root control is the outermost container of the entire screen layout, including the LinearLayout control or LinearLayout control in Android; Step 2: Starting from the root node, check the child controls contained in the root control, create a first new node for the child control of the root control, and establish a parent-child relationship between the first new node and the root node; in the Android system, obtain its child control list through its layout parameters and related APIs; Step 3: For each target child node created in step 2, check whether the control corresponding to the target child node has its own child control; if so, repeat the operation of step 2 to create a second new node for each child control, and establish a parent-child relationship between the second new node and the target child node; Step 4: In the process of creating nodes and establishing parent-child relationships, for each node, record its relative position information on the screen, and build the node tree based on the relative position information; the relative position information includes the coordinate information and size information of the control.

3. The intelligent interaction method based on Android accessibility service according to claim 2, characterized in that: The step of constructing the node tree based on the relative position information includes: Step 41: For each current node, check whether it has sibling nodes; the sibling nodes represent other nodes that have the same parent node as the current node; by traversing the child node list of the parent node, find all other nodes that have the same parent node as the current node, establish a sibling relationship between the current node and its sibling nodes, and add a reference or list pointing to the sibling nodes in the node; Step 42: After all controls have been created as nodes according to the hierarchical relationship and the parent-child and sibling relationships have been established, a depth-first traversal is performed on the node tree. During the traversal, for each node, check whether it has special attributes. If special attributes exist, the relevant information is recorded in the node. Controls with special attributes include button controls and input box controls. Step 43: For each leaf node in the node tree, check whether it has a logical association with other leaf nodes, and add a reference list pointing to the relevant leaf nodes in the node; Step 44: Optimize and adjust the entire node tree based on the hierarchical relationship, location information, special attributes and logical associations recorded in the node tree; the optimization and adjustment include node sequence optimization and adjustment and node storage structure optimization and adjustment; the node sequence optimization and adjustment makes nodes with similar functions or logical relationships adjacent; the node storage structure optimization and adjustment makes nodes with similar functions or logical relationships provide the same access interface.

4. The intelligent interaction method based on Android accessibility service according to claim 1, characterized in that: The updating of the node tree based on the result of rescanning the interface includes: Get the new control information list after rescanning the interface; Comparing each control in the new control information list with each control in the original control information list based on the original node tree, and determining whether each control is a newly added control, a removed control, or a persistent control; The nodes in the node tree are updated based on the newly added controls, the removed controls, and the persistent controls.

5. The intelligent interaction method based on Android accessibility service according to claim 4, characterized in that: The updating of nodes in the node tree based on the newly added controls, the removed controls, and the persistent controls includes: For a newly added control that exists in the new control information list but does not exist in the original control information list, determine its hierarchical position in the interface; starting from the root node, follow the hierarchical structure of the node tree and determine the corresponding parent node position based on the layout relationship between the newly added control and the controls corresponding to the existing nodes; create a new node to represent the newly added control, add it to the corresponding parent node, and establish a corresponding parent-child relationship; record complete control information for the new node, including its unique properties and layout parameters; For controls that exist in the original control information list but disappear in the new control information list, recursively remove all child nodes starting from the node in the node tree; delete the node from the child node list of its parent node, and cut off all parent-child and sibling relationship references related to the node; For controls that exist in both the new control information list and the original control information list, check whether their properties have changed; re-acquire the updated property information of the control through the Android system API, and synchronize the updated property information to the record of the corresponding node in the node tree.

6. The intelligent interaction method based on Android accessibility service according to any one of claims 1 to 5, characterized in that: The intelligent interaction method based on Android accessibility service also includes: During the control comparison check, if the hierarchical position of a persistent control in the interface changes from a child control of the first parent control to a child control of the second parent control, the node corresponding to the persistent control is determined in the node tree, removed from the child node list of the first parent node, and relocated to the second parent node to establish a new parent-child relationship, and the node's sibling node relationship is updated; If the position of a node in the node tree changes, the order of the child nodes under each parent node is re-determined, and the order of the reference relationship between sibling nodes in the node tree is adjusted according to the new interface layout; changes in node positions include adding new nodes, removing nodes, and adjusting node hierarchies.

7. The intelligent interaction method based on Android accessibility service according to claim 6, characterized in that: The intelligent interaction method based on Android accessibility service also includes: During the control comparison check process, if the structure of the node tree changes, the control nodes with logical association relationships are determined, and whether the logical association relationships between the control nodes have changed; If changes occur, the logical association relationships between control nodes are repaired and updated according to the updated node tree structure and the actual state of the control, so that the association pointers or data structures in the updated logical association relationships are consistent with the association pointers or data structures in the original logical association relationships.

8. An intelligent interactive device based on Android accessibility services, characterized in that: Applicable to the intelligent interaction method based on Android accessibility service as described in any one of claims 1 to 7; The intelligent interactive device based on Android accessibility service includes: The monitoring module is used to inherit the Accessibility Service class to create a custom accessibility service when the Android application is started, and specify the event types that the accessibility service monitors by configuring an XML file; A construction module is configured to, after the accessibility service is successfully started and a signal indicating that the interface loading is complete is received, perform a comprehensive scan of the current screen interface, obtain control information of all controls on the current screen interface based on an API provided by the Android system, and construct a node tree according to a hierarchical relationship based on the control information; An update module, configured to continuously monitor various events captured by the accessibility service, and when detecting a change in the current screen interface, suspend the current interactive operation and update the node tree based on the result of rescanning the interface; The traversal module is used to traverse the updated node tree based on the breadth-first search algorithm and the control features corresponding to the controls that need to be operated in the pre-set intelligent interaction tasks to obtain the traversal results; An intelligent interaction module is used to simulate the operation of the intelligent interaction task through the API provided by the accessibility service if the traversal result shows that there is a target control that matches the control characteristics, and perform subsequent interactions according to the completion status of the operation of the intelligent interaction task.

9. An electronic device comprising: Memory for storing computer software programs; A processor for reading and executing the computer software program, characterized in that when the processor executes the computer software program, it implements the intelligent interaction method based on Android accessibility service as claimed in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer software program, wherein: When the computer software program is executed by the processor, the intelligent interaction method based on Android accessibility service as claimed in any one of claims 1 to 7 is implemented.

Images