A method for automating user interface operations of an app

Abstract

This invention relates to the fields of automated testing and automated operation of mobile applications, and discloses a method for automating user interface operations in an app. The method uses a UI control acquisition module to obtain all clickable controls and their related information on the current page, while simultaneously monitoring real-time changes in the page's state. A UI tree construction and management module constructs and updates the UI tree based on the acquired information, recording different page states and their transition paths. An action selection module performs random action selection or shortest path selection based on the user's chosen operation mode to determine the optimal operation path. An action execution and information recording module executes the selected action and records the page state changes and path transition information before and after the action. A traffic capture module captures the network traffic of the application throughout its operation and saves the traffic data as a PCAP file. An environment monitoring module monitors the normal operation of the program, effectively improving the quality of the application.

Description

Technical Field

[0001] This invention relates to the fields of automated testing and automated operation technology for mobile applications, specifically a method for automating user interface operations in an app. Background Technology

[0002] With the rapid development and popularization of mobile applications, the requirements for application quality and stability are constantly increasing. Although traditional manual testing methods are intuitive, they are inefficient and difficult to handle large-scale application testing. In addition, manual testing lacks the real randomness of actions when simulating user operations and cannot accurately reproduce a variety of user behaviors.

[0003] Automated testing has thus become an important solution. However, existing automated testing methods still have shortcomings in terms of the stability and reliability of user interface operations. Especially when there are significant changes in devices and environments, it is difficult to guarantee the accuracy and consistency of UI operations. These automated testing tools often lack the ability to implement the actual operation path for a specific target page, which leads to uncertainty in the operation path, wastes valuable resources and time, and the operation results may not be as expected.

[0004] Therefore, there is an urgent need for a stable and reliable UI automation operation method that can achieve stable operation under different device and environmental conditions. This method should not only ensure the stability and consistency of operation, but also have the ability to intelligently analyze and randomly process user interactions, so as to more realistically simulate user operations, avoid resource waste, and accurately achieve the test objectives. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method for automating user interface operations in apps. It aims to achieve efficient and comprehensive traversal and analysis of mobile application interfaces by combining random action selection and shortest path algorithms. The system constructs and maintains a UI tree, recording different page states and their transition paths. Utilizing multiple operation modes, such as random action selection and shortest path-based action selection, it ensures comprehensive coverage of all interface states and interactive operations of the application. A probability table defines the probability of action selection, and the system supports various interactive actions, including clicking and swiping. Simultaneously, for each operation, the system records and analyzes the success rate and probability distribution of the transition path, continuously expanding and optimizing the UI tree to ensure thorough testing and traversal of all possible UI states, thus solving the aforementioned problems.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for automating user interface operations in an app, comprising the following steps:

[0009] S1. Obtain all clickable controls on the current app page through the uiautomator2 library in the UI control acquisition module, and obtain relevant information about these controls, including resource ID, class name, text content, description, and boundary information;

[0010] S2. The UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module. It updates the UI tree structure by calculating the similarity of page states and the real-time changes of the page, records the number of page visits, manages node information, and gradually builds the entire UI tree.

[0011] S3. Based on a predefined probability table, the random action selection mode in the action selection module randomly selects actions such as click, swipe, and return, mapping the actions to different operations. Based on the UI tree and edge information table in the UI tree construction and management module, the shortest path selection mode in the action selection module calculates the optimal UI operation path from the current page state to the target page state using the shortest path algorithm, and selects the corresponding optimal action sequence.

[0012] S4. The action execution and information recording module operates the device interface according to the action determined by the action selection module, including click, swipe and return operations. After each operation, the state change of the page is obtained and recorded in the edge information table, and the UI tree built in the UI tree construction and management module is updated.

[0013] S5. Capture network traffic of the application throughout its operation using the traffic capture module, and save the traffic data as a PCAP file.

[0014] Preferably, the UI control acquisition module uses an automated testing library. ,pass To find all clickable controls on the current page.

[0015] Preferably, the UI control acquisition module extracts control information from each control object and constructs it into a unique key. During the extraction of control information, each control... By unique key To identify, control information It is represented as follows:

[0016]

[0017] in, Indicates resource information, Indicates class name information, Represents text content information. This indicates descriptive information.

[0018] Preferably, after acquiring and storing all the necessary information, the UI control acquisition module generates a control list according to the control's attributes. At the same time, irrelevant package name controls are filtered out. When generating the control list for the current page, each control... Control list record Described as a triple, as shown below:

[0019]

[0020] in Indicates the center position of the control. Represents a control object. This represents the control's identification information.

[0021] Preferably, the UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module. This unique key includes an identifier constructed from the identifiers of all controls on the page. Total The identifiers for this page are as follows:

[0022]

[0023] in, Represents the page, Indicates the page A control.

[0024] Preferably, the formula for calculating the page state similarity is as follows:

[0025]

[0026] in, This represents the size of the intersection of the control collections on the new page and the old page. This represents the size of the union of the control sets on the new page and the old page.

[0027] Preferably, the UI tree construction and management module determines whether the current page already exists in the UI tree based on the calculated page state similarity. If it does not exist, it adds the page as a new node to the UI tree and initializes the node. If it exists, it updates the information of the existing node, calculates the similarity between the new page state and each existing page state, and selects the page state with the highest similarity. If the maximum similarity is lower than a preset threshold, it indicates that the page is a new page, and a new node is added to the UI tree. If the maximum similarity is higher than or equal to the preset threshold, the information of the existing node is updated.

[0028] Preferably, the probability table is constructed as follows:

[0029]

[0030] in, This represents the constructed probability table. This indicates the probability of a click operation. This indicates the probability of a return operation. This represents the probability of sliding backward. Indicates sliding forward. This represents the probability of sliding to the left. This represents the probability of a rightward swipe.

[0031] Preferably, the traffic capture module captures traffic while the action is being executed, specifically including the following process:

[0032] During the process of the application performing activities according to the above rules, the entire process uses... Command starts on device This tool captures network traffic data from a specified application and saves it as... The file will be saved after the traffic capture is complete. Files are copied from the device to local storage, and temporary files on the device are cleaned up.

[0033] A method for automating user interface operations in an app, characterized in that it further includes a device and environment monitoring module, wherein the device and environment monitoring module specifically includes the following sub-modules:

[0034] The device volume status monitoring module continuously monitors the device's volume status through a separate thread. Within this thread, it periodically checks the device's volume status; if the volume is not muted, it automatically mutes the volume. Commands to get and set device volume;

[0035] The device connection status monitoring module, through calling... Commands are used to detect and connect devices, ensuring that the devices are... It runs in a specific state and handles and reports when connection errors or timeouts occur;

[0036] The screen status management module, through execution The command is used to detect the current device window status. If an error message window, unresponsive window, status bar, or other non-target application windows are detected, the program will force close these windows and return to the target application interface, while hiding the soft keyboard by sending specific key events.

[0037] Compared with the prior art, the present invention provides a method for automating user interface operations in an app, which has the following beneficial effects:

[0038] This invention combines UI control acquisition, UI tree construction, random action selection, and intelligent path optimization. This method enables the random simulation of real user behavior under different device and environmental conditions, overcoming the limitations of traditional manual testing and existing automated testing. Specifically, it utilizes the uiautomator2 library to acquire clickable control information of the current page, dynamically constructs and manages the UI tree, ensuring consistency and stability of operations. By randomly selecting user interaction actions and using the shortest path algorithm, it optimizes operation paths, reducing resource waste and testing time. Furthermore, the traffic capture module monitors the application's network traffic in real time, providing data support for subsequent analysis, while the device and environment monitoring module ensures the consistency of the testing environment, preventing external factors from affecting test results. Overall, this invention not only improves the intelligence level of automated testing but also effectively enhances application quality and user experience, meeting increasingly stringent market demands. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the method steps of the present invention;

[0040] Figure 2 This is a diagram showing the overall modular structure of the present invention and the relationships between the modules;

[0041] Figure 3 This is a flowchart illustrating the operation of the random action selection mode in this invention.

[0042] Figure 4 This is a flowchart illustrating the operation of the shortest path selection action mode in this invention. Detailed Implementation

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] Please see Figure 1 A method for automating user interface operations in an app includes five modules: UI control acquisition module, UI tree construction and management module, action selection module, action execution and information recording module, and device traffic capture module.

[0045] It also includes an environmental monitoring module to monitor the operation of the above five modules;

[0046] UI control acquisition module: using The library retrieves all clickable controls on the current application page and collects relevant information, including resource ID, class name, text content, description, and boundary information.

[0047] UI Tree Construction and Management Module: Based on the control information obtained by the UI control acquisition module, it generates a unique key for the current page state, updates the UI tree structure by calculating the similarity of page states and real-time changes, records the number of page visits, manages node information, and gradually builds a complete UI tree;

[0048] Action selection module: includes two modes: random action selection and shortest path selection. The former randomly selects an action based on a predefined probability, while the latter uses the shortest path algorithm to calculate and execute the optimal action path from the current page state to the target page state.

[0049] Action execution and information recording module: Performs operations on the device interface according to the actions determined by the action selection module, including clicking, swiping, returning, etc. After each operation, it obtains the state changes of the page and records them in the edge information table, and updates the UI tree constructed by the UI tree construction and management module.

[0050] Traffic capture module: This module is responsible for capturing the network traffic of the application throughout its operation and saving the traffic data. The document is for subsequent analysis and includes two main functions: traffic capture and traffic collection.

[0051] The device and environment monitoring module includes device volume status monitoring, device connection status monitoring, and screen status management. Device volume monitoring continuously monitors volume via an independent thread to ensure uninterrupted testing. Device connection monitoring ensures the device... In this state, it runs and handles connection issues, while screen management closes non-target application windows and hides the soft keyboard;

[0052] The operation process for each module is described below:

[0053] The UI control acquisition module retrieves all clickable controls on the current application page and collects relevant information, including:

[0054] The UI control acquisition module uses an automated testing library. ,pass This method retrieves all clickable controls on the current page. These controls have attributes, allowing for further manipulation and analysis of them;

[0055] Collecting and storing control information: The UI control acquisition module further includes extracting control information from each control object and constructing it into a unique key;

[0056] The process of extracting control information ensures that each control... By unique key To identify, its information includes resources Class name Text content and description All control information It can be expressed as:

[0057]

[0058] Building a control list and filtering irrelevant controls: After acquiring and storing all the necessary information, the UI control acquisition module generates a control list according to the control's attributes. At the same time, it filters out irrelevant package name controls, such as system... When generating the list of controls for the current page, each control... Control list record It can be described as a triple, as shown below:

[0059]

[0060] in Indicates the center position of the control. Represents a control object. This is represented as control identification information;

[0061] This code ensures that only clickable controls belonging to a specific application are retained, excluding system-level or irrelevant application controls. It also generates a unique key for each control and stores it in the view table. In the middle, maintain the control list For subsequent use;

[0062] The UI tree construction and management module constructs and updates the UI tree structure, specifically including:

[0063] A unique key for generating the current page state is created by the UI tree construction and management module based on the control information obtained by the UI control acquisition module. This unique key includes an identifier constructed from the identifiers of all controls on the page, ensuring that the current page state can be uniquely identified. If there is a total If there are multiple controls, then the identifier for this page can be represented as:

[0064]

[0065] Calculating page state similarity: The UI tree construction and management module further includes calculating the similarity of page states. Specifically, it involves comparing the current page... state With existing pages state The similarity between the controls is calculated by comparing them using set operations. The similarity calculation formula is as follows:

[0066]

[0067] in, This represents the size of the intersection of the control collections on the new page and the old page. This represents the size of the union of the control sets on the new page and the old page.

[0068] Update the UI tree structure: Based on the calculated similarity, the UI tree construction and management module determines whether the current page already exists in the UI tree. If it does not exist, it adds the current page as a new node to the UI tree and initializes the node. If the current page exists, it updates the information of the existing node. Specific implementation includes:

[0069] Calculate the similarity between the new page state and each existing page state, and select the page state with the highest similarity. If the maximum similarity is lower than the preset threshold, it indicates that the page is a new page, and a new node is added to the UI tree. If the maximum similarity is higher than or equal to the preset threshold, the information of the existing node is updated.

[0070] Record page access counts: The UI tree building and management module maintains a page access count record, recording the number of times each page state is accessed, and uses this record to optimize the structure of the UI tree;

[0071] Management node information: The UI tree construction and management module further includes management node information, which records detailed information about each page node, including page status, parent-child relationship of nodes, etc., to ensure that the UI tree can accurately represent the transformation relationship between pages. Through subsequent operation activities and page status changes, the entire UI tree structure is gradually built and improved.

[0072] The core modules in this method are the action selection module and the action execution and information recording module. The action selection module includes two modes: random action selection and shortest path selection. Random action selection is based on a predefined probability table to randomly select an operation, while shortest path selection uses the shortest path algorithm to calculate and execute the optimal operation path from the current page state to the target page state. Since the action execution and information recording module participates in the operation and information recording of these two action selection modes, its content is introduced together with the action selection module.

[0073] Random action selection mode such as Figure 3 As shown, the random action selection mode in the action selection module selects actions based on a predefined probability table. Define the probability of choosing each action, where, This indicates the probability of a click operation. This indicates the probability of a return operation. This represents the probability of sliding backward. Indicates sliding forward. This represents the probability of sliding to the left. This indicates the probability of actions such as swiping to the right.

[0074]

[0075] The cumulative probability is generated based on the probability table, and the action is selected using random numbers. The step generates the cumulative probability by calculating the cumulative sum of the probability table, and uses the generated random numbers to determine the selected action. The calculation of the cumulative probability ensures that the previous probability sum values ​​can be reflected in the action selection, thereby realizing the random selection of actions. The corresponding action is selected and executed. Random selection includes randomly selecting and clicking a control from the UI control list, performing a swipe operation or a back button operation. Specific actions include, but are not limited to, clicking a random control, swiping backward, swiping forward, swiping left, swiping right, and returning. All clickable controls and their related information on the current page are retrieved again to detect and record changes in the page state. This includes retrieving the control information of the new page and recalculating the page state, and updating the UI tree based on this information.

[0076] Shortest path selection mode such as Figure 4 As shown, by inputting the current page state and the target page state, and based on the UI tree constructed from the previously calculated transition paths between page states, the shortest path is used. The algorithm calculates the path from the current page state to the target page state. In the UI tree, nodes represent page states, and edges represent page state transition events (such as clicks and swipes). After obtaining the optimal operation sequence through the shortest path, the algorithm checks the page state changes based on the executed actions to ensure accurate recording of the new page state changes after each operation. The algorithm records the path information of the page state changes and stores it in an edge information table. This includes, but is not limited to, the number of times this action is executed, its conversion power, and its probability distribution;

[0077] The traffic capture module is a crucial component of network monitoring and analysis systems. It is responsible for capturing network traffic in real-time throughout the entire application's operation. This module employs efficient network packet capture technology, utilizing methods such as... or The underlying libraries can accurately intercept data packets passing through network interfaces. These data packets contain rich network information, including source IP, destination IP, port number, protocol type, and transmitted data content.

[0078] During the traffic capture process, the module can be flexibly configured to support the capture needs of various network environments, such as LAN, WAN and virtual network. In order to ensure the comprehensiveness and accuracy of the capture, the module also implements a filtering mechanism, which allows users to set the capture of specific types of data packets, thereby reducing unnecessary data interference and improving the efficiency of data analysis.

[0079] The traffic collection function is responsible for saving the captured data traffic as... ( This file format is widely used in network analysis tools, such as... This facilitates in-depth analysis and troubleshooting. During traffic collection, the module can segment and store data to ensure that storage will not overflow due to excessive data volume during long-term capture. It also supports compression and encryption of captured traffic to ensure data security and integrity.

[0080] The device and environment monitoring module comprises three parts: device volume status monitoring, device connection status monitoring, and screen status management. The specific components are as follows:

[0081] Device volume status monitoring: By creating a separate thread, the device's volume status is continuously monitored to ensure that the testing process is not affected by external interference. Specifically, in the separate thread, the device's volume status is checked periodically. If the device's volume is detected to be not muted, it is automatically muted. Commands to get and set device volume;

[0082] Device connection status monitoring: by calling Commands are used to detect and connect devices, ensuring that the devices are... It runs in a specific state and handles and reports when connection errors or timeouts occur;

[0083] Screen status management: By executing The command is used to detect the current device window status. If an error message window, unresponsive window, status bar, or other non-target application windows are detected, the program will forcibly close these windows and return to the target application interface. At the same time, the soft keyboard is hidden by sending specific key events to ensure normal UI operation.

[0084] By applying the relevant modules of the above methods, random simulation of real user behavior can be achieved under different device and environmental conditions, overcoming the limitations of traditional manual testing and existing automated testing. At the same time, by randomly selecting user interaction actions and shortest path algorithms, operation paths are optimized, reducing resource waste and testing time.

[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automating user interface operations in an app, characterized in that, The app automation user interface operation method includes five modules: UI control acquisition module, UI tree construction and management module, action selection module, action execution and information recording module, and device traffic capture module. It also includes an environmental monitoring module to monitor the operation of the above five modules; Includes the following steps: S1. Obtain all clickable controls on the current app page through the uiautomator2 library in the UI control acquisition module, and obtain relevant information about these controls, including resource ID, class name, text content, description, and boundary information; S2. The UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module. It updates the UI tree structure by calculating the similarity of page states and the real-time changes of the page, records the number of page visits, manages node information, and gradually builds the entire UI tree. S3, Action Selection Module: Includes two modes: random action selection and shortest path selection. The random action selection mode is based on predefined probabilities to randomly select operations. Specifically, it involves generating a cumulative probability by calculating the cumulative sum of a probability table, and using the generated random number to determine the selected action. The calculation of the cumulative probability ensures that the previous probability sum is reflected in the action selection, thereby realizing the random selection of actions. Random action selection includes randomly selecting and clicking a control from the UI control list, performing a swipe operation, or a back button operation. The shortest path selection mode is based on the UI tree and edge information table in the UI tree construction and management module. It uses the shortest path algorithm to calculate the optimal UI operation path from the current page state to the target page state and selects the corresponding optimal action sequence. The nodes of the UI tree represent the page state, and the edges represent a page state transition event. S4. The action execution and information recording module operates the device interface according to the action determined by the action selection module, including click, swipe and return operations. After each operation, the state change of the page is obtained and recorded in the edge information table, and the UI tree built in the UI tree construction and management module is updated. S5. The traffic capture module captures the network traffic of the application throughout its operation and saves the traffic data as a PCAP file; the traffic capture module uses... or The underlying library can accurately intercept data packets passing through network interfaces. These data packets contain rich network information, including source IP, destination IP, port number, protocol type, and transmitted data content. The traffic capture module also implements a filtering mechanism, allowing users to set up the capture of specific types of data packets.

2. The method for automating user interface operations in an app according to claim 1, characterized in that: The UI control acquisition module uses an automated testing library. ,pass To find all clickable controls on the current page.

3. The method for automating user interface operations in an app according to claim 2, characterized in that: The UI control acquisition module extracts control information from each control object and constructs it into a unique key. During the control information extraction process, each control... By unique key To identify, control information It is represented as follows: in, Indicates resource information, Indicates class name information, Represents text content information. This indicates descriptive information.

4. The method for automating user interface operations in an app according to claim 3, characterized in that: After acquiring and storing all the necessary information, the UI control acquisition module generates a control list according to the control's attributes. At the same time, irrelevant package name controls are filtered out. When generating the control list for the current page, each control... Control list record Described as a triple, as shown below: in Indicates the center position of the control. Represents a control object. This represents the control's identification information.

5. A method for automating user interface operations in an app according to claim 4, characterized in that: The UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module. This unique key includes an identifier constructed from the identifiers of all controls on the page. Total The identifiers for this page are as follows: in, Represents the page, Indicates the page A control.

6. A method for automating user interface operations in an app according to claim 5, characterized in that: The formula for calculating the page state similarity is as follows: in, This represents the size of the intersection of the control collections on the new page and the old page. This represents the size of the union of the control sets on the new page and the old page.

7. A method for automating user interface operations in an app according to claim 6, characterized in that: The UI tree construction and management module determines whether the current page already exists in the UI tree based on the calculated page state similarity. If it does not exist, it adds the page as a new node to the UI tree and initializes the node. If the page exists, it updates the information of the existing node, calculates the similarity between the new page state and each existing page state, and selects the page state with the highest similarity. If the maximum similarity is lower than a preset threshold, it indicates that the page is a new page, and a new node is added to the UI tree. If the maximum similarity is higher than or equal to the preset threshold, the information of the existing node is updated.

8. A method for automating user interface operations in an app according to claim 7, characterized in that: The probability table is constructed as follows: in, This represents the constructed probability table. This indicates the probability of a click operation. This indicates the probability of a return operation. This represents the probability of sliding backward. Indicates sliding forward. This represents the probability of sliding to the left. This represents the probability of a rightward swipe.

9. A method for automating user interface operations in an app according to claim 8, characterized in that: The traffic capture module captures traffic while the action is being executed, specifically including the following process: During the application's execution activities, the entire process uses Command starts on device This tool captures network traffic data from a specified application and saves it as... The file will be saved after the traffic capture is complete. Files are copied from the device to local storage, and temporary files on the device are cleaned up.

10. A method for automating user interface operations in an app according to claim 1, characterized in that, The environmental monitoring module specifically includes the following sub-modules: The device volume status monitoring module continuously monitors the device's volume status through a separate thread. Within this thread, it periodically checks the device's volume status; if the volume is not muted, it automatically mutes the volume. Commands can be used to get and set the device volume; The device connection status monitoring module, through calling... Commands are used to detect and connect devices, ensuring that the devices are... It runs in a specific state and handles and reports when connection errors or timeouts occur; The screen status management module, through execution The command is used to detect the current device window status. If an error message window, unresponsive window, status bar, or other non-target application windows are detected, the program will force close these windows and return to the target application interface, while hiding the soft keyboard by sending specific key events.

Images