Method for automatic user interface operation of app

Abstract

The invention relates to the technical field of automatic testing and automatic operation of mobile applications, and discloses a method for automatic user interface operation of an app. According to the method, all clickable controls on a current page and related information of the clickable controls are obtained through a UI control obtaining module, meanwhile, the real-time change state of the page is monitored, a UI tree is constructed and updated by utilizing a UI tree construction and management module according to the obtained information, different page states and conversion paths of the different page states are recorded, and through an action selection module, the user interface of the current page is selected. According to an operation mode selected by a user, executing random action selection or shortest path selection, determining an optimal operation path, executing selected action operation through an action execution and information recording module, and recording page state change and path conversion information before and after the action; the traffic capturing module is responsible for capturing network traffic of the application program in the whole operation process and storing traffic data as a PCAP file, and the environment monitoring module is responsible for monitoring normal operation of the program, so that the quality of the application program is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automated testing and automated operation of mobile applications, and in particular to a method for automated user interface operation of an app. Background Art

[0002] With the rapid development and popularization of mobile applications, the requirements for application quality and stability are also constantly increasing. Although traditional manual testing methods are intuitive, they are inefficient and difficult to cope with large-scale application testing. In addition, manual testing lacks the true randomness of actions when simulating user operations and cannot accurately reproduce a variety of user behaviors.

[0003] Automated testing has therefore become an important solution. However, existing automated testing methods still have shortcomings in the stability and reliability of user interface operations. Especially when the device and environment change significantly, the accuracy and consistency of UI operations are difficult to guarantee. These automated testing tools often lack the ability to implement the actual operation path for a specific target page, which leads to uncertainty in the operation path, wastes precious resources and time, and the operation results may not be consistent with expectations.

[0004] Therefore, there is an urgent need for a method that can achieve stable and reliable UI automation operation under different equipment and environmental conditions. This method must not only ensure the stability and consistency of the operation, but also have the ability to intelligently analyze and randomly process user interactions, so as to more realistically simulate user operations, avoid resource waste, and accurately achieve test goals. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a method for automated user interface operation of an app, aiming to achieve efficient and comprehensive traversal and analysis of the mobile application interface by combining random action selection and the shortest path algorithm. The system builds and maintains a UI tree, records different page states and the conversion paths between them, and utilizes a variety of operation modes, such as random action selection and action selection based on the shortest path, to ensure comprehensive coverage of various interface states and interactive operations of the application. The probability of action selection is defined by a probability table. The system supports a variety of interactive actions, including clicking, sliding, etc. At the same time, the system records and analyzes the success rate and probability distribution of the conversion path for each step of the operation, continuously expands and optimizes the UI tree, and ensures that all possible UI states are fully tested and traversed, thereby solving the above-mentioned problems.

[0006] (II) Technical solution To achieve the above object, the present invention provides the following technical solution: a method for automating user interface operation of an app, comprising the following steps: S1. Obtain all clickable controls on the current app page through the uiautomator2 library in the UI control acquisition module, and obtain relevant information of these controls, including resource ID, class name, text content, description, and boundary information; S2. Generate a unique key for the current page state based on the control information obtained by the UI control acquisition module through the UI tree construction and management module, and update the UI tree structure by calculating the page state similarity and the real-time page change status, record the number of page visits, and manage node information, gradually building the entire UI tree; S3. Randomly select actions such as click, slide, and return based on a predefined probability table through the random action selection mode in the action selection module, map the actions to different operations, and calculate the optimal UI operation path from the current page state to the target page state using the shortest path algorithm based on the UI tree and edge information table in the UI tree construction and management module through the shortest path selection mode in the action selection module, and select the corresponding optimal action sequence; S4. The action execution and information recording module operates the device interface according to the action determined by the action selection module, including click, slide and return operations. After each operation, the state change of the page is obtained and recorded in the edge information table, and the UI tree constructed in the UI tree construction and management module is updated; S5. Capture the network traffic of the application during the entire operation process through the traffic capture module, and save the traffic data as a PCAP file.

[0007] Preferably, the UI control acquisition module uses an automated testing library ,pass To find all clickable controls on the current page.

[0008] Preferably, the UI control acquisition module extracts control information from each control object and constructs it into a unique key. By unique key To identify, control information The representation is as follows:

[0009] in, Represents resource information. Indicates class name information. Represents text content information. Indicates description information.

[0010] Preferably, after acquiring and storing all required information, the UI control acquisition module generates a control list according to the properties of the control. , and filter out irrelevant package name controls. When generating the control list of the current page, each control Control list record Described as a triple as follows:

[0011] in Indicates the center position of the control. Represents a control object. Represents control identification information.

[0012] Preferably, the UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module, and the unique key includes an identifier constructed by identifying all controls on the page. Total The page identifier is as follows:

[0013] in, Indicates the page, Indicates the page controls.

[0014] Preferably, the page status similarity calculation formula is as follows:

[0015] in, Indicates the intersection size of the new page and the old page control collection. Represents the union size of the new and old page control collections.

[0016] Preferably, the UI tree construction and management module determines whether the current page already exists in the UI tree based on the calculated page status similarity. If not, it is added to the UI tree as a new node and the node is initialized. If it exists, the information of the existing node is updated, the similarity between the new page status and each existing page status is calculated, and the page status with the greatest similarity is selected. If the maximum similarity is lower than a preset threshold, it indicates that the page is a new page, and a new node is added to the UI tree. If the maximum similarity is higher than or equal to the preset threshold, the information of the existing node is updated.

[0017] Preferably, the probability table is constructed as follows:

[0018] in, represents the constructed probability table, represents the probability of a click operation, represents the probability of returning an action, represents the probability of sliding backwards, Indicates sliding forward, represents the probability of sliding to the left, Indicates the probability of a right swipe operation.

[0019] Preferably, the traffic capturing module captures traffic while the action is being executed, specifically including the following process: When the application is performing activities according to the above rules, the whole process uses Command starts on the device Tool to capture network traffic data of specified applications and save it as After the traffic capture is completed, the saved The file is copied from the device to the local computer and the temporary files on the device are cleared.

[0020] A method for automating user interface operation of an app, characterized in that it also includes a device and environment monitoring module, wherein the device environment monitoring module specifically includes the following submodules: The device volume status monitoring module continuously monitors the volume status of the device by creating an independent thread. In the independent thread, the volume status of the device is checked regularly. If it is detected that the volume of the device is not in the muted state, the volume is automatically set to mute. Commands to get and set device volume; Device connection status monitoring module, by calling command to detect and connect to the device, ensuring that the device is Run in the same state, and handle and report when connection errors or timeouts occur; The screen status management module executes Command to detect the current device window status. If an error prompt window, unresponsive window or status bar, or other non-target application windows are detected, the program will forcefully close these windows and return to the target application interface, while hiding the soft keyboard by sending specific key events.

[0021] Compared with the prior art, the present invention provides a method for automating user interface operation of an app, which has the following beneficial effects: The present invention combines UI control acquisition, UI tree construction, random action selection and intelligent path optimization. The method can realize random simulation of real user behavior under different equipment and environmental conditions, overcome the limitations of traditional manual testing and existing automated testing. Specifically, the uiautomator2 library is used to obtain the clickable control information of the current page, dynamically build and manage the UI tree, ensure the consistency and stability of the operation, optimize the operation path by randomly selecting user interaction actions and the shortest path algorithm, reduce resource waste and test time, and the traffic capture module can monitor the network traffic of the application in real time to provide data support for subsequent analysis, while the device and environment monitoring module ensures the consistency of the test environment to prevent external factors from affecting the test results. Overall, the present invention not only improves the intelligence level of automated testing, but also effectively improves the quality of the application and user experience, meeting the increasingly stringent market demands. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the steps of the method of the present invention; Figure 2 It is a diagram showing the overall module structure of the present invention and the relationship between the modules; Figure 3 This is an operation flow chart of the random action selection mode in the present invention; Figure 4 This is an operational flow chart of the shortest path selection action mode in the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 ,A method for automated user interface operation of app includes five modules, namely: UI control acquisition module, UI tree building and management module, action selection module, action execution and information recording module, and device traffic capture module; It also includes an environmental monitoring module for monitoring the operation of the above five modules; UI control acquisition module: use The library obtains all clickable controls on the current application page and collects relevant information, including resource ID, class name, text content, description, and boundary information; UI tree construction and management module: Based on the control information obtained by the UI control acquisition module, a unique key for the current page state is generated. By calculating the similarity and real-time changes of the page states, the UI tree structure is updated, the number of page visits is recorded, node information is managed, and a complete UI tree is gradually constructed. Action selection module: includes two modes: random action selection and shortest path selection. The former randomly selects actions based on predefined probabilities, while the latter uses the shortest path algorithm to calculate and execute the optimal action path from the current page state to the target page state. Action execution and information recording module: Operate the device interface according to the action determined by the action selection module, including clicking, sliding, returning, etc. After each operation, obtain the state change of the page and record it in the edge information table, and update the UI tree constructed by the UI tree construction and management module; Traffic capture module: This module is responsible for capturing the network traffic of the application during the entire operation process and saving the traffic data as Files for subsequent analysis, specifically including two main functions: traffic capture and traffic collection; Device and environment monitoring module: including device volume status monitoring, device connection status monitoring and screen status management. Device volume monitoring continuously monitors the volume through an independent thread to ensure that the test is not disturbed. Device connection monitoring ensures that the device is Run in the state and handle connection problems, and screen management closes non-target application windows and hides the soft keyboard; The following describes the operation process of each module: The UI control acquisition module obtains all clickable controls on the current application page and collects relevant information, including: UI control acquisition module uses automated testing library ,pass To find all clickable controls on the current page, this method implements getting all clickable controls on the current page. Controls with properties, which can then be further manipulated and analyzed; Collecting and storing control information: The UI control acquisition module further includes extracting control information from each control object and building it into a unique key; The process of extracting control information ensures that each control By unique key To identify, its information includes resources , class name , text content And description , all control information It can be expressed as:

[0025] Build a control list and filter irrelevant controls: After the UI control acquisition module acquires and stores all required information, it generates a control list according to the properties of the control. , while filtering out irrelevant package name controls, such as system Controls, when generating the control list of the current page, each control Control list record Can be described as a triple as follows:

[0026] in Indicates the center position of the control. Represents a control object. Represents control identification information; This code ensures that only clickable controls belonging to a specific application are retained, and system-level or irrelevant application controls are excluded. At the same time, a unique key is generated for each control and stored in the view table. Maintain the list of controls For subsequent use; The UI tree construction and management module constructs and updates the UI tree structure, including: Generate a unique key for the current page state: The UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module. The unique key includes an identifier constructed by the identifiers of all controls on the page to ensure that the current page state can be uniquely identified. If there is a total controls, then the identifier of the page can be expressed as:

[0027] Calculate page status similarity: The UI tree construction and management module further includes calculating the similarity of page status. Specifically, the current page state With existing pages state Compare and calculate the similarity between them. Use set operation to compare the control information. The similarity calculation formula is as follows:

[0028] in, Indicates the intersection size of the new page and the old page control collection. Represents the union size of the new and old page control collections.

[0029] Update the UI tree structure: Based on the calculated similarity, the UI tree construction and management module determines whether the current page already exists in the UI tree. If not, it is added to the UI tree as a new node and the node is initialized. If it exists, the information of the existing node is updated. The specific implementation includes: Calculate the similarity between the new page state and each existing page state, and select the page state with the greatest similarity. If the maximum similarity is lower than the preset threshold, it indicates that the page is a new page, and a new node is added to the UI tree. If the maximum similarity is higher than or equal to the preset threshold, update the information of the existing node. Record page visit times: The UI tree construction and management module maintains a page visit count record, records the number of times each page state is visited, and uses this record to optimize the structure of the UI tree; Management node information: The UI tree construction and management module further includes management node information, recording detailed information of each page node, including page status, parent-child relationship of nodes, etc., to ensure that the UI tree can accurately represent the conversion relationship between pages, and gradually build and improve the entire UI tree structure through subsequent operation activities and page status changes; The core modules of this method are the action selection module and the action execution and information recording module. The action selection module includes two modes: random action selection and shortest path selection. The random action selection randomly selects operations based on a predefined probability table, while the shortest path selection uses the shortest path algorithm to calculate and execute the optimal operation path from the current page state to the target page state. Since the action execution and information recording module participates in the operation and information recording of these two action selection modes, its content is combined with the action selection module for introduction; Random action selection mode such as Figure 3 As shown, the random action selection mode in the action selection module selects actions based on a predefined probability table. , defines the selection probability of each action, where represents the probability of a click operation, represents the probability of returning an action, represents the probability of sliding backwards, Indicates sliding forward, represents the probability of sliding to the left, Indicates the probability of swiping right, etc.

[0030] Generate a cumulative probability according to the probability table, and select an action using a random number. The step generates a cumulative probability by calculating the cumulative sum of the probability table, and determines the selected action using the generated random number. The calculation of the cumulative probability ensures that the previous probability and value can be reflected in the action selection, thereby realizing random selection of actions, selecting and executing corresponding actions, and the random selection includes randomly selecting and clicking controls from the UI control list, performing sliding operations or return key operations, and specific actions include but are not limited to clicking random controls, sliding back, sliding forward, sliding left, sliding right, and returning operations, re-acquiring all clickable controls and related information of the current page to detect and record page state changes, which includes obtaining control information of the new page and recalculating the page state, and updating the UI tree based on this information; The shortest path selection mode is as follows Figure 4 As shown, by inputting the current page state and the target page state, based on the UI tree composed of the conversion paths between the previously calculated page states, the shortest path The algorithm calculates the path from the current page state to the target page state, where the node of the UI tree represents the state of the page, and the edge represents a certain page state transition event (such as click, slide, etc.). After obtaining the best operation sequence through the shortest path, the change of the page state is checked according to the executed action to ensure that the new changes of the page state after each operation are accurately recorded, and the path information of the page state change is recorded and stored in the edge information table. Including but not limited to the number of executions of this action, conversion success rate and probability distribution; The traffic capture module is an important part of the network monitoring and analysis system. It is responsible for capturing network traffic in real time during the entire operation process of the application. The module adopts efficient network packet capture technology and uses or The underlying libraries such as , can accurately intercept the data packets passing through the network interface. These data packets contain rich network information, including source IP, destination IP, port number, protocol type, and transmitted data content. During the process of traffic capture, the module can be flexibly configured to support the capture requirements of various network environments, such as LAN, WAN and virtual network. In order to ensure the comprehensiveness and accuracy of the capture, the module will also implement a filtering mechanism, allowing users to set the capture of specific types of data packets, thereby reducing unnecessary data interference and improving the efficiency of data analysis; The traffic collection function is responsible for saving the captured data traffic as ( ) file format, which is widely used in network analysis tools, such as In order to facilitate the subsequent in-depth analysis and troubleshooting, during the flow collection process, the module can store the data in segments to ensure that the storage will not overflow due to excessive data volume in the case of long-term capture. At the same time, it also supports compression and encryption of the captured traffic to ensure the security and integrity of the data. The device and environment monitoring module includes three parts: device volume status monitoring, device connection status monitoring and screen status management. The specific parts are as follows: Device volume status monitoring: By creating an independent thread, the volume status of the device is continuously monitored to ensure that the test process is not disturbed by the outside world. Specifically, in the independent thread, the volume status of the device is checked regularly. If the volume of the device is not muted, the volume is automatically set to mute. Commands to get and set device volume; Device connection status monitoring: by calling command to detect and connect to the device, ensuring that the device is Run in the same state, and handle and report connection errors or timeouts; Screen state management: By executing Command is used to detect the current device window status. If an error prompt window, an unresponsive window or status bar, or other windows of non-target applications are detected, the program will forcefully close these windows and return to the target application interface. At the same time, the soft keyboard is hidden by sending specific key events to ensure the normal operation of the UI.

[0031] Through the relevant modules of the above method, random simulation of real user behavior can be achieved under different equipment and environmental conditions, overcoming the limitations of traditional manual testing and existing automated testing. At the same time, by randomly selecting user interaction actions and the shortest path algorithm, the operation path is optimized, reducing resource waste and testing time.

[0032] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automating user interface operation of an app, characterized in that: The following steps are involved: S1. Obtain all clickable controls on the current app page through the uiautomator2 library in the UI control acquisition module, and obtain relevant information of these controls, including resource ID, class name, text content, description, and boundary information; S2. Generate a unique key for the current page state based on the control information obtained by the UI control acquisition module through the UI tree construction and management module, and update the UI tree structure by calculating the page state similarity and the real-time page change status, record the number of page visits, and manage node information, gradually building the entire UI tree; S3. Randomly select actions such as click, slide, and return based on a predefined probability table through the random action selection mode in the action selection module, map the actions to different operations, and calculate the optimal UI operation path from the current page state to the target page state using the shortest path algorithm based on the UI tree and edge information table in the UI tree construction and management module through the shortest path selection mode in the action selection module, and select the corresponding optimal action sequence; S4. The action execution and information recording module operates the device interface according to the action determined by the action selection module, including click, slide and return operations. After each operation, the state change of the page is obtained and recorded in the edge information table, and the UI tree constructed in the UI tree construction and management module is updated; S5. Capture the network traffic of the application during the entire operation process through the traffic capture module, and save the traffic data as a PCAP file.

2. A method for automating user interface operation of an app according to claim 1, characterized in that: The UI control acquisition module uses the automated testing library ,pass To find all clickable controls on the current page.

3. A method for automating user interface operation of an app according to claim 2, characterized in that: The UI control acquisition module extracts control information from each control object and constructs it into a unique key. By unique key To identify, control information The representation is as follows: in, Represents resource information. Indicates class name information. Represents text content information. Indicates description information.

4. A method for automating user interface operation of an app according to claim 3, characterized in that: After acquiring and storing all the required information, the UI control acquisition module generates a control list according to the properties of the control. , and filter out irrelevant package name controls. When generating the control list of the current page, each control Control list record Described as a triple as follows: in Indicates the center position of the control. Represents a control object. Represents control identification information.

5. A method for automating user interface operation of an app according to claim 4, characterized in that: The UI tree construction and management module generates a unique key for the current page state based on the control information obtained by the UI control acquisition module. The unique key includes an identifier constructed by the identification of all controls on the page. Total The page identifier is as follows: in, Represents a page, Indicates the page controls.

6. A method for automating user interface operation of an app according to claim 5, characterized in that: The page status similarity calculation formula is as follows: in, Indicates the intersection size of the new page and the old page control collection. Represents the union size of the new and old page control collections.

7. A method for automating user interface operation of an app according to claim 6, characterized in that: The UI tree construction and management module determines whether the current page already exists in the UI tree based on the calculated page state similarity. If not, it is added to the UI tree as a new node and the node is initialized. If it exists, the information of the existing node is updated, the similarity between the new page state and each existing page state is calculated, and the page state with the greatest similarity is selected. If the maximum similarity is lower than a preset threshold, it indicates that the page is a new page, and a new node is added to the UI tree. If the maximum similarity is higher than or equal to the preset threshold, the information of the existing node is updated.

8. A method for automating user interface operation of an app according to claim 7, characterized in that: The probability table is constructed as follows: in, represents the constructed probability table, represents the probability of a click operation, represents the probability of returning an action, represents the probability of sliding backwards, Indicates sliding forward, represents the probability of sliding to the left, Indicates the probability of a right swipe operation.

9. A method for automating user interface operation of an app according to claim 8, characterized in that: The traffic capture module captures traffic while the action is being executed, specifically including the following process: When the application is performing activities according to the above rules, the whole process uses Command starts on the device Tool to capture network traffic data of specified applications and save it as After the traffic capture is completed, the saved The file is copied from the device to the local computer and the temporary files on the device are cleared.

10. A method for automating user interface operation of an app, characterized in that: It also includes an equipment and environment monitoring module, which specifically includes the following submodules: The device volume status monitoring module continuously monitors the volume status of the device by creating an independent thread. In the independent thread, the volume status of the device is checked regularly. If it is detected that the volume of the device is not in the muted state, the volume is automatically set to mute. Commands to get and set device volume; Device connection status monitoring module, by calling command to detect and connect to the device, ensuring that the device is Run in the same state, and handle and report when connection errors or timeouts occur; The screen status management module executes Command to detect the current device window status. If an error prompt window, unresponsive window or status bar, or other non-target application windows are detected, the program will forcefully close these windows and return to the target application interface, while hiding the soft keyboard by sending specific key events.

Images