Android application testing method based on monte carlo tree search and deep reinforcement learning
By employing fine-grained state representation and Monte Carlo tree search to optimize testing strategies in Android application testing, the problem of exploration and exploitation imbalance in existing technologies is solved, achieving efficient Android application testing and high code coverage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2022-11-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing Android application testing methods fail to strike a good balance in the exploration and utilization process. State representations are too coarse-grained, resulting in low testing efficiency and difficulty in achieving high code coverage.
We employ a Monte Carlo tree search and deep reinforcement learning approach to design fine-grained state representations and novel reward functions. By combining Monte Carlo tree search with optimized testing strategies, we achieve adaptive learning and efficient state exploration for Android applications.
It achieves high code coverage testing for Android applications, avoids local optima, and improves testing efficiency and accuracy.
Smart Images
Figure CN115729828B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of software testing technology, specifically relating to an Android application testing method based on Monte Carlo tree search and deep reinforcement learning. Background Technology
[0002] With the widespread adoption of smartphones and tablets, the mobile application market has experienced explosive growth. As the most popular mobile operating system, Android holds a 70% share of the smartphone operating system market. The Google Play Store is the most popular application market on the Android platform, currently boasting over 1 million applications. Ensuring these applications function correctly to support a good user experience is crucial. Effective Android application testing techniques are key to achieving this goal.
[0003] Input generation tools based on random strategies use randomized policies to generate input for Android applications. In their simplest form, randomization only generates user interface events. Because there are so many system events, random generation is very inefficient, and applications typically only respond to a few of them and only under specific conditions. Many tools in this category aim to test inter-application communication by randomly generating event values. These tools are also effective in revealing security vulnerabilities, such as denial-of-service vulnerabilities.
[0004] Model-based Android testing tools build GUI (Graphical User Interface) models of applications to generate events and systematically explore application behavior. These models are typically finite state machines, where activities are states and events are transitions. Some tools build more precise models by distinguishing the states of active elements when representing states; for example, the same activity, such as enabling and disabling buttons, would be represented as two separate states. Most tools build such models dynamically and terminate after all events that can be triggered from the discovered states have been completed.
[0005] Reinforcement learning-based strategies have been widely applied to GUI and Android testing problems. However, in most Android application testing frameworks, only the most basic tabular reinforcement learning form is used. In tabular reinforcement learning, the state-action association values are stored in a fixed table, thus it cannot be well extended to high-dimensional problems. The emergence of deep neural networks has replaced the tabular approach with deep learning methods. The behavior value function of deep reinforcement learning is learned by one or more neural networks from past positive and negative experiences. When the state space to be explored is very large, that is, when an application has a large number of controls, deep reinforcement learning has proven to be superior to tabular reinforcement learning. However, existing deep reinforcement learning-based strategies do not strike a good balance between exploration and utilization during testing, and the representation of the application environment state and the setting of the reward function are too coarse-grained, failing to accurately reflect the state changes of the application under test. Summary of the Invention
[0006] To address the issues of poor balance between exploration and utilization during the testing process and overly coarse-grained representation of the application environment in the aforementioned technologies, this invention proposes an Android application testing method based on Monte Carlo tree search and deep reinforcement learning. During the exploration process, reinforcement learning algorithms are used to adaptively learn the exploration strategy, which can continuously reach some Android application states that were previously difficult to traverse, thereby achieving high code coverage in Android application testing.
[0007] The Android application testing method based on Monte Carlo tree search and deep reinforcement learning provided by this invention includes:
[0008] Step 1: Detect and launch the Android device, install and launch the Android application under test, and install the automated testing tools;
[0009] Step 2: Read the current state of the Android application under test through the application environment interface and obtain the XML file of the Android application interface. The XML file contains the attributes, positions and text information of all controls under the current interface of the Android application.
[0010] The Application Environment Interface (AUI) acts as the hub between the application under test (API) and the test program, handling their interaction. The AUI comprises two classes: the first is an initialization class, which includes a dictionary as an additional parameter containing the settings of the application to be tested and the Android device or emulator. The second class contains an action function that accepts an Android test action command and returns a list of objects, including the current application state and a reward value. The current application state is generated by a combination of fine-grained state representations of all controls on the current screen, and the reward value is calculated by the new reward function based on the state changes caused by the execution of the Android test action command.
[0011] Step 3: Map the control types, positions, and text information in the XML file to obtain fine-grained state representations;
[0012] A pre-designed mapping dictionary of fine-grained states for the controls of the application under test is used. Based on the control attributes, positions, and text information in the XML file, the fine-grained state representation of each control is obtained. The combined encoding of the fine-grained state representations of all controls in the current interface of the application under test is used as the state of the current application. Different fine-grained state representations are set according to whether the control is set in the current graphical user interface.
[0013] The fine-grained state representation of controls is represented using 01 encoding. The combined encoding of the fine-grained state representations of all controls on the current screen generates a unique encoding of the current application's state. The application's state representation contains the interface information of the Android application being tested.
[0014] Step 4: Start the deep reinforcement learning model. The application environment interface reads the current state of the application under test and inputs it into the deep reinforcement learning model. The deep reinforcement learning model makes a decision, outputs a test strategy, and returns it to the application environment interface. The test strategy contains the control operations that need to be performed.
[0015] Step 5: The application environment interface sends the test strategy to the test program. The test program converts the test strategy of the deep reinforcement learning model into operation instructions for the control of the application under test, and performs corresponding test operations on the application under test through the environment interface.
[0016] Step 6: After executing the test strategy, the Android application collects the environment status of the Android application. If a fault occurs, the fault information is collected, and the reward value is calculated based on the changes in the state of the tested application.
[0017] The reward value is calculated using a new reward function. This new function calculates the reward value based on the state of the application under test and whether new faults are discovered during the test. It encourages the discovery of new states or faults in the application and penalizes actions such as leaving the application or repeatedly exploring and navigating within the application without discovering new states or faults. This invention employs a finer-grained state representation, allowing the reward function to return rewards for minor changes in the position and text of controls in the GUI state.
[0018] Step 7: The deep reinforcement learning model optimizes the testing strategy based on the received rewards, and then returns to Step 2 for iterative testing and optimization until the testing time or testing step limit is reached.
[0019] The transition relationships of the interface states in the Android application under test are modeled as a directed graph. The nodes in the graph are the states of the application, that is, the encoding vectors of the fine-grained state representations of all controls on different interfaces. Each node stores a list of operable controls, and each operable control in the list points to the next node to jump to by operating that control.
[0020] The Monte Carlo Tree Search (MCS) method is employed to optimize the testing strategy. During the exploration process, the MCS records the reward value obtained after jumping to the selected action control between nodes and the number of times the action control is selected. The UCB (Upper Bound of Confidence) score is used to calculate the probability of each actionable control being selected. In this way, the MCS achieves a balance between exploration and exploitation, helping the reinforcement learning model to continuously improve and optimize its exploration strategy.
[0021] Compared with the prior art, the advantages and positive effects of the present invention are as follows:
[0022] (1) The method of the present invention designs a fine-grained state representation and a new reward function for the state search process of Android application testing. It can not only reward the state changes caused by interface jumps, but also reward the fine-grained control positions and text changes in the interface. Through the reward function under the fine-grained state representation, the benefits and costs obtained by executing the current strategy of the reinforcement learning algorithm can be more accurately quantified. The new reward function makes the exploration strategy learned during the testing process more comprehensive and detailed.
[0023] (2) The method of the present invention uses the Monte Carlo tree search method to search for states of interest to Android applications, which provides an opportunity for long-term exploration of potential states and avoids local optima;
[0024] (3) The method of the present invention uses reinforcement learning algorithm to adaptively learn the exploration strategy during the Android application testing process. During the exploration process, based on past experience, it can continuously reach some Android application states that were previously difficult to traverse, thereby achieving high code coverage Android application testing. Attached Figure Description
[0025] Figure 1 This is a flowchart illustrating the Android application testing method of the present invention;
[0026] Figure 2 This is a schematic diagram illustrating the increase in code coverage over time during the testing process of this invention. Detailed Implementation
[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0028] This invention presents an Android application testing method based on Monte Carlo tree search and deep reinforcement learning. During the exploration process, a deep reinforcement learning algorithm is used to adaptively learn the exploration strategy. Based on past experience, it can continuously reach Android application states that were previously difficult to traverse, thereby achieving high code coverage in Android application testing. This invention designs a new reward function for the state search process in Android application testing. This function not only rewards state changes caused by interface transitions but also rewards fine-grained changes in the position of controls and text within the interface. The new reward function makes the exploration strategy learned during testing more comprehensive and detailed. This invention uses Monte Carlo tree search to search for states of interest to the Android application during testing, providing opportunities for long-term exploration of potentially valuable states and avoiding local optima.
[0029] The core of this invention mainly includes: First, a new reward function is designed for the state search process in Android application testing. This function not only rewards state changes caused by interface transitions but also rewards fine-grained changes in control positions and text within the interface. This new reward function makes the exploration strategies learned during testing more comprehensive and detailed. Second, a Monte Carlo tree search method is used to search for states of interest to the Android application, providing opportunities for long-term exploration of potentially valuable states and avoiding local optima. Figure 1 As shown, the implementation of this embodiment of the invention includes the following 7 steps.
[0030] Step 1: Detect and start the Android device, and install and start the Android application under test.
[0031] Depending on the input parameters, a pre-configured Android emulator can be launched or a real Android device can be directly connected as the runtime environment for the program under test. After the Android device starts up and runs normally, the Android application under test is installed and launched on the device, and the automated testing tool Appium is installed for subsequent interaction testing with the Android application under test.
[0032] The application runtime environment provides a suitable environment for the test application to install and launch successfully. This runtime environment is either a real Android device or an Android emulator configured on the host machine. It is responsible for installing the application under test and ensuring its successful execution. The application environment interface acts as the hub between the application under test and the test algorithm, handling the interaction between them.
[0033] Step 2: The application environment interface reads the current state of the Android application under test. The Android application interface automation testing tool UIAutomator parses and obtains the XML file of the Android application interface. The XML file contains the attributes of all controls in the current Android application interface and their position information.
[0034] The Application Environment Interface (AUI) serves as the hub between the application under test (API) and the testing algorithm, handling the interaction between them. Following the guidance of the Gym emulation interface, the AUI is constructed as a class with two key functionalities. The first class is an initialization class, whose additional parameters include a dictionary containing the settings of the application to be tested and the Android device or emulator. The second class contains an action function that accepts an Android test action command and returns a list of objects, including the current application state and a reward value. The current application state is obtained by fine-grained representation of the controls in the XML file, as explained in step three; the reward value is calculated by a newly designed reward function based on the state changes caused by the Android test action command.
[0035] UIAutomator is used to parse the current interface of an Android application to obtain an XML file. The XML file includes the attributes, positions, and text information of all controls on the current interface.
[0036] Step 3: Map the control types, positions, and text information from the XML file to obtain fine-grained state representations. This embodiment of the invention pre-sets a mapping dictionary storing the fine-grained state values of each control in the application under test. Based on the XML file and the mapping dictionary, the fine-grained state values of each control are obtained, thus determining the current application state. The mapping dictionary assigns an encoding to each control by mapping a key-value pair consisting of the control type, control position, and included text information. Different fine-grained state representations will be given depending on whether the control is present in the current graphical user interface (GUI).
[0037] The current application state is obtained based on the current interface state. The state of each interface is represented by the codes of all the controls it contains. Each control is mapped to a code by a key-value pair consisting of the control type, the text information it contains, and the control position. All the control codes under the current page form a unique code representation of the current interface state of the Android application.
[0038] In this embodiment of the invention, the fine-grained state representation is represented using 01 encoding. This state representation contains the interface information of the Android application being tested. The deep reinforcement learning model takes the fine-grained state representation as input and outputs the optimal test strategy selection under this fine-grained state representation.
[0039] In this embodiment of the invention, after the test program starts, Appium is used to analyze the hierarchy of GUI controls in the activity. The analyzed hierarchy includes clickable, long-clickable, and scrollable Android controls. These controls are then stored in a dictionary containing multiple related attributes (clickable, long-clickable, and scrollable controls), and a fine-grained state representation of each control is obtained from the mapping dictionary based on information such as control type, position, and text.
[0040] For example, the current Android application includes the following controls: login.activity, main.activity, widget[intput,(30,40),'user'], widget[button,(40,50),'login'], and widget[textView,(35,50),'content'], where (a_0, a_1) and (w_0, w_1, w_2) represent the following fine-grained states:
[0041] (a_0,a_1)=(login.activity,main.activity)
[0042] (w_0,w_1,w_2)=(widget[intput,(30,40),'user'],widget[button,(40,50),'login'],widget[textView,(35,50),'content'])
[0043] Assuming the current GUI is located in the application's login.activity, including widget[intput,(30,40),'user'] and widget[button,(40,50),'login'], then a_0, w_0, w_1 are set to 1, and a_1, w_2 are set to 0. The corresponding GUI state codes are as follows:
[0044] GUI state=(a_0,a_1,w_0,w_1,w_2)=(1,0,1,1,0).
[0045] In the example above, each control is encoded using either 0 or 1. If it's in the current GUI, the control's encoding is set to 1; otherwise, it's 0. In practical applications, other numbers can also be used. Furthermore, multi-digit combinations can be used to represent control encodings. For example, if a control is located in a different position or contains different text input, the encoding can be further refined based on the control's position and the text information. For each control, a fine-grained state representation is achieved by using different encodings depending on whether it's in the current GUI.
[0046] Step 4: Start the deep reinforcement learning model and output the test strategy based on the fine-grained state representation.
[0047] The deep reinforcement learning model is the central hub of the entire automated testing framework. It receives the application state from the application environment interface and returns the control operations that need to be performed to the application environment interface. Then, it receives the new application state and reward to update and optimize the model.
[0048] In the context of Android application automated testing, the input to the environment state of the deep reinforcement learning algorithm is the GUI combinatorial state of the test program, which is the application's state code. The output of the deep reinforcement learning algorithm is the test policy, which is the sequence of operations to be executed. The actions for automated testing of Android applications are represented as user interaction events in the Android test program mapped to an MDP (Markov Decision Process) set of operations. Here, each operable action is numbered and then fed into the deep reinforcement learning model. The output of the deep reinforcement learning algorithm is the test policy. After receiving the operation action numbers contained in the test policy output by the deep reinforcement learning model, the environment interface sends the corresponding numbered operation actions to the test program for execution. Subsequently, it collects the changes in the test program's environment state and returns the action rewards. The deep reinforcement learning model updates and optimizes the test policy based on the returned rewards.
[0049] Step 5: Convert the testing strategy of the deep reinforcement learning model into operation instructions for the control of the application under test, and perform corresponding test operations on the application under test through the environment interface.
[0050] During the testing of Android applications, modeling the transition relationships between interface states in the application can yield a model that more closely resembles the real Android application environment, which can then be used for Monte Carlo tree search.
[0051] Android application models are represented using a directed graph model. Nodes in the graph are state encoding vectors for different interfaces. Each node stores a list of operable controls, and each operable control within a node points to the next node that will be navigated to by activating that control. During exploration, the Monte Carlo Tree Search method records the reward value obtained from choosing actions to navigate between nodes and the number of times each action is selected. This embodiment uses the UCB (Upper Bound of Confidence) score to calculate the probability of each operable control being selected. In this way, the Monte Carlo Tree Search method achieves a balance between exploration and exploitation, helping reinforcement learning models continuously improve and optimize their exploration strategies.
[0052] During the testing process, Monte Carlo tree search is used to select test operations. The reward value obtained by jumping between nodes through selected actions is calculated based on the fine-grained state representation of the application under test and whether any new faults are found during the test.
[0053] The reward function incentivizes the discovery of new states. This invention employs a finer-grained state representation, allowing the reward function to return rewards for minor changes in the position of controls, text, etc., within the GUI state. If the test application is exited due to certain operations, the reward function returns a larger penalty. Other situations, such as repeated exploration and roaming within the application without discovering new states or faults, will result in a smaller penalty.
[0054] The novel reward function in this embodiment of the invention is as follows: A reward of R1 = 1000 is returned when a new fine-grained state of the Android application is discovered during testing or when a new fault occurs in the Android application; a reward of R2 = -100 is returned when the test jumps to an interface outside the tested application; and a reward of R3 = -1 is returned for other situations, such as reaching a previously reached state or the state remaining unchanged. The reward value Reward can be described as follows:
[0055]
[0056] Step Six: After performing the operations in Step Five, the Android application collects its status. If a fault occurs, the fault information is collected, the reward is calculated, and the Android application is reset. If no fault occurs, the reward value is calculated based on the changes in the Android application status.
[0057] The code coverage reward for the Android application under test can be approximated using a set of Android GUI states. A larger and more complete set of Android GUI states explored during testing is considered to indicate higher code coverage for the corresponding application. Therefore, the reward function does not use the actual application code coverage as the reward feedback. Instead, it records a set of Android application GUI states searched during testing. When a new Android application GUI state is discovered during testing, an appropriate reward is returned. The GUI state set stores a fine-grained representation of the previously defined Android application states. This fine-grained reward feedback encourages reinforcement learning to explore Android application states more comprehensively and meticulously. The reward function also records a fault detection set, which stores all faults discovered during Android application testing. An appropriate reward is only returned when a new Android application fault is discovered.
[0058] Step 7: The deep reinforcement learning model optimizes the testing strategy based on the received rewards, and then returns to Step 2 for iterative testing and optimization until the testing time or testing step limit is reached.
[0059] The deep reinforcement learning model uses the Monte Carlo tree search method described in step five to optimize the testing strategy.
[0060] This invention discloses an Android application testing method based on Monte Carlo tree search and deep reinforcement learning. The main modules involved include an Android application testing environment, an environment interface between the Android application and reinforcement learning, and a reinforcement learning algorithm. The Android application testing environment must ensure that the Android application can be installed, started, and run normally within the testing environment. During testing, it tracks the running status of the Android application under test to detect test faults. If the application encounters an exception, it needs to be restarted. The environment interface between the Android application and reinforcement learning is mainly responsible for handling the interaction between the application under test and the reinforcement learning algorithm. The environment interface is responsible for representing the current GUI state of the Android application, and this state representation is in a coded input format acceptable to reinforcement learning. Simultaneously, the environment interface also needs to convert the action strategy output by reinforcement learning into the control execution strategy of the Android application and pass it to the Android application testing module. The reinforcement learning algorithm is the core testing algorithm of the entire Android application automated testing tool, responsible for learning and optimizing the testing strategy.
[0061] This invention presents an Android application testing method based on Monte Carlo tree search and deep reinforcement learning. This method was tested on multiple open-source Android applications and compared with other existing advanced Android testing tools, with performance analysis provided. Figure 2 As shown, this illustrates the increase in code coverage over time during the testing process of this invention. Figure 2 The paper compares five testing schemes: MONKEY represents a random testing strategy; ARES is a deep reinforcement learning-based testing strategy; ARES_FINE adds the fine-grained state representation and a new reward function proposed in this invention to ARES; OAT_COARSE is a deep reinforcement learning testing strategy that only uses the Monte Carlo tree search strategy proposed in this invention; and OAT is a deep reinforcement learning testing tool proposed in this invention that simultaneously employs fine-grained state representation, a new reward function, and a Monte Carlo tree search strategy. The results analysis shows that the Android application testing method based on Monte Carlo tree search and deep reinforcement learning proposed in this invention can test Android applications more efficiently.
[0062] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, and / or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.
Claims
1. An Android application testing method based on Monte Carlo tree search and deep reinforcement learning, characterized in that, Includes the following steps: (1) Install automated testing tools in the runtime environment of the Android application under test, and obtain the XML file of the interface of the application under test. The XML file contains all control attributes, positions and text information of the current interface of the application under test. (2) Pre-design the mapping dictionary of control-fine-grained state of the application under test. Based on the control attributes, position and text information in the xml file, obtain the fine-grained state representation of each control. Use the combined encoding of the fine-grained state representation of all controls under the current interface of the application under test as the state of the current application. The mapping dictionary sets different fine-grained state representations for whether a control is in the current graphical user interface (GUI); (3) The current state of the application under test is read through the application environment interface and input into the deep reinforcement learning model. The deep reinforcement learning model makes a decision and outputs a test strategy and returns it to the application environment interface. The test strategy includes the control operations to be performed. The application environment interface sends the test strategy to the test program, controls the application under test to perform the corresponding control operations, collects the changes in the environment state of the application under test and calculates the reward value. (4) The reward value is calculated using a new reward function. The reward function calculates the reward value based on the state of the application under test and whether new faults are found during the test. It encourages the discovery of new states or new faults in the application. When the application jumps out of the application under test, or when repeated exploration and wandering are performed in the application without discovering new states or faults, a penalty is imposed. (5) The jump relationship of the interface state in the application under test is modeled as a directed graph, and the deep reinforcement learning model uses the Monte Carlo tree search method to optimize the test strategy; In the directed graph, the nodes represent the state of the application. Each node corresponds to a combination of fine-grained state representations of all controls on a single interface of the application. Each node stores a list of operable controls, and each operable control in the list points to the next node to jump to after performing the operation. During the testing process, the Monte Carlo tree search method was used to optimize the testing strategy. The reward value and the number of times the selected operation control was selected were recorded after the nodes were jumped between nodes during the exploration process. The probability of each operable control being selected was calculated using the upper bound of the confidence interval UCB score.
2. The method as described in claim 1, characterized in that, In step 1, the XML file of the application interface under test is obtained by parsing the Android application interface automated testing tool UIAutomator.
3. The method as described in claim 1, characterized in that, In step 1, the application environment interface constructs two classes. The first class is an initialization class, with additional parameters including a dictionary containing the settings of the application to be tested and the Android device or emulator. The second class contains an action function that accepts an Android test action command and returns a list of objects, including the current application state and reward value. The reward value is calculated by the new reward function based on the state changes caused by the execution of the Android test action command.
4. The method as described in claim 1, characterized in that, In step 2, the fine-grained state representation of the controls is represented by 01 encoding. The combined encoding of the fine-grained state representations of all controls on the current interface generates a unique encoding of the current application's state.
5. The method as described in claim 1 or 4, characterized in that, In step 4, one implementation of the new reward function is as follows: During the test, if a new state or a new fault is found in the application under test, a reward of R1 = 1000 is returned; if the test is redirected to an interface outside the application under test, a reward of R2 = -100 is returned; if repeated exploration and traversal are performed in the application and no new state or fault is found, a reward of R3 = -1 is returned.
6. The method as described in claim 1 or 4, characterized in that, In step 4, a GUI state set and a fault detection set are set. The GUI state set records the state set of the application under test searched during the test, and the fault detection set records all faults found during the test. The reward function determines whether new application states and new faults have been found based on the GUI state set and the fault detection set.