Automatic machine-based GUI (Graphical User Interface) test tool auxiliary diagnosis and debugging method, system and application

Abstract

The invention discloses an automatic machine-based GUI (Graphical User Interface) test tool auxiliary diagnosis and debugging method, which comprises the following steps of: abstractly constructing a non-deterministic automatic machine on the basis of a known program error in an android application, performing instrumentation on the corresponding android application, collecting a log output when an automatic graphic test tool tests the android application, and debugging the log in the automatic graphic test tool. The method comprises the following steps of: performing path matching on an Android application event execution sequence recorded in a log on a crash error automatic machine, and calculating three indexes including event coverage, event pair coverage and minimum distance by utilizing data recorded in a matching process; and judging whether the test tool finds a corresponding program error or not and generating command line output or a visual report. According to the method, the effectiveness of the Android automatic graphic test tool can be evaluated, an Android automatic graphic test tool developer is assisted to diagnose and debug the test tool, and defects of the tool are found and improved. The invention further discloses a system for implementing the method and application of the method or the system.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile platform automated testing, and relates to a method, system and application for auxiliary diagnosis and debugging of an Android automated graphics testing tool based on an automaton. Background Art

[0002] The rapid development of Internet technology and software and hardware technology has made modern human life intelligent, and smartphones have now become an indispensable part of daily life. Mobile applications have greatly enriched people's daily lives, making the security issues of mobile applications increasingly important. In order to avoid serious vulnerabilities and crashes in applications during the release cycle, automated graphics testing tools have replaced traditional pure manual testing and become an efficient and low-cost solution. However, due to the defects of automated testing tools themselves and various random factors in the operation process, the results presented by the testing tools will always have false positives and false negatives. Therefore, it is very important and urgent to build an effective graphics testing tool that can find defects in practice.

[0003] In order to explore and improve the ability of Android automated graphics testing tools to find program errors, evaluating the testing tools through a benchmark test set is a sufficiently reliable and widely used technical practice. To address this problem, a recent study proposed the Themis dataset, which consists of 52 real-world Android application crash errors. At the same time, the ability of the most advanced Android automated graphics testing tools to find program errors was evaluated, and it was found that the ability of existing tools is far from meeting the requirements of real scenarios. The quantitative analysis results in the study pointed out that several of the most advanced Android automated graphics testing tools can only find 29% to 47% of the program errors in the benchmark test set. In this case, the test tool developers will fall into the dilemma of "why the tool can't find this program error". If the developer can be given prompt information about this problem, it will help the test tool developers to diagnose and improve the tool, indirectly ensuring the quality and security of the application software as well as the user experience and data security.

[0004] However, there is no direct and effective method in academia and industry to solve the above problems. The Themis dataset based on real-world program crash errors proposed for Android automated graphics testing tools can better evaluate the ability of testing tools to find program errors, but it can only determine whether the tool has found or not found a program error. It cannot give prompts for program errors that the testing tool has not detected, and developers still need to manually analyze the reasons to infer the defects of the testing tool. For automated graphics testing tools with clear test exploration strategies, developers can perform missed detection analysis by following the tool's inherent strategy in most cases when there is a lack of prompt information. However, for Monkey-based graphical testing tools, since the tool generates random, long and very fast event sequences during the test process, tool developers will have no idea when analyzing missed detections. In addition, the developer's manual analysis is not only arbitrary and time-consuming, but also cannot be guaranteed to be effective, and the results are usually coarse-grained in terms of usefulness and feasibility. Summary of the invention

[0005] In order to solve the problem that the existing technology cannot effectively help Android automated graphics testing tool developers analyze the causes of missed program crash errors, the present invention proposes an automatic machine-assisted diagnosis and debugging method for automated graphics testing tools to assist Android automated graphics testing tool developers in debugging and improving their own tools.

[0006] The steps include:

[0007] Step 1: Identification of program crash errors and crash triggering paths: Given a crash error c in an Android application software P, record one or more different crash triggering path sets T in the Android application software P that can stably reproduce the program crash error c.

[0008] Step 2: Application source code instrumentation: According to the crash trigger path set T recorded in step 1, find the response function position corresponding to the event response graphical interface component in the source code of the Android application software P and perform instrumentation to obtain the instrumented Android application software P'.

[0009] The plugging operation is to insert a specific program statement, such as a standard output statement or an error output statement, into the program code. The inserted statement will print a specified string when the program is running. The plugging operation is used in the present invention mainly to obtain the log file output when the program is running, and to obtain the behavior track of the GUI test tool therefrom, which is then used for subsequent log analysis and evaluation index calculation.

[0010] Step 3: Crash error automaton construction: A crash error automaton a is constructed based on the crash trigger path set T in step 1 and the graphical user interface migration that occurs when the events in these paths are executed in the Android application software P.

[0011] Step 4: Logging: Use the Android automated graphics testing tool G that needs to be evaluated to test on the instrumented Android application software P' and record the instrumentation output o in the process.

[0012] Step 5: Log analysis: By giving the instrumentation output o recorded in step 4 as input to the crash error automaton a built in step 3, the automaton a will accept characters for state migration, and record the runtime information I of the Android automated graphics testing tool G based on the crash error automaton a during the migration process.

[0013] Step 6: Calculate evaluation indicators: Calculate the three evaluation indicators of event coverage EC, event pair coverage EPC and minimum distance MD based on the test tool runtime information I collected from the analysis log in step 5.

[0014] Step 7: Result output: Combine the test tool runtime information I in step 5 and the three evaluation indicators calculated in step 6 to generate a command line output or HTML visualization result report.

[0015] In step 1, in the step of identifying the program crash error and the crash trigger path, any crash trigger path t in the crash trigger path set T needs to meet the following requirements:

[0016] A1: The crash triggering path t is the sequence of events that can be executed in the Android application software P, that is, t = [e1, e2, …, e n ];

[0017] A2: Event e in the executable event sequence i Defined as a triple, namely e i =(t i , w i , o i ), where ti represents the type of event (click, longclick, etc.), and w i Represents the target graphical interface component (button, text box, etc.) that executes the event. i Represents additional information associated with this event (such as a specific output string in text output);

[0018] A3: By executing the event sequence, the program crash error c in the Android application software P can be stably reproduced. If executing the event sequence multiple times does not trigger the corresponding program crash error c every time, then the event sequence cannot stably reproduce the program crash error c, and is therefore not a crash triggering path in the Android application software P.

[0019] In step 2, the application source code instrumentation step, the specific operation of instrumenting the Android application software P according to the crash trigger path set T is as follows:

[0020] B1: Select a crash trigger path t from the crash trigger path set T and obtain its specific event sequence [e1, e2, …, e n ];

[0021] B2: According to the specific event sequence corresponding to the crash triggering path obtained in step B1, the source code of the Android application P is instrumented for the events in the sequence. i =(t i ,w i ,o i ), where i∈{1,2,…,n}, find the event e in the source code of the Android application P i Graphical interface components w i The corresponding response function f i , and in the response function f i Insert a line of output statement to output the event number i;

[0022] B3: Repeat steps B1 and B2 until all events in all crash trigger paths in the crash trigger path set T have corresponding stubs in the source code of the Android application software P, and finally compile the stub-inserted source code into an APK file to obtain the stub-inserted Android application software P'.

[0023] In step 3, the crash error automaton construction step, a crash error automaton a constructed by combining the program crash trigger path set T with the graphical user interface migration that occurs when the event is executed needs to meet the following requirements:

[0024] C1: The crash error automaton a is a non-deterministic finite state automaton with ∈ transition, where ∈ transition originally means that the automaton can transition from one state to another without receiving any input, while the ∈ transition in the present invention refers to the state transition performed by receiving ∈ event;

[0025] The ∈ event refers to an event that can eliminate the execution effect of the previous event.

[0026] C2: The crash error automaton a is defined as a five-tuple, i.e., a = (Q, Σ, δ, q0, F).

[0027] The ∈ event in C1 is mainly used to summarize events such as return events and cancel operations that can eliminate the execution effects of previous events. Strictly speaking, this type of event is not in the crash trigger path, but this type of event occurs all the time when the Android automated graphics testing tool tests the Android application. In order to better track and analyze the execution path of the test tool, the ∈ event is used to refer to this type of event. For example, if the user executes event e1 in the graphical interface l1 of the Android application by clicking the top button in the graphical interface l1, the graphical interface of the Android application will jump to another interface l2, corresponding to a transition (l1, e1, l2) in the crash error automaton. If a return event is executed at this time, the graphical interface of the Android application will return from l2 to l1, and an ∈ transition occurs at this time, that is, (l2, ∈, l1), and the ∈ event refers to the return event.

[0028] The definition of the crash error automaton a in C2 is required to be described in detail as follows:

[0029] D1: Q is a finite set of abstract program states (abstract program state set), where the abstract program state q i It can be reached by executing the event represented by the output character in Σ in the Android application software P or P';

[0030] D2: Σ is a finite set of input characters and {∈}, where the input characters are composed of all the sequence numbers of the events on the crash trigger path in the program crash trigger path set T;

[0031] D3:δ is the migration function, that is in It is a power set of Q, that is, receiving an input character in an abstract program state can migrate to multiple different abstract program states;

[0032] D4: q0 is an element in the abstract program state set Q and is the starting state of the crash error automaton a;

[0033] D5: F is a set of termination states. In the present invention, F only includes an abstract program state f indicating that a program crash error occurs.

[0034] The steps to describe the abstract program state in D1 and the migration in D3 are as follows:

[0035] E1: Select a crash trigger path t from the program crash trigger path set T and obtain the specific event execution sequence [e1, e2, …, e n ];

[0036] E2: Execute the events in t in sequence on the Android application software P and record the corresponding graphical interface change sequence l = [l0, l1, …, l n ] and a series of interface migrations (l0,e1)→l1, (l1,e2)→l2,…, (l n-1 ,e n )→l n , where l0 refers to the initial interface of the Android application software P, l n Refers to a crash error in the Android application software P;

[0037] E3: Based on the graphical interface change sequence recorded in E2, execute potential ∈ events for Android application software P (this can be done by checking whether the current interface supports ∈ events including return events on each interface. Usually, ∈ events are supported). Record the newly appeared graphical interfaces and interface migrations during the process, such as (l i ,∈)→l j , where i,j∈{1,2,…,n}.

[0038] E4: Repeat steps E1 to E3 until the corresponding graphical interface change sequences and interface migrations are obtained for all crash trigger paths t in the program crash trigger path T, and the interface l in all graphical interface sequences is i The abstract program state set Q in the crash error automaton a, all interface transitions (l i ,e k )→l j The transition function δ in the crash error automaton a is composed. In the actual recording process, the event parameters in the transition function will be simplified to the corresponding event number. For example, inserting an output statement to print the string "Event k" in the corresponding function corresponds to event e k , which is convenient for corresponding instrumentation output.

[0039] In the logging step of step 4, the specific requirements for running the Android automated testing tool G to be evaluated on the instrumented Android application P' are as follows:

[0040] F1: The number of test rounds is 5, and the test duration of each round is 6 hours. This setting of test rounds and duration can better ignore the randomness shown by the Android automated graphics test tool during the test process, and also allow the tool to have a deeper exploration of the application's graphical interface space. Under this test setting, if the Android automated graphics test tool cannot find the crash error in the application, the root cause is that the possibility of test randomness is negligible;

[0041] F2: Record the instrumentation output o generated during the running of the application P during the test.

[0042] In the log analysis step of step 5, the specific steps of using the crash error automaton a built in step 3 to analyze and record the application runtime instrumentation output o recorded in step 4 are as follows:

[0043] G1: Parse the recorded application runtime instrumentation output o and obtain the event number sequence t'=[id1, id2, ..., id n' ], where id i' (i'∈{1,2,…,n'}) represents the event number output of the instrumentation in step B2, such as "Event k" in E3; this sequence is obtained when the GUI testing tool runs on the instrumented application software P', reflecting the behavior trajectory of the GUI testing tool in the process of testing the software, that is, which events in the program crash triggering path are executed.

[0044] G2: Use the equivalent transformation algorithm to transform the crash error automaton a into a deterministic finite state automaton a', calculate the minimum distance from each state of the equivalently transformed automaton a' to the terminal state f, and record it in the mapping relationship table MD: Q→N, where N represents a natural number set;

[0045] The equivalent conversion algorithm is a subset construction method. Using this method, a non-deterministic finite state automaton (NFA) can be equivalently converted into a deterministic finite state automaton (DFA). This algorithm can be replaced by the Thompson construction method and the direct construction method with the same effect.

[0046] G3: Record the current state q of the transformed deterministic finite state automaton a' in I i , if the current state is the terminal state f, reset the automaton state to the initial state q0, if the deterministic finite state automaton a' also records the last migration, then the event number id in the last migration is also recorded in I x ;

[0047] G4: Read an event number id sequentially from the event number sequence t' y As input, that is, the input character of the automaton (see step D2), a' is made to transition to q j , and record this migration (q i ,id x ,q j ), and record the event number id in I y , event number pair (id x ,id y ) and the minimum distance MD(q j ) count value is increased by one.

[0048] In step G2, it is required to convert the constructed crash error automaton a into a deterministic finite state automaton a' in order to receive the event number sequence in the log file that records the output of the plug-in, and to use the existing shortest path algorithm (for example, Dijkstra's shortest path algorithm, Floyd's algorithm, Bellman-ford algorithm) to calculate the minimum distance from each state to the terminal state f. The plug-in event does not contain the ∈ event in the non-deterministic finite state automaton, so the event number cannot be directly used as the input of the automaton a. On the other hand, processing ∈ migration has more runtime overhead, and the shortest path calculation cannot be directly and conveniently performed.

[0049] In step 6, the evaluation index calculation step, the specific indicators include the following three items:

[0050] H1: Event coverage EC. According to the event counter in the analysis information I recorded in step 5, we can know whether a certain event on the crash trigger path in the crash trigger path set T is executed and how many times it is executed in total. Based on this information, by calculating (executed events / total number of events), the result is the event coverage.

[0051] H2: Event pair coverage EPC. According to the event pair counter in the analysis information I recorded in step 5, we can know whether a certain event pair has been executed and the total number of executions. Based on this information, by calculating (the number of event pairs executed / the total number of event pairs), the result is the event pair coverage.

[0052] H3: Minimum distance MD from the crash error. According to the minimum distance counter in the analysis information I recorded in step 5, find the one whose counter value is not zero and whose distance value is the smallest, which is the minimum distance from the crash error.

[0053] In the result output step of step 7, no matter the command line format output or HTML format visual output, the record information I collected in step 5 and the three indicators calculated in step 6 need to be presented in detail, and the core part includes the triggering situation (triggering times) of the event on the crash trigger path in the crash trigger path set and the data of the three indicators. Directly presenting the analysis results to the Android automated graphics testing tool developer in the above manner is conducive to the developer to analyze the reasons for the tool's underreporting more directly.

[0054] The present invention also provides a GUI test tool auxiliary diagnosis and debugging system based on automaton, the system comprises a program error automaton extraction module, an application software plugging module, a GUI test tool driving module, and an automaton-based path analysis module.

[0055] The automaton extraction module is used to implement step 3, that is, to construct a corresponding program error automaton through the crash error triggering path;

[0056] The application software instrumentation module is used to implement step 2, that is, to perform instrumentation in the corresponding Android application according to the crash error trigger path (printing a string with an event number to a program statement of the standard output);

[0057] The GUI test tool driver module is used to implement step 4, that is, running the GUI test tool on the plugged software;

[0058] The automaton-based path analysis module is used to implement steps 5, 6 and 7, that is, the program error automaton constructed by the automaton extraction module is used to analyze the log file output by the plugged software (the software plugged by the application software plugging module);

[0059] Step 5 does not require a specific system module to complete, because the program statements used during instrumentation will print the runtime trace of the GUI test tool (i.e., the event number of execution) to the log output by the Android application runtime (such as the logcat.log file).

[0060] The present invention also provides application of the above method or system in auxiliary diagnosis and debugging of GUI testing tools.

[0061] The beneficial effects of the present invention are:

[0062] The present invention abstracts program errors into finite state machines, provides a method of reversely testing and evaluating Android automated graphics testing tools with the help of program crash errors of known Android applications, and makes good use of a large number of existing application crash errors.

[0063] The present invention proposes for the first time a method for evaluating and assisting debugging of an Android automated graphics testing tool, which effectively and directly provides testing tool developers with an analysis of the causes of tool failures in reporting program crash errors, helping testing tool developers to conduct in-depth analysis and strategy improvements on the tools, thereby indirectly ensuring the quality of mobile applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a method schematic diagram of the automaton-assisted debugging method for the automated graphics testing tool of the present invention.

[0065] Figure 2 It is a schematic diagram of the crash triggering path in the present invention.

[0066] Figure 3 Schematic diagram of the crash error automaton in the present invention.

[0067] Figure 4 It is a result output schematic diagram (command line output) in the present invention.

[0068] Figure 5 It is a result output schematic diagram (HTML visualization) in the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0070] The present invention proposes a GUI test tool-assisted diagnosis and debugging method based on automata, which abstractly constructs a non-deterministic automata based on known program errors in Android applications, and simultaneously inserts stubs into corresponding Android applications, collects logs output by automated graphics testing tools when testing Android applications, and performs path matching on the crash error automata by using the Android application event execution sequence recorded in the logs, and uses the data recorded during the matching process to calculate three indicators: event coverage, event pair coverage, and minimum distance, to determine whether the test tool has found the corresponding program error and generated a command line output or a visual report. This method can evaluate the effectiveness of Android automated graphics testing tools, assist Android automated graphics testing tool developers in diagnosing and debugging test tools, and find defects in the tools and make improvements. The present invention also proposes a system for implementing the above method and an application of the above method or system.

[0071] The main process of the present invention includes seven steps: program crash error and crash trigger path identification, application source code instrumentation, crash error automaton construction, log recording, log analysis, evaluation index calculation, and result output. The overall framework of the method is as follows Figure 1 As shown, the specific steps include:

[0072] Step 1: Program crash error and crash trigger path identification step: Given a crash error c in an Android application software P, record one or more different crash trigger path sets T in the Android application software P that can stably reproduce the program crash error c.

[0073] Step 2: Application source code instrumentation step: According to the crash trigger path set T recorded in step 1, find the response function position corresponding to the event response graphical interface component in the source code of the Android application software P and perform instrumentation to obtain the instrumented Android application software P'.

[0074] Step 3: Crash error automaton construction step: Construct a crash error automaton a according to the crash trigger path set T in step 1 and the graphical user interface migration that occurs when the events in these paths are executed in the Android application software P.

[0075] Step 4: Logging step: Use the Android automated graphics testing tool G that needs to be evaluated to test on the instrumented Android application software P', and record the instrumentation output o in the process.

[0076] Step 5: Log analysis step: By giving the instrumentation output o recorded in step 4 as input to the crash error automaton a built in step 3, the automaton will accept characters for state migration, and record the runtime information I of the Android automated graphics testing tool G based on the crash error automaton a during the migration process.

[0077] Step 6: Evaluation index calculation step: Calculate the three evaluation indicators of event coverage EC, event pair coverage EPC and minimum distance MD based on the information I collected from the log analysis in step 5.

[0078] Step 7: Result output step: Integrate the runtime information I in step 5 and the three evaluation indicators calculated in step 6 to generate a command line output or HTML visualization result report.

[0079] In step 1, in the step of identifying the program crash error and the crash trigger path, any crash trigger path t in the crash trigger path set T needs to meet the following requirements:

[0080] A1: The crash triggering path t is the sequence of events that can be executed in the Android application software P, that is, t = [e1, e2, …, e n ];

[0081] A2: Event e in the executable event sequence i Defined as a triple, namely e i =(t i , w i , o i ), where t i Represents the type of the event (click, longclick, etc.), w i Represents the target graphical interface component (Button, TextView, etc.) that executes the event. i Represents additional information associated with this event (such as a specific output string in text output);

[0082] A3: By executing the event sequence, the program crash error c in the Android application software P can be stably reproduced. If executing the event sequence multiple times does not trigger the corresponding program crash error c every time, then the event sequence cannot stably reproduce the program crash error c, and is therefore not a crash triggering path in the Android application software P.

[0083] In step 2, the application source code instrumentation step, the specific operation of instrumenting the Android application software P according to the crash trigger path set T is as follows:

[0084] B1: Select a crash trigger path t from the crash trigger path set T and obtain its specific event sequence [e1, e2, …, e n ];

[0085] B2: According to the specific event sequence corresponding to the crash triggering path obtained in step B1, the source code of the Android application P is instrumented for the events in the sequence. i =(t i ,w i ,o i ), find event e in the source code of Android application P i Graphical interface components w i The corresponding response function f i In the response function f i In the output statement, insert a line containing the event number information;

[0086] B3: Repeat steps B1 and B2 until all events in all crash trigger paths in the crash trigger path set T have corresponding stubs in the source code of the Android application software P, and finally compile the stub-inserted source code into an APK file to obtain the stub-inserted Android application software P'.

[0087] In step 3, the crash error automaton construction step, the specific steps of constructing a crash error automaton a by combining the program crash trigger path set T with the graphical user interface migration that occurs when the event is executed are:

[0088] E1: Select a crash trigger path t from the program crash trigger path set T and obtain the specific event execution sequence [e1, e2, …, e n ];

[0089] E2: Execute the events in t in sequence on the Android application software P and record the corresponding graphical interface change sequence l = [l0, l1, …, l n ] and a series of interface migrations (l0,e1)→l1, (l1,e2)→l2,…, (l n-1 ,e n )→ln , where l0 refers to the initial interface of the Android application software P, l n Refers to a crash error in the Android application software P;

[0090] E3: Based on the graphical interface change sequence recorded in E2, execute the potential ∈ event for the Android application software P, and record the newly appeared graphical interface and interface migration in the process, such as (l i ,∈)→l j .

[0091] E4: Repeat steps E1 to E3 until the corresponding graphical interface change sequence and interface migration are obtained for all crash trigger paths t in the program crash trigger path T. All interfaces li in the graphical interface sequence constitute the abstract program state set Q in the crash error automaton a. All interface migrations (l i ,e k )→l j The transition function δ in the crash error automaton a is composed. During the recording process, the event parameters in the transition function are simplified to the corresponding event numbers. The input character set Σ is composed of the sequence numbers of the events on the crash trigger path in the program crash trigger path set T. The terminal state F only contains an abstract program state f indicating that the program has a crash error.

[0092] In the logging step of step 4, the Android automated testing tool G to be evaluated is run on the instrumented Android application P' to perform the test:

[0093] F1: There are 5 test rounds, and each round lasts 6 hours.

[0094] F2: Record the instrumentation output o generated during the running of the application P during the test.

[0095] In the log analysis step of step 5, the specific steps of using the crash error automaton a built in step 3 to analyze and record the application runtime instrumentation output o recorded in step 4 are as follows:

[0096] G1: Parse the recorded application runtime instrumentation output o and obtain the event number sequence t'=[id1, id2, ..., id n' ];

[0097] G2: Use the equivalent transformation algorithm to transform the crash error automaton a into a deterministic finite state automaton a', calculate the minimum distance from each state of the equivalently transformed automaton a' to the terminal state f, and record it in the mapping relationship table MD: Q→N, where N represents a natural number set;

[0098] G3: Record the current state qi of the transformed deterministic finite state automaton a' in I. If the current state is the terminal state f, reset the automaton state to the initial state q0. If the deterministic finite state automaton a' also records the previous migration, then record the event number id in the previous migration in I. x ;

[0099] G4: Read an event number id sequentially from the event number sequence t' y As input, a' undergoes state transition to q j , and record this migration (q i ,id x ,q j ), and record the event number id in I y , event number pair (id x ,id y ) and the minimum distance MD(q j ) count value is increased by one.

[0100] In step 6, the evaluation index calculation step, the specific indicators include the following three items:

[0101] H1: The event coverage rate EC is obtained by calculating (number of executed events / total number of events).

[0102] H2: By calculating (the number of event pairs executed / the total number of event pairs), we can obtain the event pair coverage EPC.

[0103] H3: Get the minimum distance MD from the collapse error.

[0104] In step 7, the result output step, the data and evaluation results obtained in steps 5 and 6 are integrated and output in command line format or HTML format.

[0105] Example

[0106] In this embodiment, the open source software ScarletNotes (an Android application for note taking) on ​​GitHub is selected as the Android application software known to have crash errors, and the test software FastBot, which is also open source on GitHub, is selected as the Android automated graphics testing tool to be tested. The automatic machine-assisted debugging method for the automated graphics testing tool of the present invention is specifically described as follows:

[0107] Step 1 In the step of identifying the program crash error and the crash trigger path, select the program crash error given in the error report numbered #114 in the ScarletNotes GitHub repository as the target program crash error c, and use the error reproduction steps in the error report as the crash trigger path t of the program crash error. Because only the path t is mentioned in the report, the complete set of crash trigger paths T only contains path t. This path t is as follows Figure 2 As shown in:

[0108] e1: Click the c1 control on the initial application interface to create a folder;

[0109] e2: Click the newly created folder c2 control on the initial application interface;

[0110] e3: In the empty folder interface, click the c3 control in the lower left corner to pop up the option menu;

[0111] e4: Select the "Locked" option in the options menu c4 control to filter out locked notebooks;

[0112] e5: Click the "X" c5 control in the lower left corner of the interface to trigger the application crash.

[0113] In step 2, the application source code instrumentation step, the source code of the ScarletNotes application is instrumented according to the complete set of crash trigger paths T obtained in step 1. Find each event e in the source code i Corresponding trigger control c i For the response function (such as onClick), add an output statement before the first statement in the response function body, and the output content is the event number plus the event details. For example, for event e1, after finding the corresponding function onClick in the source code, add Log.i ("Event 1: Create a new folder in the initial interface of the application") before the first statement in its function body. After the instrumentation is completed, recompile the source code into an APK file to obtain the instrumented application P'.

[0114] In step 3, the crash error automaton construction step, according to the crash trigger path t, first execute the crash trigger path in the ScarletNotes application. After each event execution, the new user interface is used as a new abstract program state. The initial abstract program state is the initial interface of the application. Then, in the application graphical interface corresponding to the obtained abstract program state, the events in the executable path t and the potential ∈ events (such as pressing the return key) are executed to obtain a new abstract program state and a new interface migration, for example Figure 2 The arrow marked with back in indicates that this is a recorded ∈ migration. The constructed crash error automaton a is as follows Figure 3 shown.

[0115] In the logging step of step 4, FastBot is used to run the instrumented application P' for a total of 5 rounds, each lasting 6 hours. A log with instrumented output is obtained.

[0116] In the log analysis step of step 5, the crash error automaton a constructed in step 3 is first converted into a deterministic finite state automaton a', and then the output log obtained in step 4 is used as the input of the automaton a'. The information I during the operation of the Android automated graphics testing tool is recorded through the automaton migration process, mainly collecting the number of triggers of each event, the number of triggers of each event pair, and the number of minimum distances reached.

[0117] In step 6, the evaluation indicator calculation step, according to the information I collected in step 5, the event coverage EC is calculated by (number of triggered events / total number of events), and the event pair coverage EPC is calculated by (number of triggered event pairs / total number of event pairs). The minimum distance MD between the test tool and the triggering program crash during the test is calculated by obtaining the minimum value of the minimum distances reached, that is, how many events are needed to trigger the crash c.

[0118] In step 7, the result output step integrates the data and evaluation results obtained in steps 5 and 6, and outputs the result through the command line format ( Figure 4 ) or HTML format ( Figure 5 ) for output.

[0119] The mobile application in the embodiment can be replaced, including but not limited to various types of open source applications, commercial applications, and personal applications, as long as they have known program crash errors and editable source code.

[0120] The program crash error in the embodiment can be replaced as long as there is a stable and reproducible crash triggering path.

[0121] The Android automated graphics testing tool in the embodiment can be replaced as long as it can perform graphical interface level testing on Android application software.

[0122] The number of test rounds and test duration in the embodiment can be customized, but it is recommended to use the settings in the embodiment.

[0123] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A GUI test tool-assisted diagnosis and debugging method based on automata, characterized in that: The steps include: Step 1: Identify program crash errors and crash trigger paths: Given a crash error c in an Android application software P, record one or more different crash trigger path sets T that can stably reproduce the crash error c in the Android application software P; Step 2: Application source code instrumentation: According to the crash trigger path set T recorded in step 1, find the response function position corresponding to the event response graphical interface component in the source code of the Android application software P and perform instrumentation to obtain the instrumented Android application software P'; Step 3: Crash error automaton construction: construct a crash error automaton a according to the crash trigger path set T in step 1 and the graphical user interface migration that occurs when the events in these crash trigger paths are executed in the Android application software P; Step 4: Logging: Use the Android automated graphics testing tool G that needs to be evaluated to test the instrumented Android application software P', and record the instrumentation output o in the process; Step 5: Log analysis: The instrumentation output o recorded in step 4 is given as input to the crash error automaton a built in step 3. The automaton a will accept characters for state migration, and the runtime information I of the Android automated graphics testing tool G based on the crash error automaton a is recorded during the migration process. Step 6: Calculate evaluation indicators: Calculate the three evaluation indicators of event coverage EC, event pair coverage EPC and minimum distance MD according to the test tool runtime information I collected by analyzing the log in step 5; Step 7: Result output: Combine the test tool runtime information I in step 5 and the three evaluation indicators calculated in step 6 to generate a command line output or HTML visualization result report.

2. The method according to claim 1, characterized in that In step 1, in the step of identifying the program crash error and the crash trigger path, any crash trigger path t in the crash trigger path set T needs to meet the following requirements: A1: The crash triggering path t is the sequence of events that can be executed in the Android application software P, that is, t = [e1, e2, …, e n ]; A2: Event e in the executable event sequence i Defined as a triple, namely e i =(t i , w i , o i ), where t i Representative event i The type of i Represents the execution event e i The target graphical interface component, o i Representatives and Events i Associated additional information; A3: By executing the event sequence, the program error c in the Android application software P can be stably reproduced.

3. The method according to claim 1, characterized in that In step 2, the application source code instrumentation step, the specific operation of instrumenting the Android application software P according to the crash trigger path set T is as follows: B1: Select a crash trigger path t from the crash trigger path set T and obtain the specific event sequence [e1, e2, …, e n ]; B2: According to the specific event sequence corresponding to the crash trigger path obtained, the source code of the Android application P is instrumented for the events in the sequence in turn; for event e i =(t i ,w i ,o i ), where i∈{1,2,…,n}, find the event e in the source code of the Android application P i Graphical interface components w i The corresponding response function f i , and in the response function f i Insert a line of output statement to output the event number i; B3: Repeat steps B1 and B2 until all events in all crash trigger paths in the crash trigger path set T have corresponding stubs in the source code of the Android application software P, and finally compile the stub-inserted source code into an APK file to obtain the stub-inserted Android application software P'.

4. The method according to claim 1, characterized in that: In step 3, the crash error automaton construction step, a crash error automaton a constructed by combining the program crash path with T needs to meet the following requirements: C1: The crash error automaton a is a non-deterministic finite state automaton with ∈ transitions, where ∈ transitions refer to state transitions that are performed upon receiving ∈ events; and ∈ events refer to events that can eliminate the effects of the execution of the preceding events; C2: The crash error automaton a is defined as a five-tuple, i.e., a=(Q,Σ,δ,q0,F); where, Q is a finite set of abstract program states, where the abstract program state q i The event represented by the output character in Σ can be reached by executing it in the Android application software P or P'; Σ is a finite set of input characters and {∈}, where the input characters are composed of all the sequence numbers of the events on the crash trigger path in the program crash trigger path set T; δ is the migration function, that is, δ: in It is a power set of Q, that is, receiving an input character in an abstract program state can migrate to multiple different abstract program states; q0 is an element in the abstract program state set Q and is the starting state of the crash error automaton a; F is a set of terminal states, and F contains only one abstract program state f indicating that a program crash error has occurred.

5. The automaton-assisted debugging method for an Android automated graphics testing tool according to claim 1, characterized in that: In step 3, the specific steps of constructing a crash error automaton a by combining the program crash trigger path set T with the graphical user interface migration that occurs during event execution are as follows: E1: Select a crash trigger path t from the program crash trigger path set T, and obtain the specific event execution sequence [e1, e2, …, e n ]; E2: Execute the events in t in sequence on the Android application software P and record the corresponding graphical interface change sequence l = [l0, l1, …, l n ] and a series of interface migrations (l0,e1)→l1, (l1,e2)→l2,…, (l n-1 ,e n )→l n , where l0 refers to the initial interface of the Android application software P, l n Refers to a crash error in the Android application software P; E3: executes potential ∈ events for the application P based on the graphical interface change sequence recorded in E2, and records the newly appeared graphical interfaces and interface migrations in the process; E4: Repeat steps E1 to E3 until corresponding graphical interface sequences and interface migrations are obtained for all crash trigger paths t in the program crash trigger path T.

6. The method according to claim 1, characterized in that In the logging step of step 4, the specific requirements for running the Android automated testing tool G to be evaluated on the instrumented Android application P' are as follows: F1: There are 5 test rounds, and each round lasts 6 hours; F2: Record the instrumentation output o generated during the running of the application P during the test.

7. The method according to claim 1, characterized in that In the log analysis step of step 5, the specific steps of using the crash error automaton a built in step 3 to analyze the application runtime instrumentation output o recorded in step 4 and record the test tool runtime information I are as follows: G1: Parse the recorded application runtime instrumentation output o and obtain the event number sequence t'=[id1, id2, ..., id n' ];id i' Indicates the event number output for instrumentation in step B2, i'∈{1,2,…,n'}; G2: Use the equivalent transformation algorithm to transform the crash error automaton a into a deterministic finite state automaton a', calculate the minimum distance from each state of the equivalently transformed automaton a' to the terminal state f, and record it in the mapping relationship table MD: Q→N, where N represents a natural number set; G3: Record the current state q of the transformed deterministic finite state automaton a' in I i , if the current state is the terminal state f, reset the automaton state to the initial state q0, if the automaton a' still records the last migration, then the event number id in the last migration is also recorded in I x ; G4: Read an event number id sequentially from the event number sequence t' y As input, a' undergoes state transition to q j , and record this migration (q i ,id x ,q j ), and record the event number id in I y , event number pair (id x ,id y ) and the minimum distance MD(q j ) count value is increased by one.

8. The method according to claim 1, characterized in that: In step 6, the evaluation index calculation step, the specific indicators include the following three items: H1: Event coverage EC: Based on whether an event on the crash trigger path in the crash trigger path set T is executed and how many times it is executed in total, the result obtained by calculating "executed events / total number of events" is the event coverage; H2: Event pair coverage (EPC): Based on whether an event pair is executed and the total number of executions, the result obtained by calculating "the number of event pairs executed / the total number of event pairs" is the event pair coverage; H3: Minimum distance to crash error MD: The minimum distance to crash error is the minimum distance recorded by the minimum distance counter when the counter value is not zero and the distance value is the smallest.

9. A system for implementing the method according to any one of claims 1 to 8, characterized in that: The system includes: a program error automaton extraction module, an application software plugging module, a GUI test tool driving module, and an automaton-based path analysis module; wherein, The automaton extraction module is used to construct a corresponding program error automaton through a crash error triggering path; The application software instrumentation module is used to perform instrumentation in the corresponding Android application according to the crash error triggering path; The GUI test tool driver module is used to run the GUI test tool on the plugged software; The automaton-based path analysis module is used to analyze the log files output by the instrumented software using the program error automaton constructed by the automaton extraction module.

10. Application of the method according to any one of claims 1 to 8 or the system according to claim 9 in assisting diagnosis and debugging of GUI test tools.

Images