Black box automated testing method based on graphical user interface

By obtaining and generating test scripts and running them on the driver engine, the problem of automated testing of GUI application software is solved, efficient and accurate GUI testing is achieved, and software performance and user experience is improved.

CN113704125BActive Publication Date: 2024-11-26NANJING LINGXING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111021509.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-01
Publication Date
2024-11-26
Estimated Expiration
2041-09-01

AI Technical Summary

Technical Problem

It is difficult for the existing technology to effectively carry out automated testing of GUI application software. The traditional capture/playback mechanism cannot interact with the tested software and cannot meet the high requirements of modern software testing for automation testing efficiency and performance.

Method used

By obtaining event information and control information of testers' operations, generating a test script, and running the script on the driver engine, performing black box automation testing of GUI application software, improving control recognition and traversal algorithms to improve the accuracy and efficiency of test scripts .

Benefits of technology

It realizes efficient black box automated testing of GUI application software, improves testing efficiency and accuracy, and improves software performance and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113704125B_ABST
    Figure CN113704125B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a black box automated testing method, device, electronic device and storage medium based on a graphical user interface. The method obtains event information and control information of a tester operating the software to be tested, generates a test script according to the event information, control information and preset assertions, runs the test script on a driving engine, performs automated GUI testing on the software to be tested, and generates a test report. This implements automated GUI testing of the software to be tested, improves the efficiency and accuracy of GUI testing of the software to be tested, and is beneficial to improving the performance of the software to be tested and enhancing the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to a software automation testing method, which belongs to the technical field of computer software testing, and in particular to a black box automation testing method, device, electronic device and storage medium based on a graphical user interface. Background Art

[0002] Software testing is an important means to ensure software quality and an important part of the software development process. Graphical user interface (GUI) testing is a key part of modern software testing, and the quality of the GUI system is the key to improving the quality and reducing the cost of the entire software product. With the continuous development and progress of software technology based on the agile development model, software testing faces higher challenges; it not only requires a higher degree of automation and reduces manual participation, but also puts forward higher requirements on the efficiency and performance of automated testing. Due to the uniqueness of Windows GUI application software, the original traditional software testing methods are not suitable for GUI application software testing, and GUI manual testing can no longer meet the current GUI application software testing requirements.

[0003] In the prior art, the capture / playback (C / P) mechanism is commonly used for automated testing of GUI application software. However, the C / P mechanism does not provide good support for automated testing of GUI application software. It can only passively capture the execution information of the software under test, but cannot interact with the software under test and selectively capture the execution information of the software under test.

[0004] Therefore, effective automated testing of GUI application software has become a technical problem that urgently needs to be solved in existing software testing technologies. Summary of the invention

[0005] The embodiments of the present application provide a black box automated testing method, device, electronic device and storage medium based on a graphical user interface, which realizes black box automated testing of GUI application software.

[0006] In a first aspect, an embodiment of the present application provides a black box automated testing method based on a graphical user interface, comprising:

[0007] Obtain event information and control information of the tester's operation of the software under test;

[0008] Generate a test script according to the event information, the control information and the preset assertion;

[0009] The test script is run on the driving engine to perform a graphical user interface (GUI) test on the software to be tested and a test report is generated.

[0010] In a second aspect, an embodiment of the present application provides a black box automated testing device based on a graphical user interface, comprising:

[0011] The recording module is used to obtain the event information and control information of the tester operating the software to be tested; and generate a test script according to the event information, the control information and the preset assertion;

[0012] The playback module is used to run the test script on the driving engine, perform a graphical user interface (GUI) test on the software to be tested, and generate a test report.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the black box automated testing method based on a graphical user interface as described in the first aspect above is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the black-box automated testing method based on a graphical user interface as described in the first aspect above.

[0015] The black-box automated testing method, device, electronic device and storage medium based on a graphical user interface provided in the embodiments of the present application obtain event information and control information of the tester's operation on the software to be tested, generate a test script according to the event information, control information and preset assertions, run the test script on the driver engine, perform GUI testing on the software to be tested, and generate a test report, thereby realizing black-box automated testing of the GUI of the software to be tested, improving the efficiency and accuracy of GUI testing of the software to be tested, and being beneficial to improving the performance of the software to be tested and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a black box automated testing method based on a graphical user interface provided in Example 1 of the present application;

[0017] Figure 2 A schematic diagram of an assertion interface provided in Example 1 of the present application;

[0018] Figure 3 A flowchart of a control identification method provided in Embodiment 2 of the present application;

[0019] Figure 4 A schematic diagram of the flow of the dynamic traversal algorithm provided in Example 2 of the present application;

[0020] Figure 5 A schematic diagram of a traversal process provided in Embodiment 2 of the present application;

[0021] Figure 6 A schematic diagram of the judgment process of GUI state migration provided in the second embodiment of the present application;

[0022] Figure 7 A flowchart diagram of another dynamic traversal algorithm is provided for the second embodiment of the present application;

[0023] Figure 8 A flowchart of a control identification method provided in Embodiment 3 of the present application;

[0024] Fig. 9 A logical diagram of obtaining GUI screenshots based on a combination of automatic screenshots and manual screenshots provided in Example 3 of the present application;

[0025] Fig.10 A schematic diagram of the structure of a black box automated testing device based on a graphical user interface provided in Example 4 of the present application;

[0026] Fig.11 A schematic diagram of the structure of another black box automated testing device based on a graphical user interface provided in Example 4 of the present application;

[0027] Fig.12 A schematic diagram of the design architecture of the script management module provided in the fourth embodiment of the present application;

[0028] Fig.13 A schematic diagram of the structure of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0029] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0030] First, the concepts used in this embodiment are explained as follows:

[0031] (1) Window status

[0032] It is an abstract concept that describes the internal situation of a certain window of the application software at a certain moment. A single window state can be represented by a triple {O, P, V}, where O stands for Objects, representing the set of all GUI elements in the window state, which can be represented as O = {o l ,.....,o n}; P stands for Properties, which represents all the properties of a single GUI element in O, and can be expressed as P = {p l ,.....,pm}; V is Values, all attribute values ​​of each GUI element in the set O, which can be expressed as V = {(v 1l ,.....,v 1m ),......,(v nl ,.....,v nm )}.

[0033] (2) GUI Status

[0034] It represents the abstract concept of the overall situation of the application GUI at a certain moment. It is a state identifier that reflects the overall situation of the application software graphical user interface at a certain moment. A single GUI state is represented by {w t ,W} tuple, where w t Represents the window state of the top-level active window in the GUI state. W represents a window set, which includes the window states of all windows except the top-level active window in the GUI state.

[0035] (3) GUI state migration

[0036] The GUI state of the Window application software remains stable and unchanged at a certain moment, that is, {w t ,W} is uniquely determined, but when the application responds to and executes the user's action event, the continuous operation process of the entire application corresponds to the process of continuous migration of the GUI state, that is, {w t ,W} changes. Here w t The change of W is called GUI state migration. A common migration process is i You can use (S i ,e i ,S′ i ) is formally expressed. Among them, S i Indicates the GUI state before migration, S′ i Indicates the GUI status after migration, e i Represents a migration event.

[0037] (4) Migration events

[0038] A user event that causes a GUI state transition at a fixed time; it includes reachable events and termination events. A reachable event is a user event that opens a new window at a certain time. A termination event is a user event that closes the current top-level active window at a certain time.

[0039] (5) State sequence table, event sequence table

[0040] State sequence table: used to store all GUI states;

[0041] Event sequence table: used to store all user events.

[0042] (6) Migration status list, migration event list

[0043] Migration status list: used to store the GUI status that migrates during the traversal process;

[0044] Migration event list: used to store migration events during the traversal process.

[0045] The main idea of ​​the technical solution of this application: Based on the technical problems existing in the prior art, the embodiment of this application provides a black box automated testing solution based on Windows GUI application software. On the basis of the "recording-playback" regression test, the test personnel's operations on the software to be tested are recorded during the recording process to generate test steps, and the test script is automatically generated according to the test steps and pre-set assertions (judgment statements). During the playback process, the test script is run by the driving engine to realize the black box automated testing of the GUI of the software to be tested. In addition, in this embodiment, on the basis of improving the existing control recognition algorithm and traversal algorithm, the control object information is obtained, the control recognition accuracy and recognition efficiency are improved, which not only shortens the time required to generate the test script, improves the test efficiency, but also improves the availability and reliability of the test script, and improves the accuracy of the test.

[0046] Embodiment 1

[0047] Figure 1 This is a flowchart of a black box automated testing method based on a graphical user interface provided in the first embodiment of the present application. The method of this embodiment can be performed by a black box automated testing device based on a graphical user interface provided in the embodiment of the present application. The device can be implemented by software and / or hardware, and can be integrated into an electronic device such as a computer installed with the software to be tested. Figure 1 As shown, the black box automated testing method based on a graphical user interface of this embodiment includes:

[0048] S101. Obtain event information and control information of the tester operating the software to be tested.

[0049] In this embodiment, the software to be tested refers to GUI application software, and accordingly, the tester operation also refers to the operation performed by the tester on the GUI of the test software. In order to implement black box automated testing of the GUI of the software to be tested, at the beginning of the test, the tester is required to design a test plan for testing the software to be tested, and the test plan includes specific test steps.

[0050] Accordingly, in this embodiment, when the tester tests the software to be tested according to the test steps in the preset test scheme, the action record of the tester operating the software to be tested is recorded, and the action record includes event information and control information. Among them, event information refers to the information of the tester operating the input device such as the mouse, keyboard, touch screen, etc., such as single click, input character, etc. Control information refers to the information of the control object operated by the tester, including the name, type, position of the control object on the GUI, etc. There is a corresponding relationship between event information and control information.

[0051] Optionally, in this embodiment, the tester's operation on the input device related to the software to be tested can be monitored by a pre-installed monitoring device, so as to obtain event information of the tester's operation on the software to be tested.

[0052] Optionally, in this embodiment, the control information of the software under test operated by the tester can be obtained through a pre-set control recognition algorithm. The control recognition algorithm can be executed by calling a preset control driver, and is mainly used to identify the control object operated by the user from the GUI of the software under test and obtain the information of the control object, such as position information.

[0053] The control driver in this embodiment has the functions of both an application programming interface (API) driver and an image driver. Depending on the type of driver supported by the software to be tested, the technical solution of this application can call the control driver to adopt different control recognition algorithms to obtain control information. For details, please refer to the specific description in Example 2 and Example 3.

[0054] In this embodiment, in order to meet the needs of subsequent generation of test scripts, the event information and control information obtained can be stored accordingly in a collection manner according to the order of different operations performed by the tester on the software to be tested, such as storing in the format of {(event information 1, control information 1), (event information 2, control information 2), ..., (event information n, control information n)}.

[0055] S102: Generate a test script according to event information, control information and preset assertions.

[0056] In this step, a test script is automatically generated according to the event information and control information obtained in S101 and pre-set assertions.

[0057] Among them, assertions are essentially some Boolean expressions, which are set in advance. In the process of automated testing, assertions are mainly used to set checkpoints for test steps to realize the expected judgment of software running results. If the test results meet expectations, it means that the test step can meet the expected test requirements; otherwise, it means that the test step cannot meet the expected test requirements. Therefore, assertions can replace manual judgment of whether the GUI of the software to be tested meets the expected test results.

[0058] In order to make the assertion function more powerful and support more complex logic verification, in this embodiment, the assertions are classified in advance, so that the corresponding assertions can be selected according to the test needs to generate a test script.

[0059] In terms of assertion content, in this embodiment, assertions are divided into control-level assertions, image assertions, and combined assertions:

[0060] (1) Control-level assertion (ControlAssert): used to determine whether the relevant properties of a specified control on the page meet expectations. Specifically, it can be used to determine whether a control exists, determine the regularity of a control attribute string, determine the value of a control attribute, etc.

[0061] (2) PicAssert: When the control element cannot be obtained, the image in the picture assertion is judged by image comparison technology whether it is correctly displayed on the program running interface. Compared with the control-level assertion, the granularity of the picture assertion is coarser.

[0062] (3) Combined Assert: An assertion composed of multiple control-level assertions and image assertions according to certain logical rules. Compared with the previous two, it can realize more complex assertion functions, mainly including sequence control, if-else conditional control, etc.

[0063] In terms of assertion types, in this embodiment, assertions can be divided into hard assertions and soft assertions:

[0064] (1) Hard assertion: When a hard assertion error occurs, the test program will be terminated and an error will be thrown. The remaining test steps will not be executed.

[0065] (2) Soft assertion: When an assertion error occurs, the test program will not be terminated immediately. The remaining test steps and assertions will continue to be executed, and all errors will be thrown after all soft assertions are completed.

[0066] Optionally, according to the above assertion classification, in this embodiment, an assertion interface may be defined in advance, and a class of the assertion interface may be defined, so that different types of assertions can be acquired and executed through different classes. Figure 2 This is a schematic diagram of the assertion interface provided in Example 1 of the present application, such as Figure 2As shown, the assertion interface in this implementation includes: control-level assertion (ControlAssert) class, picture assertion (PicAssert) class, combined assertion (CombineAssert) class, assertion exception (AssertException) class and logic combination rule (LogicTemplate) class. Different classes are used to implement different interface functions of assertion. Among them:

[0067] The ControlAssert class mainly implements the related operations of control-level assertions and stores the position of controls in the control tree. During the playback phase, the control-level assertion finds the corresponding control object through the control position and verifies whether the related properties of the control object meet expectations.

[0068] The PicAssert class mainly implements the related operations of picture-level assertions. In the playback stage, the current software interface screenshot and assertion image are obtained. AssertMatch uses the template matching function in openCV to match whether the assertion image exists in the software interface.

[0069] The CombineAssert class mainly implements the related operations of combined-level assertions. The object pre-selects and loads the logic template, which implements two logics: sequential structure and branch structure. Load several Assert classes into the logic template to determine the logic of combined assertion execution.

[0070] The AssertException class mainly implements assertion exceptions, including exception handling for soft assertions and hard assertions.

[0071] The LogicTemplate class is used to formulate logic rules and provide logic templates.

[0072] In a possible implementation, in this step, the event information and control information obtained in S101 are combined to generate test steps, the control information and target assertions are combined to generate step checkpoints, and the test steps and step checkpoints are combined to generate a test script.

[0073] The target assertion refers to the assertion selected to judge the running result of the current control information, which can be a combination of one or more of the aforementioned assertion types. For different control information, the selected target assertion can be the same or different.

[0074] In this implementation, corresponding grammar rules can be pre-set, including test step generation rules, step checkpoint generation rules and test script generation rules, and according to the test step generation rules, event information and control information are combined to generate test steps, according to the step checkpoint generation rules, control information and target assertions are combined to generate step checkpoints, and according to the test script generation rules, test steps and step checkpoints are combined to generate a test script.

[0075] It can be understood that when there are multiple groups of event information and control information obtained in S101, in this step, corresponding test steps can be generated according to each group of event information and control information, and step checkpoints of corresponding test steps can be generated according to the control information and corresponding target assertions in each group. Then, all the obtained test steps and step checkpoints are combined to finally generate a test script.

[0076] Optionally, in order to strengthen the management of the test script, in this embodiment, after the test script is generated, the test script can also be split into a logic part and a data part, and at the same time, a mapping relationship between the logic part and the data part is established, wherein the logic part contains information such as event type and control path, and the data part includes data such as input variables in the test steps or variable data that the assertion needs to compare, and the logic part, the data part and the mapping relationship between the two are stored separately. Thereby, the separation of test data and test step logic in the test script is achieved, which is convenient for modular management of the test script. In addition, when there are multiple test scripts, managing the test scripts in this way can reduce the degree of coupling between the test scripts, which is conducive to improving the test efficiency of automated testing and reducing the test cost of automated testing.

[0077] In this embodiment, by generating test scripts based on assertions and performing modular management on the test scripts, not only can the test efficiency be improved, but also the test cost can be saved.

[0078] S103: Run the test script on the driver engine to perform GUI testing on the software to be tested and generate a test report.

[0079] In this step, the test script generated in S102 is sent to the driving engine, which runs the test steps and step checkpoints in the test script, performs GUI testing on the software to be tested, and obtains a corresponding test report.

[0080] Optionally, the driving engine in this embodiment is a UI driving engine.

[0081] In this embodiment, by obtaining the event information and control information of the tester operating the software to be tested, a test script is generated according to the event information, control information and preset assertions, and the test script is run on the driver engine to perform GUI testing on the software to be tested, and a test report is generated, thereby realizing the black box automated testing of the GUI of the software to be tested, improving the efficiency and accuracy of the GUI testing of the software to be tested, and being conducive to improving the performance of the software to be tested and enhancing the user experience. Control identification (i.e., the acquisition of control object information) is one of the keys to the entire GUI automated testing, and its identification effect has a direct impact on the correctness of the GUI automated testing. In the recording phase of the regression test, it is necessary to capture mouse and keyboard events. When a mouse or keyboard event occurs, the position where the mouse stays on the screen is recorded, and the operated control is identified by the mouse position. Based on this, the control identification method of the present application is also divided into two cases, that is, according to the different types of drivers supported by the software to be tested, the present application can use different control identification algorithms to perform control identification, and the following will be introduced through the second and third embodiments, respectively, the control identification method based on API drive and the control identification method based on image drive (API drive is not supported).

[0082] Embodiment 2

[0083] For example, Figure 3 A flowchart of a control identification method provided in Embodiment 2 of the present application is shown in FIG. Figure 3 As shown, if the software to be tested supports API drive, in this embodiment, the control information of the software to be tested operated by the tester is obtained through the following control identification algorithm:

[0084] S201, extracting the GUI control tree of the software to be tested through a preset API.

[0085] The control identification algorithm provided in this embodiment acquires control information based on the control tree and the mouse position. To this end, in this step, it is necessary to first extract the GUI control tree of the software to be tested by calling a pre-set dedicated API, i.e., a preset API. The GUI control tree includes all control information in the GUI currently displayed by the software to be tested when the tester operates the software to be tested, such as the position, type, name, index value, etc. of the control in the GUI interface. Accordingly, a node in the GUI control tree corresponds to the control information of a control object in the GUI interface.

[0086] S202: Obtain mouse position information when the tester operates the software to be tested.

[0087] Since the control information is related to the position information of the mouse when the tester operates the software to be tested, in this step, in order to obtain the control information related to the tester's operation, it is also necessary to obtain the position information of the mouse when the tester operates the software to be tested.

[0088] Among them, the mouse position information refers to the position coordinate information of the mouse positioning cursor on the GUI. For example, it can be expressed as (x, y), where x represents the coordinate of the mouse positioning cursor in the horizontal direction of the GUI, and y represents the coordinate of the mouse positioning cursor in the vertical direction of the GUI.

[0089] In this step, the mouse position information may be acquired through a pre-written mouse position capture algorithm or a mouse position capture tool installed on the device.

[0090] It is understandable that this step can be executed before or after S201, or simultaneously with S201. There is no strict execution order between S202 and S201, and no limitation is made here.

[0091] S203: According to the mouse position information, traverse the control objects in the GUI control tree to obtain control information.

[0092] In this step, based on the mouse position information obtained in S202, all nodes (control objects) in the entire GUI control tree are searched in a depth-first traversal manner to obtain control information relative to the mouse position information.

[0093] Specifically, determine whether there is a target control object matching the mouse position information in the GUI control tree, that is, a control object consistent with the mouse position information. If so, obtain the information of the target control object and determine the information of the target control object as the control information for the tester to operate the software to be tested.

[0094] Exemplarily, the control identification algorithm provided in this embodiment can be expressed as:

[0095]

[0096] Some GUI control objects can only be displayed in the corresponding GUI after responding to specific user events. In order to ensure that all possible GUI control objects can be traversed, generate reliable test scripts, and improve the coverage of GUI automated testing, further, if there is no target control object matching the mouse position information in the GUI control tree, in this embodiment, a pre-set dynamic traversal algorithm is adopted to send a user event to the GUI of the software under test to trigger the display of all hidden control objects in the GUI, and then all hidden control objects in the GUI are traversed according to the mouse position information to obtain the control information of the tester operating the software under test.

[0097] In one possible implementation, Figure 4 A flow chart of the dynamic traversal algorithm provided in the second embodiment of the present application is shown in FIG. Figure 4As shown, in this implementation, the dynamic traversal algorithm includes:

[0098] S301: Obtain a first GUI state of the software to be tested.

[0099] To facilitate distinction, the current GUI state obtained before sending a user event to the GUI of the software under test is called the first GUI state, and the window state set corresponding to the first GUI state is called the first window state set. The first window state set includes the window state of the first top-level active window corresponding to the first GUI state and the window states of other windows except the first top-level active window.

[0100] S302: Determine, according to the first GUI state, to send a user event sequence to the GUI of the software to be tested.

[0101] In this embodiment, all GUI states and user events related to the hidden control objects of the software to be tested can be acquired in advance, and a state sequence table and an event sequence table can be generated according to the order in which the GUI states appear and the correspondence between the GUI states and the user events.

[0102] Accordingly, in this step, according to whether the top-level active window in the current GUI state (ie, the first GUI state) contains a configuration file, different methods are used to obtain the user event sequence for GUI state traversal.

[0103] To meet the needs of dynamic traversal, in this embodiment, for the case where the name of the top-level active window is not fixed, a user action sequence is defined in advance, the sequence includes the control path and event information of the operated control, and is stored in the configuration file. Here, by searching the window state of the top-level active window in the first window state set, that is, the top-level active window {O, P, V} triple, it can be determined whether the top-level active window of the current GUI contains the configuration file.

[0104] Specifically, if it is included, the user event sequence is obtained from the configuration file; if it is not included, the position index of the first GUI state in the state sequence table is obtained, and based on the position index, the user event sequence is obtained from the event sequence table. It can be understood that the user event sequence includes the user event corresponding to the position index in the event sequence table, the last user event in the event sequence table, and all user events between the user event corresponding to the position index and the last user event.

[0105] S303: Sending user events to the GUI of the software under test according to the user event sequence to trigger display of hidden control objects in the GUI.

[0106] In this step, according to the user event sequence obtained in S302, a pre-set dynamic detection strategy is used to send user events to the GUI of the software under test to trigger the display of hidden control objects in the GUI. That is, a user event is sent to a control in the current GUI state. After the software under test completes the user event response, the current GUI state is extracted again, and the next user event is triggered. The process is repeated until the traversal of all GUI states of the software under test is completed, thereby achieving the purpose of displaying all hidden control objects in the GUI.

[0107] To further illustrate the implementation principle of S303 in this embodiment, illustratively, Figure 5 A traversal flow diagram provided in Example 2 of the present application, such as Figure 5 As shown, in this embodiment, the traversal process based on the user event sequence includes:

[0108] S401: Obtain a target user event from a user event sequence.

[0109] The target user event may be any user event in the user event sequence. It is understandable that, at the beginning of the traversal, the target user event is the first user event in the user event sequence, and in the subsequent traversal process, the target user event is the next user event of the current user event.

[0110] S402: Send a target user event to the GUI.

[0111] In this step, the target user event is sent to a certain control object in the first GUI state, and the control object may be a control object associated with the target user event.

[0112] S403: When the GUI responds to the target user event, a second GUI state of the software to be tested is obtained.

[0113] In this step, when it is monitored that the GUI responds to or executes the target user event, the current GUI state of the software under test is reacquired. For the sake of distinction, the current GUI state obtained after sending the target user event to the GUI of the software under test is called the second GUI state, and the window state set corresponding to the second GUI state is called the second window state set. The second window state set includes the window state of the second top-level active window corresponding to the second GUI state and the window states of other windows except the second top-level active window.

[0114] S404: Determine whether GUI state migration occurs in the software to be tested according to the second window state set and the first window state set.

[0115] In this step, by comparing the second window state set of the second GUI state with the first window state set of the first GUI state, it is determined whether the GUI state of the software under test has been migrated. If migration has occurred, it means that the second GUI state is different from the first GUI state, then S405 is executed, and after S405 is executed, S401 is returned to be executed, that is, new target user events are continued to be obtained, and the traversal of all GUI states is completed. If migration has not occurred, it means that the second GUI state is different from the first GUI state, and it can be seen that the target user event is a termination event, then S406 is executed.

[0116] It can be understood that the essence of this step is to determine whether two adjacent GUI states are the same, that is, whether the new GUI state is the same as the previous GUI state. When the process of S401-S404 is repeatedly executed, the newly obtained GUI state can be called the third GUI state, the fourth GUI state, the fifth GUI state, etc.

[0117] S405: Store the first GUI state in a transition state list in the cache, and store the target user event in a transition event list in the cache.

[0118] In this step, by storing the first GUI state and the target user event in the cache, it can be ensured that the first GUI state and the target user event will not be repeatedly executed in the subsequent traversal process, thereby improving the traversal efficiency.

[0119] S406: Mark the target user event as a termination event.

[0120] Through the above S401-S406, during the traversal process, by judging whether the two GUI states are the same, and storing the traversed GUI states and the corresponding migration events in the cache, it can be ensured that no loop problem occurs during the traversal process, that is, an infinite loop occurs between the two GUI states, thereby improving the traversal efficiency.

[0121] To further illustrate the implementation principle of S404, illustratively, Figure 6 A schematic diagram of the judgment process of GUI state migration provided in the second embodiment of the present application is shown in FIG. Figure 6 As shown, in this implementation, the following steps are used to determine whether the software under test has a GUI state transition:

[0122] S501. Acquire a first window status set and a second window status set.

[0123] In this step, a first window state set corresponding to the first GUI state and a second window state set corresponding to the second GUI state are respectively obtained.

[0124] S502: Determine whether the number of window states in the second window state set is the same as that in the first window state set.

[0125] Since the window state set includes the window state of the top-level active window and the window states of other windows, and the GUI state migration is usually reflected in the number of window states (i.e., the number of windows) or the difference in the top-level active window, it is relatively easy to determine the number of window states. Therefore, in this step, first determine whether the two GUI states are the same according to whether the numbers of window states in the window state sets corresponding to different GUI states are the same. If they are the same, execute S503 for further determination. If they are different, execute S505.

[0126] S503: According to the second window state set and the first window state set, respectively obtain a second top-level active window control tree of the second GUI state and a first top-level active window control tree of the first GUI state.

[0127] When it is determined according to S502 that the number of window states in the window state sets of the two GUI states is the same, in this step, further, the corresponding top-level active window control trees are generated respectively according to the window state sets of the two GUI states, so as to determine whether the two GUI states are the same according to the controls in the top-level active window. For the convenience of distinction, the top-level active window control tree corresponding to the first GUI state, that is, the top-level active window control tree generated according to the first window state set, is called the first top-level active window control tree; the top-level active window control tree corresponding to the second GUI state, that is, the top-level active window control tree generated according to the second window state set, is called the second top-level active window control tree.

[0128] S504: Determine whether the control objects in the second top-level active window control tree are the same as those in the first top-level active window control tree.

[0129] Based on S503, in this step, it is determined whether the control objects in the top-level active windows corresponding to the two GUI states are the same according to the two top-level active window control trees obtained. Here, it is mainly determined whether the number and type of control objects are the same. If it is determined that the number and type of control objects in the two control trees are the same, it means that the control objects in the two top-level active windows are the same, and S506 is executed. Otherwise, it means that the control objects in the two top-level active windows are not the same, and S505 is executed.

[0130] S505: Determine whether a GUI state transition occurs in the software to be tested.

[0131] S506: Determine that no GUI state migration occurs in the software to be tested.

[0132] In this embodiment, when the conventional traversal method cannot obtain the control information when the tester operates the software to be tested, a pre-set dynamic traversal algorithm is used to send user events to the GUI of the software to be tested to trigger the display of all hidden control objects in the GUI, and then all hidden control objects in the GUI are traversed according to the mouse position information to ensure that all control objects in the GUI are traversed and reliable control information is obtained, thereby improving the usability and reliability of the generated test script, thereby improving the coverage of GUI automated testing, and then improving the accuracy of GUI automated testing.

[0133] Optionally, before S302, in the present embodiment, it is also possible to first determine whether the first GUI state exists in the migration state list in the cache. If not, it means that the first GUI state has not been traversed, and S302-S303 is continued to be executed. If so, it means that the first GUI state has been traversed, and the corresponding migration event can be deleted from the user event sequence, even if the GUI of the software to be tested changes to the migration event of the first GUI state, thereby ensuring that the traversal process can proceed smoothly during subsequent testing and improving the adequacy of the traversal.

[0134] Optionally, before S302, in the present embodiment, the top-level active window value of the first GUI state may be obtained first, and the obtained top-level active window style value may be compared with the top-level active window style value in the style blacklist to determine whether the top-level active window value of the first GUI state is included in the style blacklist; if not, execute S302-S303; otherwise, delete the corresponding migration event from the user event sequence.

[0135] In this implementation, the top-level active window style can be used as an attribute of the top-level active window and described by the top-level active window style value. Accordingly, in this implementation, the top-level active window value of the first GUI state can be obtained according to the window state set of the first GUI state. The top-level active window value of the first GUI state can also be obtained by calling related tools, such as Spy++ provided by Microsoft, which is not limited here.

[0136] Among them, the style blacklist contains some GUI window style values ​​that are irrelevant to the main functions of the tested software, such as non-essential function GUI window style values. By setting the style blacklist, it is possible to effectively prevent the traversal program from being limited to an unimportant GUI window, thereby further improving the traversal efficiency.

[0137] For example, Figure 7 A flowchart diagram of another dynamic traversal algorithm is provided for the second embodiment of the present application. In the specific implementation process, Figure 7 As shown, the following specific steps can be used to implement a trigger-based depth-first traversal of the control tree:

[0138] S601: Initialize the migration state list and the migration event list in the cache, obtain the current GUI state and the current trigger event, and store them in the migration state list and the migration event list respectively.

[0139] S602: Determine whether the migration status list is empty.

[0140] In this step, by judging whether the migration state list is empty, it is determined whether the current GUI state is successfully acquired. If successful, S603 is executed, otherwise, the process is terminated.

[0141] S603: Whether the current GUI state contains a configuration file.

[0142] In this step, it is determined whether the current GUI state is associated with a configuration file according to the window state set of the current GUI state. The configuration file contains the corresponding user event sequence. If so, S604 is executed. If not, S605 is executed.

[0143] S604: trigger a user event in the configuration file, and record the user event to ensure that it will not be repeated in subsequent traversal processes.

[0144] S605: Using a tree-first strategy (DFS) to traverse the control nodes, ie, control objects, of the top-level active window in the current GUI state.

[0145] S606: Determine whether there are traversable control nodes and triggerable user events in the current GUI state. If both exist, execute S610; otherwise, execute S607-S609.

[0146] S607: Mark the current GUI status as traversed.

[0147] In this step, the current GUI state can be marked as traversed by modifying the state parameters of the current GUI state, such as curState.isDone=true.

[0148] S608: Trigger a termination event of the current GUI state.

[0149] S609: Return a new GUI state (NewGUIState) which is an empty set.

[0150] S610: Save the current trigger event as the last event (lastEvent) attribute of the current GUI state, and trigger this event to obtain a new GUI state.

[0151] S611: Determine whether the GUI states before and after the triggering event occurs are the same GUI state.

[0152] In this step, the method of S501-S506 can be used to determine whether the GUI state before and after the trigger event is the same GUI state, that is, whether a GUI state migration occurs. If not, execute S612, otherwise, execute S617.

[0153] S612: Return the new GUI state (NewGUIState), which is not empty.

[0154] S613: Determine whether the acquired new GUI state (NewGUIState) is an empty set. If it is an empty set, execute S614; otherwise, execute S615.

[0155] S614: Remove the last GUI state in the migration state list.

[0156] S615: Determine whether the acquired new GUI state (NewGUIState) belongs to the migration state list. If so, execute S617; otherwise, execute S616.

[0157] S616. Add the latest NewGUIState to the state transition list, and add the transition event of NewGUIState to the event list.

[0158] S617: Mark the current GUI state transition event as a termination event.

[0159] In this embodiment, the GUI control tree of the software to be tested is extracted through a preset API, and the GUI control tree includes all control information in the GUI. The mouse position information of the tester when operating the software to be tested is obtained, and based on the mouse position information, the control objects in the GUI control tree are traversed to obtain the control information, thereby realizing API-driven control recognition, improving the accuracy of control recognition, and further improving the accuracy of GUI automated testing, which is beneficial to improving the performance of the software to be tested and enhancing the user experience.

[0160] Embodiment 3

[0161] For example, Figure 8 A flowchart of a control identification method provided in Embodiment 3 of the present application is shown in FIG. Figure 8 As shown, in this embodiment, if the software to be tested supports image driver, the control information of the software to be tested operated by the tester is obtained through the following control recognition algorithm:

[0162] S701. Obtain GUI screenshots and mouse position information when the tester operates the software to be tested.

[0163] In this embodiment, a control identification algorithm is provided to obtain control information based on GUI screenshots and mouse position information. To this end, in this step, it is necessary to first obtain the GUI screenshots and mouse position information corresponding to each time the tester operates the software to be tested.

[0164] In this embodiment, the GUI screenshot can be obtained by manual screenshot or automatic screenshot. Manual screenshot means that the user manually selects the size and position of the control screenshot and manually selects the event to be triggered; automatic screenshot means that when the user normally operates the software under test, the program automatically obtains a control screenshot of a suitable size and position and automatically records the user event. Fig. 9 The logical diagram of obtaining GUI screenshots based on automatic screenshots and manual screenshots provided in Example 3 of this application. In actual operation, manual screenshots and automatic screenshots can be selected according to the scene and needs to obtain GUI screenshots and mouse position information, which is not limited here.

[0165] To improve the degree of automation and reduce manual involvement, optionally, in this step, a control area detection algorithm based on software interface screenshots obtains GUI screenshots when the tester operates the software to be tested. Each time the tester operates the software to be tested, the control area detection algorithm can automatically capture a GUI screenshot of a suitable size and position, which generally contains text information and image features.

[0166] In this step, the method of representing and obtaining the mouse position information is similar to that in S202, and reference may be made to the description of S202, which will not be repeated here.

[0167] S702: Using a preset image processing algorithm, perform image processing on the GUI screenshot to obtain a standard GUI screenshot.

[0168] To facilitate the acquisition of control information, in this step, it is necessary to first use a preset image processing algorithm to process the original GUI screenshot acquired in S701 to obtain a GUI screenshot that meets the requirements for extracting control information.

[0169] In a possible implementation, in this step, the image processing of the GUI screenshot includes binarization processing and noise reduction processing to obtain a standard GUI screenshot.

[0170] Optionally, in this embodiment, the cvtColor function is first used to convert the color GUI screenshot into a grayscale image, and then the Sobel function is used to convolve each pixel in the grayscale image in the x and y directions, and the convolution results are added to obtain the grayscale approximate value of each pixel point, and then the grayscale image is converted into a binary image based on the grayscale approximate value of each pixel point and a preset threshold, so as to obtain a contour binary image of the GUI screenshot.

[0171] Optionally, in this embodiment, based on mathematical morphology, the pixels in the contour binary image are eroded and expanded a certain number of times in the x and y directions to perform noise reduction on the contour binary image to obtain a standard GUI screenshot.

[0172] Specifically, a 3*1 convolution kernel can be used to perform several erosion operations on the contour binary image, and then several dilation operations can be performed on the contour binary image, so as to obtain long white line segments in the horizontal direction of the contour binary image; a 1*3 convolution kernel can be used to perform several erosion operations on the contour binary image, and then several dilation operations can be performed on the contour binary image, so as to obtain long white line segments in the vertical direction of the contour binary image; finally, the contour binary image is used to subtract these long white line segments (noise), and a closing operation is performed to obtain a prominent white rectangular frame.

[0173] Mathematical morphology is widely used as a branch of image analysis. It predefines structural elements with certain shapes and sizes, and uses these structural elements to extract the corresponding shapes in the image, thereby achieving the purpose of image analysis and recognition. Mathematical morphology has two basic operations, namely dilation and erosion:

[0174] Dilation operation: First, define a convolution kernel B of any shape, usually a 3*3 square, whose center point is called the anchor point. When dilating image A, slide convolution kernel B over image A and find the maximum pixel value in the convolution kernel area as the anchor point pixel value. Thus, the final effect is to enlarge the highlight area in the image. In the dilation process, the convolution kernel is equivalent to a filter. Its main function is to fill certain holes in the target area and eliminate small particle noise contained in the target area.

[0175] Corrosion operation: Corresponding to the dilation operation, when the image is corroded, the convolution kernel B will slide over the image A and find the minimum pixel value inside the kernel area as the anchor pixel value. As a result, the final effect is to reduce the highlight area in the image and expand the black area. The main function is to extract the backbone information and remove small and meaningless targets such as burrs and isolated pixels.

[0176] The erosion operation and the dilation operation are combined to form the opening operation and the closing operation. The opening operation first erodes the image and then dilates it. It is mainly used to eliminate small objects, smooth the boundaries of larger objects, and disconnect narrow connections without significantly changing the original image area. The closing operation first dilates the image and then erodes it. It is mainly used to effectively fill the black holes inside the object and connect adjacent objects without significantly changing the original image area.

[0177] In this embodiment, the standard GUI screenshot obtained by the above algorithm processing includes some prominent white rectangular boxes, and these white rectangular boxes are the edges of the controls in the identified GUI screenshot.

[0178] S703: Determine a control screenshot according to the mouse position information and the standard GUI screenshot.

[0179] In this step, the white rectangular frame corresponding to the standard GUI screenshot is obtained according to the mouse position information, and a screenshot of the position of the white rectangular frame is taken in the standard GUI screenshot to obtain a control screenshot.

[0180] S704: Determine control information according to the center point coordinate information of the control screenshot.

[0181] In this step, the center point of the control screenshot obtained in S703 is identified, the coordinate information of the center point is determined, and the center point coordinate information is used as the position of the control operated by the user in the GUI, that is, the control information.

[0182] Exemplarily, the specific steps of the control identification algorithm provided in this embodiment are as follows:

[0183] Step 1: Get a GUI screenshot when the user operates the software to be tested, and get the coordinates of the mouse in the GUI screenshot, that is, the mouse position information.

[0184] Step 2: Convert the GUI screenshot into a grayscale image using the cvtColor function.

[0185] Step 3: Use the Sobel function to perform edge detection calculation on the grayscale image obtained in step 2.

[0186] Among them, the Sobel function is an image edge detection algorithm widely used in computer graphics. It mainly uses two 3*3 matrices to perform convolution operations on the original image, thereby calculating the estimated values ​​of the grayscale values ​​in the horizontal and vertical directions, that is, the grayscale approximation.

[0187] Exemplarily, the Sobel function first convolves each pixel in the grayscale image in the x-direction and the y-direction using the following formula:

[0188]

[0189]

[0190] Among them, A is the pixel point in the grayscale image, G x , G y They are the grayscale partial derivatives in the horizontal and vertical directions, that is, the planar convolution results of the grayscale image in the horizontal and vertical directions.

[0191] Then, the grayscale approximation of each pixel is obtained through the following formula:

[0192] G=|G x |+|G y | (3)

[0193] The Sobel function detects edges by using the phenomenon that the weighted difference in grayscale between the upper and lower and left and right neighboring points of a pixel reaches an extreme value at the edge, and can provide more accurate edge direction information.

[0194] Step 4: Binarize the grayscale image after the calculation in step 3 to obtain a contour binary image. If the grayscale approximate value of each pixel in the processed grayscale image is compared with the preset threshold, the grayscale value greater than the preset threshold is recorded as 255, and the grayscale value less than the preset threshold is recorded as 0, and the contour binary image is obtained.

[0195] Step 5: Perform a certain number of erosion and expansion processes on the contour binary image obtained in step 4 in the x direction and the y direction to obtain long white line segments in the vertical direction and the horizontal direction in the contour binary image.

[0196] Step 6: Remove the long line segments in the contour binary image.

[0197] Step 7: Perform a closing operation on the contour binary image obtained in step 6 to obtain a protruding white rectangular frame, and eliminate the black dots inside the white rectangular frame in the contour binary image, so that the white rectangular frame protrudes into a complete rectangle.

[0198] Step 8: Identify all the prominent white rectangular boxes obtained in step 7 and filter out some small rectangular noises.

[0199] Step 9: According to the mouse position information, obtain the corresponding white rectangular box (such as the white rectangular box that coincides with the mouse position or is closest to the mouse position), and take a screenshot of the rectangular box position in the GUI screenshot to obtain a control screenshot.

[0200] Step 10: Determine whether the white rectangular box obtained in step 9 is unique in the GUI screenshot. If so, jump to step 12; otherwise, jump to step 11.

[0201] Step 11: Within the GUI screenshot range, expand the top, bottom, left, and right sides of the white rectangular frame obtained in step 9 by 10 pixels respectively, and take a new screenshot and return to step 10.

[0202] Step 12: Determine the center point coordinates of the control screenshot, record them, and obtain the control information.

[0203] In the implementation process, this embodiment can use OpenCV related functions to implement the above algorithm process. OpenCV has a C++ open source library that can support the implementation of computer vision related technologies, including bottom-level image processing, middle-level image analysis, and high-level visual technology.

[0204] Specifically, first, Java is used to call OpenCV, so OpenCV needs to be initialized and the dll library of OpenCV needs to be loaded before the main program is driven. Different Windows operating environments load different dll libraries in OpenCV. Secondly, after loading the dll library, a GUI screenshot of the application software to be tested is obtained, and the GUI screenshot is saved in Mat format. Mat format is a basic data type of OpenCV, which is a multi-dimensional and multi-channel array. Then, related functions are used to process the GUI screenshot, including cvtColor function, Sobel function, dilate function, erode function, and findContours function.

[0205] Among them, the cvtColor function is used to convert the image from one color space to another color space, such as realizing the conversion of RGB color to color spaces such as HSV and HSI, and can also be converted to a grayscale image. The Sobel function is used to find the derivative of the image in the x or y direction to realize edge detection calculation, the dilate function dilates the image, the erode function corrodes the image, and the findContours function is used to identify and detect the white rectangular box in the image. In this embodiment, the GUI screenshot and mouse position information when the tester operates the software to be tested are obtained, and the GUI screenshot is image processed using a preset image processing algorithm to obtain a standard GUI screenshot. According to the mouse position information and the standard GUI screenshot, the control screenshot is determined, and the control information is determined according to the center point coordinate information of the control screenshot, so as to realize the control recognition based on the image drive, improve the accuracy of the control recognition, and then improve the accuracy of the GUI automation test, which is conducive to improving the performance of the software to be tested and enhancing the user's experience.

[0206] Embodiment 4

[0207] Fig.10 A schematic diagram of the structure of a black box automated testing device based on a graphical user interface provided in Example 4 of the present application, such as Fig.10 As shown, the black box automated testing device 10 based on a graphical user interface in this embodiment includes:

[0208] Recording module 11 and playback module 12.

[0209] The recording module 11 is used to obtain the event information and control information of the tester operating the software to be tested; and generate a test script according to the event information, the control information and the preset assertion;

[0210] The playback module 12 is used to run the test script on the driving engine, perform a graphical user interface (GUI) test on the software to be tested, and generate a test report.

[0211] Optionally, the recording module 11 is specifically used for:

[0212] By monitoring the input devices, the event information of the tester operating the software under test is recorded;

[0213] Through the control identification algorithm, the control information used by the tester to operate the software under test is obtained.

[0214] Optionally, if the software to be tested supports an application program interface API driver, the recording module 11 is specifically used for:

[0215] Extracting the GUI control tree of the software under test through a preset API, wherein the GUI control tree includes all control objects of the software under test;

[0216] Obtaining mouse position information when the tester operates the software to be tested;

[0217] According to the mouse position information, the control objects in the GUI control tree are traversed to obtain the control information.

[0218] Optionally, the recording module 11 is specifically used for:

[0219] Determine whether there is a target control object matching the mouse position information in the GUI control tree;

[0220] If it exists, the information of the target control object is obtained to obtain the control information.

[0221] Optionally, the recording module 11 is further used for:

[0222] If it does not exist, a preset dynamic traversal algorithm is used to send a user event to the GUI of the software under test to trigger the display of the hidden control object in the GUI;

[0223] The control information is obtained according to the mouse position information and the hidden control object in the GUI.

[0224] Optionally, the recording module 11 is specifically used for:

[0225] Acquire a first GUI state of the software to be tested, where the first GUI state includes a first window state set of the current GUI;

[0226] Determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state;

[0227] According to the user event sequence, a user event is sent to the GUI of the software to be tested to trigger the display of hidden control objects in the GUI.

[0228] Optionally, the recording module 11 is specifically used for:

[0229] Determining, according to the first window state set, whether the top-level active window of the current GUI includes a configuration file;

[0230] If included, obtaining the user event sequence from the configuration file;

[0231] If not included, determine the position index of the first GUI state in the state sequence table, and obtain the user event sequence from the event sequence table according to the position index.

[0232] Optionally, the recording module 11 is specifically used for:

[0233] Acquire a target user event from the user event sequence;

[0234] Sending the target user event to the GUI;

[0235] When the GUI responds to the target user event, obtaining a second GUI state of the software under test, wherein the second GUI state includes a second window state set of the current GUI;

[0236] Determining whether a GUI state migration occurs in the software to be tested according to the second window state set and the first window state set;

[0237] If migration occurs, the first GUI state and the target user event are stored in a cache migration state list and a migration event list respectively, and the next target user event is obtained from the user event list.

[0238] Optionally, the recording module 11 is specifically used for:

[0239] Determining whether the second window state set has the same number of window states as the first window state set;

[0240] If the number of window states is different, it is determined that the software under test has a GUI state migration;

[0241] If the number of window states is the same, obtaining the second top-level active window control tree of the second GUI state and the first top-level active window control tree of the first GUI state respectively according to the second window state set and the first window state set;

[0242] Determine whether the control objects in the second top-level active window control tree are the same as those in the first top-level active window control tree;

[0243] If the control objects are different, it is determined that the software under test has a GUI state migration.

[0244] Optionally, the recording module 11 is further used for:

[0245] Determine whether the first GUI state exists in the migration state list in the cache;

[0246] Accordingly, the recording module 11 is specifically used for:

[0247] If it is determined that the first GUI state does not exist in the migration state list, a user event sequence sent to the GUI of the software under test is determined according to the first GUI state.

[0248] Optionally, the recording module 11 is further used for:

[0249] Obtaining the top-level active window style value of the first GUI state;

[0250] Determine whether the top-level active window style value is in a style blacklist;

[0251] Accordingly, the recording module 11 is specifically used for:

[0252] If the top-level active window style value is not in the style blacklist, a user event sequence to be sent to the GUI of the software under test is determined according to the first GUI state.

[0253] Optionally, if the software to be tested supports image driver, the recording module 11 is specifically used for:

[0254] Obtain GUI screenshots and mouse position information when the tester operates the software under test;

[0255] Using a preset image processing algorithm, performing image processing on the GUI screenshot to obtain a standard GUI screenshot;

[0256] Determining a control screenshot according to the mouse position information and the standard GUI screenshot;

[0257] The control information is determined according to the center point coordinate information of the control screenshot.

[0258] Optionally, the recording module 11 is specifically used for:

[0259] The GUI screenshot is converted by a target function to obtain a contour binary image of the GUI screenshot, wherein the target function includes a cvtColor function and a Sobel function;

[0260] The contour binary image is subjected to noise reduction processing through morphological operation to obtain the standard GUI screenshot, wherein the morphological operation includes a dilation operation and an erosion operation.

[0261] Optionally, the recording module 11 is specifically used for:

[0262] Combining the event information and the control information to generate test steps;

[0263] Combining the control information and target assertion to generate a step checkpoint;

[0264] The test steps and the step checkpoints are combined to generate the test script.

[0265] Optionally, the preset assertions include control-level assertions, image assertions, and combined assertions.

[0266] For example, Fig.11 A schematic diagram of the structure of another black box automated testing device based on a graphical user interface provided in Example 4 of the present application, such as Fig.11 As shown, the black box automated testing device 10 based on a graphical user interface provided in this embodiment further includes: a script management module 13 .

[0267] In this embodiment, the test script is optimized from the aspects of data-driven and modularization by setting the script management module 13. For example, Fig.12 The schematic diagram of the design architecture of the script management module provided in the fourth embodiment of the present application is as follows: Fig.12 As shown, in this embodiment, the test script can be divided into a logic part and a data part through the script management module 13. The logic part includes information such as event type and control path. The data part includes variable data that needs to be compared, such as input variables or assertions in the test steps.

[0268] The script management interface establishes mapping management between the script management interface and the data of the data driver layer. Testers can modify the variable data in the automated test script through the script management interface GUI.

[0269] Data-driven development mainly involves establishing a mapping relationship between the script's test steps and variable data, thereby achieving the separation of test logic and variable data.

[0270] Script modularization mainly allows a script to execute the test steps of another script by reference. Through appropriate modular script organization, a coupling relationship can be established between script functions. Scripts in different modules can reference each other, making the test script more extensible.

[0271] In this embodiment, in order to realize the data-driven and modularization of the script management module 13 and enable the script to have the ability of data separation and dynamic expansion, an XML file structure is used to store test steps and test variable data respectively. Since XML files can be structured accessed through the document object model (DOM), CURD atomic operations can be quickly implemented on XML files. In the playback stage, the test step file and the test variable file are organized together to form a complete automated test step.

[0272] In addition, this embodiment extracts variable data in the variable file through DOM, and maps these data to the graphical user interface of script management. Testers can modify the data part through the GUI interface to realize data-driven script management. In the file of the test step, the script module reference is defined through the Module tag. When converted into a script object, if the Module tag is encountered, the referenced script step will be added to the corresponding script object step, thereby realizing modularization. In the implementation process, the API provided by the JAR package of DOM4J is mainly used to parse the XML file.

[0273] The black box automated testing device based on a graphical user interface provided in this embodiment can execute the black box automated testing method based on a graphical user interface provided in the above method embodiment, and has the corresponding functional modules and beneficial effects of the execution method. The implementation principle and technical effects of this embodiment are similar to those of the above method embodiment, and will not be repeated here one by one.

[0274] Embodiment 5

[0275] Fig.13 A schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present application is shown in FIG. Fig.13 As shown, the electronic device 20 includes a memory 21, a processor 22, and a computer program stored in the memory and executable on the processor; the number of the processor 22 of the electronic device 20 may be one or more. Fig.13 A processor 22 is taken as an example; the processor 22 and the memory 21 in the electronic device 20 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0276] The memory 21, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the recording module 11 and the playback module 12 in the embodiment of the present application. The processor 22 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 21, that is, to implement the above-mentioned black box automated testing method based on the graphical user interface.

[0277] The memory 21 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 21 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 21 may further include a memory remotely arranged relative to the processor 22, and these remote memories may be connected to the electronic device via a grid. Examples of the above-mentioned grid include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0278] Embodiment 6

[0279] Embodiment 6 of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer processor, the computer program is used to execute a black box automated testing method based on a graphical user interface, the method comprising:

[0280] Obtain event information and control information of the tester's operation of the software under test;

[0281] Generate a test script according to the event information, the control information and the preset assertion;

[0282] The test script is run on the driving engine to perform a graphical user interface (GUI) test on the software to be tested and a test report is generated.

[0283] Of course, the computer-readable storage medium provided in the embodiment of the present application, whose computer program is not limited to the method operations described above, can also execute related operations in the black box automated testing method based on a graphical user interface provided in any embodiment of the present application.

[0284] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or grid device, etc.) to execute the methods described in each embodiment of the present application.

[0285] It is worth noting that in the above-mentioned embodiment of the black box automatic testing device based on a graphical user interface, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0286] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A black box automated testing method based on a graphical user interface, characterized in that: include: Obtain event information and control information of the tester's operation of the software under test; Generate a test script according to the event information, the control information and the preset assertion; Running the test script on the driver engine, performing a graphical user interface (GUI) test on the software to be tested, and generating a test report; The obtaining of event information and control information of the tester operating the software to be tested includes: By monitoring the input devices, the event information of the tester operating the software under test is recorded; Through the control identification algorithm, the control information of the tester operating the software under test is obtained; If the software to be tested supports an application program interface API driver, the control information of the software to be tested operated by the tester is obtained through a control identification algorithm, including: Extracting the GUI control tree of the software under test through a preset API, wherein the GUI control tree includes all control objects of the software under test; Obtaining mouse position information when the tester operates the software to be tested; According to the mouse position information, traverse the control objects in the GUI control tree to obtain the control information; The step of traversing the control objects in the GUI control tree according to the mouse position information to obtain the control information includes: Determine whether there is a target control object matching the mouse position information in the GUI control tree; If it exists, then obtain the information of the target control object to obtain the control information; If it does not exist, a preset dynamic traversal algorithm is used to send a user event to the GUI of the software under test to trigger the display of the hidden control object in the GUI; Obtaining the control information according to the mouse position information and the hidden control object in the GUI; The method of using a preset dynamic traversal algorithm to send a user event to the GUI of the software to be tested to trigger the display of hidden control objects in the GUI includes: Acquire a first GUI state of the software to be tested, where the first GUI state includes a first window state set of the current GUI; Determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state; According to the user event sequence, a user event is sent to the GUI of the software to be tested to trigger the display of hidden control objects in the GUI.

2. The method according to claim 1, characterized in that The step of determining a user event sequence to be sent to the GUI of the software to be tested according to the first GUI state includes: Determining, according to the first window state set, whether the top-level active window of the current GUI includes a configuration file; If included, obtaining the user event sequence from the configuration file; If not included, determine the position index of the first GUI state in the state sequence table, and obtain the user event sequence from the event sequence table according to the position index.

3. The method according to claim 2, characterized in that The sending of a user event to the GUI of the software to be tested according to the user event sequence to trigger display of a hidden control object in the GUI comprises: Acquire a target user event from the user event sequence; Sending the target user event to the GUI; When the GUI responds to the target user event, obtaining a second GUI state of the software under test, wherein the second GUI state includes a second window state set of the current GUI; Determining whether a GUI state migration occurs in the software to be tested according to the second window state set and the first window state set; If migration occurs, the first GUI state and the target user event are stored in a cache migration state list and a migration event list respectively, and the next target user event is obtained from the user event list.

4. The method according to claim 3, characterized in that The step of determining whether a GUI state migration occurs in the software to be tested according to the second window state set and the first window state set includes: Determining whether the second window state set has the same number of window states as the first window state set; If the number of window states is different, it is determined that the software under test has a GUI state migration; If the number of window states is the same, obtaining the second top-level active window control tree of the second GUI state and the first top-level active window control tree of the first GUI state respectively according to the second window state set and the first window state set; Determine whether the control objects in the second top-level active window control tree are the same as those in the first top-level active window control tree; If the control objects are different, it is determined that the software under test has a GUI state migration.

5. The method according to claim 1, characterized in that Before determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state, the method further comprises: Determine whether the first GUI state exists in the migration state list in the cache; Correspondingly, determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state includes: If it is determined that the first GUI state does not exist in the migration state list, a user event sequence sent to the GUI of the software under test is determined according to the first GUI state.

6. The method according to claim 1, characterized in that Before determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state, the method further comprises: Obtaining the top-level active window style value of the first GUI state; Determine whether the top-level active window style value is in a style blacklist; Correspondingly, determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state includes: If the top-level active window style value is not in the style blacklist, a user event sequence to be sent to the GUI of the software under test is determined according to the first GUI state.

7. The method according to claim 1, characterized in that If the software to be tested supports image driver, the control information of the software to be tested operated by the tester is obtained through the control recognition algorithm, including: Obtain GUI screenshots and mouse position information when the tester operates the software under test; Using a preset image processing algorithm, performing image processing on the GUI screenshot to obtain a standard GUI screenshot; Determining a control screenshot according to the mouse position information and the standard GUI screenshot; The control information is determined according to the center point coordinate information of the control screenshot.

8. The method according to claim 7, characterized in that The method of using a preset image processing algorithm to perform image processing on the GUI screenshot to obtain a standard GUI screenshot includes: The GUI screenshot is converted by a target function to obtain a contour binary image of the GUI screenshot, wherein the target function includes a cvtColor function and a Sobel function; The contour binary image is subjected to noise reduction processing through morphological operation to obtain the standard GUI screenshot, wherein the morphological operation includes a dilation operation and an erosion operation.

9. The method according to claim 1, characterized in that: The generating of a test script according to the event information, the control information and the preset assertion includes: Combining the event information and the control information to generate test steps; Combining the control information and target assertion to generate a step checkpoint; The test steps and the step checkpoints are combined to generate the test script.

10. The method according to any one of claims 1 to 9, characterized in that: The preset assertions include control-level assertions, image assertions, and combined assertions.

11. The method according to any one of claims 1 to 9, characterized in that: After generating a test script according to the event information, the control information and the preset assertion, the method further includes: Splitting the test script into a logic part and a data part, and establishing a mapping relationship between the logic part and the data part; The logic part, the data part and the mapping relationship are stored separately.

12. A black box automated testing device based on a graphical user interface, characterized in that: include: The recording module is used to obtain the event information and control information of the tester operating the software under test; Generate a test script according to the event information, the control information and the preset assertion; A playback module, used to run the test script on the driving engine, perform a graphical user interface (GUI) test on the software to be tested, and generate a test report; The recording module is used for: By monitoring the input devices, the event information of the tester operating the software under test is recorded; Through the control identification algorithm, the control information of the tester operating the software under test is obtained; If the software under test supports application program interface API driver; Extracting the GUI control tree of the software under test through a preset API, wherein the GUI control tree includes all control objects of the software under test; Obtaining mouse position information when the tester operates the software to be tested; According to the mouse position information, traverse the control objects in the GUI control tree to obtain the control information; Determine whether there is a target control object matching the mouse position information in the GUI control tree; If it exists, then obtain the information of the target control object to obtain the control information; If it does not exist, a preset dynamic traversal algorithm is used to send a user event to the GUI of the software under test to trigger the display of the hidden control object in the GUI; Obtaining the control information according to the mouse position information and the hidden control object in the GUI; Acquire a first GUI state of the software to be tested, where the first GUI state includes a first window state set of the current GUI; Determining a user event sequence to be sent to the GUI of the software under test according to the first GUI state; According to the user event sequence, a user event is sent to the GUI of the software to be tested to trigger the display of hidden control objects in the GUI.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the black box automated testing method based on a graphical user interface as described in any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the black box automated testing method based on a graphical user interface as described in any one of claims 1 to 11 is implemented.