Automatic test operation result auxiliary analysis system and method

Abstract

The invention provides an automatic test operation result auxiliary analysis system and method, and the system comprises a test module which is used for obtaining a test environment and a test case file, executing the test case file and an initialization file to generate a test record file, calling a WEB page packaging function, and calling an abnormal state recording and recovery module. And the screenshot function decoration module is used for capturing the current WEB page and storing the current WEB page to obtain a WEB page screenshot file. And the abnormal state recording and recovering module is used for recovering the abnormal state of the ancillary facility, deleting the residual configuration on the tested equipment and recording the process to generate a running event recording file. And the auxiliary analysis file generation module is used for generating an auxiliary analysis file according to the test record file, the WEB page screenshot file and the operation event record file. Procedures and consequences of problems can be found out through the auxiliary analysis file, reasons for failing operation can be conveniently analyzed, and the accuracy of analysis of automatic test operation results is improved.

Description

Technical Field

[0001] The present application relates to the field of automated testing technology, and in particular to an automated testing operation result auxiliary analysis system and method. Background Art

[0002] When using automated testing methods to test software, you can use a Python automated testing framework (such as nose) for testing. When executing automated testing tasks, each test case file (a collection of multiple related test case scripts) is assigned to different automated testing environments for execution according to certain rules. After the test cases in the test case file are run, an automated test report containing the test cases that passed the run and the test cases that failed the run will be formed. Developers can extract valuable information from the test cases that failed the run in the automated test report to find out the bugs in the software and fix them. Therefore, fast and accurate analysis of automated running results is also a very important part of automated testing work. However, when analyzing the results of automated testing, since automated testing involves the operation of front-end WEB pages, running automated test cases in multiple test environments, and the impact between test cases, only analyzing the failed run cases through automated test reports will result in inaccurate results.

[0003] For example, for software or devices that use a web page as a UI interface, when a test case script is run to call a web page encapsulation function and an exception occurs, the test framework will record the thrown exception information in the test record file. Since various web page software bugs or errors in the test case script itself may throw exactly the same exception information during the automated test process, it is difficult to use the exception information in the final automated test report to accurately infer the web page status when the exception occurred.

[0004] When test cases are executed continuously, the previously run test cases may generate residual configurations on the device under test due to various reasons (such as poor test case writing or software bugs, etc.), which will interfere with the test cases that are run later. When there are a large number of test cases, this interference is particularly obvious and very difficult to analyze.

[0005] When test cases are executed continuously, the previously run test cases may interfere with the status of other ancillary facilities in the test environment for various reasons, thus interfering with the test cases that are run later. For example, if a test case script modifies the configuration of a virtual machine or server in the test environment and does not restore it, all subsequent test case scripts that use these ancillary facilities will be affected. When test cases are executed continuously, the abnormal status generated in the test environment is accumulated and amplified step by step, making the results of the subsequent test process no longer of reference value. Summary of the invention

[0006] The present application provides an automated test run result auxiliary analysis system and method to solve the problem of inaccurate analysis of automated test run results.

[0007] The first aspect of the present application provides an automated test run result auxiliary analysis system, comprising:

[0008] A test module, wherein the test module is configured to:

[0009] Acquire a test environment and a test case file in an automated test case library, wherein the automated test case library includes an initialization file and a plurality of test case files, the test environment includes a device under test and ancillary facilities, the initialization file includes an initialization function and a cleanup function, and the test case file includes a plurality of test cases;

[0010] Execute the test case file and the initialization file in each set of the test environment to generate a test record file, call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file;

[0011] A screenshot function decoration module, wherein the screenshot function decoration module is configured as follows:

[0012] When an exception occurs in executing the test case file and / or calling the WEB page encapsulation function, intercept the current WEB page;

[0013] Saving the current WEB page to obtain a WEB page screenshot file;

[0014] The abnormal state recording and recovery module is configured as follows:

[0015] Restoring the abnormal state of the auxiliary facility and deleting the residual configuration on the device under test;

[0016] Recording the abnormal state and the residual configuration respectively to generate an operation event record file;

[0017] An auxiliary analysis file generation module, wherein the auxiliary analysis file generation module is configured to:

[0018] An auxiliary analysis file is generated according to the test record file, the WEB page screenshot file and the operation event record file.

[0019] A second aspect of the present application provides an automated test run result auxiliary analysis method, using the system described in the first aspect, the method comprising:

[0020] Acquire a test environment and a test case file in an automated test case library through a test module, wherein the automated test case library includes an initialization file and a plurality of test case files, the test environment includes a device under test and ancillary facilities, the initialization file includes an initialization function and a cleanup function, and the test case file includes a plurality of test cases;

[0021] Execute the test case file and the initialization file in each set of the test environment to generate a test record file, call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file;

[0022] When an exception occurs in executing the test case file and / or calling the WEB page encapsulation function, the current WEB page is intercepted by the screenshot function decoration module;

[0023] Saving the current WEB page to obtain a WEB page screenshot file;

[0024] Restoring the abnormal state of the auxiliary facility through the abnormal state recording and recovery module, and deleting the residual configuration on the device under test;

[0025] Recording the abnormal state and the residual configuration respectively to generate an operation event record file;

[0026] An auxiliary analysis file is generated by an auxiliary analysis file generation module according to the test record file, the WEB page screenshot file and the operation event record file.

[0027] The present application provides an auxiliary analysis system and method for automated test operation results, the system comprising: a test module, used to obtain test case files in the test environment and the automated test case library; execute the test case file and the initialization file in each test environment to generate a test record file, and call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file. A screenshot function decoration module, used to capture the current WEB page when an abnormality occurs, and save the current WEB page to obtain a WEB page screenshot file. An abnormal state recording and recovery module, used to restore the abnormal state of the auxiliary facilities, and delete the residual configuration on the device under test; record the abnormal state and the residual configuration respectively to generate an operation event record file. An auxiliary analysis file generation module, used to generate an auxiliary analysis file according to the test record file, the WEB page screenshot file and the operation event record file. The system generates an auxiliary analysis file by combining the test record file, the WEB page screenshot file and the operation event record file, and can find out the cause and effect of the problem through the auxiliary analysis file, facilitate the analysis and tracking of the reasons for the failure of the operation, and improve the accuracy of the analysis of the automated test operation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 A schematic diagram of the structure of the automated test operation result auxiliary analysis system provided in the embodiment of the present application;

[0030] Figure 2 A schematic diagram of the operation flow of the automated test operation result auxiliary analysis system provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the test case file and initialization file execution process in the test environment provided by an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the recovery interface configuration and the first log file generation process provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the process of restoring routing configuration and generating a second log file provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the process of recording configuration changes and generating a third log file provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of a process for deleting residual configuration and generating a fourth log file provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of a process for generating auxiliary analysis files provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram showing a test report page provided in an embodiment of the present application;

[0038] Fig.10 A schematic diagram of a test case page display showing a failed test run provided in an embodiment of the present application;

[0039] Fig.11 A schematic diagram of multiple auxiliary analysis files generated under different test environments provided in an embodiment of the present application;

[0040] Fig.12 A schematic diagram of a page display of one of the auxiliary analysis files when a test case provided in an embodiment of the present application fails to run;

[0041] Fig.13 A screenshot of the current WEB page that causes the operation failure provided in the embodiment of the present application;

[0042] Fig.14 A schematic diagram of a page display of alarm-related content in an auxiliary analysis file provided in an embodiment of the present application;

[0043] Fig.15 A schematic diagram of an alarm event related to a test case execution failure provided in an embodiment of the present application;

[0044] Fig.16 A flow chart of the auxiliary analysis method for automated test operation results provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0046] When analyzing the results of automated testing, since automated testing involves operations on front-end WEB pages, running automated test cases in multiple test environments, and the impact between test cases, only analyzing failed test cases through automated test reports will result in inaccurate results.

[0047] In order to solve the problem of inaccurate analysis of automated test results, some embodiments of the present application first provide an automated test result auxiliary analysis system, see Figure 1-Figure 2 ,The system includes a testing module, a screenshot function decoration module, an ,abnormal state recording and recovery module and an auxiliary analysis file ,generation module.

[0048] In some embodiments, the test module may be an automated test framework, such as nose software. After the test module is started, it will start several threads based on the number of currently available test environments and execute the test case files in a distributed manner. Therefore, the test module is configured to: obtain the test environment and the test case files in the automated test case library; execute the test case files and initialization files in each test environment to generate a test record file, and call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file.

[0049] The test environment is a collection of hardware and software that supports the operation of automated tests. Each set of test environments includes the equipment under test (including hardware equipment and the software running on it) and ancillary facilities (such as virtual machines, servers, instruments, etc. required for testing). The automated test case library includes initialization files and multiple test case files. The automated test case file library places test cases in different test case files according to their types and scopes. Each test case file contains several test cases, which are organized in the form of python functions or python class methods. The initialization file is the init file. When the nose software is used as the test module, the scripts in the initialization file and the test case file are written in python. Therefore, the initialization file can be represented as an _init_.py file. The initialization function and the cleanup function are set in the initialization file to call the abnormal state recording and recovery module described below.

[0050] After the test module reads all the test case files, it assigns several test case files to different test environments for execution according to the currently available test environments. For each test environment, different test case files will be randomly received during a certain execution process. For example, see Figure 3, the test module assigns test case files 5, 1 and 3 to test environment 1, and assigns test case files 2, 6 and 4 to test environment 2. It should be noted that this assignment order is not necessarily the same in different automated test tasks. In addition, at the interval of each test case file execution, the content of the _init_.py file will be executed. Before the test case file is executed, all the contents under the initialization function in the _init_.py file will be executed. After the test case file is executed, all the contents under the cleanup function in the _init_.py file will be executed.

[0051] The WEB page encapsulation function is used to abstract various page operations and page control inputs into input parameters of the Python function, which can simulate the actual page operations of the user and return the operation results. During the execution of the test case file, multiple WEB page encapsulation functions will be called to complete a series of page operations. After executing a test case file, the test module generates a test record file that records the running process, that is, the number of generated test record files is the same as the number of test case files. The test record file can be used to obtain the results of the success or failure of the test case. The test record file is a file in XML format.

[0052] The screenshot function decoration module is configured to: capture the current WEB page when an exception occurs in executing the test case file and / or calling the WEB page encapsulation function; save the current WEB page to obtain a WEB page screenshot file.

[0053] The screenshot function decoration module is used to decorate the WEB page encapsulation function. When an exception occurs when executing the test case file and / or calling the WEB page encapsulation function, the test case fails to run. At this time, the current WEB page is intercepted and saved to obtain a WEB page screenshot file. In some embodiments, the screenshot function decoration module can be a screenshot function decorator, and a decorator named save_screenshot can be defined. The screenshot method get_screenshot_as_file provided in the WEB page testing tool selenium can be used to implement the screenshot operation. Adding @save_screenshot in front of the WEB page encapsulation function can save the WEB page screenshot when the WEB page encapsulation function is called and an exception is thrown. For the saved WEB page screenshot file, it is necessary to record the test environment name, WEB page encapsulation function name, test case file name and specific test case name in the file name so that auxiliary analysis files can be generated later.

[0054] The abnormal state recording and recovery module is configured to: recover the abnormal state of the auxiliary facilities and delete the residual configuration on the tested equipment; and record the abnormal state and the residual configuration respectively to generate an operation event record file.

[0055] The auxiliary analysis file generation module is configured to generate an auxiliary analysis file according to the test record file, the WEB page screenshot file and the operation event record file.

[0056] When analyzing the test results, each test environment needs to be analyzed. That is, the system provided in this application analyzes multiple test environments, and the auxiliary analysis files are classified based on the test environment. In other words, several auxiliary analysis files are generated if several test environments are used in the automated testing process.

[0057] The system provided by the present application can facilitate subsequent serial analysis by saving the WEB page anomalies that appear when executing the test case file in the form of screenshots. By recording and deleting the residual configuration on the device under test, and recording and restoring the abnormal status of the auxiliary facilities when executing the test case file, interference with subsequent tests can be avoided. By combining the test record file, WEB page screenshot file, and operation event record file to generate an auxiliary analysis file, the cause and effect of the problem can be found through the auxiliary analysis file, which facilitates the analysis and tracking of the reasons for the failure of the operation, and improves the accuracy of the analysis of the results of the automated test operation.

[0058] In some embodiments, the ancillary facilities include interfaces of the device under test, virtual machines and virtual machine routing, the abnormal status includes interface configuration errors and routing configuration errors, and the abnormal status recording and recovery module includes: interface recovery function, routing recovery function, monitoring configuration change function and invalid configuration deletion function.

[0059] The interface recovery function can be represented as recover_interface_config(). The interface recovery function is used to: recover the interface configuration when the interface configuration of the device under test is wrong, and record the recovery process of the interface configuration to generate a first log file. Figure 4 In some embodiments, when the interface configuration of the interface of the device under test is wrong, restoring the interface configuration and recording the restoration process of the interface configuration to generate a first log file includes the following steps:

[0060] S410: Read interface configuration information of the interface of the device under test, and save the interface configuration information in a first string variable.

[0061] In some embodiments, the interface configuration information of the interface of the device under test may be read through a CLI command, and the first string variable may be represented as interface_config.

[0062] S420: Search for a special type of interface in the first string variable according to the interface configuration information.

[0063] S430: If a special type interface exists in the first string variable, delete the special type interface.

[0064] Special interfaces do not exist in the initial configuration, but may be created during the test case execution. Special interfaces include subinterfaces, aggregate interfaces, bridge interfaces, etc. Since special interfaces include one or more physical interfaces, the physical interfaces cannot be edited without deleting these special interfaces. If special interfaces exist in the first string variable interface_config, delete the special interfaces through the CLI command.

[0065] S440: Obtaining state information of the interface of the device under test from the first string variable by using a regular expression.

[0066] If the special type of interface does not exist in the first string variable interface_config, the state information of the interface of the device under test, such as the ip address, ipv6 address, open and close status, etc., is obtained from the first string variable through a regular expression.

[0067] S450: When the state information is inconsistent with the interface configuration information, the interface of the device under test is saved in an abnormal interface list.

[0068] If the acquired state information of the interface of the device under test is inconsistent with the interface configuration information in the test environment, it means that the state of the interface of the device under test is abnormal, and the interface of the device under test is saved in the abnormal interface list.

[0069] S460: Traverse the abnormal interface list, and restore the interface configuration of the interface of the device under test in the abnormal interface list.

[0070] It can be understood that the number of interfaces of the tested device in the abnormal interface list may be 0, that is, the abnormal interface list is an empty list, or it may be greater than 0. When the number of interfaces of the tested device in the abnormal interface list is greater than 0, the abnormal interface list is traversed, and the interface configuration of the interface of the tested device in the abnormal interface list is restored according to the interface configuration information through CLI commands.

[0071] S470: Record the recovery process of the interface configuration to generate a first log file.

[0072] Before this step, it is necessary to determine whether an interface recovery operation has occurred during the test. When the number of interfaces of the device under test in the abnormal interface list is greater than 0, an interface recovery operation must have occurred. At this time, the recovery process of the interface configuration is recorded in the form of a log file to generate a first log file. The title of the generated first log file includes the operation time, the name of the interface of the device under test, and the name of the test environment. If no interface recovery operation has occurred, the interface recovery function ends.

[0073] The routing recovery function can be expressed as recover_centos(). The routing recovery function is used to: restore the routing configuration when the routing configuration of the virtual machine routing is wrong, and record the recovery process of the routing configuration to generate a second log file, see Figure 5 In some embodiments, when the routing configuration of the virtual machine routing is wrong, restoring the routing configuration and recording the recovery process of the routing configuration to generate a second log file includes the following steps:

[0074] S510: Obtain routing information, and save the routing information in a second string variable.

[0075] It should be noted that before reading the routing information, you should first connect to the virtual machine in the test environment. The routing information can be read through the Linux command, and the second string variable can be represented as route_config.

[0076] S520: Determine whether there is default route information in the second string variable by using a regular expression.

[0077] S530: If the default route information does not exist in the second string variable, configure the default route of the virtual machine.

[0078] In some embodiments, a default route may be added via a Linux command.

[0079] S540: If the second string variable contains default routing information, determine whether the default routing information is correct by using a regular expression.

[0080] If the default route information is correct, determine whether the process of restoring the route configuration has occurred.

[0081] S550: When the default route information is wrong, the default route information is deleted and a new route is created to restore the route configuration.

[0082] If the default route information is incorrect, delete the default route information and create a new route through Linux commands.

[0083] S560: Record the process of restoring the routing configuration to generate a second log file.

[0084] If a routing configuration recovery operation has occurred during the test, the routing configuration recovery process is recorded in the form of a log file to generate a second log file, and the title of the generated second log file includes the operation time, the virtual machine name, and the test environment name. If no routing configuration recovery operation has occurred, the routing recovery function ends.

[0085] The monitoring configuration change function can be expressed as monitor_config_change(). The monitoring configuration change function is used to: find the configuration changes of the device under test and record the configuration changes to generate a third log file. See Figure 6 In some embodiments, searching for configuration changes of the device under test and recording the configuration changes to generate a third log file includes the following steps:

[0086] S610: Acquire initial configuration information in the initial string variable.

[0087] The initial string variable is set in the initialization file, and the initial configuration information of the device under test is stored in the initial string variable.

[0088] S620: Read current configuration information of the device under test, and save the current configuration information in a third string variable.

[0089] The current configuration information of the device under test can be read through the CLI command, and the third string variable can be represented as current_config. If the current configuration information is consistent with the initial configuration information, the monitoring configuration change function ends.

[0090] S630: When the current configuration information is inconsistent with the initial configuration information, the third string variable is divided into a plurality of first configuration lines, and the initial string variable is divided into a plurality of second configuration lines.

[0091] It can be understood that after segmentation, each first configuration row stores the first configuration information, and each second configuration row stores the second configuration information. All the first configuration information combined together is the current configuration information, and all the second configuration information combined together is the initial configuration information. That is, the first configuration information stored in multiple first configuration rows is different, and the second configuration information stored in multiple second configuration rows is also different.

[0092] S640: traverse the first configuration information in the first configuration row, and traverse the second configuration information in the second configuration row.

[0093] S650: If the first configuration information is not in the initial string variable, save the first configuration information that is not in the initial string variable in a fourth string variable.

[0094] The first configuration information is the information in the current configuration information. If a certain first configuration information is not in the initial string variable, it means that the first configuration information is residual after the test case file is executed and needs to be recorded. In some embodiments, the first configuration information that is not in the initial string variable can be saved in the fourth string variable. It can be understood that the first configuration information in the fourth string variable is the residual configuration after the test case is executed.

[0095] S660: If the second configuration information is not in the third string variable, save the second configuration information that is not in the third string variable in a fourth string variable.

[0096] The second configuration information is the first configuration information and the second configuration information in the initial configuration information. If a certain second configuration information is not in the third string variable, it means that the second configuration information is mistakenly deleted or modified after the test case file is executed, and it needs to be recorded. The second configuration information that is not in the third string variable can be saved in the fourth string variable. It can be understood that the second configuration information in the fourth string variable is the unrestored configuration after the test case is executed.

[0097] S670: Save the first configuration information and / or the second configuration information in the fourth string variable in a log file to generate a third log file.

[0098] The title of the generated third log file includes the operation time, the name of the device under test, and the name of the test environment.

[0099] The function for deleting invalid configuration can be represented as remove_invalid_config(), which is used to delete the residual configuration on the device under test and record the deletion process of the residual configuration to generate a fourth log file. It can be understood that the residual configuration is the configuration generated by running the test case. Figure 7 In some embodiments, deleting the residual configuration on the device under test and recording the deletion process of the residual configuration to generate a fourth log file includes the following steps:

[0100] S710: Construct a first dictionary.

[0101] In some embodiments, a first dictionary named recover_dict may be constructed, the first dictionary including at least one first identifier and a first element corresponding to the first identifier. The structure of the first dictionary is {str:[str,str],str:[str,str],str:[str,str]}. For example,

[0102] {Illegal configuration 1: [regular expression feature of illegal configuration 1, WEB page deletion function of illegal configuration 1],

[0103] Illegal configuration 2: [Regular expression characteristics of illegal configuration 2, WEB page deletion function of illegal configuration 2], Illegal configuration 3: [Regular expression characteristics of illegal configuration 3, WEB page deletion function of illegal configuration 3],

[0104] }. Illegal configuration 1, illegal configuration 2, illegal configuration 3 are the first identifiers in the first dictionary, the regular expression feature and the WEB page deletion function are the first elements in the first dictionary, the regular expression feature is used to find illegal configurations, and the WEB page deletion function is used to delete illegal configurations. The first identifiers illegal configuration 1, illegal configuration 2, illegal configuration 3 are the residual configurations on the device under test.

[0105] S720: Read current configuration information, and save the current configuration information in a fifth string variable.

[0106] In some embodiments, the current configuration information may be read via a CLI command.

[0107] S730: Traverse the first identifier in the first dictionary to find the residual configuration in the fifth string variable through the regular expression feature in the first element.

[0108] S740: If the fifth string variable contains a regular expression feature, call the WEB page deletion function in the first element to delete the residual configuration.

[0109] S750: Record the deletion process of the residual configuration to generate a fourth log file.

[0110] If a residual configuration deletion operation occurs during the test, the residual configuration deletion process is recorded in the form of a log file to generate a fourth log file, and the title of the generated fourth log file includes the operation time, the name of the device under test, and the name of the test environment. If the fifth string variable does not contain the first identifier, the invalid configuration deletion function ends.

[0111] It can be seen from the above embodiment that the operation event record file includes a first log file, a second log file, a third log file and a fourth log file. According to the titles of the first log file, the second log file, the third log file and the fourth log file, the operation event record file can be associated with the test record file through the test environment name and the operation time.

[0112] It should also be noted that, since the test module calls the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file, in the embodiment of the present application, the initialization function is used to call the interface recovery function and the routing recovery function before executing the test case file, so as to obtain the configuration information of the device under test before running the test case, and before the next test case file is executed, restore the interface configuration of the device under test and the virtual machine routing configuration so that the next test case file can be executed normally. The cleanup function is used to call the monitoring configuration change function and the deletion of invalid configuration function after executing the test case file, so as to detect the device configuration change after the test case file is executed, and delete the residual configuration on the device caused by running the test case.

[0113] In order to record the time of each operation during the test process, the test record file name, WEB page screenshot file name and running event record file name include a timestamp. After the test case is run, the name of the generated test record file is the same as the name of the corresponding test case file. For example, if the test case file name is test_001.py, then the test record file name is test_001.xml. When the screenshot function decoration module and the abnormal state recording and recovery module generate corresponding files, they will also record the test environment name in the file name. Therefore, in some embodiments, the test record file name includes the test case file name and a timestamp, the WEB page screenshot file name includes the test environment name, test case file name, test case name, WEB page encapsulation function name and a timestamp, and the running event record file name includes the test environment name and a timestamp. See. Figure 8 , based on the test record file, WEB page screenshot file and operation event record file, generating auxiliary analysis files includes the following steps:

[0114] S810: Read the test record file to obtain the test environment name according to the test record file.

[0115] After the test case is run, the test module will record the test environment name in the file content of the test record file, so the name of the test environment used can be obtained according to the test record file. After reading all the test record files (xml files), you can get which test environments are used during the test.

[0116] S820: Construct a second dictionary according to the test environment name.

[0117] The second dictionary is used to store the test environment name. A second dictionary named topology can be constructed. The second dictionary includes at least one second identifier and a file list, wherein the second identifier is the test environment name. The number of second identifiers in the second dictionary is the same as the number of test environments, that is, if several test environments are used during the test, there are several second identifiers in the second dictionary. A second element is set in the file list, and the second element includes the test record file name, the WEB page screenshot file name, and the operation event record file name. Each test record file name, WEB page screenshot file name, and operation event record file name in the file list are related to the test environment name corresponding to the second identifier.

[0118] Exemplarily, the second dictionary can be expressed as follows:

[0119] {

[0120] Test environment 1: [test record file name 1.xml, test record file name 2.xml, WEB page screenshot file name 1.png, WEB page screenshot file name 2.png, operation event record file name 1.txt, operation event record file name 2.txt],

[0121] Test environment 2: [test record file name 3.xml, test record file name 4.xml, WEB page screenshot file name 3.png, WEB page screenshot file name 4.png, operation event record file name 3.txt, operation event record file name 4.txt],

[0122] Test environment 3: [test record file name 5.xml, test record file name 6.xml, WEB page screenshot file name 5.png, WEB page screenshot file name 6.png, operation event record file name 5.txt, operation event record file name 6.txt],

[0123] }

[0124] S830: Sort the test record file names, WEB page screenshot file names, and running event record file names in the same file list by timestamp to generate a sorted file list.

[0125] That is, the test record file names, WEB page screenshot file names and operation event record file names are sorted in the order of the time when the files were generated. Several sorted file lists are generated when several test environments are used during the test process.

[0126] S840: Divide the sorted file list into a plurality of segments based on the test record file name to obtain a sorted sub-list.

[0127] That is, each sorting sublist includes a test record file name, and since web page screenshot files and running event record files may not be generated during the test process, if no web page screenshot files and running event record files are generated during the running of the test case file, then the sorting sublist only includes the test record file name. If web page screenshot files and running event record files are generated, then the sorting sublist includes the test record file name, web page screenshot file name, and running event record file name in chronological order.

[0128] S850: Obtain the association relationship between the test record file name, the WEB page screenshot file name, the running event record file name and the test record file in each sorted sub-list according to the timestamp.

[0129] It is understandable that before executing a test case file, when executing the initialization function in _init_.py, the name of the running event record file is associated with the test record file only if a restore interface configuration event or a restore virtual machine routing configuration event is generated. After executing a test case file, when executing the cleanup function in _init_.py, the name of the running event record file is associated with the test record file only if a device configuration change record or a deletion residual configuration record is generated. During the running of a test case file, when calling the WEB page encapsulation function, the name of the WEB page screenshot file is associated with the test record file only if a WEB page screenshot is generated.

[0130] When generating a WEB page screenshot file, the system will obtain the test case file name and the test case name through the stack module in Python, and save it in the WEB page screenshot file name. Therefore, the WEB page screenshot file can obtain the association between the WEB page screenshot file name and the test record file from the information it carries. The test record file name is the same as the test case file name, so the association between the test record file name and the test record file can be obtained through the order of timestamps. For example, the test record files arranged in the order of generation time: test001.xml (timestamp 1), test002.xml (timestamp 2), test003.xml (timestamp 3). Then, if the file generated by the initialization function in _init_.py is the type of file, it should be associated with test002.xml; if it is the type of file generated by the cleanup function, it should be associated with test001.xml. Although the test case file name cannot be directly obtained when the run event record file is generated, the above method can also be used to infer the test case file to which it belongs based on the sequence of timestamps, and then obtain the association between the run event record file name and the test record file.

[0131] S860: Traverse the second identifiers in the second dictionary to generate an auxiliary analysis file according to the test environment name based on the association relationship.

[0132] It should also be noted that the auxiliary analysis file is generated based on the test environment. The number of test environments used in the test process is the same as the number of auxiliary analysis files generated. The generated auxiliary analysis file is a file in HTML format.

[0133] The auxiliary analysis files generated according to the association relationship and the test environment name include:

[0134] S861: define HTML codes with fixed format at the beginning and end, wherein the HTML codes include CSS style information.

[0135] S862: According to the association relationship, the actual storage locations of the test record file, WEB page screenshot file, and operation event record file corresponding to the test record file name, WEB page screenshot file name, and operation event record file name in the sorted sublist are obtained respectively.

[0136] S863: Fill in the test record file name, WEB page screenshot file name, operation event record file name and actual storage location in the sorted sublist in the form of a hyperlink between the beginning and the end of the HTML code to form a code paragraph.

[0137] S864: Save code segments to generate auxiliary analysis files.

[0138] By organizing the test record files, WEB page screenshot files and operation event record files contained in the test environment in order according to the operation time sequence and taking the test case files as the unit, and generating auxiliary analysis files, the cause and consequence of the problem can be found through the auxiliary analysis files when analyzing the results of the automated test operation, which facilitates the analysis and tracking of the reasons for the failure of the operation and improves the accuracy of the analysis of the automated test operation results.

[0139] The following is an example of the system-assisted analysis process. Figure 2 After the automated test task is completed, all the test record files generated by the test module will generate a test report report.html through the test report generator. Fig. 9 , a failed test case is found in the test report, such as Fig. 9 The test case

[0528] shown in the test report can be read from the test report that this test case was executed on the test environment hudi_fw_08. For details about this test case

[0528] in the test report, see Fig.10 The reason why the test case failed to run was that an exception occurred when calling the WEB page encapsulation function new_policy. The name of the WEB page screenshot file when the exception occurred was screenshot_new_policy_20240514121259@test_policy_av_function_max_decompression_sxy_Single@test_policy_av_function_18@hudi_fw_08.png.

[0140] Call the function to generate auxiliary analysis files to generate multiple auxiliary analysis files under multiple test environments. The generated auxiliary analysis files are as follows: Fig.11 Open the auxiliary analysis file hudi_fw_08_running_history_20240521095943.html corresponding to the test environment hudi_fw_08, see Fig.12 , search for the hyperlink of the WEB page screenshot file corresponding to the failed run case

[0528] in the auxiliary analysis file. The name of the hyperlink is consistent with the name of the WEB page screenshot file. Click the hyperlink of the WEB page screenshot file in the auxiliary analysis file to open the WEB page screenshot. Fig.13,From the screenshot file of the WEB page, it was found that the cause of the problem was that a randomly appearing alarm information message box popped up, covering some controls that needed to be operated on the page, causing an exception in calling the WEB page encapsulation function new_policy.

[0141] According to the information obtained from the WEB page screenshot file, it is speculated that the system alarm configuration switch was turned on and then not turned off. Therefore, search for the English characters related to alarm in the auxiliary analysis file hudi_fw_08_running_history_20240521095943.html. Fig.14 It can be seen that after the previous test case file was executed, there was a residual configuration alarm_enable. It can be inferred that during the running of test_mobile_terminal_manage_function, the system alarm configuration switch was temporarily turned on for testing purposes, but was forgotten to be turned off.

[0142] Open the hyperlink of test_mobile_terminal_manage_function.xml, see Fig.15 , search for the word enable_alarm, and through the configuration process record of enable_alarm: True, it is found that there is indeed a step in the test process to turn on the system alarm, but there is no corresponding closing action afterwards, which causes an exception when calling the WEB page encapsulation function new_policy when running test case

[0528] .

[0143] From the above examples, we can see that the pop-up of the system alarm message box is relatively random and is determined by the system's current operating status. In the absence of auxiliary analysis tools, it is difficult to find the cause and effect of the problem. However, through the auxiliary analysis file and the WEB page screenshot file and the operation event record file that constitute the auxiliary analysis file, the WEB page screenshot file and the operation event record file can be connected in series, thereby playing the role of auxiliary analysis.

[0144] Based on the above-mentioned automated test operation result auxiliary analysis system, some embodiments of the present application also provide an automated test operation result auxiliary analysis method, using the system provided by some of the above embodiments, see Fig.16 , methods include:

[0145] S1: Obtain the test environment and test case files in the automated test case library through the test module.

[0146] The automated test case library includes an initialization file and multiple test case files. The initialization file includes an initialization function and a cleanup function. The test environment includes the device under test and ancillary facilities.

[0147] S2: Execute the test case file and initialization file in each test environment to generate a test record file, call the WEB page encapsulation function when executing the test case file, and call the abnormal status recording and recovery module through the initialization function and cleanup function when executing the initialization file.

[0148] S3: When an exception occurs in executing the test case file and / or calling the WEB page encapsulation function, the current WEB page is captured through the screenshot function decoration module.

[0149] S4: Save the current WEB page to obtain a WEB page screenshot file.

[0150] S5: Restore the abnormal state of the auxiliary facilities through the abnormal state recording and recovery module, and delete the residual configuration on the tested device.

[0151] S6: Record the abnormal status and residual configuration respectively to generate an operation event record file.

[0152] S7: Generate an auxiliary analysis file through an auxiliary analysis file generation module according to the test record file, the WEB page screenshot file and the operation event record file.

[0153] It is understandable that if no exception occurs when executing the test case file and calling the WEB page encapsulation function, there is no need to capture the current WEB page and no WEB page screenshot file is generated.

[0154] It can be seen from the above technical solutions that the embodiment of the present application provides an auxiliary analysis system and method for automated test operation results. The system includes: a test module, which is used to obtain the test environment and the test case files in the automated test case library, execute the test case files and initialization files under each test environment to generate a test record file, and call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file. A screenshot function decoration module is used to capture the current WEB page when an abnormality occurs, and save the current WEB page to obtain a WEB page screenshot file. An abnormal state recording and recovery module is used to restore the abnormal state of the auxiliary facilities and delete the residual configuration on the device under test; record the abnormal state and the residual configuration respectively to generate an operation event record file. An auxiliary analysis file generation module is used to generate an auxiliary analysis file based on the test record file, the WEB page screenshot file and the operation event record file. By combining the test record file, WEB page screenshot file and operation event record file to generate an auxiliary analysis file, the cause and effect of the problem can be found through the auxiliary analysis file, which is convenient for analyzing and tracking the reasons for the failure of the operation and improving the accuracy of the analysis of the automated test operation results.

[0155] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. An automated test run result auxiliary analysis system, characterized in that: include: A test module, wherein the test module is configured to: Acquire a test environment and a test case file in an automated test case library, wherein the automated test case library includes an initialization file and a plurality of test case files, the test environment includes a device under test and ancillary facilities, the initialization file includes an initialization function and a cleanup function, and the test case file includes a plurality of test cases; Execute the test case file and the initialization file in each set of the test environment to generate a test record file, call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file; A screenshot function decoration module, wherein the screenshot function decoration module is configured as follows: When an exception occurs in executing the test case file and / or calling the WEB page encapsulation function, intercept the current WEB page; Saving the current WEB page to obtain a WEB page screenshot file; The abnormal state recording and recovery module is configured as follows: Restoring the abnormal state of the auxiliary facility and deleting the residual configuration on the device under test; Recording the abnormal state and the residual configuration respectively to generate an operation event record file; An auxiliary analysis file generation module, wherein the auxiliary analysis file generation module is configured to: An auxiliary analysis file is generated according to the test record file, the WEB page screenshot file and the operation event record file.

2. The automated test operation result auxiliary analysis system according to claim 1, characterized in that: The ancillary facilities include a device under test interface, a virtual machine, and a virtual machine route. The abnormal state includes an interface configuration error and a route configuration error. The abnormal state recording and recovery module includes: An interface recovery function, wherein the interface recovery function is used to: When the interface configuration of the interface of the device under test is wrong, the interface configuration is restored, and the restoration process of the interface configuration is recorded to generate a first log file; the title of the first log file includes the operation time, the name of the interface of the device under test and the name of the test environment; A routing recovery function, wherein the routing recovery function is used to: When the routing configuration of the virtual machine routing is wrong, the routing configuration is restored, and the recovery process of the routing configuration is recorded to generate a second log file; the title of the second log file includes the operation time, the virtual machine name and the test environment name; A configuration change monitoring function is used to: Searching for configuration changes of the device under test, and recording the configuration changes to generate a third log file; the title of the third log file includes operation time, name of the device under test, and name of the test environment; Delete invalid configuration function, the invalid configuration deletion function is used to: Delete the residual configuration on the device under test, and record the deletion process of the residual configuration to generate a fourth log file; the title of the fourth log file includes the operation time, the name of the device under test and the name of the test environment; the residual configuration is the configuration generated by running the test case in the test case file, and the operation event record file includes the first log file, the second log file, the third log file and the fourth log file.

3. The automated test operation result auxiliary analysis system according to claim 2, characterized in that: The initialization function is used to: Before executing the test case file, calling the interface recovery function and the routing recovery function; The cleanup function is used to: After executing the test case file, the monitoring configuration change function and the deleting invalid configuration function are called.

4. The automated test operation result auxiliary analysis system according to claim 2, characterized in that: The step of restoring the interface configuration when the interface configuration of the interface of the device under test is wrong, and recording the restoration process of the interface configuration to generate a first log file comprises: Reading interface configuration information of the interface of the device under test, and storing the interface configuration information in a first string variable; Searching for a special type of interface in the first string variable according to the interface configuration information; If the special type interface exists in the first string variable, delete the special type interface; Acquire the state information of the interface of the device under test from the first string variable by using a regular expression; When the state information is inconsistent with the interface configuration information, saving the interface of the device under test in an abnormal interface list; Traversing the abnormal interface list, and restoring the interface configuration of the device under test interface in the abnormal interface list; The recovery process of the interface configuration is recorded to generate a first log file.

5. The automated test operation result auxiliary analysis system according to claim 2, characterized in that: The restoring the routing configuration when the routing configuration of the virtual machine routing is wrong, and recording the restoration process of the routing configuration to generate a second log file includes: Obtaining routing information, and storing the routing information in a second string variable; Determine whether there is default route information in the second string variable by using a regular expression; If the default route information does not exist in the second string variable, configuring the default route of the virtual machine; If the default routing information exists in the second string variable, determine whether the default routing information is correct by using a regular expression; When the default route information is wrong, deleting the default route information and creating a new route to restore the route configuration; The process of restoring the routing configuration is recorded to generate a second log file.

6. The automated test operation result auxiliary analysis system according to claim 4, characterized in that: The initialization file also includes an initial string variable, in which initial configuration information of the device under test is stored. The searching for configuration changes of the device under test and recording the configuration changes to generate a third log file includes: Obtain the initial configuration information in the initial string variable; Reading current configuration information of the device under test, and saving the current configuration information in a third string variable; When the current configuration information is inconsistent with the initial configuration information, the third string variable is divided into a plurality of first configuration lines, and the initial string variable is divided into a plurality of second configuration lines; the plurality of first configuration lines store different first configuration information, and the plurality of second configuration lines store different second configuration information; Traversing the first configuration information in the first configuration row, and traversing the second configuration information in the second configuration row; If a certain first configuration information is not in the initial string variable, the first configuration information not in the initial string variable is saved in a fourth string variable, and the first configuration information in the fourth string variable is the residual configuration after the test case is executed; If a certain second configuration information is not in the third string variable, the second configuration information not in the third string variable is saved in the fourth string variable, and the second configuration information in the fourth string variable is the unrestored configuration after the test case is executed; The first configuration information and / or the second configuration information in the fourth string variable is saved in a log file to generate a third log file.

7. The automated test operation result auxiliary analysis system according to claim 6, characterized in that: The deleting the residual configuration on the device under test and recording the deletion process of the residual configuration to generate a fourth log file includes: Constructing a first dictionary, the first dictionary including at least one first identifier and a first element corresponding to the first identifier, the first identifier is a residual configuration, and the first element includes a regular expression feature and a WEB page deletion function; Read the current configuration information, and save the current configuration information in a fifth string variable; traversing the first identifier in the first dictionary to find the residual configuration in the fifth string variable through the regular expression feature in the first element; If the fifth string variable contains the regular expression feature, calling the WEB page deletion function in the first element to delete the residual configuration; The deletion process of the residual configuration is recorded to generate a fourth log file.

8. The automated test operation result auxiliary analysis system according to claim 2, characterized in that: The content of the test record file includes the test environment name, the test record file name includes the test case file name and timestamp, the WEB page screenshot file name includes the test environment name, test case file name, test case name, WEB page encapsulation function name and timestamp, and the run event record file name includes the test environment name and timestamp; The generating of the auxiliary analysis file according to the test record file, the WEB page screenshot file and the operation event record file comprises: Read the test record file to obtain the test environment name according to the test record file; constructing a second dictionary according to the test environment name, the second dictionary including at least one second identifier and at least one file list, the second identifier being the test environment name, the file list being provided with a second element, the second element including a test record file name, a WEB page screenshot file name, and a run event record file name, and each of the test record file name, WEB page screenshot file name, and run event record file name in the file list is related to the test environment name corresponding to the second identifier; Sorting the test record file names, the WEB page screenshot file names and the operation event record file names in the same file list by the timestamp to generate a sorted file list; Based on the test record file name, the sorted file list is divided into a plurality of segments to obtain a sorted sub-list; Acquire the association relationship between the test record file name, the WEB page screenshot file name, the operation event record file name and the test record file in each of the sorted sublists according to the timestamp; The second identifiers in the second dictionary are traversed to generate the auxiliary analysis file according to the test environment name based on the association relationship.

9. The automated test operation result auxiliary analysis system according to claim 8, characterized in that: The auxiliary analysis file is a file in HTML format, and generating the auxiliary analysis file according to the association relationship and the test environment name includes: Define HTML code with fixed format at the beginning and end, and the HTML code includes CSS style information; According to the association relationship, respectively obtain the actual storage locations of the test record file, the WEB page screenshot file, and the operation event record file corresponding to the test record file name, the WEB page screenshot file name, and the operation event record file name in the sorted sublist; Fill the test record file name, the WEB page screenshot file name, the operation event record file name and the actual storage location in the sorted sublist between the beginning and the end of the HTML code in the form of a hyperlink to form a code paragraph; The code segment is saved to generate the auxiliary analysis file.

10. A method for assisting analysis of automated test operation results, characterized in that: Using the system described in any one of claims 1 to 9, the method comprises: Acquire a test environment and a test case file in an automated test case library through a test module, wherein the automated test case library includes an initialization file and a plurality of test case files, the test environment includes a device under test and ancillary facilities, the initialization file includes an initialization function and a cleanup function, and the test case file includes a plurality of test cases; Execute the test case file and the initialization file in each set of the test environment to generate a test record file, call the WEB page encapsulation function when executing the test case file, and call the abnormal state recording and recovery module through the initialization function and the cleanup function when executing the initialization file; When an exception occurs in executing the test case file and / or calling the WEB page encapsulation function, the current WEB page is intercepted by the screenshot function decoration module; Saving the current WEB page to obtain a WEB page screenshot file; Restoring the abnormal state of the auxiliary facility through the abnormal state recording and recovery module, and deleting the residual configuration on the device under test; Recording the abnormal state and the residual configuration respectively to generate an operation event record file; An auxiliary analysis file is generated by an auxiliary analysis file generation module according to the test record file, the WEB page screenshot file and the operation event record file.

Images