Method and system for analyzing execution result of test case

Abstract

The invention provides a method for analyzing execution results of test cases. The method comprises the following steps: acquiring the execution results of the test cases, and acquiring a first test case which is failed in the execution results; performing data processing on each test step of the test case to obtain a list set; according to the result list, each test step of the first test case is classified, classification results are obtained, and each classification result comprises a preset test step; and matching the parameters of the predetermined test step with the historical execution result and the existing historical analysis result, and positioning the cause of execution failure of the predetermined test step, and the method of the embodiment can improve the test efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of test cases, and in particular to a method, system, electronic device and computer storage medium for analyzing test case execution results. Background Art

[0002] As software testing projects become increasingly large, automated test cases can help testers conduct rapid regression testing and reduce manual workload. However, the phenomenon of automated test case execution failure is becoming increasingly serious, causing testers to invest a lot of time and energy in analyzing failed automated test cases. The reasons for the failure of automated test cases include data problems, network problems, service problems, equipment problems, and failure to maintain automated scripts in a timely manner. According to statistics, about 30% of automated test cases fail during execution, which not only affects test efficiency but also increases project risks. Summary of the invention

[0003] The present application provides a method for analyzing the execution results of a test case, which can improve the test efficiency. The method includes the following steps: obtaining the execution results of the test case, and obtaining the first test case that failed in the execution results; performing data processing on each test step of the test case to obtain a list set; classifying each test step of the first test case according to the result list to obtain a classification result, wherein each classification result contains a predetermined test step; matching the parameters of the predetermined test step with the historical execution results and the existing historical analysis results, and locating the cause of the failure of the predetermined test step to execute.

[0004] In some embodiments, the step of performing data processing on each test step of the test case to obtain a list set includes: obtaining execution results corresponding to each test step, performing structured conversion on each execution result to obtain a conversion result; comparing the conversion result with a predetermined result to obtain a dictionary set of execution failures; and obtaining a list set associated with each test step based on the dictionary set.

[0005] In some embodiments, the step of performing a structured conversion on each execution result includes: deserializing each execution result into JSON, converting it from a JSON string into a JSON object.

[0006] In some embodiments, the step of classifying each test step of the first test case according to the first result list to obtain a classification result includes: obtaining a first test step of a first test case as a first classification, and removing the test step from the failed test case list; obtaining the similarity between other test steps in the first test case and the first test step, classifying the test steps whose similarity is greater than a preset threshold into the first classification, and obtaining a classification result.

[0007] In some embodiments, the step of matching the parameters of the predetermined test step with the historical execution results and the existing historical analysis results includes: matching the message sending content and the return result of the predetermined test step with the historical execution results and the existing historical analysis results according to a string matching algorithm to find the historical analysis result corresponding to the maximum matching degree; and locating the cause of the failure of the execution of the predetermined test step from the historical analysis results.

[0008] In some embodiments, the method further includes: storing and displaying the classification results and their corresponding failure reasons.

[0009] In some embodiments, the present application also proposes a test case execution result analysis system, including: an acquisition module, used to obtain the execution result of the test case, and obtain the first test case that failed in the execution result; a data processing module, used to perform data processing on each test step of the test case to obtain a list set; a classification module, used to classify each test step of the first test case according to the result list to obtain a classification result, wherein each classification result includes a predetermined test step; a recommendation module, used to match the parameters of the predetermined test step with the historical execution results and the existing historical analysis results, and locate the reason for the failure of the execution of the predetermined test step.

[0010] In some embodiments, it also includes: a data module that stores and displays the classification results and their corresponding failure reasons.

[0011] In some embodiments, the present application also proposes an electronic device, including a memory and a processor, wherein the memory stores computer executable instructions, and the processor implements the analysis method of the above embodiment when running the computer executable instructions on the memory.

[0012] In some embodiments, a computer-readable storage medium is further proposed, on which a computer program is stored. When the computer program is executed by a processor, the analysis method of the above embodiment is implemented.

[0013] The test case execution results analysis method and system of the above-mentioned embodiment enable testers to classify and locate the failure causes of failed test cases when executing a large number of automated test cases, thereby saving the testers' time and energy in analyzing failed test cases, improving the efficiency of test result analysis and failure cause location, and significantly improving test efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0015] Figure 1 A schematic diagram of a flow chart of a method for analyzing test case execution results in some embodiments;

[0016] Figure 2 A schematic diagram of a data processing flow in some embodiments;

[0017] Figure 3 A schematic diagram of a classification process in some embodiments;

[0018] Figure 4 A schematic diagram of the structure of a system for analyzing test case execution results in some embodiments;

[0019] Figure 5 It is a schematic diagram of the structure of an electronic device in some embodiments. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0025] In some embodiments, the present application provides a method for analyzing test case execution results, such as Figure 1 As shown, the following steps are included:

[0026] Step S101, obtaining the execution results of the test cases, and obtaining the first failed test case in the execution results.

[0027] In this embodiment, automated test cases are executed and the execution results of these test cases are obtained. These execution results include two results: success and failure. The test case with a failed execution result is marked as the first test case.

[0028] Step S102, performing data processing on each test step of the first test case to obtain a list set.

[0029] Data processing is performed on each test step included in the first test case that failed to execute, to obtain a result list.

[0030] Step S103, classifying each test step of the first test case according to the list set to obtain classification results, each classification result including a predetermined test step.

[0031] According to the result list obtained in step S102, classification processing is performed on each test step in the first test case that failed to execute, and each test step that failed to execute is classified to obtain a classification result.

[0032] Step S104, matching the parameters of the predetermined test step with the historical execution results and the existing historical analysis results, and locating the cause of the failure of the predetermined test step to be executed.

[0033] In this embodiment, the predetermined test steps are matched according to the message sending content, the returned results, the historical execution results and the existing historical analysis result string matching algorithm to find the most matching historical analysis result, thereby recommending the most likely failure cause.

[0034] The method for analyzing the test case execution results of the above-mentioned embodiment enables the tester to classify and locate the failed test cases when executing a large number of automated test cases, thereby saving the tester's time and energy in analyzing failed test cases, improving the efficiency of test result analysis and failure cause location, and significantly improving test efficiency.

[0035] In some embodiments, Figure 2 As shown, the step of performing data processing on each test step of the first test case to obtain a list set includes:

[0036] Step S201, obtaining execution results corresponding to each test step, performing structured conversion on each execution result, and obtaining a conversion result.

[0037] In this embodiment, a test case is composed of one or more test steps. A test case is considered to have passed only when all the test steps have passed. The test return result obtained is usually a JSON string, which needs to be structured, that is, each execution result is deserialized into JSON and converted into a JSON object.

[0038] Step S202, comparing the conversion result with the predetermined result to obtain a dictionary set of failed executions.

[0039] In this embodiment, by traversing the conversion results and comparing them with the expected results, the key and value pairs that are different from the expected results are screened out and stored in the dictionary set DICT. The dictionary set is composed of the keys and values ​​that are not screened, representing the key names and key values ​​in the actual results relative to the expected results. The key names are accompanied by paths, which are used to distinguish the same key names at different levels. The expected results also exist as json strings and are written according to certain rules, containing the verified data, the verification method, and the path path of the key to be verified.

[0040] Step S203, obtaining a list set associated with each test step according to the dictionary set.

[0041] In this embodiment, the key values ​​of the dictionary set are extracted to form an identification list and a result list, and correspond to each failed test case step. That is, the key values ​​in the dictionary set are taken out to form a list corresponding to a test step of a failed test case, forming an identification list. Similarly, the value list of the dictionary set is taken out to form a list corresponding to a test step of a failed test case, forming a result list, and the identification list and the result list are stored in the list set.

[0042] In some embodiments, Figure 3 As shown, the steps of classifying the test steps of the first test case according to the list set and obtaining the classification results include:

[0043] Step S301, obtaining a first test step of a first test case as a first category, and removing the test step from a failed test case list.

[0044] In this embodiment, a test step of a failed test case is taken and marked as category A, and the test step is removed from the failed test case list.

[0045] Step S302, obtaining similarities between other test steps in the first test case and the first test step, classifying the test steps with similarities greater than a preset threshold into a first category, and obtaining a classification result.

[0046] In this embodiment, the similarities of key and value of all failed test cases in category A are calculated based on the similarity algorithm. Those with similarities greater than a preset threshold are included in category A and the test steps are removed from the failed test case test step list.

[0047] According to the preset classification rules, i.e., the similarity of a single key name, the similarity of a single key value, or the similarity of a combined key name and key value, the above steps S301 and S302 are repeatedly executed until all failed test case test steps are analyzed and classified, ensuring that the similarity of the key and value of any two failed test case test steps under the same classification is greater than the preset threshold set by the tester. The test steps in the first test case are classified in the above manner to obtain classification results, and the predetermined test steps contained in each classification result have a high similarity.

[0048] In some embodiments, the parameters of the predetermined test steps are matched with the historical execution results and the existing historical analysis results, and the step of locating the cause of the failure of the predetermined test steps includes: the predetermined test steps are the test steps of the first test case that failed to execute, and the parameters of these test steps include but are not limited to the message sending content and the return result. The test steps of the first test case that failed to execute are matched according to the message sending content, the return result, the historical execution results and the existing historical analysis results using a string matching algorithm, and the historical analysis result corresponding to the maximum matching degree is found, which is the most matching historical analysis result, so as to recommend the most likely cause of failure.

[0049] In this embodiment, each test step in a test case is equivalent to a request. Each request will have some parameters such as request address, request header, request parameter message, etc. The same request address will have different request messages, and each request corresponds to a response result. The string matching algorithm is used to determine whether it is the same request, so as to find the corresponding historical analysis results. The most likely cause of the failure of the test case execution is located from the most matching historical analysis results. Through the above steps, based on historical data and matching algorithms, the most likely cause of failure is recommended, thereby assisting testers to quickly locate the cause of the problem.

[0050] In some embodiments, the above classification results and their corresponding failure reasons are also stored and displayed.

[0051] The classification results and related recommended reasons for failure are displayed to the testers for further review and judgment. After processing, the reasons determined by the analysis are recorded in the analysis results.

[0052] By implementing the above steps, the present application can reduce the analysis time of the tester by about 40%, significantly improving the test efficiency.

[0053] Another embodiment of the present application provides a test case execution result analysis system 400, such as Figure 4 As shown, including:

[0054] The acquisition module 401 is used to acquire the execution result of the test case, and acquire the first failed test case in the execution result;

[0055] The data processing module 402 is used to perform data processing on each test step of the test case to obtain a list set;

[0056] A classification module 403, configured to classify each test step of the first test case according to the result list to obtain a classification result, wherein each classification result includes a predetermined test step;

[0057] The recommendation module 404 is used to match the parameters of the predetermined test step with the historical execution results and the existing historical analysis results to locate the reason why the predetermined test step fails to execute.

[0058] In some embodiments, the system 400 further includes: a data module for storing and displaying the classification results and their corresponding failure reasons.

[0059] The specific implementation of the analysis system in the above embodiment can be found in the above method embodiment part, which will not be repeated here.

[0060] In some embodiments, the present application also proposes an electronic device, including a memory and a processor, wherein the memory stores computer executable instructions, and the processor implements the analysis method of the above embodiment when running the computer executable instructions on the memory.

[0061] Figure 5 A schematic diagram of the structure of an electronic device is provided for another embodiment of the present application. The electronic device 4000 includes a processor 41, and may also include an input device 42, an output device 43, and a memory 44. The input device 42, the output device 43, the memory 44, and the processor 41 are interconnected via a bus.

[0062] The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0063] The input device is used to input data and / or signals, and the output device is used to output data and / or signals. The output device and the input device can be independent devices or an integrated device.

[0064] The processor may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor may also include one or more special-purpose processors, which may include GPUs, FPGAs, etc., for accelerated processing. The memory is used to store program code and data of the network device. The processor is used to call the program code and data in the memory and execute the steps in the above method embodiment. For details, please refer to the description in the method embodiment, which will not be repeated here.

[0065] Understandably, Figure 5 Only a simplified design of the electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, controllers, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0066] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0067] In some embodiments, the present application further proposes a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method of the above embodiment is implemented. Please refer to the above embodiment section for details, which will not be repeated here.

[0068] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0069] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0070] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0071] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for analyzing test case execution results, characterized in that: The following steps are involved: Obtaining the execution results of the test cases, and obtaining the first failed test case in the execution results; Performing data processing on each test step of the test case to obtain a list set; Classify each test step of the first test case according to the result list to obtain a classification result, wherein each classification result includes a predetermined test step; The parameters of the predetermined test step are matched with the historical execution results and the existing historical analysis results to locate the cause of the failure of the predetermined test step to be executed.

2. The analysis method according to claim 1, characterized in that The step of performing data processing on each test step of the test case to obtain a list set comprises: Obtain execution results corresponding to each test step, perform structured conversion on each execution result, and obtain conversion results; Compare the conversion result with the predetermined result to obtain a dictionary set of failed executions; Get the list set associated with each test step based on the dictionary set.

3. The analysis method according to claim 2, characterized in that The step of performing structural conversion on each execution result comprises: Deserialize each execution result into JSON, converting it from a JSON string into a JSON object.

4. The analysis method according to claim 1 or 2, characterized in that The step of classifying each test step of the first test case according to the first result list to obtain a classification result includes: Obtaining a first test step of a first test case as a first category, and removing the test step from the failed test case list; The similarities between other test steps in the first test case and the first test step are obtained, and the test steps with similarities greater than a preset threshold are classified into a first category to obtain a classification result.

5. The analysis method according to claim 1 or 2, characterized in that The step of matching the parameters of the predetermined test step with the historical execution results and the existing historical analysis results comprises: matching the message sending content and the return result of the predetermined test step with the historical execution results and the existing historical analysis results according to a string matching algorithm, and finding the historical analysis result corresponding to the maximum matching degree; The reason why the predetermined test step failed to execute is located from the historical analysis result.

6. The analysis method according to any one of claims 1 to 5, characterized in that Also includes: The classification results and their corresponding failure reasons are stored and displayed.

7. A test case execution result analysis system, characterized in that: include: An acquisition module, used to acquire the execution result of the test case, and acquire the first failed test case in the execution result; A data processing module, used for performing data processing on each test step of the test case to obtain a list set; A classification module, used for classifying each test step of the first test case according to the result list to obtain a classification result, wherein each classification result includes a predetermined test step; The recommendation module is used to match the parameters of the predetermined test steps with the historical execution results and the existing historical analysis results, and locate the reasons for the failure of the execution of the predetermined test steps.

8. The system according to claim 7, characterized in that Also includes: The data module stores and displays the classification results and their corresponding failure reasons.

9. An electronic device, comprising a memory and a processor, wherein the memory stores computer executable instructions, and the processor implements the method according to claims 1-6 when executing the computer executable instructions on the memory.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to claims 1 to 6 is implemented.

Images