A test case processing method and device

Abstract

The present application relates to the data processing technical field in financial technology (Fintech), especially a kind of processing method and device of test case, to improve the efficiency of test case, increase the accuracy of problem positioning.The method comprises: according to target execution times, execute target case, obtain execution result;Determine the failure case of the execution result including failure execution result in the target case;For each failure execution result, based on different analysis dimensions, determine the failure analysis information corresponding to the failure execution result;For each analysis dimension, according to the failure analysis information corresponding to the analysis dimension, the failure execution result is clustered, and the clustering reason corresponding to the clustering result is determined;Based on clustering reason, determine the failure cause of the failure case corresponding to the clustering result, and process the failure case according to the failure cause.

Description

Technical Field

[0001] This invention relates to the field of data processing technology in financial technology (Fintech), and more particularly to a method and apparatus for processing test cases. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology.

[0003] To verify software quality and reduce the possibility of defects, functional testing is generally required before software is officially released. Software automated testing involves implementing a set of test cases in code, which are then executed automatically by a computer at appropriate times to complete the software testing. A test case is a description of a testing task for a specific software product; it is a collection of scripts for automatically executing test verification. After the test cases are executed, the results are analyzed by categorizing failed test cases based on the returned messages, followed by manual analysis. This analysis method relies on manual labor, resulting in low efficiency, low accuracy in problem localization, and an inability to guarantee test quality. Summary of the Invention

[0004] This application provides a method and apparatus for processing test cases, which can improve the efficiency of test cases and increase the accuracy of problem localization.

[0005] This invention provides a method for processing test cases, the method comprising: Execute the target use case based on the target number of executions, and obtain the execution results; The target use case is identified as a failure use case whose execution result includes a failure execution result; For each failed execution result, failure analysis information is determined based on different analysis dimensions. For each analysis dimension, based on the failure analysis information corresponding to the analysis dimension, the failure execution results are clustered to determine the clustering reason corresponding to the clustering result; Based on the clustering reasons, the failure reasons of the failed test cases corresponding to the clustering results are determined, and the failed test cases are processed according to the failure reasons.

[0006] Optionally, the analysis dimensions include log analysis; The failure analysis information corresponding to the failed execution result is determined based on different analysis dimensions, including: Based on the test identifier corresponding to the failed test case, obtain the execution log of the failed execution result; Based on the target keyword, the error log is determined from the execution log, and the error log is used as the failure analysis information; The step of clustering the failed execution results based on the failure analysis information corresponding to the analysis dimension to obtain the clustered failure reasons includes: Determine the text similarity between different error logs; Cluster the failed execution results whose text similarity among error logs is greater than a similarity threshold; For each clustering result, the failure reason of the failed test cases in the clustering result is determined according to the error log corresponding to the clustering result.

[0007] Optionally, the analysis dimensions include return message analysis; The failure analysis information corresponding to the failed execution result is determined based on different analysis dimensions, including: Based on the test identifier corresponding to the failed test case, obtain the return message obtained from executing the failed test case. Here, executing the target test case once corresponds to obtaining N return messages, where N is a positive integer. Determine the target return message from all return messages, determine the return identifier corresponding to the target return message, and use the return identifier as the failure analysis information; The step of clustering the failed execution results based on the failure analysis information corresponding to the analysis dimension to obtain the clustered failure reasons includes: Cluster the failed execution results that return the same identifier; For each clustering result, the failure reason of the failed test cases in the clustering result is determined according to the return identifier corresponding to the clustering result.

[0008] Optionally, before clustering the failed execution results that return the same identifier, the method further includes: Based on the target return message, determine the test case type identifier, the failed target node identifier, and the return code corresponding to the failed test case; The clustering of failed execution results with the same return identifier includes: The failed execution results are clustered based on the use case type identifier, the failed target node identifier, and the return code.

[0009] Optionally, after clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension and determining the clustering reasons corresponding to the clustering results, before determining the failure reasons of the failed test cases corresponding to the clustering results based on the clustering reasons and processing the failed test cases according to the failure reasons, the method further includes: Merge the failure results of clusters with the same clustering reason and belonging to the same failure test case.

[0010] Optionally, determining the failure reasons for the failed test cases corresponding to the clustering results based on clustering reasons includes: Determine the number of failed test cases in each clustering result; The clustering reason corresponding to the clustering result with the most failed test cases is taken as the failure reason of the failed test cases in the clustering result. For failed execution results other than the cluster result with the most failed test cases, the steps of clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension and determining the clustering reason corresponding to the clustering result are re-executed for each analysis dimension until the clustering termination condition is met.

[0011] Optionally, processing the failed use case according to the failure reason includes: Based on the stated cause of failure, a search is performed in the database to obtain the corresponding first solution; The failed use cases are processed according to the first processing scheme; Re-execute the processed failed test cases.

[0012] Optionally, obtaining the execution result of the target use case includes: For each target use case, the target use case is executed according to the pre-set target number of executions to obtain the execution result of the target use case.

[0013] This invention also provides a test case processing apparatus, comprising: The acquisition unit is used to acquire the execution results of the target test case. A determining unit is used to determine failed use cases in the target use case whose execution results include failed execution results; The analysis unit is used to determine the failure analysis information corresponding to each failed execution result based on different analysis dimensions. The clustering unit is used to cluster the failed execution results for each analysis dimension based on the failure analysis information corresponding to the analysis dimension, and to determine the clustering reason corresponding to the clustering result. The processing unit is used to determine the failure reasons of the failed test cases corresponding to the clustering results based on the clustering reasons, and to process the failed test cases according to the failure reasons.

[0014] Optionally, the analysis dimensions include log analysis; The analysis unit is specifically used for: obtaining the execution log of the failed execution result according to the test identifier corresponding to the failed test case; determining the error log from the execution log according to the target keyword, and using the error log as the failure analysis information; The clustering unit is specifically used for: determining the text similarity between different error logs; clustering the failed execution results where the text similarity between error logs is greater than a similarity threshold; and for each clustering result, determining the failure reason of the failed test cases in the clustering result based on the error log corresponding to the clustering result.

[0015] Optionally, the clustering unit is specifically used for: Based on the target return message, determine the test case type identifier, the failed target node identifier, and the return code corresponding to the failed test case; The clustering of failed execution results with the same return identifier includes: The failed execution results are clustered based on the use case type identifier, the failed target node identifier, and the return code.

[0016] Optionally, the analysis dimensions include return message analysis; The analysis unit is specifically used for: obtaining the return message obtained from executing the failed test case according to the test identifier corresponding to the failed test case, wherein executing the target test case once corresponds to obtaining N return messages, where N is a positive integer; determining the target return message from all return messages, determining the return identifier corresponding to the target return message, and using the return identifier as the failure analysis information; The clustering unit is specifically used for: clustering failed execution results with the same return identifier; and for each clustering result, determining the failure reason of the failed test cases in the clustering result based on the return identifier corresponding to the clustering result.

[0017] Optionally, the processing unit is further configured to: Merge the failure results of clusters with the same clustering reason and belonging to the same failure test case.

[0018] Optionally, the processing unit is specifically used for: Determine the number of failed test cases in each clustering result; The clustering reason corresponding to the clustering result with the most failed test cases is taken as the failure reason of the failed test cases in the clustering result. For failed execution results other than the cluster result with the most failed test cases, the steps of clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension and determining the clustering reason corresponding to the clustering result are re-executed for each analysis dimension until the clustering termination condition is met.

[0019] Optionally, the processing unit is specifically used for: Based on the stated cause of failure, a search is performed in the database to obtain the corresponding first solution; The failed use cases are processed according to the first processing scheme; Re-execute the processed failed test cases.

[0020] Optionally, the acquisition unit is used for: For each target use case, the target use case is executed according to the pre-set target number of executions to obtain the execution result of the target use case.

[0021] This invention also provides an electronic device, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0022] This invention also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the method described above.

[0023] In this embodiment of the invention, target use cases are executed to obtain corresponding execution results. Based on the execution results, failed use cases are identified from the target use cases, where failed use cases are target use cases whose execution results include failed execution results. For each failed execution result, the result is analyzed based on different analysis dimensions to determine the corresponding failure analysis information. For each analysis dimension, the failed execution results are clustered based on the identification analysis information corresponding to that dimension to determine the clustering reason for each clustering result. Based on all clustering results, the failure reason for the failed use cases corresponding to the clustering results is determined, and the failed use cases are processed according to the failure reason. This embodiment of the invention clusters and analyzes failed execution results based on different analysis dimensions to locate the failure reasons for failed use cases, and then processes them according to the failure reasons. This reduces human intervention in the problem localization and analysis of failed use cases, effectively improving the testing efficiency of use cases. Furthermore, since there is more than one analysis dimension, the precision and accuracy of problem localization and analysis are improved. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of a possible system architecture provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a test case processing method provided in an embodiment of the present invention; Figure 3A A schematic diagram illustrating inter-node calls provided in an embodiment of the present invention; Figure 3B A flowchart illustrating the method for providing test cases in a specific embodiment of the present invention; Figure 4 A schematic diagram of the structure of a test case device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0027] For ease of understanding, the terms that may be involved in the embodiments of the present invention are defined and explained below.

[0028] Automated testing (Auto Test): To verify software quality and reduce the possibility of defects, software testing is necessary. Software testing can be divided into manual testing and automated testing according to the testing methods. Automated testing is a process that transforms human-driven testing into machine-executed testing. Compared to manual testing, automated testing can save significant human and time resources and improve testing efficiency.

[0029] Test case: A test case is a description of a test task performed on a specific software product. Automated test cases are generally a collection of scripts that automatically execute test verifications. Currently, the design and script writing of automated test cases still rely on manual operation.

[0030] Flow: Flow refers to the invocation of a request. A complete flow contains a complete request invocation path and the corresponding request message and return message.

[0031] Sandbox: In the field of computer security, a sandbox is a technology and a security mechanism that provides an isolated environment for running programs. It is typically used to experiment with programs from untrusted sources, programs with destructive potential, or programs whose intent is unknown.

[0032] See Figure 1 The diagram shown is an application architecture diagram of the test case processing method in an embodiment of the present invention. The system architecture includes a terminal device 100, a server 200, and a database 300.

[0033] Terminal device 100 can be a mobile or a fixed electronic device. Examples include mobile phones, tablets, laptops, desktop computers, various wearable devices, smart TVs, in-vehicle devices, or other electronic devices capable of performing the aforementioned functions. Terminal device 100 can send test requests to the server and receive execution results.

[0034] Server 200 can provide various network services to terminal device 100, and server 200 can use cloud computing technology for information processing. Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0035] Database 300 can be any type of database, preferably a database cluster, which can utilize cloud computing technology for information processing. Database 300 stores various failure causes and corresponding solutions.

[0036] Terminal device 100 and server 200 can be connected via the Internet to enable communication between them. Optionally, the Internet described above uses standard communication technologies and / or protocols. The Internet is typically the Internet, but can also be any network, including but not limited to any combination of Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless networks, private networks or virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0037] Specifically, server 200 may include processor 110 (Center Processing Unit, CPU), memory 120, input device 130 and output device 140, etc. Input device 130 may include keyboard, mouse, touch screen, etc., and output device 140 may include display device, such as liquid crystal display (LCD), cathode ray tube (CRT), etc.

[0038] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In this embodiment of the invention, the memory 120 may be used to store a program for processing test cases in this embodiment of the invention.

[0039] The processor 110 executes the steps of the processing method for any test case in the embodiments of the present invention according to the program instructions stored in the memory 120 by calling the program instructions.

[0040] Based on the above architecture, embodiments of the present invention provide a method for processing test cases, such as... Figure 2 As shown, the test case processing method provided in this embodiment of the invention includes the following steps: Step 201: Obtain the execution results of the target use case.

[0041] If the target test case is executed only once, the discovery rate of some problems is low, and the testing effect is poor. In a preferred embodiment of the present invention, the target test case is executed according to a set target number of executions. The execution result of the target test case is then obtained, including: For each target use case, the target use case is executed according to the pre-set target number of executions to obtain the execution result of the target use case.

[0042] In practice, a target execution count, such as N times, can be pre-set. For the same target use case, the test is repeated N times according to the target execution count, resulting in N execution results. This is because when the number of executions is less than the target execution count, the discovery rate of some issues is low, and the testing effect is poor; when the number of executions is more than the target execution count, the discovery rate of issues does not increase significantly, thus increasing the testing cost. Specifically, in this embodiment of the invention, the target execution count can be set to 3 times, that is, each target use case is executed 3 times, resulting in 3 execution results. If the number of target use cases is 1000, then 3000 execution results need to be analyzed.

[0043] Step 202: Identify the failed use cases in the target use case whose execution results include failed execution results.

[0044] In practice, the execution result of a target use case can be categorized into the following three cases: All executions of the target test case corresponding to the target number of executions resulted in success. The execution result for the target use case, corresponding to the target number of executions, is partial failure. The execution result for the target use case, corresponding to the target number of executions, is all failures.

[0045] In the three scenarios described above, target use cases that execute completely successfully are marked as success cases, and the execution results are directly reported. Target use cases that partially or completely fail are considered failure use cases. For these failures, it is necessary to analyze the problem and take appropriate action.

[0046] Step 203: For each failed execution result, determine the failure analysis information corresponding to the failed execution result based on different analysis dimensions.

[0047] In practice, failure analysis information corresponding to failed execution results is determined based on two or more analysis dimensions. For example, failure analysis information for each failed execution result can be determined based on error logs and return messages. Other information may also be included for specific analysis dimensions; this embodiment is merely an example and not a limitation.

[0048] Step 204: For each analysis dimension, based on the failure analysis information corresponding to the analysis dimension, cluster the failure execution results and determine the clustering reasons corresponding to the clustering results.

[0049] In the specific implementation process, different methods are used to process the failure analysis information for different analysis dimensions, and the processing results are classified. Under the same analysis dimension, the failure execution results will be clustered, and the clustering reason can be determined based on the failure analysis information of each cluster result.

[0050] Step 205: Based on the clustering reasons, determine the failure reasons of the failed test cases corresponding to the clustering results, and process the failed test cases according to the failure reasons.

[0051] Specific reasons for failure can be categorized into reproducible bugs (computer science vulnerabilities), environmental issues, and random bugs. For different reasons, the corresponding solutions can be retrieved and implemented based on the pre-stored correspondence between failure reasons and solutions. Additionally, there is a scenario where all target test cases fail to execute, but the logs and return codes for those test cases show no anomalies. In this case, the failure reason for the failed test case is classified as a requirement change. For failures due to requirement changes, the handling method is manual modification of the test cases.

[0052] In this embodiment of the invention, target use cases are executed to obtain corresponding execution results. Based on the execution results, failed use cases are identified from the target use cases, where failed use cases are target use cases whose execution results include failed execution results. For each failed execution result, the result is analyzed based on different analysis dimensions to determine the corresponding failure analysis information. For each analysis dimension, the failed execution results are clustered based on the identification analysis information corresponding to that dimension to determine the clustering reason for each clustering result. Based on all clustering reasons, the failure reason for the failed use case corresponding to the clustering result is determined, and the failed use case is processed according to the failure reason. This embodiment of the invention clusters and analyzes failed execution results based on different analysis dimensions to locate the failure reason for the failed use case, and then processes it according to the failure reason. This reduces human intervention in the problem localization and analysis of failed use cases, effectively improving the testing efficiency of use cases. Furthermore, since there is more than one analysis dimension, the precision and accuracy of problem localization and analysis are improved.

[0053] The embodiments of the present invention include different analysis dimensions. The following will take log analysis and return message analysis as examples for detailed explanation.

[0054] For log analysis as the analysis dimension, the method for determining the failure analysis information corresponding to the failed execution result based on different analysis dimensions includes: Based on the test identifier corresponding to the failed test case, obtain the execution log of the failed execution result; Based on the target keyword, the error log is determined from the execution log, and the error log is used as the failure analysis information; The step of clustering the failed execution results based on the failure analysis information corresponding to the analysis dimension to obtain the clustered failure reasons includes: Determine the text similarity between different error logs; Cluster the failed execution results whose text similarity among error logs is greater than a similarity threshold; For each clustering result, the failure reason of the failed test cases in the clustering result is determined according to the error log corresponding to the clustering result.

[0055] Automated test logs refer to the various data output by automated test scripts written in different scripting languages ​​within an automated testing framework, following a fixed format, including data from servers, network devices, operating systems, databases, and application software. The log analysis method in this embodiment primarily involves comparing automated test logs within the testing period and identifying, judging, and verifying them based on keywords within the logs.

[0056] In practice, the failure test case identifier and the corresponding test identifier are used as input to obtain the execution log of the failed execution result. For example, the mapping between the failure test case ID and the sequence number is used as input to obtain the execution log of that failure test case. The specific input can be the sequence number, and the processing logic is as follows: Based on the input pipeline, analyze the execution logs of all subsystems through which the failed test case was executed to see if the keyword "error" exists. For error logs containing the keyword "error", noise reduction is performed, and the text similarity of the noise-reduced error logs is compared. The error logs are then clustered based on the text similarity, for example, error logs with a similarity of more than 80% are grouped into one category.

[0057] Thus, the output is as follows: Error log 1 with similarity greater than 80%: Serial number 1, ..., Serial number M; Error log 2 with a similarity greater than 80%: Serial number 1', ..., Serial number M'; ... In this way, for each output clustering result, the clustering reason can be determined based on the error log corresponding to the clustering result, and then the failure reason of the failed execution result corresponding to the serial number in the clustering result can be determined.

[0058] It should be noted that not all logs of target use cases that fail are error logs. Error logs, i.e., logs containing the keyword "error", only occur when the program catches an exception. Therefore, clustering based solely on error logs is inaccurate, and clustering and analysis based on other analytical dimensions are also necessary.

[0059] On the other hand, the analysis dimension is the analysis of the returned message; The failure analysis information corresponding to the failed execution result is determined based on different analysis dimensions, including: Based on the test identifier corresponding to the failed test case, obtain the return message obtained from executing the failed test case. Here, executing the target test case once corresponds to obtaining N return messages, where N is a positive integer. Determine the target return message from all return messages, determine the return identifier corresponding to the target return message, and use the return identifier as the failure analysis information; The step of clustering the failed execution results based on the failure analysis information corresponding to the analysis dimension to obtain the clustered failure reasons includes: Cluster the failed execution results that return the same identifier; For each clustering result, the failure reason of the failed test cases in the clustering result is determined according to the return identifier corresponding to the clustering result.

[0060] In practice, based on the test identifier (serial number) of the failed test case, N return messages can be obtained. Each return message corresponds to a node or subsystem. These multiple return messages are traversed to find the deepest node where the exception occurred, and the return message corresponding to this deepest node is taken as the target return message. For example, if a failed test case is executed by four nodes with a calling relationship between them—the first node calling the second, the second calling the third, and the third calling the fourth—the fourth node is the deepest node. Based on the serial number of this failed test case, multiple return messages are obtained, each corresponding to a node. Only the return messages corresponding to the second and third nodes indicate execution failure. Since the third node is a deeper node based on the calling relationship, its return message is the target return message.

[0061] Then, based on the obtained target return message, the return identifier can be determined. This return identifier can be a return code or other identifier, and the number of determined return identifiers is unlimited. Based on the return identifier, the failed execution results with the same return identifier are clustered, and the clustering reason corresponding to the clustering result is determined, thereby obtaining the failure reason of the failed test case corresponding to the clustering result.

[0062] Furthermore, before clustering the failed execution results that return the same identifier, the method further includes: Based on the target return message, determine the test case type identifier, the failed target node identifier, and the return code corresponding to the failed test case; The clustering of failed execution results with the same return identifier includes: The failed execution results are clustered based on the use case type identifier, the failed target node identifier, and the return code.

[0063] In practice, the returned message contains a return code. Different return codes correspond to different processing results. For example, a return code of 0001 indicates a timeout, 9006 indicates a downstream problem, 1234 indicates a system exception, and 0000 indicates successful execution. Therefore, the return code can be used to determine whether the execution result is normal.

[0064] For each failed test case, the returned message of that test case can also identify the problematic node, making the testing more targeted. There are calling relationships between nodes, such as... Figure 3A As shown, the nodes include systems A, B, and C, with system A being the entry request node. The calling relationship between nodes is that system A calls system B, and system B calls system C. During the execution of failed test cases, nodes that do not encounter exceptions will return a code of 0000, indicating that the node executed successfully. For subsystems that encounter exceptions, the corresponding return code will not be 0000.

[0065] Therefore, we can traverse these multiple return messages to find the deepest failed target node, and use the return message corresponding to the deepest failed target node as the target return message. Figure 3A If both system B and system C malfunction, since system C is the deepest node with the problem, system C will be designated as the failure target node, and the return message corresponding to system C will be designated as the target return message.

[0066] On the other hand, each target use case also corresponds to a different use case type. For example, the use case type can be a request transaction type, and different use case types are distinguished by use case type identifiers. Therefore, failed use cases can also be clustered according to use case type.

[0067] In this embodiment of the invention, the use case type, the failed target node, and the return code are combined to jointly influence the clustering result. Specifically, failed execution results with the same use case type identifier, failed target node identifier, and return code are grouped into the same cluster.

[0068] The specific input is still the serial number, and the processing logic is as follows: Based on the input pipeline, analyze the return messages obtained by all subsystems through which the failed test case was executed; The call chain tree between subsystems is analyzed, and the deepest failed target subsystem is found. The return identifier corresponding to the failed test case is determined from the obtained return message. The specific return identifier may include the failed target subsystem identifier, the request transaction code, and the error return code. Then, the return identifiers of each transaction are compared, and transactions with the same return identifier are grouped into the same category.

[0069] Thus, the output is as follows: Failed target subsystem 1 + request transaction code 1 + return code 1: Serial number 1, ..., Serial number P; Failed target subsystem 1 + request transaction code 1 + return code 2: serial number 1', ..., serial number P'; ... In this way, for each output clustering result, the clustering reason corresponding to the clustering result can be determined according to the return identifier corresponding to the clustering result, and then the failure reason of the failed execution result corresponding to the serial number in the clustering result can be determined.

[0070] Preferably, after clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension and determining the clustering reasons corresponding to the clustering results, before determining the failure reasons of the failed test cases corresponding to the clustering results based on the clustering reasons and processing the failed test cases according to the failure reasons, the method further includes: Merge the failure results of clusters with the same clustering reason and belonging to the same failure test case.

[0071] In the specific implementation process, input failed test cases need to be preprocessed. For test cases with the same failure reason, data normalization is performed. For test cases with different failure reasons, multiple results are processed. The final preprocessed result is the failed test case ID plus the failure reason. In this way, different failure reasons for the same test case are ultimately processed into multiple failed execution results.

[0072] Preferably, determining the failure reasons for the failed test cases corresponding to the clustering results based on clustering reasons includes: Determine the number of failed test cases in each clustering result; The clustering reason corresponding to the clustering result with the most failed test cases is taken as the failure reason of the failed test cases in the clustering result. For failed execution results other than the cluster result with the most failed test cases, the steps of clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension and determining the clustering reason corresponding to the clustering result are re-executed for each analysis dimension until the clustering termination condition is met.

[0073] In practice, the clustering termination condition can be set to either the number of failed test cases in the largest clustering result being 1 or the number of clusters being equal. After clustering the failed execution results according to different analysis dimensions, the number of failed test cases in each clustering result is calculated. The clustering result with the most failed test cases is determined, i.e., the clustering reason corresponding to the cluster, and this clustering reason is used as the failure reason for all failed test cases in that cluster. Then, the failed test cases with determined failure reasons are excluded. For the remaining failed execution results, the steps described above for each analysis dimension, clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension, and determining the clustering reason corresponding to the clustering result are repeated. That is, the clustering of failed execution results is repeated until the clustering termination condition is met, at which point the clustering ends. At this point, the failure reason for each failed test case in each clustering result can be determined.

[0074] It should be noted that in this embodiment of the invention, since clustering for each failed test case is performed more than once—that is, clustering of failed execution results based on different analysis dimensions—the number of analysis dimensions corresponds to the number of clusterings of the failed execution results. For example, for the same failed execution result, based on log analysis, the failed execution result is assigned to one cluster, and based on the return identifier, the failed execution result is assigned to another cluster. Once the cluster with the largest number of failed execution results is determined, all failed execution results in that cluster need to be excluded. Therefore, the remaining failed execution results are re-clustered, excluding those for which the cause of failure has been determined.

[0075] Further, processing the failed use case according to the failure reason includes: Based on the stated cause of failure, a search is performed in the database to obtain the corresponding first solution; The failed use cases are processed according to the first processing scheme; Re-execute the processed failed test cases.

[0076] In practice, the database pre-stores the correspondence between failure reasons and handling solutions. Therefore, based on the determined failure reason, the corresponding first handling solution can be retrieved from the database, and the failure case can be automatically processed according to the first handling solution before retrying.

[0077] On the other hand, if no corresponding solution is found in the database based on the reason for failure, then processing the failed use case according to the reason for failure further includes: If no corresponding solution is found in the database based on the stated reason for failure; Then determine the second processing solution corresponding to the cause of failure; The failed test cases are processed according to the second processing scheme, and the processed failed test cases are re-executed; The second processing solution is associated with the failure reason and stored in the database.

[0078] In other words, if the database does not store a corresponding handling solution for the failure reason, a second handling solution can be determined manually or through other channels. Then, the failed test case is processed according to the second handling solution, and the processed test case is re-executed. Once the failed test case is successfully re-executed, the second handling solution and the failure reason can be associated and stored in the database so that if the same failure reason occurs again, the second handling solution can be retrieved and processed.

[0079] The following specific examples illustrate the implementation process of the test case processing method provided in this embodiment of the invention. The specific implementation process is as follows: Figure 3B As shown, the process includes the following: The automated test cases are executed a certain number of times according to the configuration. After execution, all test results are collected, including test case ID, test case execution result ID, test case running result, test case serial number, test case name and other test case profile information.

[0080] Based on the collected use case profiles, the use cases are categorized into fully successful, partially failed, and completely failed. The partially and completely failed use cases are then processed by the automated analysis module.

[0081] The automatic analysis module categorizes the input test cases, serial numbers, and other information through logs and recorded traffic, ultimately clustering them into error logs and return codes from the error reporting subsystem.

[0082] Sort the test cases by the number of test cases after clustering them according to the error reasons. Extract the failure reasons corresponding to the cluster with the most test cases (corresponding to the error log or the return code of the subsystem). Repeat the above automatic analysis steps for the remaining test cases until the number of failed test cases in the final cluster is equal or equal to 1, and then enter the final analysis state.

[0083] After test case clustering is completed, the reasons for clustering failures are recorded. If the failure matches the experience library, the solutions in the experience library are automatically processed, and the failed test cases are retried. If the failure does not match the experience library, manual processing is performed, and the processing results are recorded in the failure experience library for manual confirmation of the failure reason, whether it is a bug, an environmental issue, or a random bug. The corresponding error logs or return codes are annotated, and the clustering failure reasons and solutions are entered into the failure experience library.

[0084] This invention also provides an apparatus for test cases, the apparatus being as follows: Figure 4 As shown, it includes: Acquisition unit 401 is used to acquire the execution result of the target test case; Determining unit 402 is used to determine the failed use cases in the target use case whose execution results include failed execution results; Analysis unit 403 is used to determine the failure analysis information corresponding to each failed execution result based on different analysis dimensions; Clustering unit 404 is used to cluster the failed execution results for each analysis dimension based on the failure analysis information corresponding to the analysis dimension, and determine the clustering reason corresponding to the clustering result. The processing unit 405 is used to determine the failure reason of the failed test case corresponding to the clustering result based on the clustering reason, and to process the failed test case according to the failure reason.

[0085] Optionally, the analysis dimensions include log analysis; The analysis unit is specifically used for: obtaining the execution log of the failed execution result according to the test identifier corresponding to the failed test case; determining the error log from the execution log according to the target keyword, and using the error log as the failure analysis information; The clustering unit is specifically used for: determining the text similarity between different error logs; clustering the failed execution results where the text similarity between error logs is greater than a similarity threshold; and for each clustering result, determining the failure reason of the failed test cases in the clustering result based on the error log corresponding to the clustering result.

[0086] Optionally, the analysis dimensions include return message analysis; The analysis unit is specifically used for: obtaining the return message obtained from executing the failed test case according to the test identifier corresponding to the failed test case, wherein executing the target test case once corresponds to obtaining N return messages, where N is a positive integer; determining the target return message from all return messages, determining the return identifier corresponding to the target return message, and using the return identifier as the failure analysis information; The clustering unit is specifically used for: clustering failed execution results with the same return identifier; and for each clustering result, determining the failure reason of the failed test cases in the clustering result based on the return identifier corresponding to the clustering result.

[0087] Optionally, the clustering unit is specifically used for: Based on the target return message, determine the test case type identifier, the failed target node identifier, and the return code corresponding to the failed test case; The clustering of failed execution results with the same return identifier includes: The failed execution results are clustered based on the use case type identifier, the failed target node identifier, and the return code.

[0088] Optionally, the processing unit is further configured to: Merge the failure results of clusters with the same clustering reason and belonging to the same failure test case.

[0089] Optionally, the processing unit is specifically used for: Determine the number of failed test cases in each clustering result; The clustering reason corresponding to the clustering result with the most failed test cases is taken as the failure reason of the failed test cases in the clustering result. For failed execution results other than the cluster result with the most failed test cases, the steps of clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension and determining the clustering reason corresponding to the clustering result are re-executed for each analysis dimension until the clustering termination condition is met.

[0090] Optionally, the processing unit is specifically used for: Based on the stated cause of failure, a search is performed in the database to obtain the corresponding first solution; The failed use cases are processed according to the first processing scheme; Re-execute the processed failed test cases.

[0091] Optionally, the acquisition unit is used for: For each target use case, the target use case is executed according to the pre-set target number of executions to obtain the execution result of the target use case.

[0092] Based on the same principle, the present invention also provides an electronic device, such as... Figure 5 As shown, it includes: It includes a processor 501, a memory 502, a transceiver 503, and a bus interface 504, wherein the processor 501, the memory 502, and the transceiver 503 are connected through the bus interface 504; The processor 501 is used to read the program in the memory 502 and execute the following methods: Execute the target use case based on the target number of executions, and obtain the execution results; The target use case is identified as a failure use case whose execution result includes a failure execution result; For each failed execution result, failure analysis information is determined based on different analysis dimensions. For each analysis dimension, based on the failure analysis information corresponding to the analysis dimension, the failure execution results are clustered to determine the clustering reason corresponding to the clustering result; Based on the clustering reasons, the failure reasons of the failed test cases corresponding to the clustering results are determined, and the failed test cases are processed according to the failure reasons.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable test case device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable test case device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable test case to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable test case device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0097] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for processing test cases, characterized in that, include: For each target use case in the target use case set, the target use case is executed according to a pre-set target number of executions to obtain the execution result of the target use case; The target use case set is determined to include failed use cases whose execution results include failed execution results; For each failed execution result, based on different analysis dimensions, the failure analysis information corresponding to the failed execution result is determined. For each analysis dimension, the failed execution results are clustered according to the failure analysis information corresponding to the analysis dimension, and the clustering reason corresponding to the clustering result is determined. The analysis dimensions include log analysis and return message analysis. Determine the number of failed test cases in each clustering result; take the clustering reason corresponding to the clustering result with the most failed test cases as the failure reason of the failed test cases in the clustering result; for the failed execution results other than the clustering result with the most failed test cases, re-execute the steps of clustering the failed execution results according to the failure analysis information corresponding to the analysis dimension for each analysis dimension and determining the clustering reason corresponding to the clustering result, until the clustering termination condition is met; The failed test cases are processed according to the reasons for the failure; For each failed execution result, based on different analysis dimensions, the method determines the corresponding failure analysis information. For each analysis dimension, based on the failure analysis information corresponding to that dimension, the failed execution results are clustered to determine the clustering reason. This includes: When the analysis dimension is log analysis, the execution log of the failed execution result is obtained according to the test identifier corresponding to the failed test case. Error logs are determined from the execution logs according to the target keyword, and the error logs are used as the failure analysis information. The text similarity between different error logs is determined, and the failed execution results with text similarity greater than the similarity threshold between error logs are clustered. For each clustering result, the clustering reason of the failed test cases in the clustering result is determined according to the error logs corresponding to the clustering result. When the analysis dimension is return message analysis, the return messages obtained from executing the failed test cases are obtained according to the test identifiers corresponding to the failed test cases. Each execution of a target test case corresponds to N return messages, where N is a positive integer. The call chain between the nodes traversed during the execution of the failed test cases is determined based on the N return messages. Based on the call chain, the return message corresponding to the deepest node where the anomaly occurred is determined from the N return messages as the target return message. The return identifier corresponding to the target return message is determined, and this return identifier is used as the failure analysis information. Failed execution results with the same return identifier are clustered. For each clustering result, the clustering reason for the failed test cases in the clustering result is determined based on the return identifier corresponding to the clustering result.

2. The method as described in claim 1, characterized in that, Before clustering the failed execution results that return the same identifier, the method further includes: Based on the target return message, determine the test case type identifier, the failed target node identifier, and the return code corresponding to the failed test case; The clustering of failed execution results with the same return identifier includes: The failed execution results are clustered based on the use case type identifier, the failed target node identifier, and the return code.

3. The method as described in claim 1 or 2, characterized in that, After clustering the failed execution results based on the failure analysis information corresponding to the analysis dimension and determining the clustering reasons corresponding to the clustering results, and before processing the failed use cases based on the failure reasons, the method further includes: Merge the failure results of clusters with the same clustering reason and belonging to the same failure test case.

4. The method as described in claim 1 or 2, characterized in that, The process of handling the failed use case according to the failure reason includes: Based on the stated cause of failure, a search is performed in the database to obtain the corresponding first solution; The failed use cases are processed according to the first processing scheme; Re-execute the processed failed test cases.

5. A test case processing device, characterized in that, include: The acquisition unit is used to execute each target use case in the target use case set according to a pre-set target execution number, and obtain the execution result of the target use case; A determining unit is configured to determine failed use cases in the target use case set whose execution results include failed execution results; The analysis unit is used to determine the failure analysis information corresponding to each failed execution result based on different analysis dimensions. The clustering unit is used to cluster the failed execution results for each analysis dimension based on the failure analysis information corresponding to the analysis dimension, and to determine the clustering reason corresponding to the clustering result; wherein, the analysis dimension includes log analysis and return message analysis; The processing unit is configured to: determine the number of failed test cases in each clustering result; identify the clustering reason corresponding to the clustering result with the most failed test cases as the failure reason for the failed test cases in that clustering result; for failed execution results other than the clustering result with the most failed test cases, re-execute the steps of clustering the failed execution results for each analysis dimension based on the failure analysis information corresponding to the analysis dimension, and determining the clustering reason corresponding to the clustering result, until the clustering termination condition is met; and process the failed test cases according to the failure reason. The analysis unit and the clustering unit are specifically used for: When the analysis dimension is log analysis, the execution log of the failed execution result is obtained according to the test identifier corresponding to the failed test case. Error logs are determined from the execution logs according to the target keyword, and the error logs are used as the failure analysis information. The text similarity between different error logs is determined, and the failed execution results with text similarity greater than the similarity threshold between error logs are clustered. For each clustering result, the clustering reason of the failed test cases in the clustering result is determined according to the error logs corresponding to the clustering result. When the analysis dimension is return message analysis, the return messages obtained from executing the failed test cases are obtained according to the test identifiers corresponding to the failed test cases. Each execution of a target test case corresponds to N return messages, where N is a positive integer. The call chain between the nodes traversed during the execution of the failed test cases is determined based on the N return messages. Based on the call chain, the return message corresponding to the deepest node where the anomaly occurred is determined from the N return messages as the target return message. The return identifier corresponding to the target return message is determined, and this return identifier is used as the failure analysis information. Failed execution results with the same return identifier are clustered. For each clustering result, the clustering reason for the failed test cases in the clustering result is determined based on the return identifier corresponding to the clustering result.

6. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

Images