Test case evaluation methods, apparatus, equipment, media and procedures products

Abstract

This disclosure provides a test case evaluation method, apparatus, device, medium, and program product, relating to the field of computer technology, specifically the field of software testing technology, and applicable to the field of financial technology. The test case evaluation method includes: determining the similarity value between any two test cases in a test case set; clustering the test cases in the test case set according to the similarity value to obtain at least one test case cluster set; and determining the evaluation value of each test case in each test case cluster set according to evaluation rules, wherein the evaluation rules characterize the rules for the reference priority order of evaluating test cases.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the field of software testing technology, and more specifically to a test case evaluation method, apparatus, device, medium, and program product. Background Technology

[0002] Software testing is an important part of the current software development process, and the quality and efficiency of the work are very important. Test cases are descriptions of the testing tasks for a specific software product, reflecting the test plan, methods, techniques and strategies. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a test case evaluation method, apparatus, device, medium and program product.

[0004] According to a first aspect of this disclosure, a test case evaluation method is provided, comprising: determining a similarity value between any two test cases in a test case set; clustering the test cases in the test case set according to the similarity value to obtain at least one test case cluster set; and determining an evaluation value for each test case in each test case cluster set according to an evaluation rule, wherein the evaluation rule characterizes a rule for the reference priority order of evaluating test cases.

[0005] According to embodiments of this disclosure, test cases include input parameter data and step description data, and similarity values ​​include a first similarity value, a second similarity value, and a target similarity value. Determining the similarity value between any two test cases in the test case set includes: determining the first similarity value between any two test cases based on the input parameter data; determining the second similarity value between any two test cases based on the step description data; and determining the target similarity value between any two test cases in the test case set based on the first similarity value and the second similarity value.

[0006] According to embodiments of this disclosure, the test case cluster set includes a first cluster set and a second cluster set; clustering the test cases in the test case set according to similarity values ​​to obtain at least one test case cluster set includes: taking two test cases with a first similarity value greater than or equal to a first threshold as elements of the first cluster set to obtain at least one first cluster set; and taking two test cases with a second similarity value greater than or equal to a second threshold as elements of the second cluster set to obtain at least one second cluster set.

[0007] According to embodiments of this disclosure, the evaluation value includes a base evaluation value and a target evaluation value. Determining the evaluation value of each test case in each test case cluster set according to the evaluation rules includes: initializing the base evaluation value of each test case; and adjusting the base evaluation value according to at least one of a first dimension, a second dimension, and a third dimension to obtain the target evaluation value of each test case, wherein the first dimension represents whether the test case is executed by the execution script, the second dimension represents whether the execution script of the test case is executed successfully, and the third dimension represents whether the test case is referenced.

[0008] The test case evaluation method according to embodiments of this disclosure further includes: for any first cluster set, determining a first cluster set reference priority ratio based on the ratio between the sum of the basic evaluation values ​​of automated test cases and the sum of the target evaluation values ​​of all test cases in the first cluster set, wherein the automated test cases are test cases executed by the execution script corresponding to the first dimension; for any second cluster set, determining a second cluster set reference priority ratio based on the ratio between the sum of the basic evaluation values ​​of automated test cases and the sum of the target evaluation values ​​of all test cases in the second cluster set; and determining a target reference priority ratio based on the first cluster set reference priority ratio and the second cluster set reference priority ratio.

[0009] The test case evaluation method according to the embodiments of this disclosure further includes at least one of the following: sorting the first cluster set and the second cluster set according to the target reference priority ratio to obtain the sorted first cluster set and the second cluster set; or sorting the test cases of the first cluster set and the second cluster set according to the target similarity value to obtain the sorted test cases.

[0010] The test case evaluation method according to the embodiments of the present disclosure further includes: determining a reference test case for implementing automated test cases from a set of test cases based on the evaluation value; and determining at least one of the following based on the execution script corresponding to the reference test case: execution script reference data and execution script execution time reference data.

[0011] Another aspect of this disclosure provides a test case evaluation apparatus, comprising: a similarity value determination module, a clustering module, and an evaluation value determination module. The similarity value determination module is used to determine the similarity value between any two test cases in a test case set; the clustering module is used to cluster the test cases in the test case set according to the similarity value, obtaining at least one test case cluster set; the evaluation value determination module is used to determine the evaluation value of each test case in each test case cluster set according to evaluation rules, wherein the evaluation rules characterize the rules for the reference priority order of evaluating test cases.

[0012] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described test case evaluation method.

[0013] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described test case evaluation method.

[0014] Another aspect of this disclosure provides a computer program product, including a computer program stored on at least one of a readable storage medium and an electronic device, wherein the computer program, when executed by a processor, implements the above-described test case evaluation method.

[0015] The test case evaluation method according to the embodiments of this disclosure can achieve at least one of the following technical effects.

[0016] (1) By clustering the test cases in the test case set according to the similarity value between any two test cases in the test case set, a test case cluster set is obtained. Test cases belonging to the same test case cluster set have a higher degree of similarity. The similarity is reflected in, for example, corresponding to the same or similar test functions. This will enable testers to use the test cases in the test case cluster set as a reference when they need to find test cases in the test case set for current testing, thus efficiently providing testers with more sufficient reference resources.

[0017] (2) Evaluation rules characterize the reference priority order of evaluation test cases. Therefore, the evaluation value of a test case determined by the evaluation rules can reflect the reference significance of the test case. Thus, the reference priority of each test case in the test case set can be accurately provided to testers, making it easier for testers to distinguish and efficiently determine the test cases that are more relevant to the current testing task from the test case set. Attached Figure Description

[0018] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The diagram schematically illustrates the architecture of test case evaluation methods, apparatus, devices, media, and program products according to embodiments of the present disclosure.

[0020] Figure 2 A flowchart illustrating a test case evaluation method according to an embodiment of the present disclosure is shown schematically;

[0021] Figure 3 This schematically illustrates a flowchart of a test case evaluation method according to another embodiment of the present disclosure for determining the similarity value between any two test cases in a test case set;

[0022] Figure 4 A flowchart illustrating the process of obtaining at least one cluster set of test cases according to another embodiment of the present disclosure is shown.

[0023] Figure 5 A flowchart illustrating the determination of the evaluation value of a test case according to another embodiment of the present disclosure is shown.

[0024] Figure 6 A flowchart illustrating a test case evaluation method according to yet another embodiment of this disclosure is shown schematically;

[0025] Figure 7 A schematic block diagram of a test case evaluation apparatus according to an embodiment of the present disclosure is shown; and

[0026] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a test case evaluation method according to an embodiment of the present disclosure. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0031] With the increasing maturity of internet and computer technologies, a large number of business operations are shifting to online systems, moving away from manual paper-based processes. As information technology advances, the architecture of various application systems becomes increasingly complex, and the functions of various software systems are also increasing. For software development departments, the number of functions developed and launched in the system is growing larger and larger with each iteration of version development.

[0032] Software testing is a crucial part of the current software development process, and its quality and efficiency are paramount. Test cases describe the testing tasks performed on a specific software product, reflecting the testing plan, methods, techniques, and strategies. When developing software, companies need to test newly developed or updated versions and features. Test cases, as descriptions of software tests, are a type of testing asset. Tests can be executed based on test cases, either manually or through automated scripts.

[0033] For example, when developing and optimizing features for large systems, a large number of test cases are written for each version. This is because many features are not part of the current version's changes, but to ensure that version modifications do not affect these features, a test case asset library is typically built to improve test case reusability and reduce the workload of repeatedly writing test cases for the same features in each version.

[0034] Once a version enters the testing phase, test cases from the test case asset library are referenced by testers and incorporated into the current version to test the required content. After testers complete the functional testing and verification of the corresponding test case, they provide feedback on the test case status, forming a closed loop of test case verification results. At the end of the testing phase, test quality control personnel compile the verification status of all test cases, and after evaluation, make a decision on whether the version is ready for delivery and production deployment.

[0035] When the number of test cases in the test case asset library is small, testers can perform verification manually during version testing. However, as the functionality of each system becomes larger and the interaction between them becomes more complex, the number of test cases accumulated in the test case asset library is also increasing. With the current trend of increasingly shorter version development cycles, the traditional manual verification mode can no longer meet the needs of the current version development pace.

[0036] In this context, many large enterprises have introduced automated testing technology to resolve the contradiction between limited test execution personnel resources and the large number of test cases that need to be verified in the test case asset library. Since the test cases in the test case asset library need to be tested and verified in each version, achieving a high cost-effectiveness of automated testing has become the main goal of various enterprises in the process of implementing automated testing projects.

[0037] Due to limitations in available human resources, the number of personnel capable of developing automated test scripts is limited, while the number of test cases in the test case asset library is enormous. Automated test developers lack effective decision-making basis for implementation plans when scheduling the engineering implementation of test cases requiring automation. Furthermore, when developing automated test scripts for test cases included in the implementation plan, the inability to effectively analyze the already automated assets in the test case library leads to automated test developers relying on their own experience or subjective judgment for analysis and reuse each time they create a script, resulting in low efficiency in automated script writing.

[0038] It should be noted that the methods and apparatus determined in the embodiments of this disclosure can be used in the financial field, or in any field other than the financial field. The embodiments of this disclosure do not limit the application field of the test case evaluation methods and apparatus.

[0039] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0040] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0041] Figure 1 The schematic diagram illustrates the architecture of a test case evaluation method and apparatus according to embodiments of the present disclosure.

[0042] like Figure 1As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0043] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only). Terminal devices 101, 102, and 103 can store test case sets.

[0044] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0045] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated based on user requests) to the terminal devices. Server 105 can evaluate test cases in a test case set.

[0046] It should be noted that the test case evaluation method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the test case evaluation device provided in this disclosure embodiment can generally be located in server 105. The test case evaluation method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the test case evaluation device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] The following will be based on Figure 1 The described system architecture, through Figures 2-6 The test case evaluation method of the disclosed embodiments is described in detail.

[0049] Figure 2 A flowchart illustrating a test case evaluation method 200 according to an embodiment of the present disclosure is shown schematically.

[0050] like Figure 2 As shown, the test case evaluation method 200 of this embodiment includes operations S210 to S230.

[0051] In operation S210, the similarity value between any two test cases in the test case set is determined.

[0052] For example, in some enterprises, test case sets exist in the form of test case asset libraries. Test case sets may include, for example, existing test cases.

[0053] In operation S220, the test cases in the test case set are clustered according to the similarity value to obtain at least one test case cluster set.

[0054] In operation S230, the evaluation value of each test case in each test case cluster is determined according to the evaluation rules.

[0055] Evaluation rules characterize the rules for the reference priority of evaluation test cases.

[0056] According to the test case evaluation method of this disclosure, the test cases in the test case set are clustered based on the similarity value between any two test cases in the test case set to obtain a test case cluster set. Test cases belonging to the same test case cluster set have a higher degree of similarity. The similarity is reflected, for example, in corresponding to the same or similar test functions. This will enable testers to use the test cases in the test case cluster set as a reference when they need to find test cases in the test case set for current testing, thus efficiently providing testers with more sufficient reference resources.

[0057] According to the test case evaluation method of this disclosure, the evaluation rules characterize the rules for the reference priority of the evaluated test cases. Therefore, the evaluation value of the test cases determined by the evaluation rules can reflect the level of reference significance of the test cases. This allows for the accurate provision of the reference priority of each test case in the test case set to personnel such as testers, facilitating testers to differentiate and efficiently determine the test cases that are more relevant to the current testing task from the test case set.

[0058] Figure 3The flowchart illustrating the determination of the similarity value between any two test cases in a test case set according to another embodiment of the present disclosure is shown schematically.

[0059] Test cases include input parameter data and step description data, and similarity values ​​include first similarity value, second similarity value and target similarity value.

[0060] like Figure 3 As shown, for example, the following embodiment can be used to implement a specific example of determining the similarity value between any two test cases in the test case set in operation S310.

[0061] In operation S311, based on the input parameter data, the first similarity value between any two test cases is determined.

[0062] Test cases reflect the operational process. As we understand it, the operation of a software system involves input and output data. Input parameters can be understood as the parameters of the input data, such as constants or variables. These input parameters are related to the specific operation being performed. Based on the input parameter data, the first similarity score between test cases accurately reflects the degree of similarity between them.

[0063] In operation S312, based on the data described in the steps, determine the second similarity value between any two test cases.

[0064] Step description data can be understood as data that describes the steps of an operation. Step description data has a finer granularity. Therefore, the second similarity value between test cases determined based on the step description data can more accurately reflect the degree of similarity between test cases.

[0065] In operation S313, the target similarity value between any two test cases in the test case set is determined based on the first similarity value and the second similarity value.

[0066] Since a test case is a text describing a test operation process, it is closely related to the operation process. According to the test case evaluation method of this disclosure, given the close relationship between test cases and the software system's operation process, the operation process is comprehensively reflected based on input parameter data and step descriptions. Therefore, the target similarity value determined by combining these two dimensions (input parameter data and step description data) can accurately reflect the degree of similarity between test cases.

[0067] For example, operations S311 to S313 described above can be performed before operation S220.

[0068] Figure 4The flowchart illustrating the process of obtaining at least one cluster set of test cases according to another embodiment of the present disclosure is shown.

[0069] The test case cluster set includes a first cluster set and a second cluster set.

[0070] like Figure 4 As shown, for example, the following embodiment can be used to cluster test cases in a test case set according to similarity values ​​to obtain at least one test case cluster set.

[0071] In operation S421, two test cases with a first similarity value greater than or equal to a first threshold are used as elements of a first cluster set to obtain at least one first cluster set.

[0072] In operation S422, two test cases with a second similarity value greater than or equal to the second threshold are used as elements of the second cluster set to obtain at least one second cluster set.

[0073] For example, the first threshold and the second threshold can take values ​​in the range of 0.4 to 0.8.

[0074] The first similarity value represents the degree of similarity between test cases regarding input parameter data, and the second similarity value represents the degree of similarity between test cases regarding step description data. According to the test case evaluation method of this disclosure, the first cluster set and the second cluster set obtained by aggregating the first similarity value and the second similarity value are also related to the input parameter data and the step description data, respectively, which can realize fine-grained test case aggregation.

[0075] It should be noted that for any two test cases, the first similarity value and the second similarity value can be represented from the dimensions of the input parameter data and the step description data, respectively. Based on this, the first cluster set corresponding to the first similarity value and the second cluster set corresponding to the second similarity value can be obtained, respectively.

[0076] For example, operations S421 to S422 described above can be performed after operation S210.

[0077] Figure 5 A flowchart illustrating the determination of the evaluation value of a test case according to another embodiment of the present disclosure is shown.

[0078] The evaluation values ​​include basic evaluation values ​​and target evaluation values.

[0079] like Figure 5As shown, for example, the following embodiment can be used to implement a specific example of determining the evaluation value of each test case in each test case cluster set according to the evaluation rules.

[0080] In S531, initialize the base evaluation values ​​for each test case.

[0081] In operation S532, the base evaluation value is adjusted based on at least one of the first, second, and third dimensions to obtain the target evaluation value for each test case.

[0082] The first dimension represents whether the test case is executed by the execution script; the second dimension represents whether the execution script of the test case is executed successfully; and the third dimension represents whether the test case is referenced.

[0083] For example, the base evaluation value for each test case can be set to 5.

[0084] For example, for each test case, in the first dimension, if the test case is executed by the execution script, for example, 10 can be added to the base evaluation value. If the test case is not executed by the execution script, for example, 2 can be subtracted from the base evaluation value.

[0085] For example, for each test case, in the second dimension, the execution script of the test case can be incremented by 1 for each successful execution in the most recent 5 execution records, and can be decremented by 1 for each failed execution.

[0086] For example, for each test case, in the third dimension, if the test case is cited once in the most recent 5 historical citation records, the base evaluation value can be increased by 2, for example. If the test case is not cited, the base evaluation value is not adjusted.

[0087] Let's take determining the target evaluation value based on the first, second, and third dimensions as an example. For instance, a test case, Test-case1, has an execution script (which can be understood as automating the test case). In the most recent 5 execution records, it executed successfully twice and failed three times. This test case has been referenced in all 5 historical reference records. Therefore, the target evaluation value for Test-case1 can be determined to be 24. That is, target evaluation value 24 = 5 (base evaluation value) + 10 (first dimension) + (2-3) (second dimension) + 10 (third dimension).

[0088] The first dimension characterizes whether the test case is executed by an execution script. It can be understood that test cases executed by an execution script achieve automation. In practical application scenarios, when testers handle test tasks, the execution scripts corresponding to the automated test cases can be reused. Therefore, the test case evaluation method of this disclosure, by incorporating the first dimension into the factors determining the target evaluation value, can effectively improve the reusability of test cases and the execution scripts corresponding to the test cases.

[0089] The second dimension characterizes whether the execution script of the test case was executed successfully. It can be understood that whether the execution script was executed successfully reflects its effectiveness. The test case evaluation method of this disclosure improves the reuse efficiency of the execution script corresponding to the test case by incorporating the second dimension into the factors determining the target evaluation value.

[0090] The third dimension characterizes whether a test case is referenced. Test case citation reflects characteristics such as the importance of the test case. For example, test cases corresponding to core functions need to be referenced (tested) in every version test, or test cases corresponding to basic functions are frequently referenced in every version test. The test case evaluation method of this disclosure incorporates the third dimension into the determination factors of the target evaluation value, and the determined target evaluation value can accurately reflect the reference priority of the corresponding test cases.

[0091] According to the test case evaluation method of this disclosure, the target evaluation value of test cases, determined comprehensively using at least one of the first, second, and third dimensions, can more accurately and comprehensively assess the reference priority order of test cases. The target evaluation value can serve as a reference for testers. Testers can, for example, arrange test tasks based on the target evaluation value.

[0092] For example, operations S531 to S532 can be performed after operation S220.

[0093] Figure 6 A flowchart of a test case evaluation method 600 according to yet another embodiment of the present disclosure is illustrated.

[0094] like Figure 6 As shown, the test case evaluation method 600 according to another embodiment of this disclosure may include, for example, operations S640 to S660.

[0095] In operation S640, for any first cluster set, the reference priority ratio of the first cluster set is determined based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the first cluster set.

[0096] Automated test cases are the test cases executed by the script in the first dimension.

[0097] For example, a certain first cluster set Set1 includes 5 test cases T-case1 to T-case5. The first dimension of T-case1 to T-case3 corresponds to the execution by the script (i.e., T-case1 to T-case3 are automated test cases). The target evaluation values ​​of T-case1 to T-case5 are 12, 21, 14, 3, and 3 respectively (the target evaluation value of T-case4 and T-case5 is 3 = the base evaluation value 5-2). Therefore, the reference priority ratio of the first cluster set Set1 can be determined to be 0.8868 (rounded value).

[0098] In operation S650, for any second cluster set, the reference priority ratio of the second cluster set is determined based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the second cluster set.

[0099] In operation S660, the target reference priority ratio is determined based on the reference priority ratio of the first cluster set and the reference priority ratio of the second cluster set.

[0100] The reference priority ratio of the first cluster set reflects the overall reference priority order of the test cases in the first cluster set, and the reference priority ratio of the second cluster set reflects the overall reference priority order of the test cases in the second cluster set. According to the test case evaluation method of this disclosure, by determining a target reference priority ratio based on the reference priority ratios of the first and second cluster sets, the reference priority order of the test cases in the test case set can be characterized from the dimension of the test case cluster sets.

[0101] For example, operations S640 to S660 described above can be performed after operation S230.

[0102] Figure 6 A flowchart of a test case evaluation method according to yet another embodiment of the present disclosure is also schematically shown.

[0103] like Figure 6 As shown, the test case evaluation method according to another embodiment of this disclosure may further include at least one of the following: operation S670 to operation S680.

[0104] In operation S670, the first cluster set and the second cluster set are sorted according to the target reference priority ratio to obtain the sorted first cluster set and the second cluster set.

[0105] In operation S680, the test cases of the first cluster set and the second cluster set are sorted according to the target similarity value to obtain the sorted test cases.

[0106] According to the test case evaluation method of this disclosure, by sorting the test cases of the first cluster set and the second cluster set according to the target reference priority ratio or according to the target similarity value, more meaningful test cases or test case cluster sets can be placed first, which can provide intuitive reference for testers.

[0107] For example, the first cluster set and the second cluster set can be sorted according to the target reference priority ratio to obtain the sorted first cluster set and the second cluster set. Then, the test cases of each first cluster set and each second cluster set can be sorted according to the target similarity value to obtain the sorted test cases in each first cluster set and the sorted test cases in each second cluster set.

[0108] For example, operations S670 to S680 can be performed after operation S230.

[0109] Figure 6 A flowchart of a test case evaluation method according to yet another embodiment of the present disclosure is also schematically shown.

[0110] like Figure 6 As shown, the test case evaluation method according to another embodiment of this disclosure may include, for example, the following operations: operations S690 to S611.

[0111] When operating S690, reference test cases are determined from the test case set based on the evaluation values ​​to implement automated test cases.

[0112] When operating S611, based on the execution script corresponding to the reference test case, determine at least one of the following: execution script reference data, execution script execution time reference data.

[0113] For example, the average execution time of the execution scripts of the three test cases with the highest evaluation values ​​in the test case cluster to which the reference test case belongs can be used as the reference data for the execution time of the execution script.

[0114] According to the test case evaluation method of this disclosure, reference test cases for automated test cases can be determined from the test case set based on the evaluation value. Furthermore, reference data for the execution script and reference data for the execution time of the execution script can be determined based on the execution script corresponding to the reference test case. These data are used by testers to refer to and write the execution script for the current test task, thereby automating the test cases for the current test task and rationally planning the test task.

[0115] In summary, the test case evaluation method according to the embodiments of this disclosure can comprehensively consider factors such as test case similarity and test case citation, and provide relevant analysis information to testers for reference when dynamically adjusting the arrangement of automated test engineering plans. At the same time, during compilation and execution, it can intelligently analyze the automated script code assets corresponding to existing test cases, and form an efficient asset reuse mechanism through intelligent evaluation, thereby improving the efficiency of automated engineering implementation.

[0116] Specifically, by analyzing the similarity of the test cases themselves in the test case set, and combining the test case references of recent versions, the system can intelligently evaluate the list of test cases to be automated, forming an ordered list and an estimated completion time for each test case, which can be used as a reference for automation testers when scheduling implementation plans.

[0117] By analyzing recent script execution results and frequency, testers are spared the manual work of screening, reviewing, and comparing when developing scripts for test cases. Intelligent evaluation results generate highly reusable script code modules and automated script draft templates for automation testers to modify. Test cases in clusters have higher similarity, thus representing more reusable assets. Automated test cases with higher similarity execute more frequently and have a higher success rate, indicating higher cost-effectiveness. Prioritizing the automation of these test cases significantly improves script development efficiency and saves valuable human and time resources.

[0118] Based on the above-described test case evaluation method, this disclosure also provides a test case evaluation device. The following will be combined with... Figure 7 The device is described in detail.

[0119] Figure 7 A schematic block diagram of a test case evaluation apparatus according to an embodiment of the present disclosure is shown.

[0120] like Figure 7 As shown, the test case evaluation device 700 of this embodiment includes a similarity value determination module 710, a clustering module 720, and an evaluation value determination module 730.

[0121] The similarity value determination module 710 is used to determine the similarity value between any two test cases in the test case set.

[0122] Clustering module 720 is used to cluster test cases in the test case set according to similarity values ​​to obtain at least one test case cluster set.

[0123] The evaluation value determination module 730 is used to determine the evaluation value of each test case in each test case cluster set according to the evaluation rules.

[0124] Evaluation rules characterize the rules for the reference priority of evaluation test cases.

[0125] According to embodiments of this disclosure, test cases include input parameter data and step description data, and similarity values ​​include a first similarity value, a second similarity value, and a target similarity value. The similarity value determination module includes:

[0126] The first similarity value determination submodule is used to determine the first similarity value between any two test cases based on the input parameter data.

[0127] The second similarity value determination submodule is used to determine the second similarity value between any two test cases based on the step description data.

[0128] The target similarity value determination submodule is used to determine the target similarity value between any two test cases in the test case set based on the first similarity value and the second similarity value.

[0129] According to embodiments of this disclosure, the test case cluster set includes a first cluster set and a second cluster set. The clustering module includes: a first cluster set determination submodule and a second cluster set determination submodule.

[0130] The first cluster set determination submodule is used to take two test cases with a first similarity value greater than or equal to a first threshold as elements of the first cluster set to obtain at least one first cluster set.

[0131] The second cluster set determination submodule is used to take two test cases with a second similarity value greater than or equal to a second threshold as elements of the second cluster set to obtain at least one second cluster set.

[0132] According to embodiments of this disclosure, the evaluation value includes a basic evaluation value and a target evaluation value. The evaluation value determination module includes a basic evaluation value determination submodule and a target evaluation value determination submodule.

[0133] The Base Evaluation Value Determination Submodule is used to initialize the base evaluation value for each test case.

[0134] The target evaluation value determination submodule is used to adjust the base evaluation value based on at least one of the first, second, and third dimensions to obtain the target evaluation value for each test case.

[0135] The first dimension represents whether the test case is executed by the execution script; the second dimension represents whether the execution script of the test case is executed successfully; and the third dimension represents whether the test case is referenced.

[0136] The test case evaluation apparatus according to embodiments of the present disclosure further includes: a first cluster set reference priority ratio determination module, a second cluster set reference priority ratio determination module, and a target reference priority ratio determination module.

[0137] The module for determining the reference priority ratio of the first cluster set is used to determine the reference priority ratio of the first cluster set for any given first cluster set based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the first cluster set. The automated test cases are the test cases executed by the execution script corresponding to the first dimension.

[0138] The second cluster set reference priority ratio determination module is used to determine the second cluster set reference priority ratio for any given second cluster set based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the second cluster set.

[0139] The target reference priority ratio determination module is used to determine the target reference priority ratio based on the reference priority ratio of the first cluster set and the reference priority ratio of the second cluster set.

[0140] The test case evaluation apparatus according to embodiments of the present disclosure further includes at least one of the following: a first sorting module or a second sorting module.

[0141] The first sorting module is used to sort the first cluster set and the second cluster set according to the target reference priority ratio, so as to obtain the sorted first cluster set and the second cluster set.

[0142] The second sorting module is used to sort the test cases of the first cluster set and the second cluster set according to the target similarity value, so as to obtain the sorted test cases.

[0143] The test case evaluation apparatus according to embodiments of the present disclosure further includes a reference test case determination module and a reference data determination module.

[0144] The reference test case determination module is used to determine reference test cases from the test case set based on the evaluation values.

[0145] The reference data determination module is used to determine at least one of the following based on the execution script corresponding to the reference test case: execution script reference data and execution script execution time reference data.

[0146] According to embodiments of this disclosure, any plurality of modules among the similarity value determination module 710, clustering module 720, and evaluation value determination module 730 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the similarity value determination module 710, clustering module 720, and evaluation value determination module 730 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the similarity value determination module 710, clustering module 720, and evaluation value determination module 730 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0147] It should be understood that the embodiments of the apparatus portion of this disclosure correspond to the same or similar embodiments of the method portion of this disclosure, and the technical problems solved and the technical effects achieved are also the same or similar. This disclosure will not repeat them here.

[0148] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a test case evaluation method according to an embodiment of the present disclosure.

[0149] like Figure 8As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0150] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0151] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0152] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0153] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0154] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0155] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0156] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0157] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0158] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0161] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A test case evaluation method, comprising: Determine the similarity value between any two test cases in the test case set; Based on the similarity values, the test cases in the test case set are clustered to obtain at least one test case cluster set; as well as According to the evaluation rules, the evaluation value of each test case in each test case cluster set is determined, wherein the evaluation rules characterize the rules for evaluating the reference priority order of the test cases; The evaluation value includes a basic evaluation value and a target evaluation value; determining the evaluation value of each test case in each test case cluster according to the evaluation rules includes: Initialize the base evaluation value for each of the test cases; and The base evaluation value is adjusted based on at least one of the first dimension, the second dimension, and the third dimension to obtain the target evaluation value for each test case, wherein the first dimension indicates whether the test case is executed by the execution script, the second dimension indicates whether the execution script of the test case is executed successfully, and the third dimension indicates whether the test case is referenced. The test cases include input parameter data and step description data; the similarity values ​​include a first similarity value, a second similarity value, and a target similarity value; determining the similarity value between any two test cases in the test case set includes: Based on the input parameter data, determine the first similarity value between any two test cases; Based on the data described in the steps, determine the second similarity value between any two test cases; and Based on the first similarity value and the second similarity value, the target similarity value between any two test cases in the test case set is determined; The test case cluster set includes a first cluster set and a second cluster set; the step of clustering the test cases in the test case set according to the similarity value to obtain at least one test case cluster set includes: Two test cases with a similarity value greater than or equal to a first threshold are used as elements of a first cluster set to obtain at least one first cluster set; and Two test cases with a second similarity value greater than or equal to the second threshold are used as elements of the second cluster set to obtain at least one second cluster set; For any given first cluster set, a reference priority ratio for the first cluster set is determined based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the first cluster set, wherein the automated test cases are the test cases executed by the execution script corresponding to the first dimension; For any given second cluster set, a reference priority ratio for the second cluster set is determined based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the second cluster set; and The target reference priority ratio is determined based on the reference priority ratio of the first cluster set and the reference priority ratio of the second cluster set.

2. The method of claim 1, further comprising at least one of the following: Based on the target reference priority ratio, the first cluster set and the second cluster set are sorted to obtain the sorted first cluster set and the second cluster set; or Based on the target similarity value, the test cases in the first cluster set and the second cluster set are sorted to obtain the sorted test cases.

3. The method according to any one of claims 1-2, further comprising: Based on the evaluation value, reference test cases for the automated test cases to be implemented are determined from the test case set; as well as Based on the execution script corresponding to the reference test case, determine at least one of the following: execution script reference data, execution script execution time reference data.

4. A test case evaluation device, comprising: The similarity value determination module is used to determine the similarity value between any two test cases in the test case set; The clustering module is used to cluster the test cases in the test case set according to the similarity value to obtain at least one test case cluster set. as well as An evaluation value determination module is used to determine the evaluation value of each test case in each test case cluster set according to evaluation rules, wherein the evaluation rules characterize the rules for evaluating the reference priority order of the test cases; The evaluation value includes a basic evaluation value and a target evaluation value; determining the evaluation value of each test case in each test case cluster according to the evaluation rules includes: Initialize the base evaluation value for each of the test cases; and The base evaluation value is adjusted based on at least one of the first dimension, the second dimension, and the third dimension to obtain the target evaluation value for each test case, wherein the first dimension indicates whether the test case is executed by the execution script, the second dimension indicates whether the execution script of the test case is executed successfully, and the third dimension indicates whether the test case is referenced. The test cases include input parameter data and step description data; the similarity values ​​include a first similarity value, a second similarity value, and a target similarity value; determining the similarity value between any two test cases in the test case set includes: Based on the input parameter data, determine the first similarity value between any two test cases; Based on the data described in the steps, determine the second similarity value between any two test cases; and Based on the first similarity value and the second similarity value, the target similarity value between any two test cases in the test case set is determined; The test case cluster set includes a first cluster set and a second cluster set; the step of clustering the test cases in the test case set according to the similarity value to obtain at least one test case cluster set includes: Two test cases with a similarity value greater than or equal to a first threshold are used as elements of a first cluster set to obtain at least one first cluster set; and Two test cases with a second similarity value greater than or equal to the second threshold are used as elements of the second cluster set to obtain at least one second cluster set; For any given first cluster set, a reference priority ratio for the first cluster set is determined based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the first cluster set, wherein the automated test cases are the test cases executed by the execution script corresponding to the first dimension; For any given second cluster set, a reference priority ratio for the second cluster set is determined based on the ratio between the sum of the target evaluation values ​​of the automated test cases and the sum of the target evaluation values ​​of all test cases in the second cluster set; and The target reference priority ratio is determined based on the reference priority ratio of the first cluster set and the reference priority ratio of the second cluster set.

5. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 3.

7. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 3.

Images