Test case evaluation method and device, electronic equipment and storage medium

Abstract

The invention discloses a test case evaluation method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a test case measurement index according to a business demand, the test case measurement index being obtained based on a preset problem, and the preset problem being obtained based on an evaluation target of a test case; constructing a test case hierarchical structure according to the test case measurement index; and calculating an index weight in the hierarchical structure of the test case to obtain an evaluation result of the software test case. According to the invention, the evaluation process of the test case is optimized.

Description

Technical Field

[0001] The present application relates to the technical field of software testing and test cases, and in particular to a test case evaluation method, device, electronic device, and storage medium. Background Art

[0002] A test case is the basic unit of software testing. It is a set of inputs, expected outputs, and execution conditions used to verify a specific function or feature of the software.

[0003] The quality of test cases directly determines the effectiveness of software testing. Therefore, how to effectively evaluate the quality of test cases is an issue to be resolved in the field of software testing. Summary of the invention

[0004] The embodiments of the present application provide a test case evaluation method, device, electronic device, and storage medium to optimize the test case evaluation process.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a test case evaluation method, wherein the method comprises:

[0007] Determine test case metrics based on business needs, where the test case metrics are obtained based on preset questions, which are obtained based on evaluation objectives of the test case;

[0008] constructing a test case hierarchy according to the test case metrics; and

[0009] Calculate the indicator weights in the test case hierarchy and obtain the evaluation results of the software test cases.

[0010] In some embodiments, constructing a test case hierarchy according to the test case metrics includes:

[0011] Constructing a first-level metric element set according to the first-level metric elements in the test case metric indicators;

[0012] Decomposing the first-level metric element set according to the second-level metric elements in the test case metric indicators to obtain a second-level metric element set;

[0013] in,

[0014] There is a many-to-one mapping relationship between the secondary metric element set and the primary metric element set.

[0015] In some embodiments, the step of calculating the indicator weights in the test case hierarchy includes:

[0016] Constructing a judgment matrix corresponding to the secondary metric element set;

[0017] A matrix consistency check of the secondary metric set is performed according to the judgment matrix, and the overall indicator weight is calculated.

[0018] In some embodiments, the test case metric is determined according to the business requirements, and the test case metric is obtained based on a preset question, and the preset question is obtained based on an evaluation target of the test case, including:

[0019] Obtain the evaluation objectives of the test cases based on the business requirements of the software test cases;

[0020] Based on the evaluation objective of the test case, obtaining the preset question;

[0021] According to the preset questions, determine any one or more test case first-level metric indicators among test case design, functional test case coverage, fault type injection, fault type injection, performance test case, and automation test case,

[0022] in,

[0023] The test case design includes secondary metrics of one or more dimensions of defect discovery rate, test case accuracy, and test case scale;

[0024] The functional use case coverage includes secondary metric elements of one or more dimensions of functional requirement coverage and business process coverage;

[0025] The fault type injection is used to evaluate whether the test case fully considers potential risks and abnormal situations;

[0026] The performance test cases are used to ensure that the test cases can fully consider the performance requirements of the financial software system;

[0027] The automation use case includes secondary metrics of one or more dimensions: automation coverage, automation execution pass rate, and automation error detection rate.

[0028] In some embodiments, the determining, according to the preset questions, any one or more test case metrics of test case design, functional test case coverage, fault type injection, fault type injection, performance test case, and automation test case includes:

[0029] Any one or more answers of the test case design, functional case coverage, fault type injection, fault type injection, performance test case, and automation case in each of the preset questions are used as the first-level metric element;

[0030] The first-level metric element is decomposed into second-level metric elements according to business requirements.

[0031] In some embodiments, obtaining the evaluation target of the test case according to the business requirements of the software test case includes:

[0032] Evaluate the quality of test cases as business requirements and determine the goals of test case evaluation;

[0033] The evaluation objective based on the test case is used to obtain the preset question, including:

[0034] The preset questions are obtained by characterizing the objectives of the test case evaluation to determine whether the objectives have been achieved or to evaluate the achievement of the objectives.

[0035] In some embodiments, the method further comprises:

[0036] Collect data of multiple secondary metrics in different business projects and obtain corresponding probability values ​​in each secondary metric.

[0037] In a second aspect, an embodiment of the present application further provides a test case evaluation device, wherein the device comprises:

[0038] A determination module, used to determine test case metrics according to business requirements, wherein the test case metrics are obtained based on preset questions, and the preset questions are obtained based on evaluation objectives of the test case;

[0039] A construction module, used to construct a test case hierarchy according to the test case metrics; and

[0040] The calculation module is used to calculate the indicator weights in the test case hierarchy and obtain the evaluation results of the software test cases.

[0041] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0043] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: first, determine the test case measurement indicators according to business needs; then construct a test case hierarchy according to the test case measurement indicators; finally, calculate the indicator weights in the test case hierarchy to obtain the evaluation results of the software test case. The test case measurement indicators determined by the above method can comprehensively evaluate the quality of the test case from multiple levels, and at the same time ensure that the test case can cover the key functions and risks in financial software testing. Through the above method, according to the test case measurement indicators, a test case hierarchy is constructed to determine the relative importance of different evaluation indicators, and by establishing a judgment matrix and performing consistency checks, the objectivity and scientificity of the evaluation are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 A flow chart of a test case evaluation method in an embodiment of the present application;

[0046] Figure 2 This is a schematic diagram of the structure of the test case evaluation device in the embodiment of the present application;

[0047] Figure 3 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0049] The software test case evaluation method in the related art mainly judges the quality of the test case based on the execution results of the test case. For example, the effectiveness, completeness, and sufficiency of the test case are evaluated through indicators such as the test case pass rate, defect detection rate, and coverage rate. However, this method has a single evaluation, and there are deviations in the results of case writing and execution by different people. Moreover, the evaluation can often only be performed after the test case is executed, which has problems such as lag.

[0050] In summary, the main shortcomings of the related technologies include:

[0051] (1) The writing quality and execution results of test cases are unstable. The writing and execution of test cases are affected by factors such as the test environment, test data, and testers. There may be certain deviations and instabilities, which cannot fully and objectively reflect the quality of the test cases.

[0052] (2) The lag in test case evaluation. The test case execution results can only be obtained after the test execution is completed. The quality of the test case cannot be predicted and optimized in the test design phase, and the deficiencies and problems of the test case cannot be discovered and improved in a timely manner.

[0053] (3) Single test case evaluation method. The test case execution results can only be used to evaluate the quality of the test case from a single perspective and dimension. It cannot comprehensively consider multiple quality attributes and indicators of the test case, cannot balance the quality and quantity of the test case, and cannot meet the diversified and personalized needs of financial software testing.

[0054] In view of the above-mentioned deficiencies, a test case evaluation method is provided in an embodiment of the present application, which can be used as an evaluation indicator for financial software test cases and can be verified for feasibility in engineering practice.

[0055] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0056] The present application embodiment provides a test case evaluation method, such as Figure 1 As shown, a flow chart of a test case evaluation method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S130:

[0057] Step S110, determining a test case metric based on business requirements, wherein the test case metric is obtained based on preset questions, and the preset questions are obtained based on an evaluation objective of the test case.

[0058] The test case is explained in detail using the financial industry software test case as an example.

[0059] Test case metrics can be constructed based on the GQM method from the perspective of goal-problem-metric and the perspective of test quality managers in the financial industry. The GQM method is a goal-problem-metric software quality management method proposed by the National Institute of Standards and Technology (NIST) of the United States. It aims to help software organizations and projects determine and achieve software quality goals, solve software quality problems, and measure software quality results. The core idea of ​​the GQM method is to form a hierarchical software quality model by decomposing software quality goals into specific problems, and then decomposing problems into measurable indicators. Then, by collecting and analyzing software quality data, the status and trend of software quality can be evaluated and improved. The advantages of the GQM method are that it can define and measure software quality from multiple perspectives and dimensions, can customize and adjust the software quality model according to different software environments and requirements, and can provide a systematic and scientific framework for software quality management.

[0060] Based on the GQM method, when applied to software test cases in the financial industry, the writing quality of test cases and the stability of execution results can be guaranteed. At the same time, the quality of test cases can be predicted and optimized in the test design stage.

[0061] Step S120: construct a test case hierarchy according to the test case measurement indicators.

[0062] By adopting the test case metric, multiple quality attributes and indicators in the test case are comprehensively considered, and the quality and quantity of the test case are balanced, so as to meet the diversified and personalized needs of financial software testing.

[0063] Step S130, calculating the indicator weights in the test case hierarchy to obtain the evaluation results of the software test cases.

[0064] By calculating the weights of indicators in the test case hierarchy, objective evaluation results of software test cases can be obtained. The analytic hierarchy process (AHP) can be used to determine the relative importance of different evaluation indicators in combination with the evaluation model of the AHP method, and the objectivity and scientificity of the evaluation can be ensured by establishing a judgment matrix and performing consistency tests. This combined method makes the evaluation indicator system more accurate and practical.

[0065] It can be understood that the AHP evaluation method is an evaluation method of the hierarchical analysis method. It was proposed by Thomas Saaty, an American operations researcher, to help decision makers deal with complex multi-criteria decision-making problems. The core idea of ​​the AHP evaluation method is to decompose the decision problem into different levels such as goals, criteria and plans, and then construct a judgment matrix, use expert scoring, calculate the weights of each level, and perform consistency tests to obtain the optimal solution or optimal ranking of the decision problem. The advantages of the AHP evaluation method are that it can make full use of the knowledge and experience of experts, can process qualitative and quantitative data, can consider the relative importance of each criterion, and can provide a simple and effective decision evaluation method.

[0066] Through the above method, the test case metrics determined according to business needs can comprehensively evaluate the quality of test cases from multiple levels, and at the same time ensure that the test cases can cover the key functions and risks in financial software testing.

[0067] Through the above method, according to the test case measurement indicators, a test case hierarchy is constructed to determine the relative importance of different evaluation indicators, and the objectivity and scientificity of the evaluation are ensured by establishing a judgment matrix and performing consistency checks.

[0068] Different from the problems of unstable test case writing quality and execution results in related technologies, the above method determines the test case metric indicators according to business needs. The test case metric indicators are obtained based on preset questions, which are obtained based on the evaluation objectives of the test case, so as to comprehensively evaluate the quality of the test case from multiple levels and provide a more systematic and comprehensive quality assessment. It is more suitable for software test cases in the financial industry.

[0069] Different from the problem of lag in test case evaluation in related technologies, in the above method, a test case hierarchy is constructed according to the test case metric indicators. It adopts, but is not limited to, defect discovery rate, functional requirement coverage, fault injection case pass rate, etc., to ensure that the test case can cover the key functions and risks in financial software testing. It is more suitable for financial industry software test cases.

[0070] Different from the problem of single test case evaluation method in related technologies, the above method calculates the indicator weights in the test case hierarchy to obtain the evaluation results of the software test case. The hierarchical analysis method is used to determine the relative importance of different evaluation indicators, and the objectivity and scientificity of the evaluation are ensured by establishing a judgment matrix and performing consistency tests. This combined method makes the evaluation indicator system more accurate and practical. It is more suitable for software test cases in the financial industry.

[0071] The method in the embodiment of the present application first analyzes the goals and problems of test case evaluation, then defines the measurement indicators of test case evaluation (based on the GQM method), and finally constructs the hierarchical structure of the test case evaluation indicator system, and gives the final evaluation result in practice (AHP method). It can be understood that the above-mentioned GQM method and AHP method are only a feasible embodiment of the test case evaluation method in the embodiment of the present application, and are not used to limit the protection scope of the application embodiment.

[0072] In one embodiment of the present application, constructing a test case hierarchy according to the test case metric indicators includes: constructing a first-level metric element set according to the first-level metric elements in the test case metric indicators; decomposing the first-level metric element set according to the second-level metric elements in the test case metric indicators to obtain a second-level metric element set; wherein there is a many-to-one mapping relationship between the second-level metric element set and the first-level metric element set.

[0073] The first-level metric element set includes test case design, functional case coverage, fault type injection, performance test cases and automation cases. The second-level metric element decomposes the factors of the first layer and generates a corresponding second-level metric element set. The following is a first-level metric element set:

[0074]

[0075] The set of first-level metric elements can be taken as matrix A1.

[0076] By decomposing the first-level metric elements separately, the following second-level metric element sets can be generated:

[0077]

[0078] A4 = {fault injection use case pass rate},

[0079]

[0080] The secondary metric element set can be provided as multiple matrices A2, A3, A4, A5, A6.

[0081] It can be understood that the above-mentioned first-level measurement elements and second-level measurement elements are only examples and are not used to limit the protection scope in the embodiments of the present application. At the same time, the second-level measurement element set is obtained by decomposing the first-level measurement element set by the second-level measurement element.

[0082] In one embodiment of the present application, the calculation of indicator weights in the test case hierarchy includes: constructing a judgment matrix corresponding to the secondary metric element set; performing a matrix consistency check of the secondary metric element set according to the judgment matrix, and calculating the overall indicator weight.

[0083] Construct a judgment matrix, and use the weighted expert scoring method to construct a judgment matrix combining different metrics to ensure that expert opinions are taken into account. Among them, Table 3 is the judgment matrix of the test case design of the secondary metric.

[0084] Table 3 Judgment matrix for test case design

[0085] Test case design Defect Detection Rate Test case accuracy Test case size Defect Detection Rate 1 3 7 Test case accuracy 1 / 3 1 3 Test case size 1 / 7 1 / 3 1

[0086] Secondly, it is also necessary to perform a matrix consistency check on the judgment matrices of different metrics. If they fail, the weighted expert scores are modified. For example, the judgment matrix designed based on the secondary metric test case can be obtained to have a eigenvector of W = [0.67, 0.24, 0.09] T, and CR = 0.00675 < 0.1, which meets the matrix consistency requirements.

[0087] Finally, the overall indicator weight is calculated, and the total weight of the test case evaluation system is calculated by multiplying the weights of the first-level metric and the second-level metric, and then normalized.

[0088] Through the above method, GQM and AHP methods are combined to construct a comprehensive evaluation index system, covering multiple aspects of software testing quality, making the evaluation more comprehensive. When evaluating test cases, scientific methodology is adopted to ensure the scientificity and accuracy of the evaluation process, thereby improving the credibility and reliability of the evaluation results. This index system is designed for financial software testing scenarios, is practical and applicable, and can effectively solve the challenges of software testing in the financial industry and improve testing efficiency and quality.

[0089] Table 4 Weights of analytic hierarchy process

[0090]

[0091]

[0092] Specifically, through the hierarchical analysis method, the initial weights of its 15 indicators are calculated as follows: (0.2933, 0.1063, 0.0385, 0.1738, 0.0616, 0.0327, 0.0077, 0.0726, 0.0417, 0.0248, 0.0131, 0.0071, 0.0254, 0.014, 0.0077).

[0093] Collect secondary metric metadata and collect data of each secondary metric in two actual business projects A and B:

[0094] Project A Project B Defect Detection Rate 95.00％ 98.00％ Test case accuracy 88.60％ 90.50％ Test case size 93.37％ 98.41％ Functional requirement coverage 99.60％ 99.10％ Business process coverage 39.31％ 66.80％ Abnormal situation coverage 15.00％ 55.60％ Fault injection test case pass rate 10.00％ 66.30％ TPS compliance rate 100.00％ 98.32％ Response delay compliance rate 100.00％ 100.00％ CPU compliance rate 80.40％ 87.00％ Memory compliance rate 87.00％ 80.50％ Disk I / O compliance rate 86.30％ 85.00％ Automation coverage 98.35％ 84.00％ Automation error detection rate 87.32％ 55.30％ Automation script execution pass rate 87.32％ 75.40％

[0095] The final evaluation results are obtained by matrix multiplication of the actual data and the overall indicator weights, and the test case scoring sets of the two projects are:

[0096]

[0097] It can be seen from the above evaluation results that, through the method of the present invention, the comprehensive scores of different projects can clearly reflect the quality of test cases in practical applications. The comprehensive score of Project A is 80.73, and the comprehensive score of Project B is 84.42. According to the comprehensive score, Project B performs better than Project A in various test indicators, especially in terms of business process coverage and fault injection case pass rate.

[0098] At the same time, through the horizontal comparison with Project A, it can be seen that Project B still needs to be strengthened in automation-related indicators. Specifically, Project B's performance in automation coverage, automation error detection rate, and automation script execution pass rate is relatively weak, suggesting that further optimization and improvement can be made.

[0099] These evaluation results objectively show that the method of the present invention can effectively improve the quality and coverage of test cases, especially has significant advantages in key business processes and fault handling, but further efforts and improvements are still needed in automated testing.

[0100] In one embodiment of the present application, the test case metrics are determined according to business needs, and the test case metrics are obtained based on preset questions, and the preset questions are obtained based on the evaluation objectives of the test cases, including: obtaining the evaluation objectives of the test cases according to the business needs of the software test cases; obtaining the preset questions based on the evaluation objectives of the test cases; determining the first-level metrics of any one or more test cases in test case design, functional case coverage, fault type injection, fault type injection, performance test cases, and automation cases according to the preset questions, wherein the test case design includes secondary metrics of one or more dimensions of defect discovery rate, test case accuracy, and test case scale; the functional case coverage includes secondary metrics of one or more dimensions of functional requirement coverage and business process coverage; the fault type injection is used to evaluate whether the test cases have fully considered potential risks and abnormal situations; the performance test cases are used to ensure that the test cases can fully consider the performance requirements of the financial software system; the automation cases include secondary metrics of one or more dimensions of automation coverage, automation execution pass rate, and automation error detection rate.

[0101] In order to improve the quality of financial software test cases and ensure the stable, safe and efficient operation of the system.

[0102] Test case design, the test case is the product formed in the test design stage under the condition of covering the test requirements. It is the reference and basis for test execution. The number and quality of test cases directly affect the test coverage and have a significant impact on the quality of software testing. Therefore, test case design is selected as the first-level metric of test effectiveness.

[0103] Test case design can be decomposed into three dimensions of secondary metrics, including defect discovery rate, test case accuracy, and test case scale. The calculation method is as follows:

[0104]

[0105] Table 2 Software Scale Adjustment Factors

[0106] Lines of code Scale adjustment coefficient value 0-1000 0.79 1000-3000 0.85 3000-6000 0.90 6000-10000 0.94 10000-30000 0.97 30000-50000 0.98 50000-100000 0.99 More than 100,000 1.0

[0107] Functional use case coverage, this level of metric is designed to assess the extent to which test cases cover the key functions of financial software business. In the financial industry, it is crucial to ensure that test cases can fully and deeply cover key business functions. By measuring functional use case coverage, test quality managers can understand the test team's investment and coverage in business functions, thereby ensuring that the testing work is of practical significance to the quality improvement of financial software. Functional use case coverage can be decomposed into three dimensions of secondary metrics, including functional requirement coverage and business process coverage. The calculation method is as follows:

[0108]

[0109]

[0110] Fault type injection, as a first-level metric for test case evaluation, aims to evaluate whether the test case has fully considered potential risks and abnormal situations by simulating and injecting various types of faults. In the financial industry, system stability and security are crucial to ensure the efficient operation of financial software. By injecting fault types, test quality managers can more intuitively evaluate the coverage of test cases for different types of faults that the system may encounter, thereby ensuring that the software can respond robustly to various risks and abnormal situations.

[0111] Fault type injection can be decomposed into a secondary metric of one dimension, which is the pass rate of the fault injection case. The calculation method is as follows:

[0112]

[0113] Performance test cases, as the first-level metric for test case evaluation, are designed to ensure that test cases can fully consider the performance requirements of financial software systems. In the financial industry, the efficient operation of the system is crucial to ensure rapid and accurate transactions. Through the evaluation of performance test cases, test quality managers can evaluate the coverage of the test team when simulating high-load, high-concurrency, long-term operation and other scenarios, thereby providing strong support for the stability and efficiency of the system.

[0114] Performance test cases can be decomposed into 5 dimensions of secondary metrics, including TPS compliance rate and response delay compliance rate. If the calculated result is greater than 100%, it is taken as 100%. If the CPU, memory compliance rate and disk I / O results exceed the standard value, they are recorded as 0. The calculation method is as follows:

[0115]

[0116]

[0117] Automated use cases, as the first-level metric for test case evaluation, are designed to evaluate the degree of automation of test cases to ensure that financial software testing can be better supported in terms of stability, security, and efficiency. In the financial industry, the complexity of systems and the frequency of updates are high, so the use of automated testing is crucial to improving testing efficiency and ensuring software quality. Through the evaluation of automated use cases, test quality managers can fully understand the investment and effects of the test team in automation, providing strong support for improving the quality and execution efficiency of test cases.

[0118] Automation use cases can be broken down into three dimensions of secondary metrics, including automation coverage, automation execution pass rate, and automation error detection rate. The calculation method is as follows:

[0119]

[0120] Structure and hierarchical design of the indicator system. In the embodiments of the present application, a hierarchical test case evaluation system is proposed, which comprehensively evaluates the quality of test cases from multiple levels based on the goal-question-metric (GQM) principle. This structured evaluation method can provide a more systematic and comprehensive quality assessment.

[0121] Application of specific metrics. In the embodiments of the present application, multiple metrics are used, such as defect discovery rate, functional requirement coverage, fault injection case pass rate, etc., to ensure that the test cases can cover the key functions and risks in financial software testing.

[0122] Evaluation model combined with AHP method. In the embodiments of the present application, the analytic hierarchy process (AHP) is used to determine the relative importance of different evaluation indicators, and the objectivity and scientificity of the evaluation are ensured by establishing a judgment matrix and performing consistency tests. This combined method makes the evaluation indicator system more accurate and practical.

[0123] In one embodiment of the present application, the test case design, functional case coverage, fault type injection, fault type injection, performance test case, and automation case are determined according to the preset questions, including: taking any one or more answers of the test case design, functional case coverage, fault type injection, fault type injection, performance test case, and automation case in each of the preset questions as first-level metric elements; and decomposing the first-level metric elements into second-level metric elements according to business needs.

[0124] Table 1

[0125]

[0126] The answer to each question is used as a direct measure and the direct measures are further broken down into secondary measures, which can be subjective or objective, as needed.

[0127] In one embodiment of the present application, obtaining the evaluation objectives of the test cases according to the business requirements of the software test cases includes: evaluating the quality of the test cases as business requirements and determining the objectives of the test case evaluation; obtaining the preset questions based on the evaluation objectives of the test cases includes: obtaining the preset questions by characterizing the objectives of the test case evaluation to determine whether the objectives have been achieved or to evaluate the achievement of the objectives.

[0128] Ask questions about the goal. After determining the measurement goal, a set of questions should be used to further describe the method of achieving the specific goal in the upper layer. These questions can characterize the measurement goal from various aspects, and these questions can be used to determine whether the specific goal has been achieved or to evaluate the achievement of the goal. In order to continuously improve the quality of software testing, measure the test process and test results, and describe the goals with the following questions: 1) How to verify the effectiveness of the test case? 2) Do the test cases cover the key functions of the business? 3) Do the test cases fully consider potential risks and abnormal situations? 4) Do the test cases consider the performance of the system? 5) What is the degree of automation of the test cases?

[0129] It will be understood that the above questions and their answers are merely examples and are not intended to limit the scope of protection in the embodiments of the present application.

[0130] Secondly, clarify the goal of test case evaluation. The goal of test case evaluation is to evaluate the quality of test cases, improve the efficiency and effectiveness of test cases, thereby improving the quality of financial software and user satisfaction.

[0131] In one embodiment of the present application, the method further includes: collecting data of multiple secondary measurement elements in different business projects to obtain corresponding probability values ​​in each secondary measurement element.

[0132] Collect data from each secondary metric in actual business projects, use the answer to each question as a direct metric, and further decompose the direct metric into secondary metrics as needed. These metrics can be subjective or objective. Obtain probability values ​​for secondary metrics such as defect discovery rate, test case accuracy, functional requirement coverage, fault injection case pass rate, TPS compliance rate, and automation coverage.

[0133] The present application embodiment also provides a test case evaluation device 200, such as Figure 2 As shown, a schematic diagram of the structure of a test case evaluation device in an embodiment of the present application is provided, wherein the test case evaluation device 200 at least includes: a determination module 210, a construction module 220, and a calculation module 230, wherein:

[0134] In one embodiment of the present application, the determination module 210 is specifically used to: determine test case metrics based on business requirements, wherein the test case metrics are obtained based on preset questions, and the preset questions are obtained based on evaluation objectives of the test case.

[0135] The test case is explained in detail using the financial industry software test case as an example.

[0136] Test case metrics can be constructed based on the GQM method from the perspective of goal-problem-metric and the perspective of test quality managers in the financial industry. The GQM method is a goal-problem-metric software quality management method proposed by the National Institute of Standards and Technology (NIST) of the United States. It aims to help software organizations and projects determine and achieve software quality goals, solve software quality problems, and measure software quality results. The core idea of ​​the GQM method is to form a hierarchical software quality model by decomposing software quality goals into specific problems, and then decomposing problems into measurable indicators. Then, by collecting and analyzing software quality data, the status and trend of software quality can be evaluated and improved. The advantages of the GQM method are that it can define and measure software quality from multiple perspectives and dimensions, can customize and adjust the software quality model according to different software environments and requirements, and can provide a systematic and scientific framework for software quality management.

[0137] Based on the GQM method, when applied to software test cases in the financial industry, the writing quality of test cases and the stability of execution results can be guaranteed. At the same time, the quality of test cases can be predicted and optimized in the test design stage.

[0138] In one embodiment of the present application, the construction module 220 is specifically used to construct a test case hierarchy according to the test case metric.

[0139] By adopting the test case metric, multiple quality attributes and indicators in the test case are comprehensively considered, and the quality and quantity of the test case are balanced, so as to meet the diversified and personalized needs of financial software testing.

[0140] In one embodiment of the present application, the calculation module 230 is specifically used to calculate the indicator weights in the test case hierarchy to obtain the evaluation results of the software test cases.

[0141] By calculating the weights of indicators in the test case hierarchy, objective evaluation results of software test cases can be obtained. The analytic hierarchy process (AHP) can be used to determine the relative importance of different evaluation indicators in combination with the evaluation model of the AHP method, and the objectivity and scientificity of the evaluation can be ensured by establishing a judgment matrix and performing consistency tests. This combined method makes the evaluation indicator system more accurate and practical.

[0142] It can be understood that the AHP evaluation method is an evaluation method of the hierarchical analysis method. It was proposed by Thomas Saaty, an American operations researcher, to help decision makers deal with complex multi-criteria decision-making problems. The core idea of ​​the AHP evaluation method is to decompose the decision problem into different levels such as goals, criteria and plans, and then construct a judgment matrix, use expert scoring, calculate the weights of each level, and perform consistency tests to obtain the optimal solution or optimal ranking of the decision problem. The advantages of the AHP evaluation method are that it can make full use of the knowledge and experience of experts, can process qualitative and quantitative data, can consider the relative importance of each criterion, and can provide a simple and effective decision evaluation method.

[0143] In one embodiment of the present application, the construction module 220 is also used to

[0144] Constructing a first-level metric element set according to the first-level metric elements in the test case metric indicators;

[0145] Decomposing the first-level metric element set according to the second-level metric elements in the test case metric indicators to obtain a second-level metric element set;

[0146] in,

[0147] There is a many-to-one mapping relationship between the secondary metric element set and the primary metric element set.

[0148] In one embodiment of the present application, the construction module 220 is also used to

[0149] Constructing a judgment matrix corresponding to the secondary metric element set;

[0150] A matrix consistency check of the secondary metric set is performed according to the judgment matrix, and the overall indicator weight is calculated.

[0151] In one embodiment of the present application, the determining module 210 is further configured to:

[0152] Obtain the evaluation objectives of the test cases based on the business requirements of the software test cases;

[0153] Based on the evaluation objective of the test case, obtaining the preset question;

[0154] According to the preset questions, determine any one or more test case first-level metric indicators among test case design, functional test case coverage, fault type injection, fault type injection, performance test case, and automation test case,

[0155] in,

[0156] The test case design includes secondary metrics of one or more dimensions of defect discovery rate, test case accuracy, and test case scale;

[0157] The functional use case coverage includes secondary metric elements of one or more dimensions of functional requirement coverage and business process coverage;

[0158] The fault type injection is used to evaluate whether the test case fully considers potential risks and abnormal situations;

[0159] The performance test cases are used to ensure that the test cases can fully consider the performance requirements of the financial software system;

[0160] The automation use case includes secondary metrics of one or more dimensions: automation coverage, automation execution pass rate, and automation error detection rate.

[0161] In one embodiment of the present application, the determining module 210 is further configured to:

[0162] Any one or more answers of the test case design, functional case coverage, fault type injection, fault type injection, performance test case, and automation case in each of the preset questions are used as the first-level metric element;

[0163] The first-level metric element is decomposed into second-level metric elements according to business requirements.

[0164] In one embodiment of the present application, the determining module 210 is further configured to:

[0165] Evaluate the quality of test cases as business requirements and determine the goals of test case evaluation;

[0166] The evaluation objective based on the test case is used to obtain the preset question, including:

[0167] The preset questions are obtained by characterizing the objectives of the test case evaluation to determine whether the objectives have been achieved or to evaluate the achievement of the objectives.

[0168] In one embodiment of the present application, a collection module is also included for

[0169] Collect data of multiple secondary metrics in different business projects and obtain corresponding probability values ​​in each secondary metric.

[0170] It can be understood that the above-mentioned test case evaluation device can implement each step of the test case evaluation method provided in the above-mentioned embodiment, and the relevant explanations about the test case evaluation method are applicable to the test case evaluation device and will not be repeated here.

[0171] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0172] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0173] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0174] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a test case evaluation device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0175] Determine test case metrics based on business needs, where the test case metrics are obtained based on preset questions, which are obtained based on evaluation objectives of the test case;

[0176] constructing a test case hierarchy according to the test case metrics; and

[0177] Calculate the indicator weights in the test case hierarchy and obtain the evaluation results of the software test cases.

[0178] The above application Figure 1 The method performed by the test case evaluation device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0179] The electronic device may also perform Figure 1 A method for executing a test case evaluation device in a Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0180] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method executed by the test case evaluation device in the embodiment shown is specifically used to execute:

[0181] Determine test case metrics based on business needs, where the test case metrics are obtained based on preset questions, which are obtained based on evaluation objectives of the test case;

[0182] constructing a test case hierarchy according to the test case metrics; and

[0183] Calculate the indicator weights in the test case hierarchy and obtain the evaluation results of the software test cases.

[0184] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0189] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0190] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0191] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0192] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0193] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A test case evaluation method, wherein: The method comprises: Determine test case metrics based on business needs, where the test case metrics are obtained based on preset questions, which are obtained based on evaluation objectives of the test case; constructing a test case hierarchy according to the test case metrics; and Calculate the indicator weights in the test case hierarchy and obtain the evaluation results of the software test cases.

2. The method of claim 1, wherein: The step of constructing a test case hierarchy according to the test case metric includes: Constructing a first-level metric element set according to the first-level metric elements in the test case metric indicators; Decomposing the first-level metric element set according to the second-level metric elements in the test case metric indicators to obtain a second-level metric element set; in, There is a many-to-one mapping relationship between the secondary metric element set and the primary metric element set.

3. The method of claim 2, wherein: The calculation of the indicator weights in the test case hierarchy includes: Constructing a judgment matrix corresponding to the secondary metric element set; A matrix consistency check of the secondary metric set is performed according to the judgment matrix, and the overall indicator weight is calculated.

4. The method of claim 1, wherein: Determining the test case metric indicators according to the business requirements, wherein the test case metric indicators are obtained based on preset questions, and the preset questions are obtained based on the evaluation objectives of the test case, includes: Obtain the evaluation objectives of the test cases based on the business requirements of the software test cases; Based on the evaluation objective of the test case, obtaining the preset question; According to the preset questions, determine any one or more test case first-level metric indicators among test case design, functional test case coverage, fault type injection, fault type injection, performance test case, and automation test case, in, The test case design includes secondary metrics of one or more dimensions of defect discovery rate, test case accuracy, and test case scale; The functional use case coverage includes secondary metric elements of one or more dimensions of functional requirement coverage and business process coverage; The fault type injection is used to evaluate whether the test case fully considers potential risks and abnormal situations; The performance test cases are used to ensure that the test cases can fully consider the performance requirements of the financial software system; The automation use case includes secondary metrics of one or more dimensions: automation coverage, automation execution pass rate, and automation error detection rate.

5. The method of claim 4, wherein: Determining any one or more test case metrics among test case design, functional test case coverage, fault type injection, fault type injection, performance test case, and automation test case based on the preset questions includes: Any one or more answers of the test case design, functional case coverage, fault type injection, fault type injection, performance test case, and automation case in each of the preset questions are used as the first-level metric element; The first-level metric element is decomposed into second-level metric elements according to business requirements.

6. The method of claim 5, wherein: The step of obtaining the evaluation objectives of the test cases according to the business requirements of the software test cases includes: Evaluate the quality of test cases as business requirements and determine the goals of test case evaluation; The evaluation objective based on the test case is used to obtain the preset question, including: The preset questions are obtained by characterizing the objectives of the test case evaluation to determine whether the objectives have been achieved or to evaluate the achievement of the objectives.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Collect data of multiple secondary metrics in different business projects and obtain corresponding probability values ​​in each secondary metric.

8. A test case evaluation device, wherein: The device comprises: A determination module, used to determine test case metrics according to business requirements, wherein the test case metrics are obtained based on preset questions, and the preset questions are obtained based on evaluation objectives of the test case; A construction module, used to construct a test case hierarchy according to the test case metrics; and The calculation module is used to calculate the indicator weights in the test case hierarchy and obtain the evaluation results of the software test cases.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images