A method and device for determining software testing effectiveness measurement

Abstract

The present invention discloses a method and device for determining software testing effectiveness measurement, and relates to the field of software testing. It is used to solve the problem that there is no objective standard for the effectiveness measurement and evaluation of software testing, and most of the proposed evaluation models are theoretical models lacking practical applications. The method includes: obtaining defect records and defect coverage measurement value evaluation data of a software system project tested by each test group within the evaluation cycle; establishing a data matrix according to the defect coverage measurement value evaluation data, and obtaining the ratio of different product element measurement indicators and the weight of different product element measurement indicators under each test group sample according to the data matrix; according to the ratio of different product element measurement indicators under each test group sample, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems, the effectiveness measurement value of the software system project is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more specifically to a method and device for determining software testing effectiveness measurement. Background Art

[0002] The process of software testing is the process of measuring and evaluating the quality of the software itself. This measurement and evaluation verifies whether the software itself can meet the needs of users, which is an important basis for users to choose products. Due to the constraints of cost and construction period, it is impossible to test software exhaustively. Therefore, the effectiveness of software testing is taken as a goal to measure the process capability of a testing team and the current quality of the testing process. According to the measurement results of the goal, the advantages and disadvantages of the testing process can be discovered, and the testing strategy can be adjusted in time, ultimately improving the quality and efficiency of software testing.

[0003] At present, the research on the measurement and evaluation of software testing effectiveness by experts and scholars in the field of software testing at home and abroad is in the primary stage. The relevant measurement methods, measurement indicators, evaluation mechanisms and models are not yet mature, and a unified system for measuring and evaluating software testing effectiveness has not been formed. "The Measurement Problem of Software Quality" proposes that software measurement can be divided into several aspects: establishing software quality models and requirements, identifying software quality measurement methods suitable for selected quality factors, implementing software quality measurement, analyzing and interpreting measurement results, and confirming software quality measurement. "Evaluating the Effectiveness of Software Testing" gives a general process for evaluating the effectiveness of software testing, namely, determining the evaluation objectives, determining the measurement content, specifying the measurement responsibilities, selecting the evaluation method, determining the required facts, collecting evaluation data, and evaluating the effectiveness of the test.

[0004] Evaluation and assessment research in the field of software testing has always been in a state of lack. Although the above-mentioned prior art has proposed methods for software testing measurement or evaluation, it is still immature, and there is no relatively complete and practical application significance for the measurement and evaluation system of software testing effectiveness. Due to the particularity of software products and the complexity of software processes, the measurement of software testing effectiveness is abstract and complex. The traditional software testing effectiveness measurement method is to conduct statistics and analysis through the test defect problem record sheet of the tester. Its disadvantages are: the tester is mainly concerned with the analysis of the software problems found, and the classification of the problems is considered less, and there is no unified and detailed classification measurement standard. Statisticians cannot extract effective classification measurement information for testing technology capability evaluation. In addition, the commonly used coverage analysis technology is based on the requirements specification for business function coverage, which cannot reflect the deep testing technology capabilities; the other is to use white box testing path coverage, condition coverage, etc., which is more suitable for unit testing in the development stage; the number of missed defects is generally used in the later release stage, which is not comprehensive and cannot reflect the overall testing situation in a timely manner.

[0005] In summary, the existing software testing effectiveness measurement and evaluation have the problem that there is no objective standard for effectiveness, and most of the proposed evaluation models are theoretical models that lack practical applications. Summary of the invention

[0006] The embodiment of the present invention provides a method and device for determining software testing effectiveness measurement, so as to solve the problems that the effectiveness measurement and evaluation of existing software testing have no objective standard for effectiveness, and most of the proposed evaluation models are theoretical models lacking practical applications.

[0007] An embodiment of the present invention provides a method for determining software testing effectiveness measurement, comprising:

[0008] Obtain defect records of a software system project tested by each test group within an evaluation cycle, and obtain defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records;

[0009] A data matrix is ​​established according to the defect coverage metric value evaluation data, and the ratios of different product element metric indicators and the weights of different product element metric indicators under each test group sample are obtained according to the data matrix;

[0010] According to the ratio of different product element measurement indicators under each test group sample, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems, the effectiveness measurement value of the software system project is obtained.

[0011] Preferably, the effectiveness metric value of the software system project is obtained according to the following formula:

[0012]

[0013] Among them, U is the effectiveness measurement value of the software system project; n is a natural number, and m is a natural number; is the ratio of the jth product element metric under the i-th tester / group sample; W j is the weight of the j-th product element metric; S is the stability coefficient of different project systems; D is the difficulty coefficient of different project systems.

[0014] Preferably, the weight of the j-th product element metric is determined according to the following formula:

[0015]

[0016] Among them, d j is the redundancy of the information entropy of the j-th product element metric, d j =1-ej , j = 1, ..., n; e j is the information entropy value of the j-th product element metric, m is the number of samples, k>0, ln is the natural logarithm.

[0017] Preferably, the stability coefficients of the different project systems are determined based on the number of test cases that can be successfully executed and the total number of test cases designed to be executed; the difficulty coefficients of the different project systems are determined based on the number of test cases manually tested and the total number of test cases designed to be executed.

[0018] Preferably, obtaining defect coverage metric value evaluation data of each test group based on different product element metric dimensions according to the defect record specifically includes:

[0019] The defect records are mapped to the six dimensions included in the product elements to obtain the number of defects provided by each test group. According to the number of defects provided by each test group and the importance level coefficient of each defect, the defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions is obtained.

[0020] The embodiment of the present invention also provides a device for determining software testing effectiveness measurement, including:

[0021] The first obtaining unit is used to obtain defect records of a software system project tested by each test group within an evaluation period, and obtain defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records;

[0022] A second obtaining unit is used to establish a data matrix according to the defect coverage metric value evaluation data, and obtain the ratios of different product element metric indicators and the weights of different product element metric indicators under each test group sample according to the data matrix;

[0023] The third obtaining unit is used to obtain the effectiveness measurement value of the software system project according to the ratio of different product element measurement indicators under each test group sample, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems.

[0024] Preferably, the effectiveness metric value of the software system project is obtained according to the following formula:

[0025]

[0026] Among them, U is the effectiveness measurement value of the software system project; n is a natural number, and m is a natural number; is the ratio of the jth product element metric under the i-th tester / group sample; W jis the weight of the j-th product element metric; S is the stability coefficient of different project systems; D is the difficulty coefficient of different project systems.

[0027] Preferably, the weight of the j-th product element metric is determined according to the following formula:

[0028]

[0029] Among them, d j is the redundancy of the information entropy of the j-th product element metric, d j =1-e j , j = 1, ..., n; e j is the information entropy value of the j-th product element metric, m is the number of samples, k>0, ln is the natural logarithm.

[0030] Preferably, the third obtaining unit is specifically used for:

[0031] The stability coefficients of the different project systems are determined based on the number of test cases that can be successfully executed and the total number of test cases designed to be executed; the difficulty coefficients of the different project systems are determined based on the number of test cases manually tested and the total number of test cases designed to be executed.

[0032] Preferably, the first obtaining unit is specifically used for:

[0033] The defect records are mapped to the six dimensions included in the product elements to obtain the number of defects provided by each test group. According to the number of defects provided by each test group and the importance level coefficient of each defect, the defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions is obtained.

[0034] The embodiment of the present invention provides a method and device for determining software testing effectiveness measurement, the method comprising: obtaining defect records of a software system project tested by each test group within an evaluation period, obtaining defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records; establishing a data matrix according to the defect coverage measurement value evaluation data, obtaining the ratio of different product element measurement indicators and the weight of different product element measurement indicators under each test group sample according to the data matrix; obtaining the effectiveness measurement value of the software system project according to the ratio of different product element measurement indicators, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems under each test group sample. The method is based on the software testing effectiveness measurement method of defect coverage analysis of product elements, forms a unified software testing effectiveness measurement evaluation system, and provides an objective standard for evaluating the software testing effectiveness evaluation of each tester / group under different software system projects; solves the problem that there is no objective standard for the effectiveness of existing software testing effectiveness measurement and evaluation, and most of the proposed evaluation models are theoretical models lacking practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a method for determining software testing effectiveness measurement provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a device for determining software testing effectiveness metrics provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Figure 1 A flowchart of a method for determining software testing effectiveness measurement provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method mainly includes the following steps:

[0040] Step 101, obtaining defect records of a software system project tested by each test group within an evaluation cycle, and obtaining defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records;

[0041] Step 102, establishing a data matrix according to the defect coverage metric value evaluation data, and obtaining the ratios of different product element metric indicators and the weights of different product element metric indicators under each test group sample according to the data matrix;

[0042] Step 103, obtaining the effectiveness measurement value of the software system project according to the ratio of different product element measurement indicators under each test group sample, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems.

[0043] In step 101, each test group may be a test group including multiple persons or a test group including only one person. The evaluation period may be determined according to actual application, and in the embodiment of the present invention, the specific time of the evaluation period is not limited.

[0044] During the evaluation cycle, each test group tests a software system project and obtains the defect records of the software system project.

[0045] Furthermore, the acquired defect records are mapped to the six dimensions included in the product elements to obtain the number of defects. In the embodiment of the present invention, the six dimensions include structure, function, data, platform, operation and time.

[0046] According to the number of defects provided by each test group and the importance level coefficient corresponding to each defect, defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions can be obtained.

[0047] Each test group is determined based on the defect coverage metric evaluation data under different product element metric dimensions according to the ratio of the number of defects to the importance level coefficient of the defects. Among them, the importance level coefficient of the defect includes recommended, general, minor, serious and major. In the embodiment of the present invention, the importance level coefficient of the defect can be expressed by N, where N = {recommended, general, minor, serious, major} = {N 1 ,N 2 ,N 3 ,N 4 ,N 5}; Furthermore, the importance level coefficient corresponding to N can be assigned a value, for example, {N 1 ,N 2 ,N 3 ,N 4 ,N 5In the embodiment of the present invention, the specific value of the importance level coefficient corresponding to N is not limited.

[0048] Table 1 is a table of defect coverage metric value evaluation data of each test group based on different product element measurement dimensions provided in an embodiment of the present invention. As can be seen from Table 1, different test groups test a software system project, and the obtained defect coverage metric value evaluation data of each test group based on different product element measurement dimensions are obtained.

[0049] Table 1

[0050]

[0051] In step 102, a data matrix is ​​established based on the defect coverage metric evaluation data. For example, there are m test group samples and n product element metric indicators. This study calculates the defect coverage metric value by multiplying the number of defects provided by the test group with the defect importance level coefficient corresponding to the specific defect, and scores the evaluation result. The data matrix is ​​determined based on the evaluation result. The established data matrix X is shown in formula (1):

[0052]

[0053] Where X is the data matrix, x ij is the index value of the jth product element metric of the i-th test group, x ij It is equal to the product of the number of defects and the defect importance level coefficient.

[0054] Furthermore, the data matrix obtained by evaluating the defect coverage metric values ​​in Table 1 is shown in Table 2:

[0055] Table 2

[0056]

[0057] Furthermore, the ratio of the jth product element metric index under the i-th test group sample is determined by the following formula (2):

[0058]

[0059] Among them, i=1,2,...m, j=1,2,...n, m and n are both natural numbers.

[0060] Furthermore, the entropy values ​​of different product element metrics are determined according to formula (3):

[0061]

[0062] Among them, j is the number of indicators, m is the number of samples, and generally k>0, ln is the natural logarithm, satisfying e j ≥0, the information entropy value table is shown in Table 3:

[0063] Table 3

[0064] e1 e2 e3 e4 e5 e6 0.967 0.894 0.967 0.879 0.973 0.913

[0065] In practical applications, information entropy redundancy reflects the degree of influence of product element metrics on software testing effectiveness measurement. For a given j, the smaller the difference between the metric values, the greater the e j The larger the value, when all are equal, we can deduce e from formula (3) j =1, the indicator value has no effect at this time. When the difference in the indicator value is larger, e j The smaller the value is, the greater the effect of the indicator on the effectiveness measurement of software testing. Therefore, the coefficient of difference can be determined according to the following formula (4):

[0066] d j =1-e j (4)

[0067] Among them, d j is the redundancy of the information entropy of the j-th product element metric, j = 1,…,n. The redundancy of the information entropy of the product element metric is shown in Table 4:

[0068] Table 4

[0069] d1 d2 d3 d4 d5 d6 0.033 0.106 0.033 0.121 0.027 0.087

[0070] Furthermore, the weight of the j-th product element metric is determined according to the following formula (5):

[0071]

[0072] Among them, d j is the redundancy of the information entropy of the j-th product element metric, d j =1-e j , j = 1, ..., n;

[0073] e j is the information entropy value of the j-th product element metric, m is the number of samples, k>0, ln is the natural logarithm.

[0074] The weights of product element metrics are shown in Table 5:

[0075] Table 5

[0076] w1 w2 w3 w4 w5 w6 0.08 0.26 0.08 0.297 0.066 0.214

[0077] In step 103, according to the ratio of different product element metrics under each test group sample, the weight of different product element metrics, the stability coefficient of different project systems and the difficulty coefficient of different project systems, the effectiveness metric value of the software system project is obtained by the following formula (6):

[0078]

[0079] Among them, U is the effectiveness measurement value of the software system project; n is a natural number, and m is a natural number;

[0080] is the ratio of the jth product element metric under the i-th tester / group sample;

[0081] W j is the weight of the j-th product element metric; S is the stability coefficient of different project systems; D is the difficulty coefficient of different project systems.

[0082] It should be noted that, in the embodiments of the present invention, the stability coefficient describes the proportion of test cases that can be successfully executed under different software systems under test to the total number of test cases designed to be executed; the test difficulty coefficient describes the proportion of test cases that cannot be converted or completely converted into automated test execution under different software systems under test, and still require manual testing by testers, that is, the proportion of test cases that must rely on manual testing to the total number of test cases designed to be executed.

[0083] Among them, the stability coefficient of the software system tested by each test / group under the corresponding project is shown in Table 6, and the difficulty coefficient of the software system tested by each test group under the corresponding project is shown in Table 7:

[0084] Table 6

[0085]

[0086]

[0087] Table 7

[0088]

[0089] In the embodiment of the present invention, the comprehensive evaluation scores of software test effectiveness of each test group based on defect coverage analysis of product elements are shown in Table 8:

[0090] Table 8

[0091]

[0092] In the embodiment of the present invention, the entropy method is used to completely evaluate the software testing effectiveness of the defect coverage analysis based on product elements of the software system projects tested within the evaluation cycle, and the defect coverage is evaluated in a complete and comprehensive manner based on the product element measurement indicators, so that the software testing effectiveness evaluation of different projects for each tester / group within the evaluation cycle is more practical. For the effectiveness and importance of the software system defects proposed by each tester / group based on different product element measurement dimensions, the test effectiveness measurement value of each tester / group is well given, and can be used as an evaluation ranking of key performance, and further combined with other evaluation indicators or converted into the percentage of each evaluation object, a comprehensive evaluation of the software testing group is achieved, which has good operability and applicability in the field of software testing work effectiveness evaluation.

[0093] Based on the same inventive concept, an embodiment of the present invention provides a device for determining a measure of software testing effectiveness. Since the principle of the device for solving technical problems is similar to a method for determining a measure of software testing effectiveness, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0094] like Figure 2 As shown, the device includes a first obtaining unit 201 , a second obtaining unit 202 and a third obtaining unit 203 .

[0095] The first obtaining unit 201 is used to obtain defect records of a software system project tested by each test group within an evaluation period, and obtain defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records;

[0096] A second obtaining unit 202 is used to establish a data matrix according to the defect coverage metric value evaluation data, and obtain the ratios of different product element metric indicators and the weights of different product element metric indicators under each test group sample according to the data matrix;

[0097] The third obtaining unit 203 is used to obtain the effectiveness measurement value of the software system project according to the ratio of different product element measurement indicators under each test group sample, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems.

[0098] Preferably, the effectiveness metric value of the software system project is obtained according to the following formula:

[0099]

[0100] Among them, U is the effectiveness measurement value of the software system project; n is a natural number, and m is a natural number; is the ratio of the jth product element metric under the i-th tester / group sample; Wj is the weight of the j-th product element metric; S is the stability coefficient of different project systems; D is the difficulty coefficient of different project systems.

[0101] Preferably, the weight of the j-th product element metric is determined according to the following formula:

[0102]

[0103] Among them, d j is the redundancy of the information entropy of the j-th product element metric, d j =1-e j , j = 1, ..., n; e j is the information entropy value of the j-th product element metric, m is the number of samples, k>0, ln is the natural logarithm.

[0104] Preferably, the third obtaining unit 203 is specifically used for:

[0105] The stability coefficients of the different project systems are determined based on the number of test cases that can be successfully executed and the total number of test cases designed to be executed; the difficulty coefficients of the different project systems are determined based on the number of test cases manually tested and the total number of test cases designed to be executed.

[0106] Preferably, the first obtaining unit 201 is specifically used for:

[0107] The defect records are mapped to the six dimensions included in the product elements to obtain the number of defects provided by each test group. According to the number of defects provided by each test group and the importance level coefficient of each defect, the defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions is obtained.

[0108] It should be understood that the units included in the above device for determining software test effectiveness measurement are only logical divisions based on the functions implemented by the device. In practical applications, the above units can be superimposed or split. In addition, the functions implemented by the device for determining software test effectiveness measurement provided in this embodiment correspond to the method for determining software test effectiveness measurement provided in the above embodiment. The more detailed processing flow implemented by the device has been described in detail in the above method embodiment 1 and will not be described in detail here.

[0109] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0110] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for determining a software testing effectiveness metric, characterized in that: include: Obtain defect records of a software system project tested by each test group within an evaluation cycle, and obtain defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records; A data matrix is ​​established according to the defect coverage metric value evaluation data, and the ratios of different product element metric indicators and the weights of different product element metric indicators under each test group sample are obtained according to the data matrix; According to the ratio of different product element metrics under each test group sample, the weight of different product element metrics, the stability coefficient of different project systems and the difficulty coefficient of different project systems, the effectiveness metric value of the software system project is obtained according to the following formula: in, It is the effectiveness measure of software system projects; is a natural number; For the Tester or The first The ratio of product element metrics; For the The weight of the product element measurement indicator; S is the stability coefficient of different project systems, D is the difficulty coefficient of different project systems, is the sample size; The weight of the product element metric is determined according to the following formula: in, For the The redundancy of the information entropy of the product element measurement indicators, , ; For the The information entropy value of the product element measurement indicator, , is the sample size, , , is the natural logarithm.

2. The method according to claim 1, characterized in that The stability coefficients of the different project systems are determined based on the number of test cases that can be successfully executed and the total number of test cases designed to be executed; the difficulty coefficients of the different project systems are determined based on the number of test cases manually tested and the total number of test cases designed to be executed.

3. The method according to claim 1, characterized in that The step of obtaining defect coverage metric value evaluation data of each test group based on different product element metric dimensions according to the defect records specifically includes: The defect records are mapped to the six dimensions included in the product elements to obtain the number of defects provided by each test group. According to the number of defects provided by each test group and the importance level coefficient of each defect, the defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions is obtained.

4. A device for determining a measure of software testing effectiveness, characterized in that include: The first obtaining unit is used to obtain defect records of a software system project tested by each test group within an evaluation period, and obtain defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions according to the defect records; A second obtaining unit is used to establish a data matrix according to the defect coverage metric value evaluation data, and obtain the ratios of different product element metric indicators and the weights of different product element metric indicators under each test group sample according to the data matrix; The third obtaining unit is used to obtain the effectiveness measurement value of the software system project according to the ratio of different product element measurement indicators under each test group sample, the weight of different product element measurement indicators, the stability coefficient of different project systems and the difficulty coefficient of different project systems according to the following formula: in, It is the effectiveness measure of software system projects; is a natural number; For the Tester or The first The ratio of product element metrics; For the The weight of the product element measurement indicator; S is the stability coefficient of different project systems, D is the difficulty coefficient of different project systems, is the sample size; The weight of the product element metric is determined according to the following formula: in, For the The redundancy of information entropy of product element measurement indicators, , ; For the The information entropy value of the product element measurement indicator, , is the sample size, , , is the natural logarithm.

5. The device according to claim 4, characterized in that The third obtaining unit is specifically used for: The stability coefficients of the different project systems are determined based on the number of test cases that can be successfully executed and the total number of test cases designed to be executed; the difficulty coefficients of the different project systems are determined based on the number of test cases manually tested and the total number of test cases designed to be executed.

6. The device according to claim 4, characterized in that The first obtaining unit is specifically used for: The defect records are mapped to the six dimensions included in the product elements to obtain the number of defects provided by each test group. According to the number of defects provided by each test group and the importance level coefficient of each defect, the defect coverage measurement value evaluation data of each test group based on different product element measurement dimensions is obtained.

Images