A combined testing method based on metamorphosis relationship matching

Abstract

The present invention provides a combination test method based on metamorphosis relationship matching. The method first converts the metamorphosis relationship into a constraint on the combination test model, and then generates a set of test case sets that not only meet the combination coverage but also contain as many test case pairs matching the metamorphosis relationship as possible. Different from the traditional combination test, the method automatically determines the correctness of the software to be tested by verifying whether the output of the test case pair matching the metamorphosis relationship violates the output relationship of the metamorphosis relationship, rather than judging the correctness of the software to be tested by establishing a precise mapping between the input of each test case and the expected output.

Description

Technical Field

[0001] The invention relates to a combined testing method based on metamorphosis relationship matching. Background Art

[0002] Combinatorial testing (CT) (References: C. Nie and H. Leung, “A survey of combinatorial testing,” ACM Computing Surveys (CSUR), vol. 43, no. 2, pp. 11: 1–11: 29, Feb 2011; DR Kuhn, RN Kacker, and Y. Lei, “Practical combinatorial testing,” NIST Special Publication, vol. 800, p. 142, 2010) can efficiently detect faults caused by interacting factors. It mainly utilizes a mathematical object called a covering array (CA), which is a detailed data table that covers all t-dimensional combinations of parameters at least once. In the field of combinatorial testing, few studies have focused on the verification of the correctness of test outputs, namely the Oracle problem. In most scenarios, human labor is required to check the system behavior of the test case set generated by CT to further determine whether the software has faults. However, the labor-intensive Oracle will cause the actual cost of CT to be much higher than expected.

[0003] In addition to labor-intensive oracles, there are other types of oracles in the field of combinatorial testing. Implicit oracles (reference: ET Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, "The oracle problem in software testing: Asurvey," IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015) are software behaviors that can be directly observed, such as exceptions, crashes, and memory leaks, but the oracle limits the fault types and test scenarios. When a complete specification of the software exists, embedded assertions (reference: Y.Ledru, L.du Bousquet, O.Maury, and P.Bontron, “Filtering to bias combinatorial test suites,” in Fundamental Approaches to Software Engineering, M.Wermelinger and T.Margaria-Steffen, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp.281–294) or preconditions and postconditions (reference: T.Triki, “Filtering and reduction techniques of combinatorial tests,” Other [cs.OH]. Universit′e de Grenoble English, 2013) can also serve as oracles, but this oracle requires a complete software specification and is highly dependent on testing expertise.Another common oracle in the field of combinatorial testing is the existence of a completely correct version of the software (References: LSG Ghandehari, MN Bourazjany, Y. Lei, RN Kacker, and DR Kuhn, "Applying combinatorial testing to the siemens suite," in Software Testing, Verification and Validation Workshops (ICSTW), 2013 IEEE Sixth International Conference on. IEEE, 2013, pp. 362–371; L. Sh. Ghandehari, Y. Lei, R. Kacker, R. Kuhn, T. Xie, and D. Kung, "A combinatorial testing-based approach to fault localization," IEEE Transactions on Software Engineering, vol. 46, no. 6, pp. 616–645, 2020; Z. Zhang and J. Zhang, "Characterizing failure-causing parameter interactions by adaptive testing," in Proceedings of the 2011 International Symposium on Software Testing and Analysis.ACM, 2011, pp.331–341; N.Xintao, N.Changhai, Y.Lei, and ATSChan, “Identifying failure-inducing combinationsusing tuple relationship,” in 2013IEEE Sixth International Conference onSoftware Testing,Verification and Validation Workshops.IEEE, 2013, pp.271–280). The correctness can be directly verified by comparing the results of the current version with the results of the correct version, but there is often no completely correct version.Although the above Oracles reduce manual costs, their limited error types, the need for completely correct software versions or complete specifications, and the high reliance on the expertise of test engineers may prevent these Oracles from being applied to more general testing scenarios. Therefore, in the field of combinatorial testing, automated Oracles remain an urgent problem to be solved. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a combination test method based on metamorphic relationship matching, that is, a combination test method based on metamorphic relationship matching, to support automatic Oracle. The inspiration of this method comes from metamorphic testing (reference: T. Chen, S. Cheung, and S. Yiu, "Metamorphic testing: a new approach for generating next test cases," Technical Report HKUST CS98-01. Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong., pp. 1–11, 1998). Metamorphic testing is a recognized effective method to alleviate the Oracle problem, which automatically determines the correctness of the program by judging the output relationship of related test cases.

[0005] It should be noted that traditional combinatorial testing uses fewer test cases to test the system while ensuring the error detection rate, while metamorphic testing determines the correctness of the software to be tested through the relationship between similar test cases. Simply put, traditional combinatorial testing generates test cases that are as different as possible, while metamorphic testing expects similar test cases. In most cases, it is difficult for traditional combinatorial testing to generate test case pairs that satisfy the metamorphic relationship, that is, it is impossible to determine the correctness of the program through the metamorphic relationship. In this regard, the present invention proposes a new combinatorial test case generation algorithm that generates as many test case pairs that match the metamorphic relationship as possible under the premise of low cost.

[0006] The key idea of ​​this method is to generate as many test case pairs that match the transformation relationship as possible while satisfying the t-dimensional coverage. It includes the following steps:

[0007] Step 1, input the abstract parameter model and the transformation relationship, and transform the transformation relationship into constraints on the model;

[0008] Step 2, generate an abstract test case set;

[0009] Step 3, generate a specific test case set;

[0010] Step 4: Use the test cases to execute the software under test and use the script to automatically detect whether the program fails.

[0011] Step 1 includes the following steps:

[0012] Step 1-1, construct an abstract parameter model of the combination test of the software to be tested, including parameters, the value range and constraints of each parameter, and determine the transformation relationship of the software to be tested, the input rules and output rules of the transformation relationship.

[0013] Step 1-1 includes: setting the software to be tested to have n parameters, the i-th parameter p i The finite range of V i , where 1≤i≤n; for each parameter p i , from the set V i Take the abstract value v i Assigned to parameter p i , forming an abstract test case of the combined test, namely {(p1,v1),(p2,v2),...,(p n ,v n )}, where the abstract value v i It may correspond to an actual value, or to an equivalence class;

[0014] For each abstract value v in the abstract test case i , converting it into a concrete value c with real meaning i , by c i A specific test case that constitutes the combined test, namely {(p1,c1),(p2,c2),...,(p n ,c n )};

[0015] Step 1-1 also includes: converting the transformation relationship into constraints on the combination test model, including two cases:

[0016] In the first case, the metamorphosis relationship can be satisfied by the abstract test case. That is, if two abstract test cases satisfy the metamorphosis relationship, then the specific test cases corresponding to the two abstract test cases must also satisfy the metamorphosis relationship. In this case, the metamorphosis relationship only needs to be considered in the abstract test case generation stage. Set the source abstract test case to {(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )},p i represents the i-th abstract parameter, v si Represents the i-th parameter p i Abstract value of, where 1≤i≤n; the subsequent abstract test case is {(p1,v f1),(p2,v f2 ),...,(p n ,v fn )},v fi Represents the i-th parameter p i The abstract value of the transformation relation; the input relation r in It is expressed by the following formula:

[0017] r in (v s1 ,v s2 ,...,v sn ,v f1 ,v f2 ,...,v fn )=true

[0018] The above formula indicates that the values ​​of the two abstract test cases satisfy the input relation r in For constraints on abstract test case pairs, true represents the truth value;

[0019] In the second case, the metamorphosis relationship can only be satisfied by specific test cases. It is not enough to generate the corresponding abstract test cases. In this case, the metamorphosis relationship needs to be considered in both the abstract test case generation stage and the specific test case generation stage. The source abstract test case is set to {(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )}, the corresponding source specific test case is {(p1,c s1 ),(p2,c s2 ),...,(p n ,c sn )},c si Indicates that the i-th parameter of the source specific test case corresponds to a specific value, where 1≤i≤n; the subsequent abstract test case is {(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )}, the corresponding subsequent specific test case is {(p1,c f1 ),(p2,c f2 ),...,(p n ,c fn )},c fi Indicates the specific value corresponding to the i-th parameter of the subsequent specific test case, where 1≤i≤n. The input relation r of the transformation relation in It is expressed by the following formula:

[0020] r in (v s1 ,v s2,...,v sn ,v f1 ,v f2 ,...,v fn )=true

[0021] r in (c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn )=true

[0022] The above formula indicates that the values ​​of the two abstract test cases satisfy the input relation r in For the constraints of the abstract test case pair, the values ​​of the two specific test cases satisfy the input relation r in Constraints on specific test case pairs;

[0023] The output relationship of the transformation relationship is not considered in the test case generation stage. The output relationship of the transformation relationship is considered only when judging whether the program has a fault. The source test case output is set as output s , the subsequent test case output is output f , then the output relation of the transformation relation r out It is expressed by the following formula:

[0024] r out (output s ,output f )=true

[0025] If the test case that satisfies the transformation relationship input relationship has an output relationship that does not satisfy the transformation relationship, it is determined that the program has a fault.

[0026] Step 2 includes the following steps:

[0027] Step 2-1, first use the adaptive random sampling method to generate a set of initial abstract test case sets that meet the t-dimensional coverage rate: each time a set of candidate abstract test case sets is randomly generated, and the number of uncovered combinations covered by each candidate abstract test case is calculated (the uncovered combination is the combination that is not covered by the test case in the result set), and the abstract test case that covers the largest number of uncovered combinations is added to the result set, and the above process is repeated until the number of uncovered combinations is equal to 0, that is, the initial abstract test case set meets the t-dimensional coverage rate;

[0028] Step 2-2, traverse all abstract test case pairs (t1, t2) in the initial abstract test case set, and if (t1, t2) matches a transformation relationship, mark the abstract test cases t1 and t2;

[0029] Step 2-3, take the unlabeled abstract test case as the source abstract test case, and generate a subsequent abstract test case for the source abstract test case, transform the subsequent abstract test case generation problem into a constraint solving problem, assume that the software to be tested has n abstract parameters, and the source abstract test case t s ={(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )},p i represents the i-th abstract parameter, v si Represents the i-th parameter p of the source abstract test case i The abstract value of , where 1≤i≤n; the transformation relationship is (r in ,r out ), r in That is, the input rule of the transformation relationship, r out That is, the output rule of the transformation relationship, if there are n test cases that satisfy r in Relationship, then these n outputs must satisfy r out relationship, otherwise the software has a fault; software constraint C = {c i |c i is the Boolean constraint function of the software to be tested}, and the subsequent abstract test case t f ={(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )},v fi Indicates the i-th parameter p of the subsequent abstract test case i Abstract value of, where 1≤i≤n; subsequent abstract test case t f It is expressed by the following formula:

[0030] Variable: {v f1 ,v f2 ,...,v fn}

[0031] Range: v fi ∈V i ,

[0032] Where V i represents the value range set of the i-th abstract parameter, that is, all parameters take values ​​within the value range;

[0033] Constraint: r in (v s1 ,v s2 ,...,v sn ,v f1 ,vf2 ,...,v fn )=true

[0034] c i (v f1 ,v f2 ,...,v fn )=true,

[0035] Combine the initial abstract test case set and the generated subsequent abstract test case set into a complete abstract test case set.

[0036] Step 3 includes the following steps:

[0037] Step 3-1, traverse all abstract test case pairs (t1, t2) in the abstract test case set. If (t1, t2) matches a transformation relationship, map it to a specific test case pair that matches the transformation relationship. Set (t1, t2) to match a transformation relationship, t1 = {(p1, v s1 ),(p2,v s2 ),...,(p n ,v sn )}, the specific test case corresponding to t1 is {(p1,c s1 ),(p2,c s2 ),...,(p n ,c sn )}, t2={(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )}, the specific test case corresponding to t2 is {(p1,c f1 ),(p2,c f2 ),...,(p n ,c fn )}, software constraint C = {c i |c i is the Boolean constraint function of the software to be tested}; the transformation relationship is (r in ,r out ), r in That is, the input rule of the transformation relationship, r out That is, the output rule of the transformation relationship; then the specific test cases corresponding to t1 and t2 are expressed by the following formula:

[0038] Variable: {c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn}

[0039] Range: c si ∈v si The corresponding equivalence class set, c fi ∈v fi The corresponding equivalence class set,

[0040] Among them, the size of the equivalence class set represented by the abstract value is at least 1, and all concrete values ​​should belong to the equivalence class set corresponding to their abstract value;

[0041] Constraint: r in (c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn )=true

[0042] c i (c s1 ,c s2 ,...,c sn )=true,

[0043] c i (c f1 ,c f2 ,...,c fn )=true,

[0044] According to the above formula, a pair of specific test case pairs that satisfy the metamorphosis relationship is generated; if the metamorphosis relationship is a first-class metamorphosis relationship, that is, a metamorphosis relationship that can be satisfied by an abstract test case, then as long as the abstract test case pair satisfies the metamorphosis relationship, its corresponding specific test case pair will definitely satisfy the metamorphosis relationship, r in (c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn )=true is always true, and the above formula can still be used to represent the specific test case pair;

[0045] Step 3-2, after completing the mapping of the abstract test cases matching the metamorphosis relationship, generate corresponding concrete test cases for the remaining abstract test cases in the abstract test case set, that is, select an actual value from its corresponding equivalence class for each abstract value.

[0046] Step 4 includes: writing a test script, executing the system to be tested using the test input, and recording the actual output of each test case, automatically determining the correctness of the program by judging whether the test case pair matching the metamorphosis relationship satisfies the output rule of the metamorphosis relationship, and if the output rule is not satisfied, it is determined that the program has a fault.

[0047] In step 4, traverse any two test cases in the test case set to determine whether they satisfy the input relationship of the metamorphosis relationship. If they do, determine whether they satisfy the output rule of the metamorphosis relationship. If they do not satisfy the output rule, mark the pair of test cases, that is, the program fails.

[0048] Beneficial effects: Compared with traditional combinatorial testing, this method has two major characteristics. First, the present invention proposes a combinatorial test case generation method, which first converts the metamorphosis relationship into constraints on the combinatorial test model, and then generates a set of test case sets that not only meet the combination coverage but also contain as many test case pairs matching the metamorphosis relationship as possible; second, different from traditional combinatorial testing, this method automatically determines the correctness of the software to be tested by verifying whether the output of the test case pair matching the metamorphosis relationship violates the output relationship of the metamorphosis relationship, rather than judging the correctness of the software to be tested by establishing a precise mapping between the input of each test case and the expected output. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0050] Figure 1 It is the overall flow chart of the present invention.

[0051] Figure 2 It is an overview of abstract test case generation.

[0052] Figure 3 It is an overview of specific test case generation. DETAILED DESCRIPTION

[0053] The present invention provides a combination test method based on metamorphosis relationship matching, which inputs the combination test abstract parameter model and metamorphosis relationship of the software to be tested, generates a set of test case sets that not only meet the t-dimensional combination coverage but also contain as many matching metamorphosis relationships as possible, and outputs test case pairs that violate the metamorphosis relationship (if there are test case pairs that violate the metamorphosis relationship, it is considered that the software to be tested has a fault). The overall process uses an automated script to alleviate the automatic Oracle problem in the combination test. The overall flow chart can be referred to Figure 1 .

[0054] After inputting the combination test abstract parameter model and the transformation relationship, the transformation relationship needs to be converted into constraints on the combination test model. Two situations are mainly considered:

[0055] 1) The metamorphosis relationship can be satisfied by abstract test cases, that is, if two abstract test cases satisfy the metamorphosis relationship, then the corresponding specific test cases must also satisfy the metamorphosis relationship. For example, the input model has only one parameter, and the corresponding value range of the parameter is {0, 1}. The equivalence class set corresponding to 0 is a negative number set, and the equivalence class set corresponding to 1 is a positive number set. The metamorphosis relationship is that the output of the positive input is greater than the output of the negative input. In this case, only two abstract test cases 0 and 1 are needed, and the corresponding specific test case pair will definitely satisfy the input relationship of the metamorphosis relationship.

[0056] 2) The metamorphosis relationship can only be satisfied by specific test cases, so it is not enough to just generate the corresponding abstract test cases. At this time, the metamorphosis relationship needs to be considered in both the abstract test case generation stage and the specific test case generation stage. For example, the input model has only one parameter, and the corresponding value range of this parameter is {0, 1}, 0 corresponds to the equivalence class set as a negative number set, and 1 corresponds to the equivalence class set as a positive number set. The metamorphosis relationship is f(x) = f(-x), where x is a non-zero rational number. In this case, it is not enough to consider the metamorphosis relationship only in the abstract test case stage. It is also necessary to ensure that the two specific test case input values ​​are opposite to each other in the specific test case stage, thereby forming a pair of test cases that match the metamorphosis relationship.

[0057] The test case generation phase includes the abstract test case generation phase and the concrete test case generation phase. In the abstract test case generation phase, a set of abstract test case sets that meet the t-dimensional coverage is generated, where each test case consists of a set of integers, and each integer represents one or a set of practical values ​​in the same equivalence class. In the concrete test case generation phase, the method converts the abstract test cases into concrete test cases, and replaces the abstract integer values ​​with specific meaningful values. Both phases take into account the metamorphosis relationship in order to generate more test case pairs that match the metamorphosis relationship.

[0058] Specifically, in the abstract test case generation stage, the adaptive random sampling method (ARS) is first used (reference: R. Huang, X. Xie, TY Chen, and Y. Lu, "Adaptive random test case generation for combinatorial testing," in 2012 IEEE 36 thAnnualComputer Software and Applications Conference.IEEE,2012,pp.52–61; C.Nie,H.Leung,and K.-Y.Cai,“Adaptive combinatorial testing,”in 2013 13thInternational Conference on Quality Software(QSIC).IEEE,2013,pp.284–287) to generate a set of initial abstract test case sets that meet t-dimensional coverage. Among them, ARS is a commonly used combinatorial test case generation algorithm, which can generate a set of coverage arrays as test case sets in a shorter time. Then, traverse all abstract test case pairs (t1,t2) in the initial abstract test case set. If (t1,t2) matches a certain transformation relationship, mark the abstract test cases t1 and t2. Finally, take the unmarked abstract test cases as the source abstract test cases and generate a subsequent abstract test case for them. Here, the subsequent abstract test case generation problem is converted into a constraint solving problem. Assume that the software to be tested has n abstract parameters, and the source abstract test case t s ={(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )},p i represents the i-th abstract parameter, v si Represents the i-th parameter p i The abstract value of , where 1≤i≤n; the transformation relationship is (r in ,r out ), r in That is, the input rule of the transformation relationship, r out That is, the output rule of the transformation relationship, if there are n test cases that satisfy r in Relationship, then these n outputs must satisfy r out relationship, otherwise the software has a fault; software constraint C = {c i |c i is the Boolean constraint function of the software to be tested}, and the subsequent abstract test case t f ={(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )},v fi Represents the i-th parameter p i Abstract value of, where 1≤i≤n; subsequent abstract test case t f It is expressed by the following formula:

[0059] Variable: {v f1 ,v f2 ,...,v fn}

[0060] Range: v fi ∈V i ,

[0061] Where V i represents the value range set of the i-th abstract parameter, that is, all parameters take values ​​within the value range;

[0062] Constraint: r in (v s1 ,v s2 ,...,v sn ,v f1 ,v f2 ,...,v fn )=true

[0063] c i (v f1 ,v f2 ,...,v fn )=true,

[0064] Here, the subsequent abstract test cases need to satisfy both the constraints of the software under test and the constraints of the metamorphosis relationship, that is, they can be matched with the source abstract test cases to form a pair of abstract test cases that satisfy the metamorphosis relationship. Here, the solver used is Choco (reference: C. Prud'homme, J.-G. Fages, and X. Lorca, Choco Solver Documentation, TASC, INRIA Rennes, LINA CNRS UMR 6241, COSLING SAS, 2016. [Online]. Available: http: / / www.chocosolver.org). Choco is a general constraint programming library for Java, which is built on an event-based propagation mechanism and has a traceable structure. In our method, all parameter values ​​and constraints between abstract test cases (source abstract test cases and subsequent abstract test cases) are encoded in arithmetic form, such as s1=f1, s1>f1, etc. Then, if the constraints are satisfiable and computing resources are available, the solver will produce a solution. Choco itself guarantees the correctness of subsequent abstract test case generation, and a stable version of Choco (4.10.2) is used in the experiment. Finally, the initial abstract test case set and the generated subsequent abstract test case set are combined into a complete abstract test case set.

[0065] In order to truly test the system, it is necessary to convert abstract test cases into concrete test cases. First, traverse all abstract test case pairs (t1, t2) in the abstract test case set. If (t1, t2) matches a certain metamorphosis relationship, map it to a concrete test case pair that matches the metamorphosis relationship. Taking the sin function as an example, set the abstract model to have only one parameter, and the parameter value range includes three equivalence classes, negative, 0, and positive, and the metamorphosis relationship is sin(x) = -sin(-x). If there are two abstract test cases, and their abstract parameter values ​​are negative and positive, first randomly select a value for the first abstract test case in the equivalence class, set the mapping specific value to -23, and then combine the metamorphosis relationship to map the second abstract test case to a concrete test case with a value of 23, thereby generating a pair of concrete test cases that satisfy the metamorphosis relationship. After completing the mapping of the abstract test cases that match the metamorphosis relationship, it is also necessary to generate corresponding concrete test cases for the remaining abstract test cases in the abstract test case set, that is, select an actual value from its corresponding equivalence class for each abstract value.

[0066] After obtaining specific test cases, write a test script, use the test input to execute the system under test, and record the actual output of each test case. Unlike traditional combination testing, it does not determine the correctness of the program through assertions, but writes an automatic script to automatically determine the correctness of the program by judging whether the test case pairs matching the metamorphosis relationship meet the output rules of the metamorphosis relationship. If the output rules are not met, then it is considered that the program has a fault. Finally, the specific test case pairs that violate the metamorphosis relationship will be output.

[0067] Nouns related to this method:

[0068] Test case: Assume that the system under test is affected by n parameters, each parameter p i There is a set of possible values ​​V i , where 1≤i≤n. For the system under test, a test case in the combination test is a specific assignment sequence of n parameters, that is, from V i to p i The assignment of , where 1≤i≤n, can be represented as the set {(p1,v1), (p2,v2), …, (p n ,v n )}.

[0069] t-dimensional tuple: For the system to be tested, a tuple formed by assigning specific values ​​to t parameters is called a t-value tuple, where 1≤t≤n. It is referred to as a tuple to avoid confusion and can be expressed as {…, (p l1 ,v l1 ),…,(p lt ,v lt), …}, except for these t parameters, the values ​​of other parameters are to be determined. It should be noted that the t-dimensional tuple can be regarded as a t-dimensional subset of the corresponding test case, and the test case itself can be regarded as an n-dimensional tuple. The key target of combinatorial testing is the tuple, because many faults are caused by parameter combinations rather than single parameters.

[0070] Test case set that satisfies t-dimensional coverage: A test case set that satisfies t-dimensional coverage is a set of test cases in which each possible t-dimensional tuple appears in at least one test case. The t-dimensional tuple here refers to all possible values ​​of the t-dimensional tuple.

[0071] Transformation relation: Given a system to be tested, let x1, x2, ..., x n is a set of test cases, o1, o2, ..., o n is the corresponding output of these n test cases, where n ≥ 2. in ,r out ) declares the fact that if these n test cases satisfy r in Relationship, i.e. r in (x1,x2,…,x n )=true, then these n outputs must satisfy r out Relationship, i.e. r out (o1,o2,…,o n )=true. Formally expressed as r in (x1,x2,…,x n )→r out (o1,o2,…,o n ).

[0072] Source test cases and subsequent test cases: Given a system to be tested, the transformation relationship is (r in ,r out ). Suppose there are k test cases x1, x2, …, x k , if there are additional nk test cases x k+1 ,x k+2 , …, x n Make r in (x1,x2,…,x k , …, x n )=true, then these k test cases x1, x2, ..., x k is called source test case, additional nk test cases x k+1 ,x k+2 , …, x nIt is called a subsequent test case. In most cases, k is 1 and n is 2. For the sake of simplicity, the metamorphosis relationship used in the present invention is also this type of metamorphosis relationship, that is, there is only one source test case and only one subsequent test case. In addition, in the abstract test case stage, the source test case and the subsequent test case can be called the source abstract test case and the subsequent abstract test case; in the specific test case stage, the source test case and the subsequent test case can be called the source specific test case and the subsequent specific test case.

[0073] Table 1 shows the average data of 30 Maxsub experiments, and Table 2 shows the average data of 30 Editingdistance experiments:

[0074] Table 1

[0075]

[0076] Table 2

[0077]

[0078] Tables 1 and 2 show the two-dimensional experimental data of two software Maxsub and Editingdistance (references: H. Jin, Y. Jiang, N. Liu, C. Xu, X. Ma, and J. Lu, "Concolic metamorphic debugging," in 2015 IEEE 39th Annual Computer Software and Applications Conference, vol. 2, 2015, pp. 232–241; A. Carzaniga, A. Goffi, A. Gorla, A. Mattavelli, and M. Pezze, `"Cross-checking oracles from intrinsic software redundancy," in Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 931–942) using the traditional combinatorial testing method and the metamorphic relationship matching combinatorial testing method. To avoid contingency, the statistics here are the average data of 30 repeated experiments. The main statistical three metrics are the number of test cases generated, the number of test case pairs matching metamorphic relationships, and the number of fault detections. Compared with the traditional combination test, the new method increases the number of test cases generated by the two software by 62.41% and 66.49% respectively. As can be seen from the table, the number of test case pairs matching the metamorphosis relationship generated by the two software under test in the traditional combination test scenario is 0. This is because it is difficult to generate test case pairs that satisfy the metamorphosis relationship in the traditional combination test scenario, which makes it impossible to use the metamorphosis relationship to detect whether there is a fault, so the number of fault detections is also 0. When the new method is used, the number of test case pairs matching the metamorphosis relationship generated by the two software under test in the 2D scenario is 9.1 and 17.4 respectively. Here, the mutation tool is used to implant faults into the two software, implanting 6 and 14 faults respectively, and then automatically determine the correctness of the software under test by verifying whether the output of the test case pair matching the metamorphosis relationship violates the output relationship of the metamorphosis relationship, and finally detects 2.0 and 5.3 faults respectively.

[0079] Compared with the traditional combination test, the new method proposed by the present invention generates more test case pairs matching the metamorphosis relationship, that is, provides more opportunities to detect faults. In addition, the Oracle of the new method automatically determines the correctness of the software to be tested by judging whether the output of the test case pair matching the metamorphosis relationship violates the output relationship of the metamorphosis relationship, thereby improving the fault detection rate and better alleviating the automatic Oracle problem in the combination test.

[0080] Example

[0081] The combined testing method based on transformation relationship matching takes the software Maxsub to be tested as an example. The specific implementation steps are as follows:

[0082] (1) Construct an input parameter model. Maxsub inputs an integer array and outputs the maximum sum of the subarrays. Here, the array contains both positive and negative numbers. One or more consecutive integers in the array form subarrays. Each subarray has a sum, and the maximum value of the subarray sum is output. The input parameter model is shown in Table 3.

[0083] Table 3

[0084]

[0085] In this table, the abstract value and the corresponding concrete value of each parameter of the software to be tested are given. The basic idea of ​​this model is to consider the number of positive numbers, negative numbers, zeros, the number of identical numbers and the order of numbers in the array, and use different array structures to test the program.

[0086] (2) Determine the transformation relationship. Here, the transformation relationship of Maxsub is that the input array is in reverse order, and the output remains unchanged. Then, the transformation relationship is divided into two parts, considering abstract test cases and specific test cases respectively, as shown in Table 4. In Table 4, in the abstract test case stage, v si Represents the abstract value of the i-th parameter of the source abstract test case, v fi Indicates the abstract value of the i-th parameter of the subsequent abstract test case, where 1≤i≤5. In the specific test case stage, c i Represents the i-th number in the source specific test case input array and the i-th number from the end of the subsequent specific test case input array, where 1≤i≤n, 1≤n≤90.

[0087] Table 4

[0088]

[0089] In the abstract test case generation phase, in order to ensure that the source abstract test case input array and the subsequent abstract test case input array are in reverse order, the first four parameters of the two abstract test cases are constrained to be the same, and the fifth parameter sums to 1 (the fifth parameters of the two test cases are 0 and 1, respectively, i.e., in positive order and in reverse order). Of course, it is not enough to only consider the abstract test case phase, and it is also necessary to constrain it in the specific test case generation phase, that is, constrain the subsequent specific test case input array F to be the reverse order of the source specific test case input array S.

[0090] (3) Generate a set of abstract test cases. Figure 2In the source abstract test case generation stage, the current initial abstract test case set is set to have only one test case {0,0,0,0,0}, and then a group of test cases is randomly generated, and the randomly generated test case set is set to {{1,0,0,0,0}, {1,1,1,1,1}}. Calculation shows that the test case {1,1,1,1,1} covers more uncovered tuples, so the test case {1,1,1,1,1} is added to the initial abstract test case set, and the above process is repeated until the number of uncovered tuples is equal to 0, that is, the current initial abstract test case set covers all t-dimensional combinations.

[0091] For the abstract test case t1 in the initial abstract test case set, two cases are considered. In the first case, there is an abstract test case t2 in the initial abstract test case set, so that the test case pair (t1, t2) matches the metamorphosis relationship. In this case, there is no need to generate subsequent abstract test cases for t1 and t2. In the second case, there is no abstract test case t2 in the initial abstract test case set, so that the test case pair (t1, t2) matches the metamorphosis relationship. In this case, it is necessary to solve a subsequent abstract test case for it in combination with the metamorphosis relationship. Set the current traversal source abstract test case to {0,0,0,0,0}, and combine the metamorphosis relationship in Table 4 to generate a subsequent abstract test case {0,0,0,0,1} for it. It should be noted that not every source abstract test case can generate a corresponding subsequent abstract test case. For example, the abstract test case {0,0,0,0,2} does not satisfy the constraints of the metamorphosis relationship on the source abstract test case, but this does not affect the method results. In the worst case, this method can also degenerate into a traditional combination test method.

[0092] Finally, the source abstract test case set and the subsequent abstract test case set are combined into a complete abstract test case set.

[0093] (3) Generate a specific test case set. Figure 3 The conversion process from abstract test cases to specific test cases is shown. For the abstract test case pairs t1 and t2 that match the metamorphosis relationship, the corresponding specific test cases must meet two conditions. First, the specific value corresponds to the abstract value. Set the first parameter value of the abstract test case t1 to 1, then the number of negative numbers in the specific test case input array should be in the range [1, 20). Second, the specific test case should satisfy the metamorphosis relationship listed in Table 4. Set the specific test case input array corresponding to t1 to {1, 2, 3}, then the specific test case input of t2 should be {3, 2, 1}, thereby obtaining a pair of test cases with a matching metamorphosis relationship. When the abstract test case cannot form a test case pair with a matching metamorphosis relationship with other test cases, only a specific test case corresponding to the abstract value will be generated for it.

[0094] (4) Use the test input to execute the system under test. After obtaining a specific test case set, write a test script, use the test input to execute the system under test, and record the actual output of each test case. Set the test case input to {1,2,3}, use this input to execute the software under test Maxsub, and get the actual output of 6, then you have a complete test case that includes input and output.

[0095] (5) Use the metamorphosis relationship as an Oracle to determine whether the program has a fault. After obtaining a complete set of test cases containing inputs and outputs, write an automatic script to automatically determine the correctness of the program by determining whether the test case pairs matching the metamorphosis relationship satisfy the output rules of the metamorphosis relationship. If the output rules are not satisfied, then the program is considered to have a fault. Assume that the input of test case A is {1,2,3} and the output is 6; the input of test case B is {3,2,1} and the output is -6. When the script traverses this pair of test cases, it is found that the input satisfies the input rules of the metamorphosis relationship but the output does not satisfy the output rules of the metamorphosis relationship, then the program is considered to have a fault. Finally, the specific test case pairs that violate the metamorphosis relationship are output. It should be noted that satisfying the output rules of the metamorphosis relationship does not guarantee that the program has no faults, because this is not a full test.

[0096] The present invention provides a combined test method based on metamorphosis relationship matching. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A combined testing method based on metamorphosis relationship matching, characterized in that: The steps include: Step 1, input the abstract parameter model and the transformation relationship, and transform the transformation relationship into constraints on the model; Step 2, generate an abstract test case set; Step 3, generate a specific test case set; Step 4: Use the test cases to execute the software under test and use the script to automatically detect whether the program fails; Step 1 includes the following steps: Step 1-1, constructing an abstract parameter model of the combination test of the software to be tested, including parameters, the value range and constraints of each parameter, and determining the transformation relationship of the software to be tested, the input rules and output rules of the transformation relationship; Step 1-1 includes: setting the software to be tested to have n parameters, the i-th parameter p i The finite range of V i , where 1≤i≤n; for each parameter p i , from the set V i Take the abstract value v i Assigned to parameter p i , forming an abstract test case of the combined test, namely {(p1,v1),(p2,v2),...,(p n ,v n )}, where the abstract value v i It may correspond to an actual value, or to an equivalence class; For each abstract value v in the abstract test case i , converting it into a concrete value c with real meaning i , by c i A specific test case that constitutes the combined test, namely {(p1,c1),(p2,c2),...,(p n ,c n )}; Step 1-1 includes: converting the transformation relationship into constraints on the combination test model, including two cases: In the first case, the metamorphosis relationship can be satisfied by the abstract test case. That is, if two abstract test cases satisfy the metamorphosis relationship, then the specific test cases corresponding to the two abstract test cases must also satisfy the metamorphosis relationship. In this case, the metamorphosis relationship only needs to be considered in the abstract test case generation stage. Set the source abstract test case to {(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )},p i represents the i-th abstract parameter, v si Represents the i-th parameter p i Abstract value of, where 1≤i≤n; the subsequent abstract test case is {(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )},v fi Represents the i-th parameter p i The abstract value of the transformation relation; the input relation r in It is expressed by the following formula: r in (v s1 ,v s2 ,...,v sn ,v f1 ,v f2 ,...,v fn )=true The above formula indicates that the values ​​of the two abstract test cases satisfy the input relation r in For constraints on abstract test case pairs, true represents the truth value; In the second case, the metamorphosis relationship can only be satisfied by specific test cases. It is not enough to generate the corresponding abstract test cases. In this case, the metamorphosis relationship needs to be considered in both the abstract test case generation stage and the specific test case generation stage. The source abstract test case is set to {(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )}, the corresponding source specific test case is {(p1,c s1 ),(p2,c s2 ),...,(p n ,c sn )},c si Indicates that the i-th parameter of the source specific test case corresponds to a specific value, where 1≤i≤n; the subsequent abstract test case is {(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )}, the corresponding subsequent specific test case is {(p1,c f1 ),(p2,c f2 ),...,(p n ,c fn )},c fi Indicates the specific value corresponding to the i-th parameter of the subsequent specific test case, where 1≤i≤n; the input relationship of the transformation relationship r in It is expressed by the following formula: r in (v s1 ,v s2 ,...,v sn ,v f1 ,v f2 ,...,v fn )=true r in (c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn )=true The above formula indicates that the values ​​of the two abstract test cases satisfy the input relation r in For the constraints of the abstract test case pair, the values ​​of the two specific test cases satisfy the input relation r in Constraints on specific test case pairs; The output relationship of the transformation relationship is not considered in the test case generation stage. The output relationship of the transformation relationship is considered only when judging whether the program has a fault. The source test case output is set as output s , the subsequent test case output is output f , then the output relation of the transformation relation r out It is expressed by the following formula: r out (output s ,output f )=true If the test case that satisfies the transformation relationship input relationship has an output relationship that does not satisfy the transformation relationship, it is determined that the program has a fault.

2. The method according to claim 1, characterized in that Step 2 includes the following steps: Step 2-1, first use the adaptive random sampling method to generate a set of initial abstract test case sets that meet the t-dimensional coverage rate: each time a set of candidate abstract test case sets is randomly generated, the number of uncovered combinations covered by each candidate abstract test case is calculated, and the abstract test case with the largest number of uncovered combinations is added to the result set, and the above process is repeated until the number of uncovered combinations is equal to 0, that is, the initial abstract test case set meets the t-dimensional coverage rate; Step 2-2, traverse all abstract test case pairs (t1, t2) in the initial abstract test case set, and if (t1, t2) matches a transformation relationship, mark the abstract test cases t1 and t2; Step 2-3, take the unlabeled abstract test case as the source abstract test case, and generate a subsequent abstract test case for the source abstract test case, transform the subsequent abstract test case generation problem into a constraint solving problem, assume that the software to be tested has n abstract parameters, and the source abstract test case t s ={(p1,v s1 ),(p2,v s2 ),...,(p n ,v sn )},p i represents the i-th abstract parameter, v si Represents the i-th parameter p of the source abstract test case i The abstract value of , where 1≤i≤n; the transformation relationship is (r in ,r out ), r in That is, the input rule of the transformation relationship, r out That is, the output rule of the transformation relationship, if there are n test cases that satisfy r in Relationship, then these n outputs must satisfy r out relationship, otherwise the software has a fault; software constraint C = {c i |c i is the Boolean constraint function of the software to be tested}, and the subsequent abstract test case t f ={(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )},v fi Indicates the i-th parameter p of the subsequent abstract test case i Abstract value of, where 1≤i≤n; subsequent abstract test case t f It is expressed by the following formula: Variable: {v f1 ,v f2 ,...,v fn } range: Where V i represents the value range set of the i-th abstract parameter, that is, all parameters take values ​​within the value range; Constraint: r in (v s1 ,v s2 ,...,v sn ,v f1 ,v f2 ,...,v fn )=true Combine the initial abstract test case set and the generated subsequent abstract test case set into a complete abstract test case set.

3. The method according to claim 2, characterized in that Step 3 includes the following steps: Step 3-1, traverse all abstract test case pairs (t1, t2) in the abstract test case set. If (t1, t2) matches a transformation relationship, map it to a specific test case pair that matches the transformation relationship. Set (t1, t2) to match a transformation relationship, t1 = {(p1, v s1 ),(p2,v s2 ),...,(p n ,v sn )}, the specific test case corresponding to t1 is {(p1,c s1 ),(p2,c s2 ),...,(p n ,c sn )}, t2={(p1,v f1 ),(p2,v f2 ),...,(p n ,v fn )}, the specific test case corresponding to t2 is {(p1,c f1 ),(p2,c f2 ),...,(p n ,c fn )}, software constraint C = {c i |c i is the Boolean constraint function of the software to be tested}; the transformation relationship is (r in ,r out ), r in That is, the input rule of the transformation relationship, r out That is, the output rule of the transformation relationship; then the specific test cases corresponding to t1 and t2 are expressed by the following formula: Variable: {c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn } Range: c si ∈v si The corresponding equivalence class set, c fi ∈v fi The corresponding equivalence class set, Among them, the size of the equivalence class set represented by the abstract value is at least 1, and all concrete values ​​should belong to the equivalence class set corresponding to their abstract value; Constraint: r in (c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn )=true According to the above formula, a pair of specific test case pairs that satisfy the metamorphosis relationship is generated; if the metamorphosis relationship is a first-class metamorphosis relationship, that is, a metamorphosis relationship that can be satisfied by an abstract test case, then as long as the abstract test case pair satisfies the metamorphosis relationship, its corresponding specific test case pair will definitely satisfy the metamorphosis relationship, r in (c s1 ,c s2 ,...,c sn ,c f1 ,c f2 ,...,c fn )=true is always true, and the above formula can still be used to represent the specific test case pair; Step 3-2, after completing the mapping of the abstract test cases matching the metamorphosis relationship, generate corresponding concrete test cases for the remaining abstract test cases in the abstract test case set, that is, select an actual value from its corresponding equivalence class for each abstract value.

4. The method according to claim 3, characterized in that Step 4 includes: writing a test script, executing the system to be tested using the test input, and recording the actual output of each test case, automatically determining the correctness of the program by judging whether the test case pair matching the metamorphosis relationship satisfies the output rule of the metamorphosis relationship, and if the output rule is not satisfied, it is determined that the program has a fault.

5. The method according to claim 4, characterized in that In step 4, traverse any two test cases in the test case set to determine whether they satisfy the input relationship of the metamorphosis relationship. If they do, determine whether they satisfy the output rule of the metamorphosis relationship. If they do not satisfy the output rule, mark the pair of test cases, that is, the program fails.

Images