An efsmt test case ordering method based on multiple incremental regression trees

Abstract

The application discloses an EFSM test case sorting method based on a multiple incremental regression tree, which comprises three stages of extracting a similarity index subset, training and application; in the stage of extracting the similarity index subset, test cases are ranked according to similarity indexes, ranking information is added to a similarity index sorting set, the similarity index sorting set is used as original data for feature selection, and a whale optimization algorithm is used to screen the similarity indexes as training indexes; in the training stage, label values of test cases in a test case set TS are extracted to obtain a label value set, feature values of the test cases are extracted according to the screened indexes to obtain a feature set, a training set is constructed, the training set is applied to train the multiple incremental regression tree, and a sorting model is trained; in the application stage, the constructed test set data is input into the sorting model, and a sorting result is output, and test cases with higher scores are arranged in front. The application can obviously improve the average error detection rate of the test case set.

Description

Technical Field

[0001] This invention belongs to the field of model-based test case ranking, specifically involving an EFSM test case ranking method based on multiple incremental regression trees. Background Technology

[0002] Regression testing is a crucial part of software testing for ensuring software quality. The purpose of regression testing is to determine whether modifications to the software introduce new faults. However, achieving a high fault detection rate requires significant effort. Therefore, much research has been conducted to improve the fault detection capabilities of regression testing. To date, most test case ranking methods are based on source code, such as ranking test cases based on statement coverage or function coverage. However, as software complexity increases, the size of the source code grows, leading to greater overhead for code-based testing. To reduce the cost of regression testing, model-based test case ranking methods have been introduced. Since a model is an abstract description of software behavior, model-based methods are simpler and more efficient than code-based methods.

[0003] There are many important standardized technical models in industry, such as Extended Finite State Machine (EFSM), Finite State Machine (FSM), and Unified Modeling Language (UML). EFSM, in particular, considers both the data and control components of software, which is highly advantageous for capturing software behavior and structure. An EFSM is a behavioral model consisting of a finite number of states and state transitions. It can be described using a finite set of input / output parameters, state transition functions, and output functions, encompassing both control and data components. Currently, a large amount of research is based on EFSM, including EFSM test sequence generation, test data generation, and test sorting methods. Among these, there is considerable research on the sorting of EFSM test cases, such as sorting based on Euclidean distance, Hamming distance, Manhattan distance, and Jakeir coefficient. Previous sorting methods generally only used one aspect of EFSM, thus offering limited performance improvements. Therefore, how to utilize multiple aspects of EFSM for sorting is a hot research topic.

[0004] Learning to Rank (LTR) is a technique that uses machine learning algorithms to generate ranking models, applicable to various fields such as data retrieval, natural language processing, and fault localization. LTR consists of multiple algorithms, with multiple incremental regression trees being one of them. If a novel EFSM test case ranking method can be proposed using multiple incremental regression trees combined with multiple similarity metrics, it is expected to improve fault detection rates.

[0005] However, determining which similarity index combinations to use for optimal fault detection remains a research question. Ideally, an index combination should be complementary to ensure it provides more information. Summary of the Invention

[0006] To address the above background and problems, this invention provides an EFSM test case ranking method based on multiple incremental regression trees.

[0007] The specific steps of this invention are as follows:

[0008] (1) Sort the test cases in the test case set TS according to the similarity index to obtain the similarity index ranking set. Use the whale optimization algorithm to select features from the obtained similarity index ranking set, and use the selected similarity indexes to construct a similarity index subset.

[0009] (2) Extract the label values ​​of the test cases in the test case set TS to obtain the label value set; extract the feature values ​​of the test cases in the test case set TS according to the similarity index subset to obtain the feature set. The label value set and the feature set together constitute the training set.

[0010] (3) Use the training set to train a multiple incremental regression tree to obtain a test case ranking model.

[0011] (4) Input the feature set from step (2) as a separate test set into the test case ranking model to obtain the final ranking result.

[0012] Preferably, step (1) is as follows: The test cases in the test case set TS are sorted according to 16 similarity indicators, and the sorting results are used as data in the similarity indicator ranking set to obtain the similarity indicator ranking set. The 16 similarity indicators are Manhattan distance, Euclidean distance, Chebyshev distance, Canberra distance, Brectis distance, Hemingway distance, Levenstein distance, cosine similarity, Jaccard distance, Pearson distance, chi-square distance, Bach distance, Hellinger distance, Jensen-Shannon distance, cross-entropy, and relative entropy.

[0013] Preferably, the process of extracting label values ​​to obtain the label value set in step (2) is as follows: ① Inject some known errors into the source code of EFSM, let n=1, execute the test case set TS once to obtain the source code errors found by each test case; ② Initialize the label value set label; ③ Find the test case t in the test case set TS that can find the most source code errors, set its label value to m-n+1, ​​and update the label value set label with the label value of test case t, where m is the total number of test cases in the test case set TS; then delete test case t from the test case set TS, and delete the source code errors found by test case t from the error list; ④ Determine whether the test case set TS is empty. If the test case set TS is not empty, increase the value of n by 1 and repeat step ③. If the test case set TS is empty, output the label value set label.

[0014] Preferably, the process of extracting feature values ​​to obtain the feature set in step (2) is as follows: First, initialize the feature set feature; then sort the test cases in the test case set TS according to each similarity index in the similarity index subset, reverse and normalize each sorting result; finally, use each normalized sorted data as a feature value to update the feature set feature.

[0015] More preferably, the sorting result reversal operation is as follows: Suppose the total number of test cases in the test case set TS is k1, and the sorting result of a certain test case is k2, then the reversal operation result is k1-k2+1.

[0016] Preferably, step (3) is as follows:

[0017] 3.1 Assign the same initial weights to all training samples in the training set and initialize a training model. Here, a training sample is the ranking of a certain label value under various similarity metrics within the training set.

[0018] 3.2 Using the true values ​​of the training samples as the initial gradient values, the current weight of each training sample is calculated using an exponential loss function with the gradient value as the base.

[0019] 3.3 Train a regression tree using the current weights of each training sample, with the goal of minimizing the average gradient value of all training samples at each node of the regression tree.

[0020] 3.4 Add the newly generated regression tree to the current training model and update the predicted values ​​of the current training model; use the residual between the predicted values ​​of the current training model and the true values ​​of the corresponding training samples in the training set as the gradient values, and calculate the current weights of each training sample based on the gradient values ​​of each training sample using an exponential loss function with the gradient values ​​as the base.

[0021] 3.5 Repeat steps 3.3 and 3.4, and continue to train multiple regression trees iteratively, adding them to the current training model. When the predetermined number of iterations is reached, stop the training process and obtain the final test case ranking model.

[0022] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention analyzes the correlation between features of different EFSM test cases using multiple weighted regression trees, enabling the sorted test case set (TS) to detect potential errors in the EFSM source code more quickly. This invention transforms the EFSM test case sorting problem into a classification problem in machine learning, using a specific learning and sorting algorithm—multiple weighted regression trees—to analyze the multidimensional features of test cases and train a test case sorting model. Compared to existing single-index sorting algorithms, this significantly improves the average error detection rate of the test case set. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the framework and application of the present invention.

[0024] Figure 2 This is a flowchart of the similarity index subset selection process in this invention.

[0025] Figure 3 This is the overall flowchart of the present invention.

[0026] Figure 4 Box plots of APFD values ​​for 16 existing sorting methods and the sorting method of this invention on the M1 model.

[0027] Figure 5 Box plots of APFD values ​​for 16 existing sorting methods and the sorting method of this invention on the M2 model.

[0028] Figure 6 Box plots of APFD values ​​for 16 existing sorting methods and the sorting method of this invention on the M3 model.

[0029] Figure 7 Box plots of APFD values ​​for 16 existing sorting methods and the sorting method of this invention on the M4 model.

[0030] Figure 8 Box plots of APFD values ​​for 16 existing sorting methods and the sorting method of this invention on the M5 model. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] like Figure 1 As shown, an EFSM test case ranking method based on multiple incremental regression trees includes three main stages: extracting a subset of similarity metrics, training, and application. In the subset extraction stage, test cases are ranked according to similarity metrics, and the ranking information is added to a similarity metric ranking set. This ranking set serves as the raw data for feature selection. The whale optimization algorithm is used to effectively identify more complementary similarity metrics, which are then selected as training metrics. In the training stage, the label values ​​of test cases in the test case set TS are extracted to obtain a label value set. Feature values ​​are extracted from the test cases according to the selected metrics to obtain a feature set, and a training set is constructed. This training set is then input into a multiple incremental regression tree to train a ranking model for the specific EFSM test cases. In the application stage, the constructed test set data is input into the trained test case ranking model, and the ranking results are output, with test cases scoring higher ranked higher.

[0033] like Figure 2 and Figure 3 As shown, an EFSM test case ranking method based on multiple incremental regression trees specifically includes the following steps:

[0034] (1) Sort the test cases in the test case set TS according to the similarity index to obtain the similarity index ranking set. Use the whale optimization algorithm to select features from the obtained similarity index ranking set, and use the selected similarity indexes to construct a similarity index subset. Among them, the whale optimization algorithm can select similarity indexes with low redundancy and high complementarity.

[0035] (2) Extract the label values ​​of the test cases in the test case set TS to obtain the label value set; extract the feature values ​​of the test cases in the test case set TS according to the similarity index subset to obtain the feature set. The label value set and the feature set together constitute the training set.

[0036] (3) Use the training set to train a multiple incremental regression tree to obtain a test case ranking model.

[0037] (4) Input the feature set from step (2) as a separate test set into the test case ranking model to obtain the final ranking result.

[0038] As a preferred embodiment, step (1) is as follows: The test cases in the test case set TS are sorted according to 16 similarity indicators, and the sorting results are used as data in the similarity indicator ranking set to obtain the similarity indicator ranking set. The 16 similarity indicators are Manhattan distance, Euclidean distance, Chebyshev distance, Canberra distance, Brectis distance, Heming distance, Levenstein distance, cosine similarity, Jaccard distance, Pearson distance, Chi-square distance, Bach distance, Hellinger distance, Jensen-Shannon distance, cross-entropy, and relative entropy.

[0039] As a preferred embodiment, the process of extracting label values ​​to obtain the label value set in step (2) is as follows: ① Inject some known errors into the source code of EFSM, let n=1, execute the test case set TS once to obtain the source code errors found by each test case; ② Initialize the label value set label; ③ Find the test case t in the test case set TS that can find the most source code errors, set its label value to m-n+1, ​​and update the label value set label with the label value of test case t, where m is the total number of test cases in the test case set TS; then delete test case t from the test case set TS, and delete the source code errors found by test case t from the error list; ④ Determine whether the test case set TS is empty. If the test case set TS is not empty, increase the value of n by 1 and repeat step ③. If the test case set TS is empty, output the label value set label.

[0040] As a preferred embodiment, the process of extracting feature values ​​to obtain the feature set in step (2) is as follows: First, initialize the feature set feature; then sort the test cases in the test case set TS according to each similarity index in the similarity index subset, reverse and normalize each sorting result; finally, use each normalized sorted data as a feature value to update the feature set feature.

[0041] More preferably, the sorting result reversal operation is as follows: Suppose the total number of test cases in the test case set TS is k1, and the sorting result of a certain test case is k2, then the reversal operation result is k1-k2+1.

[0042] In a preferred embodiment, step (3) is specifically as follows:

[0043] 3.1 Assign the same initial weights to all training samples in the training set and initialize a training model. Here, a training sample is the ranking of a certain label value under various similarity metrics within the training set.

[0044] 3.2 Using the true values ​​of the training samples as the initial gradient values, the current weight of each training sample is calculated using an exponential loss function with the gradient value as the base.

[0045] 3.3 Train a regression tree using the current weights of each training sample, with the goal of minimizing the average gradient value of all training samples at each node of the regression tree.

[0046] 3.4 Add the newly generated regression tree to the current training model and update the predicted values ​​of the current training model; use the residual between the predicted values ​​of the current training model and the true values ​​of the corresponding training samples in the training set as the gradient values, and calculate the current weights of each training sample based on the gradient values ​​of each training sample using an exponential loss function with the gradient values ​​as the base.

[0047] 3.5 Repeat steps 3.3 and 3.4, and continue to train multiple regression trees iteratively, adding them to the current training model. When the predetermined number of iterations is reached, stop the training process and obtain the final test case ranking model.

[0048] During the training process, a new regression tree is generated in each iteration and integrated into the current model, thereby gradually improving the predictive ability of the trained model. By integrating multiple regression trees, overfitting can be effectively avoided, and good performance can be achieved in prediction accuracy.

[0049] In step (4), the test set data is input into the test case ranking model trained in step (3). The output ranking result is a set of real numbers. Each real number in the set represents the score of the corresponding test case. The test cases with higher scores are ranked higher.

[0050] To verify the feasibility and effectiveness of the method of this invention, a detailed experimental analysis was conducted based on five benchmark protocol EFSM test models (software models). These five EFSM test models are Network Monitor (denoted as M1), Inres initiator (denoted as M2), Class2 (denoted as M3), OLSR (denoted as M4), and SCP (denoted as M5). Specific model details are shown in Table 1.

[0051] Table 1 Five EFSM test models

[0052]

[0053] This invention uses the Average Error Detection Rate (APFD) to measure the performance of the ranking method. The APFD value ranges from 0 to 1; a larger APFD value indicates a higher average error detection rate for the reordered test case set. APFD can be formally defined as follows:

[0054]

[0055] Let T represent a test set with n test cases, and F represent the set of errors that T can find. In the above formula, TF... k This represents the sequence number of the first test case that discovers the k-th error in the reordered test set, where k ranges from 1 to m.

[0056] Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 Box plots of APFD values ​​for the proposed method on five EFSM test models were used to compare the results of 16 existing ranking methods and the proposed method using only one of the 16 similarity indices S1-S16. The results show that the proposed method achieves a higher minimum and average APFD value on each EFSM test model than any of the other 16 ranking methods. Based on the minimum, maximum, and average values, it can be concluded that the proposed method (MART) outperforms any existing ranking method using a single similarity index.

[0057] This invention employs a multiple incremental regression tree algorithm to analyze the correlation between features of different EFSM test cases, enabling faster detection of potential errors in the EFSM source code. This invention transforms the test case ranking problem into a classification problem and uses the multiple incremental regression tree algorithm to analyze the multidimensional features of test cases to train a test case ranking model. Experimental results show that, compared to single-index ranking algorithms, the proposed test case ranking model significantly improves the average error detection rate of the test case set.

[0058] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics thereof. The scope of the invention is defined by the appended claims rather than the foregoing description.

Claims

1. A test case ranking method based on multiple incremental regression trees in EFSM, characterized in that: The specific steps are as follows: (1) Sort the test cases in the test case set TS according to the similarity index to obtain the similarity index sorting set; use the whale optimization algorithm to select features from the obtained similarity index sorting set, and use the selected similarity index to construct a similarity index subset; (2) Extract the label values ​​of the test cases in the test case set TS to obtain the label value set; extract the feature values ​​of the test cases in the test case set TS according to the similarity index subset to obtain the feature set. The label value set and the feature set together constitute the training set. (3) Use the training set to train a multiple incremental regression tree to obtain a test case ranking model; (4) Input the feature set from step (2) as a separate test set into the test case ranking model to obtain the final ranking result; The specific process of extracting tag values ​​to obtain the tag value set in step (2) is as follows: ① Inject some known errors into the source code of EFSM, let n=1, execute the test case set TS completely once, and obtain the source code errors found by each test case; ② Initialize the label value set; ③ Find the test case t in the test case set TS that can find the most source code errors, set its label value to m-n+1, ​​and update the label value set label with the label value of test case t, where m is the total number of test cases in the test case set TS; Then remove test case t from the test case set TS and remove the source code error found by test case t from the error list; ④ Determine if the test case set TS is empty. If the test case set TS is not empty, increment the value of n by 1 and repeat step ③. If the test case set TS is empty, output the label value set label. The specific steps (3) are as follows: 3.1 Assign the same initial weights to each training sample in the training set and initialize a training model; where the training sample is the ranking of a certain label value in the training set under various similarity indicators. 3.2 Using the true values ​​of the training samples as the initial gradient values, the current weight of each training sample is calculated using an exponential loss function with the gradient value as the base. 3.3 Train a regression tree using the current weights of each training sample, with the goal of minimizing the average gradient value of all training samples at each node of the regression tree; 3.4 Add the newly generated regression tree to the current training model and update the predicted value of the current training model; use the residual between the predicted value of the current training model and the true value of the corresponding training sample in the training set as the gradient value, and calculate the current weight of each training sample according to the gradient value of each training sample through the exponential loss function with the gradient value as the exponential base. 3.5 Repeat steps 3.3 and 3.4, and continue to train multiple regression trees iteratively, adding them to the current training model. When the predetermined number of iterations is reached, stop the training process and obtain the final test case ranking model.

2. The EFSM test case ranking method based on multiple incremental regression trees according to claim 1, characterized in that: The specific steps (1) are as follows: The test cases in the test case set TS are sorted according to 16 similarity indicators, and the sorting results are used as the data in the similarity indicator sorting set to obtain the similarity indicator sorting set; the 16 similarity indicators are Manhattan distance, Euclidean distance, Chebyshev distance, Canberra distance, Brectis distance, Heming distance, Levenstein distance, cosine similarity, Jaccard distance, Pearson distance, Chi-square distance, Bach distance, Hellinger distance, Jensen-Shannon distance, cross entropy and relative entropy.

3. The EFSM test case ranking method based on multiple incremental regression trees according to claim 1, characterized in that: The process of extracting feature values ​​to obtain the feature set in step (2) is as follows: First, initialize the feature set feature; then sort the test cases in the test case set TS according to each similarity index in the similarity index subset, reverse and normalize each sorting result; finally, use each normalized sorted data as a feature value to update the feature set feature.

4. The EFSM test case ranking method based on multiple incremental regression trees according to claim 3, characterized in that: The specific steps for reversing the sorting result are as follows: Suppose the total number of test cases in the test case set TS is k1, and the sorting result of a certain test case is k2, then the result of the reversal operation is k1-k2+1.

Images