Semantic representation-based accidental correctness test detection method

Through the detection method based on semantic representation, semantic features of test code and focus method are extracted and hierarchical network is constructed, the problem of insufficient semantic representation of code in the existing technology is solved, and the accuracy of accidental correctness test is significantly improved, which is suitable for practical purposes Dynamic defect positioning of program development scenarios.

CN119025423APending Publication Date: 2024-11-26CHONGQING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411103508.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing accidental correctness test detection methods are limited in terms of accuracy, mainly because of insufficient semantic representation of code and failure to make full use of the semantic information of test code and focus methods.

Method used

Using a detection method based on semantic representation, the metadata of the program and test suite are obtained through a tree parser, the semantic features of the test-focus pair are extracted using UniXcoder pre-trained models and multi-layer perceptrons, and a hierarchical network is built for training and output Predicted results of accidental correctness tests.

Benefits of technology

It improves the accuracy of accidental correctness test detection, has better performance than existing popular mainstream detection algorithms, and can be effectively used for dynamic defect positioning in actual program development scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025423A_ABST
    Figure CN119025423A_ABST
Patent Text Reader

Abstract

The invention relates to a semantic representation-based accidental correctness test detection method, so that the accuracy of accidental correctness test detection is improved. Comprising the following steps: analyzing a program and a test suite in a defects4J data set based on a tree-siter analyzer to obtain basic metadata of a class and a method; positioning a test class based on the at Test annotation and the Tests suffix file name; determining a focus method in each test case based on a heuristic rule, and mapping the method to the corresponding test case to obtain a test-focus pair; taking the test-focus pair based on mapping as input, and outputting semantic features of the test-focus pair through a feature extraction model; the semantic features of the test-focus pair serve as input to train the hierarchical network, and finally the trained hierarchical network is used for predicting unknown data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of accidental correctness detection of programs, and in particular to an accidental correctness test detection method based on semantic representation. Background Art

[0002] Accidental correctness is a situation where an error or bug in a program does not cause a test case to fail. Even if there is an error in the code, when the test case is run, the test case still passes the test and returns the expected result due to the specific conditions or inputs used in the test case. Accidental correctness is very common in real-world programs and has a negative impact on the widely studied defect localization. Therefore, many accidental correctness detection methods are widely used to predict potential accidental correctness tests.

[0003] In the existing accidental correctness test detection methods, machine learning algorithms are mainly used to establish detection models, and accidental correctness test cases are detected by learning coverage features or expert features. The coverage feature refers to the information that records whether each program entity (for example, a statement, function, or class) is executed when running a test case. The original coverage data or the processed coverage data (coverage data after program slicing) can be used as or extracted as coverage features. The prior art uses coverage features to measure the similarity between failed tests and accidental correctness tests, and the coverage features of accidental correctness tests are more similar to the coverage features of failed tests. Expert features use appropriate attributes of test cases to accurately locate accidental correctness test cases. At present, researchers have proposed multiple expert features, which can be roughly divided into three types of commonly used expert features: 1) Suspicion score: By analyzing the behavior and execution path of the code, the probability or suspicion score of each code fragment (such as a statement, function, or module) that it may contain an error is calculated. The higher the score, the higher the possibility that the code fragment contains an error. This type of expert feature uses a suspicious score to detect the probability of an accidental correctness test. 2) Coverage: If a test case covers more highly suspicious statements, it is more likely to be an accidental correctness test. 3) Similarity characteristics: When the coverage information of a passed test case is similar to that of a failed test case, the passed test case is more likely to be an accidental correctness test.

[0004] Although accidental correctness detection methods based on coverage features and expert features have been widely used, these methods are still limited in the accuracy of accidental correctness detection. The main reason is that the code semantics are not well represented: the test code and the focus method contain rich semantics that may be relevant to accidental correctness testing, but they are not fully utilized. Figure 1As shown in the figure, a real example from the defects4J dataset is shown to demonstrate the necessity of combining the test code semantics with the focus method for accidental correctness test detection. Therefore, the accidental correctness detection method based on semantic representation makes up for the above shortcomings, thereby improving the accuracy of accidental correctness test detection. Summary of the invention

[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to use semantic representation to improve the detection accuracy of accidental correctness test.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: an accidental correctness test detection method based on semantic representation, comprising the following steps:

[0007] S101: Obtain an existing data set, and use a tree parser to parse each program and test suite in the data set to obtain basic metadata of classes and methods.

[0008] S102: Pinpoint the test class by the @Test annotation or the file name with a "Tests" suffix.

[0009] S103: Determine the focus method in each test case through heuristic rules, and then determine the focus method in the focus class corresponding to each test case in the test class through heuristic rules, and then map these focus methods to the corresponding test cases, thereby generating a test-focus pair, in which any focus method that does not meet the heuristic rules is excluded.

[0010] S104: Construct a feature extraction model, and extract semantic features of the test-focus pair through the feature extraction model, wherein the feature extraction model inputs embedding, UniXcoder pre-trained model and multi-layer perceptron.

[0011] S105: construct a hierarchical network, use the semantic features of the test-focus pair and its corresponding label as input to train the hierarchical network, and continuously update the parameters of the multilayer perceptron and the hierarchical network through back propagation until the maximum number of iterations is reached and the training ends.

[0012] The hierarchical network consists of four layers: an input layer, a learning layer, a fully connected layer and an output layer; the input layer receives the semantic features of the test-focus pair, the learning layer aims to learn the deep features of the semantics, the fully connected layer is used to combine the learned deep features, and finally, the output layer outputs the prediction results of the accidental correctness test.

[0013] S106: For a program P and a suite T to be predicted, the method of S102-S104 is used to obtain the semantic features of the test-focus pairs corresponding to P and T through the trained multi-layer perceptron, and then the semantic features of the test-focus pairs corresponding to P and T are input into the trained hierarchical network, and the output is the prediction result of P and T.

[0014] Furthermore, in S104, the input embedding processes the test-focus pair in the memory to generate a prefix tag <enc>It is used to identify the start, and then extract and process the test code comments [CTC] test code sequence [TCS], focus method comments [CFM], focus method sequence [FMS]. After that, the four data of CTC, TCS, CFM, and FMS are obtained for tag embedding and position embedding. The tag embedding converts the words or subword tokens in each text part, namely CTC, TCS, CFM, and FMS, into vectors. The position embedding performs position encoding and marks the position of each token.

[0015] Furthermore, in S104, the UniXcoder pre-trained model uses a BERT-based bidirectional Transformer to provide a contextual representation of each token in the input sequence, and each Transformer block includes a multi-head attention network and a fully connected feed-forward network to obtain global comprehensive semantic information;

[0016] Given the initial input of the UniXcoder pre-trained model is where s i and p i They represent the token embedding and position embedding of the input sequence respectively. After passing through N layers of Transformer blocks, the output is the contextual representation of each token in the input sequence, expressed as

[0017] Furthermore, in S104, three fully connected layers are added to the first representation output by the Transformer block as a multi-layer perceptron for extracting semantic features of the test-focus pair.

[0018] Compared with the prior art, the present invention has at least the following advantages:

[0019] The present invention proposes an accidental correctness test detection method based on semantic representation, which solves the problem of insufficient code semantic representation and failure to fully utilize the test code and focus method to enrich the semantics, so that the detection results of accidental correctness test have high accuracy. The performance of the present invention (CORESEM) is better than the current popular mainstream accidental correctness detection algorithm, and can be applied to actual program development scenarios, making contributions to the field of dynamic defect location. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a defect sample of the defects4J dataset.

[0021] Figure 2 The present invention provides an overall framework for an accidental correctness testing method based on semantic representation.

[0022] Figure 3 This is the model architecture for semantic feature extraction of the present invention.

[0023] Figure 4 It is a layered network architecture.

[0024] Figure 5 Embedding processing example for data input. DETAILED DESCRIPTION

[0025] The present invention is described in further detail below.

[0026] Example 1: See Figure 1 , an accidental correctness test detection method based on semantic representation, comprising the following steps:

[0027] S101: Obtain an existing data set, and use a tree parser to parse each program and test suite in the data set to obtain basic metadata of classes and methods.

[0028] The tree-sitter tool is used to parse the actual programs and test suites in the defects4J dataset, which contains all real defects. These programs are often used to identify accidental correctness test cases. Table 1 lists the statistics of some tested programs. Each defect is associated with a set of test cases.

[0029] Table 1 Test program statistics

[0030]

[0031]

[0032] S102: Accurately locate the test class through @Test annotation or file name with "Tests" suffix. These signs indicate that these methods can be executed as test cases. Determine whether the data set file content in S101 contains @Test annotation or the data set file name contains the "Tests" suffix to locate the test class. At the same time, define the files in the data set that do not contain the above characteristics as the focus class. This method is used to divide the data set into two parts: the focus class and the test class.

[0033] S103: The focus method is determined in each test case through heuristic rules, including name matching and identifying unique method calls in the test case context. The focus method in the focus class corresponding to each test case in the test class is then determined through heuristic rules. These focus methods are then mapped to the corresponding test cases, thereby generating test-focus pairs, in which any focus methods that do not meet the heuristic rules are excluded, ensuring high-quality data for model fine-tuning.

[0034] S104: construct a feature extraction model, and extract semantic features of the test-focus pair through the feature extraction model, such as Figure 3 As shown, the feature extraction model input embedding, UniXcoder pre-trained model and multi-layer perceptron.

[0035] S105: construct a hierarchical network, use the semantic features of the test-focus pair and its corresponding label as input to train the hierarchical network, and continuously update the parameters of the multilayer perceptron and the hierarchical network through back propagation until the maximum number of iterations is reached and the training ends.

[0036] As attached Figure 4 As shown, the hierarchical network consists of four layers: input layer, learning layer, fully connected layer and output layer. The input layer receives the semantic features of the test-focus pair, the learning layer aims to learn the deep features of the semantics, the fully connected layer is used to combine the learned deep features, and finally, the output layer outputs the prediction results of the accidental correctness test. The hierarchical network is updated by back propagation according to the test label (accidental correctness or non-accidental correctness).

[0037] S106: For a program P and a suite T to be predicted, the method of S102-S104 is used to obtain the semantic features of the test-focus pairs corresponding to P and T through the trained multi-layer perceptron, and then the semantic features of the test-focus pairs corresponding to P and T are input into the trained hierarchical network, and the output is the prediction result of P and T.

[0038] Specifically, in S104, see Figure 5 , the input embedding processes the test-focus pair in memory and generates a prefix token <enc>It is used to mark the start, then extract and process the test code comments [CTC] (extract the test code comment part), test code sequence [TCS] (extract the actual test code segment), focus method comments [CFM] (extract the focus method comment part), focus method sequence [FMS] (extract the focus method actual code segment), and then the obtained CTC, TCS, CFM, FMS four data are tokenized and embedded (token embedding) and positionally embedded (position embedding). The token embedding converts the words or subwords (tokens) in each text part (CTC, TCS, CFM, FMS) into vectors, and the position embedding performs position encoding and marks the position of each token. Prefix tags help the model distinguish and identify the input data type or structure, and better perform task prediction; test code comments, test code sequences, focus method comments, and focus method sequences help the model capture the relationship between code functions, structures, and intentions by inputting different information sources; token embedding enables the model to capture and learn the semantic relationship between different symbols; position embedding helps the model distinguish different elements in the sequence, learn the relative position of elements in the sequence, and understand the impact of order on semantics.

[0039] Specifically, in S104, the UniXcoder pre-trained model uses a BERT-based bidirectional Transformer to provide contextual representation of each token in the input sequence, and each Transformer block includes a multi-head attention network and a fully connected feedforward network to obtain global comprehensive semantic information.

[0040] Given the initial input of the UniXcoder pre-trained model is where s i and p i They represent the token embedding and position embedding of the input sequence respectively. After passing through N layers of Transformer blocks, the output is the contextual representation of each token in the input sequence, expressed as

[0041] Specifically, in S104, three fully connected layers are added to the first representation output by the Transformer block as a multilayer perceptron to extract semantic features of the test-focus pair.

[0042] The public dataset selected by the invention comes from the real programs of the defects4J dataset on the Github website, including six programs: Chart, Closure, Math, Lang, Time, and Mockito. The dataset contains all real defects. The data included contains a total of 395 real defects, with a total number of code lines of 388k, a total number of test cases of 504956, a total number of accidental correctness test cases of 51577, and a total number of non-accidental correctness test cases of 452483. Each defect is associated with a set of test cases.

[0043] When constructing the test-focus pairs, we utilize the tree-sitter tool to parse the program and test suites and extract the test-focus pairs. The output dimension of the semantic features is set to 64. For model training of semantic features, there are two learning layers and two fully connected layers with 128 and 64 neurons, respectively, and 10-fold cross validation is used to obtain the results. In order to ensure high-quality data for model fine-tuning, the focus methods that do not meet the heuristic rules are excluded.

[0044] Evaluation indicators are recall, precision, and F1-score. Recall is based on the formula Calculate, T cc is the total number of accidental correctness test cases, T′ cc is the total number of accidental correctness test cases predicted, and recall is the number of correctly identified accidental correctness test cases divided by the total number of accidental correctness test cases T cc Similarly, the accuracy The symbols have the same meaning as in the recall calculation formula. Precision is the ratio of the number of correctly identified accidental correctness test cases to the total number of predicted accidental correctness test cases. The F1 value is based on the formula Calculate, where Rell and Precision are recall and accuracy respectively.

[0045] In order to further verify the effectiveness of the present invention, it is compared with the popular mainstream accidental correctness test detection algorithms based on coverage features and expert features, which are as follows:

[0046] RF: Random forest is used to build multiple decision trees based on the coverage data, and then the final result is obtained through majority voting.

[0047] CBCFL: Incorporates failure context (statements that affect the results of failed tests) as input data for cluster analysis to improve the effectiveness of identifying accidental correctness tests.

[0048] ContraCC: A contrastive learning based method that maximizes the similarity between the same class and minimizes the similarity between different classes.

[0049] FCCI: Modeling the accidental correctness identification process as a decision-making system by building a fuzzy expert system.

[0050] NeuralCCD: A neural network based approach that uses a multilayer perceptron model to learn expert features and detect accidental correctness test cases.

[0051] MLCCI: A machine learning based approach that uses 20 spectrum-based formulas to obtain 60-dimensional expert features, which are then used as input to a random forest model to detect accidental correctness test cases.

[0052] The accidental correctness test detection method based on semantic representation proposed in this paper is called CORESEM, which combines the semantic representation of test code and focus method

[0053] ,Improving the detection of accidental correctness test cases.

[0054] The experimental results are shown in Table 2. The best performance is achieved in F1-score of all programs. The present invention is significantly better than the other six most advanced benchmarks, which shows that the combination of test code and focus method and semantic representation based on the proposed method is effective.

[0055] Table 2 Experimental results in comparison with the state-of-the-art baselines in terms of effectiveness in detecting accidental correctness test cases

[0056]

[0057]

[0058] We use recall, precision, and F1-score to evaluate the effectiveness of accidental correctness detection methods with six state-of-the-art benchmarks, including three coverage-based benchmarks (RF, CBCFL, ContraCC) and three expert-based benchmarks (FCCI, NeuralCCD, MLCCI). The "Program" column indicates the name of the program, and the "Method" column indicates the accidental correctness detection method. The "Recall", "Precision", and "F1" columns indicate the recall, precision, and F1-score of the accidental correctness detection method for the corresponding program, respectively. The best results are highlighted with bold and dark gray background, and the second-best results are highlighted with light gray background. For example, in the Chart program, CORESEM has an F1-score of 70%, which is 6% higher than the second-best benchmark MLCCI. Similarly, in the Closure program, CORESEM has an F1-score of 65%, which is 11% higher than ContraCC. CORESEM achieved the best or second-best results in 16 out of 18 cases (six programs*three evaluation metrics).

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.< / enc> < / enc>

Claims

1. A method for accidental correctness testing based on semantic representation, Features: The steps include: S101: Obtain an existing data set, and use a tree parser to parse each program and test suite in the data set to obtain basic metadata of classes and methods; S102: Use @Test annotation or file name with "Tests" suffix to pinpoint the test class; S103: determining a focus method in each test case by using a heuristic rule, and then determining a focus method in a focus class corresponding to each test case in the test class by using the heuristic rule, and then mapping these focus methods to corresponding test cases, thereby generating a test-focus pair, wherein any focus method that does not meet the heuristic rule is excluded; S104: construct a feature extraction model, and extract semantic features of the test-focus pair through the feature extraction model, wherein the feature extraction model inputs the embedding, the UniXcoder pre-trained model, and the multi-layer perceptron; S105: construct a hierarchical network, use the semantic features of the test-focus pair and its corresponding label as input to train the hierarchical network, and continuously update the parameters of the multilayer perceptron and the hierarchical network through back propagation until the maximum number of iterations is reached, and the training ends; The hierarchical network consists of four layers: input layer, learning layer, fully connected layer and output layer; The input layer receives the semantic features of the test-focus pair, the learning layer aims to learn the deep features of the semantics, the fully connected layer is used to combine the learned deep features, and finally, the output layer outputs the prediction results of the accidental correctness test; S106: For a program P and a suite T to be predicted, the method of S102-S104 is used to obtain the semantic features of the test-focus pairs corresponding to P and T through the trained multi-layer perceptron, and then the semantic features of the test-focus pairs corresponding to P and T are input into the trained hierarchical network, and the output is the prediction result of P and T.

2. The accidental correctness test detection method based on semantic representation as claimed in claim 1, Features: In S104, the input embedding processes the test-focus pair in memory to generate a prefix tag. <enc> It is used to identify the start, and then extract and process the test code comments [CTC] test code sequence [TCS], focus method comments [CFM], focus method sequence [FMS]. After that, the four data of CTC, TCS, CFM, and FMS are obtained for tag embedding and position embedding. The tag embedding converts the words or subword tokens in each text part, namely CTC, TCS, CFM, and FMS, into vectors. The position embedding performs position encoding and marks the position of each token.< / enc> 3. The accidental correctness test detection method based on semantic representation as claimed in claim 2, Features: In S104, the UniXcoder pre-trained model uses a BERT-based bidirectional Transformer to provide a contextual representation of each token in the input sequence, and each Transformer block includes a multi-head attention network and a fully connected feed-forward network to obtain global comprehensive semantic information; Given the initial input of the UniXcoder pre-trained model is where s i and p i They represent the token embedding and position embedding of the input sequence respectively. After passing through N layers of Transformer blocks, the output is the contextual representation of each token in the input sequence, expressed as 4. The accidental correctness test detection method based on semantic representation as claimed in claim 3, Features: In S104, three fully connected layers are added to the first representation output by the Transformer block as a multi-layer perceptron to extract semantic features of the test-focus pair.