Layering perception-based instability test classification detection method

Abstract

The invention discloses an instability test classification detection method based on hierarchical perception, which comprises the following steps of: based on artificial code analysis, acquiring a plurality of test code data with instability; preprocessing the test code data; an instability test classification detection model is constructed and trained, wherein the instability test classification detection model comprises a UniXcoder model used for extracting code semantic feature pre-training, a structure encoder used for extracting hierarchical structure features and a classification model used for classification; and predicting new code data by using the trained instability test classification detection model, outputting a prediction result of the stability of the code data, and if a predicted label is unstable, outputting a hierarchical root cause causing the instability test at the same time.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a method for detecting and analyzing the causes of instability of Flakytest (instability test) use cases based on hierarchical features and semantic features. Background Art

[0002] In software development, the correctness of the code is crucial, and software testing is the key link to verify this. Developers use testing to ensure the accuracy and reliability of the code. In traditional testing, test failures are usually caused by the latest changes to the code. However, in actual development, developers often encounter a specific type of test: unstable tests. These tests may show different results in different executions, and even fail sometimes and succeed sometimes when the object under test and the test conditions remain unchanged. This makes it difficult for developers to judge whether it is a problem with the code or the instability of the test itself, which not only affects the reliability of the software, but may also cause developers to spend a lot of time fixing the code. Therefore, the stability of the test is one of the keys to ensuring software quality.

[0003] To address this problem, there are currently a variety of tools and methods for detecting unstable tests. Early methods, such as rerunning the test suite under different configurations, detected test instability. However, this rerun-based detection method usually requires a lot of computing resources, which often leads to high time and computing costs, especially in complex scenarios, which may require more than 1,000 reruns. Moreover, simply determining whether a test is unstable is not enough to help developers understand the root cause of unstable tests. With the development of technology, researchers have begun to turn to static code analysis methods, trying to identify unstable tests by using the static characteristics of test code without relying on tedious reruns.

[0004] Such methods have made some progress, but most existing tools and methods focus on detecting whether the test is unstable, and do not deeply analyze the root causes of test instability, ignoring the hierarchical structure of the instability problem. Studies have shown that the causes of unstable tests can be classified according to certain hierarchies, such as "order dependency (OD)" and "non-order dependency (NOD)", and the hierarchical information of these causes is crucial to improving the detection accuracy. If this hierarchical information is not effectively utilized, it may lead to poor detection results and limit the effectiveness of its practical application. Another key problem is that existing methods usually regard the causes of instability as a single-label classification problem, however, unstable tests often involve multiple root causes. For example, some tests may be affected by multiple factors such as "shared state concurrency between components" and "not releasing acquired resources", which requires the classification method to handle multi-label classification problems. Existing tools often cannot effectively solve this complex situation where multiple root causes coexist. Finally, many existing tools rely on statistical information or rules to determine whether a test is unstable, while ignoring the semantic information of the test code itself, which makes these tools less effective when facing complex test cases and unable to deeply explore potential causes of instability. Summary of the invention

[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: to determine the test instability of static code as accurately as possible.

[0006] The present invention adopts a variety of public instability test data sets and analyzes static code based on the code semantic features of the hierarchical structure to determine whether it belongs to instability testing. If so, the root cause of the test instability is analyzed.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: a hierarchical perception-based instability test classification detection method, comprising the following steps:

[0008] S101: Acquire a number of test code data that have instability, each test code data consists of a test name, a test code and a label, the label is stable and unstable, if the label is unstable, then the label also includes a hierarchical root cause of the instability test.

[0009] S102: preprocessing the test code data, removing harmful information of the test code in each test code data, and then using the word segmentation of the byte-encoded BPE word segmenter to perform word segmentation processing.

[0010] Then construct a directed graph Where V represents a set of nodes, each node represents a hierarchical root cause of instability testing, represents the set of directed edges from the previous layer nodes to the next layer nodes, represents the set of directed edges from the nodes of the next layer to the nodes of the previous layer, and calculates the prior probability of each hierarchical root cause of the instability test;

[0011] S103: constructing and training an instability test classification detection model, wherein the instability test classification detection model includes a code semantic feature extraction model, a hierarchical feature extraction model and a classification model.

[0012] S103-1: Code semantic feature extraction model converts the test code data preprocessed by S102 into a vector representation that can be processed by a computer and extracts the code semantic features Based on the calculation edge The prior probability and edge The prior probability

[0013] S103-2: Hierarchical feature extraction model is used to extract hierarchical features At the same time and Transformed into information V represented by a label.

[0014] S103-3: Take V as the input of the classification model and output the predicted label of the test code data. If the label is unstable, the hierarchical root cause of the instability test is also output.

[0015] S103-4: Use binary cross entropy loss as the loss function of the instability test classification detection model. When the loss no longer decreases, the trained instability test classification detection model is obtained.

[0016] S104: For a piece of code data, the code data is input into a trained instability test classification detection model, and a prediction result of the stability of the code data is output.

[0017] Furthermore, the calculation of P(U j ∣∣U i ) process is as follows:

[0018] Based on Bayesian statistical inference, the prior knowledge that depends on the label is regarded as the correlation between labels, and the edge features between label nodes are represented by prior probability:

[0019]

[0020] Among them, P(U j ∣∣U i ) represents the probability of node j appearing when node i appears, P(U j∩U i ) represents the probability of node j and node i appearing at the same time, U j and U i Respectively represent the appearance of node j and the appearance of node i, P(U i ) represents the probability of node i appearing, N i and N j Respectively represent the number of times node j and node i appear, as a prior probability.

[0021] Furthermore, the code semantic features are extracted in S3-1 The process is:

[0022] The code semantic feature extraction model uses the pre-trained UniXcoder model. The UniXcoder model first converts each test code data processed by the S102 tokenizer into a token represented by a vector and the position of each token in the code; secondly, the token and the position of each token in the code are combined as the input of the BERT-based bidirectional Transformer model. in Represents the input information of the i-th test code data, x i The token representing the i-th test code, p i Indicates the position of the token of the i-th test code data in the code; finally, in the bidirectional Transformer model architecture based on BERT, after passing through N layers of Transformer layers, the context representation of each Token is output

[0023] Furthermore, in S3-1 and The calculation process is as follows:

[0024]

[0025]

[0026] Furthermore, in S103-2, The process:

[0027] Based on the bidirectional tree LSTM, the top-down and bottom-up information flows are combined to calculate the hidden state of the transmission from the bottom node to the top node.

[0028]

[0029] Among them, child(k) is the set of lower-level nodes of node k, is the hidden state of the top layer, is the hidden state of the top node j, U k Indicates that node k appears.

[0030] Calculate the hidden state of the pass from the top node to the bottom node

[0031]

[0032] in, is the underlying hidden state, is the hidden state of the underlying node p.

[0033] Will and Merge to get hierarchical features

[0034]

[0035] in Connections representing hidden states.

[0036] Furthermore, in S103-2, the process of obtaining the information V represented by the label is:

[0037] All the extracted code semantic features Encoded into a high-dimensional semantic embedding S = (s1, s2, ..., s n ),in s i represents the i-th semantic feature, n is the length of the input sequence, and the code semantic matrix is ​​obtained Through the trainable parameter matrix Perform linear transformation: V = SM, reshaped into the semantic features of the label where d v Represents the dimension of the representation vector of each label node, d u Represents the dimension of semantic features, and C is the number of label categories.

[0038] The initial information V represented by the label k Enter the bidirectional tree LSTM input gate:

[0039] i k =σ(W i ·[h p ,h l ,h r ,V k ]+b i ) where σ is the sigmoid activation function, W i is the weight matrix of the input gate, h pis the hidden state of the upper node, h l ,h r are the hidden states of the left and right child nodes, respectively, and b i is the bias of the input gate. And calculate V k Candidate state g k :g k =tanh(W g ·[h p ,h l ,h r ,V k ]+b g ), where tanh is the tanh activation function, W g is the weight matrix of the candidate state, b g is the bias of the candidate state; then calculate the forget gate f k =σ(W f ·[h p ,h l ,h r ]+b f ), where W f is the weight matrix of the forget gate, b f is the bias of the forget gate.

[0040] Get cell state c k :c k =f k ·c p +i k ·g k , where c p is the cell state of the upper node. k Summarize to the output gate to update the hidden node: h k =o k ·tanh(c k ), where o k =σ(W o ·[h p ,h l ,h r ,V k ]+b o ), b o is the bias of the output gate, W o is the weight matrix of the output gate.

[0041] When the performance of LSTM no longer improves, stop updating the hidden nodes and use the final hidden node as the final V output.

[0042] Furthermore, the loss function in S103-4 is:

[0043] L m =L c +λ·Lr

[0044] Where L m is the total loss, L c is the prediction error for each label, L r is the recursive regularization term, and λ is the weight parameter.

[0045]

[0046] where w i , w j They represent the weight vectors of the i-th node and the j-th node respectively, C is the set of all nodes, and child(i) is the set of lower-level nodes of i.

[0047]

[0048] where y ij represents the true label, y′ ij Represents the predicted label.

[0049] Compared with the prior art, the present invention has at least the following advantages:

[0050] (1) Accurate and efficient determination of instability test

[0051] By utilizing a hierarchical label structure, the model can capture the different causes of instability testing in a more fine-grained manner and improve the accuracy of detection through the propagation of hierarchical relationships. Traditional single-label models are often unable to handle complex dependencies between labels, while the hierarchical perception model of the present invention can handle such dependencies, thereby greatly improving the accuracy of instability testing. In addition, compared with traditional rerun-based detection methods, static analysis methods are not only more efficient, but also do not rely on repeated execution of tests, which improves the efficiency and feasibility of testing.

[0052] (2) Analyze the root cause of instability

[0053] Traditional methods can usually only detect stability issues in tests, but cannot analyze the root causes of instability. The present invention uses hierarchical root cause analysis to enable the model to capture dependencies between labels at multiple levels, significantly improving the effectiveness of classification tasks. It can provide a deeper understanding of instability tests and help developers find the specific source of problems. For example, different types of root causes such as "sequential dependency" or "non-sequential dependency" can be detected, thereby providing targeted improvement solutions for fixing test instability.

[0054] (3) Multi-label classification capability

[0055] The instability of test code may be affected by multiple factors. The present invention can effectively handle the coexistence of multiple root causes and provide multi-label classification capabilities. In this way, multiple factors that may affect test stability can be identified and more comprehensive guidance can be provided to developers.

[0056] (4) Enhanced model interpretability

[0057] Through the hierarchical label structure, testers can clearly see how a test sample gradually rises from the bottom-level specific problems (such as resource leakage, concurrency problems, etc.) to higher-level instability types (such as unstable network environment, test case timeout, etc.). This hierarchical structure not only enhances the classification ability of the model, but also improves the interpretability of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a framework diagram of the method of the present invention.

[0059] Figure 2 Analysis of classification results for labels that are unstable or stable.

[0060] Figure 3 Analysis of the classification results for labels belonging to unstable / non-sequential dependencies and their underlying labels.

[0061] Figure 4 Analysis of the classification results for labels belonging to unstable / sequential dependencies and their underlying labels.

[0062] Figure 5 Analysis of the classification results for the label unstable / implementation-dependent and its lower-level labels.

[0063] Figure 6 For comparison of experimental results.

[0064] Figure 7 This is the ablation experiment comparison of this method.

[0065] Figure 8 is an example of a tag hierarchy. DETAILED DESCRIPTION

[0066] The present invention is described in further detail below.

[0067] The method of the present invention mainly determines the test instability of static code as accurately as possible by the following means:

[0068] (1) Acquisition and preprocessing of instability test data

[0069] The present invention first obtains test code data marked as "stable" or "unstable" from the existing instability test data set. In the instability test case, if the label is unstable, the hierarchical root cause of the instability (such as sequential dependency, non-sequential dependency, etc.) is manually marked. Through this approach, the source of instability of each data can be clearly identified.

[0070] The key steps in the preprocessing part are:

[0071] Remove irrelevant or harmful information: Remove noise information or irrelevant parts in the test code and retain the core test code.

[0072] BPE word segmentation processing: Use byte pair encoding (BPE) to segment the test code into vocabulary units to provide standardized input for subsequent analysis.

[0073] (2) Identifying hierarchical features

[0074] The present invention organizes the labels of the test code instability into a hierarchical structure, and forms a tree or directed acyclic graph (DAG) structure by establishing dependencies between the labels. This structure can reflect the hierarchical relationship and dependency between the labels. For example, if a test sample is associated with a node (the specific cause of instability), it will also be implicitly associated with the upper layer node (the broader cause of instability).

[0075] (3) Using label dependencies based on prior probabilities

[0076] By using prior probabilities to represent the dependencies between tags, the present invention can adjust the weights of the relationships between tags based on the co-occurrence of tags in the training data. This method ensures that the model not only focuses on the current tag, but also effectively utilizes the dependency information between tags, thereby improving the accuracy of the judgment.

[0077] (4) Efficient aggregation hierarchy and semantic features

[0078] When predicting the test code, the present invention uses semantic feature extraction and structural encoder to combine semantic information and hierarchical features of labels. Specifically, the pre-trained UniXcoder model is used to extract semantic features of the test code, and then the information of the upper and lower nodes is propagated in the hierarchical structure through the hierarchical-aware structural encoder, i.e., the bidirectional Tree-LSTM, so as to accurately capture the relationship between labels at different levels. Finally, a comprehensive feature representation is generated, which will be input into the classification model.

[0079] (5) Adapt to multi-label classification

[0080] This classification model can effectively handle multi-label test code data: the model can identify whether test instability is caused by a single factor or multiple root causes working together, and ultimately optimizes through a binary cross entropy loss function to achieve accurate prediction of unstable tests.

[0081] See also Figure 1 , a classification detection method for instability test based on hierarchical perception, comprising the following steps:

[0082] S101: Acquire a number of test code data that have instability, each test code data consists of a test name, a test code and a label, the label is stable and unstable, if the label is unstable, then the label also includes a hierarchical root cause of the instability test.

[0083] S102: preprocessing the test code data, removing harmful information of the test code in each test code data, and then using the word segmentation of the byte-encoded BPE word segmenter to perform word segmentation processing.

[0084] Use the test code data and its corresponding labels as prior knowledge to build the classification model substructure, that is, to build a directed graph Where V represents a set of nodes, each node represents a hierarchical root cause of instability testing, represents the set of directed edges from the previous layer nodes to the next layer nodes, Represents the set of directed edges from nodes in the next layer to nodes in the previous layer, and calculates the prior probability of each hierarchical root cause of the instability test.

[0085] Test smells (a warning in the test code that may indicate some signs or patterns of problems) are highly correlated with whether the test is unstable. Specifically, certain test smells (e.g., Fire and Forget and AssertionRoulette) are more likely to cause test cases to exhibit instability during different executions. We call the part of the code containing test smells as harmful information. We first annotate the statements related to test smells to remove harmful information, and then tokenize the test code based on a byte pair encoding (BPE) tokenizer.

[0086] S103: constructing and training an instability test classification detection model, wherein the instability test classification detection model includes a code semantic feature extraction model, a hierarchical feature extraction model and a classification model.

[0087] S103-1: Code semantic feature extraction model converts the test code data preprocessed by S102 into a vector representation that can be processed by a computer and extracts the code semantic features Based on the calculation edge The prior probability and edge The prior probability

[0088] S103-2: Hierarchical feature extraction model is used to extract hierarchical features at the same time and Transformed into information V represented by a label.

[0089] S103-3: Take V as the input of the classification model and output the predicted label of the test code data. If the label is unstable, the hierarchical root cause of the instability test is also output.

[0090] S103-4: Use binary cross entropy loss as the loss function of the instability test classification detection model. When the loss no longer decreases, the trained instability test classification detection model is obtained.

[0091] S104: For a piece of code data, the code data is input into a trained instability test classification detection model, and a prediction result of the stability of the code data is output.

[0092] Specifically, the calculation of P(U j ∣∣U i ) process is as follows:

[0093] Based on Bayesian statistical inference, the prior knowledge that depends on the label is regarded as the correlation between labels, and the edge features between label nodes are represented by prior probability:

[0094]

[0095] Among them, P(U j ∣∣U i ) represents the probability of node j appearing when node i appears, P(U j ∩U i ) represents the probability of node j and node i appearing at the same time, U j and U i Respectively represent the appearance of node j and the appearance of node i, P(U i ) represents the probability of node i appearing, N i and N j Represent the number of times node j and node i appear respectively. As a prior probability, Figure 8 shown.

[0096] Specifically, the code semantic features are extracted in S3-1. The process is:

[0097] The code semantic feature extraction model uses the pre-trained UniXcoder model. The UniXcoder model first converts each test code data processed by the S102 tokenizer into a token represented by a vector and the position of each token in the code. Secondly, the token and the position of each token in the code are combined as the input of the BERT-based bidirectional Transformer model. in Represents the input information of the i-th test code data, x i The token representing the i-th test code, p i Indicates the position of the token of the i-th test code data in the code. Finally, in the bidirectional Transformer model architecture based on BERT, each Transformer block contains a multi-head attention network and a fully connected feedforward network, which can understand the context of the input code from both the left and right directions to capture comprehensive global semantic information. After N layers of Transformer layers, the output is the context representation of each Token. That is, the semantic features of each Token.

[0098] Specifically, in S3-1 and The calculation process is as follows:

[0099]

[0100]

[0101] Represents the data flow from the top layer to the bottom layer, It represents the data flow from the bottom layer to the top layer, ensuring that information is propagated from the lower layer nodes to the higher layer nodes.

[0102] Specifically, in S103-2, The process:

[0103] Based on the bidirectional tree LSTM, the top-down and bottom-up information flows are combined to calculate the hidden state of the transmission from the bottom node to the top node.

[0104]

[0105] Among them, child(k) is the set of lower-level nodes of node k, is the hidden state of the top layer, is the hidden state of the top node j, U k Indicates that node k appears;

[0106] Calculate the hidden state of the pass from the top node to the bottom node

[0107]

[0108] in, is the underlying hidden state, is the hidden state of the bottom-level node p;

[0109] Will and Merge to get hierarchical features

[0110]

[0111] in Connections representing hidden states.

[0112] Specifically, in S103-2, the process of obtaining the information V represented by the label is as follows: based on the UniXcoder model combined with the hierarchical perceptual encoder, an end-to-end hierarchical structure-aware global model is constructed to capture the semantic features of the test code. Interaction with tags:

[0113] All the extracted code semantic features Encoded into a high-dimensional semantic embedding S = (s1, s2, ..., s n ),in s i represents the i-th semantic feature, n is the length of the input sequence, and the code semantic matrix is ​​obtained Through the trainable parameter matrix Perform linear transformation: V = SM, map the semantic information of each test code data into the label space, align the semantic features with the semantic representation of the label, and reshape them into the semantic features of the label where d v Represents the dimension of the representation vector of each label node, d u Represents the dimension of semantic features, C is the number of label categories;

[0114] Then, the initial V is input into the bidirectional tree LSTM for propagation and feature update. Like the standard LSTM, each node in the TreeLSTM contains four gates: input gate (i k ), forget gate (f k ), candidate state (g k ), and the final output gate, which determines how the node's state is updated. The specific updates are as follows:

[0115] The initial information V represented by the labelk Enter the bidirectional tree LSTM input gate:

[0116] i k =σ(W i ·[h p ,h l ,h r ,V k ]+b i ) where σ is the sigmoid activation function, W i is the weight matrix of the input gate, h p is the hidden state of the upper node, h l ,h r are the hidden states of the left and right child nodes, respectively, and b i is the bias of the input gate. And calculate V k Candidate state g k :g k =tanh(W g ·[h p ,h l ,h r ,V k ]+b g ), where tanh is the tanh activation function, W g is the weight matrix of the candidate state, b g is the bias of the candidate state; then calculate the forget gate f k =σ(W f ·[h p ,h l ,h r ]+b f ), where W f is the weight matrix of the forget gate, b f is the bias of the forget gate.

[0117] Get cell state c k :c k =f k ·c p +i k ·g k , where c p is the cell state of the upper node. k Summarize to the output gate to update the hidden node: h k =o k ·tanh(c k ), where o k =σ(W o ·[h p ,h l ,h r ,V k ]+b o ), bo is the bias of the output gate, W o is the weight matrix of the output gate;

[0118] When the performance of LSTM no longer improves, the hidden nodes are stopped from being updated, and the final hidden nodes are output as the final V. Finally, V processed by the bidirectional tree LSTM is used as the input classification model for the final label prediction.

[0119] This transformation enables the model to utilize these semantic features for deeper propagation and updating in the hierarchical perceptual encoder: the initial node representation V is passed into the hierarchical perceptual encoder, and the resulting hidden state h embodies the semantic features of the hierarchical perception, which is then used as the input of the final classification model. This approach ensures that the hierarchical perceptual structure encoder can operate during both training and inference, thereby promoting the propagation of semantic information between layers.

[0120] Specifically, the loss function in S103-4 is:

[0121] L m =L c +λ·L r

[0122] Where L m is the total loss, L c is the prediction error for each label, L r is the recursive regularization term, and λ is the weight parameter.

[0123]

[0124] where w i , w j They represent the weight vectors of the i-th node and the j-th node respectively, C is the set of all nodes, and child(i) is the set of lower-level nodes of i.

[0125]

[0126] where y ij represents the true label, y′ ij Represents the predicted label.

[0127] Before multi-label classification, the label hierarchy is flattened and all nodes are treated as leaf nodes. Secondly, define C as the set of all nodes. For all child nodes child(i) of node i, use the parent-child node weight vector w i , w j , introduce The recursive regularization term reduces the difference between node parameters at the same level and maintains the relationship between label levels; then based on the true value (0 or 1) y of the i-th sample at the corresponding j-th label node ij The probability value y′ obtained by the sigmoid activation function of the i-th sample on the corresponding j-th label node ij

Note that nodes are labels

[0128] Experiment and analysis:

[0129] The test code use case dataset selected by the present invention comes from the dataset contributed by IDoFT, FlakeFlagger and Gruber et al. on github. The causes of instability of each test are manually obtained as labels by summarizing the modification requests in GitHub Pull Request and manual code analysis. These label hierarchies are derived from multiple latest research classification systems. The IDoFT dataset selects 2455 test cases, including 9 hierarchical labels such as Flaky, OD, and ID; the FlakeFlagger dataset selects 5450 test cases, including two labels Flaky and Not Flaky; the dataset contributed by Gruber et al. on github selects 2885 test cases, including 34 hierarchical labels such as Flaky, OD, and NOD.

[0130] Randomly select 60% of all test cases of each label as training data, 20% as validation data, and the remaining 20% ​​as test data.

[0131] The evaluation indicators are precision, recall, Macro-F1 and Micro-F1.

[0132]

[0133] Where T flaky represents the collection of actual flaky tests, Indicates the test set where the model predicts to be flaky.

[0134]

[0135]

[0136]

[0137] Where |L| represents the total number of labels, F 1i Represents the F1 score for each label.

[0138] In order to further verify the effectiveness of the present invention, it is compared with popular mainstream recommendation algorithms, which are as follows:

[0139] FlakeFlagger: A machine learning based prediction method that combines static and dynamic features to predict a binary classification algorithm for instability testing.

[0140] Flakify: A prediction method based on black-box language models that uses CodeBERT to generate embeddings for test code and predict binary classification algorithms for instability tests.

[0141] The hierarchical perception-based instability test classification detection method proposed in the present invention is called HiFlaky.

[0142] Experimental results show that the present invention is significantly superior to other recommendation algorithms in terms of both Precision and F1-score, with an accuracy improvement of 30% on the same dataset and a MicroF1 score of 0.812, indicating that the overall accuracy of all test instances is high, highlighting the effectiveness of the model in consistently detecting more frequent categories. This shows that the instability test classification detection method combined with hierarchical awareness proposed in this paper is effective. FlakeFlagger only selects features with information gain (IG) values ​​greater than 0.01 for training. Its reliance on the acquisition of static and dynamic features limits its application in certain projects, and its generalization ability is limited by the distribution of training data, and it performs poorly on other general-purpose datasets. Flakify relies entirely on the semantic and grammatical information of the test code, which affects the accuracy of the prediction when the test code cannot fully reflect the characteristics of the production code.

[0143] The present invention adds multi-level perception on the basis of instability test classification detection, refines the detection of instability tests from the perspective of hierarchical structure, effectively captures label dependencies, and enriches the feature representation of complex test scenarios. Although HiFlaky is not specifically designed for binary classification, its adaptation to this task shows excellent performance, especially in terms of accuracy and F1 score. This success is largely due to its hierarchical structure, which helps learn label dependencies and improve classification accuracy.

[0144] In short, the present invention proposes an instability test classification and detection method based on hierarchical perception. On the one hand, it improves the recognition accuracy of instability tests, and on the other hand, it solves the problem that existing methods cannot handle multiple root causes and hierarchical information, accurately locates the crux of the problem, and is superior to the current popular mainstream recommendation algorithms. It can be applied to actual development scenarios, provide framework suggestions for software developers or implement automatic repairs, improve software reliability and simplify testing.

[0145] This method evaluates each label:

[0146] See also Figure 2 , the Precision of unstable labels is 96%, Recall is 93%, and F1 is 95%, while the Precision of stable labels is 94%, Recall is 98%, and F1 is 96%, showing high accuracy and recall, and there is a good balance between the two.

[0147] See also Figure 3, the Precision of non-sequential dependency labels is 89%, Recall is 70%, and F1 is 78%, indicating that the model can better identify non-sequential dependency problems. The lower-level labels of non-sequential dependency: the Precision of concurrent labels is 50%, Recall is 40%, and F1 is 44%, showing that the model performs generally on this label, with low accuracy and recall; the Precision of concurrent / asynchronous waiting labels is 53%, Recall is 40%, and F1 is 46%, which is similar to the concurrent label, with low accuracy and limited recall; for input and output, the model's performance is more complicated. Although its Precision reaches 75%, its Recall is low, only 15%, and F1 is 25%, which shows that the accuracy is relatively good in some scenarios, but the recall ability is weak; the Precision of network labels is 70%, Recall is 56%, and F1 is 62%, indicating that the model is more accurate in identifying network-related problems, but the recall rate still has room for improvement. The Precision of the Network / Local tag is 100%, but the Recall is 80% and the F1 is 89%, which shows that the model has a very high accuracy for this tag, but there are still some missed detections; in other tags, the Precision is 77%, the Recall is 33%, and the F1 is 47%, indicating that the model performance of this tag is average, with high accuracy but weak recall. The Precision of the Other / Program Logic tag is 50%, the Recall is 20%, and the F1 is 29%, which is weak, and there is room for improvement in both accuracy and recall. The Precision of the Other / Unknown tag is 100%, the Recall is 36%, and the F1 is 53%, showing that the accuracy of this tag is high, but the recall rate is low, indicating that the model may miss detections on such issues. Finally, the Precision of the Platform Dependent tag is 80%, the Recall is 27%, and the F1 is 40%, indicating that the model has a high accuracy for this tag, but the recall ability is insufficient. The platform-dependent / runtime label has a Precision of 100%, a Recall of 27%, and an F1 of 42%, showing very high accuracy, but a low recall rate, which may fail to identify all platform-related test instabilities. The randomness label has a Precision of 59%, a Recall of 53%, and an F1 of 56%, which is a relatively balanced performance. The randomness / data label has a Precision of 64%, a Recall of 45%, and an F1 of 53%, showing high accuracy, but the recall ability needs to be improved. The time label has a Precision of 46%, a Recall of 40%, and an F1 of 43%, showing low accuracy and recall. The time / system date label has a Precision of 100%, a Recall of 33%, and an F1 of 50%, indicating that this label performs well in terms of accuracy, but has weak recall ability.The Precision of the time / test timeout label is 34%, Recall is 24%, and F1 is 28%, indicating that the model has weak recognition ability for this label.

[0148] See also Figure 4 In the sequential dependency label analysis, the Precision of the sequential dependency label is 80%, Recall is 80%, and F1 is 80%. The precision and recall of this label are very close, showing a balanced performance. The Precision of the input-output label is 75%, Recall is 15%, and F1 is 25%, indicating that the model has high accuracy in identifying input-output related problems, but the recall rate is low, and there are some missed detections. The Precision of the resource leakage label is 81%, Recall is 40%, and F1 is 54%. This label shows high accuracy, but weak recall ability, and may fail to identify all related problems. The Precision of the shared state label is 78%, Recall is 88%, and F1 is 83%. The performance is relatively balanced, with high accuracy and recall, indicating that the model can effectively identify instabilities related to shared states.

[0149] See also Figure 5 , the Precision of the implementation-dependent label is 71%, Recall is 77%, and F1 is 74%, indicating that the accuracy and recall of this label are relatively balanced, but the overall performance is medium. The lower-level label of the implementation dependency: API behavior label has a Precision of 92%, Recall of 62%, and F1 of 74%, showing high accuracy, but low recall, which means that the model may miss some relevant issues. The Precision and Recall of the data structure label are low, at 23% and 35% respectively, and F1 is 28%, indicating that the model is less effective in identifying instabilities related to data structures.

[0150] See also Figure 6,When comparing the performance of the three experimental methods (HiFlaky, Flakify, and FlakeFlagger), HiFlaky's Precision is 89%, the highest among the three, indicating that it performs best in accurately identifying test instability. However, HiFlaky's Recall is low, only 57%, which leads to its F1 score of 69%, which is at a medium level among the three methods, showing that its balance between Precision and Recall is relatively moderate. In contrast, Flakify's Precision is 65%, which is low, showing that its accuracy is insufficient, and although its Recall is high (85%), its F1 score is 74%, which still shows that its overall recognition performance is not ideal and there is a large imbalance. FlakeFlagger's Precision is 61%, the lowest among the three, indicating that its accuracy is the worst. Although FlakeFlagger's Recall is 73%, higher than HiFlaky, due to its low Precision, its F1 score is only 66%, and its performance is average. It fails to find a good balance between Precision and Recall, and its overall performance is relatively lacking.

[0151] See also Figure 7 , in the ablation experiment: Our HiFlaky performs best using Unixcoder, with a Micro-F1 value of 0.8119, which is significantly higher than the other two configurations, indicating that it has the best overall performance in all categories. Its Macro-F1 value is 0.4141, which is also high, indicating that it is well balanced across categories. The Micro-F1 using Codebert is 0.7795, which is slightly lower than Unixcoder, indicating that it is slightly inferior in testing instability recognition. Its Macro-F1 value is 0.2612, indicating that the model has poor balance between different categories. The Micro-F1 without combining hierarchical information is 0.6880, which performs the worst, especially in terms of the overall recognition effect. Its Macro-F1 value is 0.1851, which further reflects that after removing the hierarchical features, the performance of the model is unbalanced between categories. This ablation experiment shows that using Unixcoder has better performance.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A hierarchical perception-based instability test classification detection method, characterized in that: The steps include: S101: Acquire a number of test code data that have unstable behavior, where each piece of test code data consists of a test name, a test code, and a label, where the label is stable or unstable. If the label is unstable, the label also includes a hierarchical root cause of the unstable test. S102: preprocessing the test code data, removing harmful information of the test code in each test code data, and then performing word segmentation processing using the word segmenter of the byte-encoded BPE; Then construct a directed graph Where V represents a set of nodes, each node represents a hierarchical root cause of instability testing, represents the set of directed edges from the previous layer nodes to the next layer nodes, represents the set of directed edges from the nodes of the next layer to the nodes of the previous layer, and calculates the prior probability of each hierarchical root cause of the instability test; S103: constructing and training an instability test classification detection model, wherein the instability test classification detection model includes a code semantic feature extraction model, a hierarchical feature extraction model, and a classification model; S103-1: Code semantic feature extraction model converts the test code data preprocessed by S102 into a vector representation that can be processed by a computer and extracts the code semantic features Based on the calculation edge The prior probability and edge The prior probability S103-2: Hierarchical feature extraction model is used to extract hierarchical features At the same time and Transformed into information V represented by labels; S103-3: Take V as the input of the classification model, and output the predicted label of the test code data. If the label is unstable, the hierarchical root cause of the instability test is also output; S103-4: Use binary cross entropy loss as the loss function of the instability test classification detection model. When the loss stops decreasing, the trained instability test classification detection model is obtained. S104: For a piece of code data, the code data is input into a trained instability test classification detection model, and a prediction result of the stability of the code data is output.

2. The instability test classification detection method based on hierarchical perception as claimed in claim 1, characterized in that: In S102, P(U j ∣∣U i ) process is as follows: Based on Bayesian statistical inference, the prior knowledge that depends on the label is regarded as the correlation between labels, and the edge features between label nodes are represented by prior probability: Among them, P(U j ∣∣U i ) represents the probability of node j appearing when node i appears, P(U j ∩U i ) represents the probability of node j and node i appearing at the same time, U j and U i Respectively represent the appearance of node j and the appearance of node i, P(U i ) represents the probability of node i appearing, N i and N j Respectively represent the number of times node j and node i appear, as a prior probability.

3. The instability test classification detection method based on hierarchical perception as claimed in claim 2, characterized in that: Extracting code semantic features in S3-1 The process is: The code semantic feature extraction model uses the pre-trained UniXcoder model. The UniXcoder model first converts each test code data processed by the S102 tokenizer into a token represented by a vector and the position of each token in the code; secondly, the token and the position of each token in the code are combined as the input of the BERT-based bidirectional Transformer model. in Represents the input information of the i-th test code data, x i The token representing the i-th test code, p i Indicates the position of the token of the i-th test code data in the code; finally, in the bidirectional Transformer model architecture based on BERT, after passing through N layers of Transformer layers, the context representation of each Token is output 4. The instability test classification detection method based on hierarchical perception as claimed in claim 3, characterized in that: In S3-1 and The calculation process is as follows:

5. The instability test classification detection method based on hierarchical perception as claimed in claim 4, characterized in that: The obtained The process: Based on the bidirectional tree LSTM, the top-down and bottom-up information flows are combined to calculate the hidden state of the transmission from the bottom node to the top node. Among them, child(k) is the set of lower-level nodes of node k, is the hidden state of the top layer, is the hidden state of the top node j, U k Indicates that node k appears; Calculate the hidden state of the pass from the top node to the bottom node in, is the underlying hidden state, is the hidden state of the bottom-level node p; Will and Merge to get hierarchical features in Connections representing hidden states.

6. The instability test classification detection method based on hierarchical perception as claimed in claim 5, characterized in that: In S103-2, the process of obtaining the information V represented by the label is: All the extracted code semantic features Encoded into a high-dimensional semantic embedding S = (s1, s2, ..., s n ),in s i represents the i-th semantic feature, n is the length of the input sequence, and the code semantic matrix is ​​obtained Through the trainable parameter matrix Perform linear transformation: V = SM, reshaped into the semantic features of the label where d v Represents the dimension of the representation vector of each label node, d u Represents the dimension of semantic features, C is the number of label categories; The initial information V represented by the label k Enter the bidirectional tree LSTM input gate: i k =σ(W i ·[h p ,h l ,h r ,V k ]+b i ) where σ is the sigmoid activation function, W i is the weight matrix of the input gate, h p is the hidden state of the upper node, h l ,h r are the hidden states of the left and right child nodes, respectively, and b i is the bias of the input gate, and calculates V k Candidate state g k :g k =tanh(W g ·[h p ,h l ,h r ,V k ]+b g ), where tanh is the tanh activation function, W g is the weight matrix of the candidate state, b g is the bias of the candidate state; then calculate the forget gate f k =σ(W f ·[h p ,h l ,h r ]+b f ), where W f is the weight matrix of the forget gate, b f is the bias of the forget gate; Get cell state c k :c k =f k ·c p +i k ·g k , where c p is the cell state of the upper node, c k Summarize to the output gate to update the hidden node: h k =o k ·tanh(c k ), where o k =σ(W o ·[h p ,h l ,h r ,V k ]+b o ), b o is the bias of the output gate, W o is the weight matrix of the output gate; When the performance of LSTM no longer improves, stop updating the hidden nodes and use the final hidden node as the final V output.

7. The instability test classification detection method based on hierarchical perception as claimed in claim 6, characterized in that: The loss function in S103-4 is: L m =L c +λ·L r Where L m is the total loss, L c is the prediction error for each label, L r is the recursive regularization term, λ is the weight parameter; where w i , w j Represent the weight vectors of the i-th node and the j-th node respectively, C is the set of all nodes, and child(i) is the set of lower-level nodes of i; where y ij represents the true label, y′ ij Represents the predicted label.

Images