A classification detection method for instability testing based on hierarchical perception

Abstract

The present invention discloses an instability test classification detection method based on hierarchical perception, comprising the following steps: based on manual code analysis, obtaining a number of test code data showing instability; preprocessing the test code data; constructing and training an instability test classification detection model, the instability test classification detection model comprising a UniXcoder model pre-trained for extracting code semantic features, a structural encoder for extracting hierarchical structure features, and a classification model for classification; using the trained instability test classification detection model to predict new code data, outputting a prediction result of the stability of the code data, and if the predicted label is unstable, simultaneously outputting the hierarchical root cause of the instability test.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a method for detecting and analyzing the causes of instability in Flakytest (instability testing) use cases based on hierarchical features and semantic features. Background Art

[0002] In software development, code correctness is paramount, and software testing is a key step in verifying this. Developers use testing to ensure code accuracy and reliability. In traditional testing, test failures are often due to recent code changes. However, in real-world development, developers often encounter a specific type of test: unstable tests. These tests can exhibit different results in different executions, sometimes failing and sometimes succeeding, even when the test object and test conditions remain unchanged. This makes it difficult for developers to determine whether the problem lies in the code or the test itself is inherently unstable, which not only affects software reliability but can also lead to significant time-consuming code fixes. Therefore, test stability is a key factor in ensuring software quality.

[0003] To address this issue, there are currently a variety of tools and methods for detecting unstable tests. Early methods, such as rerunning the test suite under different configurations, detected test instability. However, this rerun-based detection method usually requires a lot of computing resources, which often results in high time and computing costs, especially in complex scenarios, where more than 1,000 reruns may be required. Moreover, simply determining whether a test is unstable is not enough to help developers understand the root cause of unstable tests. With the development of technology, researchers have begun to turn to static code analysis methods, trying to identify unstable tests by using the static characteristics of the test code without relying on tedious reruns.

[0004] While some progress has been made in this area, most existing tools and methods focus on detecting test instability without deeply analyzing the root causes of test instability, overlooking the hierarchical structure of instability. Research has shown that the causes of test instability can be categorized hierarchically, such as "order dependency" (OD) and "non-order dependency" (NOD). This hierarchical information is crucial for improving detection accuracy. Failure to effectively utilize this hierarchical information can lead to poor detection results, limiting their practical application. Another key issue is that existing methods typically treat the causes of instability as a single-label classification problem. However, instability tests often involve multiple root causes. For example, some tests may be affected by multiple factors, such as "shared state concurrency between components" and "failure to release acquired resources." This requires classification methods to be able to handle multi-label classification. Existing tools often fail to effectively address the complexities of these multiple root causes. Finally, many existing tools rely on statistical information or rules to determine whether a test is unstable, ignoring the semantic information of the test code itself. This results in poor performance when dealing with complex test cases and an inability to deeply identify potential causes of instability. Summary of the Invention

[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is to determine the test instability of static code as accurately as possible.

[0006] The present invention uses a variety of public instability test data sets to analyze static code based on the hierarchical code semantic features to determine whether it belongs to instability testing. If so, the root cause of the test instability is analyzed.

[0007] To solve the above technical problems, the present invention adopts the following technical solution: a hierarchical perception-based instability test classification detection method, comprising the following steps:

[0008] S101: Acquire a number of test code data that exhibit instability, where each piece of test code data consists of a test name, a test code, and a label, where the label is stable or unstable. If the label is unstable, the label also includes a hierarchical root cause of the instability test.

[0009] S102: pre-processing the test code data, removing harmful information of the test code in each test code data, and then performing word segmentation processing using the word segmenter of the byte-encoded BPE.

[0010] Then construct the directed graph Where V represents a set of nodes, each node represents a hierarchical root cause of instability testing, represents the set of directed edges from the nodes in the previous layer to the nodes in the next layer, represents the set of directed edges from the nodes in the next layer to the nodes in the previous layer, and calculates the prior probability of each hierarchical root cause of the instability test;

[0011] S103: Constructing and training an instability test classification detection model, wherein the instability test classification detection model includes a code semantic feature extraction model, a hierarchical structure feature extraction model, and a classification model.

[0012] S103-1: Code semantic feature extraction model converts the test code data pre-processed by S102 into a vector representation that can be processed by a computer and extracts the code semantic features Based on the calculation edge The prior probability of and the edge The prior probability of

[0013] S103-2: Hierarchical feature extraction model is used to extract hierarchical features At the same time and Converted into information V represented by labels.

[0014] S103-3: Take V as the input of the classification model and output the predicted label of the test code data. If the label is unstable, the hierarchical root cause of the instability test is also output.

[0015] S103-4: Use binary cross entropy loss as the loss function of the instability test classification detection model. When the loss no longer decreases, the trained instability test classification detection model is obtained.

[0016] S104: For a piece of code data, the code data is input into a trained instability test classification detection model, and the prediction result of the stability of the code data is output.

[0017] Furthermore, the calculation of P(U j ∣∣U i ) process is as follows:

[0018] Based on Bayesian statistical inference, the prior knowledge that depends on the label is regarded as the correlation between labels, and the edge features between label nodes are represented by prior probability:

[0019]

[0020] Among them, P(U j ∣∣U i ) represents the probability of node j appearing when node i appears, P(U j∩U i ) represents the probability of node j and node i appearing at the same time, U j and U i Respectively represent the occurrence of node j and node i, P(U i ) represents the probability of node i appearing, N i and N j Represent the number of times node j and node i appear, as a prior probability.

[0021] Furthermore, the code semantic features are extracted in S3-1. The process is:

[0022] The code semantic feature extraction model uses the pre-trained UniXcoder model. The UniXcoder model first converts each test code data processed by the S102 word segmenter into a token represented by a vector and the position of each token in the code; secondly, the token and the position of each token in the code are combined as the input of the BERT-based bidirectional Transformer model. in Represents the input information of the i-th test code data, x i Token representing the i-th test code, p i Indicates the position of the token of the i-th test code data in the code; finally, in the bidirectional Transformer model architecture based on BERT, after passing through N layers of Transformer layers, the context representation of each token is output

[0023] Furthermore, in S3-1 and The calculation process is as follows:

[0024]

[0025]

[0026] Furthermore, in S103-2, The process:

[0027] Based on the bidirectional tree LSTM, the hidden state of the transmission from the bottom point to the top node is calculated by combining the top-down and bottom-up information flow.

[0028]

[0029] Among them, child(k) is the set of lower-level nodes of node k, is the hidden state of the top layer, is the hidden state of the top node j, U k Indicates that node k appears.

[0030] Calculate the hidden state of the transfer from the top node to the bottom node

[0031]

[0032] in, is the underlying hidden state, is the hidden state of the bottom-level node p.

[0033] Will and Merge to get hierarchical features

[0034]

[0035] in Connections representing hidden states.

[0036] Furthermore, in S103-2, the process of obtaining the information V represented by the label is:

[0037] All the extracted code semantic features Encoded into a high-dimensional semantic embedding S = (s1, s2, ..., s n ),in s i Represents the i-th semantic feature, n is the length of the input sequence, and the code semantic matrix is ​​obtained Through the trainable parameter matrix Perform linear transformation: V = SM, reshaped into the semantic features of the label where d v Represents the dimension of the representation vector of each label node, d u Represents the dimension of semantic features, and C is the number of label categories.

[0038] The initial information V represented by the label k Enter the bidirectional tree LSTM input gate:

[0039] i k =σ(W i ·[h p ,h l ,h r ,V k ]+b i ) where σ is the sigmoid activation function, W i is the weight matrix of the input gate, h pis the hidden state of the upper node, h l ,h r are the hidden states of the left and right child nodes, respectively, b i is the bias of the input gate. And calculate V k Candidate state g k :g k =tanh(W g ·[h p ,h l ,h r ,V k ]+b g ), where tanh is the tanh activation function, W g is the weight matrix of the candidate state, b g is the bias of the candidate state; then calculate the forget gate f k =σ(W f ·[h p ,h l ,h r ]+b f ), where W f is the weight matrix of the forget gate, b f is the bias of the forget gate.

[0040] Get cell state c k :c k =f k c p +i k ·g k , where c p is the cell state of the upper node. k Summarize to the output gate to update the hidden node: h k =o k tanh(c k ), where o k =σ(W o ·[h p ,h l ,h r ,V k ]+b o ), b o is the bias of the output gate, W o is the weight matrix of the output gate.

[0041] When the performance of LSTM no longer improves, stop updating the hidden nodes and use the final hidden node as the final V output.

[0042] Furthermore, the loss function in S103-4 is:

[0043] L m =L c +λ·Lr

[0044] Among them L m is the total loss, L c The prediction error for each label, L r is the recursive regularization term, and λ is the weight parameter.

[0045]

[0046] where w i , w j They represent the weight vectors of the i-th node and the j-th node respectively, C is the set of all nodes, and child(i) is the set of lower-level nodes of i.

[0047]

[0048] where y ij represents the true label, y′ ij Represents the predicted label.

[0049] Compared with the prior art, the present invention has at least the following advantages:

[0050] (1) Accurate and efficient determination of instability test

[0051] By utilizing a hierarchical label structure, the model can capture the different causes of instability tests in a more fine-grained manner and improve the accuracy of detection through the propagation of hierarchical relationships. Traditional single-label models are often unable to handle complex dependencies between labels, while the hierarchical perception model of the present invention can handle such dependencies, thereby significantly improving the accuracy of instability test determination. In addition, compared with traditional re-run-based detection methods, static analysis methods are not only more efficient, but also do not rely on repeated test execution, which improves the efficiency and feasibility of testing.

[0052] (2) Analyze the root cause of instability

[0053] Traditional methods can usually only detect stability issues in tests, but cannot analyze the root causes of instability. However, the present invention uses hierarchical root cause analysis to enable the model to capture the dependencies between labels at multiple levels, significantly improving the effectiveness of classification tasks. It can provide a deeper understanding of instability tests and help developers find the specific source of the problem. For example, it can detect different types of root causes such as "sequential dependency" or "non-sequential dependency", thereby providing targeted improvement solutions for fixing test instability.

[0054] (3) Multi-label classification capability

[0055] Test code instability can be caused by multiple factors. This invention effectively addresses the coexistence of multiple root causes and provides multi-label classification capabilities. This allows identification of multiple factors that may affect test stability and provides developers with more comprehensive guidance.

[0056] (4) Enhanced model interpretability

[0057] Through the hierarchical labeling structure, testers can clearly see how a test sample progresses from low-level, specific issues (such as resource leaks and concurrency issues) to higher-level instability types (such as unstable network environments and test case timeouts). This hierarchical structure not only enhances the model's classification capabilities but also improves the interpretability of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a framework diagram of the method of the present invention.

[0059] Figure 2 Analyze the classification results for whether the label is unstable or stable.

[0060] Figure 3 Analysis of the classification results of labels belonging to unstable / non-sequential dependencies and their underlying labels.

[0061] Figure 4 Analysis of the classification results of labels belonging to unstable / sequential dependence and their underlying labels.

[0062] Figure 5 Analysis of the classification results for the label unstable / implementation-dependent and its underlying labels.

[0063] Figure 6 For comparison of experimental results.

[0064] Figure 7 This is the ablation experiment comparison of this method.

[0065] Figure 8 This is an example of a tag hierarchy. DETAILED DESCRIPTION

[0066] The present invention is described in further detail below.

[0067] The method of the present invention mainly determines the test instability of static code as accurately as possible through the following means:

[0068] (1) Acquisition and preprocessing of instability test data

[0069] This method first obtains test code data labeled "stable" or "unstable" from an existing instability test dataset. For instability test cases labeled "unstable," the hierarchical root causes of the instability (e.g., sequential dependency, non-sequential dependency, etc.) are manually annotated. This approach allows us to clearly identify the source of instability for each piece of data.

[0070] The key steps in the preprocessing part are:

[0071] Remove irrelevant or harmful information: Remove noise information or irrelevant parts in the test code and retain the core test code.

[0072] BPE word segmentation processing: Use byte pair encoding (BPE) to perform word segmentation and convert the test code into vocabulary units to provide standardized input for subsequent analysis.

[0073] (2) Identifying hierarchical features

[0074] This paper organizes the labels used to test code instability into a hierarchical structure, establishing dependencies between the labels to form a tree or directed acyclic graph (DAG) structure. This structure reflects the hierarchical relationships and dependencies between the labels. For example, if a test sample is associated with a node (a specific cause of instability), it is also implicitly associated with the node in the previous layer (a broader cause of instability).

[0075] (3) Using label dependencies of prior probabilities

[0076] By using prior probabilities to represent dependencies between tags, the present invention can adjust the weights of the relationships between tags based on their co-occurrence in the training data. This approach ensures that the model not only focuses on the current tag but also effectively utilizes the dependencies between tags, improving the accuracy of the decision.

[0077] (4) Efficient aggregation hierarchy and semantic features

[0078] When predicting test code, this paper uses semantic feature extraction and structural encoders to combine semantic information and the hierarchical structure of labels. Specifically, a pre-trained UniXcoder model is used to extract semantic features from the test code. Then, a hierarchically aware structural encoder, namely a bidirectional Tree-LSTM, propagates information from upper and lower nodes in the hierarchical structure, accurately capturing the relationship between labels at different levels. Ultimately, a comprehensive feature representation is generated, which is then input into the classification model.

[0079] (5) Adapt to multi-label classification

[0080] This classification model can effectively handle multi-label test code data: the model can identify whether test instability is caused by a single factor or multiple root causes working together, and ultimately optimizes through a binary cross-entropy loss function to achieve accurate prediction of unstable tests.

[0081] See also Figure 1 , a hierarchical perception-based instability test classification detection method, comprising the following steps:

[0082] S101: Acquire a number of test code data that exhibit instability, where each piece of test code data consists of a test name, a test code, and a label, where the label is stable or unstable. If the label is unstable, the label also includes a hierarchical root cause of the instability test.

[0083] S102: pre-processing the test code data, removing harmful information of the test code in each test code data, and then performing word segmentation processing using the word segmenter of the byte-encoded BPE.

[0084] Use the test code data and its corresponding labels as prior knowledge to build the classification model substructure, that is, to build a directed graph Where V represents a set of nodes, each node represents a hierarchical root cause of instability testing, represents the set of directed edges from the nodes in the previous layer to the nodes in the next layer, Represents the set of directed edges from nodes in the next layer to nodes in the previous layer, and calculates the prior probability of each hierarchical root cause of instability testing.

[0085] Test smells (a warning in test code that may indicate signs or patterns of a problem) are highly correlated with test instability. Specifically, certain test smells (e.g., Fire and Forget and AssertionRoulette) are more likely to cause test cases to exhibit instability across different executions. We call portions of code containing test smells "harmful information." We first annotate statements related to test smells to remove harmful information, and then tokenize the test code using a Byte Pair Encoding (BPE)-based tokenizer.

[0086] S103: Constructing and training an instability test classification detection model, wherein the instability test classification detection model includes a code semantic feature extraction model, a hierarchical structure feature extraction model, and a classification model.

[0087] S103-1: Code semantic feature extraction model converts the test code data pre-processed by S102 into a vector representation that can be processed by a computer and extracts the code semantic features Based on the calculation edge The prior probability of and the edge The prior probability of

[0088] S103-2: Hierarchical feature extraction model is used to extract hierarchical features at the same time and Converted into information V represented by labels.

[0089] S103-3: Take V as the input of the classification model and output the predicted label of the test code data. If the label is unstable, the hierarchical root cause of the instability test is also output.

[0090] S103-4: Use binary cross entropy loss as the loss function of the instability test classification detection model. When the loss no longer decreases, the trained instability test classification detection model is obtained.

[0091] S104: For a piece of code data, the code data is input into a trained instability test classification detection model, and the prediction result of the stability of the code data is output.

[0092] Specifically, the calculation of P(U j ∣∣U i ) process is as follows:

[0093] Based on Bayesian statistical inference, the prior knowledge that depends on the label is regarded as the correlation between labels, and the edge features between label nodes are represented by prior probability:

[0094]

[0095] Among them, P(U j ∣∣U i ) represents the probability of node j appearing when node i appears, P(U j ∩U i ) represents the probability of node j and node i appearing at the same time, U j and U i Respectively represent the occurrence of node j and node i, P(U i ) represents the probability of node i appearing, N i and N j Represent the number of times node j and node i appear respectively. As a prior probability, such as Figure 8 shown.

[0096] Specifically, the code semantic features are extracted in S3-1. The process is:

[0097] The code semantic feature extraction model uses the pre-trained UniXcoder model. The UniXcoder model first converts each test code data processed by the S102 word segmenter into a token represented by a vector and the position of each token in the code. Secondly, the token and the position of each token in the code are combined as the input of the BERT-based bidirectional Transformer model. in Represents the input information of the i-th test code data, x i Token representing the i-th test code, p i Indicates the position of the token of the i-th test code data in the code. Finally, in the bidirectional Transformer model architecture based on BERT, each Transformer block contains a multi-head attention network and a fully connected feedforward network, which can understand the context of the input code from both the left and right directions to capture comprehensive global semantic information. After N layers of Transformer layers, the output is the context representation of each token. That is, the semantic features of each Token.

[0098] Specifically, in S3-1 and The calculation process is as follows:

[0099]

[0100]

[0101] Indicates the data flow from the top layer to the bottom layer, Represents the data flow from the bottom layer to the top layer, ensuring that information is propagated from lower-level nodes to higher-level nodes.

[0102] Specifically, the step S103-2 obtains The process:

[0103] Based on the bidirectional tree LSTM, the hidden state of the transmission from the bottom point to the top node is calculated by combining the top-down and bottom-up information flow.

[0104]

[0105] Among them, child(k) is the set of lower-level nodes of node k, is the hidden state of the top layer, is the hidden state of the top node j, U k Indicates that node k appears;

[0106] Calculate the hidden state of the transfer from the top node to the bottom node

[0107]

[0108] in, is the underlying hidden state, is the hidden state of the bottom-level node p;

[0109] Will and Merge to get hierarchical features

[0110]

[0111] in Connections representing hidden states.

[0112] Specifically, in S103-2, the process of obtaining the information V represented by the label is as follows: Based on the UniXcoder model and the hierarchical perceptual encoder, an end-to-end hierarchical structure perceptual global model is constructed to capture the semantic features of the test code. Interaction with tags:

[0113] All the extracted code semantic features Encoded into a high-dimensional semantic embedding S = (s1, s2, ..., s n ),in s i Represents the i-th semantic feature, n is the length of the input sequence, and the code semantic matrix is ​​obtained Through the trainable parameter matrix Perform linear transformation: V = SM, map the semantic information of each test code data into the label space, align the semantic features with the semantic representation of the label, and reshape it into the semantic features of the label where d v Represents the dimension of the representation vector of each label node, d u Represents the dimension of semantic features, C is the number of label categories;

[0114] Then, the initial V is input into the bidirectional tree LSTM for propagation and feature update. Like the standard LSTM, each node in the TreeLSTM contains four gates: input gate (i k ), forget gate (f k ), candidate state (g k ), and the final output gate, which determines how the node's state is updated. The specific updates are as follows:

[0115] The initial information V represented by the labelk Enter the bidirectional tree LSTM input gate:

[0116] i k =σ(W i ·[h p ,h l ,h r ,V k ]+b i ) where σ is the sigmoid activation function, W i is the weight matrix of the input gate, h p is the hidden state of the upper node, h l ,h r are the hidden states of the left and right child nodes, respectively, b i is the bias of the input gate. And calculate V k Candidate state g k :g k =tanh(W g ·[h p ,h l ,h r ,V k ]+b g ), where tanh is the tanh activation function, W g is the weight matrix of the candidate state, b g is the bias of the candidate state; then calculate the forget gate f k =σ(W f ·[h p ,h l ,h r ]+b f ), where W f is the weight matrix of the forget gate, b f is the bias of the forget gate.

[0117] Get cell state c k :c k =f k c p +i k ·g k , where c p is the cell state of the upper node. k Summarize to the output gate to update the hidden node: h k =o k tanh(c k ), where o k =σ(W o ·[h p ,h l ,h r ,V k ]+b o ), bo is the bias of the output gate, W o is the weight matrix of the output gate;

[0118] When the performance of LSTM stops improving, the hidden node update stops and the final hidden node is output as the final V. Finally, V processed by the bidirectional tree LSTM is used as the input to the classification model for the final label prediction.

[0119] This transformation enables the model to utilize these semantic features for deeper propagation and updates in the hierarchical perceptual encoder: the initial node representation V is passed into the hierarchical perceptual encoder, and the resulting hidden state h embodies the semantic features of the hierarchical perception, which is then used as the input of the final classification model. This approach ensures that the hierarchical perceptual structure encoder can operate during both training and inference, thereby promoting the propagation of semantic information between layers.

[0120] Specifically, the loss function in S103-4 is:

[0121] L m =L c +λ·L r

[0122] Among them L m is the total loss, L c The prediction error for each label, L r is the recursive regularization term, and λ is the weight parameter.

[0123]

[0124] where w i , w j They represent the weight vectors of the i-th node and the j-th node respectively, C is the set of all nodes, and child(i) is the set of lower-level nodes of i.

[0125]

[0126] where y ij represents the true label, y′ ij Represents the predicted label.

[0127] Before multi-label classification, the label hierarchy is flattened and all nodes are treated as leaf nodes. Secondly, define C as the set of all nodes, and for all child nodes child(i) of node i, use the parent-child node weight vector w i , w j , introduced The recursive regularization term reduces the difference between node parameters at the same level and maintains the relationship between label levels; then based on the true value (0 or 1) y of the i-th sample at the corresponding j-th label node ij The probability value y′ obtained by the sigmoid activation function of the i-th sample on the corresponding j-th label node ij

Note that nodes are labels

[0128] Experiment and analysis:

[0129] The test code use case dataset selected in the present invention comes from the dataset contributed by IDoFT, FlakeFlagger, and Gruber et al. on GitHub. By summarizing the modification requests in the GitHub Pull Request and manual code analysis, the causes of the instability of each test are manually obtained as labels. These label hierarchies are derived from the classification systems of multiple latest studies. Among them, the IDoFT dataset selects 2455 test cases, including 9 hierarchical labels such as Flaky, OD, and ID; the FlakeFlagger dataset selects 5450 test cases, including two labels of Flaky and Not Flaky; the dataset contributed by Gruber et al. on GitHub selects 2885 test cases, including 34 hierarchical labels such as Flaky, OD, and NOD.

[0130] Randomly select 60% of all test cases of each label as training data, 20% as validation data, and the remaining 20% ​​as test data.

[0131] The evaluation indicators are precision, recall, Macro-F1 and Micro-F1.

[0132]

[0133] Where T flaky represents the set of actual flaky tests, Indicates that the model predictions are flaky on the test set.

[0134]

[0135]

[0136]

[0137] Where |L| represents the total number of labels, F 1i represents the F1 score for each label.

[0138] To further verify the effectiveness of this invention, we compared it with popular mainstream recommendation algorithms, which are as follows:

[0139] FlakeFlagger: A machine learning-based prediction method that combines static and dynamic features to predict a binary classification algorithm for instability testing.

[0140] Flakify: A prediction method based on black-box language models that uses CodeBERT to generate embeddings for test code and predict a binary classification algorithm for instability tests.

[0141] The hierarchical perception-based instability test classification detection method proposed in the present invention is called HiFlaky.

[0142] Experimental results show that the present invention is significantly superior to other recommendation algorithms in terms of both Precision and F1-score, with an accuracy improvement of 30% on the same dataset and a MicroF1 score of 0.812, indicating that the overall accuracy of all test instances is high, highlighting the effectiveness of the model in consistently detecting more frequent categories. This shows that the instability test classification detection method combined with hierarchical perception proposed in this paper is effective. FlakeFlagger only selects features with information gain (IG) values ​​greater than 0.01 for training. Its reliance on the acquisition of static and dynamic features limits its application in certain projects, and its generalization ability is limited by the distribution of training data, and it performs poorly on other general-purpose datasets. Flakify relies entirely on the semantic and grammatical information of the test code, which affects the accuracy of the prediction when the test code cannot fully reflect the characteristics of the production code.

[0143] This paper incorporates multi-level perception into the detection of instability tests, refining the detection of instability tests from a hierarchical perspective, effectively capturing label dependencies and enriching the feature representation of complex test scenarios. Although HiFlaky is not specifically designed for binary classification, its adaptation to this task demonstrates excellent performance, particularly in terms of accuracy and F1 score. This success is largely due to its hierarchical structure, which helps learn label dependencies and improve classification accuracy.

[0144] In short, the present invention proposes a classification and detection method for instability tests based on hierarchical perception. On the one hand, it improves the recognition accuracy of instability tests, and on the other hand, it solves the problem that existing methods cannot handle multiple root causes and hierarchical information, accurately locates the crux of the problem, and is superior to the current popular mainstream recommendation algorithms. It can be applied to actual development scenarios, provide framework suggestions or implement automatic repairs for software developers, improve software reliability and simplify testing.

[0145] This method evaluates each label:

[0146] See also Figure 2 , the Precision of unstable labels is 96%, Recall is 93%, and F1 is 95%, while the Precision of stable labels is 94%, Recall is 98%, and F1 is 96%, showing higher accuracy and recall, and a good balance between the two.

[0147] See also Figure 3The Precision of the non-sequential dependency label is 89%, the Recall is 70%, and the F1 is 78%, indicating that the model can effectively identify non-sequential dependency issues. The Precision of the concurrent label is 50%, the Recall is 40%, and the F1 is 44%, indicating that the model's performance on this label is average, with low accuracy and recall. The Precision of the concurrent / asynchronous wait label is 53%, the Recall is 40%, and the F1 is 46%, similar to the concurrent label, with low accuracy and limited recall. For input and output, the model's performance is more complex. Although its Precision reaches 75%, its Recall is low, only 15%, and its F1 is 25%. This shows that its accuracy is relatively good in some scenarios, but its recall ability is weak. The Precision of the network label is 70%, the Recall is 56%, and the F1 is 62%, indicating that the model is relatively accurate in identifying network-related issues, but the recall rate still needs improvement. The Precision for the Network / Local label was 100%, but the Recall was 80% and the F1 was 89%, indicating that the model's accuracy for this label was very high, but there were still some missed detections. For the Other label, the Precision was 77%, the Recall was 33%, and the F1 was 47%, indicating that the model's performance for this label was average, with high accuracy but weak recall. The Precision for the Other / Program Logic label was 50%, the Recall was 20%, and the F1 was 29%, indicating weak performance and room for improvement in both precision and recall. The Precision for the Other / Unknown label was 100%, the Recall was 36%, and the F1 was 53%, indicating high accuracy for this label but low recall, indicating that the model may have missed detections for this type of problem. Finally, the Precision for the Platform-Dependent label was 80%, the Recall was 27%, and the F1 was 40%, indicating that the model's accuracy for this label was high but its recall was insufficient. The platform-dependent / runtime label achieved a Precision of 100%, a Recall of 27%, and an F1 of 42%, demonstrating very high accuracy, but low recall, potentially failing to identify all platform-related test instabilities. The randomness label achieved a Precision of 59%, a Recall of 53%, and an F1 of 56%, showing a relatively balanced performance. The randomness / data label achieved a Precision of 64%, a Recall of 45%, and an F1 of 53%, demonstrating high accuracy, but potential for improvement in recall. The time label achieved a Precision of 46%, a Recall of 40%, and an F1 of 43%, demonstrating low precision and recall. The time / system date label achieved a Precision of 100%, a Recall of 33%, and an F1 of 50%, indicating excellent accuracy but weak recall.The Precision of the time / test timeout label is 34%, Recall is 24%, and F1 is 28%, indicating that the model has weak recognition ability for this label.

[0148] See also Figure 4 In the sequential dependency label analysis, the sequential dependency label has a Precision of 80%, a Recall of 80%, and an F1 of 80%. The precision and recall of this label are very close, showing a balanced performance. The Precision of the input-output label is 75%, the Recall is 15%, and the F1 is 25%, indicating that the model has high accuracy in identifying input-output related issues, but the Recall is low, and there are some missed detections. The Precision of the resource leakage label is 81%, the Recall is 40%, and the F1 is 54%. This label shows high accuracy, but weak recall ability, and may not identify all related issues. The Precision of the shared state label is 78%, the Recall is 88%, and the F1 is 83%, which is a relatively balanced performance with high precision and recall, indicating that the model can effectively identify instabilities related to shared states.

[0149] See also Figure 5 The implementation-dependent label achieved a Precision of 71%, a Recall of 77%, and an F1 of 74%, indicating that its accuracy and recall were relatively balanced, but its overall performance was mediocre. The API behavior label, a lower-level label for implementation-dependency, achieved a Precision of 92%, a Recall of 62%, and an F1 of 74%, demonstrating high accuracy but low recall, suggesting that the model may miss some relevant issues. The Data Structure label achieved lower Precision and Recall of 23% and 35%, respectively, with an F1 of 28%, indicating that the model was less effective in identifying instabilities related to data structure.

[0150] See also Figure 6When comparing the performance of the three experimental methods (HiFlaky, Flakify, and FlakeFlagger), HiFlaky's Precision is 89%, the highest of the three, indicating that it performs best in accurately identifying test instabilities. However, HiFlaky's Recall is low, only 57%, resulting in an F1 score of 69%, which is in the middle of the three methods and shows that its balance between Precision and Recall is relatively moderate. In contrast, Flakify's Precision is low at 65%, indicating that its accuracy is insufficient. Despite its high Recall (85%), its F1 score is 74%, still indicating that its overall recognition performance is not ideal and there is a large imbalance. FlakeFlagger's Precision is 61%, the lowest of the three, indicating that its accuracy is the worst. Although FlakeFlagger's Recall is 73%, higher than HiFlaky, its low Precision results in an F1 score of only 66%, which is average performance. It fails to find a good balance between Precision and Recall, and its overall performance is relatively lacking.

[0151] See also Figure 7 In the ablation experiment, our HiFlaky performed best using Unixcoder, with a Micro-F1 value of 0.8119, significantly higher than the other two configurations, indicating that it has the best overall performance across all categories. Its Macro-F1 value is 0.4141, also relatively high, indicating good balance across categories. The Micro-F1 using Codebert is 0.7795, slightly lower than Unixcoder, indicating that it is slightly inferior in identifying test instability. Its Macro-F1 value is 0.2612, indicating that the model has poor balance across different categories. The Micro-F1 without incorporating hierarchical information is 0.6880, which is the worst performance, especially with a significant decline in overall recognition effect. Its Macro-F1 value is 0.1851, further reflecting that after removing hierarchical features, the model's performance is unbalanced across categories. This ablation experiment shows that using Unixcoder has better performance.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A hierarchical perception-based instability test classification detection method, characterized by: The steps include: S101: Acquire a number of test code data items that exhibit instability. Each test code data item consists of a test name, a test code, and a label. The labels are stable and unstable. If the label is unstable, the label also includes a hierarchical root cause of the instability test. S102: pre-processing the test code data, removing harmful information from each test code data, and then performing word segmentation processing using the word segmenter of the byte-encoded BPE; Then construct the directed graph Where V represents a set of nodes, each node represents a hierarchical root cause of instability testing, represents the set of directed edges from the nodes in the previous layer to the nodes in the next layer, represents the set of directed edges from the nodes in the next layer to the nodes in the previous layer, and calculates the prior probability of each hierarchical root cause of the instability test; S103: Constructing and training an instability test classification detection model, wherein the instability test classification detection model includes a code semantic feature extraction model, a hierarchical structure feature extraction model, and a classification model; S103-1: Code semantic feature extraction model converts the test code data pre-processed by S102 into a vector representation that can be processed by a computer and extracts the code semantic features Based on the calculation edge The prior probability of and the edge The prior probability of S103-2: Hierarchical feature extraction model is used to extract hierarchical features At the same time and Converted into information V represented by labels; S103-3: Take V as the input of the classification model and output the predicted label of the test code data. If the label is unstable, the hierarchical root cause of the instability test is also output; S103-4: Use binary cross entropy loss as the loss function of the instability test classification detection model. When the loss stops decreasing, the trained instability test classification detection model is obtained. S104: For a piece of code data, the code data is input into a trained instability test classification detection model, and the prediction result of the stability of the code data is output.

2. The hierarchical perception-based instability test classification detection method according to claim 1, characterized in that: In the step S102, P(U j ∣∣U i ) process is as follows: Based on Bayesian statistical inference, the prior knowledge that depends on the label is regarded as the correlation between labels, and the edge features between label nodes are represented by prior probability: Among them, P(U j ∣∣U i ) represents the probability of node j appearing when node i appears, P(U j ∩U i ) represents the probability of node j and node i appearing at the same time, U j and U i Respectively represent the occurrence of node j and node i, P(U i ) represents the probability of node i appearing, N i and N j Represent the number of times node j and node i appear, as a prior probability.

3. The instability test classification detection method based on hierarchical perception according to claim 2, characterized in that: Extracting code semantic features in S3-1 The process is: The code semantic feature extraction model uses the pre-trained UniXcoder model. The UniXcoder model first converts each test code data processed by the S102 word segmenter into a token represented by a vector and the position of each token in the code; secondly, the token and the position of each token in the code are combined as the input of the BERT-based bidirectional Transformer model. in Represents the input information of the i-th test code data, x i Token representing the i-th test code, p i Indicates the position of the token of the i-th test code data in the code; finally, in the bidirectional Transformer model architecture based on BERT, after passing through N layers of Transformer layers, the context representation of each token is output 4. The hierarchical perception-based instability test classification detection method according to claim 3, characterized in that: In S3-1 and The calculation process is as follows:

5. The hierarchical perception-based instability test classification detection method according to claim 4, characterized in that: Obtained in S103-2 The process: Based on the bidirectional tree LSTM, the hidden state of the transmission from the bottom point to the top node is calculated by combining the top-down and bottom-up information flow. Among them, child(k) is the set of lower-level nodes of node k, is the hidden state of the top layer, is the hidden state of the top node j, U k Indicates that node k appears; Calculate the hidden state of the transfer from the top node to the bottom node in, is the underlying hidden state, is the hidden state of the bottom-level node p; Will and Merge to get hierarchical features in Connections representing hidden states.

6. The hierarchical perception-based instability test classification detection method according to claim 5, characterized in that: In S103-2, the process of obtaining the information V represented by the label is as follows: All the extracted code semantic features Encoded into a high-dimensional semantic embedding S = (s1, s2, ..., s n ),in s i Represents the i-th semantic feature, n is the length of the input sequence, and the code semantic matrix is ​​obtained Through the trainable parameter matrix Perform linear transformation: V = SM, reshaped into the semantic features of the label where d v Represents the dimension of the representation vector of each label node, d u Represents the dimension of semantic features, C is the number of label categories; The initial information V represented by the label k Enter the bidirectional tree LSTM input gate: i k =σ(W i ·[h p ,h l ,h r ,V k ]+b i ) where σ is the sigmoid activation function, W i is the weight matrix of the input gate, h p is the hidden state of the upper node, h l ,h r are the hidden states of the left and right child nodes, respectively, b i is the bias of the input gate, and calculate V k Candidate state g k :g k =tanh(W g ·[h p ,h l ,h r ,V k ]+b g ), where tanh is the tanh activation function, W g is the weight matrix of the candidate state, b g is the bias of the candidate state; then calculate the forget gate f k =σ(W f ·[h p ,h l ,h r ]+b f ), where W f is the weight matrix of the forget gate, b f is the bias of the forget gate; Get cell state c k :c k =f k c p +i k ·g k , where c p is the cell state of the upper node, c k Summarize to the output gate to update the hidden node: h k =o k tanh(c k ), where o k =σ(W o ·[h p ,h l ,h r ,V k ]+b o ), b o is the bias of the output gate, W o is the weight matrix of the output gate; When the performance of LSTM no longer improves, stop updating the hidden nodes and use the final hidden node as the final V output.

7. The hierarchical perception-based instability test classification detection method according to claim 6, characterized in that: The loss function in S103-4 is: L m =L c +λ·L r Among them L m is the total loss, L c The prediction error for each label, L r is the recursive regularization term, λ is the weight parameter; where w i , w j Represent the weight vectors of the i-th node and the j-th node respectively, C is the set of all nodes, and child(i) is the set of lower-level nodes of i; where y ij represents the true label, y′ ij Represents the predicted label.

Images