A recurrent neural network component test case selection method based on clustering and uncertainty
By employing a clustering and uncertainty-based test case selection method, and utilizing HDBSCAN and minimum confidence sampling to screen efficient test cases, the problem of low fault detection rate in recurrent neural network component testing is solved, achieving cost reduction and efficiency improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2023-04-11
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies suffer from low fault detection rates, poor inclusiveness, and a lack of diversity in the selection of test cases for recurrent neural network components, resulting in high costs for test case annotation.
A test case selection method based on clustering and uncertainty recurrent neural network components is adopted. Through HDBSCAN clustering and minimum confidence sampling, combined with the hidden state change rate, an efficient set of test cases is selected.
It effectively reduced the cost of test case annotation, improved the fault detection rate, inclusiveness and diversity, and optimized the testing efficiency of recurrent neural network components.
Smart Images
Figure CN116450513B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of software testing technology, specifically to a test case selection method for recurrent neural network components based on clustering and uncertainty. By selecting test cases for recurrent neural network components in software, the method reduces the cost of labeling large-scale test cases. Background Technology
[0002] Recurrent neural networks (RNNs), as innovation-driven components, have been integrated into application software across many different fields. Their security and quality issues have also attracted significant attention from academia and industry. One of the most common methods to ensure the quality of RNN-driven software systems is to test and optimize the RNN components. The core idea is to invest heavily in annotating test cases, thereby triggering faults in the RNN components using a large number of labeled test cases. However, this incurs a huge overhead for testing software with RNN components. To alleviate the problem of high overhead in annotating test cases for RNN components, existing research focuses on selecting test cases most likely to trigger faults in the RNN components for annotation.
[0003] Recently, the paper DeepState (Z. Liu, Y. Feng, Y. Yin, et al. DeepState: Selecting TestSuites to Enhance the Robustness of Recurrent Neural Networks[C] / / 2022IEEE / ACM 44th International Conference on Software Engineering (ICSE).2022.) proposed a test case selection method for recurrent neural network components. This method selects data based on the state perspective of the recurrent neural network component. By capturing the state changes of neurons in the recurrent neural network component model, it analyzes the rate of change and trend of change of hidden states in the recurrent neural network component model to identify test cases that may trigger faults in the recurrent neural network component model, thereby effectively selecting test cases in large-scale unlabeled datasets. However, the test cases selected by this method still suffer from problems such as low fault detection rate, poor inclusiveness, and lack of diversity. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention proposes a test case selection method for recurrent neural network components based on clustering and uncertainty, RNNtcs, which can select as many small-scale test cases as possible from large-scale test cases that can detect faults in the recurrent neural network model under test.
[0005] The technical solution of this invention is as follows:
[0006] The method for selecting test cases for recurrent neural network components based on clustering and uncertainty includes the following steps:
[0007] Step 1: Use the recurrent neural network component under test to predict the original test case set and obtain the output confidence features corresponding to each test case; then perform feature processing on the output confidence features; then perform HDBSCAN clustering calculation on the processed features to obtain the test cases D.cluster corresponding to the normal cluster and the test cases D.noise corresponding to the outliers;
[0008] Step 2: Based on the HDBSCAN clustering results, calculate the uncertainty scores of the test cases corresponding to the normal clusters and the test cases corresponding to the outliers after clustering. Based on the predicted uncertainty score threshold, pre-screen the test case set.
[0009] Step 3: For each test case in the test case set obtained in Step 2, calculate the hidden state change rate when the recurrent neural network component under test predicts each test case. Sort the test cases according to the hidden state change rate to obtain the first test case set sorted by the probability of triggering the failure of the recurrent neural network component under test. Select test cases from the first test case set in order to obtain the final required test case set DataSelected.
[0010] Furthermore, in step 1, the feature processing of the output confidence features includes feature normalization and feature dimensionality reduction.
[0011] Furthermore, in step 1, the process of normalizing the output confidence features is as follows:
[0012] Suppose the original test case set is represented as T, and its size is denoted as st, T = {t1, t2, ..., t}. st}, t i The eigenvectors are denoted as F i ={f i1 ,f i2 ,...f ir}, using variable x before normalization ij to represent t i j th The value of the feature, and using variables To represent the normalized t i j th The value of the feature;
[0013] 1≤i≤st and 1≤j≤r.
[0014] Furthermore, in step 1, the FastICA algorithm is used for feature dimensionality reduction.
[0015] Furthermore, in step 2, the uncertainty score of the test cases is calculated using the lowest confidence level sampling method.
[0016] Furthermore, in step 2, the lowest confidence level sample is the difference between the highest confidence level prediction and the 100% confidence level.
[0017] Furthermore, in step 2, test cases are first selected from the test cases corresponding to outliers, and the uncertainty score of all test cases corresponding to outliers is calculated. Then, test cases with uncertainty scores greater than a threshold are pre-screened. If the size of the pre-screened test cases with uncertainty scores greater than the threshold is greater than or equal to the size n of the final test case set to be selected, then a pre-screened test case set is obtained. If the size of the pre-screened test cases with uncertainty scores greater than the threshold is less than the size n of the final test case set to be selected, then the uncertainty score of the test cases corresponding to normal clusters is further calculated, and test cases with uncertainty scores greater than the threshold are pre-screened.
[0018] Furthermore, in step 2, the threshold is 0.01.
[0019] Furthermore, in step 3, the test cases are sorted in reverse order from high to low according to the hidden state change rate, thereby obtaining the first test case set with the probability of triggering the failure of the recurrent neural network model under test in a forward order. Test cases are then selected from the first test case set in order from high to low according to the hidden state change rate to obtain the final required test case set.
[0020] Furthermore, in step 3, if the size of the test cases with uncertainty scores greater than the threshold in the test cases corresponding to the pre-selected outliers is greater than or equal to the size n of the final test case set to be selected, then the test cases with uncertainty scores greater than the threshold in the test cases corresponding to the pre-selected outliers are sorted according to the hidden state change rate, and then the test cases are selected in order to obtain the final required test case set DataSelected.
[0021] If the number of test cases with uncertainty scores greater than the threshold in the pre-selected outlier test cases is smaller than the final required test case set size n, then the test cases with uncertainty scores greater than the threshold in the pre-selected outlier test cases will be included as part of the final required test case set DataSelected.
[0022] Then, test cases with uncertainty scores greater than the threshold that are pre-screened in each normal cluster are sorted according to the hidden state change rate, and test cases are selected sequentially from the pre-screened test cases in each normal cluster according to the proportion, forming another part of the final required test case set DataSelected; the proportion is the percentage of the number of pre-screened test cases in each normal cluster to the total number of pre-screened test cases in all normal clusters.
[0023] Beneficial effects
[0024] The test case selection method RNNtcs proposed in this invention, based on clustering and uncertainty, can prioritize the selection of a small subset of test cases with high error testing capabilities from a large number of test cases, thereby reducing the cost of test case annotation and enabling more efficient testing and optimization of recurrent neural network components.
[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0027] Figure 1 It is the overall architecture of a test case selection method based on clustering and uncertainty in recurrent neural networks. Detailed Implementation
[0028] This invention designs a test case selection method for recurrent neural networks (RNNtcs) based on clustering and uncertainty, which can select as many small-scale test cases as possible from large-scale test cases that can detect faults in the recurrent neural network model under test.
[0029] This method first uses the recurrent neural network model under test to predict the output confidence features of the original test case set, which serve as the feature representation of all test cases in the original test case set. Then, feature processing is performed on these output confidence features based on the data characteristics of these test case output confidence features. Next, HDBSCAN (Hierarchical Density-Based Spatial Clustering of Noisy Applications) clustering calculation is performed based on the processed features.
[0030] Secondly, based on the HDBSCAN clustering results, the uncertainty score of each test case in different normal clusters and outliers after clustering is calculated, and the test case set is pre-screened according to the uncertainty score threshold.
[0031] Finally, for the pre-selected test case set, the hidden state change rate when the recurrent neural network component under test predicts each test case is calculated. The required test case set is then selected using this hidden state change rate. Alternatively, test cases can be sorted in reverse order of hidden state change rate from high to low to obtain a test case set that is sorted in forward order of probability of triggering a fault in the recurrent neural network model under test. Test cases are then selected sequentially from this test case set according to their hidden state change rate from high to low to obtain the required test case set. Experiments demonstrate that this method can effectively select test cases from a large-scale test case set that can trigger recurrent neural network faults, thereby reducing annotation costs.
[0032] The specific steps are as follows:
[0033] Step 1: Test case identification based on HDBSCAN clustering. This stage includes two inputs and two outputs:
[0034] Input: Original test case set D = d0, d1, ..., d n The recurrent neural network component model M to be tested;
[0035] Output: Test case D.cluster corresponding to normal clusters and test case D.noise corresponding to outliers after clustering.
[0036] Step 1.1: Clustering Input Feature Selection. The recurrent neural network component model gradually learns features from the test case input to predict labels. Layers closer to the input layer represent more basic features, while layers closer to the output layer represent more higher-order features. Comparatively, more higher-order features can more accurately capture the relationship between the test case input and the label, while basic features cannot reflect the complex input feature patterns for predicting labels. Considering that the output layer features are closely integrated with both the model and the test case input, and can more fully reflect the performance of the test case input in the model, this invention selects the confidence features of the output layer when the recurrent neural network component model predicts test cases as the clustering input features.
[0037] Step 1.2: Normalize the output confidence features.
[0038] Before clustering, the confidence features of the output layer of the recurrent neural network component model predicting test cases are first normalized to adjust the measurements at different scales to a common scale. Since the confidence features of the output layer of the recurrent neural network component model predicting these test cases are all numerical, min-max normalization is used to adjust each value of these features to the interval [0,1]. Assume a complete test case set is represented as T, with its size denoted as st, and the test case set in T is represented as T = {t1, t2, ..., t...}. st}, ti The eigenvectors are denoted as F i ={f i1 ,f i2 ,...f ir}, using variable x before normalization ij to represent t i j th The value of the feature, and using variables To represent the normalized t i j th The values of the features are (1≤i≤st and 1≤j≤r). Formula 1 is the normalization calculation method.
[0039]
[0040] Step 1.3: Output Confidence Feature Dimensionality Reduction. The performance of HDBSCAN can significantly decrease as the dimensionality of features increases. Therefore, dimensionality reduction of high-dimensional features is performed before clustering. This invention uses the FastICA algorithm for dimensionality reduction. The goal of FastICA is to find independent components and help identify potential factors by maximizing the negative entropy defined in Equation 2. In Equation 2, Y... Gauss Y is a Gaussian random variable with the same variance as the random variable Y. E[.] is used to calculate the average value, and g(.) is a nonlinear function used to approximate the differential entropy.
[0041] N g (Y)={E[g(Y))-E[g(Y Gauss )]]} 2 (Formula 2)
[0042] Step 1.4: Output Confidence Feature Clustering. The HDBSCAN algorithm is used to cluster the output layer confidence features of the normalized and dimensionality-reduced test cases. HDBSCAN divides the features corresponding to the test cases into different clusters, and test cases in different clusters are more likely to have different testing capabilities. In addition, some test cases do not cluster into any cluster, forming a minority space, i.e., outliers.
[0043] Step 1.5: Output the test case D.cluster corresponding to the normal clusters and the test case D.noise corresponding to the outliers after clustering.
[0044] Step 2: Test case selection based on prediction uncertainty. This stage includes four inputs and two outputs:
[0045] Input: The recurrent neural network model M to be tested, the number of test cases n to be selected, the test cases D.cluster corresponding to the normal clusters after clustering, and the test cases D.noise corresponding to the outliers;
[0046] Output: The pre-filtered test case set D.cluster.LS for normal clusters and the pre-filtered test case set D.noise.LS for outliers.
[0047] Step 2.1: Confidence Sampling Method Selection. To select test cases that reflect the overall prediction uncertainty of the recurrent neural network component model, the least confidence score is used to calculate the uncertainty score of the test cases. The least confidence score is the difference between the highest confidence prediction and the 100% confidence score. The least confidence score represents the degree of uncertainty of the prediction result, and its calculation formula is as follows:
[0048]
[0049] Step 2.2: Calculate the uncertainty and pre-screen test cases. Calculate the uncertainty score for each test case corresponding to the normal clusters and outliers after clustering using the lowest confidence level sampling. Then, pre-screen the test cases based on the uncertainty score threshold.
[0050] First, select test cases from the test cases corresponding to the cluster outliers. Calculate the prediction uncertainty score of all test cases corresponding to the outliers according to Formula 3. Then, pre-screen the test cases with uncertainty scores greater than 0.01. If the size of the pre-screened test cases with uncertainty scores greater than 0.01 is greater than or equal to the final size n of the test case set to be selected, then the pre-screened test case set is obtained.
[0051] If the number of test cases with an uncertainty score greater than 0.01 in the pre-selected outliers is smaller than the size n of the required test case set, then the uncertainty score of the test cases corresponding to the normal clusters is further calculated according to Formula 3, and test cases with an uncertainty score greater than 0.01 in the pre-selected test cases corresponding to the normal clusters are pre-selected.
[0052] Step 2.3: Output the pre-filtered test case set D.noise.LS for test cases corresponding to outliers; or output the pre-filtered test case set D.cluster.LS for test cases corresponding to normal clusters and the pre-filtered test case set D.noise.LS for test cases corresponding to outliers.
[0053] Step 3: Test case ranking based on hidden state change rate. This stage includes four inputs and one output:
[0054] Input: The recurrent neural network model M to be tested, the number of test cases n to be selected, the test case set D.cluster.LS after pre-screening of test cases corresponding to normal clusters, and the test case set D.noise.LS after pre-screening of test cases corresponding to outliers;
[0055] Output: Test case set DataSelected.
[0056] The hidden state change rate, based on the state perspective of a recurrent neural network (RNN), calculates the rate of change of the predicted labels of the hidden states within the RNN, reflecting the uncertainty of the hidden layers within the RNN model. Calculating the hidden state change rate for each test case in the test case set yields the degree of uncertainty of the hidden layers within the RNN for each test case. This degree of uncertainty indicates the probability that a test case will trigger a failure in the RNN model. The hidden state change rate is defined as follows:
[0057]
[0058] For the test cases pre-selected in step 2, if the size of the test cases with an uncertainty score greater than 0.01 corresponding to the pre-selected outliers is greater than or equal to the size n of the final test case set to be selected, then Formula 4 is used to directly calculate the hidden state change rate of the test cases with an uncertainty score greater than 0.01 corresponding to the pre-selected outliers, and the test cases are sorted in reverse order from high to low according to the hidden state change rate, thereby obtaining the first test case set in the forward order of the probability of triggering the failure of the recurrent neural network model under test. Then, n test cases are selected sequentially from the first test case set according to the hidden state change rate from high to low to obtain the final required test case set DataSelected.
[0059] If the number of test cases with an uncertainty score greater than 0.01 in the pre-selected outlier test cases is smaller than the final required test case set size n, then the test cases with an uncertainty score greater than 0.01 in the pre-selected outlier test cases will be included as part of the final required test case set DataSelected.
[0060] Next, use Formula 4 to calculate the hidden state change rate of test cases with an uncertainty score greater than 0.01 pre-selected in each normal cluster. For the test cases with an uncertainty score greater than 0.01 pre-selected in each normal cluster, sort them from high to low according to the hidden state change rate, and then select test cases in order from the test cases with an uncertainty score greater than 0.01 pre-selected in each normal cluster according to the proportion, forming another part of the final test case set DataSelected. The proportion is the percentage of the number of test cases pre-selected in each normal cluster to the total number of test cases pre-selected in all normal clusters.
[0061] The two parts are combined to obtain the final set of test examples, DataSelected.
[0062] Results Evaluation
[0063] The effectiveness of the clustering and uncertainty-based recurrent neural network test case selection method designed in this invention is verified by comparing it with the latest test case selection method for recurrent neural networks (DeepState).
[0064] 1. Model Evaluation Datasets. In this invention, the performance of the LSTM model (a variant of a recurrent neural network) is evaluated on four datasets: MNIST, Fashion, CIFAR-10, and SVHN. The initial performance of the LSTM models trained on the four datasets is shown in Table 1.
[0065] Table 1 Distribution of the performance evaluation experiment dataset
[0066] Dataset Training set size Test set size Accuracy (%) Mnist 60,000 10,000 98.92 Fashion 60,000 10,000 90.59 CIFAR-10 50,000 10,000 53.65 SVHN 73,257 26,032 88.66
[0067] 2. Evaluation Metrics. To effectively evaluate the performance of different methods, a multi-dimensional evaluation is conducted, including fault detection rate, fault detection coverage, fault detection diversity, and model optimization effectiveness. The specific calculation formulas for these evaluation metrics are as follows:
[0068] • Fault Detection Rate: This parameter represents the ability of the test case set selected by the test case selection method to detect errors in recurrent neural networks. Similar to traditional software testing, for a given test case selection method, a selected test case set that triggers more faults means it can reveal more defects in the software. The goal of test case selection in deep learning models is also to select test cases that may trigger potential defects. The original test case set is represented as D. test The selected test case set is represented as D. select Test cases that are misclassified are represented as D. select,bug ,|Dselect,bug | represents the number of test cases that were misclassified, |D select | indicates the size of the selected test case set. The specific calculation formula is:
[0069]
[0070] • Fault Detection Inclusivity: This parameter represents the extent to which the test case selection method reveals potential faults in the recurrent neural network model. The original test case set is represented as D. test The selected test case set is represented as D. select And will be able to detect D test Test cases for erroneous behavior are represented as D. test,bug It can detect D select Test cases for erroneous behavior are represented as D. select,bug ,|D select,bug | represents the number of test cases in the selected test case set that were misclassified, |D test,bug | indicates the number of test cases that were incorrectly classified in the original test case set. The specific calculation formula is:
[0071]
[0072] • Fault Detection Diversity: This parameter represents the degree to which the test case set selected by the test case selection method reveals the diversity of potential faults in the recurrent neural network model. The original test case set is represented as D. test The selected test case set is represented as D. select And will be able to detect D test The fault types included in the test cases for erroneous behavior are represented as D. test,type It can detect D select The fault types included in the test cases for erroneous behavior are represented as D. select,type ,|D select,type | indicates the number of fault types contained in the test cases that are incorrectly classified in the selected test case set, |D test,type | indicates the number of fault types contained in the test cases that were incorrectly classified in the original test case set. The specific calculation formula is:
[0073]
[0074] • Model Optimization Effectiveness: This parameter represents the degree to which the test case set selected by the test case selection method guides the recurrent neural network (RNN) model retraining. Theoretically, test cases with high failure detection rates can be used to optimize the RNN model. Therefore, model optimization effectiveness is proposed to further evaluate whether retraining the RNN model using selected test cases can enhance its robustness. In experiments, the RNN model was retrained using the selected test case set, and the effectiveness of model optimization was compared based on the performance of the RNN model on the test case set before and after retraining.
[0075] 3. Experimental Results
[0076] Performance comparison: The method of this invention outperforms DeepState in all four aspects: fault detection rate, fault detection coverage, fault detection diversity, and model optimization effectiveness, achieving excellent performance.
[0077] Table 2 Performance evaluation results of the method
[0078]
[0079] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.
Claims
1. A method for selecting test cases for recurrent neural network components based on clustering and uncertainty, characterized in that: Includes the following steps: Step 1: Use the recurrent neural network component under test to predict the original test case set and obtain the output confidence features corresponding to each test case; then perform feature processing on the output confidence features; finally, perform HDBSCAN clustering calculation on the processed features to obtain the test cases corresponding to the normal clusters. Test cases corresponding to outliers ; Step 2: Based on the HDBSCAN clustering results, calculate the uncertainty scores of the test cases corresponding to the normal clusters and the test cases corresponding to the outliers after clustering. Based on the predicted uncertainty score threshold, pre-screen the test case set. Step 3: For each test case in the test case set obtained in Step 2, calculate the rate of change of the hidden state when the recurrent neural network component under test predicts each test case; The hidden state change rate is calculated based on the state of the recurrent neural network. The change rate of the predicted label of the hidden state inside the recurrent neural network is calculated, and the hidden state change rate of each test case in the test case set is calculated to obtain the uncertainty of the hidden layer inside the recurrent neural network for each test case. This uncertainty indicates the probability that the test case triggers the recurrent neural network model failure. Test cases are sorted according to the rate of change of hidden states to obtain a first set of test cases sorted by the probability of triggering a failure of the recurrent neural network component under test. Test cases are then selected sequentially from the first set of test cases to obtain the final set of test cases. ; Specifically, if the size of the test cases with uncertainty scores greater than a threshold among the pre-selected outliers is greater than or equal to the size of the final set of test cases to be selected. Then, based on the rate of change of hidden states, the test cases with uncertainty scores greater than the threshold corresponding to the pre-screened outliers are sorted, and the test cases are selected in order to obtain the final set of test cases. ; If the number of test cases with uncertainty scores greater than the threshold among the pre-selected outliers is smaller than the final size of the test case set to be selected. Then, the test cases with uncertainty scores greater than the threshold among the pre-selected outlier test cases will be used as the final set of test cases required. Part of; Next, test cases with uncertainty scores greater than a threshold in each normal cluster are sorted according to their hidden state change rate, and test cases are selected sequentially from the pre-selected test cases in each normal cluster according to the proportion, forming the final required test case set. The other part; the ratio is the percentage of the number of pre-selected test cases in each normal cluster to the total number of pre-selected test cases in all normal clusters.
2. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 1, characterized in that: In step 1, the feature processing of the output confidence features includes feature normalization and feature dimensionality reduction.
3. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 2, characterized in that: In step 1, the process of normalizing the output confidence features is as follows: Assume the original test case set is represented as Its size is recorded as , , The eigenvectors are denoted as Use variables before normalization to indicate of The value of the feature, and using variables To represent the normalized result of The value of the feature; ， 。 4. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 2 or 3, characterized in that: In step 1, the FastICA algorithm is used to perform feature dimensionality reduction.
5. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 1, characterized in that: In step 2, the uncertainty score of the test cases is calculated using the lowest confidence level sampling method.
6. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 5, characterized in that: In step 2, the lowest confidence level sample is the difference between the highest confidence level prediction and the 100% confidence level.
7. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 1, characterized in that: In step 2, test cases are first selected from the test cases corresponding to the outliers, and the uncertainty score of all test cases corresponding to the outliers is calculated. Then, test cases with uncertainty scores greater than a threshold are pre-screened. If the size of the pre-screened test cases with uncertainty scores greater than the threshold is greater than or equal to the size of the final set of test cases to be selected, the process is repeated. If the number of test cases with uncertainty scores greater than a threshold among the pre-selected outliers is smaller than the final size of the test case set to be selected, then a pre-selected test case set is obtained. Then, the uncertainty score of the test cases corresponding to the normal cluster is further calculated, and test cases with uncertainty scores greater than the threshold are pre-screened.
8. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 7, characterized in that: In step 2, the threshold is 0.
01.
9. The method for selecting test cases for recurrent neural network components based on clustering and uncertainty as described in claim 7, characterized in that: In step 3, the test cases are sorted in reverse order from high to low according to the hidden state change rate, thereby obtaining the first test case set with the probability of triggering the failure of the recurrent neural network model under test in a forward order. Test cases are then selected from the first test case set in order from high to low according to the hidden state change rate to obtain the final required test case set.