An adaptive test case optimization and selection method based on deep neural network

Abstract

The application discloses a kind of adaptive test case optimization and selection method based on deep neural network, for the DNN model for N classification task, first, test case set is input to DNN model, according to output result, it is divided into N set, then based on the prediction confidence, each set is divided into selection set, candidate set and removal set.Second, the similarity of image in candidate set and image in selection set is obtained by calculating, and the selection set and candidate set are used as initial set to select, guarantee the balanced distribution of test case in selection set, and return the selection set finally obtained.The present application selects test case based on the idea of uniform distribution of test case and model uncertainty, only marks the test case selected, not only can save test case marking cost, improve DNN model test efficiency, but also can retrain DNN model using the test case after selection, improve the robustness of DNN model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of testing technology in software engineering, and particularly relates to an adaptive test case selection and optimization method based on a deep neural network. BACKGROUND

[0002] Deep learning (DL) has made rapid progress in the past few decades and has become a key technology in various fields. In the field of computer vision, DNN models have made revolutionary progress, including image recognition, object detection, image segmentation, and other tasks. Through a large amount of image data training, these models can accurately identify objects, faces, scenes, and other objects in pictures, and are applied to autonomous vehicles, surveillance systems, medical image analysis, and other fields. Like traditional software, software based on neural network (DNN) models also has defects and errors, for example, the Tesla Autopilot system failed to recognize certain obstacles or dangerous proximity to other vehicles in some traffic environments, resulting in a collision accident. Therefore, the safety, reliability of intelligent software guided by DNN models and the explainability of their decision-making process have attracted widespread attention and discussion.

[0003] There is a huge difference between traditional software testing and intelligent software testing based on DNN models in discovering errors. Therefore, how to efficiently and accurately test intelligent software has become a hot topic in the field of software testing. Intelligent software based on DNN models can continuously adapt to new programming patterns and bug types through continuous learning, while traditional methods may need to manually update rules to adapt to new situations. In theory, the method based on DNN models can provide higher accuracy and efficiency, but the success of this method depends largely on a large amount of high-quality training data and test data. In addition, the DNN model is a complex machine learning framework that can simulate and learn very complex nonlinear relationships by utilizing the powerful combination capabilities of millions to billions of neurons. This results in a lack of explainability of the DNN model, making it difficult for developers to understand the specific reasons for the model's specific decisions. In practical applications, developers often need to collect a large amount of data from application scenarios to retrain the DNN model, and then correct the DNN model's wrong predictions to improve the accuracy of the model. In the process of collecting data, a large number of labor is needed to mark the data, which consumes a lot of time and effort of the developers.

[0004] In order to alleviate the labeling cost problem, a feasible solution is to select unmarked test cases, select test cases that may cause the DNN to make wrong predictions, and only mark the selected test cases. This not only saves the labeling cost, but also improves the efficiency of DNN testing. SUMMARY

[0005] The application aims to provide a deep neural network-based adaptive test case selection and optimization method, which can provide a more uniformly distributed test case set to highlight the proportion of errors predicted by the DNN model, and the test cases selected by the application can be labeled, which not only saves labeling costs and improves the efficiency of DNN testing, but also improves the robustness of the DNN model through guided retraining.

[0006] Technical scheme: A deep neural network-based adaptive test case optimization and selection method, which processes the test cases of a DNN model used for N classification tasks as follows:

[0007] S1, input the unlabeled test case set to be selected into the DNN model, divide it into N sets according to the DNN prediction result and the image classification target, and then divide each set into a selection set, a candidate set and a removal set according to the prediction confidence variance;

[0008] S2, for the image data sets in the selection set and the candidate set, calculate the similarity between any two images based on the model uncertainty image similarity calculation method, specifically including:

[0009] S21, prediction: randomly select one picture from the selection set and the candidate set, respectively, denoted as picture a and picture b, and input them into the DNN model for classification prediction, and the probability vectors P and Q output by the DNN model represent the prediction probabilities of pictures a and b, respectively;

[0010] S22, reorganization and extraction: randomly select 3 prediction probability values from the probability vectors P and Q, respectively, thereby reorganizing the vectors P and Q into three-dimensional sub-vectors, respectively;

[0011] S23, calculate the image similarity: let any one group of three-dimensional sub-vectors after reorganizing the vectors P and Q be P'(p i , p j , p k ) and Q'(q i , q j , q k ), then regard the two vectors as coordinate points in a three-dimensional space, and project them onto the plane where p i and q i are located, thereby obtaining three-dimensional coordinate points P' and Q', according to which the point P', Q' and the origin 0 form an angle, denoted as angle α, then calculate sin(α) using the corresponding coordinates of P' and Q', and at the same time obtain the absolute value of the difference between p i and q i ;

[0012] Similarly, this also includes projecting the aforementioned two-dimensional coordinate points P′ and Q′ onto p respectively. j and q j p k and q k Calculate the angles β and β' and the sine on the plane in which they are located, including their absolute values;

[0013] Based on this, a formula is defined to measure the similarity between probability vectors P and Q, which is used to obtain information about angles and differences in the above process. The expression is as follows:

[0014] similarity (i，j，k) =sin(α)*(|p i -q i |)+sin(β)*(|p j -q j |)+sin(γ)*(|p k -q k |)

[0015] S24. Similarity Accumulation: The recombined results... The similarity of each vector is accumulated. For a specific image input x1 and x2, when performing a classification task with n categories, the similarity between images x1 and x2 is calculated using the following formula:

[0016] Similarity(x1,x2)=∑similarity (i,j，k) ;

[0017] S3. Using the selection set and candidate set optimized in step S2 as the initial set, select images by calculating the similarity between the images in the candidate set and the images in the selection set, and return the final selection set.

[0018] Further, in step S1, the test case set C is divided into N subsets based on the prediction results output by the DNN model, and each subset C... i The test cases in the dataset are predicted by the DNN to be of the same category, and then for each subset C... i The test cases in the dataset calculate the prediction confidence variance, and then use the prediction confidence variance and the subset C as a basis for the calculation. i The test cases within the set are sorted, and then the test cases are divided into an initial selection set, an initial candidate set, and an exclusion set according to the number of target test cases required.

[0019] Furthermore, the prediction confidence variance is calculated as follows:

[0020] In image classification tasks, given an input test case x, the predicted output of the DNN model is a vector with n values, denoted as P =<p1, p2, ..., p nwhere each p i denotes the probability that the input x is predicted to be the i-th class, and The vector P is sorted to obtain a vector P' = <p'1, p'2,..., p'n>, where p'1≥ p'2≥... ≥ p'n. n n’ The probability difference between n predicted probabilities is defined as follows:

[0021]

[0022] The confidence variance:

[0023] Further, step S23 is to evaluate the similarity between each test case x in the candidate set and all test cases in the selected set by an image similarity calculation method, and the shortest distance from the test case x in the candidate set to the selected set is obtained. After obtaining these minimum similarity values, the test case corresponding to the maximum value is selected to ensure the uniform distribution of the selected test cases in the set.

[0024] The method of the present application can select a test case set with balanced distribution based on the idea of balanced distribution of test cases and the uncertainty of the model, which has balanced distribution of test case categories and higher probability of inducing DNN model to produce error prediction, is more likely to expose the defects of DNN, and through retraining, helps to improve the robustness of DNN. The beneficial effects of the present application include the following three points:

[0025] (1) The method of the present application can effectively improve the test efficiency of the DNN model by using the image similarity calculation method;

[0026] (2) By using the idea of balanced distribution of test cases and the uncertainty of the model, a test case selection method is designed, which can improve the defect detection rate of DNN, and only marking the selected test cases can save the marking cost and optimize the DNN model;

[0027] (3) The present application preferentially selects test cases with higher probability of inducing DNN model to produce error prediction, and then re-trains DNN by marking these test cases and adding them to the original training set, which can significantly improve the robustness of DNN. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flow framework diagram of the method of the present application. DETAILED DESCRIPTION

[0029] ​The main technologies used by the present application are respectively deep neural network (Deep Neural Network) and adaptive test case selection technology. The flow of the method is introduced below and the implementation and specific implementation steps of each technology used by the present application are specifically explained.

[0030] Deep neural network (DNN) is a kind of artificial neural network, including application in the field of image recognition processing, the structure of DNN model includes an input layer, multiple hidden layers and an output layer. The internal hidden layer includes convolution layer, pooling layer and full connection layer and other parts. The main function of the convolution layer is to extract features from the input data, and different convolution kernels can capture different feature information; after several convolution layers, a pooling layer is usually added, which helps to reduce the total amount of neural network parameters and helps to prevent the overfitting phenomenon of neural network; the role of the full connection layer is to convert the extracted feature information to the label space of the sample, and then the data is transmitted to the output layer. After the convolution layer and the full connection layer, an activation function such as ReLU or Sigmoid is usually applied to enhance the representation ability of the neural network.

[0031] Generally speaking, the deep neural network is used to map the input image data x (test case) to the output result y. For example, for an N-classification task, a test case x is given, and after the internal neuron processing of the DNN, an N-dimensional vector P = {p1, p2, p3,..., p N} is obtained at the output layer, and then the softmax function is used for normalization processing, so that a set of probability vectors P = {p1, p2, p3,..., p N} is obtained, and p i represents the probability that the picture is predicted by the neural network as the i-th class, and the final prediction result of the DNN is the class corresponding to the maximum value in P.

[0032] The role of the deep neural network is to convert the input data x into the output result y. Taking an N-classification task as an example, for a given test input x, the data is processed through multiple layers of the neural network, and finally an N-dimensional vector V = {v1, v2, v3,..., v N} is formed at the output layer. Next, the softmax function is applied to normalize this vector to generate a set of probability vectors P = {p1, p2, p3,..., p N}, where each p i represents the probability that the neural network predicts that the input x belongs to the i-th class. The final prediction result of the neural network is the class with the maximum probability value in the vector P.

[0033] Further introduction, test case selection is a hot research problem in the field of software testing, how to select better quality test data from a large amount of data becomes the key of intelligent software testing. Only the selected test data with better quality is marked to correct the wrong prediction behavior of the DNN model, so as to improve the robustness and reliability of the model. Previous test case selection methods are basically divided into two kinds, which are test case selection method based on neuron coverage guidance and test case selection method based on priority.

[0034] Test case selection method based on neuron coverage guidance: the method based on neuron coverage is to guide the selection of test cases by using the neuron coverage rate in the deep neural network model. This kind of method usually adopts greedy algorithm, and uses the previous neuron activation value as feedback to guide the selection of the next input.

[0035] Test case selection method based on priority: many efficient test case selection methods adopt the priority-based guidance strategy, which defines a set of priority rules, and the data is sorted according to the priority rules, and the data at the top of the sorting is selected. Most of the strategies are based on the uncertainty of the model, and the high-quality test cases are selected according to the prediction behavior of the DNN model.

[0036] Based on the above, we further introduce the test case selection method provided by the present application.

[0037] The adaptive test case optimization and selection method based on deep neural network provided by the present application considers that the more uniform the distribution of the test case set is, the greater the proportion of the test cases predicted by the DNN is, and the more DNN prediction error categories are covered, and the robustness of the DNN model can be effectively improved through retraining of the DNN model. For a DNN model trained for N classification tasks, the present application first inputs the unmarked test case set to be selected into the DNN model, divides it into N sets according to the DNN prediction result, and then proposes a prediction confidence (PCV) method, which is used to optimize each set, and divides each set into a selection set, a candidate set and a removal set. Secondly, we further process the selection set and the candidate set, that is, we propose an image similarity calculation method based on model uncertainty, which can calculate the similarity between two images. Finally, the optimized selection set and candidate set are used as the initial set, the similarity between the images in the candidate set and the images in the selection set is calculated to select the images, the distribution of the test cases in the selection set is balanced, and the final selection set is returned.

[0038] Specifically, the adaptive test case optimization and selection method based on deep neural network provided by the present application comprises the following steps:

[0039] Step 1: Optimize the test case set.

[0040] We first address the processing of unlabeled test sets by the DNN model. For each subset of the N sets output by the DNN model in the N-class image classification task, we introduce the calculation of Prediction Confidence Variance (PVC). Based on the PVC coefficients, we sort the test cases within each set, dividing them into three categories: the selection set, the candidate set, and the exclusion set. This process constitutes the initial selection; the resulting initial selection set consists of the selection set and the candidate set. The initial selection set includes test cases that have a high probability of causing the model to make incorrect predictions, while the exclusion set contains test cases that reduce the probability of incorrect predictions.

[0041] In this process, we introduce a new metric called Prediction Confidence Variance (PCV) to classify test cases based on the uncertainty of model predictions. Prediction Confidence Variance (PCV) partitions the set of test cases based on the uncertainty of model predictions. In image classification tasks, given an input x, the predicted output of the DNN model is a vector P = <p1, p2, ..., pn> with n values. n >, where each p i Let x represent the probability that input x is predicted to be the i-th class, and Sort the vector P to obtain the vector P′=<p′1,p′2,...,p′ n >, where p′1≥p′2≥...≥p′ n PCV measures the probability difference between n predicted probabilities. It is defined as follows:

[0042]

[0043] Step 2: Calculate image similarity.

[0044] This step considers that the selected test cases can maintain the balance of their distribution. Therefore, we designed an image similarity evaluation method based on directionality and uncertainty. This image similarity is achieved using the data image sets from the candidate set and the selection set in step 1, i.e., the initial test case set. The specific steps are as follows:

[0045] 2.1 Prediction.

[0046] Select one picture from the data set contained in the selection set and the candidate set respectively, which are picture a and picture b respectively, and input them into a DNN model for classification prediction. Assuming that a data set containing 10 categories is selected, therefore, for each picture, the DNN model will output a probability vector containing 10 elements, representing the predicted probability of the picture belonging to each category. Let the two probability vectors be P = <p1, p2,..., p10> and Q = <q1, q2,..., q10> respectively, where vector P represents the classification prediction result of picture a, and vector Q represents the classification prediction result of picture b. The elements p 10 and q 10 in each vector represent the predicted probability of picture a and picture b belonging to the corresponding category respectively. i i

[0047] 2.2, recombination and extraction.

[0048] Recombine vectors P and Q into vectors, each of which contains 3 prediction probability values after recombination, denoted as vectors P' and Q', both of which exist sub-vectors obtained by combination arrangement.

[0049] 2.3 Image similarity calculation.

[0050] For vector P', any one of them is represented as P'(p i , p j , p k ) and Q'(q i , q j , q k ), then the two vectors are regarded as coordinate points in three-dimensional space, and are projected onto the plane where p i and q i are located, and then three-dimensional coordinate points P' and Q' are obtained, according to which point P', Q' and the origin 0 form an angle, denoted as angle α, then sin(α) is calculated using the corresponding coordinates of P' and Q', and the absolute value of the difference between p i and q i is obtained;

[0051] Similarly, it also includes projecting the above two-dimensional coordinate points P' and Q' onto the planes where p j and q j , p k and q k are located to calculate angles β and γ and sines, including absolute values;

[0052] On this basis, a formula for measuring the similarity of probability vectors P and Q is defined, which is used for the information obtained in the above process, and the expression is as follows:​​

[0053] similarity (i，j，k) = sin(a) * (|p i -q i |) + sin(b) * (|p j -q j |) + sin(g) * (|p k -q k |)

[0054] For further illustration, if we take one set of P and Q as A(p1, p2, p3) and B(q1, q2, q3) respectively. Then, we project the coordinates of A and B onto the horizontal plane corresponding to p3 and q3, and obtain two-dimensional coordinate points A' and B'. Accordingly, points A', B' and the origin 0 form an angle, denoted as angle a. Using the coordinates of A' and B', we can calculate sin(a). At the same time, we can obtain the absolute value of the difference between p3 and q3. Similarly, we can project A and B onto the horizontal plane corresponding to p2 and q2, and p1 and q1, and thus obtain angles b and g. On this basis, we define a formula for measuring the similarity of vectors P and Q, which integrates the information of the angles and differences obtained in the above process.

[0055] similarity (1，2,3) = sin(a) * (|p3-q3|) + sin(b)

[0056] * (|p2-q2|) + sin(g) * (|p1-q1|)

[0057] Similarly, starting from vectors <p1, p2, p3> and <q1, q2, q3>.

[0058] 2.4, Similarity accumulation.

[0059] Similarity accumulation: the similarity of the reorganized vectors is accumulated. For specific image inputs x1 and x2, when performing a classification task of n categories, the similarity between images x1 and x2 is calculated by the formula:

[0060] Similarity(x1, x2) = ∑similarity (i，j，k) ;

[0061] Set the example in step 2.3, and accumulate the similarity of the reorganized vectors. For specific image inputs x1 and x2, when performing a classification task of n categories, we define the following formula to calculate the similarity between images x1 and x2:

[0062] Similarity(x1, x2) = similarity (1，2,3) + similarity (1，2,4) +... + similarity (n-2，n-1，n)

[0063] Step 3: Adaptive test case selection.

[0064] According to step 1, the original selection set and the candidate set are obtained. In the process of selecting test cases, the image similarity calculation method is used to evaluate the similarity between each test case x in the candidate set and all test cases in the selection set. Specifically, the focus will be on the minimum similarity between x and any test case in the selection set, that is, the shortest distance of x to the selection set is calculated. After obtaining these minimum similarity values, the maximum value is selected to ensure the uniform distribution of the selected test cases in the set.

[0065] Step 4: The adaptive test case selection method is evaluated using four commonly used data sets and four DNN models, and the selected test case set is used to retrain the DNN model to improve the robustness of the model.

[0066] The specific implementation steps of the present application will be described below through specific examples:

[0067] (1) Data set and model

[0068] Four classic datasets were selected: MNIST, CIFAR-10, Fashion, and SVHN. Simultaneously, four widely used DNN models of different sizes for image classification tasks were chosen: LeNet-1, LeNet-5, ResNet-20, and VGG-16. Two different DNN models were used for each dataset to ensure stable experimental results, as shown in Table 1. The MNIST dataset is a large database of handwritten digits, commonly used to train various image processing systems. This dataset contains 70,000 images of handwritten digits (0 to 9), each a 28x28 pixel grayscale image. CIFAR-10 is a benchmark dataset widely used in machine learning and computer vision research, containing 60,000 32x32 pixel color images, divided into 50,000 training images and 10,000 test images. The Fashion-MNIST dataset aims to provide a dataset that more closely approximates real-world problems. It contains 70,000 grayscale images from 10 categories, with 60,000 samples used to train the model and the remaining 10,000 samples used to evaluate the model's performance and generalization ability. The SVHN dataset is a large-scale dataset for digit recognition, sourced from Google Street View data. SVHN contains over 600,000 color images, divided into a training set, a test set, and an additional dataset. The training set contains 73,257 images, the test set contains 26,032 images, and the additional dataset contains 531,131 images.

[0069] Table 1 Datasets and DNN Models

[0070]

[0071]

[0072] (2) Test case optimization: The test set data is input into the trained DNN model, and the data is divided into 10 sets C = {C1, C2, C3, ..., C...} based on the prediction results of the DNN. 10}, that is, each set C i Test cases in (i∈[1,10]) are predicted by the DNN to be of the same category. Then we respectively test the set C i The test cases in the dataset have PCV coefficients calculated. Based on these PCV coefficients, the test cases in the dataset are sorted into three categories: the initial selection set, the initial candidate set, and the exclusion set. The initial selection set includes test cases that significantly increase the probability of incorrect predictions by the model, while the exclusion set contains test cases that decrease the probability of incorrect predictions. The remaining test cases are then assigned to the candidate set.

[0073] (3) Test case selection process. The selected set and candidate set obtained in (2) are reserved as the initial selected set and candidate set in our selection process. First, we set the end condition of the selection process, which is when the number of selected test cases reaches the target we set, the algorithm stops running and returns the selected test case set. Next, in each prediction classification category, we calculate the minimum image similarity of each test case x in the candidate test case set to the selected test case set, and store this minimum image similarity value in the corresponding similarity queue. Finally, we sort the similarity queue in descending order, and then reorder the candidate set according to this sorting order. Then, select the first batch elements from the candidate set and add them to the selected set. When the number of test cases in the test case set reaches the target number, return the test case set.

Claims

1. An adaptive test case optimization and selection method based on deep neural networks, characterized in that, This method performs the following processing on test cases of DNN models used for N-class classification tasks: S1. Input the unlabeled set of test cases to be selected into the DNN model, divide it into N sets according to the DNN prediction results and the image classification target, and then divide each set into a selection set, a candidate set and a removal set by predicting the confidence variance. S2. For the image datasets in the selection set and candidate set, calculate the similarity between any two images using an image similarity calculation method based on model uncertainty, specifically including: S21. Prediction: Select one image from the selection set and the candidate set respectively, denoted as image a and image b, and input them into the DNN model for classification prediction. The probability vectors P and Q output by the DNN model represent the predicted probabilities of image a and image b, respectively. S22, Recombination and Extraction: Randomly select 3 predicted probability values ​​from probability vectors P and Q respectively, and thus recombine vectors P and Q into... A three-dimensional sub-direction; S23. Calculate image similarity: Denote any set of three-dimensional sub-vectors resulting from the recombination of vectors P and Q as P'(p i ,p j ,p k ) and Q'(q i ,q j ,q k Then, these two vectors are treated as coordinate points in three-dimensional space, and their projection is fixed on p. i and q i On the plane in which it lies, we obtain three-dimensional coordinate points p′ and Q′. These points p′ and Q′ form an angle with the origin O, denoted as angle α. Then, we use the coordinates corresponding to P′ and Q′ to calculate sin(α), and simultaneously obtain p′. i and q i The absolute value of the difference; Similarly, this also includes projecting the aforementioned two-dimensional coordinate points P′ and Q′ onto p respectively. j and q j p k and q k Calculate angles β and γ and sine on the plane in which they are located, including absolute values; Based on this, a formula is defined to measure the similarity between probability vectors P and Q, which is used to obtain information about angles and differences in the above process. The expression is as follows: similarity (i,j,k) sin(α)*(|p i -q i |)+sin(β)*(|p j -q j |)+sin(γ)*(|p k -q k |) S24. Similarity Accumulation: The recombined results... The similarity of each vector is accumulated. For a specific image input x1 and x2, when performing a classification task with n categories, the similarity between images x1 and x2 is calculated using the following formula: Similarity(x1,x2)=∑similartty (i,j,k) ; S3. Using the selection set and candidate set optimized in step S2 as the initial set, select images by calculating the similarity between the images in the candidate set and the images in the selection set, and return the final selection set.

2. The adaptive test case optimization and selection method based on deep neural networks according to claim 1, characterized in that, In step S1, the test case set C is divided into N subsets based on the prediction results output by the DNN model. i The test cases in the dataset are predicted by the DNN to be of the same category, and then for each subset C... i The test cases in the dataset calculate the prediction confidence variance, and then use the prediction confidence variance and the subset C as a basis for the calculation. i The test cases within the set are sorted, and then the test cases are divided into an initial selection set, an initial candidate set, and an exclusion set according to the number of target test cases required.

3. The adaptive test case optimization and selection method based on deep neural networks according to claim 1 or 2, characterized in that, The prediction confidence variance is calculated as follows: In image classification tasks, given an input test case x, the predicted output of the DNN model is a vector with n values, denoted as P = <p1,p2,…,p n >, where each p i Let x represent the probability that input x is predicted to be the i-th class, and Sort the vector P to obtain the vector P′= <p′1,p′2,...,p′ n >, where p′1≥p′2≥...≥p′ n The probability difference between n predicted probabilities is defined as follows: Confidence variance:

4. The adaptive test case optimization and selection method based on deep neural networks according to claim 1, characterized in that, Step S23 evaluates the similarity between each test case x in the candidate set and all test cases in the selection set by using image similarity calculation methods. After obtaining the minimum similarity values ​​by finding the shortest distance from each test case x in the candidate set to the selection set, the test case corresponding to the maximum value is selected to ensure that the selected test cases are evenly distributed in the set.

Images