Adaptive test case optimization and selection method based on deep neural network

Abstract

The invention discloses an adaptive test case optimization and selection method based on a deep neural network, and the method comprises the steps: firstly inputting a test case set into a DNN model for N classification tasks, dividing the test case set into N sets according to an output result, and carrying out the N sets; each set is then divided into a selection set, a candidate set, and a removal set based on the predicted confidence. Secondly, selecting by taking a selection set obtained by calculating the similarity between the images in the candidate set and the images in the selection set and the candidate set as an initial set, ensuring that the test cases in the selection set are distributed uniformly, and returning to the finally obtained selection set; according to the method, the test cases are selected based on the idea of uniform distribution of the test cases and the model uncertainty, and only the selected test cases are marked, so that the marking cost of the test cases can be saved, the test efficiency of the DNN model is improved, the DNN model can be retrained by using the selected test cases, and the robustness of the DNN model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of testing technology in software engineering, and in particular relates to an adaptive test case selection and optimization method based on deep neural network. Background Art

[0002] Deep learning (DL) has made rapid progress in the past few decades and has become a critical technology in various fields. In the field of computer vision, DNN models have made revolutionary progress, including image recognition, object detection, image segmentation and other tasks. Through training with a large amount of image data, these models can accurately identify objects, faces, scenes, etc. in pictures, and are applied to many fields such as self-driving cars, monitoring systems, and medical image analysis. Like traditional software, software based on neural network (DNN) models also has defects and errors. For example, Tesla's Autopilot system failed to identify specific obstacles in the traffic environment or dangerously close to other vehicles in some cases, resulting in collision accidents. Therefore, the safety and reliability of intelligent software guided by DNN models and the interpretability of their decision-making processes have attracted widespread attention and discussion.

[0003] There is a huge difference between traditional software testing and intelligent software testing guided by DNN models in terms of error detection. Therefore, how to test intelligent software efficiently and accurately has become a hot topic in the field of software testing. Intelligent software based on DNN models can continuously adapt to new programming patterns and bug types through continuous learning, while traditional methods may require manual rule updates to adapt to new situations. In theory, methods based on DNN models can provide higher accuracy and efficiency, but the success of this method depends largely on a large amount of high-quality training data and test data. In addition, the DNN model is a complex machine learning framework that can simulate and learn very complex nonlinear relationships by leveraging the powerful combination of millions to billions of neurons. As a result, the DNN model is not interpretable enough, making it difficult for developers to understand the specific reasons why the model makes a specific decision. In practical applications, developers often need to collect a large amount of data from application scenarios to retrain the DNN model, and then correct the DNN model's incorrect predictions to improve the model's accuracy. In the process of collecting data, a large amount of labor needs to be hired to label the data, which consumes a lot of time and energy of developers.

[0004] In order to alleviate the labeling cost problem, a feasible solution is to select unlabeled test cases. For test cases that may cause DNN to make incorrect predictions, we can only label the selected test cases, which can not only save labeling costs but also improve the efficiency of DNN testing. Summary of the invention

[0005] Purpose of the invention: The present invention aims to provide an adaptive test case selection and optimization method based on deep neural networks, which can provide a more evenly distributed test case set to highlight the proportion of incorrect predictions by the DNN model, and mark the test cases selected by the present invention, which can not only save marking costs and improve the efficiency of DNN testing, but also improve the robustness of the DNN model by guiding retraining.

[0006] Technical solution: An adaptive test case optimization and selection method based on deep neural network. This method performs the following processing on the test cases of DNN model for N classification tasks:

[0007] S1. Input the unlabeled test case set to be selected into the DNN model, divide it into N sets according to the DNN prediction results and the image classification target, and then divide each divided set into a selection set, a candidate set and a removal set by predicting the confidence variance;

[0008] S2. For the image datasets in the selection set and the candidate set, the similarity between any two images is calculated using an image similarity calculation method based on model uncertainty, specifically including:

[0009] S21, prediction: select one picture from the selection set and the candidate set respectively, denoted as picture a and picture b, and input them into the DNN model for classification prediction. The probability vectors P and Q output by the DNN model represent the prediction probabilities of picture a and picture b respectively;

[0010] S22, reorganization and extraction: arbitrarily select three predicted probability values ​​from the probability vectors P and Q, thereby reorganizing the vectors P and Q into 3D subvectors;

[0011] S23, calculate the image similarity: record any set of three-dimensional subvectors after the vectors P and Q are reorganized as P'(p i , p j , p k ) and Q'(q i ,q j ,q k ), then consider these two vectors as coordinate points in three-dimensional space and project them to p i and q i On the plane where the three-dimensional coordinate points P' and Q' are located, the three-dimensional coordinate points P' and Q' are obtained. According to this point P', Q' and the origin 0 form an angle, which is recorded as angle α. Then, the coordinates corresponding to P' and Q' are used to calculate sin(α), and p is obtained at the same time. i and q i The absolute value of the difference;

[0012] Similarly, it also includes projecting the above two-dimensional coordinate points P′ and Q′ onto p j and q j 、p k and q k Angle β and β and sine calculations are performed on the plane where the angle is located, including absolute values;

[0013] On this basis, a formula to measure the similarity of probability vectors P and Q is defined for the angle and difference information obtained in the above process. The expression is as follows:

[0014] similarity (i，j，k) = sin(α)*(|p i -q i |)+sin(β)*(|p j -q j |)+sin(γ)*(|p k -q k |)

[0015] S24, similarity accumulation: recombining The similarity of the vectors is accumulated. For specific image inputs x1 and x2, when performing n-category classification tasks, the similarity between images x1 and x2 is calculated by the formula:

[0016] Similarity(x1,x2)=∑similarity (i,j，k) ;

[0017] S3. The selection set and candidate set optimized in step S2 are used as the initial set, and the selection is performed by calculating the similarity between the images in the candidate set and the images in the selection set, and the final selection set is returned.

[0018] Furthermore, in step S1, the test case set C is divided into N subsets according to the prediction results output by the DNN model. i The test cases in are predicted by DNN to be of the same category, and then for each subset C i The test cases in the prediction confidence variance are calculated, and the prediction confidence variance and the subset C are calculated based on the prediction confidence variance and the prediction confidence variance. i The test cases in the set are sorted, and then the test cases are divided into an initial selection set, an initial candidate set, and an exclusion set according to the number of target test cases.

[0019] Furthermore, the prediction confidence variance is calculated as follows:

[0020] In the image classification task, given an input test case x, the predicted output of the DNN model is a vector with n values, denoted as P = <p1, p2, ..., p n>, where each p i represents the probability that the input x is predicted to be the i-th category, and Sorting the vector P yields the vector P′=<p′1,p′2,...,p′ n >, where p′1≥p′2≥...≥p′ n’ ; The probability difference between n predicted probabilities is defined as follows:

[0021]

[0022] Confidence variance:

[0023] Furthermore, step S23 is to evaluate the similarity between each test case x in the candidate set and all test cases in the selection set through an image similarity calculation method, and to obtain the shortest distance from the test case x in the candidate set to the selection set. After obtaining these minimum similarity values, the test case corresponding to the maximum value is selected to ensure that the selected test cases are evenly distributed in the set.

[0024] The method of the present invention is based on the idea of ​​balanced distribution of test cases and uncertainty of the model, and can select a balanced test case set, in which the test case categories are evenly distributed. At the same time, the probability of the test case inducing the DNN model to make an incorrect prediction is higher, and the more likely it is to reveal the defects of the DNN, and through retraining, it is more helpful to improve the robustness of the DNN. The beneficial effects of the present invention include the following three points:

[0025] (1) The method of the present invention can effectively improve the testing efficiency of the DNN model by adopting an image similarity calculation method;

[0026] (2) Based on the idea of ​​balanced test case distribution and model uncertainty, a test case selection method is designed to improve the defect detection rate of DNN. At the same time, only the selected test cases are marked, which can save marking costs and optimize the DNN model.

[0027] (3) The present invention preferentially selects test cases that have a higher probability of inducing the DNN model to make incorrect predictions, and then retrains the DNN by marking these test cases and adding them to the original training set, which can significantly improve the robustness of the DNN. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The flowchart of the method of the present invention is shown in FIG. DETAILED DESCRIPTION

[0029] The main technologies used in the present invention are Deep Neural Network and Adaptive Test Case Selection Technology. The following introduces the process of the method and specifically explains the implementation methods and specific implementation steps of each technology used in the present invention.

[0030] Deep neural network (DNN) is a type of artificial neural network, including the field of image recognition and processing. The structure of the DNN model includes an input layer, multiple hidden layers, and an output layer. The hidden layer includes convolutional layers, pooling layers, and fully connected layers. The main function of the convolutional layer is to extract features from the input data. Different convolution kernels can capture different feature information. After several convolutional layers, a pooling layer is usually added, which helps to reduce the total number of parameters of the neural network and prevent overfitting of the neural network. The function of the fully connected layer is to convert the extracted feature information into the label space of the sample, and then the data will be passed to the output layer. After the convolutional layer and the fully connected layer, an activation function, such as ReLU or Sigmoid, is usually applied to enhance the representation ability of the neural network.

[0031] Generally speaking, deep neural networks are used to map input image data x (test case) to output result y. For example, in an N-classification task, given a test case x, after being processed by the internal neurons of the DNN, an N-dimensional vector P = {p1, p2, p3, ..., p N}, and then use the softmax function to normalize it, so that a set of probability vectors P = {p1, p2, p3, ..., p N}, p i It indicates the probability that the image is predicted by the neural network as the i-th category. The final prediction result of DNN is the category corresponding to the value with the largest probability in P.

[0032] The function of a deep neural network is to transform input data x into output result y. Taking the N classification task as an example, for a given test input x, the data is processed through multiple layers of the neural network and finally forms an N-dimensional vector V = {v1, v2, v3, ..., v N Next, the vector is normalized by applying the softmax function to generate a set of probability vectors P = {p1, p2, p3, ..., p N}, where each p i Represents the probability that the neural network predicts that the input x belongs to the i-th category. The final prediction result of the neural network is the category with the largest probability value in the vector P.

[0033] Further introduction, test case selection is a hot research issue in the field of software testing. How to select better quality test data from a large amount of data has become the key to intelligent software testing. Simply marking the selected better quality test data can correct the incorrect prediction behavior of the DNN model, thereby improving the robustness and reliability of the model. Previous test case selection methods are basically divided into two types, namely, test case selection methods based on neuron coverage guidance and test case selection methods based on priority.

[0034] Test case selection method based on neuron coverage guidance: The neuron coverage-based selection method guides the selection of test cases by utilizing the neuron coverage in the deep neural network model. This type of method usually adopts a greedy algorithm to guide the selection of the next input by using the previous neuron activation value as feedback.

[0035] Priority-based test case selection method: Many efficient test case selection methods use a priority-based guidance strategy, which defines a set of priority rules, sorts the data according to the priority rules, and selects the top-ranked data. Most of these strategies are based on model uncertainty and select high-quality test cases based on the predicted behavior of the DNN model.

[0036] Based on the above content, we further introduce the test case selection method provided by the present invention.

[0037] The present invention provides an adaptive test case optimization and selection method based on a deep neural network. Considering that the more evenly distributed test case set is, the greater the proportion of DNN error prediction is, the more DNN prediction error categories are covered at the same time, and the robustness of the DNN model can be effectively improved by retraining the DNN model. For a trained DNN model for N classification tasks, the present invention first inputs the unlabeled test case set to be selected into the DNN model, divides it into N sets according to the DNN prediction results, and then proposes a prediction confidence (PCV) method, optimizes each set using this method, and divides each set into a selection set, a candidate set, and a removal set. Secondly, we further process the selection set and the candidate set, that is, propose an image similarity calculation method based on model uncertainty, which can calculate the similarity between two images. Finally, the optimized selection set and candidate set are used as the initial set, and the selection is performed by calculating the similarity between the image in the candidate set and the image in the selection set to ensure that the test cases in the selection set are evenly distributed, and the final selection set is returned.

[0038] Specifically, the present invention provides a method for adaptive test case optimization and selection based on a deep neural network, comprising the following steps:

[0039] Step 1: Test case set optimization.

[0040] We first introduce the calculation of prediction confidence variance (PVC) for each subset of the N classification tasks of images and the N sets output by the DNN model based on the processing of the unlabeled test set by the DNN model. We sort the test cases in the set according to the PCV coefficient and divide the test set into three categories: selection set, candidate set and removal set. This process is a preliminary selection, that is, the initial selection set is composed of the selection set and the candidate set. The initial selection set includes those test cases that have a high probability of causing the model to make wrong predictions, while the exclusion set includes those test cases that reduce the probability of the model making wrong predictions.

[0041] In this process, we introduce a new metric, called prediction confidence variance (PCV), to classify test cases based on the uncertainty of model predictions. Prediction confidence variance (PCV) partitions the test case set based on the uncertainty of model predictions. In the image classification task, given an input x, the predicted output of the DNN model is a vector P with n values ​​​​= <p1, p2, ..., p n >, where each p i represents the probability that the input x is predicted to be the i-th category, and Sorting the vector P yields the vector P′=<p′1,p′2,...,p′ n >, where p′1≥p′2≥...≥p′ n ,. PCV measures the probability difference between n predicted probabilities. It is defined as follows:

[0042]

[0043] Step 2: Calculation of image similarity.

[0044] This step considers that the selected test cases can maintain the balance of their distribution. Therefore, we designed an image similarity evaluation method based on directionality and uncertainty. The image similarity is the data image set in the candidate set and the selected set in step 1, that is, the preliminary test case set. The specific steps are as follows:

[0045] 2.1. Prediction.

[0046] Select one image from each of the datasets in the selection set and candidate set, namely image a and image b, and input them into a DNN model for classification prediction. Suppose a dataset containing 10 categories is selected. Therefore, for each image, the DNN model will output a probability vector containing 10 elements, representing the predicted probability that the image belongs to each category. Let these two probability vectors be P = <p1, p2, ..., P 10 >And Q= <q1,q2,...,q 10 >, where vector P represents the classification prediction result of image a, and vector Q represents the classification prediction result of image b. The element p in each vector i and q i Respectively represent the predicted probability that picture a and picture b belong to the corresponding category.

[0047] 2.2. Reorganization and extraction.

[0048] Rearrange vectors P and Q into vectors, each of which contains 3 predicted probability values ​​after reorganization, which are denoted as vectors P' and Q'. Both exist The sub-vectors obtained by permutation of combinations.

[0049] 2.3 Image similarity calculation.

[0050] For vector P', any one of them is represented as P'(p i , p j , p k ) and Q'(q i ,q j ,q k ), then consider these two vectors as coordinate points in three-dimensional space and project them to p i and q i On the plane where the three-dimensional coordinate points P' and Q' are located, the three-dimensional coordinate points P' and Q' are obtained. According to this point P', Q' and the origin 0 form an angle, which is recorded as angle α. Then, the coordinates corresponding to P' and Q' are used to calculate sin(α), and p is obtained at the same time. i and q i The absolute value of the difference;

[0051] Similarly, it also includes projecting the above two-dimensional coordinate points P′ and Q′ onto p j and q j 、p k and q k Angles β and γ and sine calculations are performed on the plane, including absolute values;

[0052] On this basis, a formula to measure the similarity of probability vectors P and Q is defined for the angle and difference information obtained in the above process. The expression is as follows:

[0053] similarity (i，j，k) = sin(α)*(|p i -q i |)+sin(β)*(|p j -q j |)+sin(γ)*(|p k -q k |)

[0054] To further illustrate, if one of the quantities P and Q is A(p1, p2, p3) and B(q1, q2, q3) respectively. Then, project the coordinates of A and B onto the horizontal planes corresponding to p3 and q3, and obtain the two-dimensional coordinate points A′ and B′. Based on this, points A′ and B′ form an angle with the origin 0, denoted as angle α. Using the coordinates of A′ and B′, we can calculate sin(α). At the same time, we can get the absolute value of the difference between p3 and q3. Similarly, we can project A and B onto the horizontal planes corresponding to p2 and q2 and p1 and q1, and obtain angles β and β. On this basis, we define a formula to measure the similarity of vectors P and Q, which combines the information of angles and differences obtained in the above process.

[0055] similarity (1，2,3) = sin(α)*(|p3-q3|)+sin(β)

[0056] *(|p2-q2|)+sin(β)*(|p1-q1|)

[0057] Start with the vectors <p1, p2, p3> and <q1, q2, q3>, and so on.

[0058] 2.4. Similarity accumulation.

[0059] Similarity accumulation: recombining to get The similarity of the vectors is accumulated. For specific image inputs x1 and x2, when performing n-category classification tasks, the similarity between images x1 and x2 is calculated by the formula:

[0060] Similarity(x1,x2)=∑similarity (i，j，k) ;

[0061] The example in step 2.3 is reorganized to obtain For specific image inputs x1 and x2, when performing a classification task of n categories, we define the following formula to calculate the similarity between images x1 and x2:

[0062] Similarity(x1,x2)=similarity (1，2,3) +similarity (1，2,4) +...+similarity (n-2，n-1，n)

[0063] Step 3: Adaptive test case selection.

[0064] According to step 1, the original selection set and the candidate set are obtained. In the process of selecting test cases, the image similarity calculation method is used to evaluate the similarity between each test case x in the candidate set and all test cases in the selection set. Specifically, the focus will be on the minimum similarity between x and any test case in the selection set, that is, the shortest distance from x to the selection set is calculated. After obtaining these minimum similarity values, the maximum value will be selected to ensure the uniform distribution of the selected test cases in the set.

[0065] Step 4: Use four common datasets and four DNN models to evaluate the adaptive test case selection method, and use the selected test case sets to retrain the DNN model to improve the robustness of the model.

[0066] The specific implementation steps of the present invention are described below by means of specific examples:

[0067] (1) Dataset and Model

[0068] Four classic datasets were selected, namely: MNIST, CIFAR-10, Fashion, and SVHN. At the same time, four widely used DNN models of different sizes for image classification tasks were selected, namely: LeNet-1, LeNet-5, ResNet-20, and VGG-16. Two different DNN models were equipped for each dataset to ensure the stability of our experimental results, as shown in Table 1. The MNIST dataset is a large handwritten digit database that is commonly used to train various image processing systems. The dataset contains 70,000 images of handwritten digits (0 to 9), each of which is a grayscale image of 28x 28 pixels. CIFAR10 is a benchmark dataset widely used in machine learning and computer vision research. It contains 60,000 32x32 pixel color images. The dataset is divided into 50,000 training images and 10,000 test images. The Fashion-MNIST dataset aims to provide a dataset that is closer to real-world problems. It contains 70,000 grayscale images from 10 categories, of which 60,000 samples are used to train the model and the remaining 10,000 samples are used to evaluate the performance and generalization ability of the model. The SVHN dataset is a large-scale dataset for digital recognition, which comes from Google Street View data. SVHN contains more than 600,000 color images, divided into a training set, a test set and an additional dataset, of which the training set contains 73,257 images, the test set contains 26,032 images, and the additional dataset contains 531,131 images.

[0069] Table 1 Datasets and DNN models

[0070]

[0071]

[0072] (2) Test case optimization: The test set data is input into the trained DNN model and divided into 10 sets C = {C1, C2, C3, ..., C 10}, that is, each set C i The test cases in (i∈[1,10]) are predicted by DNN to be of the same category. Then we i The PCV coefficient is calculated for the test cases in the set, and the test cases in the set are sorted according to the PCV coefficient, and the test cases are divided into three categories: the initial selection set, the initial candidate set, and the exclusion set. The initial selection set includes those test cases that have a high probability of causing the model to make wrong predictions, while the exclusion set includes those test cases that reduce the probability of wrong predictions of the model. The remaining test cases are classified into the candidate set.

[0073] (3) Test case selection process. The selection set and candidate set obtained after optimization in (2) are retained as the initial selection set and candidate set in our selection process. First, we set the end condition of the selection process. This condition is that when the number of selected test cases reaches the target we set, the algorithm stops running and returns the selected test case set. Next, in each predicted classification category, we will calculate the minimum image similarity of each test case x in the candidate test case set within the category to the selected test case set, and store this minimum image similarity value in the corresponding similarity queue. Finally, we sort the similarity queue in descending order and then re-sort the candidate set according to this sorting order. Next, select the first batch of elements from the candidate set and add them to the selection set. When the number of test cases in the test case set reaches the target number, the test case set is returned.

Claims

1. An adaptive test case optimization and selection method based on deep neural network, characterized in that: This method performs the following processing on the test case of the DNN model for N classification tasks: S1. Input the unlabeled test case set to be selected into the DNN model, divide it into N sets according to the DNN prediction results and the image classification target, and then divide each divided set into a selection set, a candidate set and a removal set by predicting the confidence variance; S2. For the image datasets in the selection set and the candidate set, the similarity between any two images is calculated using an image similarity calculation method based on model uncertainty, specifically including: S21, prediction: select one picture from the selection set and the candidate set respectively, denoted as picture a and picture b, and input them into the DNN model for classification prediction. The probability vectors P and Q output by the DNN model represent the prediction probabilities of picture a and picture b respectively; S22, reorganization and extraction: arbitrarily select three predicted probability values ​​from the probability vectors P and Q, thereby reorganizing the vectors P and Q into Three-dimensional sub-directions; S23, calculate the image similarity: record any set of three-dimensional subvectors after the vectors P and Q are reorganized as P'(p i ,p j ,p k ) and Q'(q i ,q j ,q k ), then consider these two vectors as coordinate points in three-dimensional space and project them to p i and q i The three-dimensional coordinate points p' and Q' are then obtained. The points p' and Q' form an angle with the origin O, which is denoted as angle α. Then, the coordinates corresponding to P' and Q' are used to calculate sin(α), and p is obtained. i and q i The absolute value of the difference; Similarly, it also includes projecting the above two-dimensional coordinate points P′ and Q′ onto p j and q j 、p k and q k Angles β and γ and sine calculations are performed on the plane, including absolute values; On this basis, a formula to measure the similarity of probability vectors P and Q is defined for the angle and difference information obtained in the above process. The expression is as follows: similarity (i,j,k) sin(α)*(|p i -q i |)+sin(β)*(|p j -q j |)+sin(γ)*(|p k -q k |) S24, similarity accumulation: recombining The similarity of the vectors is accumulated. For specific image inputs x1 and x2, when performing n-category classification tasks, the similarity between images x1 and x2 is calculated by the formula: Similarity(x1,x2)=∑similartty (i,j,k) ; S3. The selection set and candidate set optimized in step S2 are used as the initial set, and the selection is performed by calculating the similarity between the images in the candidate set and the images in the selection set, and the final selection set is returned.

2. The method for adaptive test case optimization and selection based on deep neural network according to claim 1, characterized in that: In step S1, the test case set C is divided into N subsets according to the prediction results output by the DNN model. i The test cases in are predicted by DNN to be of the same category, and then for each subset C i The test cases in the prediction confidence variance are calculated, and the prediction confidence variance and the subset C are calculated based on the prediction confidence variance and the prediction confidence variance. i The test cases in the set are sorted, and then the test cases are divided into an initial selection set, an initial candidate set, and an exclusion set according to the number of target test cases.

3. The method for adaptive test case optimization and selection based on deep neural network according to claim 1 or 2, characterized in that: The prediction confidence variance is calculated as follows: In the image classification task, given an input test case x, the predicted output of the DNN model is a vector with n values, denoted as P = <p1,p2,…,p n >, where each p i represents the probability that the input x is predicted to be the i-th category, and Sorting the vector P yields the vector P′= <p′1,p′2,...,p′ n >, where p′1≥p′2≥...≥p′ n ,; the probability difference between n predicted probabilities is defined as follows: Confidence variance:

4. The method for adaptive test case optimization and selection based on deep neural network according to claim 1, characterized in that: Step S23 is to evaluate the similarity between each test case x in the candidate set and all test cases in the selection set by using an image similarity calculation method. The shortest distance from the test case x in the candidate set to the selection set is obtained. After obtaining these minimum similarity values, the test case corresponding to the maximum value is selected to ensure a uniform distribution of the selected test cases in the set.

Images