A multi-method hybrid selection input method for neural network testing based on clustering

Abstract

The present invention discloses a multi-method hybrid selection input method for neural network testing based on clustering, which aims to combine traditional machine learning with traditional software testing algorithms. By using clustering algorithms, all test cases are divided into two categories according to their clustering effects: a part of multiple performance classes that can be clustered, and another part of noise classes that cannot be clustered. In the two categories, algorithms that have outstanding performance in traditional machine learning and traditional software testing are used respectively. While ensuring the local optimum of both, the global optimization of test case selection is achieved, thereby effectively improving the performance of the retrained neural network model. Compared with ARS‑DNNT, the method proposed in the present invention is better than ARS‑DNNT in terms of the selection effect, regardless of whether the value of the selected test case is high or low. The experimental effect of the effectiveness of CMS‑DNNT is better than that of ARS‑DNNT. The experimental results show that the accuracy of the retraining model of the case selected by CMS‑DNNT is at least 26% higher than that of ARS‑DNNT.

Description

Technical Field

[0001] The present invention belongs to the technical field of automated testing in neural network testing, and specifically proposes a neural network testing multi-method hybrid selection input method based on clustering, namely CMS-DNNT. Background Art

[0002] With the rapid development of deep neural networks, fields such as image classification, face recognition, and even natural language processing have been widely used. However, as more safety-critical fields (such as autonomous driving, medical diagnosis, etc.) begin to gradually use deep neural networks, people have put forward a new requirement for deep neural networks, namely robustness. As a neural network test that has emerged in recent years, it does have certain practical significance in these aspects. In the process of neural network testing, in order to improve the generalization of the model under diverse abnormal modes, expanding samples can be said to be a good strategy, but at the same time, expanding samples will also bring a new problem, that is, the model under test usually has a relatively large image input. Therefore, it is of great significance to select a part of the test input that can effectively identify model failure as a test case. At the same time, the development of the scale of deep learning systems is also constantly promoting the development of the diversity of neural network testing technology. Among the many testing methods, random testing (RT) is widely used in the quality assurance of various deep learning systems because of its simple concept and application methods. More specifically, RT is to randomly generate test cases in the input domain to drive the execution of the neural network under test, and determine whether the test case causes abnormal output of the neural network. However, RT only randomly generates test cases blindly, ignoring the relevant information of the test cases that have been executed, and also not paying attention to the properties of the neural network input itself that triggers the fault. Therefore, researchers have a certain controversy about the effectiveness of RT. In view of the fault detection capability of RT, Adaptive Random Testing (ART) is one of the improvement methods proposed for RT.

[0003] ART's appearance in the literature can be traced back to a journal paper published by Chen et al. in 2001. ART was inspired by observations independently reported by many researchers from many different fields about software failure patterns, which are: inputs that trigger program failures tend to cluster into continuous regions, called failure regions. In their paper, Chan et al. defined three failure modes: block failure mode, strip failure mode, and point failure mode, such as Figure 1The figure shows the three failure modes in a two-dimensional input domain. The bounding box represents the input domain boundary. The black bars, blocks, or dots represent the input distribution that causes the failure. Early studies have shown that bar and block failure modes are more common than dot failure modes.

[0004] According to the above analysis, if the fault region is continuous, then the non-fault region should also be continuous in the entire input domain. Specifically: if a test case t c If is an input that triggers a program failure, then its neighbors are also likely to be inputs that trigger program failures. Based on this, the goal of ART is to achieve a uniform distribution of test cases on the input domain while ensuring the randomness of RT. It usually includes two processes: one is the random generation of test inputs, and the other is to ensure the uniform distribution of generated test cases in the entire input domain.

[0005] ART is a cluster of testing methods based on various intuitions and criteria, among which the Fixed Size Candidates Set Version of ART (FSCS-ART) is the most classic and widely accepted algorithm. The core technologies used by FSCS-ART are: ① The concept of similarity between two test cases, i.e., distance measurement; ② Two test case sets are used: the executed test case set E (a set of test cases that have been executed and have not captured any exceptions in the program under test) and the candidate test case set C (a set of k test cases randomly generated in the input domain). If c′∈C wants to become the next test case, it must meet the following conditions:

[0006] c′∈C,min e∈E d(c′, e)≥min e∈E d(c, e),

[0007] Among them, d(x, y) is used to describe the similarity between test cases x and y, and represents the Euclidean distance (Euclidean distance) between x and y.

[0008] The Euclidean distance is defined as follows: The Euclidean distance between test cases tc1 and tc2 is denoted as: d(tc1, tc2). For an m-dimensional input domain, there is a test case a = {a1, a2, ..., a m} and test case b = {b1, b2, ..., b m},but:

[0009]

[0010] The introduction of ART into deep learning systems can be traced back to a journal paper published by Yan et al. in 2019. In order to improve the effectiveness of RT, ARTDL, which combines the fixed candidate set adaptive randomized algorithm FSCS-ART with a deep learning system, is proposed based on the algorithm, which has outstanding performance in traditional software testing. The basic idea of ​​the algorithm is to design an automated black-box testing method based on a deep learning system to improve and replace the use of RT. In the experimental design process, based on the feature vectors extracted from the output of each convolutional layer of the neural network, the Feature Euclidean distance (FED) based on the deep learning system is defined, and then the failure mode of the DL model is visualized by the feature vector, that is, the three-dimensional spatial diagram of the fault-causing input and the non-fault-causing input. Figure 2 It is shown that the feature Euclidean distance can be used as a measure of similarity between fault-causing inputs and non-fault-causing inputs.

[0011] The characteristic Euclidean distance is defined as follows: Test case t a and t b The Euclidean distance between them is denoted as FED(t a , t b ). For the m-dimensional feature vector, there are test cases t a =v a ={a1, a2...a m} and test case t b =v b ={b1, b2...b m},but:

[0012]

[0013] According to the above analysis, the ARTDL method simply introduces the fixed size candidate set adaptive random testing method (FSCS-ART), which has outstanding performance in traditional software testing, into the deep learning system. As an improved version of the randomness test of deep learning systems, the ARTDL proposed by MIN YAN has indeed made a qualitative leap in performance. However, in response to the high robustness requirements in more safety-critical fields today, the performance is still slightly insufficient. Considering that the greater the similarity of the extracted image feature data, the more consistent the corresponding test capabilities, the present invention will be based on the traditional clustering algorithm in the deep learning system, and optimize ARTDL to further improve the performance of the deep learning system.

[0014] After dividing the test inputs with different test capabilities into different groups, test cases are selected from each group to form a small set of test inputs. For multi-performance classes with relatively consistent test performance, it is difficult to perfectly distinguish different test capabilities through clustering. Each group has noise, that is, the test input should not be divided into this group. In order to avoid the influence of noise in each group, the test input that best represents the test capability is an interpretable group, also called a prototype. The most advanced instance-based Maximum Mean Discrepancy (MMD) algorithm in machine learning has certain advantages in increasing the interpretability of the selected test input. The algorithm calculates the difference between the prototype distribution and the group distribution, and determines the selected prototype while ensuring that the difference value is minimized. MMD calculates the expected value of the prototype and the group in the function space F, and uses it as a measure of the difference between the prototype and the group. The specific expression is as follows:

[0015] Summary of the invention

[0016] In view of the above problems, in order to effectively improve the efficiency of the adaptive random neural network testing method, the present invention proposes a multi-method hybrid selection input method based on clustering neural network testing. The technical solution of the present invention includes the following steps:

[0017] Step 1, determine the pre-trained neural network model D′ to be tested and the adversarial sample set S;

[0018] Step 2, extract the adversarial sample feature set V;

[0019] Step 3: Get two different categories: several performance categories l1, l2, ...l that account for the vast majority of the total number of samples j and the minority noise class l0;

[0020] Step 4, when the number of selected use cases is determined to be n, the selection ratio of the two categories is set to α=0.6;

[0021] Step 5: For two different categories, the divide-and-conquer approach is used to combine traditional machine learning with traditional software testing algorithms, and α*n and (1-α)*n test cases are selected respectively;

[0022] Step 6: Use the selected use cases as the training set T, and then train the old model D′ to obtain the new model D″;

[0023] Step 7, finally return the number of prediction errors of the new model D″ for adversarial samples, which is used as a metric to judge the performance of various selection algorithms;

[0024] Further, the specific process of step 1 is as follows:

[0025] Step 1.1, according to the previous software design documents, determine the neural network model D to be tested and the corresponding seed image database (N seed image sets I);

[0026] Step 1.2, according to the neural network model D to be tested determined in step 1.1 and the corresponding seed image database (N seed image set I), the neural network model to be tested is initialized, that is, the neural network model D to be tested is pre-trained through the corresponding seed image database, and then the pre-trained neural network model to be tested D′ is determined;

[0027] Step 1.3, based on the seed image database (N seed image set I) determined in step 1.1, perform sample expansion on the 33 transformation methods shown in Table 1 to generate a 33*N adversarial sample set S.

[0028] The specific process of step 2 above is as follows:

[0029] Step 2.1, according to D′ determined in 1.2, obtain all its convolutional layers cov1, cov2, ...;

[0030] Step 2.2, based on S determined in 1.3, extract its feature set V1, V2, ... in cov1, cov2, ..., and select a variable V that will affect the performance i . The experimental results verify that the feature set with the best effect is the optimal feature set V;

[0031] The specific process of step 3 above is as follows:

[0032] Step 3.1, use the density hierarchical clustering algorithm (HDBSCAN). Use distance as the similarity metric, and classify all test cases into different categories according to test performance based on the idea that the more consistent the test performance, the more concentrated the corresponding test cases. The specific steps are as follows:

[0033] Step 3.1.1, transform the space according to density / sparseness. The core of the clustering algorithm is single-link clustering, which is very sensitive to noise: a single noise data point in the wrong location can act as a bridge between islands, gluing them together. To eliminate the impact of noise, dense points (low core distance) are kept at the same distance from each other, but sparser points are pushed away so that their core distance is at least as far away from any other point. The mutual reachability distance is defined as follows:

[0034] d mreach-k (a, b) = max{core k(a), core k (b), d(a, b)}

[0035] Among them, core k (a) core k (b) represents the distance between point a and point b to their respective k closest points, and d(a, b) represents the original distance between points a and b.

[0036] Step 3.1.2, build the minimum spanning tree of the distance-weighted graph. Treat the data as a weighted graph, where the data points are vertices and the weight of the edge between any two points is equal to the mutual reachable distance between these points, that is, the value determined in step 3.1.1. Find a minimum set of edges such that removing any edge from the set will cause the component to be disconnected, that is, the minimum spanning tree is very efficiently constructed through Prim's algorithm.

[0037] Step 3.1.3, construct a cluster hierarchy of connected components. Sort the edges of the tree by distance (in increasing order), then traverse and create a new merged cluster for each edge. By union-finding the data structure, determine each edge that connects two clusters together. Finally, a hierarchy of connected components is obtained at different threshold levels (from fully connected to completely disconnected).

[0038] Step 3.1.4, compress the cluster hierarchy according to the minimum cluster size. Introduce the concept of a minimum cluster size and use it as a parameter of HDBSCAN. During the traversal of the hierarchy, determine whether there is a new cluster created by the split with fewer points than the minimum cluster size at each split. If there are fewer points than the minimum cluster size, declare them as "points removed from the cluster" and keep the larger cluster as the parent cluster. If the sizes of the clusters obtained by the split are all larger than the minimum cluster size, keep their splits in the tree. The result is a much smaller tree with a small number of nodes, each of which has data about how the cluster size of the node decreases with different distances.

[0039] Step 3.1.5, extract stable clusters from the compressed tree. To ensure that the selected clusters can persist and have a longer life cycle, that is, if a cluster is selected, any cluster of its descendants cannot be selected. Consider using a metric method different from distance to consider the persistence of clusters, that is, define the density metric of each point as:

[0040]

[0041] Where distance represents the shortest distance between the point and the remaining cluster points.

[0042] For a given cluster, define the value λ birth and λ deat is the value of λ when the cluster separates and becomes its own cluster, where λbi represents the λ value when the cluster is formed; λ death represents the value of λ when the cluster splits into two subclusters; and the value of λ when the cluster is split into smaller clusters (if any). Conversely, for a given cluster, for each point P in the cluster, we can set the value λ P is defined as the value of λ that “falls out of the cluster”, which is between λ bir and λ death Now, for each cluster, the stability is calculated as follows:

[0043] S cluster =∑ p∈cluster (λ p -λ birt )

[0044] λ P is the lambda value when separating from the cluster. All leaf nodes are declared as selected clusters. By traversing the tree (in reverse topological sorting order), it is determined whether the sum of the stabilities of the subclusters is greater than the stability of the cluster. If so, the cluster stability is set to the sum of the stabilities of the subclusters. Otherwise, the cluster is declared as the selected cluster and all its subclusters are deselected. When the root node is traversed, the set of all currently selected clusters is returned.

[0045] Step 3.2, according to step 3.1.4, set the minimum cluster parameter value to 50, determine whether the number of test cases in the obtained cluster is less than the threshold of 50, and aggregate all clusters with a threshold of ≤50, that is, all the "points removed from the cluster" into a noise class l0, and the remaining clusters l1, l2, ..., l j Defined into several performance classes.

[0046] The specific process of step 5 above is as follows:

[0047] Step 5.1, that is, for several performance categories, select the Maximum Mean Discrepancy (MMD) algorithm with outstanding performance in machine learning from the use cases with relatively consistent test capabilities. In the selection process, the distribution P1, P2...P of the selected use cases in each performance category within the performance category is calculated respectively. j , and the class distribution G of each performance class in the total performance class, take MMD as the test statistic. In order to make the two distributions as similar as possible, the selected α*n test cases need to make the MMD value as small as possible. In addition, the number of test cases selected in each performance class is selected based on the ratio of the number of test cases in the performance class to the total number of test cases in the total performance class;

[0048] Step 5.2: For the noise class, select the Adaptive Random Selection (FSCS-ARS) method based on a fixed candidate set, which has a strong performance in relatively scattered test cases in traditional software testing, and select the number of test cases (1-α)*n;

[0049] Step 5.2.1, the present invention uses Feature Euclidean distance (FED) to measure the similarity between test cases. The closer the distance, the higher the similarity between the two test cases.

[0050] Step 5.2.2, based on V obtained in step 2, initialize the executed test case set E and the candidate test case set C; the initial value t0 of E is a randomly selected non-abnormal adversarial sample vector feature vector v0, and the size of C is fixed to 10.

[0051] Step 5.2.3, calculate each candidate test case c in C i With every element e in E i (Currently there is only one element t0 in E) characteristic Euclidean distance, select the minimum distance as the similarity value of the candidate test case, and select the test case with the lowest similarity value (maximum minimum distance) among the candidate test cases as the next test case t1;

[0052] The characteristic Euclidean distance is defined as follows: Test case t a and t b The Euclidean distance between them is denoted as FED(t a , t b ). For the m-dimensional feature vector, there are test cases t a =v a ={a1, a2, ..., a m} and test case t b =v b ={b1, b2, ..., b m}, where a i 、b i Represent test cases t a ,t b In the eigenvector of the i-th dimension, then:

[0053]

[0054] Step 5.2.4, if t1 detects an anomaly, add t1 to E and repeat step 5.2.3 until the number of selected use cases reaches (1-α)*n.

[0055] The specific process of step 6 above is as follows:

[0056] Step 6.1: Use the n test cases selected in step 5 as the training set T = {t0, t1, ..., t n};

[0057] Step 6.2, retrain the model D′ according to T obtained in step 6.1 to obtain the final new model D″;

[0058] The specific process of step 7 above is as follows:

[0059] Step 7.1: Use the remaining adversarial samples in step 1 as the test set T′={t′0, t′1, ..., t′ n};

[0060] Step 7.2: Based on T′ obtained in step 7.1, input the new model D″ to determine whether its predicted value is consistent with the label value, and then judge the performance of the test case selection algorithm based on the number of errors obtained.

[0061] Table 1 33 transformations performed when expanding the sample of the original database

[0062]

[0063] Beneficial effects of the present invention:

[0064] 1. The present invention constructs a multi-method hybrid selection input method for neural network testing based on clustering, which can improve the algorithm performance of ARTDL in application and neural network testing when the number of test cases is certain.

[0065] 2. In the test case selection process, the present invention adopts the divide-and-conquer idea based on the clustering algorithm, and properly combines the FSCS-ARS and MMD algorithms in traditional tests, so that in the process of neural network test input selection, there is an essential improvement compared with the prior art ARTDL directly using FSCS-ARS. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 These are the three failure modes in the two-dimensional case;

[0067] Figure 2 It is the failure mode of the feature vector visualization DL model under different convolutional layers;

[0068] Figure 3 It is a flowchart of the FSCS_ARS method;

[0069] Figure 4 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0070] The present invention combines traditional machine learning with traditional software testing algorithms. By using a clustering algorithm, all test cases are divided into two categories according to their clustering effects: a part of multiple performance classes that can be clustered, and another part of noise classes that cannot be clustered. In the two categories, the algorithms that have outstanding performance in traditional machine learning and traditional software testing are used respectively. Under the condition of ensuring the local optimum of both, the global optimization of test case selection is achieved, thereby effectively improving the performance of the retrained neural network model. The present invention mainly includes: 1. Sample expansion to generate adversarial samples; 2. Feature extraction of adversarial samples based on different convolutional layers of neural networks; 3. Using a density hierarchical clustering algorithm, all test cases are classified into different categories according to test performance. Since a few test cases have special performance, the clustered classes are too small, and these small classes are classified into one large class, namely the noise class. At this time, there are several different classes as follows: several performance classes that account for the majority of the total; and noise classes that account for a minority. 4. When the number of selected cases is determined to be n, the selection ratio of the two categories is set to α=0.6; 5. For two different categories, traditional machine learning and traditional software testing algorithms are used respectively, that is, for several performance categories, the Maximum Mean Discrepancy (MMD) algorithm with outstanding selection ability in machine learning cases with relatively consistent test ability is selected; for the noise category, the Adaptive Random Selection (FSCS-ARS) method based on a fixed candidate set with strong performance in relatively dispersed cases is selected in software testing. 6. The selected cases are used as training sets, and the old model is retrained to achieve the use of a small number of adversarial samples, while reducing the interference caused by artificial labels, and at the same time, the recognition accuracy of the neural network in various abnormal environments can be improved to the greatest extent. Through experimental verification, the method proposed by the present invention is compared with ARS-DNNT. In terms of the selection effect, regardless of whether the value of the selected test case is high or low, the effectiveness experimental effect of CMS-DNNT is better than ARS-DNNT. Experimental results show that the accuracy of the retrained model for the use cases selected by CMS-DNNT is at least 26% higher than that of ARS-DNNT.

[0071] The present invention will be further described below in conjunction with the accompanying drawings.

[0072] The present invention aims to improve the selection performance of the FSCS_ARS method and provides a multi-method hybrid selection input method for neural network testing based on clustering. To verify the method of the present invention, an experiment with 10, 20, 30, 40, and 50 selection cases and an input database of MNIST is used to illustrate it.

[0073] like Figure 4 As shown, the algorithm proposed by the present invention comprises the following steps:

[0074] Step 1, determine the pre-trained neural network model D′ to be tested and the adversarial sample set S;

[0075] Step 2, extract the adversarial sample feature set V;

[0076] Step 3: Obtain two different categories: several performance categories l1, l2, ..., l that account for the vast majority of the total number of samples. j and the minority noise class l0;

[0077] Step 4, when the number of selected use cases is determined to be n, the selection ratio of the two categories is set to α=0.6;

[0078] Step 5: For two different categories, the divide-and-conquer approach is used to combine traditional machine learning with traditional software testing algorithms, and α*n and (1-α)*n test cases are selected respectively;

[0079] Step 6: Use the selected use cases as the training set T, and then train the old model D′ to obtain the new model D″;

[0080] Step 7, finally return the number of prediction errors of the new model D″ for adversarial samples, which is used as a metric to judge the performance of various selection algorithms;

[0081] The specific process of step 1 above is as follows:

[0082] Step 1.1, according to the previous software design documents, determine the neural network model D to be tested and the corresponding seed image database (N seed image sets I);

[0083] Step 1.2, according to the neural network model D to be tested determined in step 1.1 and the corresponding seed image database (N seed image set I), the neural network model to be tested is initialized, that is, the neural network model D to be tested is pre-trained through the corresponding seed image database, and then the pre-trained neural network model to be tested D′ is determined;

[0084] Step 1.3, based on the seed image database (N seed image set I) determined in step 1.1, perform sample expansion on the 33 transformation methods shown in Table 1 to generate a 33*N adversarial sample set S.

[0085] The specific process of step 2 above is as follows:

[0086] Step 2.1, according to D′ determined in 1.2, obtain all its convolutional layers cov1, cov2, ...;

[0087] Step 2.2, based on S determined in 1.3, extract its feature set V1, V2, ... in cov1, cov2, ..., and select a variable V that will affect the performance i . The experimental results verify that the feature set with the best effect is the optimal feature set V;

[0088] The specific process of step 3 above is as follows:

[0089] Step 3.1, use the density hierarchical clustering algorithm (HDBSCAN). Use distance as the similarity metric, and classify all test cases into different categories according to test performance based on the idea that the more consistent the test performance, the more concentrated the corresponding test cases. The specific steps are as follows:

[0090] Step 3.1.1, transform the space according to density / sparseness. The core of the clustering algorithm is single-link clustering, which is very sensitive to noise: a single noise data point in the wrong location can act as a bridge between islands, gluing them together. To eliminate the impact of noise, dense points (low core distance) are kept at the same distance from each other, but sparser points are pushed away so that their core distance is at least as far away from any other point. The mutual reachability distance is defined as follows:

[0091] d mreach-k (a, b) = max{core k (a), core k (b), d(a, b)}

[0092] Among them, core k (a) core k (b) represents the distance between point a and point b to their respective k closest points, and d(a, b) represents the original distance between points a and b.

[0093] Step 3.1.2, build the minimum spanning tree of the distance-weighted graph. Treat the data as a weighted graph, where the data points are vertices and the weight of the edge between any two points is equal to the mutual reachable distance between these points, that is, the value determined in step 3.1.1. Find a minimum set of edges such that removing any edge from the set will cause the component to be disconnected, that is, the minimum spanning tree is very efficiently constructed through Prim's algorithm.

[0094] Step 3.1.3, construct a cluster hierarchy of connected components. Sort the edges of the tree by distance (in increasing order), then traverse and create a new merged cluster for each edge. By union-finding the data structure, determine each edge that connects two clusters together. Finally, a hierarchy of connected components is obtained at different threshold levels (from fully connected to completely disconnected).

[0095] Step 3.1.4, compress the cluster hierarchy according to the minimum cluster size. Introduce the concept of a minimum cluster size and use it as a parameter of HDBSCAN. During the traversal of the hierarchy, determine whether there is a new cluster created by the split with fewer points than the minimum cluster size at each split. If there are fewer points than the minimum cluster size, declare them as "points removed from the cluster" and keep the larger cluster as the parent cluster. If the sizes of the clusters obtained by the split are all larger than the minimum cluster size, keep their splits in the tree. The result is a much smaller tree with a small number of nodes, each of which has data about how the cluster size of the node decreases with different distances.

[0096] Step 3.1.5, extract stable clusters from the compressed tree. To ensure that the selected clusters can persist and have a longer life cycle, that is, if a cluster is selected, any cluster of its descendants cannot be selected. Define a metric method different from distance to consider the persistence of clusters, that is, define the density metric of each point as:

[0097]

[0098] Where distance represents the shortest distance between the point and the remaining cluster points.

[0099] For a given cluster, define the value λ birt and λ death is the value of λ when the cluster separates and becomes its own cluster, and the value of λ when the cluster is split into smaller clusters (if any). Conversely, for a given cluster, for each point P in the cluster, we can set the value of λ P is defined as the value of λ that “falls out of the cluster”, which is between λ birth and λ deat Now, for each cluster, the stability is calculated as follows:

[0100] S cluster =∑ p∈cluster (λ p -λ birth)

[0101] where λ birth : λ value when clusters are formed; λ death : the λ value when the cluster splits into two sub-clusters; λ PThe lambda value when separating from the cluster. Declare all leaf nodes as selected clusters, and determine whether the sum of the stabilities of the subclusters is greater than the stability of the cluster by traversing the tree (in reverse topological sort order). If so, set the cluster stability to the sum of the subclusters' stabilities, otherwise declare the cluster as the selected cluster and deselect all its subclusters. When traversing to the root node, return the set of all currently selected clusters.

[0102] Step 3.2, according to step 3.1.4, set the minimum cluster parameter value to 50, determine whether the number of test cases in the obtained cluster is less than the threshold of 50, and aggregate all clusters with a threshold of ≤50, that is, all the "points removed from the cluster" into a noise class l0, and the remaining clusters l1, l2, ..., l j Defined into several performance classes.

[0103] The specific process of step 5 above is as follows:

[0104] Step 5.1, that is, for several performance categories, select the Maximum Mean Discrepancy (MMD) algorithm with outstanding performance in machine learning from the use cases with relatively consistent test capabilities. In the selection process, the distribution P1, P2...P of the selected use cases in each performance category within the performance category is calculated respectively. j , and the class distribution G of each performance class in the total performance class, take MMD as the test statistic. In order to make the two distributions as similar as possible, the selected α*n test cases need to make the MMD value as small as possible. In addition, the number of test cases selected in each performance class is selected based on the ratio of the number of test cases in the performance class to the total number of test cases in the total performance class;

[0105] Step 5.2: For the noise class, select the Adaptive Random Selection (FSCS-ARS) method based on a fixed candidate set, which has a strong performance in relatively scattered test cases in traditional software testing, and select the number of test cases (1-α)*n;

[0106] Step 5.2.1, the present invention uses Feature Euclidean distance (FED) to measure the similarity between test cases. The closer the distance, the higher the similarity between the two test cases.

[0107] Step 5.2.2, based on V obtained in step 2, initialize the executed test case set E and the candidate test case set C; the initial value t0 of E is a randomly selected non-abnormal adversarial sample vector feature vector v0, and the size of C is fixed to 10.

[0108] Step 5.2.3, calculate each candidate test case c in C i With every element e in E i (Currently there is only one element t0 in E) characteristic Euclidean distance, select the minimum distance as the similarity value of the candidate test case, and select the test case with the lowest similarity value (maximum minimum distance) among the candidate test cases as the next test case t1;

[0109] The characteristic Euclidean distance is defined as follows: Test case t a and t b The Euclidean distance between them is denoted as FED(t a , t b ). For the m-dimensional feature vector, there are test cases t a =v a ={a1, a2, ..., a m} and test case t b =v b ={b1, b2, ..., b m}, where a i 、b i Represent test cases t a ,t b In the eigenvector of the i-th dimension, then:

[0110]

[0111] Step 5.2.4, if t1 detects an anomaly, add t1 to E and repeat step 5.2.3 until the number of selected use cases reaches (1-α)*n.

[0112] The specific process of step 6 above is as follows:

[0113] Step 6.1: Use the n test cases selected in step 5 as the training set T = {t0, t1, ..., t n};

[0114] Step 6.2, retrain the model D′ according to T obtained in step 6.1 to obtain the final new model D″;

[0115] The specific process of step 7 above is as follows:

[0116] Step 7.1: Use the remaining adversarial samples in step 1 as the test set T′={t′0, t′1, ..., t′ n};

[0117] Step 7.2: Based on T′ obtained in step 7.1, input the new model D″ to determine whether its predicted value is consistent with the label value, and then judge the performance of the test case selection algorithm based on the number of errors obtained.

[0118] Table 1 33 transformations performed when expanding the sample of the original database

[0119]

[0120] Algorithm Verification Results Analysis

[0121] The present invention aims to improve the performance of selection when the number of test cases is certain. In neural network testing, the correctness of the test case output result is a significant feature to measure the performance of the model. Therefore, after the selected adversarial samples are retrained with the neural network model, it is of great practical significance to predict the number of errors caused by the remaining adversarial samples as the inspection standard. The average number of errors caused by the remaining adversarial samples predicted during the neural network testing process is used as a measurement standard, which is called Error_num. For the method of the present invention, an accuracy rate of 5% and a confidence level of 95% are adopted. According to the central limit theorem, the number of experiments required is at least 2000 times.

[0122] The present invention sets the input sample database as MNIST, and sets the number of selected use cases to 10, 20, 30, 40, and 50. These five groups of experiments obtain the number of prediction errors of the new model obtained by retraining the samples selected by the RT method, the MMD method, the FSCS-ARS method, and the method of the present invention. The specific comparison results are shown in Table 2.

[0123] Through experimental verification, although the performance of MMD-DNNT is better than that of the method proposed in this invention when the sample size is small, in general, the selection effect of CMS is better than that of MMD. Compared with ARS-DNNT, the method proposed in this invention is better than ARS-DNNT in terms of the selection effect, regardless of whether the value of the selected test case is high or low. The experimental results show that the accuracy of the retraining model of the case selected by CMS-DNNT is at least 26% higher than that of ART-DNNT.

[0124] Table 2 Error_num experimental results comparison

[0125] Select the number n RT MMD FSCS-ART Method of the present invention Improvement rate n＝10 860 93 136 99 27.20％ n＝20 524 67 68 49 27.90％ n＝30 386 74 50 37 26.00％ n＝40 297 89 41 30 26.80％ n＝50 252 105 34 25 26.40％

[0126] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-method hybrid selection input method for neural network testing based on clustering, characterized in that: The steps include: S1, determine the pre-trained neural network model D′ to be tested and the adversarial sample set S; S2, extract the adversarial sample feature set V; S3, obtain two different categories: the first category is a number of performance classes l1, l2...l that are in the majority j , the second category is the noise category l0 which is a minority; the specific process of step S3 includes: S3.1, using density hierarchical clustering algorithm, all test cases are classified into different categories according to test performance; S3.2, determine whether the number of test cases in the obtained class is less than the threshold 50, aggregate all classes with the threshold ≤ 50 into a noise class l0, and group the remaining classes l1, l2, ..., l j Defined as a number of performance classes; S4, set the selection ratio value α of the two categories, and α>0.5; S5, for two different categories, combines traditional machine learning with traditional software testing algorithms, and selects α*n and (1-α)*n test cases respectively when the number of selected test cases is n; The specific process of S5 includes the following: S5.1, that is, for several performance classes, select the maximum mean difference algorithm MMD with outstanding ability from the test cases with relatively consistent test ability in machine learning, select the number of test cases α*n, and select the number of test cases in each performance class according to the ratio of the number of test cases of the performance class to the total number of test cases of all performance classes; S5.2, for the noise category, select the FSCS-ART method based on the fixed candidate set adaptive random test, which has strong performance in relatively scattered test cases in traditional software testing, and select the number of test cases (1-α)*n; specifically: S5.2.1, use the feature Euclidean distance to measure the similarity between test cases. The closer the distance, the higher the similarity between the two test cases. S5.2.2, based on V obtained in step S2, initialize the executed test case set E and the candidate test case set CX; the initial value t0 of E is a randomly selected non-anomalous adversarial sample vector feature vector v0, and the size of C is fixed to 10; S5.2.3, calculate each candidate test case c in C i With every element e in E i The feature Euclidean distance is selected, and the minimum distance is selected as the similarity value of the candidate test case. The test case with the lowest similarity value among the candidate test cases, that is, the test case with the largest minimum distance, is selected as the next test case t1; The characteristic Euclidean distance is defined as follows: Test case t a and t b The Euclidean distance between them is denoted as FED(t a ,t b ), for the m-dimensional feature vector, there are test cases t a =v a ={a1,a2,…,a m } and test case t b =v b ={b1,b2,…,b m },but: S5.2.4, if t1 detects an anomaly, add t1 to E and repeat step 5.2.3 until the number of selected use cases reaches (1-α)*n.

2. According to claim 1, a clustering-based neural network test multi-method hybrid selection input method is characterized in that: The specific process of step S1 includes: S1.1, according to the software design document, determine the neural network model D to be tested and the corresponding seed image database, wherein the database is a set of N seed images I; S1.2, according to the neural network model D to be tested determined in S1.1 and the corresponding seed image database, the neural network model to be tested is initialized, and the neural network model to be tested is pre-trained through the corresponding seed image database to obtain the pre-trained neural network model to be tested D′; S1.3, based on the corresponding seed image database determined in S1.1, expand the samples by changing brightness, translating, scaling, and blurring M operations to generate a set S of M*N adversarial samples.

3. The multi-method hybrid selection input method for neural network testing based on clustering according to claim 1 is characterized in that: The specific process of step S2 includes: S2.1, according to D′ determined in S1.2, determines its different convolutional layers cov1, cov2, ...; S2.2, based on S determined in S1.3, extract its feature set V1, V2, ... in cov1, cov2, ..., and select the variable V that will affect the performance i , determine the optimal feature set V.

4. The multi-method hybrid selection input method for neural network testing based on clustering according to claim 1 is characterized in that: The specific steps of step S3.1 include: Step 3.1.1, transform the space according to density / sparseness: The core of the clustering algorithm is single-link clustering, which is very sensitive to noise: a single noise data point in the wrong location can act as a bridge between islands, gluing them together to eliminate the impact of noise, that is, dense points keep the same distance from each other, but sparser points are pushed away so that their core distance is at least as far away from any other point; define the mutual reachable distance as follows: d mreach-k (a,b)=max{core k (a),core k (b),d(a,b)} Step 3.1.2, build the minimum spanning tree of the distance-weighted graph: treat the data as a weighted graph, where the data points are vertices and the weight of the edge between any two points is equal to the mutually reachable distance between these points, that is, step 3.1.1 determines the value to find a minimum set of edges, so that removing any edge from the set will cause the component to be disconnected, that is, the minimum spanning tree is very efficiently constructed through Prim's algorithm; Step 3.1.3, construct a cluster hierarchy of connected components: sort the edges of the tree in increasing order according to distance, then traverse and create a new merged cluster for each edge, and determine each edge that connects two clusters together by union-finding the data structure; finally, a hierarchy of connected components is obtained at different threshold levels, from fully connected to completely disconnected; Step 3.1.4, compress the cluster hierarchy according to the minimum cluster size: introduce the concept of a minimum cluster size and use it as a parameter of HDBSCAN. During the traversal of the hierarchy, determine whether there is a new cluster created by the split with fewer points than the minimum cluster size at each split; if there are fewer points than the minimum cluster size, declare it as a 'point removed from the cluster' and keep the larger cluster as the parent cluster. If the sizes of the split clusters are all larger than the minimum cluster size, keep the split in the tree; finally, you get a much smaller tree with a small number of nodes, each node has data about how the cluster size of the node decreases with different distances; Step 3.1.5, extract stable clusters from the compressed tree: To ensure that the selected clusters can persist and have a longer life cycle, that is, if a cluster is selected, any of its descendant clusters cannot be selected. A measurement method different from distance is defined to consider the persistence of the cluster. The specific expression is as follows: For a given cluster, define the value λ birth and λ death is the value of λ when the cluster separates and becomes its own cluster, and the value of λ when the cluster is split into smaller clusters; conversely, for a given cluster, for each point P in the cluster, set the value of λ P is defined as the value of λ that “falls out of the cluster”, which is between λ birth and λ death Now, for each cluster, the stability is calculated as follows: where λ birth : λ value when clusters are formed; λ death : the λ value when the cluster splits into two sub-clusters; λ P The lambda value when separating from the cluster declares all leaf nodes as selected clusters. By traversing the tree in reverse topological sorting order, it is determined whether the sum of the stabilities of the subclusters is greater than the stability of the cluster. If so, the cluster stability is set to the sum of the stabilities of the subclusters. Otherwise, the cluster is declared as the selected cluster and all its subclusters are deselected. When traversing to the root node, the set of all currently selected clusters is returned.

5. The multi-method hybrid selection input method for neural network testing based on clustering according to claim 1 is characterized in that: In the step S4, α=0.

6.

6. The multi-method hybrid selection input method for neural network testing based on clustering according to claim 1, characterized in that: Also included are the following method steps to verify the test performance: S6, using the selected use cases as the training set T, training the neural network model D′, and obtaining a new model D″; S7, according to the new model D″, the number of prediction errors for adversarial samples is used as a metric to judge the performance of various selection algorithms.

7. The multi-method hybrid selection input method for neural network testing based on clustering according to claim 6 is characterized in that: The specific implementation of S6 includes: S6.1, use the n test cases selected in step S5 as the training set T = {t0, t1, ..., t n }; S6.2, retrain the model D′ according to T obtained in step S6.1 to obtain the final new model D″.

8. The multi-method hybrid selection input method for neural network testing based on clustering according to claim 6 is characterized in that: The specific implementation of S7 includes: S7.1, use the remaining adversarial samples in step S1 as the test set T′={t′0,t′1,…,t′ n }; S7.2, according to T′ obtained in step S7.1, input the new model D″, determine whether its predicted value is consistent with the label value, and judge the performance of the test case selection algorithm based on the number of errors obtained.

Images