A deep learning software testing adequacy metric method based on input feature diversity

Abstract

The application relates to a test sufficiency measurement method based on input feature diversity. Firstly, images in a training set are segmented and a pixel region set is generated; secondly, pixels in the set are clustered to select pixel features, important scores of all the pixel features are calculated, and an important feature set is formed; thirdly, important features of different test data are calculated and marked; then, the test set is artificially clustered according to the important features, a centroid set is calculated and generated; then, the distance between the centroids is calculated by using the Euclidean distance; finally, the centroid set is converted into a multi-dimensional space graph, the diversity of the test set based on distance entropy is calculated, and the sufficiency of the test set is effectively evaluated. The application aims at the evaluation problem of the sufficiency of deep learning software, through analyzing the feature diversity of the test set, the sufficiency and interpretability of software testing can be improved, and the method for guaranteeing the quality of deep learning software is provided.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of software testing, especially to the field of test sufficiency measurement of deep learning software (DL software for short), and aims to explain the sufficiency of data in the test set by using the internal structure of deep neural network. It is a method for helping software testers to improve the sufficiency and interpretability of software testing and guaranteeing the quality of deep learning software. BACKGROUND

[0002] In the current era of big data, deep learning (DL) is a key technology in the field of new generation artificial intelligence, which is driving the rapid development of artificial intelligence. Deep learning adopts a three-level structure model of "neuron-layer-network". Among them, the neuron is the basic computing unit in the neural network, a number of neurons are combined in the width direction to form a layer to realize the extraction and expression of features, and the input layer, the hidden layer, the activation function and the output layer are connected along the depth direction to form a network, which is finally mapped to the output end. At the current stage, various deep neural networks (DNN) have gradually become the key technology in the fields of automatic driving, face unlocking, etc. Due to the diversification of DL practical application scenarios and the complication of DNN internal, the software is extremely likely to output incorrect results in application, causing serious consequences. For example, in 2020, Google and Tesla's autonomous vehicles had multiple car accidents in different scenarios, causing serious casualties. Therefore, the software quality problem of deep learning needs to be paid attention to.

[0003] To carry out sufficient testing on software and find potential defects is the key to guarantee the quality of software system. In the field of traditional software engineering, researchers design test schemes through white-box code understanding or black-box function understanding. Deep learning defines a new data-driven program paradigm. The system is usually completed by a large number of data training on the model framework constructed by human, to realize a series of intelligent processing such as high-dimensional space nonlinear transformation. This also leads to the characteristics of deep learning software system such as running un-understandability and requirement uncertainty, so the traditional software testing method is difficult to meet the sufficiency testing requirement. Therefore, testers construct test sufficiency standards for deep learning software by migrating test coverage measurement indicators of traditional software such as statement coverage, judgment coverage and condition coverage. These standards measure the output, combination or path coverage of neurons in DNN in the test set, to illustrate the sufficiency of the test process.

[0004] However, given the paradigm of deep neural networks, the existing neuron-based coverage measurement method still has effectiveness doubts, mainly including the following two aspects: ① The coverage method lacks fine-grained performance of the test, and the test index only measures the overall performance of the dataset on the model, ignoring the influence of a single data on the model defect detection; ② The coverage method lacks reasonable explanation of the deep learning software, and only shows the coverage percentage of the dataset on the model, ignoring the characteristics of the data itself and the decision influence on the model. These two factors seriously restrict the reliability and explainability of the deep learning software test sufficiency method. Therefore, the DL software test considers the influence of different data in the test set on the result, and evaluates the sufficiency of the test process through the explainable performance on the model. In traditional software, input features and coverage features are two features of traditional software testing. The input feature does not depend on the coverage degree of the data in the software, but more concerned about the characteristics of the data itself. The tester observes whether the test data has enough features to illustrate the sufficiency of the test case set. In deep learning software, the input feature is particularly obvious. Taking a picture dataset as an example, the features of the picture are sorted from low to high, corresponding to texture, color, scene, object and other contents. In the process of deep neural network training, these features can be learned by neurons in different layers of the deep neural network, and make different contributions to the final output of the model.

[0005] Therefore, the present application proposes a test sufficiency measurement method based on input feature diversity. The basic idea of the method is: introducing the concept of input features in DL software testing, using the explainable method of DNN model to find the input features important to model decision, using distance entropy measurement method to evaluate the diversity of test set covering input features as the evaluation standard of test set sufficiency. First, the training data in the training set is taken out in turn, the pixel is segmented and the set is generated according to the image pixel interval; second, the pixel region set is clustered, and the importance score of all features is calculated using the training set, and the top m features with the highest score are selected to form a feature set; third, the test data in the test set is traversed, the top feature with the highest score is calculated and marked as an important feature, and a feature label array of the test data is generated; fourth, the test set is artificially clustered according to the feature label array, and a centroid set is calculated and generated; fifth, the Euclidean distance between the centroids is calculated to generate a corresponding distance matrix; finally, the centroid set is converted into a graph in a multi-dimensional space, and the diversity of the test set based on distance entropy is calculated to effectively evaluate the sufficiency of the test set.

[0006] In this method, testers use the training set to mine input features that are highly correlated with the model's decisions, measure the distribution of test set data in the multidimensional space of input features, and then conduct thorough testing on the DNN model to discover potential defects within the model. Ultimately, this effectively reduces the testing time of DL software testing and improves the system's development efficiency. Summary of the Invention

[0007] This invention effectively addresses the current challenge of evaluating the adequacy of deep learning (DL) software tests by providing a test adequacy measurement method based on input feature diversity. Utilizing a measurement index based on input feature diversity helps software testers accurately assess the adequacy of the test set, ensuring the quality of DL software.

[0008] To achieve the above objectives, this invention proposes a sufficiency measurement method for deep learning software testing based on input feature diversity. First, training data from the training set TR is sequentially extracted, and pixels are segmented according to image pixel intervals s to generate a pixel region set T. Second, T is clustered, and the importance scores of all features are calculated using the training set TR. The m features with the highest scores are selected to reconstruct a feature set F. Third, data from the test set T is sequentially extracted, and the features with the highest scores are calculated and labeled as important features, generating a feature label array L for the test data. Next, the test set T is manually clustered based on the feature label array L, and a centroid set C is calculated and generated. Then, the distance between centroids is calculated using Euclidean distance, generating the corresponding distance matrix D. Finally, the centroid set is converted into a multi-dimensional graph G, and the test set diversity dis_entropy based on distance entropy is calculated to effectively evaluate the sufficiency of the test set. Specifically, this method includes the following steps:

[0009] 1) Image segmentation of training data. Given a training set TR = {tr1, tr2, ..., tr...} n Generate a set of pixel region centers and update it using the lowest gradient algorithm, then calculate the pixel center c. j to pixel p k Euclidean distance (c) j ,p k The distance is stored in the distance array D. The formula for calculating the Euclidean distance is as follows:

[0010]

[0011] Generate a label array L to record the pixel intervals corresponding to the pixels, complete the pixel segmentation, and generate a set of pixel regions T.

[0012] 2) CAV Important Feature Selection. Given a training set TR = {tr1, tr2, ..., tr...} n} and the pixel region set T, clustering the pixel region set, selecting the pixel region closest to the centroid in each cluster as the feature, and generating a feature set F = {f1, f2, …, f l}, calculating the importance scores of all features using the training set, and the calculation formula is as follows:

[0013]

[0014] Where tr i represents the i-th data in the training set, f j represents the j-th feature, represents the gradient of layer, f layer represents the activation function of the hidden layer layer in the model, represents the activation feature vector (Concept Activation Vectors, CAV) of the feature f i . And the m features with the highest scores are selected to reconstitute the feature set.

[0015] 3) Test data feature mining. Input test set T = {t1, t2, …, t n}, initialize the feature label array L = <0, 0, …, 0> and the score array S = <0, 0, …, 0>, and according to the important feature set F = {f1, f2, …, f l} obtained by the training data image cutting and CAV important feature selection step and the trained model M, the importance scores of the test data in the test set to the features are calculated in turn. The calculation formula is as follows:

[0016]

[0017] Where t i represents the i-th data in the test set, f j represents the j-th feature, represents the gradient of layer, f layer represents the activation function of the hidden layer layer in the model, represents the CAV of the feature f i . Mark the feature with the highest score as the important feature of f j , and then generate the feature label array L of the test data accordingly.

[0018] 4) Multi-dimensional space data clustering. Input test set T = {t1, t2, …, t n} and the obtained feature label array L, initialize the centroid set C = {c1, c2, …, c lThe test data is divided into centroids based on the feature label array, and the set of centroids of each cluster is updated until all test data and the set of centroids of each cluster have been completely traversed. This is done using c. k =c k The method for calculating / num yields the centroid c of the k-th new cluster. k , where num represents the number of test data for the k-th new cluster.

[0019] 5) Inter-cluster centroid distance measurement, using Euclidean distance(c i ,c j Calculate the distance between any two centroids in the cluster centroid set and create a distance matrix D. Store the two endpoints and the distance magnitude of each distance segment in matrix D: each row stores one distance segment, where the first two columns store the two endpoints of the distance and the third column stores the distance magnitude. Sort the rows in matrix D in ascending order according to the data in their third column, i.e., the distance magnitude, and then output matrix D.

[0020] 6) Calculation of distribution diversity entropy. Given a distance matrix D containing stored distance information and sorted in ascending order of distance, create a multidimensional spatial graph G =<N,E> Add the edges and their endpoints from matrix D to graph G in ascending order of weight. After each edge addition, check that graph G does not contain cycles. If adding an edge results in a cycle in graph G, remove the edge from graph G and continue adding the next edge. If the edges in the graph have connected all the centroids in the centroid set to the graph, stop adding edges. Calculate the weight set Γ = {γ1, γ2, ..., γ...} corresponding to the edges in graph G. l-1 The sum of the elements in} γ Sum The diversity dis_entropy of the test set is calculated based on the distance entropy of the centroid set calculated using graph G. The calculation formula is as follows:

[0021]

[0022] Where, γ i Let represent the i-th element in set Γ; finally, the sufficiency of the test set is effectively evaluated using dis_entropy.

[0023] Furthermore, the specific steps of step 1) above are as follows:

[0024] Step 1)-1: Initial state;

[0025] Step 1)-2: Input training set TR= <tr1,tr2,…,tr n >, image pixel range s;

[0026] Steps 1)-3: Create a set of pixel regions T = {};

[0027] Step 1 )-4: i = 1, execute Step 1 )-6;

[0028] Step 1 )-5: i ++;

[0029] Step 1 )-6: Calculate tr i , the number of divisible pixel regions, denoted as num, is calculated as follows:

[0030] num = (tr i .size / s) 2

[0031] C = <c1, c2,..., c num > represents all the pixel region centers;

[0032] Step 1 )-7: Update C using the lowest gradient algorithm with a step size of 3*3;

[0033] Step 1 )-8: Initialize the pixel region set R = <R1, R2,..., R num >;

[0034] Step 1 )-9: Initialize the label array L [] = <-1, -1,..., -1> and the distance array D [] = <∞, ∞,..., ∞> with a length of size 2

[0035] Step 1 )-10: j = 1, execute Step 1 )-12;

[0036] Step 1 )-11: j ++;

[0037] Step 1 )-12: k = 1, execute Step 1 )-14;

[0038] Step 1 )-13: k ++;

[0039] Step 1 )-14: Take out c j in the 2S*2S region of a pixel point p k ;

[0040] Step 1 )-15: Calculate the distance between c j and p k , denoted as distance, calculated as follows:

[0041] distance = dis(c j , p k )

[0042] Step 1 )-16: Determine if distance < D [k], if yes, execute Step 1 )-17, otherwise execute Step 1 )-13;

[0043] Step 1 )-17: From R​L[k] p is deleted from R k ;

[0044] Step 1)-18: D[k] = distance, L[k] = j;

[0045] Step 1)-19: put p k into R j ;

[0046] Step 1)-20: judge whether the pixel points in the 2S*2S region are traversed, if yes, execute Step 1)-21, otherwise execute Step 1)-13;

[0047] Step 1)-21: judge whether j>num, if yes, execute Step 1)-22, otherwise execute Step 1)-11;

[0048] Step 1)-22: T = T U R;

[0049] Step 1)-23: judge whether i>n, if yes, execute Step 1)-24, otherwise execute Step 1)-5;

[0050] Step 1)-24: output the pixel region set T;

[0051] Step 1)-25: end state.

[0052] Further, the specific steps of the above Step 2) are as follows:

[0053] Step 2)-1: initial state;

[0054] Step 2)-2: input the training set TR = {tr1, tr2, …, tr n}, the pixel region set T, and the number of important features m;

[0055] Step 2)-3: cluster the pixel regions in P;

[0056] Step 2)-4: select the pixel region closest to the centroid in each cluster as a feature to generate the feature set F = {f1, f2, …, f l};

[0057] Step 2)-5: initialize the score array S with length m;

[0058] Step 2)-6: i = 1, execute Step 2)-8;

[0059] Step 2)-7: i++;

[0060] Step 2)-8: j = 1, execute Step 2)-10;

[0061] Step 2)-9: j++;

[0062] Step 2)-10: Select layer in model M, calculate f j The important score of t j , denoted as score, is calculated as follows:

[0063]

[0064] Step 2)-11: Determine whether score > S[i] is true, if yes, execute step 2)-13, otherwise execute step 2)-12;

[0065] Step 2)-12: S[i] = score;

[0066] Step 2)-13: Determine whether j > n is true, if yes, execute step 2)-14, otherwise execute step 2)-9;

[0067] Step 2)-14: Determine whether i > l is true, if yes, execute step 2)-15, otherwise execute step 2)-7;

[0068] Step 2)-15: Select the top m features in the score set S to reconstitute the feature set F = {f1, f2, …, f m};

[0069] Step 2)-16: Output the feature set F;

[0070] Step 2)-17: End state.

[0071] Further, the specific steps of the above step 3) are as follows:

[0072] Step 3)-1: Start state;

[0073] Step 3)-2: Input test set T = {t1, t2, …, t n}, important feature set F = {f1, f2, …, f m} and trained model M;

[0074] Step 3)-3: Initialize the feature label array L = <0, 0, …, 0> and the score array S = <0, 0, …, 0> with length n;

[0075] Step 3)-4: i = 1, j = 1;

[0076] Step 3)-5: Select layer in model M, calculate f j The important score of t i , denoted as score, is calculated as follows:

[0077]

[0078] Step 3)-8: judge whether the important feature set is traversed, if yes, execute step 3)-9, otherwise execute step 3)-10;

[0079] Step 3)-7: S[i] = score;

[0080] Step 3)-8: judge whether the important feature set is traversed, if yes, execute step 3)-9, otherwise execute step 3)-10;

[0081] Step 3)-9: judge whether the test data set is traversed, if yes, execute step 3)-12, otherwise execute step 3)-11;

[0082] Step 3)-10: j++, execute step 3)-5;

[0083] Step 3)-11: i++, execute step 3)-5;

[0084] Step 3)-12: output the feature label array L;

[0085] Step 3)-13: end state.

[0086] Further, the specific steps of the above step 4) are as follows:

[0087] Step 4)-1: start state;

[0088] Step 4)-2: input the test set T = {t1, t2, …, t n} and the feature label array L;

[0089] Step 4)-3: initialize the test data partition cluster centroid set C = {c1, c2, …, c l};

[0090] Step 4)-4: k = 1, i = 1, num = 0;

[0091] Step 4)-5: judge whether L[i] == k is true, if yes, execute step 4)-6, otherwise execute step 4)-7;

[0092] Step 4)-6: c k = c k + t i , num++;

[0093] Step 4)-7: judge whether the test data is traversed, if yes, execute step 4)-9, otherwise execute step 4)-8;

[0094] Step 4) -8: i++, execute step 4) -5;

[0095] Step 4) -9: c k = c k / num;

[0096] Step 4) -10: judge whether the centroid set of the cluster is completed, if yes, execute step 4) -12, otherwise execute step 4) -11;

[0097] Step 4) -11: i=1, k++, execute step 4) -5;

[0098] Step 4) -12: output the cluster centroid set C={c1, c2, …, c l};

[0099] Step 4) -13: end state;

[0100] Further, the specific steps of the above step 5) are as follows:

[0101] Step 5) -1: start state;

[0102] Step 5) -2: input the cluster centroid set C={c1, c2, …, c l};

[0103] Step 5) -3: create a new l(l-1) / 2 row 3 column matrix D[][];

[0104] Step 5) -4: i=1, j=1, k=1;

[0105] Step 5) -5: D[k][1]=i;

[0106] Step 5) -6: D[k][2]=j;

[0107] Step 5) -7: calculate the Euclidean distance of c i , c j two data, saved in D[k][3], the calculation formula is as follows:

[0108] D[k][3]=distance(c i , c j )

[0109] Step 5) -8: j++, k++;

[0110] Step 5) -9: judge whether j>l is true, if yes, execute step 5) -10, otherwise execute step 5) -5;

[0111] Step 5) -10: i++, j=i+1;

[0112] Step 5)-11: Determine if i>l-1 is true. If yes, execute step 5)-12; otherwise, execute step 5)-5.

[0113] Step 5)-12: Sort the rows in matrix D in ascending order based on the data in their 3rd column;

[0114] Step 5)-13: Output matrix D;

[0115] Step 5)-14: End state.

[0116] Furthermore, the specific steps of step 6) above are as follows:

[0117] Step 6)-1: Initial state;

[0118] Step 6)-2: Input matrix D;

[0119] Step 6)-3: Initialize the multidimensional space graph G =<N,E> ;

[0120] Step 6)-4: i = 0, j = 0;

[0121] Step 6)-5: i++;

[0122] Step 6)-6: j++;

[0123] Steps 6)-7: G = G∪ <c D[j][2] ,c D[j][2] ,D[j][3]>;

[0124] Steps 6)-8: Determine if there is a loop in graph G. If so, proceed to steps 6)-9; otherwise, proceed to steps 6)-10.

[0125] Step 6)-9: G = G- <c D[j][2] ,c D[j][2] ,D[j][3]>,Execute steps 6)-6;

[0126] Step 6)-10: Determine if i>l-1 is true. If yes, execute step 6)-11; otherwise, execute step 6)-5.

[0127] Step 6)-11: Calculate the weight set Γ = {γ1, γ2, ..., γ} corresponding to the edges in graph G. l-1 The sum of the elements in} γ Sum The calculation formula is as follows:

[0128]

[0129] Steps 6)-12: Based on the distance entropy of set Γ, calculate the diversity dis_entropy of the test set. The calculation formula is as follows:

[0130]

[0131] Step 6)-13: Output the diversity dis_entropy of the test set.

[0132] Step 6)-14: End state. BRIEF DESCRIPTION OF DRAWINGS

[0133] Figure 1 A flowchart of a test sufficiency measurement method based on input feature diversity in the embodiment of the present application.

[0134] Figure 2 A flowchart of training data image segmentation in the embodiment of the present application. Figure 1

[0135] A flowchart of CAV important feature selection in the embodiment of the present application. Figure 3 Figure 1 A flowchart of test data feature mining in the embodiment of the present application.

[0136] Figure 4 Figure 1 A flowchart of multi-dimensional space data clustering in the embodiment of the present application.

[0137] Figure 5 A flowchart of inter-cluster centroid distance measurement in the embodiment of the present application. Figure 1

[0138] Figure 6 A flowchart of distribution diversity entropy value calculation in the embodiment of the present application. Figure 1

[0139] Figure 7 Figure 1 DETAILED DESCRIPTION

[0140] In order to better understand the technical content of the present application, specific embodiments are described below with the aid of the accompanying drawings.

[0141] A test sufficiency measurement method based on input feature diversity, characterized by comprising the following steps.

[0142] S1 training data image segmentation, sequentially taking out the training data in the training set, using the lowest gradient algorithm to update the pixel region center, using the Euclidean distance to calculate the distance of the pixel point to the pixel center and saving it in the distance array, adding the pixel point to the corresponding pixel interval, and dividing the pixels according to the image pixel interval and generating a set;

[0143] S2 CAV important feature selection, clustering the pixel region set, selecting the pixel region closest to the centroid in each cluster to generate a feature set, and using the training set to calculate the important scores of all features, and selecting the m features with the highest scores to reconstitute the feature set;​​​​​​

[0144] S3 test data feature mining: For test data in the test set, calculate the importance score of all features in the feature set in turn, and mark the feature with the highest score as the important feature of the test data, thereby generating a feature label array for the test data;

[0145] S4 multidimensional spatial data clustering manually clusters the test set based on the feature label array, divides the centroids of the test data in the test set until the test data and the centroid set of the clusters are traversed sequentially, and calculates and generates a new set of centroids of the clusters.

[0146] The S5 inter-cluster centroid distance metric enumerates the centroids in the generated centroid set one by one according to the principle of no repetition and no omission, and calculates the Euclidean distance between each pair of centroids in turn; a distance matrix is ​​created, and the two endpoints of each distance segment and the distance size information are stored in the distance matrix row by row, and then the distance information stored in the matrix is ​​sorted in ascending order of distance size.

[0147] The S6 distribution diversity entropy calculation involves creating a multidimensional spatial graph. For a distance matrix already sorted in ascending order of distance, edges are added to the graph in ascending order of weight. If adding an edge would create a cycle in the graph, that edge is discarded. This process continues until all centroids in the centroid set are connected to the graph. The diversity of the test set is calculated based on the distance entropy of the centroid set obtained using this graph, thus effectively evaluating the sufficiency of the test set.

[0148] Figure 1 This is a flowchart illustrating a test sufficiency measurement method based on input feature diversity, as implemented in this invention.

[0149] Figure 2 The flowchart for training image segmentation is as follows:

[0150] Step 1: Initial state; Step 2: Input training set TR= <tr1,tr2,…,tr n >, image pixel interval s; Step 3: Create pixel region set T = {}; Step 4: i = 1, execute steps 1)-6; Step 5: i++; Step 6: Calculate tr i The number of divisible pixel regions, denoted as num, is calculated using the following formula: num = (tr i .size / s), C = <c1,c2,…,c num > represents the center of all pixel regions; Step 7: Update C using the minimum gradient algorithm with a step size of 3*3; Step 8: Initialize the pixel region set R = <R1,R2,…,R num Step 9: Initialize the length to size. 2the label array L[] = <-1,-1,...,-1> and the distance array D[] = <∞,∞,...,∞>; step 10: j = 1, execute step 12; step 11: j++, step 12: k = 1, execute step 14; step 13: k++, step 14: take out c j a pixel point p in the 2S*2S region k ; step 15: calculate the distance between c j and p k , denoted as distance, the calculation formula is as follows: distance = dis(c j , p k ); step 16: judge distance < D[k], if yes, execute step 17, otherwise execute step 13; step 17: delete p L[k] from R k ; step 18: D[k] = distance, L[k] = j; step 19: put p k into R j ; step 20: judge whether the pixel points in the 2S*2S region are traversed, if yes, execute step 21, otherwise execute step 13; step 21: judge j > num, if yes, execute step 22, otherwise execute step 11; step 22: T = T∪R; step 23: judge whether i > n is true, if yes, execute step 24, otherwise execute step 5; step 24: output the pixel region set T; step 25: end state.

[0151] Figure 3 a flow chart for selecting important features of CAV, the specific steps are as follows:

[0152] step 1: start state; step 2: input the training set TR = {tr1, tr2,..., tr n}, the pixel region set T, and the number of important features m; step 3: cluster the pixel regions in P; step 4: select the pixel region closest to the centroid in each cluster as a feature to generate the feature set F = {f1, f2,..., f l}; step 5: initialize the score array S with length m; step 6: i = 1, execute step 8; step 7: i++, step 8: j = 1, execute step 10; step 9: j++, step 10: select the layer in the model M, calculate the important score of f j to tr j , denoted as score, the calculation formula is as follows: Step 11: determine whether score > S[i] is true, if yes, execute step 13, otherwise execute step 12; Step 12: S[i] = score; Step 13: determine whether j > n is true, if yes, execute step 14, otherwise execute step 9; Step 14: determine whether i > 1 is true, if yes, execute step 15, otherwise execute step 7; Step 15: select the highest m features in the score set S to reconstitute the feature set F = {f1, f2, …, f m}; Step 16: output the feature set F; Step 17: end state.

[0153] Figure 4 For the flowchart of feature mining of test data, the specific steps are as follows:

[0154] Step 1: start state; Step 2: input the test set T = {t1, t2, …, t n}, the important feature set F = {f1, f2, …, f m} and the trained model M; Step 3: initialize the feature label array L = <0, 0, …, 0> and the score array S = <0, 0, …, 0> with length n; Step 4: i = 1, j = 1; Step 5: select the layer in the model M, calculate the important score of f j for t i , denoted as score, and the calculation formula is as follows: Step 6: determine whether score > S[i] is true, if yes, execute step 7, otherwise execute step 8; Step 7: S[i] = score; Step 8: determine whether the important feature set is completed, if yes, execute step 9, otherwise execute step 10; Step 9: determine whether the test data set is completed, if yes, execute step 12, otherwise execute step 11; Step 10: j++, execute step 5; Step 11: i++, execute step 5; Step 12: output the feature label array L; Step 13: end state.

[0155] Figure 5 For the flowchart of multi-dimensional space data clustering, the specific steps are as follows:

[0156] Step 1: start state; Step 2: input the test set T = {t1, t2, …, t n} and the feature label array L; Step 3: initialize the test data partition cluster centroid set C = {c1, c2, …, c l}; Step 4: k = 1, i = 1, num = 0; Step 5: determine whether L[i] == k is true, if yes, execute step 6, otherwise execute step 7; Step 6: c k = c k + t i, num++; step 7: judging whether the test data is traversed, if yes, executing step 9, otherwise executing step 8; step 8: i++, executing step 5; step 9: c k = c k / num; step 10: judging whether the centroid set of the cluster is traversed, if yes, executing step 12, otherwise executing step 11; step 11: i=1, k++, executing step 5; step 12: outputting the centroid set of the cluster C={c1, c2, …, c l}; step 13: ending state.

[0157] Figure 6 The flow chart for the inter-cluster centroid distance measurement is shown in Fig. 6, and the specific steps are as follows:

[0158] Step 1: starting state; step 2: inputting the centroid set of the cluster C={c1, c2, …, c l}; step 3: newly creating an l(l-1) / 2 row 3 column matrix D[][]; step 4: i=1, j=1, k=1; step 5: D[k][1]=i; step 6: D[k][2]=j; step 7: calculating the Euclidean distance between c i and c j , and saving it in D[k][3], and the calculation formula is as follows: D[k][3]=distance(c i , c j );

[0159] Step 8: j++, k++; step 9: judging whether j>l is true, if yes, executing step 10, otherwise executing step 5; step 10: i++, j=i+1; step 11: judging whether i>l-1 is true, if yes, executing step 12, otherwise executing step 5; step 12: sorting the rows in the matrix D according to the data in the 3rd column in ascending order; step 13: outputting the matrix D; step 14: ending state.

[0160] Figure 7 The flow chart for the distribution diversity entropy value calculation is shown in Fig. 7, and the specific steps are as follows:

[0161] Step 1: starting state; step 2: inputting the matrix D; step 3: initializing the multi-dimensional space graph G=<N, E>; step 4: i=0, j=0; step 5: i++; step 6: j++; step 7: G=G∪<c D[j][2] , c D[j][2] , D[j][3]>; step 8: judging whether there is a loop in the graph G, if yes, executing step 9, otherwise executing step 10; step 9: G=G-<c D[j][2] , c D[j][2]D[j][3] >, execute step 6;Step 10: determine whether i>l-1 is true, if yes, execute step 11, otherwise execute step 5;Step 11: calculate the sum of the elements in the weight set Γ corresponding to the edge in the graph G = {γ1, γ2, …, γ l-1}γ Sum , the calculation formula is as follows: Step 12: based on the distance entropy of the set Γ, the diversity dis_entropy of the test set is calculated, and the calculation formula is as follows: Step 13: output the diversity dis_entropy of the test set;Step 14: end state.

[0162] In summary, the present application aims at the evaluation problem of the sufficiency of deep learning software testing. Analyzing the feature diversity of the test set can help software testers improve the sufficiency and interpretability of software testing, and guarantee the quality of deep learning software.

Claims

1. A deep learning software test adequacy measure method based on input feature diversity, characterized in that, First, the training data in the training set TR is taken out in turn, the pixels are segmented according to the image pixel interval s, and the pixel region set T is generated; secondly, T is clustered, and the importance scores of all features are calculated using the training set TR, and the m features with the highest scores are selected to reconstitute the feature set F; thirdly, the data in the test set T is taken out in turn, and the features with the highest scores are calculated and labeled as important features to generate the feature label array L of the test data; then, the test set T is artificially clustered according to the feature label array L, and the centroid set C is calculated and generated; then, the distance between the centroids is calculated using the Euclidean distance, and the corresponding distance matrix D is generated; finally, the centroid set is converted into a graph G in a multi-dimensional space, and the diversity dis_entropy of the test set based on the distance entropy is calculated to effectively evaluate the sufficiency of the test set, and the method comprises the following steps: 1) Training data image segmentation, given the training set TR = {tr1, tr2, …, tr n}, generate pixel region center set and update with the lowest gradient algorithm, calculate the Euclidean distance distance(c j , p k ) of pixel center c j to pixel point p k , and save it in distance array D; the calculation formula of Euclidean distance is as follows: The label array L records the pixel interval corresponding to the pixel point, the segmentation of the pixels is completed, and the pixel region set T is generated. 2) CAV important feature selection, given training set TR = {tr1, tr2, …, tr n} and pixel region set T, cluster the pixel region set, select the pixel region closest to the cluster center in each cluster as the feature, generate feature set F = {f1, f2, …, f l}, calculate the importance score of all features using the training set, the calculation formula is as follows: where tr i represents the i-th data in the training set, f j represents the j-th feature, represents the gradient of layer, f layer represents the activation function of the hidden layer layer in the model, represents the activation feature vector (Concept Activation Vectors, CAV) of the feature f i ; and the m features with the highest selection scores are reorganized into a feature set; 3) Test data feature mining, input test set T = {t1, t2, …, t n}, initialize feature label array L = <0, 0, …, 0> and score array S = <0, 0, …, 0>, according to the important feature set F = {f1, f2, …, f l} obtained by the image cutting and CAV important feature selection steps of the training data and the trained model M, calculate the important scores of the test data in the test set on the features in turn; the calculation formula is as follows: where t i represents the ith data in the test set, f j represents the jth feature, represents the gradient of layer, f layer represents the activation function of the hidden layer layer in the model, represents the CAV of the feature f i ; the feature with the highest score is marked as the important feature of f j , and then the feature label array L of the test data is generated accordingly; 4) Multi-dimensional space data clustering, input test set T = {t1, t2, …, t n} and the obtained feature label array L, initialize the cluster center set of test data division C = {c1, c2, …, c l}, according to the feature label array, sequentially divide the test data belonging to the cluster center, update the cluster center set until all test data and the cluster center set are completely traversed, and the cluster center c k of the kth new cluster is obtained through the calculation method of c k = c k / num, wherein num represents the number of test data of the kth new cluster; 5) Cluster inter-centroid distance metric, which calculates the distance between each two centroids in the centroid set using the Euclidean distance distance(c i ,c j ) and creates a distance matrix D, which stores the two endpoints and the distance size information of each distance segment in the matrix D: each row stores a distance information, where the first 2 columns store the two endpoints of the distance, and the 3rd column stores the size of the distance; sort the rows in the matrix D in ascending order according to the 3rd column data, i.e. the size of the distance, and then output the matrix D; 6) Distribution diversity entropy calculation, given the stored distance information, and the distance matrix D has been sorted in ascending order of distance size, create a multi-dimensional space graph G = <N, E>; the edges and their two endpoints in the matrix D are added to the graph G in order from small to large, and after each edge is added, it is checked that there is no loop in the graph G, if the addition of a certain edge leads to the existence of a loop in the graph G, then delete the edge from the graph G, and continue to add the next edge. If the edges in the graph have connected all the centroids in the centroid set into the graph, stop adding edges; calculate the sum of the elements in the weight set Γ = {γ1, γ2, …, γ l-1} calculated based on the graph G, and the diversity dis_entropy of the test set is calculated based on the distance entropy of the centroid set calculated by the graph G, and the calculation formula is as follows: Sum ​ where γ i denotes the i-th element in the set Γ; finally, the adequacy of the test set is effectively evaluated by dis_entropy.

2. The deep learning software test adequacy measure method based on input feature diversity of claim 1, wherein, In step 1), the training data image segmentation is performed, the training data in the training set is taken out in turn, the pixel region center is updated using the lowest gradient algorithm, the distance between the pixel point and the pixel center is calculated using the Euclidean distance and is saved in the distance array, the pixel point is added to the corresponding pixel interval, and the pixels are segmented according to the image pixel interval and the set is generated.

3. The deep learning software test adequacy measure method based on input feature diversity of claim 1, wherein, In step 2), the CAV important feature selection is performed, the pixel region set is clustered, the feature set is generated by selecting the pixel region closest to the centroid in each cluster, and the importance scores of all features are calculated using the training set, and the m features with the highest scores are selected to reconstitute the feature set.

4. The deep learning software test adequacy measure method based on input feature diversity of claim 1, wherein, In step 3), the test data feature mining is performed, for the test data in the test set, the importance scores of all features in the feature set are calculated in turn, and the feature with the highest score is marked as the important feature of the test data, and the feature label array of the test data is generated.

5. The deep learning software test adequacy measure method based on input feature diversity of claim 1, wherein, In step 4), the multi-dimensional space data clustering is performed, the test set is artificially clustered according to the feature label array, and the test data in the test set is divided into the centroid until the test data in the test set and the centroid set of the cluster are traversed in turn, and the centroid set of the new cluster is calculated and generated.

6. The deep learning software test adequacy measure method based on input feature diversity of claim 1, wherein, In step 5), the centroid distance between clusters is measured, the centroids in the generated centroid set are enumerated one by one according to the principle of not repeating and not missing, and the Euclidean distance between each two centroids is calculated in turn; A distance matrix is created, and the two endpoints and distance size information of each distance are stored in the distance matrix in rows, and then the distance information stored in the matrix is sorted in ascending order according to the distance size.

7. The deep learning software test adequacy measure method based on input feature diversity of claim 1, wherein, In step 6), the distribution diversity entropy value calculation is performed, a multi-dimensional space graph is created, the edges in the distance matrix sorted in ascending order according to the distance size are added to the graph in turn according to the weight value; if the addition of an edge will cause a loop in the graph, the edge is discarded; until the edges in the graph connect all the centroids in the centroid set; the diversity of the test set is calculated based on the distance entropy of the centroid set calculated by using the graph, thereby effectively evaluating the sufficiency of the test set.

Images