Deep neural network oriented natural language processing model test case reduction method

Abstract

The application discloses a natural language processing technology test case reduction method for a deep neural network, and comprises the following steps: step 1), a model to be tested is determined, and a natural language processing model based on a deep neural network is trained by using original data; step 2), a feature vector of an original test set is extracted by using the trained model, and the extracted feature vector of the text data is stored in an xlsx file; and step 3), an evaluation function is defined to sort text data test cases, and the greater the evaluation function value, the greater the possibility that the test case is misclassified, and the test cases with greater evaluation function values are sequentially arranged in front. The application defines a function for measuring the possibility that a test case is misclassified from the unique features of the used text data, and with the popularization of the natural language processing technology model based on the deep neural network, the test efficiency of the model can be effectively improved, and the test cost can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software testing optimization technology in software testing technology, and particularly relates to a natural language processing model test case reduction method for deep neural networks, and particularly relates to a function for natural language processing model test optimization for deep neural networks. BACKGROUND

[0002] Software testing is essential for software development and is an important means to ensure software quality and reliability. Software testing optimization technology is an important part of software testing. Different testing optimization technologies can help testers quickly find defects and reduce resource occupation, aiming to greatly improve testing efficiency and reduce testing cost.

[0003] There are three main methods of software testing optimization technology: test case sorting, test case selection and test case reduction. Test case selection achieves test optimization by selecting a small test set with the same test function as the original test set. Test case sorting arranges the test cases that are more likely to cause software errors in the front, helping testers to find defects as soon as possible. Test case reduction removes redundant test cases and concentrates test cases that are easy to cause algorithm errors in a small test set to improve testing efficiency and reduce testing cost.

[0004] Most of the existing research on deep neural network-oriented test optimization methods are for non-textual data. However, the particularity of natural language data requires the model to have stronger information extraction capability, resulting in a large difference between deep neural network-based natural language processing technology and other deep neural network technologies. Therefore, the existing test optimization methods cannot be directly used on this type of model, and a test optimization method specifically suitable for deep neural network-based natural language processing technology model needs to be found by combining the characteristics of text data and this type of model. In addition, due to the fuzziness and non-standardization of text data, text data has higher labeling cost and fewer labeled samples. Reducing this cost in the test optimization work in the NLP field is of practical significance. Finding a deep neural network test optimization method specifically suitable for NLP and reducing the labeling cost of text data in the test optimization work are the two main problems faced by the test optimization work at present.

[0005] Natural language is edited by people, so natural language has subjectivity, inaccuracy and non-standard characteristics, and contains more abundant information than picture data, which requires natural language processing models based on deep neural networks to have stronger information extraction capabilities. Natural language processing models based on deep neural networks can extract abundant information contained in the text and output a distributed vector representation, which is a dense feature vector, each element has no specific meaning, but the whole contains a lot of information extracted by the model. According to the above characteristics of the text data and the output of this kind of model, the application uses Shannon entropy and Gini impurity to screen the test cases that are more likely to cause the model to make mistakes, and realizes the test case reduction of the deep neural network natural language processing model.

[0006] Natural language is edited by people, so the data itself has subjectivity, ambiguity and non-standard, when the text contains too much redundant information, it will cause the model to make mistakes when extracting important information from the text because of excessive confusion of information. Gini impurity is a standard for measuring the degree of system confusion, and the application uses it to measure the confusion degree of the vector set extracted by the model, that is, the confusion degree of the information contained in the text. The higher the confusion degree, the more likely the test case is misclassified. In addition, if the information contained in the text is not sufficient, the model will not be able to clearly identify the important information of the text, resulting in model errors. Shannon entropy measures the amount of information of an information, the larger the amount of information, the stronger the certainty of the information, and the less likely the model will make mistakes. Based on this idea, we use Shannon entropy to measure the likelihood of misclassification of test cases, and the smaller the entropy, the less uncertain the text, and the less likely it will be misclassified. SUMMARY

[0007] An object of the present application is to improve the test efficiency of natural language processing models based on deep neural networks and reduce the test cost. The present application proposes a function for optimizing the test of natural language processing models based on deep neural networks to measure the likelihood of misclassification of test cases. The present application combines the feature vector output by the model and the characteristics of the text data, and uses Gini impurity and Shannon entropy to define a function that measures the likelihood of misclassification of test cases. The present application first sorts the test cases according to the two functions, and puts the test cases that are more likely to be misclassified by the model at the front.

[0008] Another object of the present application is to remove redundant test cases in the test set according to Shannon entropy, and to realize test case reduction for deep neural network natural language processing model. We take the entire test case set as a large set, and each test case as a single element of the set. We take the evaluation function of each test case being misclassified as the information carried by each element. The certainty of the misclassification of the first k test cases is measured by Shannon entropy. The smaller the entropy value, the stronger the certainty of the misclassification of the test case set. We automatically divide the intercepted test set by adjusting the threshold of Shannon entropy.

[0009] The technical solution of the present application is a deep neural network-oriented natural language processing test case reduction method, comprising the following steps:

[0010] Step 1), specify the model to be tested, and train the deep neural network-based natural language processing model using the original data;

[0011] Step 2), use the model trained in step 1) to extract the feature vector of the original test set, and store the extracted feature vector of the text data in an xlsx file;

[0012] Wherein, different layers of DNN represent different types of input features. Layers closer to the input layer represent more basic features, i.e. the test input itself and the basic features extracted from the test input. Layers closer to the output layer represent more high-order features, and higher-order features can more accurately capture the relationship between input and label. We take the last hidden layer feature here, i.e. the output of the last hidden layer in DNN, which is a high-order feature that can directly infer the prediction result of the input.

[0013] Step 3), define an evaluation function to sort the text data test cases. The larger the evaluation function value, the greater the possibility of misclassification of the test case. We sort the test cases with large evaluation function values in turn; the evaluation function is a Gini impurity-based evaluation function or a Shannon entropy-based evaluation function;

[0014] The Gini impurity-based evaluation function is:

[0015]

[0016] The Shannon entropy-based evaluation function is:

[0017]

[0018] Where p t,iFor each element in the feature vector, N is the number of elements contained in the feature vector, t is the row where each element is located, i is the column where each element is located. m is the parameter value to be set, ξ(t) is the function value of the evaluation function defined based on Gini impurity, H(P) is the function value of the evaluation function defined based on Shannon entropy.

[0019] Further, the method of the present application further comprises:

[0020] Step 4), all test cases are taken as a set, the evaluation function value of each test case is taken as a piece of information, Shannon entropy is used to measure the certainty of the first K test cases being misclassified, and the size of the required test set is automatically intercepted according to the set parameter m and the threshold value. The threshold value is the size of the growth of the misclassification evaluation value of the first K test cases, and the parameter m is set in the range, wherein the effective range of the parameter m is different for different models, for example, for the model Bert, the effective range of the parameter m under two kinds of sorting indexes is [70, 100] and [270, 340] respectively, and the threshold value range is [0, 2], so that the test set after optimization and simplification can be obtained;

[0021] Further, the method of the present application further comprises:

[0022] Step 5), the test set after the test case reduction obtained in step 4) is reorganized into a test set in text form, the model is tested again, and the effectiveness of the method is evaluated by means of the accuracy reduction rate of the test set after reduction.

[0023] The test case reduction method for the natural language processing model of the deep neural network provided by the present application has the advantages compared with the existing test case reduction technology for the deep neural network, that is, the existing test optimization work based on the deep neural network is concentrated in the field of computer vision, and there is a lack of test optimization technology for the deep neural network in the field of natural language processing, especially in the narrow category, there is no test optimization technology research for text data. The present application provides a test case reduction method specially for the model based on the deep neural network natural language processing technology. Moreover, the present application starts from the unique features of the text data used, and defines a function for measuring the size of the possibility of misclassification of test cases. With the popularization of the model based on the deep neural network natural language processing technology, the test efficiency of this kind of model can be effectively improved, and the test cost can be reduced, and the present application has practical significance. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Flow chart of test case reduction method for deep neural network natural language processing model DETAILED DESCRIPTION

[0025] The application will be further described below with reference to the drawings. The application will be described below with reference to the drawings. Figure 1 The technical solutions of the application will be described in detail. A test case reduction method for a natural language processing model of a deep neural network has the following implementation steps.

[0026] 1. Define the target model and the data set, and train the model using the data set. The application is a test case set reduction method, and the test set of the target model is the target to be tested and optimized. In order to extract the feature vector of the target data set, after the target model and the used data set are determined, the original data set is divided using existing technology to train the model. The data set is divided into a training set, a validation set and an original test set, and the training set and the validation set are used to train the target model. The trained target model to be tested is obtained.

[0027] 2. Use the trained model to obtain the feature vector representation of the original data set. The original test case set is input into the trained model, and the application edits the code after the last hidden layer of the model structure to extract the feature vector output by the last hidden layer of the model structure. The last hidden layer outputs, and the rich text information is stored in the feature vector. This is a dense feature vector, and each element has no specific meaning, but the whole contains a large amount of information in the text data. The more information contained in the dense feature vector, the closer the relationship between the text data and the feature vector.

[0028] Among them, different layers of DNN represent different types of input features. Layers closer to the input layer represent more basic features, i.e. the test input itself and the basic features extracted from the test input. Layers closer to the output layer represent more high-order features, and higher-order features can more accurately capture the relationship between the input and the label. We take the last hidden layer feature here, i.e. the output of the last hidden layer in the DNN, which is a high-order feature that can directly infer the prediction result of the input.

[0029] 3. Sort the test cases. Based on the extracted feature vector, we use an evaluation function based on Gini impurity or based on Shannon entropy to measure the possibility of misclassification of the test cases, and sort the test cases according to the possibility of misclassification. The evaluation functions for sorting the test cases are formula (1) and formula (2) respectively.

[0030]

[0031]

[0032] where p t,iFor each element in the feature vector, m is the parameter value we need to set, ξ(t) is the function value of the evaluation function defined based on Gini impurity, and H(P) is the function value of the evaluation function defined based on Shannon entropy.

[0033] 4, Shannon entropy is used to intercept the test set from the first to the Kth test case according to the probability of each test case being misclassified. According to the set parameters m and the threshold, the size of the required test set is automatically intercepted. The threshold is the size of the increase in the misclassification evaluation value of the first K test cases, and the parameter m is set in the range, wherein the effective range of the parameter m is different for different models, for example, for the model Bert, the effective range of the parameter m under two sorting indicators is [70, 100] and [270, 340] respectively. When the Shannon entropy is small enough, it means that the test cases contained in the set make the model more deterministic. The present application considers that the Shannon entropy threshold range is small enough when it is in the range [0, 2]. For the two sorting methods, the calculation formulas are as formula 3 and

[0034] Formula 4:

[0035]

[0036]

[0037] Wherein α is the entropy value, ξ k is the measurement value of the kth test case based on formula (1), H k (P) is the measurement value of the kth test case based on formula (2). According to the threshold range of the Shannon entropy being small enough, the first K test cases under two sorting strategies are intercepted as the reduced test set according to formula (3) and formula (4).

[0038] 5, the effectiveness of the test set obtained by optimization is input into the model verification method. The reduced test cases obtained in step 4 are input into the target model, and a smaller test set is used to detect the defects existing in the target model. The purpose of the present application is to remove redundant test cases through test case reduction, and to concentrate the test cases that are easy to make the model error in a smaller test set. The experimental effect of test case reduction is measured by using accuracy and accuracy drop rate. The definitions are as follows:

[0039]

[0040]

[0041] Wherein True Positive (TP) is predicted as positive, and actually positive; False Positive (FP) is predicted as positive, and actually negative; True Negative (TN) is predicted as negative, and actually negative; False Negative (FN) is predicted as negative, and actually positive. The original accuracy is the accuracy of the classification result obtained by the original test set, and the accuracy is the test accuracy obtained by the test reduction strategy on the small test set.

[0042] Embodiments:

[0043] The present application carries out experiments on the Bert model, and the experimental results are shown in Tables 1 and 2.

[0044] Table 1 Bert based on Gini impurity experimental results

[0045]

[0046]

[0047] Table 1 records the experimental results of the Bert model based on Gini impurity, the accuracy of the original test set is 77.34%, the parameter m takes no value between 70 and 100, and the interval is 10. When m takes the value of 70, the accuracy decreases to 33.24%, and the maximum decrease rate is 57.02%. When m is 100, the accuracy decreases to 61.67%, and the minimum decrease rate is 20.26%. The time used is also reduced from the original 5 seconds to 0 seconds. The size of the original test set is 11968 test cases, and our method reduces the data set to 311 in the case of the largest decrease rate. The experimental results prove that the present application is effective.

[0048] Table 2 Bert based on Shannon entropy experimental results

[0049]

[0050] Table 2 records the experimental results of the Bert model based on Shannon entropy, the accuracy of the original test set is 77.34%, the parameter m takes no value between 270 and 345, and the interval is 5. When the parameter m is 270, the accuracy is 77.41%, at this time the size of the test set is not reduced, and the accuracy value has a slight increase. When the parameter m takes the value of 345, the accuracy decreases to 42.60%, and the maximum decrease rate is 44.92%. The time used is also reduced from the original 5 seconds to 0 seconds. The size of the original test set is 11968 test cases, and the present application reduces the data set to 223 in the case of the largest decrease rate. The experimental results prove that the method of sorting test cases based on Shannon entropy for test case reduction is effective for the model Bert.

[0051] The foregoing describes in detail the natural language processing model test case reduction method for the deep neural network of the present application, but obviously the specific implementation form of the present application is not limited thereto. Various obvious changes made by those skilled in the art without departing from the spirit and scope of the claims of the present application are within the scope of protection of the present application.

Claims

1. A method for deep neural network oriented natural language processing test case reduction, characterized in that: The method comprises the following steps: Step 1, the target model and the original text data set are determined, and the original text data set is used to train the model; The model to be tested is determined, and the deep neural network-based natural language processing model is trained using the original text data set; the original text data set is divided into a training set, a verification set and an original test case set, the training set and the verification set are used to train the target model; and a trained target model to be tested is obtained; Step 2, using the trained model, the feature vector representation of the original test case set is obtained; The original test case set is input into the trained model, and the code is edited after the last hidden layer of the model structure, and the feature vector output by the last hidden layer of the model structure is extracted; the last hidden layer outputs, and the extracted text information is stored in the feature vector; Step 3, the test cases are sorted; The evaluation function based on Gini impurity or based on Shannon entropy is used to measure the possibility of error classification of the test cases, and the test cases are sorted according to the possibility of error classification; the evaluation function for sorting test cases is formula (1) and formula (2); where p t，i is the value of each element in the feature vector, m is the parameter value to be set, ξ(t) is the function value of the evaluation function defined based on Gini impurity, and H(P) is the function value of the evaluation function defined based on Shannon entropy. Step 4, intercepting the test set based on Shannon entropy; After sorting the test cases, the test cases that are easy to make the model error are arranged in the front; then, using the probability of each test case error as information, Shannon entropy is used to intercept the first K test cases, and the size of the required test set is automatically intercepted according to the set parameters m and threshold; Step 5, input the optimized test set into the model to verify the effectiveness of the method; The test cases are input into the target model, and a smaller test set is used to detect defects in the target model.

2. The natural language processing technique test case reduction method for deep neural networks according to claim 1, wherein: In step 2, different layers of DNN represent different types of input features; Layers closer to the input layer represent more basic features, i.e. the test input itself and the basic features extracted from the test input; layers closer to the output layer represent more high-order features; here, the last hidden layer feature, i.e. the output of the last hidden layer in DNN, directly infers the high-order features of the prediction result of the input.

3. The natural language processing technique test case reduction method for deep neural networks according to claim 1, wherein: In step 4, the Shannon entropy threshold range is considered to be small enough when it is in [0, 2]; for the two sorting methods, the calculation formulas are formula (3) and formula (4): wherein a is an entropy value, ξ k Hk is a measurement value of the kth test case based on formula (1), H k Hk is a measurement value of the kth test case based on formula (2); according to a threshold range in which the Shannon entropy is small enough, the first K test cases under two kinds of sorting strategies are intercepted according to formula (3) and formula (4) to be the reduced test set.

Images