Deep learning variation test method based on weak variation analysis

Abstract

The invention discloses a deep learning variation test method based on a weak variation technology, and belongs to the technical field of software engineering. According to the method, all layers directly forming a deep learning model serve as a series of components forming the deep learning model, and a weighted graph capable of reflecting different features is constructed according to an output value of a last convolutional layer in the model to measure the expression difference of a test case on an original model and a variant. Therefore, the killing condition of the variant can be judged in advance. The method serves as a deep learning variation test technology, the similarity degree of the variants and the original model on the test set is measured by calculating the weighted graph with the image feature expression ability, and therefore the killing condition of the variants can be predicted in advance, the defect of DL weak variation test in the prior art is overcome, and the method has the advantages of being high in practicability and the like. The time for executing the variant is shortened, so that the cost of a DL variation test is greatly reduced, and the efficiency of the variation test is improved.

Description

Technical Field

[0001] The present invention belongs to the field of software engineering technology, and specifically relates to deep learning algorithms and software testing, and in particular to a deep learning variation testing method based on weak variation technology. Background Art

[0002] Deep learning (DL) has been widely used in many fields such as image classification, speech recognition, and natural language processing, and has achieved great success. These achievements are expected to make DL components an important part of complex intelligent software systems. However, much like traditional software, DL software may also fail to run and crash due to defects or errors in the software, causing serious consequences, even resulting in property damage and casualties. Therefore, in safety-critical fields, DL software also needs to be fully tested to ensure its reliability and safety.

[0003] Based on the data-driven nature of DL models, software testing's evaluation of their behavior is largely dependent on the software's test data. A high-quality test set can guarantee the quality of the software to a great extent; conversely, an insufficient test set will make it difficult to detect software defects and errors. Therefore, evaluating the quality of the test set is crucial to testing DL software.

[0004] In the traditional software testing field, mutation testing has been proven to be an effective technique for evaluating the quality of test sets. It can help testers find effective tests, evaluate the quality of test sets, and accurately locate defective data in test sets. Mutation testing designs mutation operators for specific systems and uses these mutation operators to artificially inject errors into the software under test to generate mutants. These mutants can fully simulate the errors that developers make when programming. Then, by calculating the corresponding mutation scores, the quality of the test set can be reflected. That is, test sets with higher mutation scores are more likely to find faults in the original program, while test sets with lower mutation scores need further improvement to achieve better error detection capabilities. Unlike traditional software systems, the behavior of DL systems depends not only on the source code, but also on the training data, neural network structure, and other various hyperparameters. At present, there are many methods that apply mutation testing to DL systems. These methods combine the characteristics of DL systems and design multiple mutation operators to generate mutant models, which can help measure the quality of test sets.

[0005] Deep learning mutation testing: In traditional software testing, mutation testing has been proven to be an effective method to measure the quality of test sets. It can design targeted mutation operators for different software systems under test to artificially inject errors into the software. By generating a large number of mutants to simulate the mistakes that developers often make during the software development process, testers can measure the quality of the test set based on the performance of the test set on the mutants.

[0006] However, compared with traditional software systems, the behavior of DL software depends not only on the source code, but also on the settings and adjustments of training data, neural network structure, hyperparameters, etc. Therefore, DL mutation operators have different properties. Ma et al. have proposed a DL mutation testing framework DeepMutation and designed a set of source-level mutation operators and a set of model-level mutation operators. The source-level mutation operator introduces possible errors into the training data and training program before model training, and then uses them to retrain and generate variants; while the model-level mutation operator generates variants by directly modifying the weights, neuron parameters, and model structure in the trained model. The proposed source-level mutation operators include: data duplication (DR), label error (LE), data loss (DM), data disorder (DS), noise perturbation (NP), layer addition (LAs), and activation function removal (AFRs), which are used to modify training data and programs to generate variants. The specific description of the source-level mutation operator is as follows:

[0007] 1) Data Repetition (DR): Randomly repeat a portion of the training data

[0008] 2) Label Error (LE): Randomly change the labels of some data

[0009] 3) Data loss (DM): Randomly remove some training data

[0010] 4) Data scrambling (DS): shuffle the original order of training data

[0011] 5) Noise Perturbation (NP): Adding noise to the training data

[0012] 6) Layer Addition (LAs): Randomly add a layer that does not affect the normal operation of the model

[0013] 7) Activation Function Removal (AFRs): Randomly remove the activation function of a layer

[0014] The existing DL mutation testing methods are all based on the "strong mutation" idea in traditional mutation testing. The deep learning model consists of several layers, including convolutional layers, pooling layers, activation layers, and fully connected layers. These layers play different functions and roles in the back propagation process of the model. For a given test input, when it is pushed into the DNN model at the input layer, its value will eventually get a predicted classification result after the weight matrix calculation and activation operations of each layer. This method can indeed evaluate the quality of the test set efficiently and accurately, but when using this method for mutation testing, its cost is often very high. In this process, the test case needs to be calculated and processed by all layers of the model, and this process takes a lot of time.

[0015] Weak mutation technology: Weak mutation technology was first proposed and applied in traditional mutation testing to reduce the execution time of mutants and alleviate the problem of high mutation testing costs. This technology considers that program p is composed of a series of components C = {c1,...,cn}. When the state of any ci of mutant m is different from the state of the corresponding ci in the original program p, it can be considered that the mutant is killed. Therefore, in the weak mutation testing process, there is no need to wait for the program and mutant to be fully executed before comparing the results. Instead, you only need to monitor the state of a certain execution point between them to determine whether it has been killed.

[0016] However, the components in traditional weak mutation testing technology are variables, functions, classes, etc., which cannot be directly applied to DL mutation testing. In order to realize weak mutation analysis in DL mutation testing, it is necessary to re-examine the structure of the DL model. Consider the software being tested as a whole composed of a series of components, where these components can be variables, functions, classes, and other elements in the program. During the weak mutation testing process, the tester monitors the status and variables of these components in real time. If the status of the corresponding components in the original program and the variant is different, it can be judged that the variant is killed by the test case. However, in traditional mutation testing, these components are very trivial and small, and it is very difficult and energy-consuming for testers to monitor the status of these components. Therefore, this concept cannot be directly used for DL ​​weak mutation testing.

[0017] In order to solve the above problems, a deep learning mutation testing method based on weak mutation analysis is proposed in the present invention. By considering all layers in the model as a series of components that directly constitute the model, the output value of the last convolutional layer is taken as the processing object, and the weighted graphs of the test cases on the original model and the variant are calculated respectively, so as to judge the running results of the variant in advance, which can effectively alleviate the high cost problem in DL mutation testing. Summary of the invention

[0018] Purpose of the invention: The present invention aims to provide a deep learning mutation testing method based on weak mutation analysis, which can determine the killing status of the mutant in advance, reduce the system running time, and reduce cost overhead.

[0019] Technical solution: A deep learning mutation testing method based on weak mutation analysis. This method regards all layers that directly constitute the deep learning model as a series of components, and constructs a weighted graph that can reflect different features based on the output value of the last convolutional layer in the model to measure the performance difference of the test case on the original model and the variant, so as to judge the killing status of the variant in advance. The method includes the following steps:

[0020] S1, use the selected data set and neural network model for training to obtain the corresponding original deep learning model;

[0021] S2. By using the source-level mutation operator in the deep learning mutation operator, the training data and the training program are modified, and then they are used for retraining to obtain a set of deep learning mutation models, called variants;

[0022] S3. Based on weak mutation analysis, by deconstructing and analyzing the structure of the original deep learning model and the mutation model, we focus on the convolutional layers and their positions that play a key role in image feature extraction in the two models, and calculate the corresponding weighted graphs based on the output values ​​of the test cases on the convolutional layers of the two models to measure the similarity of the behaviors of the test cases on the original deep learning model and the mutation model, and finally determine whether the mutant is killed.

[0023] The method also includes obtaining the feature map output values ​​of the test case on the deep learning model and the mutation model by focusing on the convolutional layers that affect the feature extraction of the test image, and calculating a weighted map that can show the contribution to the final model prediction result, thereby reducing the execution cost during the mutation test.

[0024] Furthermore, step S2 includes modifying the training data by copying the training data, changing the data label, randomly discarding the data and / or disrupting the data order;

[0025] Modifications to the training procedure include deleting layers, adding layers, and / or removing activation functions.

[0026] Furthermore, the weak variation analysis described in step S3 also includes the following process:

[0027] S31, analyzing the model structures of the existing original model and the variant, and reconstructing two new models according to their structures to replace the old original model and the variant model respectively;

[0028] Specifically, all layer outputs of the existing model are temporarily stored in a list, and then this list is combined with the layer input of the existing model to obtain a new model;

[0029] The new model can accept inputs that are exactly the same as the old model, and can also obtain the output values ​​of each layer in the old model, that is, obtain the first set of data;

[0030] S32, specifying the index of the last convolutional layer in the newly constructed model, and respectively extracting the output values ​​of the test case on the layer of the original model and the variant, to obtain a second set of data, which is a set of feature maps containing multiple channels;

[0031] S33, for each set of feature graphs in the two sets of data, respectively calculate the average value of the values ​​in each feature graph to assign a weight to the feature graph, and then, according to the values ​​in the feature graph and the assigned weight values, weightedly superimpose them and calculate them in the same channel, to obtain weighted graphs of the test case on the original model and the variant, respectively;

[0032] S34. After obtaining the weighted graphs of the test case on the original model and the variant, the SSIM method is used to calculate their similarity, and the value range is [-1, 1]. When the value is set to be greater than 0, it is considered that the variant is alive, otherwise it is considered that the variant is killed by the test case;

[0033] This makes it possible to predict the killing of variants in advance without waiting for the final run results.

[0034] Furthermore, the method also includes providing analysis and guidance for deep learning mutation testing based on the results of the deep learning original model and the variants in the weak mutation method, including selecting and generating test cases based on the results.

[0035] Beneficial effect: Compared with the prior art, the deep learning mutation testing method of the present invention further obtains the feature map output value of the test case on it by focusing on the convolution layer that plays a more important role in feature extraction of the test image, and calculates a weighted graph based on this that can fully demonstrate the contribution to the final model prediction result, thereby alleviating the problem of high execution cost in the mutation testing process caused by executing variants of the complete structure in the prior art; on the other hand, since this method uses the technical idea of ​​weak mutation, the entire model is regarded as a series of components, and the output value of each component in the model can be monitored in real time; at the same time, the weighted graph calculated by these output values ​​can be used to judge the killing status of the variant in advance based on these data without waiting for the final prediction result, which helps to reduce the time of executing the variant, thereby reducing the cost of DL mutation testing and improving testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of the implementation framework of the method of the present invention. DETAILED DESCRIPTION

[0037] In order to explain the technical solution disclosed by the present invention in detail, a further introduction is given below in conjunction with the accompanying drawings of the specification.

[0038] The present invention provides a deep learning mutation test method based on weak mutation analysis, firstly, the selected data set and neural network model are used for training to obtain the corresponding original model, and the training data and training program are modified by using the source-level mutation operator in the deep learning mutation operator, and then they are used for retraining to obtain a group of deep learning mutation models, called variants; then, a deep learning variant killing discrimination method based on weak mutation technology is adopted, by analyzing the model structure, calculating the weighted graph according to the output value of the internal layer of the model, and measuring the similarity of the weighted graph of the test case on the original model and the variant to judge whether the variant is killed; finally, according to the results of the deep learning original model and the variant in the weak mutation method, further analysis and guidance are provided for deep learning mutation testing. As a deep learning mutation test technology, the present invention measures the similarity between the variant and the original model on the test set by calculating the weighted graph with the ability to express image features, so that the killing of the variant can be predicted in advance, which makes up for the deficiency of DL weak mutation test in the prior art, helps to reduce the time of executing the variant, thereby greatly reducing the cost of DL mutation test and improving the efficiency of mutation test.

[0039] Specifically, this embodiment discusses the implementation process of the present invention from the following four aspects.

[0040] 1. Train the original deep learning model.

[0041] The MNIST dataset was selected as the experimental object for evaluation. MNIST is a dataset that is open to the public and widely used in research. It consists of handwritten digit images from 0 to 9. There are 70,000 images, including 60,000 training images and 10,000 test images, all of which are grayscale images. Two models A and B with different structural parameters were trained using the MNIST dataset. Table 1 shows the details of the model structure and the accuracy of each model on the dataset.

[0042] Table 1 Deep learning model structure

[0043]

[0044] 2. Use mutation operators to generate mutants.

[0045] For source-level mutation operators that target data, the variants are generated by randomly modifying 1% of the training data; for source-level mutation operators that target the training program, it is necessary to rebuild the qualified model according to the original model design structure. For example, the layer addition operation randomly adds a layer that does not affect the normal operation of the model, that is, the input shape and output shape of the added layer must match the input and output structure of the inserted position; and the activation function removal randomly removes the activation function of a layer. Then the variants are obtained through training. It should be noted that all generated variants need to maintain a prediction accuracy of more than 90% on the test set, otherwise they need to be regenerated. In addition, in order to eliminate randomness, we generated 20 variants for each mutation operator, so there will be 140 variants for both groups A and B.

[0046] 3. Determine the killing status of the variants based on the analysis method based on weak mutation technology.

[0047] First, analyze the existing model structure, create a list containing the output of all layers of the model based on its structure, and then combine this list with the input of the old model to build a new model to replace the old model. The new model can accept inputs that are exactly the same as the old model, and can also get the output values ​​of each layer in the model. Specify the last convolutional layer in the model, and intercept the output values ​​of the test image in the original model and the variant on this layer. The data is a set of feature maps containing multiple channels.

[0048] The shape of the feature map obtained is usually (size, height, width, channels), where size represents the size of the test set, channels represents the number of channels in the feature map, and height and width represent the height and width of the channel respectively. For each set of feature maps, the weights are calculated channel by channel according to formula 1, where value i,j Represents a single pixel value in a channel:

[0049]

[0050] Then calculate the corresponding weighted graph according to Formula 2. Specifically, channel i represents the i-th channel of the feature map, and weight i Then it represents the weight value corresponding to the channel. After weighted calculation of the feature map of each channel and the corresponding weight value, all weighted results are accumulated to generate a weighted map that integrates all channel information.

[0051]

[0052] After obtaining the weighted graphs on the original model and the variant respectively, the indicator SSIM is used to calculate their similarity. Its value range is [-1,1]. When its value is set to be greater than 0, it can be considered that the variant is alive, otherwise it is considered that the variant is killed by the test case.

[0053] Table 2. Run time results of single variant and all variants in weak mutation test

[0054]

[0055] As shown in Table 2, the running time of a single variant and all variants in the weak mutation test is compared with the results in the control experiment. Taking model A as an example, the running time of a single variant in the method of the present invention is 1.3706 seconds on average, while the average running time of the variants in the control experiment is 1.5756 seconds, that is, judging the survival of a single variant only takes 86.99% of the time required by the original method. In addition, judging the survival of all variants in the method of the present invention requires 193.1264 seconds, while the control experiment requires 220.5788 seconds to run, so the method of the present invention saves about 12.45% of the running time.

[0056] 4. Evaluation test set.

[0057] The mutation scores of the test set on a set of generated deep learning mutation models are calculated to evaluate the quality of the test set and obtain the corresponding mutation analysis results, which further provide guidance and feedback for the testing of the deep learning system.

[0058] In summary, this method regards all the layers that directly constitute the deep learning model as a series of components, and constructs a weighted graph that can reflect different features based on the output value of the last convolutional layer in the model to measure the performance difference of the test case on the original model and the variant, so that the killing status of the variant can be judged in advance, and ultimately the running time is reduced and the cost overhead is introduced.

Claims

1. A deep learning variation testing method based on weak variation analysis, characterized in that: This method regards all layers that directly constitute the deep learning model as a series of components, and constructs a weighted graph that can reflect different features based on the output value of the last convolutional layer in the deep learning model to measure the performance difference of the test case on the original model and the variant, so as to determine the killing status of the variant in advance, including the following steps: S1, use the selected data set and neural network model for training to obtain the corresponding original deep learning model; S2. By using the source-level mutation operator in the deep learning mutation operator, the training data and the training program are modified, and then they are used for retraining to obtain a set of deep learning mutation models, called variants; S3. Based on weak mutation analysis, by deconstructing and analyzing the structure of the original deep learning model and the mutation model, we focus on the convolutional layers and their positions that play a key role in image feature extraction in the two models, and calculate the corresponding weighted graphs based on the output values ​​of the test cases on the convolutional layers of the two models to measure the similarity of the behaviors of the test cases on the original deep learning model and the mutation model, and finally determine whether the mutant is killed.

2. The deep learning variation testing method according to claim 1, characterized in that: The method also includes analyzing the convolutional layers that affect the feature extraction of the test image, obtaining the feature map output values ​​of the test case on the deep learning model and the mutation model, and calculating the feature map output values ​​to obtain a weighted graph that can contribute to the final model prediction result, thereby reducing the execution cost during the mutation test.

3. The deep learning variation testing method according to claim 1, characterized in that: Step S2 includes modifying the training data by copying the training data, changing the data label, randomly discarding the data and / or disrupting the data order; Modifications to the training procedure included deleting layers, adding layers, and removing activation functions.

4. The deep learning variation testing method according to claim 1, characterized in that: The weak variation analysis process described in step S3 is specifically as follows: S31, analyzing the model structures of the existing original model and the variant, and reconstructing two new models according to their structures to replace the old original model and the variant model respectively, specifically, temporarily storing all layer outputs of the existing model in a list, and then combining the list with the layer input of the existing model to obtain a new model; The new model can accept inputs that are exactly the same as the old model, and can also obtain the output values ​​of each layer in the old model, that is, obtain the first set of data; S32, specifying the index of the last convolutional layer in the newly constructed model, and respectively extracting the output values ​​of the test case on the layer of the original model and the variant, to obtain a second set of data, which is a set of feature maps containing multiple channels; S33, for each set of feature graphs in the two sets of data, respectively calculate the average value of the values ​​in each feature graph to assign a weight to the feature graph, and then, according to the values ​​in the feature graph and the assigned weight values, weightedly superimpose them and calculate them in the same channel, to obtain weighted graphs of the test case on the original model and the variant, respectively; S34. After obtaining the weighted graphs of the test case on the original model and the variant, the SSIM method is used to calculate their similarity, and the value range is [-1, 1]. When the value is set to be greater than 0, the variant is considered to be alive, otherwise it is considered that the variant is killed by the test case; This makes it possible to predict the killing of variants in advance without waiting for the final run results.

5. The deep learning variation testing method according to claim 1, characterized in that: The method also includes further providing analysis and guidance for deep learning mutation testing based on the results of the deep learning original model and the variant in the weak mutation method, including selecting and generating test cases based on the results.

Images