A differential testing method and system for a deep learning framework in an industrial internet application scenario
By screening high-quality mutation operators and using specific result distance metrics, the problems of low testing efficiency and low defect detection accuracy in deep learning framework testing are solved, achieving efficient and accurate defect detection for deep learning frameworks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF SOFTWARE - CHINESE ACAD OF SCI
- Filing Date
- 2022-03-09
- Publication Date
- 2026-04-14
AI Technical Summary
In existing deep learning framework testing methods, the test case generation stage contains low-quality or redundant mutation operators, resulting in poor testing efficiency. Furthermore, the output comparison stage relies on model accuracy and cannot be applied to mutation models, leading to low defect detection accuracy.
By selecting a high-quality set of mutation operators, deep learning models are mutated. The output differences are measured using specific result distance metrics for classification and regression models to identify inconsistencies and improve testing efficiency and defect detection accuracy.
It achieves efficient and accurate defect detection using a deep learning framework, which can quickly identify potential defects and improve testing efficiency and defect detection rate.
Smart Images
Figure CN116795665B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a differential testing method and system for deep learning frameworks in industrial internet application scenarios, belonging to the field of computer software testing technology. Background Technology
[0002] With the rapid expansion of industrial internet application scenarios, various application scenarios at the device layer, edge layer, platform layer, and application layer of industrial internet platforms are driving the transformation of traditional production models into intelligent production models characterized by real-time dynamic perception, scientific decision-making, and iterative optimization. The massive amounts of data and high-efficiency computing power used in these models rely on deep learning frameworks optimized for artificial intelligence. By using deep learning frameworks, a data-sharing network that fully connects equipment, products, systems, and services can be built, effectively improving the intelligent characteristics of industrial internet production, such as self-learning, self-decision-making, and self-adaptation.
[0003] Deep learning frameworks have significantly lowered the technical barriers to building industrial internet platforms. In recent years, the most popular deep learning frameworks, such as Tensorflow, Caffe, Theano, MXNet, Torch, and PyTorch, have been widely applied in related industrial internet fields such as healthcare, finance, retail, security, transportation, and energy. However, deep learning systems in industrial internet platforms may have software defects and exhibit a lack of robustness (Kurakin A, Goodfellow IJ, Bengio S. Adversarial examples in the physical world[C] / / Adversarial examples in the physical world.2016.), and may cause real-world accidents, thus threatening the safety of production and daily life. Compared to traditional software systems, deep learning systems in industrial internet platforms typically contain more complex components, such as hardware infrastructure, software infrastructure, deep learning frameworks, models, source code for training, and training and testing corpora (Guo Q, Chen S, Xie X, et al. An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms[C] / / 2019 34th IEEE / ACM International Conference on Automated Software Engineering(ASE).2019:810–822.). The typical structure of a deep learning system mainly consists of an application layer, a framework layer, and an infrastructure layer, each of which can potentially introduce errors (Islam MJ, NguyenG, Pan R, et al. A comprehensive study on deep learning bug characteristics[C] / / Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.2019:510–520.).As the foundational function library for deep learning systems in industrial internet platforms, deep learning frameworks directly influence the behavior of these systems. Potential software defects can affect millions of applications built upon them (Pang Tao. Research on the Development Status and Trends of Open Source Deep Learning Frameworks [J]. Internet World, 2018, 4: 46–54.). Therefore, ensuring the correctness of deep learning frameworks is crucial. Currently, most research on quality assurance for deep learning systems focuses on testing methods at the application layer (Zhang JM, Harman M, Ma L, et al. Machine learning testing: Survey, landscapes and horizons [J]. IEEE Transactions on Software Engineering, IEEE, 2020.), such as testing deep learning models by generating adversarial examples or measuring the test adequacy of deep learning models. However, few studies have focused on the verification and testing at the framework level within deep learning systems.
[0004] Unlike traditional software testing, test cases for deep learning frameworks are combinations of deep learning models and corresponding input data. Due to the complexity of the model structure and the need for manual annotation of input data, it is difficult to obtain a large number of such test cases. Simply selecting open-source deep learning models and datasets to test deep learning frameworks is insufficient to cover all framework behaviors, resulting in some code locations remaining untested. Furthermore, due to the poor interpretability of deep learning models, it is difficult to obtain the expected output of a deep learning framework for a given input instance, thus lacking test assertions for identifying potential defects. Deep learning framework testing faces these challenges. How to better test deep learning frameworks to ensure their correctness, thereby providing a reliable software foundation for deep learning systems in industrial internet platforms, is the problem this invention aims to solve.
[0005] Currently, a few academic studies have focused on the table cloning problem and proposed different methods for testing deep learning frameworks. Existing deep learning framework testing methods mainly focus on using differential testing techniques to detect inconsistencies in the inference results of the same instance on different deep learning frameworks. For example, the existing work CRADLE (Pham HV, Lutellier T, Qi W, et al. CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries [C] / / 2019 IEEE / ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019: 1027-1038.) proposes to apply differential testing techniques to deep learning framework testing and proposes two metrics for measuring the degree of inconsistency as test assertions, so as to more accurately detect the problem of inconsistent results of the same instance on different frameworks. Audee (Guo Q, Xie X, Li Y, et al.Audee: Automated Testing for Deep Learning Frameworks[C] / / 202035th IEEE / ACM International Conference on Automated Software Engineering(ASE).IEEE,2020:486-498.) proposed expanding the test cases for deep learning frameworks by varying the dimensions of model layer parameters, model weights, and input data, thereby enabling more comprehensive testing of deep learning frameworks.LEMON (Wang Z, Yan M, Chen J, et al. Deep learning library testing via effective model generation[C] / / Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.2020:788-799.) also uses mutation to expand test cases. Unlike Audee, LEMON proposes a heuristic mutation strategy to amplify the inconsistency in inference results of different frameworks for the same given test case, making it easier to distinguish inconsistencies caused by software defects. While the above work uses mutation to test test cases to more fully test deep learning frameworks, some areas still require improvement. This invention patent mainly addresses the following problems existing in the above-mentioned testing methods: 1) In the test case generation stage, existing methods usually expand the test case set based on mutation. However, for the mutation of test cases in deep learning frameworks, existing mutation methods have some low-quality or redundant mutation operators, resulting in the generation of low-quality test cases and thus poor testing efficiency; 2) In the output result comparison stage, the calculation method based on average distance relies on the high prediction accuracy of the model, while model mutation will reduce the prediction accuracy of the model. Therefore, this method cannot be applied to test scenarios using mutated models, resulting in low accuracy of defect detection in mutated scenarios. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of current deep learning framework testing techniques for differential testing, which suffer from poor testing efficiency and low defect detection accuracy. This invention proposes a new deep learning framework testing method that can perform more efficient testing of deep learning frameworks and improve the correctness and detection rate of defect detection.
[0007] The key to the deep learning framework differential testing method based on test case mutation is that the deep learning frameworks implement the same functionalities, and for a given input instance, different deep learning frameworks should have very similar output results. By measuring whether the difference in the output results of two frameworks for the same input instance exceeds a certain threshold, test cases that trigger inconsistent results can be identified, thereby detecting potential defects in the deep learning frameworks and maximizing the correctness of the deep learning framework functions.
[0008] The technical solution of the present invention is as follows:
[0009] A differential testing method for deep learning frameworks in industrial internet application scenarios mainly includes the following three steps:
[0010] Test case acquisition: Acquire test cases, which include deep learning models and their corresponding input data. The deep learning models include original models and mutated models.
[0011] Output extraction: Using a deep learning framework, the inference process of the test cases is loaded and executed to obtain the execution results of each test case under multiple different deep learning frameworks;
[0012] Inconsistency issue determination: Based on the execution results of the deep learning framework, calculate the degree of difference in the execution results of the same test case under different deep learning frameworks, and determine whether the test case triggers an inconsistency error based on whether the degree of difference is greater than a threshold.
[0013] The steps of the present invention are described in detail below.
[0014] 1. Test Case Acquisition
[0015] Test cases for deep learning frameworks consist of a deep learning model and corresponding input data. The deep learning model plays a crucial role in determining the scope of code execution logic that the test can cover. A deep learning model is a deep neural network with at least one hidden layer, including an input layer, several hidden layers, and an output layer, with each layer consisting of multiple neurons.
[0016] This invention primarily targets mutation of deep learning models. Firstly, based on the structural characteristics of deep learning models and existing model mutation schemes, a set of model mutation operators capable of mutation in multiple dimensions is derived. These mutation dimensions can be categorized as: 1) model structure mutation, 2) layer parameter mutation, 3) model weight mutation, and 4) neuron mutation. Table 1 lists the model mutation operators used.
[0017] Table 1: Model mutation operators used
[0018]
[0019] To improve model mutation efficiency and reduce the generation of invalid test cases, mutation operators need to be screened, and some low-quality mutation operators need to be removed. To evaluate the quality of the above mutation operators, experimental evaluation is needed to understand their impact on test results. This invention uses the following method to evaluate the quality of mutation operators: 1) An original model and the mutation models generated from this original model are called a race; 2) On the same pair of deep learning frameworks and the same race of models, for an input sample, there is always a mutation model that causes the input to trigger inconsistency on the pair of deep learning frameworks and has the greatest degree of inconsistency on the race; 3) By finding a set of mutation models that have the greatest degree of inconsistency through multiple input samples, the number of wins of each mutation operator is defined as the sum of the number of times the corresponding mutation operator was used to mutate this set of mutation models; 4) After mutating the model multiple times through experiments, the number of wins of each mutation operator is counted. The more wins, the more effective the mutation operator is.
[0020] Test cases that trigger inconsistency issues are selected through small-scale experiments. The number of wins for each mutation operator is counted, and the mutation operators are ranked according to the statistical results. The 10 mutation operators with the highest number of wins (or other numbers) are selected as the set of mutation operators used for the final test. A set of existing deep learning models is selected, and the original models are randomly mutated using the selected set of mutation operators to generate a set of mutated models as needed.
[0021] 2. Output Result Extraction
[0022] This invention performs inference on multiple different deep learning frameworks using all acquired deep learning models (including original and mutated models) and their corresponding input datasets to obtain the output result for each input instance. By configuring the features of multiple backends of different deep learning frameworks, a given set of deep learning models and their corresponding input data can be inferred on different deep learning frameworks, and the inference results can be used for difference result determination.
[0023] 3. Inconsistency issue determination
[0024] This invention inputs the output results extracted from different deep learning frameworks into a result comparison module to determine inconsistencies. The output result determination methods are categorized into three different types based on the type of deep learning model:
[0025] 1) For deep learning models that are primitive models and whose output belongs to classification models, the metric D_CLASS is calculated using the distance from the classification result. C，Y，Y As a measure of the degree of difference between the results, the threshold T C Set to 8. If D_CLASSC，Y，Y >T C If the results are inconsistent, it is determined that the results are inconsistent. The result distance calculation method based on classification is specifically designed for classification models. It calculates the difference in ranking of the output results of the two frames relative to the sample labels.
[0026] Specifically, a classification model with N classes outputs a vector of size N, containing data related to class C. i The corresponding confidence level p i Where 0 < i ≤ N, confidence level p i The model represents class C. i The confidence value for predicting the correct label for the input instance. Given the output vector Y of the classification model, where the label corresponding to the input sample is C, calculate the classification score S of the backend (the backend refers to the low-level deep learning framework, such as TensorFlow, Theano, and CNTK). C，Y :
[0027]
[0028] rank C，Y is the rank of label C in the output vector Y, and k is a constant value (empirically chosen based on the specific model) used to control the score range of the classification result, indicating that a score is awarded only when the rank of label C in the output vector is less than k. Given an output vector Y′ of the same model but with different backends, calculate its classification score S. C，Y′ Then, the distance calculation index D_CLASS based on the classification results is calculated. C，Y，Y′ The calculation formula is as follows:
[0029] D_CLASS C，Y，Y′ =|S C，Y -S C，Y′ |
[0030] 2) For deep learning models that are regression models, the result distance metric D_MAD, based on the average distance, is used. O，Y，Y′ As a measure of the degree of difference in outcomes, the threshold T M Set to 0.4. If D_MAD O，Y，Y′ >T M If the results are inconsistent, then the results are determined to be inconsistent.
[0031] Specifically, for the same model and two different backends of the input samples, given two prediction vectors Y and Y′ of size N, the result distance calculation method based on average distance first calculates the average absolute distance δ between the two output vectors and the ground-truth vector O. Y，O and δ Y′，O δ Y，O The calculation formula is as follows:
[0032]
[0033] Then calculate the distance D_MAD between the two results. O，Y，Y′ The calculation formula is as follows:
[0034]
[0035] 3) For deep learning models that are mutation models and whose outputs belong to classification models, this invention proposes a distance calculation index based on multi-backend classification results. As a measure of the degree of difference in outcomes, the threshold T vote Set to 8. If If the results are inconsistent, then the results are determined to be inconsistent.
[0036] Specifically, given an input sample I, a model M, and C... vote The result is obtained by combining the inference results of I using model M across multiple frames with the original sample labels through voting. Y and Y′ are the prediction result vectors obtained from any two frames. and Let represent the prediction scores of the two frameworks, respectively. The formula for calculating the distance metric for the optimized multi-backend classification result is as follows:
[0037]
[0038] Based on the same inventive concept, this invention also provides a differential testing system for a deep learning framework in an industrial internet application scenario using the above-described method, comprising:
[0039] The test case acquisition module is used to acquire test cases, which include deep learning models and their corresponding input data. The deep learning models include original models and mutated models.
[0040] The output result extraction module is used to load and execute the inference process of test cases using a deep learning framework, and obtain the execution results of each test case under multiple different deep learning frameworks.
[0041] The inconsistency problem determination module is used to calculate the degree of difference in the execution results of the same test case under different deep learning frameworks based on the execution results of the deep learning framework, and to determine whether the test case triggers an inconsistency error based on whether the degree of difference is greater than a threshold.
[0042] The advantages of this invention compared to existing technologies are as follows: Using the method of this invention, a set of high-quality mutation operators is selected based on their contribution to the test results. This allows for the efficient generation of a large number of deep learning models for differential testing of deep learning frameworks. Furthermore, by cleverly utilizing classification result information to assist in the determination of inconsistencies, this method can accurately detect inconsistencies in deep learning frameworks caused by their potential defects. Compared to existing table clone detection methods, this method can identify potential defects in deep learning frameworks more quickly and accurately. Attached Figure Description
[0043] Figure 1 This is a schematic diagram of the testing method framework of the deep learning framework based on test case variation of the present invention.
[0044] Figure 2 This is a schematic diagram of the mutation operator set selection process for the deep learning model of this invention.
[0045] Figure 3 This is a schematic diagram of the D_CLASS_VOTE index calculation process of the present invention. Detailed Implementation
[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0047] The differential testing method for deep learning frameworks in industrial internet application scenarios proposed in this invention is mainly aimed at testing the framework level of deep learning systems in industrial internet application scenarios. Based on the mutation of test cases, differential testing technology is used to execute test cases on deep learning frameworks, and the correctness of deep learning frameworks is verified by comparing whether the difference in the execution results of the same test cases on different deep learning frameworks is greater than a threshold.
[0048] Given two or more deep learning frameworks, this invention first acquires deep learning models and their corresponding input data in existing industrial internet application scenarios. Then, using a high-level deep learning framework, the model is loaded. By configuring the low-level frameworks used by the high-level framework, test cases consisting of the model and input data are executed on different low-level frameworks to obtain inference results. Finally, the difference distance metric is used to calculate the difference distance between the execution results of the same test case on different low-level deep learning frameworks. By comparing the difference distance with a threshold, it is determined whether the test case triggered an inconsistency error. This method includes three parts: test case acquisition, output result extraction, and inconsistency problem determination.
[0049] Test case acquisition refers to obtaining the test cases needed for differential testing of deep learning frameworks. Test cases consist of deep learning models and their corresponding input data. Test cases are mainly acquired through two methods: one is to collect commonly used deep learning models and datasets from the Internet; the other is to expand the number of deep learning models by mutating the deep learning models collected through the first method.
[0050] Output extraction utilizes a deep learning framework to load and execute the inference process of test cases, ultimately obtaining the execution results of each test case under multiple different deep learning frameworks. Deep learning frameworks can generally be divided into high-level frameworks and low-level frameworks. Low-level frameworks are responsible for implementing deep learning algorithms according to definitions, while high-level frameworks unify and abstract the programming interfaces provided by low-level frameworks, providing high-level function libraries for rapid development of deep learning programs and calling functions in low-level frameworks. Users can configure and use different low-level frameworks.
[0051] Inconsistency problem determination is based on the execution results of deep learning frameworks. It calculates the degree of difference in the execution results of the same test case under different deep learning frameworks, and finally determines whether the test case has triggered an inconsistency error based on whether the degree of difference is greater than a threshold.
[0052] The mutation model of the deep learning model is obtained through the following steps: First, the mutation operators of the deep learning models commonly used in existing industrial internet application scenarios are regarded as an initial mutation operator set; the existing deep learning models are mutated to a certain extent using this mutation operator set, and then these mutated models are tested. The score of each mutation operator is calculated based on the test results, that is, the cumulative number of mutation operators used by the model that results in the greatest inconsistency in each family of models is calculated; the initial mutation operator set is reduced based on the score, and the mutation operators with lower scores are removed to obtain a new mutation operator set; the new mutation operator set is used to mutate the deep learning models to obtain a large number of mutated models that can be used for testing.
[0053] The output extraction process involves first loading the acquired deep learning model using a high-level framework of the deep learning framework, and then utilizing the high-level framework's ability to configure and use different low-level frameworks to execute test cases on different low-level frameworks.
[0054] An embodiment of the present invention provides a differential testing method for deep learning frameworks in industrial internet application scenarios, the overall process of which is as follows: Figure 1As shown. The test objects selected in this example are deep learning frameworks, including the high-level framework Keras and the low-level frameworks TensorFlow, Theano, CNTK, and MXNet. Test cases consist of deep learning models and their corresponding inputs. This example mainly includes three parts: test case acquisition, output result extraction, and output result comparison.
[0055] 1. Test Case Acquisition
[0056] In the deep learning framework differential testing method of this invention, test case acquisition mainly focuses on mutating deep learning models to obtain more deep learning models. To efficiently mutate deep learning models, this invention filters existing model mutation operators. The process for filtering mutation operators is as follows: Figure 2 As shown, we first obtain the commonly used model mutation operators, for example, the 12 mutation operators in Table 2.
[0057] Table 2: Model mutation operators
[0058] Serial Number Model mutation operator Number of wins 1 Delete layer 50 2 Switching layer 6 3 Add a layer 30 4 Batch add layers 146 5 Weight swap 36 6 Neuron effect blocking 141 7 Gaussian blur 92 8 Copy layer 0 9 neuronal exchange 3 10 Neuron activation reversal 5 11 Activation function replacement 7 12 Activation function removal 5
[0059] Then, these mutation operators are used to mutate existing deep learning models, generating a batch of mutated models. These mutated models are then combined with some existing input data to form test cases, executed under the aforementioned deep learning framework, and the number of wins for each mutation operator is calculated. First, the mutated models are grouped according to their respective "races." For each pair of deep learning low-level frameworks and for each input data point, the execution results of these models differ across different deep learning frameworks. Once the degree of difference exceeds a threshold, models within each race are selected based on the degree of difference, choosing only the model that results in the greatest degree of difference within that race. Second, based on these selected models, the number of mutation operators used to generate these models is calculated, and this count is taken as the number of wins for each mutation operator. For example, model M is the original model, with mutation operators a, b, c, a pair of deep learning frameworks, and 2 input samples. Through mutation, mutated models Ma, Mb, Mc, Mac, and Mcbb are generated (the lowercase letters in the names represent the mutation operators used to obtain the mutated model). Suppose that Mac and Mcbb are input samples 1 and 2 respectively, which cause the pair of deep learning frameworks to trigger inconsistencies in this race, and the degree of inconsistency is the largest among models of the same race. Then the number of wins of mutation operator a is 1, the number of wins of mutation operator b is 2, and the number of wins of mutation operator c is 2.
[0060] Subsequently, after calculating the score of each mutation operator based on the test results, the mutation operators with lower scores were removed from the original mutation operator set. In this example, the four mutation operators with the lowest scores (replication layer, neuron swap, neuron activation inversion, and activation function removal) were removed, resulting in the filtered model mutation operator set, which contains a total of eight mutation operators.
[0061] After obtaining the selected mutation operators, the existing model is mutated using the set of mutation operators to generate a batch of mutated models for use in subsequent output result extraction and inconsistency problem determination.
[0062] 2. Output Result Extraction
[0063] To extract the output results, the deep learning model file is first loaded using the high-level deep learning framework Keras, and the input data corresponding to the deep learning model is prepared. Keras is then configured to use the low-level frameworks TensorFlow, Theano, CNTK, and MXNet to perform the inference process on the model. Finally, the execution results of the test cases (deep learning model and corresponding input samples) under four different deep learning frameworks are obtained.
[0064] 3. Inconsistency issue determination
[0065] In the inconsistency problem determination section, the input consists of a series of execution results obtained in the "output result extraction" stage and the original labels of the input data. Different distance calculation formulas are selected based on the model's attributes for calculation. Once the result distance exceeds a threshold, an inconsistency error is determined to have been triggered.
[0066] Distance calculation methods will be discussed in three main cases:
[0067] (1) The regression model uses the result based on the mean distance to calculate the index D_MAD, with a threshold set to 0.4. For example, Dave-2
[29] is a model that outputs the steering angle of a car (measured in radians) given a dashboard camera image as input. For a given input image I, the recorded steering angle (ground-truth) is O = 0.0. Using the same image as input, Dave-2 outputs Y = 0.4 and Y′ = -0.1 using two different deep learning low-level frameworks. Therefore, there is δ Y，O =|0.4-0.0|=0.4 and δ Y，O =|-0.1-0.0|=0.1, and then we can calculate Since 0.6 < 0.4, this test case is determined to have triggered an inconsistency issue.
[0068] (2) In the classification model, when using the original model, the result distance metric D_CLASS based on classification is adopted, with a threshold set to 8. For example, when k=5, the label of a certain input instance ranks first in the inference results under TensorFlow, i.e., rank. C，Y =1,S C，Y =2 5-1 =16. The label for this instance ranks 6th in the inference results under CNTK, i.e., rank. C，Y =6, S C，Y =0, then D_CLASS C，Y，Y =16. Because D_CLASS C，Y，Y If the result is greater than 8, it is determined that the instance has triggered a result inconsistency issue.
[0069] (3) In the classification model, when using the mutation model, an optimized metric D_CLASS_VOTE based on the distance calculation of multi-backend classification results is adopted, with a threshold set to 8. If the number of low-level frames is 4, the calculation process of the D_CLASS_VOTE metric is as follows: Figure 3 As shown. First, the inference result vectors of the test cases under the four low-level frames are obtained. The top-1 labels are calculated based on the inference results, and these four top-1 labels, together with the original labels, form five label categories. Then, a voting process is performed to select the label category C with the most occurrences among the five labels. vote Then use C vote Calculate the rank of the input data's label in each of the four results, and then calculate S according to the formula. C，Y and Finally, according to The results are compared with the threshold to determine whether the test case triggered an inconsistency issue.
[0070] Based on the same inventive concept, another embodiment of the present invention provides a differential testing system for a deep learning framework in an industrial internet application scenario using the above-described method, comprising:
[0071] The test case acquisition module is used to acquire test cases, which include deep learning models and their corresponding input data. The deep learning models include original models and mutated models.
[0072] The output result extraction module is used to load and execute the inference process of test cases using a deep learning framework, and obtain the execution results of each test case under multiple different deep learning frameworks.
[0073] The inconsistency problem determination module is used to calculate the degree of difference in the execution results of the same test case under different deep learning frameworks based on the execution results of the deep learning framework, and to determine whether the test case triggers an inconsistency error based on whether the degree of difference is greater than a threshold.
[0074] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.
[0075] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.
[0076] The various threshold and other constant values, as well as the specific types and quantities of deep learning frameworks mentioned in the above embodiments, are merely illustrative examples and are not fixed values of this invention. When implementing this invention, values can be selected based on experience or as needed, according to the actual situation.
[0077] Although specific embodiments and accompanying drawings of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments and accompanying drawings.
Claims
1. A differential testing method for deep learning frameworks in industrial internet application scenarios, characterized in that, Includes the following steps: Obtain test cases, which include deep learning models and their corresponding input data, and the deep learning models include original models and mutated models; Using a deep learning framework, the inference process of test cases is loaded and executed to obtain the execution results of each test case under multiple different deep learning frameworks; Based on the execution results of the deep learning framework, calculate the degree of difference in the execution results of the same test case under different deep learning frameworks, and determine whether the test case triggers an inconsistency error based on whether the degree of difference is greater than a threshold. The mutation model was obtained using the following steps: First, the mutation operators of commonly used deep learning models in existing industrial internet application scenarios are regarded as an initial mutation operator set; The existing deep learning model is mutated using the initial set of mutation operators, and then the mutated model is tested. The score of each mutation operator is calculated based on the test results, that is, the cumulative number of mutation operators used by the model that results in the greatest inconsistency in the model set for each race is calculated. The initial set of mutation operators is reduced based on the scores, and mutation operators with lower scores are removed to obtain a new set of mutation operators; The deep learning model is mutated using a new set of mutation operators to obtain a large number of mutated models that can be used for testing. The cumulative number of mutation operators used by the model that results in the greatest inconsistency in the model set for each race includes: 1) A race is defined as an original model and the variant models generated from it. 2) On the same pair of deep learning frameworks and the same set of models of the same race, for an input sample, there is always a variant model that causes the input sample to trigger inconsistency on the pair of deep learning frameworks and has the greatest degree of inconsistency on the race. 3) Find a set of mutation models that maximizes the degree of inconsistency through multiple input samples. The number of wins of each mutation operator is defined as the sum of the number of times the corresponding mutation operator was used to obtain this set of mutation models; 4) The number of wins of each mutation operator after multiple mutations of the model is statistically analyzed through experiments. The more wins, the more effective the mutation operator is.
2. The method according to claim 1, characterized in that, The deep learning framework is divided into a high-level framework and a low-level framework. The low-level framework is responsible for implementing deep learning algorithms according to the definition. The high-level framework unifies and abstracts the programming interface provided by the low-level framework, provides a high-level function library for rapid development of deep learning programs, and calls functions in the low-level framework to implement them.
3. The method according to claim 2, characterized in that, The process of loading and executing test cases using a deep learning framework to obtain the execution results of each test case under multiple different deep learning frameworks includes: first, loading the obtained deep learning model using the high-level framework of the deep learning framework, and then using the feature of the high-level framework that can be configured to use different low-level frameworks to execute the test cases on different low-level frameworks.
4. The method according to claim 1, characterized in that, The calculation of the difference in execution results of the same test case under different deep learning frameworks, and the determination of whether the test case triggers an inconsistency error based on whether the difference exceeds a threshold, includes: firstly, calculating the difference distance of the output results of the same test case on different deep learning frameworks based on the execution results of the deep learning frameworks; secondly, selecting different distance calculation formulas based on the attributes of the deep learning model; and thirdly, using a result distance calculation metric based on the average distance for the regression model. D_MAD The threshold is set to 8; in the classification model, the original model uses the distance from the classification results to calculate the metric. D_CLASS The mutation model uses an optimized distance calculation metric based on multi-backend classification results. D_CLASS_VOTE The threshold is set to 0.4; if the result is greater than the value of the calculated index, it is judged that an inconsistency error has been triggered.
5. A differential testing system for a deep learning framework in an industrial internet application scenario employing the method described in any one of claims 1 to 4, characterized in that, include: The test case acquisition module is used to acquire test cases, which include deep learning models and their corresponding input data. The deep learning models include original models and mutated models. The output result extraction module is used to load and execute the inference process of test cases using a deep learning framework, and obtain the execution results of each test case under multiple different deep learning frameworks. The inconsistency problem determination module is used to calculate the degree of difference in the execution results of the same test case under different deep learning frameworks based on the execution results of the deep learning framework, and to determine whether the test case triggers an inconsistency error based on whether the degree of difference is greater than a threshold.
6. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 4.