Developer experience-based deep learning framework variation test method

Abstract

The invention discloses a deep learning framework variation test method based on developer experience, and the method comprises the steps: generating a model through a variation operator and a constraint condition based on the professional knowledge of a developer, and simulating the common operation of the developer in a development process; and more diversified defects are detected in multiple stages (such as model training and reasoning) of the life cycle of the deep learning model. The method comprises two parts: 1, model variation, which is used for generating variants; and secondly, defect detection is carried out, wherein defect detection is executed. Wherein the model variation comprises the following steps of: firstly, selecting a variation operator and a seed model by adopting two deep Q networks; variation is carried out on the seed model under the constraint of a criterion, and a Q network is updated according to a reward calculated by current target evaluation; then, the variants violating any constraint are filtered out, and the legal variants are added into the generated model pool; the process is iteratively carried out until variation is completed. The defect detection comprises the following steps: firstly, detecting defects exposed in variants and recording a defect report; then, based on the rewards, the present invention selects and further executes partial models to detect defects in resource scheduling, execution crash, model performance, and output accuracy. By means of the method, meaningless models in defect detection can be filtered, diversified defects can be detected in the execution stage of the models, the effectiveness of defect detection is improved, and the method has important application value and popularization prospects.

Description

Technical Field

[0001] The present invention relates to the fields of computer science and software engineering, with a particular focus on testing and defect detection of deep learning frameworks. The method proposed in the present invention is an innovative deep learning framework testing technology that leverages the expertise of developers to design mutation operators and constraints to detect more diverse defects at multiple stages such as model training and reasoning. The method has been evaluated on widely used deep learning frameworks such as PyTorch, JAX, and MindSpore, and has shown superior performance over existing baseline methods. The present invention not only improves the diversity and legitimacy of generated models, but also successfully detects multiple high-value defects and has been put into practical use in the MindSpore community, proving its effectiveness in detecting important defects in actual development scenarios. Background Art

[0002] In the current digital age, deep learning technology has achieved remarkable success in multiple industries, especially in safety-critical fields such as autonomous driving, medical diagnosis, and face recognition. The successful development, maintenance, and deployment of these applications are highly dependent on deep learning frameworks. However, defects in the frameworks can lead to serious safety accidents and economic losses. Therefore, it is critical to rigorously test and detect defects in deep learning frameworks to ensure their quality and safety.

[0003] Although existing research has adopted deep learning models as test inputs and generated diverse models through mutation operators to improve test results, these methods often have difficulty detecting the defects that developers are most concerned about. Developers are more inclined to fix defects that are common in real applications rather than defects exposed by rare models generated by fuzz testing tools. This shows that there is a certain gap between existing testing methods and the actual needs of developers.

[0004] Specifically, existing testing methods have three major limitations: First, the defects they detect are often unrealistic, which are rare in the real world and are usually automatically closed after confirmation without any solution; Second, they ignore key defects in other stages of the model life cycle (such as training), which leads to the failure to discover defects related to unreasonable resource scheduling, substandard efficiency, abnormal performance, etc., thus limiting the detection ability. These defects are more frequently exposed during the development process and usually interfere with developers using the framework; Third, they consume a lot of resources and lack constraints to filter illegal models, which may cause crashes, abnormal outputs (not caused by framework defects), and trigger a large number of false positives. This wastes time analyzing low-value test inputs and requires additional manual work to identify these false positives, thereby reducing testing efficiency.

[0005] Therefore, a new deep learning framework testing method is needed that is closer to the actual needs of developers and detects common and critical defects in practical applications. This method should be able to simulate common operations in the development process and perform comprehensive defect detection at multiple stages of the model life cycle, thereby improving the accuracy and efficiency of testing and ensuring the quality and security of the deep learning framework. Summary of the invention

[0006] This paper aims to propose a mutation testing method for deep learning frameworks based on developer experience, which detects diverse defects by effectively exploring valuable test input spaces and combining developer expertise. Specifically, based on the actual experience of industrial partners in developing deep learning models, we designed seven mutation operators and nine constraints to simulate common operations of developers and guide model mutation. In addition, we consulted industrial partners about their concerns during model building, training, and reasoning to design corresponding test oracles and detect diverse defects in these stages. To further improve the effectiveness, we used the double Q learning algorithm to select mutation operators and seed models during the mutation process. This method has been evaluated on multiple widely used deep learning frameworks and has shown superior performance to existing baseline methods. It not only improves the diversity and legitimacy of the generated models, but also successfully detects multiple high-value defects, reflecting the effectiveness of detecting defects in actual development scenarios.

[0007] The technical solution of the present invention is as follows: A deep learning framework mutation testing method based on developer experience, including the following main steps: The whole process is iterative, and the mutation operator and seed model selection are first performed in each round of iteration. The present invention adopts dual Q learning, and uses two deep Q networks to select seed models and mutation operators, efficiently identifies valuable parts in the test input space, and continues to explore these parts with the help of two Q networks. Then there is model mutation. In each round of mutation, the seed model is mutated using the selected mutation operator to determine whether the mutation fails or an illegal model that violates the constraint is generated, and the corresponding reward calculation and model pool update are performed to select the seed model for the next round. Then defect detection is performed, the legal models in the model pool are analyzed and defects are detected, and defective models are deleted. Finally, the models in the model pool are sorted according to the reward value, and the top k models are trained and inferred to further detect defects, and the defect set is returned as output.

[0008] Through the above steps, the present invention realizes a deep learning framework mutation testing method based on developer experience, which solves the shortcomings of existing methods in deep learning model testing, greatly improves the accuracy and efficiency of defect detection, and provides reliable technical support for the quality assurance of deep learning frameworks. It has important application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the drawings of the specification Figure 1 This is the workflow of the deep learning framework mutation testing method based on developer experience in the present invention. The left frame part of the figure is the model mutation process, and the right frame part is the defect detection part. DETAILED DESCRIPTION

[0010] The present invention provides a deep learning framework mutation testing method based on developer experience, and its specific implementation steps are as follows:

[0011] 1. Parameter initialization

[0012] The present invention first initializes parameters, including: (1) the selected seed model d to be mutated; (2) the two networks Q1 and Q2 of the mutation operators; (3) the set DefectSet storing the detected defects; (4) the mutation counter n; and (5) the set D storing the generated models.

[0013] 2. Selection of seed model and mutation operator

[0014] After initialization, n rounds of mutation are performed. In each round of mutation, a probability p is generated using the ∈-greedy policy and compared with the preset threshold. If p is less than the threshold, a mutation operator is randomly selected, otherwise the average output of Q1 and Q2 is calculated to select the mutation operator.

[0015] 3. Model variation and network update

[0016] Next, we perform mutation on the seed model d with the selected mutation operator s to generate a mutated model d'. If the current mutation fails (e.g., generates a crash model) or generates an illegal model that violates the three constraints, the present invention sets the reward r to -1 to punish the behavior of selecting s based on d, and reselects a seed model with the maximum UCB value. If the generated model d' does not violate the constraints, the reward r between the current seed model d and d' is calculated, and the model pool D is updated.

[0017] When calculating the reward, we use the last layer as the basis. Then, the present invention randomly updates one of the two Q networks. We follow the original update process of double Q learning, that is, combining part of the current expected reward with part of the future expected reward and controlling the update speed. Finally, d' is used as the seed model for the next round.

[0018] 4. Defect Detection

[0019] After that, defects are further detected by analyzing the generated legitimate models stored in the model pool D, and defective models are deleted from them. The models in the model pool are then sorted according to the reward value, and the top k models are selected for further analysis. For these models, further training and reasoning are performed to detect the defects in them. Finally, the set of all detected defects DefectSet is taken as output. Due to different input data, scenario task types, etc., these detection subjects have different value ranges on different models, so we execute all models across different frameworks to collect preliminary results on these research subjects (for example, memory usage size). Then, we further set thresholds to determine whether the generated model exposes relevant defects.

[0020] The present invention combines the dual Q learning algorithm and uses two deep Q networks to select mutation operators. It can efficiently identify valuable parts in the test input space and continue to explore these parts with the help of two Q networks. After the model is generated, the model with better performance is selected to perform training and reasoning, and multiple high-value defects are detected according to different test predictions.

[0021] Through the above-mentioned implementation steps and specific technologies, the present invention realizes a deep learning framework mutation testing method based on developer experience, improves the diversity and legitimacy of the generated model, and effectively detects defects that are more relevant to real scenarios.

Claims

1. A deep learning framework mutation testing method based on developer experience, comprising the following steps: (1) Parameter initialization, including initializing the seed model, selecting two networks of mutation operators, the set of defects to be detected, the mutation counter, and the set of stored generation models. (2) The whole process is carried out iteratively. In each round of iteration, the mutation operator and seed model are first selected. The present invention adopts dual Q learning and uses two deep Q networks to select seed models and mutation operators, efficiently identify valuable parts in the test input space, and continuously explore these parts with the help of two Q networks. (3) Model mutation: In each round of mutation, the seed model is mutated using the selected mutation operator to determine whether the mutation fails or generates an illegal model that violates the constraints. The corresponding reward calculation and model pool update are performed to select the seed model for the next round. (4) Defect detection: Analyze the legitimate models in the model pool and detect defects, and delete the defective models. (5) The models in the model pool are sorted according to the reward value, and the top k models are trained and inferred to further detect defects. Finally, the defect set is returned as output.

2. The deep learning framework testing method according to claim 1, characterized in that: In the iterative process, the present invention uses two deep Q networks to select seed models and mutation operators. After each round of mutation, the Q network is updated and the seed model for the next round is selected according to the reward calculated by the current target evaluation.

3. The deep learning framework testing method according to claim 1, characterized in that: The mutation operators described above are types that use model structure scripts as mutation subjects. Their mutation methods are to add new structures to the original model, delete existing structures, and modify parameters to improve model performance.

4. The deep learning framework testing method according to claim 1, characterized in that: The mutation operators described above are types that use execution scripts as mutation subjects. Their mutation methods include data expansion of the model, changing the loss function and optimizer type, and modifying their parameters.

5. The deep learning framework testing method according to claim 1, characterized in that: In the model mutation step, the seed model is mutated using the selected mutation operator, and six mutation constraints need to be followed, including structural function consistency constraints, data calculation scale constraints, serial structure addition / deletion constraints, parallel structure addition / deletion constraints, parameter value range constraints and loss / optimizer modification constraints.

6. The deep learning framework testing method according to claim 1, characterized in that: The filtering constraints used to determine whether the generated model is illegal in the model variation step include three types: execution time constraints, output accuracy constraints, and gradient value range constraints.

7. The deep learning framework testing method according to claim 1, characterized in that: In the defect detection step, four types of defects need to be detected: performance defects, resource defects, precision defects and crash defects.

Images