A deep learning framework mutation testing method based on developer experience

Abstract

The application discloses a deep learning framework mutation testing method based on developer experience, which adopts a mutation operator and a constraint condition generation model based on the professional knowledge of developers to simulate the common operation of developers in the development process and detect more diversified defects in multiple stages (such as model training and inference) of the life cycle of a deep learning model. The method comprises two parts: first, model mutation, which is used to generate mutants; second, defect detection, which is used to perform defect detection. The model mutation comprises the following steps: first, the application adopts two deep Q networks to select a mutation operator and a seed model; then, the seed model is mutated under the constraint of criteria, and the Q network is updated according to the reward calculated by the current target evaluation; then, it filters out the mutants that violate any constraint, and adds the legal mutants to the generated model pool. The process is iterated until the mutation is completed. The defect detection comprises the following steps: first, the application detects the defects exposed in the mutants and records the defect reports. Then, based on the reward, the application selects part of the models and further executes them to detect defects in resource scheduling, execution crash, model performance and output precision. Through the method of the application, meaningless models in defect detection can be filtered, diversified defects can be detected in the execution stage of the model, the effectiveness of defect detection is improved, and the method has important application value and popularization prospect.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of computer science and software engineering, and is particularly concerned with testing and defect detection of deep learning frameworks. The method proposed by the present invention is an innovative deep learning framework testing technique that utilizes developers' expertise to design mutation operators and constraints, thereby detecting a more diverse range of defects in multiple stages such as model training and inference. The method has been evaluated on widely used deep learning frameworks such as PyTorch, JAX, and MindSpore, and has shown superior performance over existing baseline methods. The present invention not only improves the diversity and legality of generated models, but also successfully detects multiple high-value defects, and has been practically applied in the MindSpore community, proving its effectiveness in detecting important defects in real development scenarios. BACKGROUND

[0002] In the current digital era, deep learning technology has achieved remarkable success in multiple industries, particularly in safety-critical areas such as autonomous driving, medical diagnosis, and facial recognition. The successful development, maintenance, and deployment of these applications rely heavily on deep learning frameworks. However, defects in the framework can lead to serious safety incidents and economic losses. Therefore, it is crucial to rigorously test and detect defects in deep learning frameworks to ensure their quality and safety.

[0003] Although existing research has adopted deep learning models as test inputs and generated diverse models through mutation operators to improve testing effectiveness, these methods often fail to detect defects that are most concerning to developers. Developers are more inclined to fix defects that are common in real-world applications rather than those exposed by rare models generated by fuzzing tools. This indicates that there is a certain gap between existing testing methods and the actual needs of developers.

[0004] Specifically, there are three main limitations of existing testing methods: first, they detect defects that are often unrealistic, which are rarely seen in the real world and are usually automatically closed without any solution after confirmation; second, they ignore key defects in other stages of the model lifecycle, such as training, which limits their detection capabilities. These defects are more frequently exposed during the development process and often interfere with developers using the framework; third, they consume a large amount of resources and lack constraints to filter out illegal models that may cause crashes, abnormal outputs (not caused by framework defects), and trigger a large number of false positives. This wastes time analyzing low-value test inputs and requires additional manual work to identify these false positives, thereby reducing testing efficiency.

[0005] Therefore, there is a need for a new deep learning framework testing method that can better meet the actual needs of developers and detect common and critical defects in practical applications. This method should be able to simulate common operations during development and conduct comprehensive defect detection at multiple stages of the model lifecycle, thereby improving the accuracy and efficiency of testing and ensuring the quality and security of deep learning frameworks. SUMMARY

[0006] The present invention aims to propose a deep learning framework mutation testing method based on developer experience, which effectively explores valuable test input space and detects diverse defects by combining developer expertise. Specifically, we designed seven mutation operators and nine constraints based on the actual experience of industrial partners developing deep learning models to simulate common operations of developers and guide model mutation. In addition, we consulted industrial partners about their concerns during model construction, training, and inference to design corresponding test predictions and detect diverse defects at these stages. To further improve effectiveness, we use a double Q-learning algorithm to select mutation operators and seed models during mutation. This method has been evaluated on multiple widely used deep learning frameworks and has shown better performance than existing baseline methods, not only improving the diversity and legality of generated models, but also successfully detecting multiple high-value defects, demonstrating the effectiveness of defect detection in real-world development scenarios.

[0007] The technical solution of the present invention is as follows: a deep learning framework mutation testing method based on developer experience, including the following main steps: the whole process is iterated, and in each iteration, first, the mutation operator and seed model selection are performed, the present invention uses double Q-learning to select seed models and mutation operators using two deep Q networks, efficiently identifying valuable parts of the test input space, and continuously exploring these parts with the help of two Q networks. Then model mutation, in each mutation, the selected mutation operator is used to mutate the seed model, to determine whether the mutation fails or generates an illegal model that violates the constraints, and to perform corresponding reward calculation and model pool update to select the seed model for the next round. Then defect detection, analyze the legal models in the model pool and detect defects, and delete the models with defects. Finally, the models in the model pool are sorted according to the reward value, and the top k models are trained and inferred to further detect defects, returning the defect set as output.

[0008] Through the above steps, the present invention realizes a deep learning framework mutation testing method based on developer experience, solves the shortcomings of existing methods in deep learning model testing, greatly improves the accuracy and efficiency of defect detection, provides reliable technical support for quality assurance of deep learning frameworks, and has important application value and promotion prospects. BRIEF DESCRIPTION OF DRAWINGS

[0009] In the drawings: Figure 1 For the workflow of the developer experience-based deep learning framework mutation testing method in the application, the left box part is the model mutation process, and the right box part is the defect detection part. DETAILED DESCRIPTION

[0010] The application provides a developer experience-based deep learning framework mutation testing method, and the specific implementation steps are as follows:

[0011] 1. Parameter initialization

[0012] The application first initializes parameters, including: (1) a selected seed model d to be mutated; (2) two networks Q1 and Q2 selected as mutation operators; (3) a set DefectSet storing detected defects; (4) a mutation counter n; and (5) a set D storing generated models.

[0013] 2. Selection of seed model and mutation operator

[0014] After the initialization is completed, n rounds of mutation are performed. In each round of mutation, an probability p is generated by adopting an epsilon-greedy policy strategy, and the probability p is compared with a preset threshold value. If the probability p is smaller than the threshold value, a mutation operator is randomly selected, otherwise the output average values of Q1 and Q2 are calculated to select a mutation operator.

[0015] 3. Model mutation and network update

[0016] Next, the application performs mutation on the seed model d by using the selected mutation operator s to generate a mutated model d'. If the current mutation fails (for example, a crash model is generated) or a non-legal model violating three constraints is generated, the application sets a reward r as -1 to punish the behavior of selecting s based on d, and reselects a seed model with a maximum UCB value. If the generated model d' does not violate the constraints, the reward r between the current seed model d and d' is calculated, and the model pool D is updated.

[0017] In the calculation of the reward, the last layer is adopted as the basis. Then, the application randomly updates one of the two Q networks. We follow the original update process of double q learning, that is, a part of the current expected reward is combined with a part of the future expected reward, and the update speed is controlled. Finally, d' is used as the seed model for the next round.

[0018] 4. Defect detection

[0019] Afterwards, the generated legitimate models are further analyzed by the model pool D to detect defects and remove the defective models from it. Then the models in the model pool are ranked according to the reward values and the top k models are selected for further analysis. For these models, further training and inference are performed to detect defects therein. Finally, all the detected defects are collected into a set DefectSet as the output. Since the input data, scenario task types, etc. are different, these detection subjects have different value ranges on different models, so we execute all the models across different frameworks to collect preliminary results (e.g., memory usage size) about these research subjects. Then, we further set thresholds to determine whether the generated models expose relevant defects.

[0020] The present application uses two deep Q networks to select mutation operators by combining the double Q learning algorithm. It can efficiently identify valuable parts in the test input space and continuously explore these parts with the help of two Q networks. After generating models, the model with better performance is selected to perform training and inference, and various high-value defects are detected according to different test predictions.

[0021] Through the above implementation steps and specific techniques, the present application realizes a deep learning framework mutation testing method based on developer experience, improves the diversity and legitimacy of the generated models, and effectively detects defects more relevant to real scenarios.

Claims

1. A mutation testing method for deep learning frameworks based on developer experience, comprising the following steps: (1) Parameter initialization, including the selected seed model to be mutated, the two networks to be selected for mutation operators, the set of detected defects to be stored, the mutation counter, and the set of generated models to be stored. (2) The whole process is carried out iteratively. In each iteration, the mutation operator and seed model are selected first. Two deep Q networks are used to select the seed model and mutation operator. (3) Model mutation: In each round of mutation, the selected mutation operator is used to mutate the seed model, and it is determined whether the mutation fails or an illegal model that violates the constraints is generated. The reward calculation for the selected mutation operator and the model pool are updated to select the seed model for the next round. (4) Defect detection: Analyze the legal models in the model pool and detect defects, and delete the defective models; (5) Sort the models in the model pool according to the reward value, train and infer the top k models to further detect defects, and finally return the defect set as the output.

2. The deep learning framework testing method according to claim 1, characterized in that: During the iterative process, the present invention uses two deep Q-networks to select the seed model and mutation operator. After each round of mutation, the Q-network is updated based on the reward calculated according to the current target evaluation and the seed model for the next round is selected.

3. The deep learning framework testing method according to claim 1, characterized in that: Among the mutation operators, those that use model structure scripts as the main mutation subject mutate by adding new structures to the original model, deleting existing structures, and modifying parameters to improve model performance.

4. The deep learning framework testing method according to claim 1, characterized in that: Among the mutation operators, those that use script execution as the main mutation subject mutate by augmenting the model with data, changing the loss function and optimizer type, and modifying their parameters.

5. The deep learning framework testing method according to claim 1, characterized in that: The model mutation step described above uses a selected mutation operator to mutate the seed model, which needs to follow six mutation constraints, including structural-functional consistency constraints, data computation scale constraints, series structure addition / deletion constraints, parallel structure addition / deletion constraints, parameter value range constraints, and loss / optimizer modification constraints.

6. The deep learning framework testing method according to claim 1, characterized in that: The filtering constraints used to determine whether the generated model is illegal in the model mutation step include three types: execution time constraints, output accuracy constraints, and gradient value range constraints.

7. The deep learning framework testing method according to claim 1, characterized in that: The defect detection steps described above require the detection of four types of defects: performance defects, resource defects, accuracy defects, and crash defects.

Images