A mutation operator simplification method and system for deep learning models

Abstract

The present invention relates to the field of software testing, and in particular to a mutation operator simplification method and system for a deep learning model. The present invention uses each mutation operator to mutate the original model, obtains the hidden state sequence of each test sample in the original model and each variant during the training process, and after dimensionality reduction of each hidden state vector, clusters all the reduced hidden state vectors, regards each cluster cluster as an abstract state, and obtains the abstract state trace of each test sample in the corresponding model; for each original abstract state trace in the set of original abstract state traces, calculates the average value of the distance between it and each mutated abstract state trace in the set of mutated abstract state traces of each variant, obtains the mean difference between the abstract state traces of the original deep learning model and each variant, and determines the mutation operator to be simplified. The present invention reduces the execution cost of mutation testing and improves the effectiveness of evaluating the quality of image or text test case sets.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a mutation operator simplification method and system for a deep learning model. Background Art

[0002] Traditional program mutation testing measures the effectiveness of a test suite by introducing artificial errors into the original program being tested. These programs with artificial errors are called variants. Similarly, mutation testing is increasingly being applied in the field of deep learning. Mutation testing of deep learning models involves making subtle modifications (equivalent to introducing errors) to the model structure, parameters, and state to generate mutated models (also called variants). The difference in prediction results between the original and mutated models after executing the test sample determines whether the test sample can detect changes in the mutated model, ultimately measuring the test sample's error detection capability. However, the large number of test samples and mutation operators generates a large number of variants, leading to high execution costs for mutation testing, which hinders its practical application. Mutation operator reduction methods, including mutation operator selection, higher-order mutation operators, and mutation operator selection, are commonly used. These methods allow targeted screening and optimization of mutation operators, reducing unnecessary mutation operators and ultimately reducing the number of generated variants. This reduces the number of generated variants and lowers the cost of mutation testing.

[0003] Taking face recognition models as an example, these models rely on deep learning models to learn and recognize massive amounts of facial images. If a large number of mutation operators are used during mutation testing, a large number of variants will be generated. Excessive variants not only consume enormous computing resources and time during the training and testing phases, but can also easily lead to overfitting, causing the model to focus on non-critical image details and ignore key recognition features. Mutation operator reduction can accurately select the most valuable mutation operations, eliminate unnecessary variants, and more effectively evaluate whether test samples fully and accurately reflect the model's performance in different face recognition scenarios. This improves the quality assessment of face image test samples, effectively tests the robustness of face recognition models, and ensures that face recognition systems can quickly and accurately identify different faces in real-world use. Intelligent customer service systems rely on deep models to understand and respond to large amounts of user-entered text, which covers a variety of natural language expressions and semantics. Without mutation operator simplification, a large number of duplicate or invalid variants are likely to be generated. However, through mutation operator simplification, insignificant mutation operations can be eliminated, retaining only variants that can reveal potential problems in the model's language processing. This improves the efficiency of quality assessment of test samples and ensures the accuracy and stability of the intelligent customer service system.

[0004] Currently, various mutation operator reduction methods are commonly used in program mutation testing. Lin et al. identified the minimum mutation set based on a coverage approach and found that mutation operators have different importances. Do et al. evaluated the quality of high-order mutation operators and mutants, ranking test cases based on their ability to kill mutants. Zhang et al. categorized mutation selection into operator-based, random, and combinatorial mutation selection, and proposed sampling mutation. Kurtz et al. analyzed various selective mutation testing methods using dominant mutants and minimal mutation testing. Learning-based strategies use code features to assign different importances to mutants and eliminate redundant ones. Wei et al. proposed a mutation reduction method based on spectral clustering to select representative mutants. In mutation testing of deep learning models (deep neural networks (DNNs), Zhang et al. proposed a mutation operator reduction method for deep learning systems. The method consists of three steps: first, classifying mutation operators according to their scope of application; then, combining different types of mutation operators; and finally, analyzing the mutation scores of different mutation operator combinations to obtain a sufficient subset of mutation operators. Feng et al. proposed using DocEntropy to measure the change in the model decision boundary between the generated variants and the original deep learning model for each group of deep learning operators. They then selected the operator group with the highest entropy and used the included operators for further mutation testing. Wang et al. introduced the classification results of deep learning models to conduct a fine-grained evaluation of deep learning operators. They found that some mutation operators were redundant or useless in guiding the generation of high-quality test cases. They then introduced two metrics to measure the usefulness of deep learning mutation operators: the redundancy score (RS) and the quality score (QS). When using mutation operators in a selective mutation strategy, prioritizing mutation operators based on RS can reduce the number of variants; similarly, prioritizing mutation operators based on QS can also reduce the number of variants. Li et al. proposed a mutation testing framework for deep learning models called mutation classification tuples. By applying mutation operators twice, they construct first-order and higher-order (second-order) tuples and search for strongly included high-order tuples from the high-order tuples.

[0005] However, existing mutation operator reduction methods for deep learning models fail to consider the semantic changes they cause to the model. Some mutation operators, while altering the model structure, remain highly similar to other mutation operators from a semantic perspective. Without analyzing these semantic changes, these redundant variants remain unreduced, resulting in wasted computational resources in mutation testing on repeated testing of similar model variations. This makes it difficult to accurately assess the effectiveness of the test set. In image deep learning models, this results in poor image processing accuracy, making it difficult to accurately identify key features and content within the image. In text deep learning models, this leads to poor text processing accuracy, making it difficult to accurately understand and respond to the semantic information within the text, severely impacting the model's performance in practical applications. Furthermore, existing mutation operator reduction methods for deep learning models fail to analyze the impact of mutation operators on prediction results. This can result in the retention of variants that appear to be distinguishable in the test set but actually negatively impact prediction results. The fact that the test set can distinguish predictions from those of the original and mutated models gives the impression that the test set is valid. However, a deeper analysis reveals that some mutation operators subtly alter the model's recognition logic. For example, in a crop pest and disease monitoring scenario, these mutation operators may cause the model to misclassify crops with mild symptoms as healthy. Thus, while the test set may appear to pass the test for distinguishing variants, its effectiveness assessment is actually distorted, failing to truly reflect potential issues with the model when faced with complex real-world situations. Summary of the Invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the fact that the existing methods do not consider the changes caused by the mutation operator to the model semantics and the impact of the mutation operator on the prediction results, resulting in excessively high execution costs of mutation testing and inability to effectively evaluate the quality of the test set, resulting in poor accuracy of the deep learning model's processing results for images or texts.

[0007] To solve the above technical problems, the present invention provides a mutation operator reduction method for a deep learning model, comprising the following steps:

[0008] Use each mutation operator to mutate the original deep learning model and generate a variant of the original deep learning model corresponding to each mutation operator;

[0009] Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and perform dimensionality reduction on each hidden state vector in each hidden state sequence;

[0010] Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and its variants, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state;

[0011] For each test sample's hidden state sequence in the original deep learning model and its variants, each simplified hidden state vector in each hidden state sequence is mapped to its corresponding abstract state, obtaining the abstract state trace of each test sample in the original deep learning model and its variants.

[0012] Summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model; summarize the mutated abstract state traces of all test samples under each variant to obtain the mutated abstract state trace set of each variant;

[0013] For each original abstract state trace in the original deep learning model's original abstract state trace set, calculate the average of the distances between it and each variant abstract state trace in the variant abstract state trace set of each variant, and obtain the mean difference between the abstract state traces of the original deep learning model and each variant;

[0014] According to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, the mutation operator to be simplified is determined, and the mutation operator to be simplified is simplified.

[0015] Preferably, for each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant;

[0016] Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order of frequency, and take the first preset number of subsequences as high-frequency subsequences;

[0017] For each variant, if all subsequences of the current variant do not contain high-frequency subsequences, the mutation operator that generates the current variant will be used as the mutation operator to be reduced.

[0018] Preferably, for each variant, a pattern matching algorithm is used to determine whether all subsequences of the current variant contain a high-frequency subsequence.

[0019] Preferably, determining the mutation operator to be reduced based on the mean difference between the abstract state traces of the original deep learning model and each variant includes:

[0020] If the mean difference in the abstract state traces between the original deep learning model and the current variant is less than a preset threshold, the mutation operator that generates the current variant will be used as the mutation operator to be reduced, and the mutation operator to be reduced will be simplified.

[0021] Preferably, determining the mutation operator to be reduced based on the mean difference between the abstract state traces of the original deep learning model and each variant includes:

[0022] According to the size of the difference between the abstract state trace of the original deep learning model and each variant, the variants are sorted in ascending order. The mutation operator corresponding to the variant in the range is used as the mutation operator to be reduced; among them, The preset ratio.

[0023] Preferably, for each original abstract state trace in the set of original abstract state traces of the original deep learning model, the distance between it and each variant abstract state trace in the set of variant abstract state traces of each variant is calculated, and the calculation method is any one of Levenshtein distance and Jaccard similarity.

[0024] Preferably, for each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated by Jaccard similarity, and the formula is:

[0025] ,

[0026] ,

[0027] in, is the set of original abstract state traces of the original deep learning model calculated by Jaccard similarity With the The set of abstract state traces of variants The average distance between all pairs of abstract state traces in , is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The Jaccard similarity between is the set of original abstract state traces of the original deep learning model Middle A trace of the original abstract state, For the The set of abstract state traces of variants Middle A variation of the abstract state trace, express and The intersection size of express and The union size of is the number of test samples, is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The distance between them.

[0028] Preferably, in the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

[0029] Preferably, when the test sample is image data, the original deep learning model is any one of an image segmentation model, an object detection model, and an image defogging model;

[0030] When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model.

[0031] The present invention also provides a mutation operator reduction system for a deep learning model, comprising:

[0032] The mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator;

[0033] The vector reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence;

[0034] The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and its variants, treating each cluster as an abstract state and mapping the simplified hidden state vectors to the corresponding abstract state;

[0035] An abstract state trace acquisition module is used to map each hidden state vector in each hidden state sequence of each test sample in the original deep learning model and each variant to its corresponding abstract state, thereby obtaining the abstract state trace of each test sample in the original deep learning model and each variant;

[0036] An abstract state trace set construction module is used to aggregate the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model, and aggregate the mutated abstract state traces of all test samples under each variant to obtain the mutated abstract state trace set of each variant;

[0037] An abstract state trace difference mean calculation module is used to calculate the average value of the distance between each original abstract state trace in the original abstract state trace set of the original deep learning model and each variant abstract state trace in the variant abstract state trace set of each variant, thereby obtaining the mean difference between the abstract state traces of the original deep learning model and each variant;

[0038] The mutation operator simplification module is used to determine the mutation operator to be simplified based on the size of the mean difference between the abstract state traces of the original deep learning model and each variant, and to simplify the mutation operator to be simplified.

[0039] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0040] The mutation operator simplification method and system of a deep learning model described in the present invention obtains the hidden state sequence of each test sample in the original deep learning model and each variant during the training process. The hidden state sequence contains the intermediate information of the model during the training process, which can reflect the model's processing process and feature extraction of the test sample, and embodies the changes in the model semantics during the processing process; each hidden state vector is dimensionally simplified, and redundant dimensions are removed without losing key information, so that subsequent analysis is more efficient; all simplified hidden state vectors are clustered, and each cluster cluster is regarded as an abstract state, and the transition sequence between abstract states is used as an abstract state trace; the clustering operation classifies similar hidden state vectors into one category to form an abstract state, so that complex hidden state information can be simplified and abstracted, and the abstract state trace can intuitively show the state change process of the model when processing the test sample; for the original abstract state trace set of the original deep learning model, For each original abstract state trace, the average value of the distance between it and each mutated abstract state trace in the set of mutated abstract state traces of each variant is calculated to obtain the mean difference between the abstract state traces of the original deep learning model and each variant; by calculating the average distance, the degree of semantic difference between the original model and each variant is quantified. The mean difference of the abstract state trace can intuitively reflect the change of each variant at the semantic level relative to the original model, providing a quantitative basis for evaluating the effectiveness and importance of the variant. Based on the quantified difference mean, the mutation operators that contribute less to the semantic change of the model are accurately identified, and the mutation operators to be simplified are determined, avoiding unnecessary repeated testing of these variants in the mutation test, thereby effectively reducing the number of variants and reducing computing resource consumption, so that the mutation test focuses more on the mutations that have a significant impact on the semantics of the model, and can more effectively evaluate the quality of the test samples and improve the accuracy of the model's processing results for images or texts.

[0041] In addition, existing methods do not analyze the impact of mutation operators on prediction results, which may result in the retention of some variants that can be distinguished by the test set on the surface, but actually produce adverse changes to the prediction results. The present invention uses the current abstract state trace as the rotation point of the current variant abstract state trace, and the rotation point as the separation point for each variant abstract state trace in the set of variant abstract state traces of the current variant. This lays the foundation for in-depth research on the state change process of the model between different prediction results, meticulously analyzes the decision-making process of the model, and accurately locates the rotation points of key state changes that lead to prediction deviations. Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order according to frequency from high to low, take the first preset number of subsequences as high-frequency subsequences, and by judging whether the subsequences of the variants contain high-frequency subsequences, it is possible to accurately determine which mutation operators generate variants that have less impact on the model semantics and prediction results, and thus use these mutation operators as objects to be reduced. This avoids retaining invalid or bad variants, reduces the waste of computing resources, and makes mutation testing more focused on mutations that have a significant impact on model performance, thereby improving the efficiency and pertinence of mutation testing, ensuring the effectiveness of test samples, improving the accuracy of image recognition of image deep learning models, and making the text processing results of text deep learning models more semantically consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 This is a flowchart of the steps of a mutation operator simplification method for a deep learning model of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0045] Reference Figure 1 As shown, this embodiment provides a mutation operator reduction method for a deep learning model, including the following steps:

[0046] Step S1: Use each mutation operator to mutate the original deep learning model to generate a variant of the original deep learning model corresponding to each mutation operator;

[0047] In this embodiment, specifically, during the training process of the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

[0048] In this embodiment, specifically, when the test sample is image data, the original deep learning model is any one of an image segmentation model, an object detection model, and an image defogging model;

[0049] If the original deep learning model is an image segmentation model, the original deep learning model is any one of a U-Net model, a Mask R-CNN model, a SegNet model, a recurrent neural network (RNN), and a GRU model;

[0050] If the original deep learning model is an object detection model, the original deep learning model is any one of the Faster R-CNN model and the SSD (Single Shot MultiBox Detector) model;

[0051] If the original deep learning model is an image dehazing model, the original deep learning model is any one of the AOD-Net (All-in-One Dehazing Network) and MSCNN (Multi-Scale Convolutional Neural Network) models;

[0052] When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model;

[0053] If the original deep learning model is a text classification model, then the original deep learning model is either a TextCNN model or an LSTM-based Classifier model;

[0054] If the original deep learning model is a named entity recognition model, it can be either an LSTM-CRF (Long Short-Term Memory with Conditional Random Field) model or a BERT-LSTM-CRF model.

[0055] If the original deep learning model is a text matching model, it can be either the Deep Structured Semantic Model (DSSM) or the Enhanced LSTM for Natural Language Inference (ESIM) model.

[0056] Step S2: Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant , for each hidden state vector in each hidden state sequence Perform dimensionality reduction;

[0057] in, For the Test samples in the model The hidden state sequence during the training process of , whose dimensions are , for each hidden state vector Perform dimensionality reduction, and the dimension after simplification is The hidden state vector , is the total number of time steps, For the Test samples in the model During the training process The hidden state vector corresponding to the time step, For the Test samples in the model During the training process The reduced hidden state vector corresponding to the time steps, and .

[0058] In this embodiment, preferably, principal component analysis is used to perform dimensionality reduction on each hidden state vector in the current hidden state sequence to obtain each reduced hidden state vector in the current hidden state sequence, including:

[0059] For each hidden state vector in the current hidden state sequence, calculate the average value of the corresponding elements of each dimension;

[0060] For each element of each dimension in each hidden state vector, subtract the average value of the elements corresponding to that dimension to obtain each hidden state vector after removing the mean;

[0061] Calculate the covariance matrix of the current hidden state sequence based on the hidden state vectors after removing the mean in the current hidden state sequence;

[0062] By performing eigendecomposition on the covariance matrix of the current hidden state sequence, multiple eigenvalues ​​corresponding to the current hidden state sequence and the eigenvector corresponding to each eigenvalue are obtained;

[0063] According to the size of the eigenvalue corresponding to the current hidden state sequence, the corresponding eigenvectors are sorted, and the hidden state vectors after removing the mean in the current hidden state sequence are projected onto the eigenvector corresponding to the maximum eigenvalue to obtain the simplified hidden state vectors in the current hidden state sequence.

[0064] In this embodiment, principal component analysis is used to Test samples in the model The hidden state vectors in the hidden state sequence during the training process are dimensionality reduced to obtain the first Test samples in the model The hidden state vectors after reduction in the hidden state sequence during the training process include:

[0065] For the Test samples in the model For each hidden state vector in the hidden state sequence during the training process, calculate the average value of the elements corresponding to each dimension as the average value corresponding to each dimension. The formula is:

[0066] ,

[0067] in, For the Test samples in the model In the hidden state sequence during the training process, the first hidden state vector The average value of the elements corresponding to the dimensions, For the Test samples in the model No. The hidden state vector corresponding to the time step Dimensions correspond to elements, is the dimension index, .

[0068] Subtract the average value of each dimension from the elements of each hidden state vector to obtain the hidden state vector after removing the mean. The formula is:

[0069] ,

[0070] in, For the Test samples in the model During the training process The hidden state vector after removing the mean corresponding to the time step Dimensions correspond to elements.

[0071] Based on the Test samples in the model The hidden state vectors after removing the mean in the hidden state sequence during the training process are calculated. Test samples in the model The corresponding covariance matrix;

[0072] ,

[0073] in, For the Test samples in the model The corresponding covariance matrix, For the Test samples in the model During the training process The hidden state vector after removing the mean corresponding to the time step, is transposed.

[0074] Through the Test samples in the model The corresponding covariance matrix is ​​decomposed to obtain the Test samples in the model Corresponding multiple eigenvalues The eigenvector corresponding to each eigenvalue ;

[0075] According to Test samples in the model The size of the corresponding eigenvalue is sorted, and the corresponding eigenvectors are sorted. Test samples in the model In the hidden state sequence during the training process, each hidden state vector after removing the mean is projected onto the eigenvector corresponding to the maximum eigenvalue On, get Test samples in the model The formula for each hidden state vector after reduction in the hidden state sequence during the training process is:

[0076] ,

[0077] in, For the Test samples in the model During the training process The reduced hidden state vector corresponding to the time step is For the Test samples in the model The eigenvector corresponding to the maximum eigenvalue of The dimension is .

[0078] Principal Component Analysis (PCA) can extract key information from high-dimensional data, condensing numerous complex original features into a few comprehensive features. In the context of mutation operator reduction, it can greatly simplify the data structure of hidden state sequences, significantly reducing the amount of data while preserving core information, thereby significantly reducing the complexity of subsequent analysis and processing. This not only speeds up data processing but also enables the model to more efficiently learn and understand the key features of samples, effectively improving the overall efficiency and accuracy of mutation operator reduction.

[0079] Step S3: Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant to obtain Clusters , regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state;

[0080] In this embodiment, specifically, the method for clustering the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant is any one of Gaussian mixture model, hierarchical clustering, K-means clustering, DBSCAN, and spectral clustering.

[0081] In this embodiment, preferably, the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant are clustered using a Gaussian mixture model, including:

[0082] According to the set number of clusters , randomly assign mean and variance to the same number of Gaussian distributions, and based on the mean and variance of each Gaussian distribution, calculate the probability of each reduced hidden state vector under each Gaussian distribution through the probability density function of the Gaussian distribution;

[0083] The formula for calculating the probability of each hidden state vector after reduction under each Gaussian distribution based on the mean and variance of each Gaussian distribution and the probability density function of the Gaussian distribution is:

[0084] ,

[0085] in, For the Test samples in the model During the training process The reduced hidden state vector corresponding to the time step is in The probability under a Gaussian distribution, is the Gaussian distribution index, For the The mean of a Gaussian distribution, For the The variance of a Gaussian distribution, is an exponential function with a natural constant as its base.

[0086] Based on the probability of each simplified hidden state vector under each Gaussian distribution, the mean and variance of each Gaussian distribution are updated until the mean and variance of each Gaussian distribution are in a convergent state. According to the probability of each simplified hidden state vector under each Gaussian distribution in a convergent state, the category of each simplified hidden state vector is divided.

[0087] The distribution of the hidden state vectors after simplification in deep learning models is complex, making simple clustering methods difficult to effectively handle. The Gaussian mixture model, composed of multiple Gaussian distributions, offers high flexibility. It can dynamically adjust its parameters to accurately fit complex data distributions, even under complex and diverse hidden state vector distributions.

[0088] When hidden state vectors exhibit multimodal distributions or complex geometric shapes, the Gaussian mixture model effectively captures these distributional characteristics by superimposing different Gaussian distributions, accurately grouping hidden state vectors with similar characteristics into the same cluster. Compared to traditional simple clustering methods, the Gaussian mixture model does not require a fixed pre-set clustering pattern and can adaptively determine cluster boundaries and characteristics based on the inherent laws of the data. This enables it to demonstrate excellent performance when processing hidden state vectors with different characteristics, stably obtaining high-quality clustering results, providing a solid and reliable data foundation for subsequent mutation operator reduction, and significantly improving the accuracy and effectiveness of mutation operator reduction.

[0089] Step S4: For each hidden state sequence of each test sample in the original deep learning model and each variant, the simplified hidden state vectors in each hidden state sequence are are mapped to their corresponding abstract states , , get the abstract state trace of each test sample in the original deep learning model and each variant ;in, For the Test samples in the model The abstract state trace of For the Test samples in the model During the training process The abstract state corresponding to the simplified hidden state vector corresponding to the time step;

[0090] Step S5: Summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model , summarize the mutation abstract state traces of all test samples under each variant, and obtain the mutation abstract state trace set of each variant ;

[0091] Step S6: For each original abstract state trace in the set of original abstract state traces of the original deep learning model, calculate the average value of the distance between it and each variant abstract state trace in the set of variant abstract state traces of each variant, and obtain the mean difference between the abstract state traces of the original deep learning model and each variant;

[0092] In this embodiment, specifically, for each original abstract state trace in the set of original abstract state traces of the original deep learning model, the distance between it and each variant abstract state trace in the set of variant abstract state traces of each variant is calculated, and the calculation method is any one of Levenshtein distance and Jaccard similarity.

[0093] Jaccard similarity is a commonly used similarity calculation method that can be used to compare the similarity between two sets. Specifically, Jaccard similarity measures the similarity between two sets by calculating the ratio of the size of the intersection to the size of the union.

[0094] In this embodiment, preferably, for each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated by Jaccard similarity, and the formula is:

[0095] ,

[0096] ,

[0097] in, is the set of original abstract state traces of the original deep learning model calculated by Jaccard similarity With the The set of abstract state traces of variants The average distance between all pairs of abstract state traces in , is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The Jaccard similarity between is the set of original abstract state traces of the original deep learning model Middle A trace of the original abstract state, For the The set of abstract state traces of variants Middle A variation of the abstract state trace, express and The intersection size of express and The union size of is the number of test samples, is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The distance between them.

[0098] Levenshtein distance is a type of edit distance, which refers to the minimum number of edit operations required to convert one string into another.

[0099] In this embodiment, preferably, for each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set is calculated using the Levenshtein distance, and the formula is:

[0100] ,

[0101] in, is the set of original abstract state traces of the original deep learning model calculated by Levenshtein distance With the The set of abstract state traces of variants The average distance between all pairs of abstract state traces in , is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The Levenshtein distance between them.

[0102] Step S7: Determine the mutation operator to be simplified based on the mean difference between the abstract state traces of the original deep learning model and each variant, and simplify the mutation operator to be simplified.

[0103] In this embodiment, specifically, determining the mutation operator to be reduced based on the mean difference between the abstract state traces of the original deep learning model and each variant includes:

[0104] If the mean difference between the abstract state traces of the original deep learning model and the current variant is less than a preset threshold, the mutation operator of the current variant is used as the mutation operator to be reduced.

[0105] In this embodiment, specifically, determining the mutation operator to be reduced based on the mean difference between the abstract state traces of the original deep learning model and each variant includes:

[0106] According to the size of the difference between the abstract state trace of the original deep learning model and each variant, the variants are sorted in ascending order. The mutation operator corresponding to the variant in the range is used as the mutation operator to be reduced; among them, The preset ratio.

[0107] Existing methods ignore the impact of mutation operators on model semantics, which not only causes serious waste of computing resources on repeated detection during mutation testing, but also fails to accurately evaluate the effectiveness of the test sample set. This leads to poor accuracy of image deep learning models in identifying key image features, and text deep learning models also find it difficult to accurately understand and respond to text semantics.

[0108] By obtaining the hidden state sequences of test samples during training of the original and variant models, the present invention can demonstrate the dynamic changes in semantics when the model processes the sample. For each original abstract state trace in the set of original abstract state traces of the original deep learning model, the average distance between it and each variant abstract state trace in the set of variant abstract state traces is calculated to obtain the mean difference in abstract state traces between the original deep learning model and each variant. By calculating the average distance, the degree of semantic difference between the original model and each variant is quantified. The mean difference in abstract state traces can intuitively reflect the magnitude of the change in the semantic level of each variant relative to the original model, providing a quantitative basis for evaluating the effectiveness and importance of the variants. Based on the quantified mean difference, mutation operators that contribute less to the semantic change of the model are accurately identified, and mutation operators to be simplified are determined, avoiding unnecessary repeated testing of these variants in mutation testing, thereby effectively reducing the number of variants and reducing computing resource consumption. This allows mutation testing to focus more on mutations that have a significant impact on the model semantics, more effectively evaluating the quality of test samples, and improving the accuracy of the model's processing results for images or text.

[0109] In this embodiment, preferably, for each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant;

[0110] Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order of frequency, and take the first preset number of subsequences as high-frequency subsequences;

[0111] For each variant, if all subsequences of the current variant do not contain high-frequency subsequences, the mutation operator that generates the current variant will be used as the mutation operator to be simplified, and the mutation operator to be simplified will be simplified.

[0112] Pattern matching is a fundamental operation on strings in data structures. Given a substring, finding all substrings within a string that match it is a pattern matching problem. Let P be the given substring, and Y be the string to be searched. Finding all substrings within Y that match P is a pattern matching problem. P is called the pattern, and Y is called the target. If one or more substrings in Y match the pattern P, the position of the substring in Y is output, and the match is considered a success. Otherwise, the match fails.

[0113] In this embodiment, specifically, for each variant, a pattern matching algorithm is used to determine whether all subsequences of the current variant contain a high-frequency subsequence.

[0114] Define each high-frequency subsequence as a pattern , , For the preset number, define each subsequence in the current variant as target Y, and then Perform pattern matching search. If one or more matches with any pattern are found in the subsequence A consistent substring means a successful match, which means the current variant contains a high-frequency subsequence.

[0115] Traditional approaches to simplifying mutation operators only scratch the surface, completely ignoring their deeper impact on model predictions. This results in the retention of many variants that appear distinguishable by the test set. However, these variants actually negatively impact the model's predictions, reducing its accuracy and stability, and significantly compromising its performance in real-world applications.

[0116] In sharp contrast, the method of the present invention is innovative and has significant advantages. For each variant abstract state trace in the set of variant abstract state traces of the current variant, the present invention conducts in-depth mining from the perspective of prediction results. Once a difference is detected between the prediction results corresponding to the current abstract state and the next abstract state, the current abstract state is immediately and accurately located as the rotation point. Using the rotation point as the basis for division, each variant abstract state trace is carefully split to obtain subsequences of all variants. It can go deep into the core link of model decision-making and accurately find the key state change points that lead to prediction deviations, that is, the rotation points. Using the rotation point as a delimiter, each variant abstract state trace is divided to obtain subsequences of all variants. The subsequences of all variants are then aggregated, and the frequency of each subsequence in the set of variant abstract state traces is counted. All subsequences are then sorted in descending order of frequency, with a preset number of subsequences being taken as high-frequency subsequences. By determining whether a variant's subsequence contains a high-frequency subsequence, it is possible to accurately determine which mutation operators generate variants with minimal impact on model semantics and prediction results. These mutation operators are then selected as candidates for reduction. This precise screening and reduction significantly reduces computational resource consumption, allowing mutation testing to focus on mutations that truly have a significant impact on model performance. This not only significantly improves the efficiency of mutation testing, enabling it to complete testing tasks in a shorter time, but also enhances the targeted nature of the test, ensuring that the test results more accurately reflect the actual performance of the model. At the same time, the present invention effectively ensures the validity of the test samples, thereby greatly improving the accuracy of the image deep learning model in image recognition tasks, making image recognition more accurate and reliable; for the text deep learning model, it also makes its text processing results more semantically consistent, showing higher practical value in actual applications.

[0117] This second embodiment provides a mutation operator reduction system for a deep learning model, including:

[0118] The mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator;

[0119] The vector reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence;

[0120] The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and its variants, treating each cluster as an abstract state and mapping the simplified hidden state vectors to the corresponding abstract state;

[0121] An abstract state trace acquisition module is used to map each hidden state vector in each hidden state sequence of each test sample in the original deep learning model and each variant to its corresponding abstract state, thereby obtaining the abstract state trace of each test sample in the original deep learning model and each variant;

[0122] An abstract state trace set construction module is used to aggregate the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model, and aggregate the mutated abstract state traces of all test samples under each variant to obtain the mutated abstract state trace set of each variant;

[0123] An abstract state trace difference mean calculation module is used to calculate the average value of the distance between each original abstract state trace in the original abstract state trace set of the original deep learning model and each variant abstract state trace in the variant abstract state trace set of each variant, thereby obtaining the mean difference between the abstract state traces of the original deep learning model and each variant;

[0124] The mutation operator simplification module is used to determine the mutation operator to be simplified based on the size of the mean difference between the abstract state traces of the original deep learning model and each variant, and to simplify the mutation operator to be simplified.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0129] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A mutation operator reduction method for a deep learning model, characterized in that: The following steps are involved: Use each mutation operator to mutate the original deep learning model and generate a variant of the original deep learning model corresponding to each mutation operator; Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and perform dimensionality reduction on each hidden state vector in each hidden state sequence; Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and its variants, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; For each test sample's hidden state sequence in the original deep learning model and its variants, each simplified hidden state vector in each hidden state sequence is mapped to its corresponding abstract state, obtaining the abstract state trace of each test sample in the original deep learning model and its variants. Summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model; summarize the mutated abstract state traces of all test samples under each variant to obtain the mutated abstract state trace set of each variant; For each original abstract state trace in the original deep learning model's original abstract state trace set, calculate the average of the distances between it and each variant abstract state trace in the variant abstract state trace set of each variant, and obtain the mean difference between the abstract state traces of the original deep learning model and each variant; Determine the mutation operator to be simplified based on the mean difference between the abstract state traces of the original deep learning model and each variant, and simplify the mutation operator to be simplified; For each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, then the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant; Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order of frequency, and take the first preset number of subsequences as high-frequency subsequences; For each variant, if none of the subsequences of the current variant contain high-frequency subsequences, the mutation operator that generated the current variant will be used as the mutation operator to be reduced, and the mutation operator to be reduced will be simplified; Wherein, the test sample is image data or text data; When the test sample is image data, the original deep learning model is any one of the image segmentation model, target detection model, and image dehazing model; When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model.

2. The mutation operator reduction method for a deep learning model according to claim 1, characterized in that: For each variant, a pattern matching algorithm is used to determine whether all subsequences of the current variant contain high-frequency subsequences.

3. The mutation operator reduction method for a deep learning model according to claim 1, characterized in that: The step of determining the mutation operator to be simplified based on the mean difference between the abstract state traces of the original deep learning model and each variant includes: If the mean difference between the abstract state traces of the original deep learning model and the current variant is less than a preset threshold, the mutation operator of the current variant is used as the mutation operator to be reduced.

4. The mutation operator reduction method for a deep learning model according to claim 1, characterized in that: The step of determining the mutation operator to be simplified based on the mean difference between the abstract state traces of the original deep learning model and each variant includes: According to the size of the difference between the original deep learning model and each variant’s abstract state trace, the former is sorted in ascending order. The mutation operator corresponding to the variant in the range is used as the mutation operator to be reduced; among them, The preset ratio.

5. The mutation operator reduction method for a deep learning model according to claim 1, characterized in that: For each original abstract state trace in the original abstract state trace set of the original deep learning model, the distance between it and each mutated abstract state trace in the mutated abstract state trace set of each variant is calculated, and the calculation method is any one of Levenshtein distance and Jaccard similarity.

6. The mutation operator reduction method for a deep learning model according to claim 1, characterized in that: For each original abstract state trace in the original deep learning model’s original abstract state trace set, the average distance between it and each variant abstract state trace in the variant abstract state trace set is calculated using the Jaccard similarity. The formula is: ， ， in, is the set of original abstract state traces of the original deep learning model calculated by Jaccard similarity With the The set of abstract state traces of variants The average distance between all pairs of abstract state traces in , is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The Jaccard similarity between is the set of original abstract state traces of the original deep learning model Middle A trace of the original abstract state, For the The set of abstract state traces of variants Middle A variation of the abstract state trace, express and The intersection size of express and The union size of is the number of test samples, is the set of original abstract state traces of the original deep learning model Middle Original abstract state trace With the The set of abstract state traces of variants Middle Variation abstract state trace The distance between them.

7. The mutation operator reduction method for a deep learning model according to claim 1, characterized in that: In the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

8. A mutation operator reduction system for a deep learning model, characterized in that: include: The mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator; The vector reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence; The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and its variants, treating each cluster as an abstract state and mapping the simplified hidden state vectors to the corresponding abstract state; An abstract state trace acquisition module is used to map each hidden state vector in each hidden state sequence of each test sample in the original deep learning model and each variant to its corresponding abstract state, thereby obtaining the abstract state trace of each test sample in the original deep learning model and each variant; An abstract state trace set construction module is used to aggregate the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model, and aggregate the mutated abstract state traces of all test samples under each variant to obtain the mutated abstract state trace set of each variant; An abstract state trace difference mean calculation module is used to calculate the average value of the distance between each original abstract state trace in the original abstract state trace set of the original deep learning model and each variant abstract state trace in the variant abstract state trace set of each variant, thereby obtaining the mean difference between the abstract state traces of the original deep learning model and each variant; The mutation operator reduction module is used to determine the mutation operator to be reduced based on the mean difference between the abstract state traces of the original deep learning model and each variant, and to reduce the mutation operator to be reduced; For each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, then the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant; Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order of frequency, and take the first preset number of subsequences as high-frequency subsequences; For each variant, if none of the subsequences of the current variant contain high-frequency subsequences, the mutation operator that generated the current variant will be used as the mutation operator to be reduced, and the mutation operator to be reduced will be simplified; Wherein, the test sample is image data or text data; When the test sample is image data, the original deep learning model is any one of the image segmentation model, target detection model, and image dehazing model; When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model.

Images