Variation operator reduction method and system of deep learning model

Abstract

The invention relates to the field of software testing, in particular to a mutation operator reduction method and system of a deep learning model. According to the method, variation operators are used for carrying out variation on an original model, a hidden state sequence of each test sample in the original model and variation bodies in the training process is obtained, after dimension reduction is carried out on each hidden state vector, clustering is carried out on all the reduced hidden state vectors, each cluster is regarded as an abstract state, and then the hidden state vectors are clustered. Obtaining an abstract state trace of each test sample in the corresponding model; and for each original abstract state trace in the original abstract state trace set, calculating the average value of the distance between the original abstract state trace and each variant abstract state trace in the variant abstract state trace set of each variant, obtaining the abstract state trace difference average value of the original deep learning model and each variant, and determining a mutation operator to be reduced. According to the method, the execution cost of variation testing is reduced, and the effectiveness of evaluating the quality of the image or text test case set is improved.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a mutation operator simplification method and system for a deep learning model. Background Art

[0002] Traditional program mutation testing measures the effectiveness of the test case set by implanting artificial errors in the original program to be tested. The program with artificial errors is called a mutant. Similarly, mutation testing is increasingly being used in the field of deep learning. Mutation testing of deep learning models generates mutation models (also called mutants) by making subtle modifications (equivalent to implanting errors) to the model structure, model parameters, model state, etc. Based on whether there is a difference in the prediction results after the test sample is executed on the original model and the mutation model, it is determined whether the test sample can detect the changes on the mutation model, and finally the error detection ability of the test sample is measured. However, since a large number of test samples and mutation operators will generate a large number of mutants, the execution cost of mutation testing is too high, which affects the actual application effect of mutation testing. In terms of mutation operator simplification, methods including mutation operators, high-order mutation operators, and mutation operator selection are usually used. Through these methods, mutation operators can be screened and optimized in a targeted manner to reduce unnecessary mutation operators, thereby reducing the number of generated mutants from the source. In this way, the number of generated mutants is reduced and the cost of mutation testing is reduced.

[0003] Taking the face recognition model as an example, the model relies on a deep model to learn and recognize massive face images. If a large number of mutation operators are used in mutation testing, a large number of mutants will be generated. Too many mutants will not only consume huge computing resources and time in the training and testing stages, but also easily cause overfitting, causing the model to focus on non-critical image details and ignore key recognition features. With the help of mutation operator simplification, the most valuable mutation operations can be accurately screened, unnecessary mutants can be reduced, and the test samples can be more effectively evaluated to see whether they can fully and accurately reflect the performance of the model in different face recognition scenarios, thereby improving the quality evaluation effect of face image test samples, effectively detecting the robustness of face recognition models, and ensuring that face recognition systems can quickly and accurately identify different faces in actual use. The intelligent customer service system relies on deep models to understand and respond to a large amount of user input text, which covers a variety of natural language expressions and semantics. If mutation operator simplification is not performed, a large number of duplicate or invalid variants are likely to be generated. Through mutation operator simplification, mutation operations with little effect can be eliminated, and only variants that can reveal potential problems of the model in language processing are retained, thereby improving the quality assessment efficiency of test samples and ensuring the accuracy and stability of the intelligent customer service system.

[0004] At present, in the mutation testing of programs, a variety of mutation operator reduction methods are often used. Lin et al. identified the minimum mutation set based on the coverage method and found that the importance of mutation operators is different; Do et al. evaluated the quality of high-order mutation operators and mutants, and sorted test cases according to the ability to kill mutants; Zhang divided mutation selection into operator-based, random and combination mutation selection, and proposed sampling mutation; Kurtz et al. analyzed a variety of selective mutation testing methods through dominance mutants and minimum mutation testing; learning-based strategies use code features to give different importance to mutants and eliminate redundant mutants; Wei et al. proposed a mutant reduction method based on spectral clustering to select representative mutants. In the mutation testing of deep learning models (deep neural networks, DNNs), Zhang et al. proposed a mutation operator reduction method for deep learning systems. The method is divided into three steps: first, classify mutation operators according to their scope of action; then, combine different types of mutation operators; finally, obtain a sufficient subset of mutation operators by analyzing the mutation scores of different mutation operator combinations. Feng et al. proposed using DocEntropy to measure the model decision boundary changes between the generated variants and the original deep learning model for each group of deep learning operators. Then, the operator group with the highest entropy value was selected and the included operators were used for further mutation testing. Wang et al. introduced the classification results of deep learning models to conduct fine-grained evaluation of deep learning operators. They found that some mutation operators were redundant or useless in guiding the generation of high-quality test cases. They then introduced two indicators to measure the usefulness of deep learning mutation operators: redundancy score RS and quality score QS. When using mutation operators in selective mutation strategies, prioritizing mutation operators based on RS can reduce the number of variants; similarly, prioritizing mutation operators based on QS can reduce the number of variants. Li et al. proposed a deep learning model mutation testing framework called mutation classification tuples, which constructs first-order and high-order (second-order) tuples by applying mutation operators twice, and searches for strongly included high-order tuples from high-order tuples.

[0005] However, the existing mutation operator simplification methods for deep learning models do not consider the changes in the semantics of the model caused by the mutation operator. Although some mutation operators change the model structure, they are still highly similar to other mutation operators from a semantic perspective. Since the semantic changes are not analyzed, these redundant variants are not simplified, which wastes computing resources in mutation testing on repeated detection of similar model changes, making it impossible to accurately evaluate the effectiveness of the test sample set. In image deep learning models, this will lead to poor accuracy of the model's image processing results and difficulty in accurately identifying the key features and content in the image; in text deep learning models, this will lead to poor accuracy of the model's text processing results and inability to accurately understand and respond to the semantic information in the text, which seriously affects the performance of the model in practical applications. In addition, the existing mutation operator simplification methods for deep learning models do not analyze the impact of mutation operators on prediction results. This may result in some variants that can be distinguished by the test set on the surface, but actually have adverse changes to the prediction results being retained. From a phenotypic point of view, the test set can distinguish the prediction results of the original model and the variant model, which makes people feel that the test set is effective. However, a deeper analysis reveals that some mutation operators have quietly changed the recognition logic of the model. For example, in the crop pest monitoring scenario, these mutation operators may cause the model to misjudge some crops with mild symptoms as healthy. In this way, although the test set seems to have passed the test of distinguishing variants, in fact, the evaluation of its own effectiveness has been distorted and cannot truly reflect the potential problems of the model when facing actual complex situations. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problem that the existing methods do not consider the changes caused by the mutation operator to the model semantics and the impact of the mutation operator on the prediction results, resulting in excessively high execution costs of mutation testing and inability to effectively evaluate the quality of the test set, resulting in poor accuracy of the deep learning model's processing results for images or texts.

[0007] In order to solve the above technical problems, the present invention provides a mutation operator simplification method for a deep learning model, comprising the following steps: Use each mutation operator to mutate the original deep learning model to generate a variant of the original deep learning model corresponding to each mutation operator; Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and perform dimensionality reduction on each hidden state vector in each hidden state sequence; Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; For each test sample's hidden state sequence in the original deep learning model and each variant, map each simplified hidden state vector in each hidden state sequence to its corresponding abstract state, and obtain the abstract state trace of each test sample in the original deep learning model and each variant; The original abstract state traces of all test samples under the original deep learning model are summarized to obtain the original abstract state trace set of the original deep learning model, and the mutated abstract state traces of all test samples under each variant are summarized to obtain the mutated abstract state trace set of each variant; For each original abstract state trace in the original abstract state trace set of the original deep learning model, calculate the average value of the distance between it and each mutated abstract state trace in the mutated abstract state trace set of each variant, and obtain the mean difference between the abstract state trace of the original deep learning model and each variant; According to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, the mutation operator to be simplified is determined, and the mutation operator to be simplified is simplified.

[0008] Preferably, for each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant; Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order from high to low frequency, and take the first preset number of subsequences as high-frequency subsequences; For each variant, if all subsequences of the current variant do not contain high-frequency subsequences, the mutation operator that generates the current variant is used as the mutation operator to be reduced.

[0009] Preferably, for each variant, a pattern matching algorithm is used to determine whether all subsequences of the current variant contain a high-frequency subsequence.

[0010] Preferably, determining the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant includes: If the mean difference between the abstract state traces of the original deep learning model and the current variant is less than a preset threshold, the mutation operator that generates the current variant is used as the mutation operator to be simplified, and the mutation operator to be simplified is simplified.

[0011] Preferably, determining the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant includes: According to the size of the difference between the abstract state trace of the original deep learning model and each variant, the variants are sorted in ascending order. The mutation operator corresponding to the variant in the range is used as the mutation operator to be reduced; among them, The preset ratio.

[0012] Preferably, for each original abstract state trace in the set of original abstract state traces of the original deep learning model, the distance between it and each variant abstract state trace in the set of variant abstract state traces of each variant is calculated, and the calculation method is any one of Levenshtein distance and Jaccard similarity.

[0013] Preferably, for each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated by Jaccard similarity, and the formula is: , , in, is the set of original abstract state traces of the original deep learning model calculated by Jaccard similarity With The set of variant abstract state traces The average distance between all pairs of abstract state traces in , is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The Jaccard similarity between is the original abstract state trace set of the original deep learning model Middle A trace of the original abstract state, For the The set of variant abstract state traces Middle A variation of the abstract state trace, express and The size of the intersection, express and The size of the union of is the number of test samples, is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The distance between.

[0014] Preferably, in the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

[0015] Preferably, when the test sample is image data, the original deep learning model is any one of an image segmentation model, a target detection model, and an image defogging model; When the test sample is text data, the original deep learning model is any one of a text classification model, a named entity recognition model, and a text matching model.

[0016] The present invention also provides a mutation operator reduction system for a deep learning model, comprising: A mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator; The vector reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence; The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; An abstract state trace acquisition module is used to map each hidden state vector after reduction in each hidden state sequence to its corresponding abstract state for each test sample in the original deep learning model and each variant, so as to obtain the abstract state trace of each test sample in the original deep learning model and each variant; An abstract state trace set construction module is used to summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model, and summarize the variant abstract state traces of all test samples under each variant to obtain the variant abstract state trace set of each variant; An abstract state trace difference mean value calculation module is used to calculate the average value of the distance between each original abstract state trace in the original abstract state trace set of the original deep learning model and each variant abstract state trace in the variant abstract state trace set of each variant, so as to obtain the abstract state trace difference mean value between the original deep learning model and each variant; The mutation operator simplification module is used to determine the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, and to simplify the mutation operator to be simplified.

[0017] The above technical solution of the present invention has the following beneficial effects compared with the prior art: The mutation operator simplification method and system of a deep learning model described in the present invention obtains the hidden state sequence of each test sample in the original deep learning model and each variant during the training process. The hidden state sequence contains the intermediate information of the model during the training process, can reflect the model's processing process and feature extraction of the test sample, and embodies the changes in the semantics of the model during the processing process; dimensionality is simplified for each hidden state vector, and redundant dimensions are removed without losing key information, so that subsequent analysis is more efficient; all simplified hidden state vectors are clustered, each cluster cluster is regarded as an abstract state, and the transition sequence between abstract states is used as an abstract state trace; the clustering operation classifies similar hidden state vectors into one category to form an abstract state, so that complex hidden state information can be simplified and abstracted, and the abstract state trace can intuitively show the state change process of the model when processing the test sample; for the original abstract state trace set of the original deep learning model, For each original abstract state trace, the average value of the distance between it and each mutated abstract state trace in the set of mutated abstract state traces of each variant is calculated to obtain the mean difference in abstract state traces between the original deep learning model and each variant; by calculating the average distance, the degree of semantic difference between the original model and each variant is quantified. The mean difference in abstract state traces can intuitively reflect the change in the semantic level of each variant relative to the original model, providing a quantitative basis for evaluating the effectiveness and importance of variants. Based on the quantified difference mean, those mutation operators that contribute less to the semantic change of the model are accurately identified, and the mutation operators to be simplified are determined to avoid unnecessary repeated testing of these variants in mutation testing, thereby effectively reducing the number of variants and reducing computing resource consumption, so that mutation testing can focus more on mutations that have a significant impact on the semantics of the model, and can more effectively evaluate the quality of test samples and improve the accuracy of the model's processing results for images or texts.

[0018] In addition, the existing methods do not analyze the impact of mutation operators on the prediction results, which may result in some variants that can be distinguished by the test set on the surface, but actually produce adverse changes to the prediction results, being retained. For each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain subsequences of all variants; this lays the foundation for in-depth research on the state change process of the model between different prediction results, meticulously analyzes the decision-making process of the model, and accurately locates the rotation points of key state changes that cause prediction deviations. Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order from high to low frequency, take the first preset number of subsequences as high-frequency subsequences, and by judging whether the subsequences of the variants contain high-frequency subsequences, it is possible to accurately determine which mutation operators generate variants that have less impact on the model semantics and prediction results, and thus use these mutation operators as objects to be reduced. This avoids retaining invalid or bad variants, reduces the waste of computing resources, and makes mutation testing more focused on mutations that have a significant impact on model performance, thereby improving the efficiency and pertinence of mutation testing, ensuring the effectiveness of test samples, and improving the accuracy of image recognition of image deep learning models, making the text processing results of text deep learning models more semantically consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 It is a step flow chart of a mutation operator simplification method for a deep learning model of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0021] Reference Figure 1 As shown, this embodiment 1 provides a mutation operator simplification method for a deep learning model, including the following steps: Step S1: using each mutation operator to mutate the original deep learning model, and generating a variant of the original deep learning model corresponding to each mutation operator; In this embodiment, specifically, in the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

[0022] In this embodiment, specifically, when the test sample is image data, the original deep learning model is any one of an image segmentation model, a target detection model, and an image defogging model; If the original deep learning model is an image segmentation model, the original deep learning model is any one of a U-Net model, a Mask R-CNN model, a SegNet model, a recurrent neural network (RNN), and a GRU model; If the original deep learning model is an object detection model, the original deep learning model is any one of the Faster R-CNN model and the SSD (Single Shot MultiBox Detector) model; If the original deep learning model is an image dehazing model, the original deep learning model is any one of the AOD-Net (All-in-One Dehazing Network) and MSCNN (Multi-Scale Convolutional Neural Network) models; When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model; If the original deep learning model is a text classification model, the original deep learning model is any one of the TextCNN model and the LSTM-based Classifier model; If the original deep learning model is a named entity recognition model, the original deep learning model is any one of an LSTM-CRF (Long Short-Term Memory with Conditional Random Field) model and a BERT-LSTM-CRF model; If the original deep learning model is a text matching model, the original deep learning model is any one of the Deep Structured Semantic Model (DSSM) model and the Enhanced LSTM for Natural Language Inference (ESIM) model.

[0023] Step S2: Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant , for each hidden state vector in each hidden state sequence Perform dimensionality reduction; in, For the The test samples in the model The hidden state sequence during the training process of , whose dimensions are , for each hidden state vector Perform dimensionality reduction, and the dimension after simplification is The hidden state vector , is the total number of time steps, For the The test samples in the model During the training process The hidden state vector corresponding to the time step is For the The test samples in the model During the training process The reduced hidden state vector corresponding to the time step is .

[0024] In this embodiment, preferably, principal component analysis is used to reduce the dimension of each hidden state vector in the current hidden state sequence to obtain each reduced hidden state vector in the current hidden state sequence, including: For each hidden state vector in the current hidden state sequence, calculate the average value of the elements corresponding to each dimension; For each element of each dimension in each hidden state vector, subtract the average value of the element corresponding to that dimension to obtain each hidden state vector after removing the mean value; Based on each hidden state vector after removing the mean in the current hidden state sequence, calculate the covariance matrix of the current hidden state sequence; By performing eigendecomposition on the covariance matrix of the current hidden state sequence, a plurality of eigenvalues ​​corresponding to the current hidden state sequence and an eigenvector corresponding to each eigenvalue are obtained; According to the size of the eigenvalue corresponding to the current hidden state sequence, the corresponding eigenvectors are sorted, and each hidden state vector after removing the mean in the current hidden state sequence is projected onto the eigenvector corresponding to the maximum eigenvalue to obtain each simplified hidden state vector in the current hidden state sequence.

[0025] In this embodiment, principal component analysis is used to The test samples in the model The hidden state vectors in the hidden state sequence during the training process are dimensionally reduced to obtain the first The test samples in the model The hidden state vectors after simplification in the hidden state sequence during the training process include: For The test samples in the model For each hidden state vector in the hidden state sequence during the training process, calculate the average value of the elements corresponding to each dimension as the average value corresponding to each dimension. The formula is: , in, For the The test samples in the model In the hidden state sequence during the training process, the first hidden state vector The average value of the elements corresponding to the dimensions, For the The test samples in the model No. The hidden state vector corresponding to the time step Dimensions correspond to elements, is the dimension index, .

[0026] Subtract the average value corresponding to each dimension from the elements of each hidden state vector to obtain the hidden state vectors after removing the mean. The formula is: , in, For the The test samples in the model During the training process The hidden state vector after removing the mean corresponding to the time step Dimensions correspond to elements.

[0027] Based on The test samples in the model The hidden state vectors after removing the mean in the hidden state sequence during the training process are calculated. The test samples in the model The corresponding covariance matrix; , in, For the The test samples in the model The corresponding covariance matrix is, For the The test samples in the model During the training process The hidden state vector after removing the mean corresponding to the time step, is transposed.

[0028] Through the The test samples in the model The corresponding covariance matrix is ​​decomposed to obtain The test samples in the model The corresponding multiple eigenvalues The eigenvector corresponding to each eigenvalue ; According to The test samples in the model The corresponding eigenvalues ​​are sorted according to their corresponding eigenvectors. The test samples in the model In the hidden state sequence during the training process, each hidden state vector after removing the mean is projected to the eigenvector corresponding to the maximum eigenvalue On, get The test samples in the model The formula for each hidden state vector after simplification in the hidden state sequence during the training process is: , in, For the The test samples in the model During the training process The reduced hidden state vector corresponding to the time step is For the The test samples in the model The eigenvector corresponding to the maximum eigenvalue of The dimension is .

[0029] Principal Component Analysis (PCA) can extract key information from high-dimensional data and condense numerous complex original features into a few comprehensive features. In the scenario of mutation operator reduction, it can greatly simplify the data structure of the hidden state sequence, greatly reduce the amount of data while retaining the core information, and thus significantly reduce the complexity of subsequent analysis and processing. This not only speeds up data processing, but also enables the model to learn and understand the key features of samples more efficiently, effectively improving the overall efficiency and accuracy of mutation operator reduction.

[0030] Step S3: Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant to obtain Clusters , regard each cluster as an abstract state, and map each simplified hidden state vector to the corresponding abstract state; In this embodiment, specifically, the method for clustering the hidden state vectors of all test samples after simplification during the original deep learning model and the training process of each variant is any one of Gaussian mixture model, hierarchical clustering, K-means clustering, DBSCAN, and spectral clustering.

[0031] In this embodiment, preferably, the Gaussian mixture model is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant, including: According to the set number of clusters , randomly assign mean and variance to the same number of Gaussian distributions, and calculate the probability of each hidden state vector after reduction under each Gaussian distribution based on the mean and variance of each Gaussian distribution through the probability density function of the Gaussian distribution; The formula for calculating the probability of each hidden state vector after simplification under each Gaussian distribution based on the mean and variance of each Gaussian distribution and the probability density function of the Gaussian distribution is: , in, For the The test samples in the model During the training process The reduced hidden state vector corresponding to the time step is The probability under a Gaussian distribution is is the Gaussian distribution index, For the The mean of a Gaussian distribution, For the The variance of a Gaussian distribution, is an exponential function with a natural constant as its base.

[0032] Based on the probability of each hidden state vector after simplification under each Gaussian distribution, the mean and variance of each Gaussian distribution are updated until the mean and variance of each Gaussian distribution are in a convergent state, and each hidden state vector after simplification is classified according to the probability of each hidden state vector after simplification under each Gaussian distribution in a convergent state.

[0033] The hidden state vector distribution after simplification of the deep learning model is complex, and simple clustering methods are difficult to effectively handle. The Gaussian mixture model is composed of multiple Gaussian distributions and has high flexibility. In the face of complex and diverse hidden state vector distributions, it can dynamically adjust its own parameters and accurately fit various complex data distributions.

[0034] When the hidden state vector presents a multi-peak distribution or complex geometric shape, the Gaussian mixture model effectively captures these distribution characteristics by superimposing different Gaussian distributions, and accurately divides the hidden state vectors with similar characteristics into the same cluster cluster. Compared with the traditional simple clustering method, the Gaussian mixture model does not need to preset a fixed clustering mode, and can adaptively determine the clustering boundaries and characteristics according to the intrinsic laws of the data. This enables it to demonstrate excellent performance when processing hidden state vectors with different characteristics, stably obtain high-quality clustering results, and provide a solid and reliable data foundation for subsequent mutation operator simplification, which greatly improves the accuracy and effectiveness of mutation operator simplification.

[0035] Step S4: For each hidden state sequence of each test sample in the original deep learning model and each variant, the simplified hidden state vectors in each hidden state sequence are are mapped to their corresponding abstract states , , and obtain the abstract state traces of each test sample in the original deep learning model and each variant ;in, For the The test samples in the model The abstract state trace of For the The test samples in the model During the training process The abstract state corresponding to the reduced hidden state vector corresponding to the time step; Step S5: Summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model , summarize the mutation abstract state traces of all test samples under each variant, and obtain the mutation abstract state trace set of each variant ; Step S6: for each original abstract state trace in the original abstract state trace set of the original deep learning model, calculate the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant, and obtain the mean difference between the abstract state trace of the original deep learning model and each variant; In this embodiment, specifically, for each original abstract state trace in the set of original abstract state traces of the original deep learning model, the distance between it and each mutated abstract state trace in the set of mutated abstract state traces of each variant is calculated, and the calculation method is any one of the Levenshtein distance and Jaccard similarity.

[0036] Jaccard similarity is a commonly used similarity calculation method, which can be used to compare the similarity between two sets. Specifically, Jaccard similarity measures the similarity between two sets by calculating the ratio of the size of the intersection to the size of the union.

[0037] In this embodiment, preferably, for each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated by Jaccard similarity, and the formula is: , , in, is the set of original abstract state traces of the original deep learning model calculated by Jaccard similarity With The set of variant abstract state traces The average distance between all pairs of abstract state traces in , is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The Jaccard similarity between is the original abstract state trace set of the original deep learning model Middle A trace of the original abstract state, For the The set of variant abstract state traces Middle A variation of the abstract state trace, express and The size of the intersection, express and The size of the union of is the number of test samples, is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The distance between.

[0038] Levenshtein distance is a type of edit distance, which refers to the minimum number of edit operations required to convert one string into another.

[0039] In this embodiment, preferably, for each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated by Levenshtein distance, and the formula is: , in, is the set of original abstract state traces of the original deep learning model calculated by Levenshtein distance With The set of variant abstract state traces The average distance between all pairs of abstract state traces in , is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The Levenshtein distance between .

[0040] Step S7: Determine the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, and simplify the mutation operator to be simplified.

[0041] In this embodiment, specifically, determining the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant includes: If the mean difference between the abstract state traces of the original deep learning model and the current variant is less than a preset threshold, the mutation operator that generates the current variant is used as the mutation operator to be reduced.

[0042] In this embodiment, specifically, determining the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant includes: According to the size of the difference between the abstract state trace of the original deep learning model and each variant, the variants are sorted in ascending order. The mutation operator corresponding to the variant in the range is used as the mutation operator to be reduced; among them, The preset ratio.

[0043] Existing methods ignore the impact of mutation operators on model semantics, which not only causes serious waste of computing resources on repeated detection during mutation testing, but also fails to accurately evaluate the effectiveness of the test sample set, resulting in poor accuracy of image deep learning models in identifying key image features, and text deep learning models also find it difficult to accurately understand and respond to text semantics.

[0044] The present invention can show the dynamic changes of semantics when the model processes samples by obtaining the hidden state sequence of the test samples in the original and variant model training. For each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated to obtain the mean difference of the abstract state trace between the original deep learning model and each variant. By calculating the average value of the distance, the degree of semantic difference between the original model and each variant is quantified. The mean value of the abstract state trace difference can intuitively reflect the change size of each variant at the semantic level relative to the original model, providing a quantitative basis for evaluating the effectiveness and importance of the variant. According to the quantified difference mean, the mutation operators that contribute less to the semantic change of the model are accurately identified, and the mutation operators to be simplified are determined, so as to avoid unnecessary repeated detection of these variants in the mutation test, thereby effectively reducing the number of variants and reducing the consumption of computing resources, so that the mutation test is more focused on the mutation situation that has a significant impact on the semantics of the model, and can more effectively evaluate the quality of the test sample, and improve the accuracy of the model's processing results for images or texts.

[0045] In this embodiment, preferably, for each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant; Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order from high to low frequency, and take the first preset number of subsequences as high-frequency subsequences; For each variant, if all subsequences of the current variant do not contain high-frequency subsequences, the mutation operator that generates the current variant is used as the mutation operator to be simplified, and the mutation operator to be simplified is simplified.

[0046] Pattern matching is a basic operation of strings in data structures. Given a substring, it is required to find all substrings that are identical to the substring in a string. This is called pattern matching. Let P be the given substring, Y is the string to be searched, and it is required to find all substrings that are identical to P from Y. This problem is called pattern matching. P is called the pattern, and Y is called the target. If there are one or more substrings in Y with the pattern P, the position of the substring in Y is output, which is called a successful match. Otherwise, the match fails.

[0047] In this embodiment, specifically, for each variant, a pattern matching algorithm is used to determine whether all subsequences of the current variant contain a high-frequency subsequence.

[0048] Define each high-frequency subsequence as a pattern , , For the preset number, define each subsequence in the current variant as target Y, and then Perform a pattern matching search. If one or more matches with any pattern are found in the subsequence, A consistent substring means a successful match, which means that the current variant contains a high-frequency subsequence.

[0049] The traditional method only stays on the surface when simplifying the mutation operator, and completely ignores the deep impact of the mutation operator on the model prediction results. This results in many variants that seem to be distinguishable by the test set being retained, but these variants actually have a negative impact on the model's prediction results, reducing the accuracy and stability of the model, and greatly reducing the performance of the model in practical applications.

[0050] In sharp contrast, the method of the present invention is innovative and has significant advantages. For each variant abstract state trace in the set of variant abstract state traces of the current variant, the present invention conducts in-depth mining from the perspective of prediction results. Once it is detected that there is a difference in the prediction results corresponding to the current abstract state and the next abstract state, the current abstract state is immediately and accurately located as the rotation point. Using the rotation point as the basis for division, each variant abstract state trace is carefully split to obtain subsequences of all variants. It is possible to go deep into the core link of the model decision and accurately find the key state change points that lead to prediction deviations, that is, the rotation points. The rotation point is used as a dividing point to divide each mutation abstract state trace to obtain the subsequences of all variants; the subsequences of all variants are summarized, and the frequency of each subsequence in the set of mutation abstract state traces of all variants is counted. All subsequences are arranged in descending order from high to low frequency, and the first preset number of subsequences are taken as high-frequency subsequences. By judging whether the subsequence of the variant contains a high-frequency subsequence, it is possible to accurately determine which mutation operators generate variants that have less impact on the model semantics and prediction results, so that these mutation operators are used as objects to be simplified. Through precise screening and simplification, the consumption of computing resources is greatly reduced, allowing mutation testing to focus on those mutations that really have a significant impact on model performance. This not only significantly improves the efficiency of mutation testing, enabling it to complete the test task in a shorter time, but also enhances the pertinence of the test, ensuring that the test results can more accurately reflect the actual performance of the model. At the same time, the present invention effectively ensures the validity of the test samples, thereby greatly improving the accuracy of the image deep learning model in image recognition tasks, making image recognition more accurate and reliable; for the text deep learning model, it also makes its text processing results more semantically consistent, showing higher practical value in actual applications.

[0051] This second embodiment provides a mutation operator reduction system for a deep learning model, including: A mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator; The vector reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence; The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; An abstract state trace acquisition module is used to map each hidden state vector after reduction in each hidden state sequence to its corresponding abstract state for each test sample in the original deep learning model and each variant, so as to obtain the abstract state trace of each test sample in the original deep learning model and each variant; An abstract state trace set construction module is used to summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model, and summarize the variant abstract state traces of all test samples under each variant to obtain the variant abstract state trace set of each variant; An abstract state trace difference mean value calculation module is used to calculate the average value of the distance between each original abstract state trace in the original abstract state trace set of the original deep learning model and each variant abstract state trace in the variant abstract state trace set of each variant, so as to obtain the abstract state trace difference mean value between the original deep learning model and each variant; The mutation operator simplification module is used to determine the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, and to simplify the mutation operator to be simplified.

[0052] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0053] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0056] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A mutation operator reduction method for a deep learning model, characterized in that: The following steps are involved: Use each mutation operator to mutate the original deep learning model to generate a variant of the original deep learning model corresponding to each mutation operator; Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and perform dimensionality reduction on each hidden state vector in each hidden state sequence; Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; For each test sample's hidden state sequence in the original deep learning model and each variant, map each simplified hidden state vector in each hidden state sequence to its corresponding abstract state, and obtain the abstract state trace of each test sample in the original deep learning model and each variant; The original abstract state traces of all test samples under the original deep learning model are summarized to obtain the original abstract state trace set of the original deep learning model, and the mutated abstract state traces of all test samples under each variant are summarized to obtain the mutated abstract state trace set of each variant; For each original abstract state trace in the original abstract state trace set of the original deep learning model, calculate the average value of the distance between it and each mutated abstract state trace in the mutated abstract state trace set of each variant, and obtain the mean difference between the abstract state trace of the original deep learning model and each variant; According to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, the mutation operator to be simplified is determined, and the mutation operator to be simplified is simplified.

2. The mutation operator simplification method of a deep learning model according to claim 1, characterized in that: For each variant abstract state trace in the variant abstract state trace set of the current variant, if the prediction result corresponding to the current abstract state is different from the prediction result corresponding to the next abstract state, the current abstract state is used as the rotation point of the current variant abstract state trace, and the rotation point is used as the separation point to divide each variant abstract state trace to obtain multiple subsequences of the current variant; Summarize the subsequences of all variants, count the frequency of each subsequence in the set of variant abstract state traces of all variants, sort all subsequences in descending order from high to low frequency, and take the first preset number of subsequences as high-frequency subsequences; For each variant, if all subsequences of the current variant do not contain high-frequency subsequences, the mutation operator that generates the current variant is used as the mutation operator to be simplified, and the mutation operator to be simplified is simplified.

3. The mutation operator reduction method of a deep learning model according to claim 2, characterized in that: For each variant, a pattern matching algorithm is used to determine whether all subsequences of the current variant contain a high-frequency subsequence.

4. The mutation operator reduction method of a deep learning model according to claim 1, characterized in that: The step of determining the mutation operator to be simplified according to the size of the difference between the abstract state traces of the original deep learning model and each variant includes: If the mean difference between the abstract state traces of the original deep learning model and the current variant is less than a preset threshold, the mutation operator that generates the current variant is used as the mutation operator to be reduced.

5. The mutation operator reduction method of a deep learning model according to claim 1, characterized in that: The step of determining the mutation operator to be simplified according to the size of the difference between the abstract state traces of the original deep learning model and each variant includes: According to the size of the difference between the original deep learning model and each variant’s abstract state trace, the former is sorted in ascending order. The mutation operator corresponding to the variant in the range is used as the mutation operator to be reduced; among them, The preset ratio.

6. The mutation operator reduction method of a deep learning model according to claim 1, characterized in that: For each original abstract state trace in the original abstract state trace set of the original deep learning model, the distance between it and each mutated abstract state trace in the mutated abstract state trace set of each variant is calculated, and the calculation method is any one of Levenshtein distance and Jaccard similarity.

7. The mutation operator reduction method of a deep learning model according to claim 1, characterized in that: For each original abstract state trace in the original abstract state trace set of the original deep learning model, the average value of the distance between it and each variant abstract state trace in the variant abstract state trace set of each variant is calculated by Jaccard similarity. The formula is: ， ， in, is the set of original abstract state traces of the original deep learning model calculated by Jaccard similarity With The set of variant abstract state traces The average distance between all pairs of abstract state traces in , is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The Jaccard similarity between is the original abstract state trace set of the original deep learning model Middle A trace of the original abstract state, For the The set of variant abstract state traces Middle A variation of the abstract state trace, express and The intersection size of express and The size of the union of is the number of test samples, is the original abstract state trace set of the original deep learning model Middle Original abstract state trace With The set of variant abstract state traces Middle Variation abstract state trace The distance between.

8. The mutation operator reduction method of a deep learning model according to claim 1, characterized in that: In the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

9. The mutation operator reduction method of a deep learning model according to claim 1, characterized in that: When the test sample is image data, the original deep learning model is any one of the image segmentation model, target detection model, and image dehazing model; When the test sample is text data, the original deep learning model is any one of a text classification model, a named entity recognition model, and a text matching model.

10. A mutation operator reduction system for a deep learning model, characterized in that: include: A mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator; The vector reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence; The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; An abstract state trace acquisition module is used to map each hidden state vector after reduction in each hidden state sequence to its corresponding abstract state for each test sample in the original deep learning model and each variant, so as to obtain the abstract state trace of each test sample in the original deep learning model and each variant; An abstract state trace set construction module is used to summarize the original abstract state traces of all test samples under the original deep learning model to obtain the original abstract state trace set of the original deep learning model, and summarize the variant abstract state traces of all test samples under each variant to obtain the variant abstract state trace set of each variant; An abstract state trace difference mean value calculation module is used to calculate the average value of the distance between each original abstract state trace in the original abstract state trace set of the original deep learning model and each variant abstract state trace in the variant abstract state trace set of each variant, so as to obtain the abstract state trace difference mean value between the original deep learning model and each variant; The mutation operator simplification module is used to determine the mutation operator to be simplified according to the size of the mean difference between the abstract state traces of the original deep learning model and each variant, and to simplify the mutation operator to be simplified.

Images