A method and system for detecting redundant variants of deep learning models

Abstract

The present invention relates to the field of software testing, and in particular to a method and system for detecting redundant variants of a deep learning model. The present invention obtains the hidden state sequence of each test sample in the original deep learning model and each variant, performs dimensionality reduction on each hidden state vector, clusters all the reduced hidden state vectors, regards each cluster cluster as an abstract state, and uses the transition sequence between the abstract states of the hidden state sequence of each test sample in each model as the abstract state trace of each test sample in the corresponding model; constructs the finite state machine of each model based on the abstract state trace of each test sample in the original deep learning model and each variant; and utilizes the equivalence and inclusion relationship between the finite state machine languages ​​of the original deep learning model and each variant to identify redundant variants. The present invention improves the effectiveness of redundant variant detection and ensures the accuracy of evaluating the quality of image or text test case sets.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a method and system for detecting redundant variants of a deep learning model. Background Art

[0002] In today's technology-driven industrial development, traditional program mutation testing measures the effectiveness of a test suite by introducing artificial errors into the original program being tested. These programs, which have been artificially errored, are called variants. Mutation testing is now increasingly widespread in the field of deep learning. Mutation testing of deep learning models generates mutated models (also called variants) by making subtle modifications (equivalent to introducing errors) to the model structure, parameters, and state. The difference in prediction results between the original deep learning model and the variant model is used to determine whether the test sample can detect the changes in the variant model, ultimately measuring the test sample's error detection capability. The large number of test samples and generated variants makes mutation testing prohibitively expensive, especially due to the presence of redundant variants. Redundant variants fall into two categories: equivalent variants and inclusion variants. Equivalent variants exhibit the same behavior (prediction results) as the original program (model) for all possible test cases (samples). Inclusion variants exhibit behavior that is included in the original program (model) for all possible test cases (samples). Furthermore, equivalent variants introduce significant bias in mutation-based analysis. Therefore, detecting and reducing such redundant variants can significantly reduce the amount of code executed for mutation testing and eliminate the bias they introduce.

[0003] For example, in medical imaging diagnosis, mutation testing and redundant variant detection techniques can verify the stability and accuracy of medical imaging diagnostic models. Because medical imaging diagnosis is crucial to patient health, the reliability of medical imaging diagnostic models is crucial. Mutation testing can promptly identify potential errors in the model when exposed to diverse imaging data, while addressing redundant variants optimizes the testing process, ensuring the efficiency and reliability of medical imaging diagnostic models in clinical applications. In intelligent customer service systems, natural language processing models are responsible for understanding user questions and generating appropriate responses. Mutation testing plays a crucial role in ensuring that intelligent customer service systems can accurately understand and respond to various user queries. By making subtle modifications to the structure and parameters of the natural language processing model, a variant model is generated. A large number of different user questions are used as test samples. The differences in responses between the original and variant models are used to determine whether the test samples can detect model changes. This serves to measure the error detection capability of user questions. By identifying these variants, the natural language processing model's ability to understand user questions is further improved.

[0004] To address the problem of redundant variant detection, researchers have explored a variety of approaches, including compiler technology, static analysis, dynamic analysis, machine learning, and large language models (LLMs). For example, Papadakis et al. proposed the Trivial Compiler Equivalence (TCE) technique, which leverages existing compiler technology to identify equivalent variants in programs. Researchers have also used mutation classification strategies to automatically isolate equivalent variants. Papadakis et al. experimentally evaluated potential mutation classification strategies, demonstrating that mutation classification can only partially improve low-quality test sets. Kintis et al. utilized a series of data flow patterns to automatically identify equivalent and partially equivalent variants in Java and JavaScript programs. Ghiduk et al. used second-order mutation testing and dynamic symbolic execution to automatically detect and reduce equivalent second-order variants. Naeem et al. considered three metrics based on constraint-based testing (CBT) theory to extract characteristic constraints for mutated programs. They used program dependency graphs and binary classification models to obtain these constraints and applied machine learning algorithms to predict equivalent variants in Java programs. Peacock et al. designed and trained an abstract syntax tree recursive neural network model based on machine learning to automatically classify equivalent variants during mutation testing. Tian et al. studied the effectiveness of large language models (LLMs) in detecting equivalent variants on method-level Java variant pairs.

[0005] However, existing methods for detecting redundant variants all have limitations. Static analysis methods cannot accurately capture the program's runtime state, resulting in numerous false positives and difficulty detecting semantically equivalent variants. Compiler-based and dynamic analysis methods can capture the program's runtime state, but the sheer number of program states and the sheer volume of test data make the number of state traces difficult to process, hindering the practical application of mutation testing. The challenges faced by deep learning models are particularly daunting. Taking recurrent neural networks (RNNs) as an example, the number and dimensions of their hidden states far exceed the state scale of general programs, making it a challenge to detect their equivalent variants. Although methods based on machine learning and large language models (LLMs) have certain potential, they have high requirements for a large number of training samples and manual labeling. In reality, the number of samples of equivalent variants is relatively small. Therefore, methods based on machine learning and large language models (LLMs) find it difficult to accurately and efficiently identify redundant variants, resulting in the inability to accurately evaluate the quality of image test case sets or text test case sets, resulting in poor accuracy in the image processing results of image deep learning models or poor accuracy in the text processing results of text deep learning models, and insufficient robustness. Summary of the Invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome certain limitations of existing redundant variant detection methods, which make it difficult to efficiently identify redundant variants, resulting in the inability to accurately judge the quality of image test case sets or text test case sets, causing the image deep learning model to have poor accuracy when processing images, and the text deep learning model to have insufficient accuracy when processing text, and the robustness of both is relatively weak.

[0007] To solve the above technical problems, the present invention provides a method for detecting redundant variants of a deep learning model, comprising the following steps:

[0008] Use each mutation operator to mutate the original deep learning model and generate a variant of the original deep learning model corresponding to each mutation operator;

[0009] Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and perform dimensionality reduction on each hidden state vector in each hidden state sequence;

[0010] Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and its variants, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state;

[0011] For each test sample's hidden state sequence in the original deep learning model and its variants, each simplified hidden state vector in each hidden state sequence is mapped to its corresponding abstract state, obtaining the abstract state trace of each test sample in the original deep learning model and its variants.

[0012] Based on the original abstract state traces of all test samples in the original deep learning model, a finite state machine of the original abstract state traces is constructed;

[0013] According to the mutation abstract state traces of all test samples in each variant, a finite state machine of the mutation abstract state traces of each variant is constructed;

[0014] Based on the abstract state traces of each test sample in each variant, the finite state machine of the original abstract state trace is used to identify redundant variants of the original deep learning model.

[0015] Based on the variation abstract state trace of each test sample in each variant except the current variant, the redundant variants of the current variant are identified using the finite state machine of the variation abstract state trace of the current variant, and the redundant variants of each variant are obtained.

[0016] Preferably, constructing a finite state machine of the original abstract state traces according to the original abstract state traces of all test samples in the original deep learning model includes:

[0017] All test samples in each abstract state of the original deep learning model are regarded as the abstract state set of the original deep learning model;

[0018] If the test sample is text data, the words in the test sample are used as the input character set of the original deep learning model; if the test sample is image data, the image features in the test sample are used as the input character set of the original deep learning model;

[0019] Count the first abstract state of all test samples in the original abstract state trace of the original deep learning model as the initial abstract state of the original deep learning model;

[0020] Count the last abstract state of all test samples in the original abstract state trace of the original deep learning model as the terminal abstract state of the original deep learning model;

[0021] The abstract state in the abstract state set of the original deep learning model , according to a character in the input character set, transfer to the next abstract state The mapping relationship is used as the abstract state transfer function of the original deep learning model;

[0022] Based on the abstract state set, input character set, initial abstract state, terminal abstract state, and abstract state transfer function of the original deep learning model, a finite state machine of the original abstract state trace is constructed.

[0023] Preferably, the method of identifying redundant variants of the original deep learning model based on the variant abstract state trace of each test sample in each variant and using a finite state machine of the original abstract state trace includes:

[0024] Generate the regular language of the current variant based on the abstract state traces of all test samples in the current variant;

[0025] If the acceptable language of the finite state machine of the original abstract state trace is equivalent to the regular language of the current variant, then the current variant is an equivalent variant of the original deep learning model;

[0026] If the acceptable language of the finite state machine of the original abstract state trace contains the regular language of the current variant, then the current variant is an included variant of the original deep learning model.

[0027] Preferably, the method of identifying redundant variants of the current variant based on the variant abstract state trace of each test sample in variants other than the current variant and utilizing a finite state machine of the variant abstract state trace of the current variant to obtain redundant variants of each variant includes:

[0028] Based on all test samples in The abstract state trace of the variants generates the A regular language with 1 variant;

[0029] Jordi The acceptable language of the variant finite state machine of the first variant is the same as that of the The regular language of variants is an equivalence relation, then The variant is Equivalent variants of a variant;

[0030] Jordi The acceptable language of the variant finite state machine of the variant contains The regular language of variants, then The variant is variants comprising variants; among them, , is the number of variants.

[0031] Preferably, in the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

[0032] Preferably, the method for dimensionality reduction of each hidden state vector in the current hidden state sequence is any one of principal component analysis, correlation coefficient matrix method, and linear discriminant analysis.

[0033] Preferably, principal component analysis is used to perform dimensionality reduction on each hidden state vector in the current hidden state sequence to obtain each reduced hidden state vector in the current hidden state sequence, including:

[0034] For each hidden state vector in the current hidden state sequence, calculate the average value of the corresponding elements of each dimension;

[0035] For each element of each dimension in each hidden state vector, subtract the average value of the elements corresponding to that dimension to obtain each hidden state vector after removing the mean;

[0036] Calculate the covariance matrix of the current hidden state sequence based on the hidden state vectors after removing the mean in the current hidden state sequence;

[0037] By performing eigendecomposition on the covariance matrix of the current hidden state sequence, multiple eigenvalues ​​corresponding to the current hidden state sequence and the eigenvector corresponding to each eigenvalue are obtained;

[0038] According to the size of the eigenvalue corresponding to the current hidden state sequence, the corresponding eigenvectors are sorted, and the hidden state vectors after removing the mean in the current hidden state sequence are projected onto the eigenvector corresponding to the maximum eigenvalue to obtain the simplified hidden state vectors in the current hidden state sequence.

[0039] Preferably, the method for clustering the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant is any one of Gaussian mixture model, hierarchical clustering, K-means clustering, DBSCAN, and spectral clustering.

[0040] Preferably, clustering the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant is performed using a Gaussian mixture model, including:

[0041] According to the set number of clusters, the mean and variance are randomly assigned to the same number of Gaussian distributions. Based on the mean and variance of each Gaussian distribution, the probability density function of the Gaussian distribution is used to calculate the probability of each hidden state vector after reduction under each Gaussian distribution.

[0042] Based on the probability of each simplified hidden state vector under each Gaussian distribution, the mean and variance of each Gaussian distribution are updated until the mean and variance of each Gaussian distribution are in a convergent state. According to the probability of each simplified hidden state vector under each Gaussian distribution in a convergent state, the category of each simplified hidden state vector is divided.

[0043] Preferably, when the test sample is image data, the original deep learning model is any one of an image segmentation model, an object detection model, and an image defogging model;

[0044] When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model.

[0045] The present invention also provides a redundant variant detection system for a deep learning model, comprising:

[0046] The mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator;

[0047] The reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence;

[0048] The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and its variants, treating each cluster as an abstract state and mapping the simplified hidden state vectors to the corresponding abstract state;

[0049] An abstract state trace acquisition module is used to map each hidden state vector in each hidden state sequence of each test sample in the original deep learning model and each variant to its corresponding abstract state, thereby obtaining the abstract state trace of each test sample in the original deep learning model and each variant;

[0050] A first finite state machine construction module is used to construct a finite state machine of the original abstract state trace according to the original abstract state trace of all test samples in the original deep learning model;

[0051] The second finite state machine construction module is used to construct a finite state machine of the abstract state trace of each variant according to the abstract state trace of each variant of all test samples;

[0052] The original model redundant variant detection module is used to identify redundant variants of the original deep learning model based on the abstract state trace of each test sample in each variant and the finite state machine of the original abstract state trace;

[0053] The variant redundant variant detection module is used to identify redundant variants of the current variant based on the variant abstract state traces of each test sample in each variant except the current variant, using the finite state machine of the variant abstract state trace of the current variant, and obtain the redundant variants of each variant.

[0054] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0055] The redundant variant detection method and system of a deep learning model described in the present invention breaks through the previous limitations of only focusing on the surface output or simple internal features of the deep learning model. Starting from the core of the dynamic changes within the model, by obtaining the hidden state sequence of each test sample in the original deep learning model and the training process of each variant, it can reflect the semantic changes of the model when processing data in real time, greatly improving the accuracy of obtaining the program running state. However, the hidden state sequence is highly dynamic and continuous, and is always in complex changes, making it difficult to grasp its laws. In order to overcome the continuous and dynamic characteristics of the hidden state sequence, the present invention converts the hidden state sequence into a discrete, abstract state with clear characteristics, and converts the originally disordered state information into a finite number of identifiable and distinguishable state categories; and considering the problems of high dimensionality and large data volume of the hidden state sequence, by simplifying the dimension of each hidden state vector, the complexity of data processing is reduced, and redundant information is accurately removed. By clustering all the simplified hidden state vectors, each cluster cluster is regarded as an abstract state, and the transition sequence between abstract states is used as the abstract state trace of each test sample in the corresponding model. The state representation is simplified in one step. These abstract states are then used as the basic elements of a finite state machine, and the transition logic between abstract states is mapped to the state transition rules of the finite state machine. This creates a structured state expression system that can be understood by the finite state machine. This system incorporates semantic information (i.e., state information) and utilizes the equivalence and inclusion relationships between finite state machine languages ​​to clearly analyze and judge redundant variants of deep learning models, overcoming the undecidability of detecting redundant variants in deep learning models. Furthermore, the present invention does not require reliance on a large number of externally labeled samples, and thus can effectively detect redundant variants even when the number of equivalent variant samples is relatively small in reality, reducing reliance on large-scale samples and manual labeling. By effectively detecting redundant variants in the original deep learning model, the accuracy of evaluating the effectiveness of test samples is improved, the error rate of image or text deep learning models is reduced, and the precision and robustness of image or text deep learning models are improved, making image recognition clearer and more accurate, and text processing more semantically accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 This is a flowchart of the steps of a redundant variant detection method for a deep learning model. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0059] Reference Figure 1 As shown, this embodiment provides a method for detecting redundant variants of a deep learning model, including the following steps:

[0060] Step S1: Use each mutation operator to mutate the original deep learning model to generate a variant of the original deep learning model corresponding to each mutation operator;

[0061] In this embodiment, specifically, during the training process of the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

[0062] In this embodiment, specifically, when the test sample is image data, the original deep learning model is any one of an image segmentation model, an object detection model, and an image defogging model;

[0063] If the original deep learning model is an image segmentation model, the original deep learning model is any one of a U-Net model, a Mask R-CNN model, a SegNet model, a recurrent neural network (RNN), and a GRU model;

[0064] If the original deep learning model is an object detection model, the original deep learning model is any one of the Faster R-CNN model and the SSD (Single Shot MultiBox Detector) model;

[0065] If the original deep learning model is an image dehazing model, the original deep learning model is any one of the AOD-Net (All-in-One Dehazing Network) and MSCNN (Multi-Scale Convolutional Neural Network) models;

[0066] When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model;

[0067] If the original deep learning model is a text classification model, then the original deep learning model is either a TextCNN model or an LSTM-based Classifier model;

[0068] If the original deep learning model is a named entity recognition model, it can be either an LSTM-CRF (Long Short-Term Memory with Conditional Random Field) model or a BERT-LSTM-CRF model.

[0069] If the original deep learning model is a text matching model, it can be either the Deep Structured Semantic Model (DSSM) or the Enhanced LSTM for Natural Language Inference (ESIM) model.

[0070] Step S2: Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant , for each hidden state vector in each hidden state sequence Perform dimensionality reduction;

[0071] in, For the Test samples in the model The hidden state sequence during the training process of , whose dimensions are , for each hidden state vector Perform dimensionality reduction, and the dimension after simplification is The hidden state vector , is the total number of time steps, For the Test samples in the model During the training process The hidden state vector corresponding to the time step, For the Test samples in the model During the training process The reduced hidden state vector corresponding to the time steps, and .

[0072] In this embodiment, specifically, the method for performing dimensionality reduction on each hidden state vector in the current hidden state sequence is any one of principal component analysis, correlation coefficient matrix method, and linear discriminant analysis.

[0073] In this embodiment, preferably, principal component analysis is used to perform dimensionality reduction on each hidden state vector in the current hidden state sequence to obtain each reduced hidden state vector in the current hidden state sequence, including:

[0074] For each hidden state vector in the current hidden state sequence, calculate the average value of the corresponding elements of each dimension;

[0075] For each element of each dimension in each hidden state vector, subtract the average value of the elements corresponding to that dimension to obtain each hidden state vector after removing the mean;

[0076] Calculate the covariance matrix of the current hidden state sequence based on the hidden state vectors after removing the mean in the current hidden state sequence;

[0077] By performing eigendecomposition on the covariance matrix of the current hidden state sequence, multiple eigenvalues ​​corresponding to the current hidden state sequence and the eigenvector corresponding to each eigenvalue are obtained;

[0078] According to the size of the eigenvalue corresponding to the current hidden state sequence, the corresponding eigenvectors are sorted, and the hidden state vectors after removing the mean in the current hidden state sequence are projected onto the eigenvector corresponding to the maximum eigenvalue to obtain the simplified hidden state vectors in the current hidden state sequence.

[0079] In this embodiment, principal component analysis is used to Test samples in the model The hidden state vectors in the hidden state sequence during the training process are dimensionality reduced to obtain the first Test samples in the model The hidden state vectors after reduction in the hidden state sequence during the training process include:

[0080] For the Test samples in the model For each hidden state vector in the hidden state sequence during the training process, calculate the average value of the elements corresponding to each dimension as the average value corresponding to each dimension. The formula is:

[0081] ,

[0082] in, For the Test samples in the model In the hidden state sequence during the training process, the first hidden state vector The average value of the elements corresponding to the dimensions, For the Test samples in the model No. The hidden state vector corresponding to the time step Dimensions correspond to elements, is the dimension index, .

[0083] Subtract the average value of each dimension from the elements of each hidden state vector to obtain the hidden state vector after removing the mean. The formula is:

[0084] ,

[0085] in, For the Test samples in the model During the training process The hidden state vector after removing the mean corresponding to the time step Dimensions correspond to elements.

[0086] Based on the Test samples in the model The hidden state vectors after removing the mean in the hidden state sequence during the training process are calculated. Test samples in the model The corresponding covariance matrix;

[0087] ,

[0088] in, For the Test samples in the model The corresponding covariance matrix, For the Test samples in the model During the training process The hidden state vector after removing the mean corresponding to the time step, is transposed.

[0089] Through the Test samples in the model The corresponding covariance matrix is ​​decomposed to obtain the Test samples in the model Corresponding multiple eigenvalues The eigenvector corresponding to each eigenvalue ;

[0090] According to Test samples in the model The size of the corresponding eigenvalue is sorted, and the corresponding eigenvectors are sorted. Test samples in the model In the hidden state sequence during the training process, each hidden state vector after removing the mean is projected onto the eigenvector corresponding to the maximum eigenvalue On, get Test samples in the model The formula for each hidden state vector after reduction in the hidden state sequence during the training process is:

[0091] ,

[0092] in, For the Test samples in the model During the training process The reduced hidden state vector corresponding to the time step is For the Test samples in the model The eigenvector corresponding to the maximum eigenvalue of The dimension is .

[0093] Principal Component Analysis (PCA) is an unsupervised machine learning algorithm used to explore high-dimensional data structures. It is primarily used for data dimensionality reduction, which can help identify more understandable features and accelerate the processing of valuable sample information. It can also be applied to visualization (reducing to two dimensions) and denoising.

[0094] By calculating the average value of the elements corresponding to each dimension of each hidden state vector in the current hidden state sequence, removing the mean of the elements in each dimension of each hidden state vector, and then calculating the covariance matrix and performing eigendecomposition, the originally complex high-dimensional data is transformed into a more manageable form, greatly reducing the amount of subsequent calculations and improving computational efficiency. Furthermore, principal component analysis can effectively preserve key information, sort eigenvectors by eigenvalue, and project the data onto the eigenvector with the largest eigenvalue, ensuring that the simplified data captures the main features of the original data. This provides a reliable foundation for subsequent operations such as redundant variant detection based on this data, thereby improving detection accuracy. This not only helps enhance model stability but also reduces the impact of noise and interference information in the original hidden state sequence on data analysis and judgment, avoiding misjudgments caused by noise, and improving the accuracy and efficiency of redundant variant detection.

[0095] Step S3: Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant to obtain Clusters , regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state;

[0096] In this embodiment, specifically, the method for clustering the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant is any one of Gaussian mixture model, hierarchical clustering, K-means clustering, DBSCAN, and spectral clustering.

[0097] In this embodiment, preferably, the hidden state vectors of all test samples after simplification during the training of the original deep learning model and each variant are clustered using a Gaussian mixture model, including:

[0098] According to the set number of clusters , randomly assign mean and variance to the same number of Gaussian distributions, and based on the mean and variance of each Gaussian distribution, calculate the probability of each reduced hidden state vector under each Gaussian distribution through the probability density function of the Gaussian distribution;

[0099] The formula for calculating the probability of each hidden state vector after reduction under each Gaussian distribution based on the mean and variance of each Gaussian distribution and the probability density function of the Gaussian distribution is:

[0100] ,

[0101] in, For the Test samples in the model During the training process The reduced hidden state vector corresponding to the time step is in The probability under a Gaussian distribution, is the Gaussian distribution index, For the The mean of a Gaussian distribution, For the The variance of a Gaussian distribution, is an exponential function with a natural constant as its base.

[0102] Based on the probability of each simplified hidden state vector under each Gaussian distribution, the mean and variance of each Gaussian distribution are updated until the mean and variance of each Gaussian distribution are in a convergent state. According to the probability of each simplified hidden state vector under each Gaussian distribution in a convergent state, the category of each simplified hidden state vector is divided.

[0103] Gaussian mixture models can effectively address the complexity of hidden state vector distributions. Hidden state vectors after simplification by deep learning models may exhibit complex distribution patterns. However, Gaussian mixture models, composed of multiple Gaussian distributions, can flexibly fit various complex data distributions and accurately group hidden state vectors with similar characteristics into the same cluster. Compared to some simple clustering methods, this can more accurately capture the inherent structure of the data and adaptively adjust the boundaries and characteristics of clusters. This adaptive capability enables the Gaussian mixture model to achieve relatively ideal clustering results when processing hidden state vectors with different characteristics, effectively improving the accuracy and effectiveness of the entire redundant variant detection process.

[0104] Step S4: For each hidden state sequence of each test sample in the original deep learning model and each variant, the simplified hidden state vectors in each hidden state sequence are are mapped to their corresponding abstract states , , get the abstract state trace of each test sample in the original deep learning model and each variant ;in, For the Test samples in the model The abstract state trace of For the Test samples in the model During the training process The abstract state corresponding to the simplified hidden state vector corresponding to the time step;

[0105] Step S5: constructing a finite state machine of the original abstract state traces according to the original abstract state traces of all test samples in the original deep learning model;

[0106] A finite state machine (FSM) is a mathematical model that represents a finite number of states and the transitions and actions between these states. , where: Σ is the input character set (finite, non-empty), S is the state set (finite, non-empty), S0 is the initial state, belonging to the set S, is the state transition function, , that is, the state in the state set S , when receiving a character from the input character set Σ as input, the state transition function Depending on the status and input characters to determine the next state to which the finite state machine will transfer , F is the terminal state.

[0107] In this embodiment, specifically, a finite state machine of the original abstract state trace is constructed based on the original abstract state trace of all test samples in the original deep learning model, including:

[0108] All test samples in each abstract state of the original deep learning model are regarded as the abstract state set of the original deep learning model;

[0109] If the test sample is text data, the words in the test sample are used as the input character set of the original deep learning model; if the test sample is image data, the image features in the test sample are used as the input character set of the original deep learning model;

[0110] Count the first abstract state of all test samples in the original abstract state trace of the original deep learning model as the initial abstract state of the original deep learning model;

[0111] Count the last abstract state of all test samples in the original abstract state trace of the original deep learning model as the terminal abstract state of the original deep learning model;

[0112] The abstract state in the abstract state set of the original deep learning model , according to a character in the input character set, transfer to the next abstract state The mapping relationship is used as the abstract state transfer function of the original deep learning model;

[0113] Based on the abstract state set, input character set, initial abstract state, terminal abstract state, and abstract state transfer function of the original deep learning model, a finite state machine of the original abstract state trace is constructed.

[0114] in, , S is the abstract state set of the original deep learning model.

[0115] Step S6: constructing a finite state machine of the abstract state trace of each variant according to the abstract state trace of each variant of all test samples;

[0116] In this embodiment, specifically, a finite state machine of the abstract state trace of the current variant is constructed based on the abstract state trace of the variant of all test samples in the current variant, including:

[0117] All test samples in each abstract state of the current variant are regarded as the abstract state set of the current variant;

[0118] If the test sample is text data, the words in the test sample are used as the input character set of the current variant; if the test sample is image data, the image features in the test sample are used as the input character set of the current variant;

[0119] Count the first abstract state of all test samples in the variant abstract state trace of the current variant as the initial abstract state of the current variant;

[0120] Count the last abstract state of all test samples in the mutation abstract state trace of the current variant as the terminal abstract state of the current variant;

[0121] The abstract state in the abstract state set of the current variant , according to a character in the input character set, transfer to the next abstract state The mapping relationship is used as the abstract state transfer function of the current variant;

[0122] in, , A is the abstract state set of the current variant.

[0123] The language that a finite state machine (FSM) can accept is the set of all input character sequences that can, during the state transition process of the finite state machine, start from the initial state, follow the rules of the state transition function, and finally reach the terminal state through the guidance of a series of input characters.

[0124] The language that a finite state machine can process is a sequence of strings. If the finite state machine reaches the terminal state after reading a string, then the string is a language that the finite state machine can accept. , if the language set L(M) consisting of all string sequences that the finite state machine M can accept is the same as the regular language The set of all string sequences contained Equal, that is , then the language acceptable to the finite state machine is equivalent to the regular language. Similarly, if the language set L(M) consisting of all string sequences that the finite state machine M can accept contains the regular language The set of all string sequences contained, that is , then the language L(M) acceptable to the finite state machine contains the regular language , which belongs to the inclusion relationship.

[0125] Step S7: Based on the variant abstract state traces of each test sample in each variant, the redundant variants of the original deep learning model are identified using the finite state machine of the original abstract state traces;

[0126] In this embodiment, specifically, the method of identifying redundant variants of the original deep learning model based on the variant abstract state trace of each test sample in each variant and using a finite state machine of the original abstract state trace includes:

[0127] Generate the regular language of the current variant based on the abstract state traces of all test samples in the current variant;

[0128] If the acceptable language of the finite state machine of the original abstract state trace is equivalent to the regular language of the current variant, then the current variant is an equivalent variant of the original deep learning model;

[0129] If the acceptable language of the finite state machine of the original abstract state trace contains the regular language of the current variant, then the current variant is an included variant of the original deep learning model.

[0130] Step S8: Based on the variant abstract state traces of each test sample in each variant except the current variant, the redundant variants of the current variant are identified using the finite state machine of the variant abstract state trace of the current variant to obtain the redundant variants of each variant.

[0131] In this embodiment, specifically, the method of identifying redundant variants of the current variant based on the variant abstract state trace of each test sample in each variant other than the current variant and using a finite state machine of the variant abstract state trace of the current variant to obtain redundant variants of each variant includes:

[0132] Based on all test samples in The abstract state trace of the variants generates the A regular language with 1 variant;

[0133] Jordi The acceptable language of the variant finite state machine of the first variant is the same as that of the The regular language of variants is an equivalence relation, then The variant is Equivalent variants of a variant;

[0134] Jordi The acceptable language of the variant finite state machine of the variant contains The regular language of variants, then The variant is variants comprising variants; among them, , is the number of variants.

[0135] In the field of deep learning, traditional methods are mostly limited to focusing on the surface outputs or simple internal features of deep learning models, but fail to gain a deeper understanding of the complex operating mechanisms within the models. This invention, however, focuses on the core of the dynamic changes within the model, obtaining the hidden state sequence of each test sample during the training process of the original deep learning model and its variants, reflecting the dynamic changes of the model as it processes data in real time.

[0136] However, the hidden state sequence is highly dynamic and continuous, and is always in complex changes, making it difficult to grasp its laws. In order to overcome the continuous and dynamic characteristics of the hidden state sequence, the present invention converts the hidden state sequence into discrete abstract states with clear characteristics, and converts the originally disordered state information into a finite number of identifiable and distinguishable state categories; considering the problems of high state dimension and large data volume of the hidden state sequence, the hidden state sequence has high state dimension and large data volume, by simplifying the dimension of each hidden state vector in the hidden state sequence, the complexity of data processing is reduced, and redundant information is accurately removed. By clustering all the simplified hidden state vectors, each cluster cluster is regarded as an abstract state, and the transition sequence between abstract states is used as an abstract state trace, which further simplifies the state representation.

[0137] These abstract states are then used as the basic elements of the finite state machine, and the conversion logic between abstract states is mapped to the state transfer rules of the finite state machine, thus becoming a structured state expression system that can be understood by the finite state machine. By incorporating semantic information (i.e., state information) and utilizing the equivalence and inclusion relationships between finite state machine languages, it is possible to clearly analyze and judge the redundant variants of the deep learning model, overcoming the undecidability of the problem of detecting redundant variants of the deep learning model.

[0138] At the same time, the present invention does not need to rely on a large number of external labeled samples, so that even when the number of equivalent variant samples is relatively small in reality, it can still effectively detect redundant variants, reducing the dependence on large-scale samples and manual labeling, and overcoming the problem of a relatively small number of equivalent variant samples.

[0139] By effectively detecting redundant variants of the original deep learning model, the present invention improves the accuracy of evaluating the effectiveness of test samples, reduces the misjudgment rate of the image or text deep learning model, and improves the precision and robustness of the image or text deep learning model, making image recognition clearer and more accurate, and text processing more semantically consistent.

[0140] This second embodiment provides a redundant variant detection system for a deep learning model, including:

[0141] The mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator;

[0142] The reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence;

[0143] The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and its variants, treating each cluster as an abstract state and mapping the simplified hidden state vectors to the corresponding abstract state;

[0144] An abstract state trace acquisition module is used to map each hidden state vector in each hidden state sequence of each test sample in the original deep learning model and each variant to its corresponding abstract state, thereby obtaining the abstract state trace of each test sample in the original deep learning model and each variant;

[0145] A first finite state machine construction module is used to construct a finite state machine of the original abstract state trace according to the original abstract state trace of all test samples in the original deep learning model;

[0146] The second finite state machine construction module is used to construct a finite state machine of the abstract state trace of each variant according to the abstract state trace of each variant of all test samples;

[0147] The original model redundant variant detection module is used to identify redundant variants of the original deep learning model based on the abstract state trace of each test sample in each variant and the finite state machine of the original abstract state trace;

[0148] The variant redundant variant detection module is used to identify redundant variants of the current variant based on the variant abstract state traces of each test sample in each variant except the current variant, using the finite state machine of the variant abstract state trace of the current variant, and obtain the redundant variants of each variant.

[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0153] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for detecting redundant variants of a deep learning model, characterized in that: The following steps are involved: Use each mutation operator to mutate the original deep learning model and generate a variant of the original deep learning model corresponding to each mutation operator; Obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and perform dimensionality reduction on each hidden state vector in each hidden state sequence; Cluster the hidden state vectors of all test samples after simplification during the training of the original deep learning model and its variants, regard each cluster as an abstract state, and map the simplified hidden state vectors to the corresponding abstract state; For each test sample's hidden state sequence in the original deep learning model and its variants, each simplified hidden state vector in each hidden state sequence is mapped to its corresponding abstract state, obtaining the abstract state trace of each test sample in the original deep learning model and its variants. Based on the original abstract state traces of all test samples in the original deep learning model, a finite state machine of the original abstract state traces is constructed; According to the mutation abstract state traces of all test samples in each variant, a finite state machine of the mutation abstract state traces of each variant is constructed; Based on the abstract state trace of each test sample in each variant, the finite state machine of the original abstract state trace is used to identify redundant variants of the original deep learning model, including: Generate the regular language of the current variant based on the abstract state traces of all test samples in the current variant; If the acceptable language of the finite state machine of the original abstract state trace is equivalent to the regular language of the current variant, then the current variant is an equivalent variant of the original deep learning model; If the acceptable language of the finite state machine of the original abstract state trace contains the regular language of the current variant, then the current variant is the included variant of the original deep learning model; Based on the abstract state traces of each test sample in each variant except the current variant, the redundant variants of the current variant are identified using the finite state machine of the abstract state trace of the current variant, and the redundant variants of each variant are obtained, including: Based on all test samples in The abstract state trace of the variants generates the A regular language with 1 variant; Jordi The acceptable language of the variant finite state machine of the first variant is the same as that of the The regular language of variants is an equivalence relation, then The variant is Equivalent variants of a variant; Jordi The acceptable language of the variant finite state machine of the variant contains The regular language of variants, then The variant is variants comprising variants; among them, , , , is the number of variants; When the test sample is image data, the original deep learning model is any one of the image segmentation model, target detection model, and image dehazing model; When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model.

2. The method for detecting redundant variants of a deep learning model according to claim 1, wherein: The method of constructing a finite state machine of the original abstract state traces of all test samples in the original deep learning model includes: All test samples in each abstract state of the original deep learning model are regarded as the abstract state set of the original deep learning model; If the test sample is text data, the words in the test sample are used as the input character set of the original deep learning model; if the test sample is image data, the image features in the test sample are used as the input character set of the original deep learning model; Count the first abstract state of all test samples in the original abstract state trace of the original deep learning model as the initial abstract state of the original deep learning model; Count the last abstract state of all test samples in the original abstract state trace of the original deep learning model as the terminal abstract state of the original deep learning model; The abstract state in the abstract state set of the original deep learning model , according to a character in the input character set, transfer to the next abstract state The mapping relationship is used as the abstract state transfer function of the original deep learning model; Based on the abstract state set, input character set, initial abstract state, terminal abstract state, and abstract state transfer function of the original deep learning model, a finite state machine of the original abstract state trace is constructed.

3. The method for detecting redundant variants of a deep learning model according to claim 1, wherein: In the process of training the original deep learning model and each variant using the test sample, by setting the parameters return_sequences=True and return_state=True, the hidden state sequence of each test sample in the original deep learning model and each variant during the training process is obtained.

4. The method for detecting redundant variants of a deep learning model according to claim 1, wherein: The method for dimensionality reduction of each hidden state vector in the current hidden state sequence is any one of principal component analysis, correlation coefficient matrix method, and linear discriminant analysis.

5. The method for detecting redundant variants of a deep learning model according to claim 1, wherein: The principal component analysis is used to reduce the dimensions of each hidden state vector in the current hidden state sequence to obtain the reduced hidden state vectors in the current hidden state sequence, including: For each hidden state vector in the current hidden state sequence, calculate the average value of the corresponding elements of each dimension; For each element of each dimension in each hidden state vector, subtract the average value of the elements corresponding to that dimension to obtain each hidden state vector after removing the mean; Calculate the covariance matrix of the current hidden state sequence based on the hidden state vectors after removing the mean in the current hidden state sequence; By performing eigendecomposition on the covariance matrix of the current hidden state sequence, multiple eigenvalues ​​corresponding to the current hidden state sequence and the eigenvector corresponding to each eigenvalue are obtained; According to the size of the eigenvalue corresponding to the current hidden state sequence, the corresponding eigenvectors are sorted, and the hidden state vectors after removing the mean in the current hidden state sequence are projected onto the eigenvector corresponding to the maximum eigenvalue to obtain the simplified hidden state vectors in the current hidden state sequence.

6. The method for detecting redundant variants of a deep learning model according to claim 1, wherein: The method for clustering the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and each variant is any one of Gaussian mixture model, hierarchical clustering, K-means clustering, DBSCAN, and spectral clustering.

7. The method for detecting redundant variants of a deep learning model according to claim 1, wherein: The Gaussian mixture model is used to cluster the hidden state vectors of all test samples after being simplified during the training of the original deep learning model and each variant, including: According to the set number of clusters, the mean and variance are randomly assigned to the same number of Gaussian distributions. Based on the mean and variance of each Gaussian distribution, the probability density function of the Gaussian distribution is used to calculate the probability of each hidden state vector after reduction under each Gaussian distribution. Based on the probability of each simplified hidden state vector under each Gaussian distribution, the mean and variance of each Gaussian distribution are updated until the mean and variance of each Gaussian distribution are in a convergent state. According to the probability of each simplified hidden state vector under each Gaussian distribution in a convergent state, the category of each simplified hidden state vector is divided.

8. A redundant variant detection system for a deep learning model, characterized in that: include: The mutation module is used to mutate the original deep learning model using each mutation operator to generate a variant of the original deep learning model corresponding to each mutation operator; The reduction module is used to obtain the hidden state sequence of each test sample during the training process of the original deep learning model and each variant, and to reduce the dimension of each hidden state vector in each hidden state sequence; The clustering module is used to cluster the hidden state vectors of all test samples after simplification during the training process of the original deep learning model and its variants, treating each cluster as an abstract state and mapping the simplified hidden state vectors to the corresponding abstract state; An abstract state trace acquisition module is used to map each hidden state vector in each hidden state sequence of each test sample in the original deep learning model and each variant to its corresponding abstract state, thereby obtaining the abstract state trace of each test sample in the original deep learning model and each variant; A first finite state machine construction module is used to construct a finite state machine of the original abstract state trace according to the original abstract state trace of all test samples in the original deep learning model; The second finite state machine construction module is used to construct a finite state machine of the abstract state trace of each variant according to the abstract state trace of each variant of all test samples; The original model redundant variant detection module is used to identify redundant variants of the original deep learning model based on the abstract state trace of each test sample in each variant and the finite state machine of the original abstract state trace, including: Generate the regular language of the current variant based on the abstract state traces of all test samples in the current variant; If the acceptable language of the finite state machine of the original abstract state trace is equivalent to the regular language of the current variant, then the current variant is an equivalent variant of the original deep learning model; If the acceptable language of the finite state machine of the original abstract state trace contains the regular language of the current variant, then the current variant is the included variant of the original deep learning model; The variant redundant variant detection module is used to identify redundant variants of the current variant based on the variant abstract state traces of each test sample in each variant other than the current variant, using the finite state machine of the variant abstract state trace of the current variant, and obtain the redundant variants of each variant, including: Based on all test samples in The abstract state trace of the variants generates the A regular language with 1 variant; Jordi The acceptable language of the variant finite state machine of the first variant is the same as that of the The regular language of variants is an equivalence relation, then The variant is Equivalent variants of a variant; Jordi The acceptable language of the variant finite state machine of the variant contains The regular language of variants, then The variant is variants comprising variants; among them, , , , is the number of variants; When the test sample is image data, the original deep learning model is any one of the image segmentation model, target detection model, and image dehazing model; When the test sample is text data, the original deep learning model is any one of the text classification model, named entity recognition model, and text matching model.

Images