Automatic data defect detection method for deep learning classification task

Abstract

The invention relates to an automatic data defect detection method for a deep learning classification task, and belongs to the field of artificial intelligence. According to the method, a trained deep neural network (DNN) model is used as an original model, and a plurality of variation models are generated. Then, data features are extracted from output of the variation model, the data features are mapped into a credibility score, and the score indicates the probability that data input into the model do not contain data defects and reflects the degree that the data is beneficial to training an accurate classification model; and finally, sorting the data in the training data set based on the credibility score, setting a threshold value, and determining that the data is defective data when the data is lower than the threshold value. According to the method, a tester can perform emphasized processing on the detected defect data, and a high-quality training data set can be efficiently obtained without performing traversal inspection on the whole data set.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to an automated data defect detection method for deep learning classification tasks. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning models are becoming increasingly complex and rely more heavily on large amounts of data for training and judgment to achieve key tasks such as target recognition and battlefield situational awareness. However, data collected in real-world scenarios often suffers from labeling errors and data noise, resulting in lower accuracy for deep learning models trained on such datasets. This directly impacts the reliability and effectiveness of the intelligent system's output. Therefore, improving the quality of deep neural network (DNN) models is crucial for reducing misjudgments in intelligent systems and ensuring they make rapid and correct decisions in complex and ever-changing environments. Several methods have been proposed to improve DNN model quality, with two main approaches: first, adjusting the weights of the neural network, i.e., locating and modifying defective neurons in the DNN to improve model accuracy; and second, improving the quality of training data. Data quality can be evaluated using reliability metrics, which are formed through the combined effects of ensuring the reliability of data sources, the standardization of data collection and processing, and the security and integrity of the data. These methods collectively guarantee the accuracy of DNN models, providing accurate and reliable information support for model decision-making.

[0003] The main purpose of current DNN weight adjustment methods is to discover neuronal defects in trained models. This is because the prediction results of a neural network are related to the neurons in the intermediate layers. Once the model is trained and deployed to an application scenario, its internal weights are frozen. Therefore, for a trained deep learning model, potential defects may be hidden in the training optimization parameters of neurons. Existing weight adjustment techniques include neuron abnormal behavior detection, weight modification of abnormal neurons, model retraining, and model fine-tuning. However, the fundamental reason affecting the accuracy of DNN models is data defects in the training dataset, namely, label errors, noise, and other problems. These data defects propagate through the model's internal parameters, thus affecting the performance of the DNN model. Therefore, only by detecting and removing defective data and improving data quality can the model quality be improved at its root. Data that is conducive to training an accurate classification model is called reliable data. Several methods for measuring data reliability have been proposed, using manual review of the inherent characteristics of the dataset to derive reliability scores, such as dataset content accuracy and dataset standard accuracy. However, for manually reviewed datasets, without a standard reference dataset, data defects are sparse, and finding these defective samples is time-consuming when the dataset has a large number of samples. Furthermore, due to differences in data collection methods and the habits of data labelers, the distribution of data defects varies across different datasets, which can lead to significant manpower and time expenditure in detecting data defects. In summary, there is currently a lack of a universal credibility assessment method based on data defect characteristics that can adaptively detect data defective samples. Summary of the Invention

[0004] (a) Technical problems to be solved The technical problem to be solved by this invention is how to provide an automated data defect detection method for deep learning classification tasks, so as to solve the problem of adaptively detecting data defect samples.

[0005] (II) Technical Solution To address the aforementioned technical problems, this invention proposes an automated data defect detection method for deep learning classification tasks, comprising the following steps: Step 1: Training the original model Before generating a mutation model, the dataset needs to be fully trained to build a suitable DNN model, thereby obtaining a reliable original model. Step 2: Model Variation After obtaining the original model, different model mutation strategies are used to generate mutated models. The mutated model is generated by first obtaining a new training set based on the original training set, and then using this new training set to fine-tune the original model. The new training set is obtained by filtering the original dataset through specific criteria. Three model mutation strategies are used, namely, mutation at the input layer, intermediate layer and output layer, to remove outliers in the input data, select neurons with abnormal activation states and exclude data with large prediction losses. Step 3: Feature Extraction Methods for the Model After generating the mutation model, two types of key features were extracted for each model: one is the output of the DNN model, and the other is the loss value predicted by the model. Step 4: Data Reliability Calculation An effective credibility prediction model is established, implemented using a neural network model. The model is denoted as Cred, which is a fully connected binary classification neural network model. The input to the Cred model is... , representing a data sample extracted from the dataset. Feature set, using This represents the computational operations of each layer in a neural network model. Let the model contain a total of... s There are 1 fully connected layer, and the weights corresponding to each variable are 1. The model can then be represented as a list of combinations of fully connected layers:

[0006] Among them, feature set It includes the two types of features mentioned in step three: one is the model output, that is, the probability distribution of the input sample belonging to each category; the other is the loss value of the mutation model. Step 5: Defect Data Selection The updated Cred model is used to calculate the confidence score of the original training dataset samples. The samples are then ranked based on the confidence score, and samples with scores below a threshold are identified as defective data. These low-scoring samples result in poor model performance.

[0007] (III) Beneficial Effects This invention proposes an automated data defect detection method for deep learning classification tasks, used to detect low-quality data samples that negatively impact model accuracy during DNN training due to data defects. The proposed data defect detection method borrows from traditional code error localization methods based on model mutation, leveraging the differences between mutated models to amplify and identify potential data errors. First, using the trained DNN model as the original model, multiple mutated models are generated. Then, data features are extracted from the output of the mutated models and mapped to a confidence score. This score indicates the probability that the input data does not contain data defects, reflecting the degree to which the data is conducive to training an accurate classification model. Finally, defective data in the training dataset can be ranked based on confidence scores, and a threshold is set; data below this threshold is considered defective. This allows researchers to check and correct data based on the ranking results without reviewing the entire dataset.

[0008] The data defect detection method proposed in this invention allows testers to focus on processing detected defective data without having to traverse the entire dataset, thus efficiently obtaining a high-quality training dataset. Retraining the model using the processed training dataset effectively improves the classification accuracy of deep learning, further enhancing the performance of the intelligent software system. Attached Figure Description

[0009] Figure 1 This is a technical roadmap for the data defect detection method of the present invention; Figure 2 A schematic diagram of the VAE model; Figure 3 This is a schematic diagram of the Cred model. Detailed Implementation

[0010] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0011] The purpose of this invention is to propose an automated data defect detection method for deep learning classification tasks, used to detect low-quality data samples that negatively impact model accuracy during DNN training due to data defects. This data defect detection method draws on traditional code error localization methods based on model mutation, leveraging the differences between mutated models to amplify and identify potential data errors. First, using the trained DNN model as the original model, multiple mutated models are generated. Then, data features are extracted from the output of the mutated models and mapped to a confidence score. This score indicates the probability that the input data does not contain data defects, reflecting the degree to which the data is conducive to training an accurate classification model. Finally, defective data in the training dataset can be ranked based on the confidence score, and a threshold is set; data below this threshold is considered defective. This allows researchers to check and correct data based on the ranking results without reviewing the entire dataset.

[0012] The overall technical approach of this invention is as follows: Figure 1 As shown, the data defect detection method includes five steps: original model training, model mutation, feature extraction, confidence calculation, and defect data selection.

[0013] This invention first trains an original DNN model using a dataset, and then generates a mutated model based on the original model. The mutation method includes three strategies: input layer mutation, intermediate layer mutation, and output layer mutation. This step aims to amplify the difference between reliable and defective data, facilitating the subsequent selection of defective data. Subsequently, the dataset is input into the mutated model, and the model's output probability distribution and predicted loss value are extracted as data features. Based on these features, the reliability of each sample in the dataset is calculated. Reliability reflects the degree to which the data can be used to train an accurate model. Finally, the samples are ranked according to their calculated reliability scores. The lower the score, the more likely the sample is to be defective data. A threshold is set empirically; if the reliability score is below this threshold, the sample is selected as defective data.

[0014] This invention provides an automated data defect detection method for deep learning classification tasks, the method comprising the following steps: Step 1: Training the original model Before generating a mutation model, the dataset needs to be fully trained to build a suitable DNN model, thereby obtaining a reliable original model. Step 2: Model Variation After obtaining the original model, different model mutation strategies are used to generate mutated models. The mutated model is generated by first obtaining a new training set based on the original training set, and then using this new training set to fine-tune the original model. The new training set is obtained by filtering the original dataset through specific criteria. Three model mutation strategies are used, namely, mutation at the input layer, intermediate layer and output layer, to remove outliers in the input data, select neurons with abnormal activation states and exclude data with large prediction losses. Step 3: Feature Extraction Methods for the Model After generating the mutation model, two types of key features were extracted for each model: one is the output of the DNN model, and the other is the loss value predicted by the model. Step 4: Data Reliability Calculation An effective credibility prediction model is established, implemented using a neural network model. The model is denoted as Cred, which is a fully connected binary classification neural network model. The input to the Cred model is... , representing a data sample extracted from the dataset. Feature set, using This represents the computational operations of each layer in a neural network model. Let the model contain a total of... s There are 1 fully connected layer, and the weights corresponding to each variable are 1. The model can then be represented as a list of combinations of fully connected layers:

[0015] Among them, feature set It includes the two types of features mentioned in step three: one is the model output, that is, the probability distribution of the input sample belonging to each category; the other is the loss value of the mutation model. Step 5: Defect Data Selection The updated Cred model is used to calculate the confidence score of the original training dataset samples. The samples are then ranked based on the confidence score, and samples with scores below a threshold are identified as defective data. These low-scoring samples result in poor model performance.

[0016] Example 1: Step 1: Training the original model Before generating the mutation model, the dataset needs to be sufficiently trained to construct a suitable DNN model, thereby obtaining a reliable original model. The performance of the original model will directly affect the effect and stability of the subsequent mutation model. Training is considered complete once the original model has been confirmed to have reached the required convergence state.

[0017] The model architecture is selected based on the characteristics of the dataset and the task requirements. For image classification datasets, convolutional neural networks (CNNs) and visual Transformers can be chosen; for text and audio classification tasks, LSTM and BiLSTM models can be selected; if the dataset is complex, a DNN model needs to be built according to the specific task requirements. After selecting the model, it is initialized and relevant hyperparameters, such as learning rate, batch size, and training epochs, are set. Then, the original model is trained until it converges. The convergence is determined by observing changes in the model's accuracy and loss value.

[0018] Step 2: Model Variation After obtaining the original model, different model mutation strategies are used to generate mutated models. The mutated model is generated by first obtaining a new training set based on the original training set, and then fine-tuning the original model using this new training set. The new training set is obtained by filtering the original dataset using specific criteria. The starting point for designing the mutation strategies is to amplify the difference between reliable and defective data. This invention proposes three model mutation strategies, which involve mutation at the input layer, intermediate layers, and output layer. These methods include removing outliers from the input data, selecting neurons with abnormal activation states, and excluding data with high prediction loss. These three mutation strategies are implemented simultaneously to maximize the difference between reliable and defective data.

[0019] (1) Mutation strategy based on input layer The mutation strategy based on the input layer detects outliers in the input data and generates a mutation model by removing these outliers. This strategy uses a variational autoencoder (VAE) to detect outliers in the input space.

[0020] Visual Imagery (VAE) is a generative model primarily used for data compression, anomaly detection, and generating new data. Its core objective is to map complex high-dimensional data (such as images and text) to a low-dimensional space using an encoder, and then reconstruct the original data from this low-dimensional space using a decoder. The structure of a VAE includes an encoder and a decoder, and introduces randomness through a normal distribution to ensure the smoothness and generative nature of the mapping space. Its structure is as follows: Figure 2 As shown, using VAE (Variational Analysis) allows for data reconstruction by compressing data into a low-dimensional space. If the reconstruction error of a certain input data point is large, it indicates that the data is an outlier. Removing the data with large reconstruction errors generates a subset of the dataset with outliers removed, which can then be used to train the mutation model.

[0021] An encoder's role is to map input data (such as images, text, or other types of data) into a low-dimensional spatial representation. Encoders typically consist of multi-layered neural networks that progressively extract features from the input data through a series of non-linear transformations. Unlike traditional autoencoders, VAE encoders not only output a fixed low-dimensional representation but also output two parameters: the mean (…). μ ) and variance ( σ ²). These parameters together define a normal distribution in a low-dimensional space. μ This indicates the center position of a data point in a low-dimensional space. σ ² represents the dispersion of data points around that location. After obtaining the mean and variance in the low-dimensional space, VAEs utilize reparameterization techniques to ensure the model's trainability. This process involves transforming random noise (typically represented as ) from a standard normal distribution... Combined with mean and variance, it represents low-dimensional spatial variables. z for:

[0022] in It is noise sampled from a standard normal distribution. This mechanism ensures the continuity and randomness of the low-dimensional space, thereby enabling the model to capture the underlying structure of the data.

[0023] The decoder's role is to restore the mapping space representation generated by the VAE's encoder back to the high-dimensional original data. The decoder receives variables from the mapping space. z ,because z The presence of a certain degree of randomness allows the decoder to generate diverse outputs. Decoders typically consist of multiple non-linear transformation neural network layers that progressively transform the mapped low-dimensional space variables into results similar to the original input data, making the generated output data as close as possible to the real input data.

[0024] During training, the decoder's performance is evaluated using reconstruction error, which is derived from loss functions such as mean squared error and cross-entropy. By transforming low-dimensional data in the mapping space into high-dimensional data, the decoder can not only effectively reconstruct the training data but also create diverse outputs with a similar distribution to the training data.

[0025] In summary, the mutation strategy at the input layer uses a Variational Object (VAE) to detect outliers in the model input. By compressing the data into a low-dimensional mapping space and reconstructing it, the VAE identifies data samples with large reconstruction errors, which are often outliers in the training set. After removing these outliers, a new dataset can be generated for training the mutation model based on the input layer.

[0026] (2) Mutation strategy based on intermediate layer The mutation strategy based on intermediate layers aims to analyze the activation states of neurons in the intermediate layers, identify anomalous features in the data, and then remove data that causes abnormal neuron activation states. This invention first uses the activation states of neurons in the last linear layer to represent the feature extraction of the input data by the model's intermediate layers. Then, the K-means clustering algorithm is used to cluster the neuron activation states into two clusters.

[0027] K-means is a common clustering algorithm that divides a dataset into two clusters by minimizing the squared error between samples within each cluster and the centroid of that cluster. The specific steps of the algorithm are as follows: 1) Initialization: Randomly select two samples from the dataset as the initial centroids of the cluster: ,in, It is d A dimensional vector, representing the 3rd dimension vector. i The center of each cluster.

[0028] 2) Allocation: Calculate each sample Distance to all centroids. A commonly used distance metric is Euclidean distance, assuming the sample vector is... The centroid vector is The formula for calculating the distance is:

[0029] Then, each sample is assigned to the cluster of the nearest centroid. This step will be performed for each sample. Specify a cluster label This ensures that all samples are assigned to k Clusters.

[0030] 3) Update: After all samples have been assigned, update the centroid of each cluster. New centroid It is calculated from the mean of all samples in the cluster:

[0031] in It is the first i All samples within a cluster, This represents the number of samples in the cluster.

[0032] 4) Iteration: Repeat the partitioning and updating steps until the change in each cluster center is less than a preset threshold, or the preset number of iterations is reached. The convergence condition can be expressed as:

[0033] in It is a positive threshold value used to determine whether the change in the centroid meets the convergence requirement.

[0034] The objective function of K-means is to minimize the sum of the squares of the distances between all samples within a cluster and their corresponding centroids, defined as:

[0035] By iteratively updating the centroid and redistributing samples, the value of the objective function will gradually decrease.

[0036] Ultimately, K-means categorizes the activation states of intermediate layer neurons into two classes: one containing relatively dense, reliable samples, and the other potentially containing relatively sparse outlier samples. Assuming noisy data corresponds to sparse clusters, noisy data is removed to generate a new subset of training data. The original model is then fine-tuned on this subset to obtain a mutated model.

[0037] (3) Output layer-based mutation strategy The mutation strategy based on the output layer involves removing samples with large predicted loss values ​​from the dataset, based on the model's predicted loss value for each data sample. Large predicted losses are often due to inaccurate labels or inherent problems with the data itself, making it difficult for the model to learn correctly. Therefore, this portion of data is deleted to generate a new subset of training data. The remaining training data is then used to fine-tune the original model to obtain the mutated model.

[0038] In classification tasks, the loss function of the model is generally the cross-entropy loss function. The cross-entropy loss function measures the difference between the probability distribution predicted by the model and the true label distribution. For each sample, if the true label is represented using one-hot encoding... y The output of the model is Then the cross-entropy loss value L This can be expressed as the following formula:

[0039] in, C It is the total number of categories. It is the first i The true label of the class, The data predicted by the model belongs to the first... i The cross-entropy loss method determines the probability of a class. A key characteristic of cross-entropy loss is that the loss value is small when the model's predicted probabilities match the true labels; conversely, the loss value increases significantly if the predictions do not match the true labels. This mechanism allows the model to prioritize correctly classified samples during training. By reducing cross-entropy loss, the model can better learn the relationship between input data and corresponding classes.

[0040] The three mutation strategies described above generate three different mutation models. The fine-tuned mutation models retain the predictive ability of the original model on reliable data, but because outlier data is removed, their behavior when handling defective data differs from the original model, thus amplifying the difference between defective and reliable data. Furthermore, the three mutation strategies process each class in the training set separately. This approach allows the model to better identify data differences, as the data distribution, features, and noise levels may vary significantly between each class.

[0041] Step 3: Feature Extraction Methods for the Model After generating the mutation models, this invention extracts two key features for each model: one is the output of the DNN model, and the other is the model's predicted loss value. The model output reflects the model's predicted probability of a sample belonging to each category, which is a probability representation based on the model's feature extraction results from the input data. The loss value measures the difference between the model output and the true label, representing the model's error on the current task.

[0042] The output of a typical DNN model is calculated using the Softmax function. The Softmax function is commonly used in multi-class classification tasks, and its main function is to transform the model's output into the predicted probability for each class, such that the sum of all probabilities equals 1. Given a vector... Softmax output Calculated using the following formula:

[0043] in, It is the base of the natural logarithm. Represents an exponential function. It is a model for categories i The original output score, It is a category i The predicted probability. Softmax uses The model's output score is converted into a probability. If the input data learns to score highly in a certain category through the model, its corresponding probability will become higher after processing; if the score is low, the probability will be relatively low. This characteristic is beneficial for the model to classify the data.

[0044] Another feature is the model's loss value, which is derived from the loss function used during the model training process. The loss function is different for different models and tasks.

[0045] This invention uses the original model to extract two key features from the data, and then uses a mutation model to perform the same feature extraction operation on the data. That is, the same dataset is input into different models, and the model outputs and loss values ​​are calculated separately, serving as input to the subsequently constructed credibility model.

[0046] Step 4: Data Reliability Calculation Data credibility calculations utilize the mutation of deep neural network models to generate quantifiable credibility scores, which represent the probability that the data can be used to train the model. Data credibility primarily stems from the authenticity, accuracy, and completeness of the data, ensuring that the data accurately reflects the facts it represents and that there has been no human intervention or data loss.

[0047] To achieve a universal credibility scoring metric, an effective credibility prediction model needs to be established, which can be implemented using a neural network model. The model used is denoted as Cred, which is a fully connected binary classification neural network model. The model structure is as follows: Figure 3 As shown. The input to the Cred model is , representing a data sample extracted from the dataset. Feature set, using This represents the computational operations of each layer in a neural network model. Let the model contain a total of... s There are 1 fully connected layer, and the weights corresponding to each variable are 1. Then the model can be represented as a list of combinations of multiple fully connected layers:

[0048] Among them, feature set It includes the two types of features mentioned in step three: one is the model output, that is, the probability distribution of the input sample belonging to each category; the other is the loss value of the mutation model.

[0049] The weights of the Cred model are optimized using a dataset that has undergone data defect checking. We set defective data to be labeled 0 and reliable data to be labeled 1. The Cred model is tuned using the checked data to achieve correct classification of samples. The reliability score of a sample is the probability that the Cred model classifies it as reliable data. Specifically, assuming a single sample is input, the model output is... ,in This represents the predicted probability that the sample is defective data. Let represent the predicted probability that a sample is reliable data, and satisfy . Then the sample's credibility score is The lower the confidence score, the lower the sample quality and the greater the probability of data defects. Conversely, the higher the confidence score, the higher the sample quality and the more conducive it is to training an accurate classification model.

[0050] The Cred model building process includes two steps: model initialization and weight update. (1) Initialization. Initially, the entire dataset was not manually checked, so there were not enough reliable samples to train the Cred model. To solve this problem, a defect dataset was first generated by randomly sampling samples from the original training set and adding data noise and label noise to transform some reliable data in the dataset into defect data. Then, the Cred model was initialized and trained using the manually generated defect dataset.

[0051] (2) Update. Using the Cred model obtained from the initial training, calculate the confidence score of the unchecked data, sort these confidence scores, and provide the data with low confidence scores to the annotators. The annotators review these samples, provide feedback on whether they are defective data, and then re-annotate the checked dataset, selecting an appropriate confidence threshold. Data below the threshold is considered detected defective data. Finally, the Cred model is retrained using the re-annotated defective dataset. The model is updated multiple times until the model's defect detection accuracy meets the requirements, at which point the model update is considered complete.

[0052]

[0053] Step 5: Defect Data Selection The updated Cred model is used to calculate the confidence score of the original training dataset samples. The samples are then ranked based on the confidence score, and samples with scores below a threshold are identified as defective data. These low-scoring samples result in poor model performance.

[0054] The purpose of this invention is to propose an automated data defect detection method for deep learning classification tasks, used to detect low-quality data samples that negatively impact model accuracy during DNN training due to data defects. This data defect detection method draws on traditional code error localization methods based on model mutation, leveraging the differences between mutated models to amplify and identify potential data errors. First, using the trained DNN model as the original model, multiple mutated models are generated. Then, data features are extracted from the output of the mutated models and mapped to a confidence score. This score indicates the probability that the input data does not contain data defects, reflecting the degree to which the data is conducive to training an accurate classification model. Finally, defective data in the training dataset can be ranked based on the confidence score, and a threshold is set; data below this threshold is considered defective. This allows researchers to check and correct data based on the ranking results without reviewing the entire dataset.

[0055] The data defect detection method proposed in this invention allows testers to focus on processing detected defective data without having to traverse the entire dataset, thus efficiently obtaining a high-quality training dataset. Retraining the model using the processed training dataset effectively improves the classification accuracy of deep learning, further enhancing the performance of the intelligent software system.

[0056] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An automated data defect detection method for deep learning classification tasks, characterized in that, The method includes the following steps: Step 1: Training the original model Before generating a mutation model, the dataset needs to be fully trained to build a suitable DNN model, thereby obtaining a reliable original model. Step 2: Model Variation After obtaining the original model, different model mutation strategies are used to generate mutated models. The mutated model is generated by first obtaining a new training set based on the original training set, and then using this new training set to fine-tune the original model. The new training set is obtained by filtering the original dataset through specific criteria. Three model mutation strategies are used, namely, mutation at the input layer, intermediate layer and output layer, to remove outliers in the input data, select neurons with abnormal activation states and exclude data with large prediction losses. Step 3: Feature Extraction Methods for the Model After generating the mutation model, two types of key features were extracted for each model: one is the output of the DNN model, and the other is the loss value predicted by the model. Step 4: Data Reliability Calculation An effective credibility prediction model is established, implemented using a neural network model. The model is denoted as Cred, which is a fully connected binary classification neural network model. The input to the Cred model is... , representing a data sample extracted from the dataset. Feature set, using This represents the computational operation of each layer in a neural network model. Let the model contain *s* fully connected layers, and the weights corresponding to each variable be... The model can then be represented as a list of combinations of fully connected layers: Among them, feature set It includes the two types of features mentioned in step three: one is the model output, that is, the probability distribution of the input sample belonging to each category; the other is the loss value of the mutation model. Step 5: Defect Data Selection The updated Cred model is used to calculate the confidence score of the original training dataset samples. The samples are then ranked based on the confidence score, and samples with scores below a threshold are identified as defective data. These low-scoring samples result in poor model performance.

2. The automated data defect detection method for deep learning classification tasks as described in claim 1, characterized in that, Step one includes: selecting the model structure based on the characteristics of the dataset and the task requirements. For image classification datasets, convolutional neural networks and visual Transformer model architectures are selected; for text and audio dataset classification tasks, LSTM and BiLSTM models are selected; if the dataset is more complex, a DNN model needs to be constructed according to the specific task requirements.

3. The automated data defect detection method for deep learning classification tasks as described in claim 1, characterized in that, In step two, the mutation strategy based on the input layer includes: detecting outliers in the input data and generating a mutation model by removing these outliers; this strategy uses a variational autoencoder (VAE) to detect outliers in the input space; the VAE maps complex high-dimensional data to a low-dimensional space through the encoder, and reconstructs the original data from the low-dimensional space through the decoder; the VAE structure includes an encoder and a decoder, and introduces randomness through a normal distribution to ensure that the mapping space is smooth and generative; if the reconstruction error of a certain input data is large, it indicates that the data is an outlier, and the data with large reconstruction errors is removed, thereby generating a subset of the dataset with outliers removed, which is used to train the mutation model.

4. The automated data defect detection method for deep learning classification tasks as described in claim 3, characterized in that, The encoder's function is to map the input data into a low-dimensional spatial representation. Encoders typically consist of multi-layered neural networks that progressively extract features from the input data through a series of nonlinear transformations. The VAE's encoder not only outputs a fixed low-dimensional representation but also two parameters: the mean μ and the variance σ². These parameters collectively define a normal distribution in the low-dimensional space. μ represents the center position of the data point in the low-dimensional space, and σ² represents the dispersion of the data point around that position. After obtaining the mean and variance in the low-dimensional space, the VAE utilizes reparameterization techniques to ensure the model's trainability. This process involves removing random noise from the standard normal distribution. Combined with mean and variance, the low-dimensional spatial variable z can be represented as: in It is noise sampled from the standard normal distribution; this mechanism ensures the continuity and randomness of the low-dimensional space, thereby enabling the model to capture the underlying structure of the data; The decoder's role is to restore the mapping space representation generated by the VAE's encoder back to the high-dimensional original data; The decoder receives the variable z in the mapping space. Because z has a certain degree of randomness, the decoder can generate diverse outputs. The decoder consists of multiple non-linear transformation neural network layers that gradually transform the mapped low-dimensional space variables into results similar to the original input data, making the generated output data as close as possible to the real input data. During training, the performance of the decoder is evaluated by the reconstruction error, which is obtained by calculating the loss function. The decoder transforms low-dimensional data in the mapping space into high-dimensional data, which not only effectively reconstructs the training data, but also creates diverse outputs with a similar distribution to the training data.

5. The automated data defect detection method for deep learning classification tasks as described in claim 1, characterized in that, The mutation strategy based on the intermediate layer includes: analyzing the activation state of neurons in the intermediate layer, identifying abnormal features of the data, and then removing data that causes abnormal neuron activation states; firstly, the activation state of neurons in the last linear layer is used to represent the feature extraction of the input data by the intermediate layer of the model; then, the K-means clustering algorithm is used to cluster the activation states of neurons to obtain two clusters, one containing relatively densely distributed reliable samples, and the other potentially containing relatively sparsely distributed outlier samples.

6. The automated data defect detection method for deep learning classification tasks as described in claim 5, characterized in that, The mutation strategy based on the intermediate layer specifically includes: Initialization: Randomly select two samples from the dataset as the initial centroids of the cluster: ,in, It is a d-dimensional vector representing the center of the i-th cluster; Allocation: Calculate each sample , Distances to all centroids; assuming the sample vector is... The centroid vector is The formula for calculating the distance is: Then, each sample is assigned to the cluster of the nearest centroid. This step will be performed for each sample. Specify a cluster label This results in all samples being assigned to k clusters; Update: After all samples have been assigned, update the centroid of each cluster; new centroids. It is calculated from the mean of all samples in the cluster: in It refers to all samples within the i-th cluster. It is the number of samples in the cluster; Iteration: Repeat the partitioning and updating steps until the change in each cluster center is less than a preset threshold, or the preset number of iterations is reached; the convergence condition is expressed as: in It is a positive threshold value used to determine whether the change in the centroid satisfies the convergence requirement; The objective function of K-means is to minimize the sum of the squares of the distances between all samples within a cluster and their corresponding centroids, defined as: By iteratively updating the centroids and redistributing samples, the value of the objective function will gradually decrease; Ultimately, K-means categorizes the activation states of intermediate layer neurons into two classes: one containing relatively densely distributed reliable samples, and the other potentially containing relatively sparsely distributed outlier samples. Assuming that noisy data corresponds to sparse clusters, noisy data is removed to generate a new subset of training data, and the original model is fine-tuned on the training subset to obtain a mutated model.

7. The automated data defect detection method for deep learning classification tasks as described in claim 1, characterized in that, The mutation strategy based on the output layer predicts the loss value for each data sample according to the model and removes samples with large loss values ​​in the dataset. In classification tasks, the model uses the cross-entropy loss function, which measures the difference between the probability distribution predicted by the model and the true label distribution. For each sample, if the true label is represented by y using one-hot encoding, and the model output is... Then the cross-entropy loss value L is expressed by the following formula: Where C is the total number of categories, It is the true label of class i. It is the probability that the data predicted by the model belongs to the i-th class.

8. The automated data defect detection method for deep learning classification tasks as described in claim 1, characterized in that, In step three, the original model was used to extract two types of key features from the data, and the same feature extraction operation was performed on the data using a mutation model. That is, the same dataset was input into different models, and the model output and loss value were calculated respectively, which were then used as inputs for the credibility model to be constructed later.

9. The automated data defect detection method for deep learning classification tasks as described in claim 1, characterized in that, In step four, the weights of the Cred model are optimized using a dataset that has undergone data defect checks, with defective data labeled as 0 and reliable data labeled as 1. The Cred model is adjusted using the checked data to achieve correct classification of the samples. The reliability score of a sample is then the probability that the Cred model classifies the sample as reliable data. Assuming a single sample is input, the model's output is... ,in This represents the predicted probability that the sample is defective data. Let represent the predicted probability that a sample is reliable data, and satisfy . Then the sample's credibility score is The lower the confidence score, the lower the sample quality and the greater the probability of data defects. The higher the confidence score, the higher the sample quality and the more conducive it is to training an accurate classification model.

10. The automated data defect detection method for deep learning classification tasks as described in claim 9, characterized in that, In step four, the Cred model building process includes two steps: model initialization and weight update. (1) Initialization: Initially, the entire dataset was not manually checked, so there were not enough reliable samples to train the Cred model. First, a defect dataset was generated. Samples were randomly drawn from the original training set to add data noise and label noise. Some reliable data in the dataset was transformed into defect data. Then, the artificially generated defect dataset was used to initialize and train the Cred model. (2) Update: Using the Cred model obtained from the initial training, calculate the confidence score of the unchecked data, sort the confidence scores of these data, and provide the data with low confidence scores to the annotators; the annotators review these samples, provide feedback on whether they are defective data, and then re-annotate the checked dataset, and select an appropriate confidence threshold. Data below the threshold is considered as detected defective data; finally, the Cred model is retrained using the re-annotated defective dataset; the model is updated multiple times until the model's defect detection accuracy meets the requirements, at which point the model update is considered complete. 。

Images