A data cleaning outlier processing method based on multi-model divergence detection

Abstract

The application discloses a data cleaning abnormal value processing method based on multi-model divergence detection, and comprises the following steps: S1, data division and configuration initialization; S2, training of under-fitting model; S3, prediction and consistency aggregation; S4, artificial review and closed loop; and S5, auditing and reproducibility. The application can realize the data cleaning problem with high precision, high robustness and full-process auditing without relying on heavy artificial review.

Description

Technical Field

[0001] This invention relates to the fields of data cleaning and machine learning technology, specifically to a data cleaning outlier handling method based on multi-model divergence detection. Background Technology

[0002] In machine learning and data mining, high-quality training data is crucial for model performance. However, real-world datasets commonly contain labeling errors, noise, or outliers (collectively referred to as "erroneous data"). Current techniques for correcting erroneous training data primarily rely on rule filtering, manual sampling, and simple statistical cleaning. A typical approach involves initial screening of candidate samples based on preset keywords or field thresholds, followed by manual review and correction; or using fixed rules to batch replace field values. For data matching, character-level fuzzy matching or simple similarity scoring is often used to help locate suspected erroneous samples. In terms of quality auditing, only a small amount of summary statistics is typically recorded, lacking detailed, reproducible parameters and a complete process checklist.

[0003] The existing technical solutions have the following drawbacks:

[0004] (1) Insufficient precision: Filtering based on static rules is difficult to understand complex semantic contexts (such as negation, limitation, degree modification, etc.), while character-level fuzzy matching is prone to mismatch in the case of homographs or domain term variations, resulting in incorrect correction (correcting the correct data) or omission (not correcting the incorrect data).

[0005] (2) Poor robustness: The existing process lacks dynamic verification of data label parsing and numerical out-of-bounds, and abnormal samples are easy to enter subsequent stages, which will disrupt the data distribution and introduce bias. At the same time, due to the lack of a mechanism for actively detecting uncertainty (such as multi-model divergence detection), it is difficult to efficiently focus on high-value samples that the model cannot judge.

[0006] (3) Weak auditability and reproducibility: Most processes fail to systematically record input and output paths, parameters used, column name positioning strategies and detailed correction statistics, making subsequent quality audits, problem backtracking and migration and reproduction in different environments very difficult.

[0007] The root cause of these shortcomings lies in the fact that existing technical processes are mainly based on static rules, have limited matching capabilities, and their governance of data quality remains at the surface format level. Furthermore, they lack intelligent detection capabilities for uncertainties and structured metadata management capabilities. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a data cleaning outlier processing method based on multi-model divergence detection, so as to solve the problem of achieving high accuracy, high robustness and full auditability of data cleaning without relying on heavy manual review.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows.

[0010] A data cleaning and outlier handling method based on multi-model divergence detection includes the following steps:

[0011] S1. Data partitioning and configuration initialization: Load parameters from a specified read-only configuration file; divide the dataset to be cleaned into multiple different training subsets according to the partitioning method and number defined in the parameters, and initialize a unified configuration object;

[0012] S2. Training the underfitting model: Train on each training subset obtained in step S1. N An underfitting model is used to enhance the model's sensitivity to divergence from outliers by limiting its learning ability.

[0013] S3. Prediction and Consistency Aggregation: Using the data trained in step S2... N Each underfitting model makes a prediction for each sample in the dataset, and the results are summarized. N The prediction results of each underfitting model are used to calculate the consistency index between models, and divergent samples are marked according to preset judgment conditions, and a list of divergent samples is output.

[0014] S4. Manual review and loop closure: Manually review the data marked as divergent samples, update the original dataset based on the review results, calculate and record the divergence rate and correction amount of this iteration; if the iteration termination condition is not met, the training subset is re-divided based on the updated dataset, and steps S2 and S3 are executed again in sequence for the next round of iteration cleaning until the iteration termination condition is met.

[0015] S5. Audit and Reproducibility: Record the audit metadata used in this cleaning task in a structured audit log to ensure reproducibility across environments and auditability throughout the entire process.

[0016] Preferably, the configuration file in step S1 is stored in YAML or JSON format; the method for dividing the training subset in step S1 is either K-fold cross-validation or random partitioning, and when K-fold cross-validation is used, the number of partitions is the number of folds. K When using the random partitioning method, the number of partitions is an independently defined parameter for the number of partitions.

[0017] Preferably, the unified configuration object in step S1 includes the partitioning method and quantity, and the number of models. NThe parameters include: number of training rounds, number of frozen layers, regularization coefficient, random seed, divergence threshold, confidence threshold, variance threshold, entropy threshold, data version number, and iteration termination threshold.

[0018] Preferably, the specific method for training the underfitting model in step S2 includes at least one of the following:

[0019] Limit training cycles: Use an early stop mechanism to limit training cycles to 1 to 2 cycles;

[0020] Freeze pre-trained layers: Freeze some network layer parameters at the bottom of the pre-trained model;

[0021] Enhanced regularization: Set a weight decay coefficient of not less than 0.01 or a dropout rate of not less than 0.5.

[0022] Preferably, step S3 specifically includes:

[0023] S31. Prediction Summary: For each sample in the dataset, collect the data trained in step S2. N Predicted class labels and corresponding prediction confidence scores for each underfitting model;

[0024] S32. Consistency computation: based on N For each predicted category label and its corresponding prediction confidence level, a consistency index is calculated among the models. This consistency index includes the majority vote ratio and at least one of confidence level variance and information entropy as an auxiliary index. The calculation formulas for each index are as follows:

[0025] majority vote The calculation formula is as follows:

[0026]

[0027] in, The number of models predicting the primary category;

[0028] Confidence variance The calculation formula is as follows:

[0029]

[0030] in, For the first The confidence level of each model's prediction for the target category. This represents the average prediction confidence level.

[0031] Information entropy The calculation formula is as follows:

[0032]

[0033] in, This represents the total number of categories; for N The model for the first The average predicted probability of each category;

[0034] S33. Divergent Sample Labeling and Derivation: Based on the auxiliary indicators selected in step S32, execute the following corresponding judgment logic; a sample is labeled as a divergent sample if any condition is met:

[0035] When using confidence variance: if the majority vote ratio of a sample is less than the divergence threshold, or if its confidence variance is greater than the variance threshold when the majority vote ratio is not less than the divergence threshold, or if its prediction result is inconsistent with the original label and the average confidence is not less than the confidence threshold, then it is marked as a divergent sample.

[0036] When using information entropy: if the majority vote ratio of a sample is less than the divergence threshold, or if its information entropy is greater than the entropy threshold when the majority vote ratio is not less than the divergence threshold, or if its prediction result is inconsistent with the original label and the average confidence is not less than the confidence threshold, then it is marked as a divergence sample.

[0037] Finally, summarize all labeled samples to output a list of divergent samples.

[0038] Preferably, the list of divergent samples output in step S33 includes sample ID, original label, N The predicted class labels and prediction confidence of each underfitting model, the calculated consistency index value, and the divergence decision are used. Before outputting the list of divergent samples, the divergent samples are sorted according to the divergence score to determine the priority of manual review.

[0039] Preferably, the divergence score is based on the normalized confidence variance or information entropy, and... The weighted summation is calculated using the following formula:

[0040]

[0041] in, The divergence score has a range of [0,1], with higher values ​​indicating greater sample divergence. To indicate the degree of disagreement in voting; This represents the normalized confidence variance. Normalized information entropy; This is a weighting coefficient for the degree of disagreement in voting. These are the weighting coefficients for the normalized confidence variance. The weighting coefficients are the normalized information entropy, and , ;

[0042] and ,in, This represents the minimum confidence variance of all samples in the dataset. This represents the maximum confidence variance of all samples in the dataset.

[0043] And the entropy is normalized to [0,1].

[0044] Preferably, the manual review in step S4 specifically includes:

[0045] S41. Perform phrase matching and semantic retrieval, accurately correct divergent samples, and record the correction results and evidence.

[0046] S42. Based on the corrected sample extraction rules in step S41, perform deterministic rewriting on other samples in the dataset that are hit based on the corrected rules, and generate a corrected list and difference statistics; the corrected rules include phrase rules or binary key rules.

[0047] S43. Compile the correction results, correction evidence, and correction rules into a data dictionary or change list, and record version differences to support continuous iteration.

[0048] Preferably, the iteration termination threshold parameters in step S1 are the divergence rate threshold and the maximum number of iterations threshold; the iteration termination condition in step S4 is that the divergence rate of the current iteration is lower than the divergence rate threshold or the number of iterations reaches the maximum number of iterations threshold.

[0049] Preferably, the audit metadata in step S5 includes: configuration file path, all parameters and values, all input and output paths, intermediate statistical results, data version number and execution timestamp, and the intermediate statistical results include prediction results, confidence level, list of divergent samples, review result information, divergence rate and correction amount; the structured audit log is recorded in CSV format.

[0050] Due to the adoption of the above technical solutions, the technical progress achieved by this invention is as follows.

[0051] This invention features high precision and controllability: through a multi-model divergence detection mechanism, it can accurately locate deep-seated outliers (such as semantic annotation errors) that are difficult to detect by traditional methods, and restrict the automatic correction operation to high-confidence rules, which significantly reduces the risk of incorrect and missed corrections.

[0052] This invention offers high efficiency and robustness: by focusing limited and expensive manual review resources on high-value samples with the greatest model discrepancies, it significantly improves the efficiency of manual review. Simultaneously, the use of underfitting models and closed-loop correction processes enhance the system's robustness to noise.

[0053] This invention boasts strong auditability and reproducibility: through parameterized configuration and structured audit log recording throughout the entire process, every step of data cleaning is traceable and auditable, and can be reproduced in a completely consistent manner in different computing environments, greatly facilitating project quality management, collaboration, and iteration.

[0054] In summary, this invention achieves high-confidence automatic correction and efficient manual closed-loop collaboration without relying on complex computing resources, and is suitable for large-scale training data governance and continuous iteration scenarios. Attached Figure Description

[0055] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0057] A data cleaning and outlier handling method based on multi-model divergence detection is proposed. This method is implemented using a multi-model divergence detection module and an audit traceability module. These modules are decoupled and connected through a standardized CSV interface and parameter configuration. Figure 1 As shown, the method includes the following steps:

[0058] S1. Data partitioning and configuration initialization: Load parameters from a specified read-only configuration file; divide the dataset to be cleaned into multiple different training subsets according to the partitioning method and number defined in the parameters, and initialize a unified configuration object.

[0059] In this step, the training subset is divided using either K-fold cross-validation or random partitioning. When K-fold cross-validation is used, the number of partitions is equal to the number of folds. K When using the random partitioning method, the number of partitions is an independently defined parameter for the number of partitions.

[0060] In this step, the configuration file is typically stored in YAML or JSON format, and the following parameters are explicitly set:

[0061] N Number of models, for example N =3 indicates that 3 independent models are trained.

[0062] Training epochs: Controls the number of training iterations.

[0063] Number of layers to freeze: For example, for the BERT model, freeze the parameters of the first 9 layers and only fine-tune the higher layers.

[0064] Regularization coefficients: such as weight decay coefficients or dropout rates, are used to enhance regularization.

[0065] Random seed (Seed): For example, Seed=42, to ensure that the random process is reproducible.

[0066] Data version number: For example, v1.0, which identifies the dataset version.

[0067] Partitioning method and number: The partitioning method is either K-fold cross-validation or random partitioning; for example: if the partitioning method is K-fold cross-validation, the number of folds... K =5, meaning the dataset is divided into 5 parts, with 4 parts used for training and 1 part for validation; the partitioning method is random partitioning, and the number of partitions is... S =5, meaning the dataset is divided into 5 parts. Using this partitioning method helps in evaluating the model's generalization ability.

[0068] Divergence threshold : Used to participate in the judgment of divergent samples in step S3.

[0069] Confidence threshold : Used to participate in the judgment of divergent samples in step S3.

[0070] Variance threshold : Used to participate in the judgment of divergent samples in step S3.

[0071] Entropy threshold : Used to participate in the judgment of divergent samples in step S3.

[0072] Iteration termination threshold: This is a combination of the divergence rate threshold and the maximum number of iterations threshold, used to control the termination of iterative cleaning in step S4.

[0073] The unified configuration object includes the partitioning method and quantity, and the number of models. N Number of training epochs, number of frozen layers, regularization coefficient, random seed, and bifurcation threshold. Confidence threshold Variance threshold Entropy threshold Data version number and iteration termination threshold parameters.

[0074] S2. Training the underfitting model: Train on each training subset obtained in step S1. N An underfitting model is used to enhance the model's sensitivity to discrepancies in outlier samples by limiting the model's learning ability.

[0075] The specific methods for training an underfitting model include at least one of the following:

[0076] Limit training epochs: Use early stopping mechanisms, such as limiting the number of training epochs to 1 to 2 cycles, to prevent the model from overfitting to noise in the training set;

[0077] Freeze pre-trained layers: Freeze some network layer parameters at the bottom of the pre-trained model, such as freezing the parameters of layers 6-9 at the bottom of the BERT model, and only fine-tuning the higher layers to limit the model capacity and make its learning ability insufficient;

[0078] Enhanced regularization: Apply a larger weight decay factor or a higher dropout rate, such as setting a weight decay factor of not less than 0.01 or a dropout rate of not less than 0.5, to force the model to focus on key features rather than noisy details.

[0079] S3. Prediction and Consistency Aggregation: Using the data trained in step S2... N Each underfitting model makes a prediction for each sample in the dataset, and the results are summarized. N The prediction results of each underfitting model are used to calculate the consistency index between models, and divergent samples are marked according to preset judgment conditions, and a list of divergent samples is output.

[0080] This step specifically includes:

[0081] S31. Prediction Summary: For each sample in the dataset, collect the data trained in step S2. N The predicted class labels and corresponding prediction confidence scores of an underfitting model.

[0082] S32. Consistency computation: based on N For each predicted category label and its corresponding prediction confidence level, a consistency index is calculated among the models. The consistency index includes the majority vote ratio and at least one of confidence level variance and information entropy as an auxiliary index. The calculation formulas for each index are as follows:

[0083] (1) Majority voting ratio The calculation formula is as follows:

[0084]

[0085] in, The majority vote percentage is used to predict the number of models that fall into the primary category. That is, the proportion of models that predict the main category out of the total number of models;

[0086] The closer to 1, the higher the consistency between models.

[0087] (2) Confidence variance The calculation formula is as follows:

[0088]

[0089] in, For the first The confidence level of each model's prediction for the target category. This represents the average prediction confidence level.

[0090] The larger the value, the greater the uncertainty fluctuation between models.

[0091] (3) Information entropy The calculation formula is as follows:

[0092]

[0093] in, This represents the total number of categories; for N The model for the first The average predicted probability of each category;

[0094] The larger the value, the more uniform the predicted distribution, and the higher the uncertainty.

[0095] S33. Divergent Sample Labeling and Derivation: Based on the auxiliary indicators selected in step S32, execute the following corresponding judgment logic; a sample is labeled as a divergent sample if any condition is met:

[0096] (1) When using confidence variance: if the majority vote ratio of the sample is less than the divergence threshold (model opinion conflict), or if the confidence variance is greater than the variance threshold when the majority vote ratio is not less than the divergence threshold (model consensus is unstable), or if the prediction result is inconsistent with the original label and the average confidence is not less than the confidence threshold (high confidence error), then it is marked as a divergent sample.

[0097] (2) When using information entropy: if the majority vote ratio of a sample is less than the divergence threshold, or if its information entropy is greater than the entropy threshold when the majority vote ratio is not less than the divergence threshold (high uncertainty in model prediction), or if its prediction result is inconsistent with the original label and the average confidence is not less than the confidence threshold, then it is marked as a divergence sample.

[0098] Finally, summarize all labeled samples to output a list of divergent samples.

[0099] The output list of divergent samples includes sample ID, original label, N The predicted class labels and prediction confidence of each underfitting model, the calculated consistency index value, and the divergence decision are used. Before outputting the list of divergent samples, the divergent samples are sorted according to the divergence score to determine the priority of manual review.

[0100] Specifically, the divergence score is based on the normalized confidence variance or information entropy, and... The weighted summation is calculated using the following formula:

[0101]

[0102] in, is the divergence score, with a value range of [0,1]. The higher the value, the greater the sample divergence (i.e., the more likely it is to be an outlier or a difficult case). To indicate the degree of disagreement in voting; This represents the normalized confidence variance. Normalized information entropy; This is a weighting coefficient for the degree of disagreement in voting. These are the weighting coefficients for the normalized confidence variance. The weighting coefficients are the normalized information entropy, and , ,For example: , (Confidence variance as an auxiliary indicator) or , (Information entropy is used as an auxiliary indicator);

[0103] and (Normalization from minimum to maximum value), where, This represents the minimum confidence variance of all samples in the dataset. This represents the maximum confidence variance of all samples in the dataset.

[0104] And the entropy is normalized to [0,1].

[0105] In steps S2 and S3, step S2 constructs an "underfitting model" with limited capacity or insufficient training, enabling it to reach consensus (low variance) on simple samples but exhibiting significant uncertainty and divergence (high variance) on noisy labels and ambiguous difficult samples. Step S3 then aggregates these divergences and uses the differential performance of the weak learner to accurately locate potential labeling errors or difficult examples (i.e., divergent samples).

[0106] S4. Manual review and loop closure: Manually review the data marked as divergent samples, update the original dataset based on the review results, calculate and record the divergence rate and correction amount for this iteration; if the iteration termination condition is not met, the training subset is re-divided based on the updated dataset, and steps S2 and S3 are executed again for the next round of iteration cleaning until the iteration termination condition is met.

[0107] In this step, the iteration terminates when the divergence rate of the current iteration is lower than the divergence rate threshold or the number of iterations reaches the maximum number of iterations threshold.

[0108] Manual review specifically includes:

[0109] S41. Perform phrase matching and semantic retrieval, accurately correct divergent samples, and record the correction results and evidence.

[0110] S42. Based on the corrected sample extraction rules in step S41, perform deterministic rewriting on other samples in the dataset that are hit based on the corrected rules, and generate a corrected list and difference statistics; wherein, the corrected rules include phrase rules or binary key rules;

[0111] S43. The correction results, correction evidence, and correction rules are compiled into a clear data dictionary or change list, and version differences are recorded to support continuous iteration.

[0112] S5. Audit and Reproducibility: Record the audit metadata used in this cleaning task in a structured audit log to ensure reproducibility across environments and auditability throughout the entire process.

[0113] Audit metadata includes: configuration file path, all parameters and values, all input / output paths, intermediate statistical results, data version number and execution timestamp. Intermediate statistical results include prediction results, confidence levels, a list of divergent samples, verification results, divergence rate, and correction amounts, thus ensuring the consistency of experimental conditions. Structured audit logs are recorded in CSV format.

[0114] This invention has significant innovations and advantages in three aspects:

[0115] First, the controllable correction process driven by divergence focus directly screens high-value samples based on uncertainty signals and outputs a controlled list, significantly reducing the risk of incorrect corrections and maintaining controllable boundaries.

[0116] Secondly, the multi-model discrepancy detection centered on uncertainty automatically focuses difficult cases on a small number of high-value samples, greatly improving the efficiency of manual review and the overall output quality.

[0117] Third, unified parameterized configuration and structured audit log output are integrated throughout the process, ensuring reproducibility across environments and auditability throughout the entire process, thereby enhancing the robustness and adaptability of the project.

Claims

1. A data cleaning and outlier handling method based on multi-model divergence detection, characterized in that: Includes the following steps: S1. Data partitioning and configuration initialization: Load parameters from a specified read-only configuration file; Based on the partitioning method and number defined in the parameters, the dataset to be cleaned is divided into multiple different training subsets, and a unified configuration object is initialized. S2. Training the underfitting model: Train on each training subset obtained in step S1. N An underfitting model is used to enhance the model's sensitivity to divergence from outliers by limiting its learning ability. S3. Prediction and Consistency Aggregation: Using the data trained in step S2... N Each underfitting model makes a prediction for each sample in the dataset, and the results are summarized. N The prediction results of each underfitting model are used to calculate the consistency index between models, and divergent samples are marked according to preset judgment conditions, and a list of divergent samples is output. Step S3 specifically includes: S31. Prediction Summary: For each sample in the dataset, collect the data trained in step S2. N Predicted class labels and corresponding prediction confidence scores for each underfitting model; S32. Consistency computation: based on N For each predicted category label and its corresponding prediction confidence level, a consistency index is calculated among the models. This consistency index includes the majority vote ratio and at least one of confidence level variance and information entropy as an auxiliary index. The calculation formulas for each index are as follows: majority vote The calculation formula is as follows: in, The number of models predicting the primary category; Confidence variance The calculation formula is as follows: in, For the first The confidence level of each model's prediction for the target category. This represents the average prediction confidence level. Information entropy The calculation formula is as follows: in, This represents the total number of categories; for N The model for the first The average predicted probability of each category; S33. Divergent Sample Labeling and Derivation: Based on the auxiliary indicators selected in step S32, execute the following corresponding judgment logic; a sample is labeled as a divergent sample if any condition is met: When using confidence variance: if the majority vote ratio of a sample is less than the divergence threshold, or if its confidence variance is greater than the variance threshold when the majority vote ratio is not less than the divergence threshold, or if its prediction result is inconsistent with the original label and the average confidence is not less than the confidence threshold, then it is marked as a divergent sample. When using information entropy: if the majority vote ratio of a sample is less than the divergence threshold, or if its information entropy is greater than the entropy threshold when the majority vote ratio is not less than the divergence threshold, or if its prediction result is inconsistent with the original label and the average confidence is not less than the confidence threshold, then it is marked as a divergence sample. Finally, summarize all labeled samples and output a list of divergent samples; S4. Manual review and loop closure: Manually review the data marked as divergent samples, update the original dataset based on the review results, calculate and record the divergence rate and correction amount of this iteration; if the iteration termination condition is not met, the training subset is re-divided based on the updated dataset, and steps S2 and S3 are executed again in sequence for the next round of iteration cleaning until the iteration termination condition is met. S5. Audit and Reproducibility: Record the audit metadata used in this cleaning task in a structured audit log to ensure reproducibility across environments and auditability throughout the entire process.

2. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 1, characterized in that: In step S1, the configuration file is stored in YAML or JSON format; the method for dividing the training subset in step S1 is either K-fold cross-validation or random partitioning, and when K-fold cross-validation is used, the number of partitions is the number of folds. K When using the random partitioning method, the number of partitions is an independently defined parameter for the number of partitions.

3. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 1, characterized in that: The unified configuration object in step S1 includes the partitioning method and quantity, and the number of models. N The parameters include: number of training rounds, number of frozen layers, regularization coefficient, random seed, divergence threshold, confidence threshold, variance threshold, entropy threshold, data version number, and iteration termination threshold.

4. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 3, characterized in that: The specific methods for training the underfitting model in step S2 include at least one of the following: Limit training cycles: Use an early stop mechanism to limit training cycles to 1 to 2 cycles; Freeze pre-trained layers: Freeze some network layer parameters at the bottom of the pre-trained model; Enhanced regularization: Set a weight decay coefficient of not less than 0.01 or a dropout rate of not less than 0.

5.

5. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 1, characterized in that: The list of divergent samples output in step S33 includes sample ID, original label, N The predicted class labels and prediction confidence of each underfitting model, the calculated consistency index value, and the divergence decision are used. Before outputting the list of divergent samples, the divergent samples are sorted according to the divergence score to determine the priority of manual review.

6. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 5, characterized in that: The divergence score is based on the normalized confidence variance or information entropy, and... The weighted summation is calculated using the following formula: in, The divergence score has a range of [0,1], with higher values ​​indicating greater sample divergence. To indicate the degree of disagreement in voting; This represents the normalized confidence variance. Normalized information entropy; This is a weighting coefficient for the degree of disagreement in voting. These are the weighting coefficients for the normalized confidence variance. The weighting coefficients are the normalized information entropy, and , ; and ,in, This represents the minimum confidence variance of all samples in the dataset. This represents the maximum confidence variance of all samples in the dataset; And the entropy is normalized to [0,1].

7. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 3, characterized in that: The manual review in step S4 specifically includes: S41. Perform phrase matching and semantic retrieval, accurately correct divergent samples, and record the correction results and evidence. S42. Based on the corrected sample extraction rules in step S41, perform deterministic rewriting on other samples in the dataset that are hit based on the corrected rules, and generate a corrected list and difference statistics; the corrected rules include phrase rules or binary key rules. S43. Compile the correction results, correction evidence, and correction rules into a data dictionary or change list, and record version differences to support continuous iteration.

8. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 7, characterized in that: In step S1, the iteration termination threshold parameters are the divergence rate threshold and the maximum number of iterations threshold; in step S4, the iteration termination condition is that the divergence rate of the current iteration is lower than the divergence rate threshold or the number of iterations reaches the maximum number of iterations threshold.

9. The data cleaning and outlier processing method based on multi-model divergence detection according to claim 1, characterized in that: The audit metadata in step S5 includes: configuration file path, all parameters and values, all input and output paths, intermediate statistical results, data version number and execution timestamp, and the intermediate statistical results include prediction results, confidence level, list of divergent samples, review result information, divergence rate and correction amount; the structured audit log is recorded in CSV format.

Images