On-demand data cleaning method and system for support vector machine
By proposing an on-demand data cleaning method for support vector machines, this paper utilizes Lagrange multipliers to partition the data and constructs a multidimensional grouped knapsack problem. Combining greedy strategies and reinforcement learning, the data cleaning is optimized, solving the problem of low reliability of cleaning results in existing methods and improving data quality and model performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-12
- Publication Date
- 2026-04-14
AI Technical Summary
Existing data cleaning methods have low reliability in cleaning results, making it difficult to correctly evaluate the quality of cleaning results without accessing real data. Furthermore, traditional methods have failed to effectively improve the model performance and data quality of downstream tasks.
We adopt an on-demand data cleaning method for support vector machines. By training an SVM model, we use Lagrange multipliers to partition the data, construct a multidimensional grouped knapsack problem, optimize the data cleaning strategy, and combine greedy strategies and reinforcement learning to gradually improve data quality and enhance model performance.
It significantly improves the reliability of data cleaning results and the model performance of downstream tasks with low time cost, optimizes data quality and model performance, and meets the generalization requirements of specific models.
Smart Images
Figure CN121858863A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data cleaning technology, specifically to an on-demand data cleaning method and system for support vector machines. Background Technology
[0002] Data cleaning aims to identify and correct noise in data (including incomplete, conflicting, and erroneous data), minimizing its adverse effects on data analysis or application tasks. Most existing data cleaning techniques start with the data itself, considering factors such as consistency, completeness, and accuracy, with the fundamental goal of improving data quality. Due to the lack of a true value, data cleaning often assumes that most data is clean, then uses the characteristics of clean data to clean dirty data; the smaller the difference between the cleaned dataset and the original dataset, the higher the data's accuracy. Guided by this idea, automated data cleaning methods use heuristic algorithms to repair data, using information such as integrity constraints or rules to determine the choice of cleaning operations, including rule-based, statistical feature-based, and knowledge-based methods. These methods are often easy to implement and have low cleaning costs because they can be automated without additional effort, but the reliability of the cleaning results is relatively low. Summary of the Invention
[0003] The purpose of this invention is to address the problem of low reliability of existing data cleaning methods by providing an on-demand data cleaning method and system for support vector machines.
[0004] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0005] An on-demand data cleaning method for support vector machines includes the following steps:
[0006] Step 1: Obtain the original dataset and train the SVM model using it. Record the SVM model's performance and the Lagrange multipliers for each data point in the original dataset. ;
[0007] Step 2: Using Lagrange multipliers The data in the dataset to be cleaned is divided into high-impact data, low-impact data, and no-impact data.
[0008] Step 3: Based on high-impact and low-impact data, obtain the quality score and cleaning cost for each data point;
[0009] Step 4: Based on low-impact data, the data is regarded as items in a knapsack, the performance improvement of the SVM model is regarded as the value of the items, the quality score and the cleaning cost, and a multidimensional grouped knapsack problem is constructed. Then, the multidimensional grouped knapsack problem is solved, and the data is cleaned according to the solution results.
[0010] Step 5: Based on high-impact data, obtain the SVM model performance improvement per unit cleaning cost, and clean the data one by one from largest to smallest performance improvement;
[0011] Step 6: Determine whether the quality score of the cleaned data reaches the threshold. If it does, proceed to step 9; otherwise, proceed to step 7.
[0012] Step 7: Input the cleaned data into the trained SVM model, and record the model performance and the Lagrange multiplier for each data point. ;
[0013] Step 8: Based on the model performance obtained in Step 1 and Step 7, obtain the model performance change. If the model performance change is less than the threshold, proceed to Step 9; if the model performance change is not less than the threshold, proceed to Step 10.
[0014] Step 9: End;
[0015] Step 10: Determine if the maximum number of iterations has been reached. If not, use the cleaned data as the original dataset in Step 1 and repeat Steps 1 to 8. Otherwise, end the process.
[0016] Furthermore, the specific steps of step 2 are as follows:
[0017] Step 11: Obtain the original dataset and preprocess the samples in the original dataset;
[0018] Step 12: Train the SVM model using the preprocessed samples and record the model performance. and the Lagrange multipliers for each sample in the original dataset. ;
[0019] Step 13: Based on the set threshold and Lagrange multipliers The data is divided as follows:
[0020] like Then it is a high-impact sample.
[0021] like If so, it is a low-impact sample.
[0022] like If the sample is unaffected, then it is considered an unaffected sample.
[0023] Furthermore, the preprocessing includes data standardization.
[0024] Furthermore, the effect of the model is expressed as follows:
[0025] ,
[0026] in, For the model effect, For tuples in the dataset, As a weighted term, This is the displacement term.
[0027] Furthermore, the cleaning cost is expressed as:
[0028] ,
[0029] in, The cleaned tuples, For the first The cost of cleaning individual data.
[0030] A system for implementing the method of any one of claims 1 to 5, the system being based on a data cleaning module, the data cleaning module specifically performing the following steps:
[0031] Step 1: Obtain the original dataset and train the SVM model using it. Record the SVM model's performance and the Lagrange multipliers for each data point in the original dataset. ;
[0032] Step 2: Using Lagrange multipliers The data in the dataset to be cleaned is divided into high-impact data, low-impact data, and no-impact data.
[0033] Step 3: Based on high-impact and low-impact data, obtain the quality score and cleaning cost for each data point;
[0034] Step 4: Based on low-impact data, the data is regarded as items in a knapsack, the performance improvement of the SVM model is regarded as the value of the items, the quality score and the cleaning cost, and a multidimensional grouped knapsack problem is constructed. Then, the multidimensional grouped knapsack problem is solved, and the data is cleaned according to the solution results.
[0035] Step 5: Based on high-impact data, obtain the SVM model performance improvement per unit cleaning cost, and clean the data one by one from largest to smallest performance improvement;
[0036] Step 6: Determine whether the quality score of the cleaned data reaches the threshold. If it does, proceed to step 9; otherwise, proceed to step 7.
[0037] Step 7: Input the cleaned data into the trained SVM model, and record the model performance and the Lagrange multiplier for each data point. ;
[0038] Step 8: Based on the model performance obtained in Step 1 and Step 7, obtain the model performance change. If the model performance change is less than the threshold, proceed to Step 9; if the model performance change is not less than the threshold, proceed to Step 10.
[0039] Step 9: End;
[0040] Step 10: Determine if the maximum number of iterations has been reached. If not, use the cleaned data as the original dataset in Step 1 and repeat Steps 1 to 8. Otherwise, end the process.
[0041] Furthermore, the specific steps of step 2 are as follows:
[0042] Step 11: Obtain the original dataset and preprocess the samples in the original dataset;
[0043] Step 12: Train the SVM model using the preprocessed samples and record the model performance. and the Lagrange multipliers for each sample in the original dataset. ;
[0044] Step 13: Based on the set threshold and Lagrange multipliers The data is divided as follows:
[0045] like Then it is a high-impact sample.
[0046] like If so, it is a low-impact sample.
[0047] like If the sample is unaffected, then it is considered an unaffected sample.
[0048] Furthermore, the preprocessing includes data standardization.
[0049] Furthermore, the effect of the model is expressed as follows:
[0050] ,
[0051] in, For the model effect, For tuples in the dataset, As a weighted term, This is the displacement term.
[0052] Furthermore, the cleaning cost is expressed as:
[0053] ,
[0054] in, The cleaned tuples, For the first The cost of cleaning individual data.
[0055] The beneficial effects of this invention are:
[0056] This application proposes a new paradigm for on-demand data cleaning. It treats the data itself and the downstream task model as a whole, further optimizing data quality for specific models while meeting the effectiveness and generalization requirements of the downstream tasks. This paradigm breaks through the limitations of traditional methods that only focus on the quality of the data itself, greatly improving the reliability of data cleaning results. Attached Figure Description
[0057] Figure 1 This is the overall flowchart of this application;
[0058] Figure 2 To illustrate the improvement of model performance Figure 1 ;
[0059] Figure 3 To illustrate the improvement of model performance Figure 2 . Detailed Implementation
[0060] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.
[0061] Specific Implementation Method 1: This implementation method describes an on-demand data cleaning method for support vector machines, including:
[0062] Step 1: The data acquisition system retrieves the raw dataset to be processed from the database. ,Include Each sample can contain numerical attributes, categorical attributes, and label fields. For example, a classification task might include numerical attributes, character attributes, and category labels.
[0063] Step 2: Construct the initial SVM model
[0064] Training the initial support vector machine model ,
[0065] Using the preprocessed dataset The SVM is initially trained to obtain the model. Record the model's performance and Lagrange multipliers information.
[0066] The expression that records the model's effect:
[0067] ,
[0068] in, It is a dataset tuples in These are the cleaned tuples. It is a displacement term.
[0069] According to the Lagrange multipliers of SVM and threshold The sample is divided into:
[0070] High-impact samples ( ): Data located near the decision boundary that contributes significantly to the model's decision;
[0071] Low-impact samples ( ): Data that has a certain impact on model performance;
[0072] No-impact sample ( ): Data that is far from the decision boundary and whose cleaning has limited effect on improving model performance.
[0073] Calculate data quality and cleaning cost metrics;
[0074] For each sample Calculate data quality score and cleaning costs .
[0075] The expressions for quality score and cleaning cost:
[0076] ,
[0077] express There are missing values or outliers.
[0078] ,
[0079] Step 3: Build an on-demand data cleaning and optimization model
[0080] Establish a multi-objective optimization function
[0081] This application focuses on SVM model performance, data quality, and Lagrange multipliers. With the objective as the objective and cleaning cost as the constraint, the optimization model can be expressed as:
[0082] ,
[0083] use - The constraint method transforms multi-objective optimization into a single-objective optimization problem.
[0084] By taking SVM performance as the single objective function and data quality, data importance, and data cleaning cost as constraints, a formalized single-constraint optimization problem is obtained.
[0085] ,
[0086] Transform the cleaning task into a multidimensional grouped knapsack problem;
[0087] The sample is considered as an item in the backpack;
[0088] Performance improvements in SVM models are considered as the value of an item.
[0089] Data quality and cleaning costs are considered as capacity;
[0090] Whether the data was cleaned and the cleaning results.
[0091] High-impact samples should be cleaned first.
[0092] Based on the model performance per unit cleaning cost of the samples The improvement involves performing cleaning or repair operations one by one in descending order of size to achieve the best model effect.
[0093] Model effect Data quality score ,
[0094] Dynamic programming is used to screen low-impact samples. The group knapsack dynamic programming algorithm selects the optimal sample subset and performs repair operations under the constraints of data quality and cleaning cost. Since the data size is much smaller than the full data, the computational cost is significantly reduced.
[0095] Pruning strategy for unaffected samples
[0096] For samples deemed to have no impact, no cleaning operation is performed; they are skipped directly. This strategy can significantly reduce cleaning time and resource consumption.
[0097] The specific cleaning operation is performed according to the error type of each sample, using the appropriate cleaner for repair, such as:
[0098] Missing values are filled using interpolation, regression prediction, or multi-source constraint completion.
[0099] Inconsistent values are corrected using rules or fixed using a knowledge base.
[0100] Step 5: SVM Model Update and Iterative Optimization
[0101] Train an updated SVM model using the cleaned dataset Record changes in model performance, including validation set accuracy, hinge loss, and support vector distribution.
[0102] Data cleaning methods typically follow a rule-based principle of minimum repair, aiming to minimize human intervention, while paying less attention to the impact of data cleaning on downstream analytical tasks. This approach may ensure that the cleaned data meets quality standards in form (e.g., no missing values, no violation of constraints, no obvious outliers), but its performance in downstream tasks may not be significantly improved, thus failing to truly realize the role of data cleaning in improving data quality and optimizing data-driven tasks. Downstream task-oriented data cleaning methods also have certain limitations. Existing mainstream downstream task-oriented data cleaning methods also have problems. Some methods focus solely on improving the effect of downstream tasks, often aiming to minimize the performance of downstream models, neglecting the characteristics of the data itself, and thus ignoring the improvement of data quality. They often only produce optimized downstream models, without improving data quality. Furthermore, the cleaning process often requires various forms of external information, such as validation sets or (correctly cleaned data results), making it difficult to accurately evaluate the quality of the cleaning results without accessing the actual data.
[0103] The goal of on-demand data cleaning for SVM is to improve the quality of SVM input data while simultaneously enhancing the performance of the downstream SVM model. This primarily involves optimizing both the data quality for the model and the requirements of SVM. These two aspects are mutually reinforcing: improving the information required by SVM helps optimize the data quality for the model, and improved data quality also contributes to obtaining a better SVM model. Therefore, iterative updates to both the model and the data are essential.
[0104] The optimization objectives of this cleaning process are the requirements of SVM and the data quality oriented towards the model. SVM requirements include performance requirements and the generalization requirements of the data to SVM. Data quality oriented towards the model includes data quality and data importance. Data quality is approached from a traditional data quality perspective, while data importance is considered from the perspective of the relationship between SVM and the data, aiming to enhance the positive impact of the data on SVM. The constraint is the cleaning cost. We aim to achieve high model accuracy with low time cost. However, multi-objective optimization problems are complex and prone to getting trapped in local optima. To simplify the problem, we use the ε-constraint method to transform it into a single-objective optimization problem, adjusting the model requirements to a unified optimization objective (including model performance and generalization requirements), with data quality and cleaning cost as constraints.
[0105] To solve this optimization problem, we first model it as a multidimensional grouped knapsack problem. Here, the tuples in the dataset are modeled as items in the knapsack, the model's requirements are modeled as the value of the items in the knapsack as the optimization objective, and data quality, data importance, and cleaning cost are modeled as constraints as the capacity of the items in the knapsack. Thus, the on-demand data cleaning problem for SVM is modeled as a multidimensional grouped knapsack problem, and a dynamic programming problem is designed to solve it. First, we design an exact solution based on dynamic programming, using a recursive equation to obtain the maximum value under a given capacity. However, although dynamic programming can theoretically achieve high accuracy, it still has exponential time complexity, resulting in excessive time cost. Therefore, we design an approximate algorithm to optimize the time cost. We design an approximate on-demand data cleaning based on a greedy strategy, using the Lagrange multipliers of SVM to approximate data importance, dividing the data into high / low / no-impact data. For high-impact data, a greedy strategy is designed to prioritize cleaning samples where the model effect varies greatly per unit of data quality. For low-impact data, an exact algorithm based on dynamic programming is used. Since the data size is significantly reduced compared to the entire dataset, the time cost is also significantly reduced. For data with no impact, a pruning strategy is designed, without any cleaning operation. Finally, when a validation set of the dataset is available, an on-demand data cleaning based on reinforcement learning is designed to optimize the effect of transforming the objective into the validation set. Other constraints are adjusted into penalty terms, and both serve as the reward component of reinforcement learning. Actions are set as cleaning operations, and states are designed as cleaning results. Through continuous interaction with the agent, the final cleaning result is obtained.
[0106] Through continuous iteration of the SVM training and cleaning optimization modules, both data quality and SVM requirements have been optimized. Data quality has been improved, and SVM has improved the final results, enabling on-demand data cleaning for SVM.
[0107] Compared to traditional data cleaning focused solely on data quality, we propose a new paradigm of on-demand data cleaning. Instead of solely focusing on the quality of the data itself or the effectiveness of downstream task models, it treats the data and downstream task models as a whole. While meeting the effectiveness and generalization requirements of downstream tasks, it further optimizes data quality for specific models. This paradigm breaks through the limitations of traditional methods that only focus on the quality of the data itself, further optimizing model-oriented data quality.
[0108] To address the model requirements of SVM and model-oriented data quality, the on-demand data cleaning problem is formally defined as a multi-objective optimization problem, and then... The constraint method transforms the problem into a single-constraint problem, prioritizing the optimization of the linearly additive hinge loss function of SVM. NP-completeness analysis is performed on this problem, and a single-objective optimization problem solution, HHLClean, for on-demand data cleaning oriented towards SVM is designed based on the knapsack problem. Furthermore, to mitigate overfitting of SVM during the cleaning process, an innovative overfitting estimation strategy using a Gaussian mixture model is introduced, significantly improving the data's generalization ability to SVM.
[0109] For the knapsack problem solution of on-demand data cleaning for SVM, an optimization strategy to reduce time cost is proposed. Starting from the sample importance in SVM, the knapsack problem strategy is pruned, prioritizing the cleaning of samples with large Lagrange multipliers (which have a significant impact on boundary decisions). Simultaneously, an approximate algorithm HHLClean-APPRO based on a greedy strategy is designed (i.e., processing high-impact data steps) to reduce the dimensionality of the multidimensional knapsack problem, thereby reducing the time cost of cleaning from... Reduce to .
[0110] in, Indicates the lower limit of data quality. The table represents the lower bound of data importance. Indicates the size of the sample candidate values. Indicates the sample size, while These represent the corresponding values after pruning.
[0111] (4) For the single-objective constraint of on-demand data cleaning, an on-demand data cleaning strategy based on reinforcement learning is designed. In the scenario where the validation set is available, the optimization objective is transformed from the loss function of the training set to the effect of the validation set. A policy network is used to learn how to make the optimal cleaning operation so that the proxy model performs best on the validation set, further optimizing the effect of on-demand data cleaning and significantly reducing the time cost.
[0112] (5) Experiments were conducted from three dimensions: the relationship between data quality and model performance, cleaning effect, and algorithm efficiency.
[0113] The impact of data quality on the performance of downstream tasks was verified.
[0114] The effectiveness of HHLClean was verified from the perspectives of SVM performance and model-oriented data quality.
[0115] The impact of error rate, number of cleaning iterations, and approximation ratio on on-demand data results was analyzed through ablation experiments.
[0116] This application outperforms other data cleaning methods focused on improving model performance and data quality. While HHLClean-APPRO performs slightly worse than HHLClean on most datasets, it still surpasses other cleaning methods. HHLClean enables on-demand data cleaning that improves both data quality and model performance. Regarding model performance, for example... Figure 2 HHLClean can improve the model's performance under most conditions, meeting the model's requirements.
[0117] Regarding data quality, such as Figure 3 As shown, HHLClean's RMSE is lower than other methods, indicating that it performs better in improving data quality.
[0118] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.
Claims
1. An on-demand data cleaning method for support vector machines, characterized in that... Includes the following steps: Step 1: Obtain the original dataset and train the SVM model using it. Record the SVM model's performance and the Lagrange multipliers for each data point in the original dataset. ; Step 2: Using Lagrange multipliers The data in the dataset to be cleaned is divided into high-impact data, low-impact data, and no-impact data. Step 3: Based on high-impact and low-impact data, obtain the quality score and cleaning cost for each data point; Step 4: Based on low-impact data, the data is regarded as items in a knapsack, the performance improvement of the SVM model is regarded as the value of the items, the quality score and the cleaning cost, and a multidimensional grouped knapsack problem is constructed. Then, the multidimensional grouped knapsack problem is solved, and the data is cleaned according to the solution results. Step 5: Based on high-impact data, obtain the SVM model performance improvement per unit cleaning cost, and clean the data one by one from largest to smallest performance improvement; Step 6: Determine whether the quality score of the cleaned data reaches the threshold. If it does, proceed to step 9; otherwise, proceed to step 7. Step 7: Input the cleaned data into the trained SVM model, and record the model performance and the Lagrange multiplier for each data point. ; Step 8: Based on the model performance obtained in Step 1 and Step 7, obtain the model performance change. If the model performance change is less than the threshold, proceed to Step 9; if the model performance change is not less than the threshold, proceed to Step 10. Step 9: End; Step 10: Determine if the maximum number of iterations has been reached. If not, use the cleaned data as the original dataset in Step 1 and repeat Steps 1 to 8. Otherwise, end the process.
2. The on-demand data cleaning method for support vector machines according to claim 1, characterized in that... The specific steps of step 2 are as follows: Step 11: Obtain the original dataset and preprocess the samples in the original dataset; Step 12: Train the SVM model using the preprocessed samples and record the model performance. and the Lagrange multipliers for each sample in the original dataset. ; Step 13: Based on the set threshold and Lagrange multipliers The data is divided as follows: like Then it is a high-impact sample. like If so, it is a low-impact sample. like If the sample is unaffected, then it is considered an unaffected sample.
3. The on-demand data cleaning method for support vector machines according to claim 2, characterized in that... The preprocessing includes data standardization.
4. The on-demand data cleaning method for support vector machines according to claim 3, characterized in that... The model's effect is represented as follows: ， in, For the model effect, For tuples in the dataset, As a weighted term, This is the displacement term.
5. The on-demand data cleaning method for support vector machines according to claim 4, characterized in that... The cleaning cost is expressed as: ， in, The cleaned tuples, For the first The cost of cleaning individual data.
6. A system for implementing the method of any one of claims 1 to 5, characterized in that... The system is based on a data cleaning module, which specifically performs the following steps: Step 1: Obtain the original dataset and train the SVM model using it. Record the SVM model's performance and the Lagrange multipliers for each data point in the original dataset. ; Step 2: Using Lagrange multipliers The data in the dataset to be cleaned is divided into high-impact data, low-impact data, and no-impact data. Step 3: Based on high-impact and low-impact data, obtain the quality score and cleaning cost for each data point; Step 4: Based on low-impact data, the data is regarded as items in a knapsack, the performance improvement of the SVM model is regarded as the value of the items, the quality score and the cleaning cost, and a multidimensional grouped knapsack problem is constructed. Then, the multidimensional grouped knapsack problem is solved, and the data is cleaned according to the solution results. Step 5: Based on high-impact data, obtain the SVM model performance improvement per unit cleaning cost, and clean the data one by one from largest to smallest performance improvement; Step 6: Determine whether the quality score of the cleaned data reaches the threshold. If it does, proceed to step 9; otherwise, proceed to step 7. Step 7: Input the cleaned data into the trained SVM model, and record the model performance and the Lagrange multiplier for each data point. ; Step 8: Based on the model performance obtained in Step 1 and Step 7, obtain the model performance change. If the model performance change is less than the threshold, proceed to Step 9; if the model performance change is not less than the threshold, proceed to Step 10. Step 9: End; Step 10: Determine if the maximum number of iterations has been reached. If not, use the cleaned data as the original dataset in Step 1 and repeat Steps 1 to 8. Otherwise, end the process.
7. The system according to claim 6, characterized in that... The specific steps of step 2 are as follows: Step 11: Obtain the original dataset and preprocess the samples in the original dataset; Step 12: Train the SVM model using the preprocessed samples and record the model performance. and the Lagrange multipliers for each sample in the original dataset. ; Step 13: Based on the set threshold and Lagrange multipliers The data is divided as follows: like Then it is a high-impact sample. like If so, it is a low-impact sample. like If the sample is unaffected, then it is considered an unaffected sample.
8. The system according to claim 7, characterized in that... The preprocessing includes data standardization.
9. The system according to claim 8, characterized in that... The model's effect is represented as follows: ， in, For the model effect, For tuples in the dataset, As a weighted term, This is the displacement term.
10. The system according to claim 9, characterized in that... The cleaning cost is expressed as: ， in, The cleaned tuples, For the first The cost of cleaning individual data.