A data error correction method and system based on repair action grammar and global constraint optimization

Abstract

The application discloses a data error correction method and system based on repair action grammar and global constraint optimization, which is suitable for automatic error correction and safe write-back of multi-source heterogeneous table data and semi-structured records. The method constructs a combinable atomic repair action and its parameter space, locates suspected error units based on constraint checking, generates a candidate repair set with evidence reference; combines hard constraints and soft constraints to evaluate the cost of the candidate, and models the candidate selection as a global consistency optimization problem of minimizing the total cost under the condition of meeting the hard constraint, and solves the optimal repair solution through integer programming, maximum satisfiability or satisfiability module theory. Further based on the solution gap between the optimal solution and the suboptimal solution, the multi-optimality and the evidence credibility, risk rejection is triggered, and the integrated repair suggestion and audit evidence are output to support manual review. The application reduces the risk of false write-back while ensuring global consistency and auditability, and is suitable for data governance and intelligent data service scenarios.

Description

Technical Field

[0001] This invention relates to the field of data governance and data quality control technology, and in particular to an automatic error correction method and system for enterprise multi-source heterogeneous data. Specifically, it is a data error correction technology based on candidate generation of repair action syntax, global consistency optimization solution based on hard / soft constraints, and auditable evidence output and risk rejection control. Background Technology

[0002] Enterprise data assets typically originate from multiple heterogeneous systems and processing chains. The processes of collection, synchronization, processing, and manual entry are prone to introducing data quality issues such as missing data, formatting errors, semantic conflicts, and inconsistencies across fields / tables. Data quality problems can impact statistical analysis and business decisions, and may amplify errors in downstream model training and inference, as well as accumulate risks in automated processes. Existing technologies commonly employ data correction methods including rule-based or constraint-based detection and repair, statistical imputation and correction, and repair suggestion generation based on learning models. Rule-based or constraint-based methods offer determinism and controllability, but have limited coverage of semantic errors and cross-field association errors. Learning models can expand coverage, but in the absence of global consistency constraints and auditable evidence, they pose a risk of local reasonableness but global inconsistency, and direct write-back under high uncertainty may trigger cascading business impacts. Therefore, a data correction solution is needed that can maintain controllable candidate repairs, verifiable global consistency, and auditable output results, while also possessing the ability to reject risks and provide aggregated suggestions. Summary of the Invention

[0003] This invention provides a data error correction method and system based on repair action syntax and global constraint optimization, aiming to solve the problems of uncontrollable candidate repairs, difficulty in guaranteeing global consistency, and uncontrollable risk of error correction write-back. The method of the present invention includes at least the following steps: (1) Constructing a repair action syntax: limiting the repair process to composable atomic repair actions and their parameter space, and establishing combination rules such as action chaining, mutual exclusion, dependency and applicable domain; (2) Suspected error location: verifying the data based on hard constraints and soft constraints to obtain a set of suspected error field values; (3) Candidate repair generation: generating a set of candidate repairs based on the suspected error field values ​​according to the repair action syntax, and binding evidence references to each candidate; (4) Cost evaluation: calculating the candidate cost value related to the modification range, statistical consistency, constraint satisfaction impact and evidence credibility of the candidate repairs; (5) Global optimization solution: modeling the candidate selection as a global constraint optimization problem that satisfies hard constraints and minimizes the total cost value, and obtaining the optimal repair solution through solvers such as MILP / MaxSAT / SMT; (6) Output and rejection: outputting the error correction results and audit evidence, and triggering risk rejection based on conditions such as solution gap, multiple optimality or evidence credibility, outputting a set of candidate repair suggestions and entering the closed loop of manual review. In contrast to the above methods, the present invention also provides a data error correction system, an electronic device, and a storage medium for implementing the above methods.

[0004] Specifically as follows:

[0005] A data error correction method based on repair action syntax and global constraint optimization includes: constructing a repair action syntax, which includes at least a set of atomic repair actions, a parameter space for each atomic repair action, and combination rules for atomic repair actions; determining a set of suspected errors based on constraint verification, which includes at least one suspected error field value; generating a candidate repair set for the suspected error field value according to the repair action syntax, wherein each candidate repair includes at least a target field identifier, a repair action identifier, repair action parameters, a candidate repair value, and a candidate evidence reference; constructing a constraint set including hard constraints and soft constraints, and evaluating the cost of the candidate repair set to obtain the candidate cost value, wherein the candidate cost value is at least related to the modification magnitude, statistical consistency, and impact on constraint satisfaction; constructing the candidate repair selection process as a global constraint optimization problem, and calling a solver to minimize the total cost value under the premise of satisfying the hard constraints to obtain the optimal repair solution; outputting error correction results and generating audit evidence based on the optimal repair solution; wherein, when the optimal repair solution meets the preset risk rejection conditions, a set of candidate repair suggestions is output and a manual review process is triggered.

[0006] The atomic repair action set includes at least one of the following: format normalization action, unit conversion action, enumeration normalization action, alias mapping action, string editing action, missing filling action, cross-field deduction action, and external code table alignment action.

[0007] The combination rules include at least: action chainability rules, action mutual exclusion rules, action parameter dependency rules, and action application domain rules; wherein the action application domain rules are used to limit the applicability of atomic repair actions to field types, data domains, or business entities. By verifying the action application domain, parameter dependencies, and mutual exclusion relationships during the candidate generation stage, repair action sequences that are not applicable to the target field or cannot be legally combined can be excluded, thereby limiting the candidate range for global solution.

[0008] Among them, the set of suspected errors determined based on constraint verification includes at least one of the following: field value range constraint verification, regular expression pattern constraint verification, enumeration set constraint verification, key constraint verification, functional dependency constraint verification, containment dependency constraint verification, cross-table reference constraint verification, and cross-field consistency constraint verification. The verification results also retain the correspondence between the suspected error field identifier and the trigger constraint identifier, which can be reused for candidate generation and global optimization, so that the repair object and its associated constraints remain consistent in subsequent processing.

[0009] The generation of the candidate repair set includes candidate fusion from at least two sources; the sources include at least one of the following: rule or dictionary generation, business code table or master data retrieval generation, generation based on historical clean sample statistics, and candidate generation or ranking based on a learning model;

[0010] The candidate evidence references include at least one of the following: a triggered constraint identifier, a code table entry identifier, a search hit document identifier, a historical sample cluster identifier, a field lineage identifier, and a sequence of repair actions used to generate candidate repair values. Evidence references are bound and stored with candidate repair values ​​and action sequences for cost assessment to extract evidence credibility, and an audit record is generated along with the output repair results to verify the source and generation process of the selected repair values.

[0011] The candidate replacement value is calculated from a multi-component cost vector, which includes at least one of the following: edit distance cost, semantic similarity cost, statistical consistency cost, constraint violation increment cost, and evidence credibility penalty cost. The candidate replacement value is obtained by weighting and summing the multi-component cost vectors using non-negative weights. By normalizing and weighting each cost component, the modification magnitude, statistical consistency, and evidence credibility of the candidates can be compared on the same scale, avoiding the selection of repair values ​​solely based on string modification distance.

[0012] The global constraint optimization problem includes at least the following: a selection constraint that chooses a unique candidate from its candidate repair set for each suspected erroneous field value; and constraints to ensure record-level or table-level consistency. The solver is one of the following: an integer linear programming solver, a maximum satisfiability solver, or a satisfiability modular theory solver. For multiple suspected erroneous fields associated with the same constraint, the solver jointly selects their candidate combination, rather than selecting the lowest-cost candidate for each field separately, so that the local candidate selection conforms to the overall consistency requirement of the repaired data.

[0013] The audit evidence includes at least: the identifier of the modified field, the value before modification, the value after modification, the identifier and parameters of the repair action used, the list of hard constraints satisfied, the list of soft constraints violated and their penalties, and the corresponding evidence references.

[0014] The preset risk rejection conditions include at least one of the following: the difference in total algebraic value between the optimal repair solution and the suboptimal repair solution is less than a preset threshold; there are multiple optimal repair solutions; the credibility of the evidence for the optimal repair solution is lower than a preset threshold; the risk of the optimal repair solution causing a violation of the constraint of the key field is greater than a preset threshold.

[0015] The review results of the manual review process are used to update at least one of the following: the combination rules in the action syntax, the candidate generation dictionary or code table, the weight parameters of the candidate cost values, and the weight parameters of the soft constraints. The review acceptance, modification, and rejection results are used to identify reusable mapping relationships, inapplicable action combinations, and candidate ranking biases, respectively, providing a basis for subsequent candidate generation and cost parameter adjustment for similar errors.

[0016] A data error correction system, implementing the aforementioned data error correction method, comprises: an action syntax construction module for constructing repair action syntax; an error location module for determining a set of suspected errors based on constraint verification; a candidate generation module for generating a candidate repair set for suspected error field values ​​based on the repair action syntax; a constraint construction module for constructing a constraint set including hard and soft constraints; a cost evaluation module for evaluating the cost of the candidate repair set to obtain candidate cost values; a global solution module for constructing the candidate selection into a global constraint optimization problem and calling a solver to obtain the optimal repair solution; and an audit and rejection module for outputting error correction results and generating audit evidence, and outputting aggregated candidate repair suggestions and triggering a manual review process when risk rejection conditions are met.

[0017] The candidate generation module includes: a rule generation unit, a code table retrieval unit, and a learning and sorting unit; the learning and sorting unit is used to reorder or prune the candidate repairs from the rule generation unit and the code table retrieval unit.

[0018] The constraint construction module is used to represent the constraint set as a constraint graph structure or a constraint relationship table; the nodes of the constraint graph structure correspond to fields or entities, and the edges correspond to constraint relationship types.

[0019] The global solution module is used to output the optimal repair solution and the suboptimal repair solution, and the audit and rejection module is used to determine whether to trigger risk rejection based on the difference in the total value of the two solutions.

[0020] It also includes a write-back control module, which writes the error correction results back to the data source when no risk rejection is triggered, and writes the aggregated candidate repair suggestions into the queue to be reviewed and disables automatic write-back when a risk rejection is triggered.

[0021] Beneficial effects

[0022] This invention limits the applicable domain, parameter space, and combination relationships of actions by repairing action syntax, eliminating illegal action sequences during the candidate generation stage, and ensuring that candidates entering the solution have a clear generation path. Candidate repair values ​​are bound and saved with action sequences and evidence references, supporting candidate cost calculation and post-repair auditing. Global constraint optimization jointly selects candidate combinations of related fields, comparing modification costs with soft constraint penalties while satisfying hard constraints, avoiding cross-field or cross-table conflicts caused by independent selection of each field. Furthermore, it determines whether automatic write-back is allowed based on the cost difference between the optimal and suboptimal solutions, multiple optimal states, and evidence credibility, outputting aggregated suggestions when candidate competition or insufficient evidence occurs, reducing the risk of uncertain repairs being directly written to the data source. Manual review results are used to correct existing mappings, action rules, and cost parameters, providing feedback for subsequent similar errors. The above process links candidate legality, overall consistency, and write-back risk control, while saving a verifiable record from the original value, action, to the repair value. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a block diagram of the overall structure of the data error correction system of the present invention;

[0025] Figure 2 This is a flowchart of the data error correction method of the present invention;

[0026] Figure 3 A schematic diagram of the constraint set;

[0027] Figure 4 Generate a flowchart for candidate repairs;

[0028] Figure 5 This is a schematic diagram for calculating the candidate substitution value.

[0029] Figure 6 Flowchart for global constraint optimization solution;

[0030] Figure 7 A diagram illustrating the data structure for outputting audit evidence;

[0031] Figure 8 This is a schematic diagram of the closed-loop manual review and online update process;

[0032] Figure 9 A schematic diagram of an electronic device structure for implementing the method of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] This invention provides a data error correction method and system based on repair action syntax and global constraint optimization. It achieves secure error correction and write-back of multi-source heterogeneous data through a "candidate controllable generation - global consistency solution - audit evidence output - risk rejection and manual review closed loop".

[0035] Implementation plan: See Figure 1 , Figure 2 A data error correction method based on repair action syntax and global constraint optimization, the method includes the following steps:

[0036] S1. Construct the syntax for repair actions.

[0037] S11. Define the set of atomic repair actions. Atomic repair actions should include at least: format normalization, unit conversion, enumeration normalization, alias mapping, string editing, missing data filling, cross-field derivation, and external code table alignment.

[0038] S12. Define the parameter space and constraints for each atomic repair action, such as the upper limit of edit distance, unit conversion ratio, enumeration mapping table, code table version number, etc.

[0039] S13. Define action combination rules, including action chaining rules, mutual exclusion rules, dependency rules, and application domain rules, to limit the applicable conditions of actions on field types and business entities.

[0040] S2. Locating suspected error sets based on constraint verification (see [link]). Figure 3 ,

[0041] S21. Construct a set of hard constraints and a set of soft constraints. Hard constraints are used for consistency conditions that must be met, while soft constraints are used for business preferences or statistical consistency conditions.

[0042] S22. Perform constraint validation on the data to be corrected, and output the set of suspected erroneous field values ​​and the triggering constraint identifiers.

[0043] S3. Generate a candidate repair set and bind evidence references, see [link to relevant documentation]. Figure 4 ,

[0044] S31. Based on the repair action syntax, generate candidate repairs for each suspected erroneous field value. Candidate sources include at least rules or dictionaries, business code tables or master data retrieval, historical clean sample statistics, and learning model generation or sorting. In one implementation, applicable atomic repair actions are first filtered based on the target field type, data domain, and business entity. Then, it is checked whether the action parameters belong to their parameter space, and a legal action sequence is constructed based on concatenation, mutual exclusion, and parameter dependency rules. The legal action sequence is then transformed accordingly to obtain candidate repair values. Identical fields and repair values ​​generated from multiple sources are merged, and their action sequences and evidence references are retained. Single field value ranges, enumerations, and format constraints can be used for pre-screening; association constraints involving multiple suspected erroneous fields are jointly judged in S5 to avoid prematurely deleting feasible candidates due to the current erroneous values ​​of other fields to be repaired.

[0045] S32. Bind an evidence reference to each candidate repair. The evidence reference includes at least a trigger constraint identifier, a code table entry identifier, a search hit identifier, a historical sample cluster identifier, a field lineage source identifier, and an action sequence. Each candidate is saved as a candidate record with the target field identifier, repair action sequence and parameters, candidate repair value, and evidence reference. In this embodiment, the evidence credibility q is between 0 and 1, determined by the preset credibility level of the main evidence source on which the repair value is based; the fusion candidate retains the original evidence references from each source to support subsequent cost calculation and manual review.

[0046] S4. Cost assessment, see [link / reference] Figure 5 .

[0047] S41. Calculate a multi-component cost vector for each candidate repair, wherein the multi-component cost vector includes at least: edit distance cost, semantic similarity cost, statistical consistency cost, constraint violation increment cost, and evidence credibility penalty cost;

[0048] S42. The candidate cost value is obtained by weighting and summing the multi-component cost vector using non-negative weights. Let the j-th candidate cost value of the i-th suspected error field be... ,but ,in and . To account for the edit distance cost, the edit distance between the original value and the candidate value is normalized by dividing the length of the two strings; when both are empty, the component is recorded as 0. For semantic similarity cost, To calculate the consistency cost, both values ​​are converted to a range of 0 to 1 according to the evaluation method used; the smaller the cost, the higher the consistency. To constrain the incremental cost of violations, the number of new soft constraint violations added by a single candidate replacement is normalized by dividing by the total number of soft constraints involved in that candidate; if there are no related soft constraints, it is recorded as 0. This component is only used for local cost estimation of candidates and does not replace the joint constraint judgment in S5. The cost of penalizing the credibility of evidence. The credibility of the evidence for this candidate is determined. Components not used in this implementation have a weight of 0; the remaining weights are determined by preset parameters or a validation set and can be adjusted based on historical review results.

[0049] S5. Global consistency optimization solution. See also Figure 6 ,

[0050] S51. Model the candidate selection as a globally constrained optimization problem. Suppose there are n suspected error fields, and the i-th field has... There are 10 candidates, and a binary variable is set for the j-th candidate. , This indicates that the candidate is selected, otherwise it is 0, and it is set. Fields not included in the suspected error set retain their original values ​​in this solution. In an implementation using an integer linear programming solver, for each hard constraint, a combination t of candidate related fields that would lead to constraint violation is listed, and the following settings are configured: This ensures that the violating combination cannot be selected simultaneously; where Let be the number of candidates included in this combination. For soft constraints s that allow for penalties, set a binary violation variable. Set for each violation combination t Violations are allowed, but penalties are included in the objective function. Figure 5 The hard constraint pre-screening in this step only applies to single-field constraints or fixed references that can be judged independently. Candidate combinations involving multiple fields to be repaired are uniformly coded in this step.

[0051] S52. The objective function is to minimize the total cost. ,in The candidate substitution value obtained by S42, The penalty weight for the soft constraint s. The integer linear programming solver is invoked to obtain the optimal candidate combination and objective value that satisfy the hard constraints. A maximum satisfiability solver or a satisfiability modular theory solver can be used as other implementation methods; this implementation takes integer linear programming encoding as an example. When the solver returns an infeasible solution or fails to obtain a confirmed optimal solution, it enters manual review instead of automatically writing back.

[0052] S53. To obtain a suboptimal candidate combination that differs from the optimal candidate combination, let the optimal solution be the candidate number selected for each field. Add to the original model Then solve it again. If a new optimal feasible solution is obtained, its objective value is denoted as... The gap is denoted as ;like If multiple optimal solutions exist for different candidate combinations, then there are multiple optimal solutions corresponding to different candidate combinations. If the exclusion constraint is added and then proven infeasible, then there is no second feasible candidate combination, and the solution gap threshold comparison is not performed, but other risk conditions such as the credibility of the evidence still need to be assessed.

[0053] S6. Output the error correction results, audit evidence, and risk rejection. See [link / reference] Figure 7 , Figure 8 .

[0054] S61. Output the error correction results and generate audit evidence. The audit evidence shall include at least: field identifier, value before modification, value after modification, identifier and parameters of the repair action used, list of hard constraints satisfied, list of soft constraints violated and penalty items, and corresponding evidence references.

[0055] S62. When the risk rejection condition is met, output aggregated candidate repair suggestions and trigger the manual review process, prohibiting automatic write-back; the risk rejection condition includes at least the following: the cost difference between the optimal solution and the suboptimal solution is less than a threshold, multiple optimal solutions exist, the credibility of the evidence is lower than a threshold, or the risk of violating key field constraints is higher than a threshold. In this embodiment, when or Rejection is triggered at a certain time, where Δ is calculated by S53. A preset gap threshold is set; for the fields actually modified by the optimal solution, the minimum value of the selected candidate evidence credibility is taken as the evidence credibility Q for this repair. When a rejection is triggered, is the preset evidence credibility threshold. The evidence credibility judgment for this modification is not executed when no field modification occurs. The threshold is preset according to the field risk level or determined through verification data. When rejection is triggered, the obtained optimal and alternative candidate combinations are retained, and the candidate value, action sequence, evidence reference and rejection reason of each field are listed to form a review pending suggestion; the repair result is allowed to be written back to the data source only when no rejection is triggered and the optimal feasible solution has been obtained.

[0056] S63. The manual review result is fed back to update the dictionary or code table, action combination rules, cost weights and soft constraint weights. In one embodiment, for a repair value that is accepted or confirmed after manual modification, the corresponding relationship between it and the original field value, field type and business domain is taken as an entry to be updated for the dictionary or code table; for a candidate rejected due to inapplicable action, the applicable domain or mutually exclusive rule of the corresponding action is adjusted according to the review reason; for a candidate whose sorting does not conform to the review conclusion, the review record is used to form a verification sample, the candidate cost weight and soft constraint weight are manually adjusted, the adjusted candidate sorting and constraint satisfaction result are checked on the verification sample, and after confirmation, they are used for subsequent batches.

[0057] Example 1

[0058] This embodiment is applicable to consistency repair of relational table data. The constraint set may include hard constraints such as primary key uniqueness, foreign key reference, value range, enumeration set, functional dependency and cross-field consistency, as well as soft constraints such as co-occurrence preference based on historical distribution. The number K of candidates for each suspected error field can be set to 2 to 50, preferably 10 to 20; the actual number is determined according to the candidates generated by the legal action sequence. The cost weights satisfy and , and the weight of the unused component is 0.

[0059] To illustrate the calculation process, an example record is adopted, where the original value of the province field is "A" and the original value of the city field is "B". The example code table only allows two sets of corresponding relationships between provinces and cities: "Province A — City A" and "Province B — City B". Neither of the original values belongs to the field enumeration set, so the two fields are included in the suspected error set. Province candidates "Province A" and "Province B" and city candidates "City A" and "City B" are obtained through alias mapping, string editing and code table alignment, and corresponding dictionary, code table or historical sample evidences are bound thereto.

[0060] This calculation example adopts the weight vector Edit distance is calculated per unit cost for character insertion, deletion, and replacement, and normalized according to S42. The statistical consistency cost and evidence credibility in this example use preset demonstration parameters. Following the order of edit distance, semantic similarity, statistical consistency, constraint violation increment, and evidence penalty, the component vector for candidate province "Province A" is (0.5,0,0,0,0), with a total cost of 0.1 and evidence credibility of 1; the component vector for candidate province "Province B" is (1,0,0.4,0,0.2), with a total cost of 0.4 and evidence credibility of 0.8; the component vector for candidate city "City A" is (1,0,0,0,0), with a total cost of 0.2 and evidence credibility of 1; and the component vector for candidate city "City B" is (0.5,0,0,0,0), with a total cost of 0.1 and evidence credibility of 1. This example does not set any additional soft constraint penalties, and its total global penalty is 0.

[0061] If we select the lowest-cost candidate for each of the two fields, we get "Province A - City B", with a total cost of 0.2. However, this combination violates the hard constraint of province-city correspondence. We need to set a candidate variable for the province. , and city candidate variables , , respectively, corresponding to the above candidate order, excluding the unique selection constraint , In addition, add and Two invalid province-city combinations were eliminated. A global solution yielded the optimal combination "Province A - City A". Add constraints to exclude the optimal combination. Subsequently, the second-best combination was obtained: "Province B - City B". , solution gap .

[0062] This example is only for demonstrating the determination process. Set to 0.15. Set to 0.85; the minimum evidence confidence of the candidates selected by the optimal combination. Therefore, this example did not trigger the gap resolution or evidence credibility rejection conditions, and write-back is allowed when other risk conditions are not triggered. If... If we set it to 0.25, then The system prohibits automatic write-back and submits the "Province A - City A" and "Province B - City B" data, along with their costs, evidence citations, and reasons for rejection, for manual review. The audit log saves the original values ​​"A" and "B", the selected repair value, the action sequence and parameters, the code table correspondence, the target value, and whether rejection was triggered. When the solver returns infeasibility, it outputs conflict information and initiates manual review.

[0063] Example 2

[0064] This embodiment is applicable to semantic consistency repair of semi-structured records containing short text fields and enumerated fields. Candidate repairs are prioritized based on alignment from the business entity database / knowledge base and code table, using the search hit identifier, entity ID, and evidence fragment identifier as evidence references. For records containing short text fields, the evidence credibility threshold or the resolution gap threshold can be increased to make the system more likely to trigger risk rejection, thereby outputting aggregated candidate suggestions and initiating manual review when uncertain.

[0065] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data error correction method based on repair action syntax and global constraint optimization, characterized in that, include: Construct a repair action syntax, which includes at least a set of atomic repair actions, a parameter space for each atomic repair action, and a combination rule for atomic repair actions; A set of suspected errors is determined based on constraint verification, wherein the set of suspected errors contains at least one suspected error field value; For suspected erroneous field values, a candidate repair set is generated based on the repair action syntax. Each candidate repair includes at least the target field identifier, repair action identifier, repair action parameters, candidate repair value, and candidate evidence reference. A constraint set containing hard and soft constraints is constructed, and the candidate repair set is evaluated for cost to obtain the candidate cost value. The candidate cost value is related to at least the modification magnitude, statistical consistency, and impact on constraint satisfaction. The candidate repair selection process is constructed as a global constraint optimization problem, and the solver is called to minimize the total cost value while satisfying the hard constraints to obtain the optimal repair solution. Based on the optimal repair solution, the error correction result is output and audit evidence is generated. When the optimal repair solution meets the preset risk rejection condition, a set of candidate repair suggestions is output and a manual review process is triggered.

2. The method according to claim 1, characterized in that, The atomic repair action set includes at least one of the following: format normalization action, unit conversion action, enumeration normalization action, alias mapping action, string editing action, missing data filling action, cross-field deduction action, and external code table alignment action. The combination rules include at least: action chainability rules, action mutual exclusion rules, action parameter dependency rules, and action application domain rules; wherein the action application domain rules are used to limit the applicable conditions of atomic repair actions for field types, data domains, or business entities.

3. The method according to claim 1, characterized in that, Based on constraint validation, the suspected error set should include at least one of the following: field value range constraint validation, regular expression pattern constraint validation, enumeration set constraint validation, key constraint validation, functional dependency constraint validation, inclusion dependency constraint validation, cross-table reference constraint validation, and cross-field consistency constraint validation.

4. The method according to claim 1, characterized in that, The generation of the candidate repair set includes candidate fusion from at least two sources; the sources include at least one of the following: rule or dictionary generation, business code table or master data retrieval generation, generation based on historical clean sample statistics, and candidate generation or ranking based on a learning model; The candidate evidence references include at least one of the following: a triggered constraint identifier, a code table entry identifier, a search hit document identifier, a historical sample cluster identifier, a field lineage identifier, and a sequence of repair actions used to generate candidate repair values.

5. The method according to claim 1, characterized in that, The candidate generation value is calculated from a multi-component cost vector, which includes at least one of the following: edit distance cost, semantic similarity cost, statistical consistency cost, constraint violation increment cost, and evidence credibility penalty cost. Furthermore, the candidate generation value is obtained by weighting and summing the multi-component cost vectors with non-negative weights.

6. The method according to claim 1, characterized in that, The global constraint optimization problem includes at least the following: selection constraints for each suspected erroneous field value to select a unique candidate from its candidate repair set; and constraints to ensure record-level or table-level consistency; the solver is one of the following: an integer linear programming solver, a maximum satisfiability solver, or a satisfiability modular theory solver.

7. The method according to claim 1, characterized in that, The audit evidence shall include at least: the identifier of the modified field, the value before modification, the value after modification, the identifier and parameters of the repair action used, the list of hard constraints satisfied, the list of soft constraints violated and their penalties, and the corresponding evidence references. The preset risk rejection conditions include at least one of the following: the difference in total algebraic value between the optimal repair solution and the suboptimal repair solution is less than a preset threshold; there are multiple optimal repair solutions; the credibility of the evidence for the optimal repair solution is lower than a preset threshold; the risk of the optimal repair solution causing a violation of the constraint of the key field is greater than a preset threshold. The review results of the manual review process are used to update at least one of the following: the combination rules in the action syntax, the candidate generated dictionary or code table, the weight parameters of the candidate cost values, and the weight parameters of the soft constraints.

8. A data error correction system, characterized in that, The system for implementing the method according to any one of claims 1-7, comprising: an action syntax construction module for constructing a repair action syntax; an error location module for determining a set of suspected errors based on constraint verification; a candidate generation module for generating a set of candidate repairs based on the repair action syntax for the values ​​of suspected error fields; a constraint construction module for constructing a set of constraints including hard constraints and soft constraints; a cost evaluation module for evaluating the cost of the candidate repair set to obtain the candidate cost value; a global solution module for constructing the candidate selection into a global constraint optimization problem and calling a solver to obtain the optimal repair solution; and an audit and rejection module for outputting error correction results and generating audit evidence, and outputting a set of candidate repair suggestions and triggering a manual review process when the risk rejection condition is met. The candidate generation module includes: a rule generation unit, a code table retrieval unit, and a learning and sorting unit; the learning and sorting unit is used to reorder or prune the candidate repairs from the rule generation unit and the code table retrieval unit. The constraint construction module is used to represent the constraint set as a constraint graph structure or a constraint relationship table; the nodes of the constraint graph structure correspond to fields or entities, and the edges correspond to constraint relationship types. The global solution module is used to output the optimal repair solution and the suboptimal repair solution, and the audit and rejection module is used to determine whether to trigger risk rejection based on the difference in the total value of the two solutions. It also includes a write-back control module, which writes the error correction results back to the data source when no risk rejection is triggered, and writes the aggregated candidate repair suggestions into the queue to be reviewed and disables automatic write-back when a risk rejection is triggered.

9. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images