A rule-based image-based data incremental cleaning method

Abstract

The present application relates to the technical field of data cleaning, and particularly relates to a data incremental cleaning method based on rule portrait, which first generates a field-level rule portrait, converts data source attributes, field characteristics, format rules, business constraints and historical correction trajectories into ordered features and marks stable states; secondly, matches new batch fields with the rule portrait to generate a field-level cleaning task list; thirdly, scans the incremental data in partitions, determines change partitions and associated partitions through fingerprint comparison; fourthly, performs incremental cleaning on the change segments according to field dependency relationships; and finally, writes the field state index and data version relationship, updates the rule portrait and outputs the cleaning result.

Description

Technical Field

[0001] This invention relates to the field of data cleaning technology, and more specifically, to a data incremental cleaning method based on rule profiling. Background Technology

[0002] Data cleaning is a core component of data governance and data quality control. With the continuous incremental access of data from various business systems, the demand for batch incremental data cleaning is gradually increasing, which places higher demands on the adaptability of cleaning rules and the execution efficiency of the processing process.

[0003] In existing incremental data cleaning technologies, on the one hand, cleaning rules mostly rely on fixed field names or preset mapping relationships. When faced with multi-source heterogeneous data and scenarios with changes in field names, the adaptability of rule matching is limited. Moreover, rule updates often directly overwrite the original rules with the current batch features, which can easily solidify the abnormal data features of a single batch into general cleaning rules, affecting the long-term stability of the cleaning rules. On the other hand, incremental processing often performs scanning verification on a whole batch or full record basis, without identifying changes at the partition granularity of the data. This can easily lead to unchanged data being repeatedly entered into the cleaning process, increasing unnecessary computational overhead. At the same time, existing solutions lack the ability to collaboratively process cross-field and cross-partition business constraints, and the status traceability and dynamic rule iteration mechanism of the cleaning process are not perfect.

[0004] To address the shortcomings of the existing technologies, this invention proposes a data incremental cleaning method based on rule-based profiling. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a data incremental cleaning method based on rule-based profiling.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A data incremental cleaning method based on rule-based profiling includes the following steps:

[0008] Step S1: Generate field-level rule profiles, transforming data source attributes, field structures, field values, format rules, business constraints, and historical correction trajectories into field-level ordered features, and marking the stable state of the features in the rule profiles.

[0009] Step S2: Match the new batch fields with the rule profile, generate temporary features for the new batch fields, match the temporary features with the existing rule profile, and generate a field-level cleaning task list based on the matching results.

[0010] Step S3: Perform partition scanning on the incremental data and locate the changed segments. Determine the changed partitions and necessary associated partitions to enter the cleaning process through stable partitions and fingerprint comparison.

[0011] Step S4: Perform field-level incremental cleaning on the changed fragments, determine the processing order according to the field dependencies, and after saving the state before cleaning, perform null value filling, format normalization, duplicate item merging, outlier replacement or conflict field correction, and determine the state after cleaning based on the verification results.

[0012] Step S5: Write the field status index and data version relationship, update the rule profile according to the cleaning results, and output the incremental data after cleaning, field status index, rule profile update record and version relationship record.

[0013] Furthermore, in step S1, generating a field-level rule profile includes: parsing the access data to obtain a field set, field declaration type, field original name, field position, field sample value, and record primary key; parsing the actual type distribution of the fields based on the field sample values; performing only regularization processing for feature extraction on the original field values ​​to obtain the field missing status, format status, and field value fingerprint, and determining the field quality status features based on the field value fingerprint.

[0014] Furthermore, step S1 also includes: determining the allowed value range of the field according to the field type, forming a field format automaton according to the field length constraint, character category constraint, date template, numerical precision template, encoding bit length template and separator template, and configuring a set of business constraints; the set of business constraints includes field uniqueness constraints, enumeration constraints, range constraints, cross-field dependency constraints, version time constraints and source priority constraints, and the set of business constraints is divided into strong constraints and weak constraints.

[0015] Furthermore, the field-level rule profile includes data source type, field declaration type, field actual type distribution, field allowed value range, field missing status, field duplicate status, field illegal format status, field format automaton, business constraint set, and field historical cleaning trajectory. After forming the field-level rule profile, the field actual type distribution, field allowed value range, field missing status, field duplicate status, field illegal format status, and field format automaton are marked as stable statuses. Stable features participate in the main matching, while features to be confirmed participate in the auxiliary judgment.

[0016] Furthermore, in step S2, matching new batch fields with rule profiles includes: performing a feature extraction process corresponding to the field-level rule profile on the new batch fields to obtain a temporary feature group; matching the temporary feature group with the candidate rule profile, the matching process taking into account the data source type, field type, value range, format status and business constraint hit status; when the field sample size is insufficient to stably determine a certain feature, the feature is marked as a default comparable state, and the comparison weight corresponding to the feature is reduced during the matching process.

[0017] Furthermore, when multiple candidate rule profiles meet the matching conditions but cannot be uniquely determined, they are compared sequentially based on the number of strong constraint hits, data source type consistency, historical cleaning trajectory stability, format automaton pass rate, and profile version timeliness. If they still cannot be uniquely determined, the corresponding field is marked as a multiple candidate field, and its execution is restricted to detection and annotation that does not change the business meaning of the field. When no candidate rule profile meets the matching conditions, the corresponding field is marked as a new field, and a new field-level rule profile is formed after it continuously meets the stable conditions.

[0018] Furthermore, the field-level cleaning task list includes field identifiers, adaptation profile identifiers, allowed actions, prohibited actions, triggering conditions, dependent fields, associated partitions, write-back locations, and failure handling methods. When a field is subject to strong constraints, pre-correction and post-correction checks are set in the field-level cleaning task list. When a field involves cross-field dependencies, the dependent field is listed as a prerequisite field. When a field involves cross-partition uniqueness constraints, the associated partition is listed as a read-only check partition.

[0019] Furthermore, in step S3, the partition scanning includes: forming partitions according to the data source identifier, business primary key, time window, field constraint relationship, and hash bucket; when a record lacks a stable business primary key, generating a temporary comparison key based on the identifier field, time field, and source field marked as stable in the rule profile, the temporary comparison key is only used for partition and fingerprint comparison; determining the partition change status based on the record fingerprint and field fingerprint of the previous valid version and the current batch, writing the version inheritance relationship into the unchanged partition, and marking the changed partition as a change fragment; when a field in the change fragment is affected by cross-partition constraints, locating the relevant partition and making the relevant partition participate in constraint verification and candidate value search.

[0020] Furthermore, in step S4, performing field-level incremental cleaning on the changed fragments includes: determining the field cleaning order based on business constraints, and writing a snapshot of the state before cleaning for each field; for null value fields, forming a candidate value set based on the allowed value range of the field, historical valid values ​​of the same business object, non-null values ​​within the same context in the current changed fragment, and cross-field constraint derivation values, and writing the filler value when the candidate value meets the filler condition; for formatted fields, performing format normalization only when the field value can be accepted by the field format automaton and does not conform to the target output template, and writing the cleaned field value after format normalization when it passes the field format automaton and business constraint verification and the semantic parsing value is consistent.

[0021] Furthermore, step S4 also includes: for candidate duplicate groups, determining the master record based on record completeness, number of strong constraint conflicts, update time, source credibility, and historical correction stability, and merging non-conflicting fields of non-master records into the master record at the field granularity; for candidate outliers, performing outlier replacement when they meet the outlier judgment conditions and violate business constraints, and the candidate replacement value must be verified by the field format automaton and business constraints; for conflicting fields, forming candidate correction rules based on the field format automaton, business constraint set, and historical cleaning trajectory, and selecting correction rules based on version time relationship, constraint coverage, source credibility, and rollback status.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. To address the issues of existing cleaning rules relying on fixed field mappings and being susceptible to the influence of single-batch outliers, this invention employs a technique of generating field-level rule profiles and marking stable states. This integrates data source attributes, field structures, value characteristics, format automata, business constraints, and historical cleaning trajectories into ordered feature groups, distinguishing between stable features and features awaiting confirmation. New batches of data are then adapted to the corresponding rule profiles through multi-dimensional feature matching. This method eliminates the need for completely identical field names to complete rule matching, improving the rule adaptability for multi-source heterogeneous fields. Furthermore, the stable state marking mechanism prevents single-batch outliers from being solidified into cleaning rules, contributing to improved stability and adaptability of the cleaning rules.

[0024] 2. To address the issues of repetitive processing of unchanged data and high computational overhead in existing incremental data cleaning methods, this invention employs partition scanning and fingerprint comparison to locate changed segments. Data partitions are formed based on data source identifiers, business primary keys, and time windows. The degree of partition change is calculated by recording fingerprints and field fingerprints. Only changed partitions and necessary associated partitions are included in the cleaning process, while unchanged partitions are only written with version inheritance relationships. This approach skips repeated cleaning operations on unchanged data, reducing unnecessary computational load. Furthermore, for cross-partition constraint association with corresponding verification partitions, the integrity of constraint verification can be ensured while reducing processing overhead, thus improving the efficiency of incremental data cleaning. Attached Figure Description

[0025] Figure 1 This is a flowchart of a data incremental cleaning method based on rule-based profiling. Detailed Implementation

[0026] Example, refer to Figure 1 This embodiment of a rule-based data incremental cleaning method specifically includes the following steps:

[0027] Step S1: Generate field-level rule profiles.

[0028] The data source attributes, field structure, field values, formatting rules, business constraints, and historical correction history are uniformly transformed into field-level ordered features.

[0029] S11. Collect and standardize the original state of the fields:

[0030] For data source At the batch time The accessed data is parsed to obtain the field set, field declaration type, field original name, field position, field sample value and record primary key. The field declaration type is obtained from the access task description or the source data structure, and the actual field type is obtained by parsing the field sample value. When parsing the actual type, numerical parsing, date parsing, boolean parsing, enumeration parsing, encoded string parsing and ordinary text parsing are used in sequence to obtain the distribution of the actual field type.

[0031] For fields The original values ​​are only normalized for feature extraction and do not directly overwrite the original data. The normalization includes removing invisible control characters, unifying character encoding, identifying empty values ​​in the database, identifying empty strings, identifying source-end agreed-upon empty tags, preserving the original delimiter, and preserving the original case status. This yields the field missing status, format status, and comparable field value fingerprints.

[0032] Field missing rate By field The field duplication rate is calculated as the ratio of records identified as null to the total number of records included in the statistics. Determined by the recurrence of field value fingerprints within the same business primary key range or the same partition range; illegal format rate. The ratio is obtained by comparing the number of records whose field values ​​did not pass the format recognition rules with the number of records that participated in the format recognition. The above ratio is used as a field quality status feature and not as an indicator of cleaning effectiveness.

[0033] S12. Determine the value range, format automaton, and business constraints:

[0034] For fields The range of values The data is categorized and described as follows: numerical fields are described by stability boundaries, quantile intervals, median values, and dispersion; date fields are described by time boundaries, time granularity, inter-period status, and acceptable date templates; enumerated fields are described by the set of enumerated values, the order of occurrence of enumerated values, and low-frequency enumeration markers; and text fields are described by length range, character category, delimiter, prefix and suffix structure, and encoding rules. If a field value changes continuously, not all historical values ​​are saved individually; instead, the boundaries, distribution summary, and anomaly markers that can constrain the cleaning action are saved.

[0035] Formatting Automaton It is derived from field length constraints, character category constraints, date templates, numeric precision templates, encoding bit length templates, and separator templates; for cases where the same field has multiple valid formats, a format automaton sequence is formed according to the source format priority, historical pass frequency, and business effective time; when a field value is accepted by any format automaton, it is marked as a recognizable format; only when a field value is accepted but does not conform to the target output template will it enter the format normalization process;

[0036] Business constraints At a minimum, constraints should include field uniqueness, enumeration, range, cross-field dependency, version time, and source priority. Constraints are categorized as strong or weak. Strong constraints originate from fields that cannot be arbitrarily modified, such as business primary keys, legal codes, order numbers, account numbers, and document numbers. Weak constraints originate from statistically stable ranges, historical correction trends, and field relationships that can be interpreted by other fields. Strong constraints are prioritized for profile matching and conflict correction, while weak constraints are only considered as candidate judgment factors.

[0037] S13. Create a regular profile and mark its stability:

[0038] For data source Middle field At the batch time The regular portrait is denoted as an ordered feature group:

[0039] ;

[0040] in, Indicates data source Middle field At the batch time The rules of the portrait; This indicates the data source type, which is determined by the source channel, data organization format, and access protocol. The field declaration type is determined by the access task description; This indicates the actual data type distribution of the field, obtained by parsing the field sample values. The allowed value range of the field is determined by the range limited by business constraints, the historically stable value range, and the identifiable value range of the current batch. Indicates the missing rate; Indicates the repetition rate; Indicates the illegal format rate; Represents a field formatting automaton; Represents the set of business constraints; This indicates the historical cleaning trajectory of the field, which is found in the historical cleaning operation records related to the field. The corresponding completion, normalization, merging, replacement, and conflict correction actions are extracted in batch order.

[0041] After the portrait was created, and Each feature is marked as stable. If a feature changes within a preset stable band across multiple consecutive valid batches and the cleaned data passes the corresponding business constraint verification, then the feature is marked as a stable feature. If a feature only appears in the current batch or its changes cannot be explained by business constraints, then it is marked as a feature to be confirmed. Stable features participate in the main matching, while features to be confirmed only participate in auxiliary judgments to avoid outliers from a single batch being solidified into subsequent cleaning rules.

[0042] Step S2: Match the new batch fields with the rule profile.

[0043] Temporary features are generated for the fields in the new batch and matched with the existing rule profiles. The matching does not depend on the exact consistency of the field names, but rather on the source type, field type, value range, format status, and business constraint hit.

[0044] S21. Generate temporary features for new batch fields:

[0045] When new batch of data After entering the access buffer, for each field Perform the same feature extraction process as S11 and S12 to obtain a temporary feature group. The temporary feature group includes data source type, field name segmentation result, field declaration type, field actual type distribution, value range summary, missing rate, duplication rate, illegal format rate, format automaton hit result, and business constraint hit result.

[0046] If field If the sample size is insufficient to stably determine a certain feature, then that feature is marked as a default comparable state. The default comparable state does not directly reduce the matching result, but reduces the weight of the feature in the similarity calculation to prevent small sample fields from being incorrectly excluded.

[0047] S22. Calculate the matching degree between the calculated field and the candidate profile:

[0048] fields to be matched With candidate portrait field Calculate the matching degree:

[0049] ;

[0050] in, Representation field With candidate portrait field The matching degree is normalized to a closed interval. Dimensionless values ​​within; This indicates the number of features involved in the comparison, determined by the comparable features of this field. Indicates the first The weight of each feature item is determined by the feature's stable state, the strength of business constraints, and the relevance of the field type. Stable features and strongly constrained features correspond to higher weights. Indicates the first Similarity function of each feature term; Representation field The A temporary feature; Represents candidate profile field The One corresponding feature;

[0051] When the feature term is an enumerated set The feature is determined based on the degree of overlap of enumerated values ​​and the consistency of high-frequency enumerations; when the feature is a numerical range, it is determined based on the degree of overlap of the ranges and the closeness of the median value; when the feature is a format automaton, it is determined based on the situation where the sample values ​​are accepted by the same format automaton; when the feature is a type distribution, it is determined based on the closeness of the actual type confidence distribution; when the feature is a business constraint, it is determined based on the hit situation of strong constraints and the interpretation situation of weak constraints.

[0052] The matching threshold is denoted as This threshold is determined by the field fault tolerance requirements of the access task, the strong constraint field consistency requirements, and the acceptable matching boundaries in confirmed historical batches, and is not a fixed hard number; if achieve Then the candidate portrait can be used as a field. Adapted image;

[0053] S23. Handling multiple candidate and unmatched fields:

[0054] When multiple candidate portraits reach When the matching degree is close, the following factors are compared in turn: the number of strong constraint hits, the consistency of data source type, the stability of historical cleaning trajectory, the pass rate of the format automaton, and the timeliness of the profile version; if a unique determination still cannot be made, the field will be... Fields marked as having multiple candidates are only allowed to perform detection and annotation without changing the business meaning of the field; completion, replacement, or conflict correction are not allowed.

[0055] When no candidate portraits reach When, the field Mark it as a new field, and only perform null value annotation, format recognition, type recognition and non-destructive exception prompts on the new field; a new field-level rule profile is formed only after the new field appears consecutively in multiple valid batches and the field type, format and value range reach stable conditions;

[0056] S24. Generate a field-level cleaning task list:

[0057] After the profile matching is completed, a field-level cleaning task list is generated for the current batch. This list not only records the cleaning type, but also the field identifier, the matching profile identifier, the allowed actions, the prohibited actions, the triggering conditions, the dependent fields, the associated partitions, the write-back location, and the failure handling method.

[0058] If a field is subject to strong constraints, pre-correction and post-correction validations are set in the list; if a field involves cross-field dependencies, the dependent field is listed as a prerequisite field; if a field involves cross-partition uniqueness constraints, the associated partition is listed as a read-only validation partition, and the task list then enters the partition scanning phase.

[0059] Step S3: Perform a partitioned scan on the incremental data and locate the changed segments.

[0060] By using stable partitioning and fingerprint comparison to skip unchanged data, only changed partitions and their necessary associated partitions are cleaned at the field level, thus avoiding the need to repeatedly perform cleaning actions on the entire dataset.

[0061] S31. Generate stable partitions:

[0062] For new batches of data, partitions are formed according to data source identifier, business primary key, time window, field constraint relationship and hash bucket; if the business primary key is stable, the partition is formed based on the hash result of the business primary key; if the data has a business time field, the partition is formed by combining the time window; if a business constraint involves multiple fields, the relevant fields are included in the same scan unit.

[0063] When a record lacks a stable business primary key, a temporary comparison key is generated based on the identifier field, time field, and source field marked as stable in the field profile. The temporary comparison key is only used for partition and fingerprint comparison and is not used as a business primary key to write back the cleaned data.

[0064] S32. Generate record fingerprints and field fingerprints:

[0065] For each partition The records in the previous valid version and the current batch are read respectively. The record fingerprint is obtained by hashing the data source identifier, partition identifier, business primary key or temporary comparison key, the normalized original value of the cleaned field, the field position identifier and the batch identifier in a fixed order.

[0066] Field fingerprints are generated based on recorded fingerprints, and only cover fields that participate in rule profile matching and field-level cleaning. Fields not included in the cleaning task list do not participate in field fingerprint comparison. This can avoid unnecessary cleaning processes triggered by changes in irrelevant fields such as notes, import time, and non-business descriptions.

[0067] S33. Calculate the degree of zoning variation and determine the variation segments:

[0068] For partitions Calculate the degree of change:

[0069] ;

[0070] in, Indicates partition The degree of change; Indicates the partition in the previous valid version A collection of recorded fingerprints; Indicates the partitions in the current batch A collection of recorded fingerprints; Indicates the intersection, used to identify unchanged records; Represents the union, used to identify all records involved in the comparison; Indicates the number of elements in the set;

[0071] The partition change threshold is denoted as This threshold is determined by the data source update frequency, partition size, and the scope of the field cleaning task; if Not achieved This partition only writes version inheritance relationships and does not undergo field-level cleaning; if achieve The partition was marked as a changed segment, and further identification was performed on newly added records, modified records, deleted records, and suspected duplicate records.

[0072] If a field in a changed segment is affected by cross-partition constraints, a reverse search is established based on the constraint field to locate the relevant partition. The relevant partition only participates in constraint verification and candidate value search, and is not directly rewritten, thereby avoiding the destruction of global consistency due to local changes.

[0073] Step S4: Perform field-level incremental cleaning on the changed fragments:

[0074] This phase performs cleaning actions within the changed fragments and determines the processing order based on field dependencies. Cleaning actions include null value padding, format normalization, duplicate item merging, outlier replacement, and conflict field correction. Fields that do not meet the triggering conditions remain in their original state and are marked with observation tags.

[0075] S41. Determine the field cleaning order and save the state before cleaning:

[0076] Based on field business constraints Generate field dependencies; if fields The value of the field is determined If the validity is verified, the field will be processed first. Then process the fields If there is a circular dependency between multiple fields, the field with strong constraints will be processed first, followed by the field with weak constraints. If the constraint strength is the same, the field with higher source credibility and fewer historical conflicts will be processed first.

[0077] Before each field is cleaned, a snapshot of its state before cleaning is written. The snapshot includes at least the original field value, the actual field type, the format recognition result, the missing state, the duplicate state, the abnormal state, the conflict state, the matching profile identifier, the partition, the batch, and the record fingerprint. If cleaning fails or verification fails, the original state of the field is restored based on the snapshot, and the field is marked as pending review.

[0078] S42. Perform null padding and format normalization:

[0079] Records Middle field For null values, first consider the range of allowed values ​​for the field. The candidate value set is formed from the historical valid values ​​of the same business object, the non-null values ​​within the same context of the current change segment, and the deduced values ​​of cross-field constraints. The supplementary value is determined by the following formula:

[0080] ;

[0081] in, Representing records Middle field The padding value; Indicates the candidate values; Representation field The allowed range of values; Indicates in the recording context and rule portrait Candidate values ​​under certain conditions The conditional probability, which is a dimensionless value; Representing records In and field The set of non-empty fields with dependencies, whose scope is determined by Cross-field dependencies are determined within the context;

[0082] Conditional probability The data is obtained by combining historical stable batches from the same data source, non-empty samples within the current change segment, the previous valid version of the same business object, and records grouped with the same constraints. If the sample size is insufficient, smoothing is applied, with the smoothing strength determined by the data source size and the field's stability. The confidence threshold is then added. The risk level and field constraint strength that allow automatic completion by the business are determined; if the maximum conditional probability is not reached. Retain null values ​​and write them to the review pending status;

[0083] The formatting automaton only applies to field values ​​that are reversible and do not change the business meaning; the field values ​​first enter the formatting automaton. If the field value can be identified, If accepted, but does not conform to the target output template, then perform format conversions such as removing leading and trailing whitespace, standardizing date templates, standardizing code length, standardizing amount precision, standardizing case sensitivity, or standardizing separators; re-enter after conversion. and Only after verification, and if the semantic parsing values ​​before and after the transformation are consistent, will the cleaned field value be written. If the field value cannot be verified... Accept, only write the invalid format status, do not directly delete or replace;

[0084] S43. Perform duplicate item merging and outlier replacement:

[0085] Duplicate item merging first distinguishes between intra-field duplicates and record-level duplicates. Intra-field duplicates are used to handle synonyms that appear multiple times in the same field, while record-level duplicates are used to handle situations where multiple records correspond to the same business object. Candidate duplicate groups are jointly determined by the business primary key, stable identifier field, source field, time field, and core business field.

[0086] After identifying candidate duplicate groups, compare the completeness, number of strong constraint conflicts, update time, source credibility, and historical correction stability of each record; the main record should meet the requirements of fewer strong constraint conflicts, more complete core fields, and interpretable version time relationship; non-main records are not directly deleted, and their non-conflicting fields can be merged into the main record at the field granularity; merged fields save the source identifier and version identifier, and unmerged fields are saved as conflict candidates and enter into subsequent conflict correction;

[0087] For numeric fields Candidate outliers Calculate outlier using robust statistical methods:

[0088] ;

[0089] in, Representing records Middle field The degree of abnormality; This indicates the value of the field to be evaluated; Indicates reference window inner field The median value; Indicates reference window inner field The median absolute deviation; It is determined by the current change segment, the effective batches of homologous adjacent segments, and the records within the stable value range in the profile;

[0090] when When the value is zero or cannot be calculated stably, the anomaly degree formula is not used to directly replace the field value. Instead, anomaly labeling is performed solely based on business constraints, and the anomaly threshold is denoted as [value missing]. The degree of dispersion of the field, the business tolerance range, and the historical stability are determined; only when achieve And the field value also violates Replaceable outlier handling only occurs when there are interval constraints or cross-field constraints in the data; if the field value only shows statistical deviation but the business constraints can still be interpreted, the original value is retained and an anomaly observation mark is added.

[0091] Outlier replacement value from The candidate replacement value is selected from the previous valid value of the same business object, the nearest value in the same constraint group, and the deduced value of the business constraint; the candidate replacement value must also pass through the format automaton. and business constraints If multiple candidate values ​​meet the constraints, the candidate value that is more consistent with the midpoint value in the reference window or the historical change trend of the same business object will be selected first. If no candidate value passes the verification, the replacement will not be performed.

[0092] S44. Correct conflicting fields:

[0093] Conflict field correction covers multi-source conflicts of the same field, cross-field logical conflicts, and version conflicts. Multi-source conflicts of the same field refer to multiple data sources providing different values ​​for the same field of the same business object. Cross-field logical conflicts refer to situations where fields within the same record do not satisfy... Version conflict refers to a conflict between the field value of the current batch and the field value of the previous valid version, which cannot be explained by business time relationships.

[0094] For fields The set of candidate correction rules is denoted as This set is generated by the field formatting automaton. Business constraints and historical cleansing trajectory The selection process is completed, and the final execution rules are chosen using the following formula:

[0095] ;

[0096] in, Representation field The final selected correction rule; Indicates the candidate modification rule; Representation field The set of candidate correction rules; Indicates candidate modification rules The degree of matching with the current data version time is determined by the business activation time, data update time, and batch order; Indicates candidate modification rules The coverage of the current conflicting constraints is determined by the strong and weak constraints that are satisfied. Indicates candidate modification rules The credibility of a source is determined by source priority, field-level historical error status, and consistency with the most recent valid version. Indicates candidate modification rules The reversibility is determined by whether the state before the correction is completely saved, whether the state after the correction can be reversed by the rules, and whether it affects other fields; These are adjustment indices for time matching, constraint coverage, source credibility, and rollback, respectively. Their values ​​are determined by business rule configuration and field risk level, and are all dimensionless parameters.

[0097] The rule execution threshold is denoted as The value is determined by the importance of the conflicting field, the strength of the constraints, and the risk level that allows for automatic correction; if the comprehensive value of the candidate rule does not reach... If the automatic correction is not performed, the field will be written to the review pending state; if a new strong constraint conflict occurs after the correction is performed, the correction will be withdrawn and the relevant fields will be marked as conflict pending.

[0098] Step S5: Write the field status index, update the profile, and output the results.

[0099] By writing the cleaning actions, rule hits, and data version relationships into a traceable index, subsequent data fusion and value analysis can read data by version, field status, or rule identifier without rescanning the entire original data.

[0100] S51. Write the field status index and data version identifier:

[0101] For each field that has been inherited, observed, completed, unified, merged, replaced, conflict corrected, or suspended for review, write a field status index item. The index item shall include at least the data source identifier, batch identifier, partition identifier, record fingerprint, field identifier, field value summary before cleaning, field value summary after cleaning, field status change type, hit profile identifier, execution rule identifier, rule confidence status, cleaning action type, previous version identifier, and current version identifier.

[0102] Unchanged partitions are only written to the version inheritance index, and field-level correction details are not written repeatedly; fields that have changed but have not triggered cleaning are written to the observation status; fields that have undergone cleaning are written to both the differences before and after cleaning and the rule execution record; fields that are suspended for review retain their original field values ​​and are not written to the valid data area after cleaning.

[0103] The data version identifier for the current batch is generated using the following formula:

[0104] ;

[0105] in, This indicates the version identifier after the current batch of cleaning. Indicates the identifier of the previous valid version; This represents a summary of the field status before the current batch is cleaned. It is generated from the original field value summary, record fingerprint, and partition identifier in a fixed order. This represents a summary of the field status after the current batch is cleaned. It is generated from the summary of cleaned field values, the field status change type, and the output partition identifier. This represents a summary of the rules actually executed in the current batch, generated from the profile identifier, rule identifier, action type, and verification status. This indicates that the fields are concatenated according to a predetermined order. The hash function can be implemented using conventional irreversible hashing algorithms in this field; all items involved in the hashing are byte sequences, so both sides of the formula are version identifiers;

[0106] S52. Update the rule profile based on the cleaning results:

[0107] After the batch cleaning is completed, the rule profile is updated based on the data that has passed the cleaning and the review results. The update does not directly overwrite the original profile with the statistical results of the current batch. Instead, the changes in value range, format, missing status, duplicate status, abnormal status and conflict correction are first written into the confirmation area.

[0108] The features in the unconfirmed area consistently appear in multiple valid batches, and the cleaning results consistently pass through the formatting automaton. and business constraints After verification, the rules are transferred to the valid profile area. If a rule is frequently triggered but the number of pending reviews increases, the priority of that rule in profile matching and automatic execution will be reduced. If a rule has not been triggered for a long time and the corresponding field no longer appears or the business constraint has been replaced, the rule will be marked as low-activity. Low-activity rules do not participate in the main matching, but are retained in the historical version backtracking chain.

[0109] For new fields, a new field-level rule profile is generated only after its field type, format automaton, value range, and business constraints have stabilized in a continuous batch of valid data. Temporary fields, misaligned fields, test fields, and occasional abnormal columns are not included in the valid profile area.

[0110] S53. Output the cleaning results and provide subsequent calls:

[0111] After completing the above processing, the output includes cleaned incremental data, field status index, rule profile update record, and version relationship record. The cleaned incremental data only contains data fragments that have changed and passed the verification. The field status index is used to track the change process of a field from its original state to its cleaned state. The rule profile update record is used to explain the impact of the current batch on stable features, features to be confirmed, and low-activity rules.

[0112] Subsequent fusion processing can be based on The field status change type and rule identifier can be used to read data of a specified version; value analysis can directly read records in the field status index related to completion, normalization, merging, replacement, conflict correction and review suspension to determine the data quality changes of the current batch. Since only the version inheritance relationship is written to the unchanged partition, subsequent processing does not require repeated scanning of the full original data.

[0113] Through the detailed description of the above embodiments, the data incremental cleaning method based on rule profiling of the present invention integrates the multi-dimensional features of data fields with business constraints and historical correction trajectories into ordered feature groups by constructing a field-level rule profiling system, and is equipped with a stable state marking mechanism to achieve feature-based adaptation and dynamic iteration of cleaning rules. It achieves precise positioning of incremental change segments through partition-level fingerprint comparison, performing cleaning processing only on change areas and necessary related partitions, reducing the additional overhead of full-volume scanning. During the cleaning process, the processing order is determined according to field dependencies, and the accompanying state snapshot and multi-level verification mechanism ensure that the processing process is rollbackable and verifiable. Finally, the entire process is traceable through field state indexes and data version relationships, and the rule profiling is continuously optimized based on the cleaning results. This method can adapt to the cleaning needs of multi-source heterogeneous incremental data, balancing the adaptability of cleaning rules and the execution efficiency of incremental processing.

[0114] The preset parameters in the above formulas shall be set by those skilled in the art according to the actual situation.

[0115] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0116] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data incremental cleaning method based on rule-based profiling, characterized in that, The method flow is as follows: Step S1: Generate field-level rule profiles, transforming data source attributes, field structures, field values, format rules, business constraints, and historical correction trajectories into field-level ordered features, and marking the stable state of the features in the rule profiles. Step S2: Match the new batch fields with the rule profile, generate temporary features for the new batch fields, match the temporary features with the existing rule profile, and generate a field-level cleaning task list based on the matching results. Step S3: Perform partition scanning on the incremental data and locate the changed segments. Determine the changed partitions and necessary associated partitions to enter the cleaning process through stable partitions and fingerprint comparison. Step S4: Perform field-level incremental cleaning on the changed fragments, determine the processing order according to the field dependencies, and after saving the state before cleaning, perform null value filling, format normalization, duplicate item merging, outlier replacement or conflict field correction, and determine the state after cleaning based on the verification results. Step S5: Write the field status index and data version relationship, update the rule profile according to the cleaning results, and output the incremental data after cleaning, field status index, rule profile update record and version relationship record.

2. The data incremental cleaning method based on rule-based profiling according to claim 1, characterized in that, In step S1, generating a field-level rule profile includes: parsing the access data to obtain the field set, field declaration type, field original name, field position, field sample value, and record primary key; parsing the actual type distribution of the field based on the field sample value; performing only regularization processing for feature extraction on the original field value to obtain the field missing state, format state, and field value fingerprint, and determining the field quality state feature based on the field value fingerprint.

3. The data incremental cleaning method based on rule-based profiling according to claim 2, characterized in that, Step S1 further includes: determining the allowed value range of the field according to the field type, forming a field format automaton according to the field length constraint, character category constraint, date template, numerical precision template, encoding bit length template and separator template, and configuring a set of business constraints; the set of business constraints includes field uniqueness constraints, enumeration constraints, range constraints, cross-field dependency constraints, version time constraints and source priority constraints, and the set of business constraints is divided into strong constraints and weak constraints.

4. The data incremental cleaning method based on rule-based profiling according to claim 3, characterized in that, The field-level rule profile includes data source type, field declaration type, actual field type distribution, allowed field value range, field missing status, field duplicate status, illegal field format status, field format automaton, business constraint set, and field historical cleaning trajectory. After forming the field-level rule profile, the actual field type distribution, allowed field value range, field missing status, field duplicate status, illegal field format status, and field format automaton are marked as stable. Stable features participate in the main matching, while features to be confirmed participate in the auxiliary judgment.

5. The data incremental cleaning method based on rule-based profiling according to claim 1, characterized in that, In step S2, matching new batch fields with rule profiles includes: performing a feature extraction process corresponding to the field-level rule profile on the new batch fields to obtain a temporary feature group; matching the temporary feature group with the candidate rule profile, the matching process taking into account the data source type, field type, value range, format status and business constraint hit status; when the field sample size is insufficient to stably judge a certain feature, the feature is marked as a default comparable state, and the comparison weight corresponding to the feature is reduced during the matching process.

6. The data incremental cleaning method based on rule-based profiling according to claim 5, characterized in that, When multiple candidate rule profiles meet the matching conditions but cannot be uniquely determined, they are compared sequentially based on the number of strong constraint hits, data source type consistency, historical cleaning trajectory stability, format automaton pass rate, and profile version timeliness. If they still cannot be uniquely determined, the corresponding field is marked as a multiple candidate field, and its execution is restricted to detection and annotation that does not change the business meaning of the field. When no candidate rule profile meets the matching conditions, the corresponding field is marked as a new field, and a new field-level rule profile is formed after it continuously meets the stable conditions.

7. The data incremental cleaning method based on rule-based profiling according to claim 1, characterized in that, The field-level cleaning task list includes field identifier, adaptation profile identifier, allowed actions, prohibited actions, triggering conditions, dependent fields, associated partitions, write-back location, and failure handling method; When a field is subject to strong constraints, set up pre-correction and post-correction validations in the field-level cleaning task list; When a field involves cross-field dependency, the field that is depended on is listed as the preceding field; When a field involves a unique constraint across partitions, the associated partition is listed as a read-only validation partition.

8. The data incremental cleaning method based on rule-based profiling according to claim 1, characterized in that, In step S3, the partition scan includes: forming partitions according to the data source identifier, business primary key, time window, field constraint relationship, and hash bucket; when a record lacks a stable business primary key, a temporary comparison key is generated based on the identifier field, time field, and source field marked as stable in the rule profile. The temporary comparison key is only used for partition and fingerprint comparison; the partition change status is determined based on the record fingerprint and field fingerprint of the previous valid version and the current batch. Unchanged partitions are written into the version inheritance relationship, and changed partitions are marked as changed fragments; when a field in a changed fragment is affected by cross-partition constraints, the relevant partition is located, and the relevant partition participates in constraint verification and candidate value search.

9. The data incremental cleaning method based on rule-based profiling according to claim 1, characterized in that, In step S4, performing field-level incremental cleaning on the changed fragments includes: determining the field cleaning order based on business constraints, and writing a snapshot of the state before cleaning for each field; for null value fields, forming a candidate value set based on the allowed value range of the field, historical valid values ​​of the same business object, non-null values ​​in the same context within the current changed fragment, and cross-field constraint derivation values, and writing the fill-in value when the candidate value meets the fill-in condition; for formatted fields, performing format normalization only when the field value can be accepted by the field format automaton and does not conform to the target output template, and writing the cleaned field value after format normalization when it passes the field format automaton and business constraint verification and the semantic parsing value is consistent.

10. The data incremental cleaning method based on rule-based profiling according to claim 9, characterized in that, Step S4 further includes: for candidate duplicate groups, determining the master record based on record completeness, number of strong constraint conflicts, update time, source credibility, and historical correction stability, and merging non-conflicting fields of non-master records into the master record at the field granularity; for candidate outliers, performing outlier replacement when they meet the outlier judgment conditions and violate business constraints, and the candidate replacement value must be verified by the field format automaton and business constraints; for conflicting fields, forming candidate correction rules based on the field format automaton, business constraint set, and historical cleaning trajectory, and selecting correction rules based on version time relationship, constraint coverage, source credibility, and rollback status.

Images