Data quality optimization methods based on dynamic rule matching
By generating data context fingerprints and constructing rule scope graphs, the problems of rule mismatch and rule oscillation in existing technologies are solved, realizing dynamic rule matching for data quality optimization methods and improving the accuracy and efficiency of data processing.
Patent Information
- Application Number
- CN202611339307.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-01
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-09-01
AI Technical Summary
Existing data quality optimization methods cannot identify whether a rule is applicable based on the actual context of the data to be optimized, resulting in persistent rule mismatches and rule oscillations. They also cannot determine compatible execution paths when multiple rules are hit simultaneously, and statically bound rules easily identify legitimate data changes as quality defects.
By acquiring the field structure features, statistical distribution features, source identifiers, and processing stage identifiers of the dataset to be optimized, a data context fingerprint is generated. Preset data quality rules are converted into rule nodes, a rule scope graph is constructed, an applicable rule set is determined based on the data context fingerprint, candidate rule paths are generated, data state changes are deduced, field constraint satisfaction and distribution offset are calculated, and a target rule path is selected to perform data quality optimization.
This system eliminates the reliance on fixed field bindings or fixed priorities for rule selection, reduces the chance of incorrect corrections due to changes in legitimate distributions, avoids round-trip corrections on the same field, and ensures that rule application boundaries and relationships are updated with data status, preventing similar conflicts from recurring in consecutive batches.
Smart Images

Figure CN122862330B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing and data quality control technology, specifically a data quality optimization method based on dynamic rule matching. Background Technology
[0002] Data quality optimization typically relies on pre-configured data validation rules to perform field type validation, null value validation, value range validation, uniqueness validation, format validation, and field association consistency validation on datasets entering the processing flow. Existing systems generally bind various rules according to fixed priorities, fixed field ranges, or fixed processing stages. At runtime, the corresponding rule is invoked based on the data table name, field name, or data source identifier, and deletion, replacement, completion, format conversion, or association correction is performed upon detecting anomalies. For scenarios with a large number of rules, some solutions adjust the invocation order based on historical hit counts, manually set weights, or rule execution time. Other solutions divide data fields into several rule sets and then select the appropriate set based on field type. After rule execution, the system typically determines the correction result based on changes in the number of anomalies or the rule pass rate and writes the result back to the original dataset. Such solutions can complete routine quality checks, but the scope of application of the rules is mainly determined by static configuration. Changes in data sources, processing stages, joint distribution of fields, and upstream and downstream processing relationships are usually not uniformly encoded. The dependencies, conflicts, and common constraints between rules also lack structured expression, making it difficult to redetermine the rule combination and execution order when the data status changes.
[0003] Fixed-rule matching methods cannot identify the applicability of rules based on the actual context of the data to be optimized, nor can they determine compatible execution paths when multiple rules are matched simultaneously. The same field may have different constraints in different source versions, processing stages, or joint distribution states; statically bound rules easily identify legitimate data changes as quality defects. When multiple rules act on the same or related fields, the output of a previous rule may become the trigger condition for a subsequent rule, and rules with opposite correction directions can cause round-trip modifications. If only fixed priorities are applied, the constraint satisfaction of local fields may improve, but inter-field dependencies or data distribution may be disrupted. Because existing solutions lack data representations that simultaneously describe structural characteristics, distribution characteristics, source states, and processing stages, and also lack association structures that express rule dependencies, overlapping effects, and correction conflicts, the rule selection process cannot form a closed loop from applicability judgment, path construction, state deduction to boundary updates, leading to persistent rule mismatches and rule oscillations. Summary of the Invention
[0004] The purpose of this invention is to provide a data quality optimization method based on dynamic rule matching, which can solve the problems in the background art mentioned above.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] The data quality optimization method based on dynamic rule matching includes: acquiring the dataset to be optimized, extracting field structure features, statistical distribution features, source identifiers, processing stage identifiers and historical quality status, and generating data context fingerprints;
[0007] The preset data quality rules are converted into rule nodes that include applicable conditions, target fields, input constraints, output constraints, and correction directions. A rule scope diagram is constructed based on field overlap, prerequisite dependencies, and correction conflict relationships.
[0008] Based on the data context fingerprint, a set of applicable rules is determined, and candidate rule paths are generated in the rule scope graph.
[0009] The data state changes corresponding to each candidate rule path are deduced, the field constraint satisfaction, field association consistency and distribution offset are calculated, the target rule path is selected to perform data quality optimization, and the applicable conditions of the rule node are updated according to the difference between the actual output state and the corresponding output constraints.
[0010] Preferably, generating the data context fingerprint includes: constructing a structure sub-fingerprint, a distribution sub-fingerprint, a source sub-fingerprint, and a stage sub-fingerprint according to field granularity;
[0011] Encode the field type, null value pattern, uniqueness, value range, distribution skewness, inter-field dependency strength, data source version, and processing node sequence number respectively;
[0012] Fingerprint association edges are formed based on the dependencies between fields, and each sub-fingerprint and the fingerprint association edges are combined into a data context fingerprint with a hierarchical index.
[0013] When some features are missing, candidate representations of the missing features are determined based on data fingerprints from the same source and adjacent processing stages, and the uncertainty markers corresponding to the candidate representations are retained.
[0014] Preferably, constructing the rule scope graph includes: configuring a context application range, a set of field functions, pre-input constraints, post-execution output constraints, and a correction direction vector for each rule node;
[0015] A conflict edge is established when the field action sets of two rule nodes intersect and the correction direction vectors are opposite; a dependency edge is established when the output constraint of one rule node satisfies the pre-input constraint of another rule node; and a collaborative edge is established when multiple rule nodes jointly constrain the association relationship of the same field.
[0016] The rule nodes are layered according to the conflict edges, the dependency edges, and the cooperation edges to form a directed rule scope graph for the retrieval of candidate rule paths.
[0017] Preferably, generating the candidate rule path and deducing the data state change includes: taking a node with no preceding dependent rule in the applicable rule set as the starting node, extending the path along the dependent edge, generating a mutually exclusive branch when encountering the conflicting edge, and generating a joint execution branch when encountering the cooperative edge;
[0018] For each branch, a corresponding data state copy is created, and rule constraints in the path are applied sequentially to obtain the field constraint satisfaction, field association consistency, local distribution offset, and number of rule repetitions.
[0019] Paths that violate hard constraints are filtered out according to the order of multidimensional constraints, and the target rule path is determined from the remaining paths.
[0020] Preferably, the data context fingerprint with the hierarchical index is organized in the order of dataset layer, field group layer, and field layer;
[0021] The dataset layer records the source version, processing stage, and global distribution status; the field group layer records the set of fields with associated constraints and their joint distribution; and the field layer records the structure and value characteristics of a single field.
[0022] When matching rule nodes, rule nodes with inconsistent sources or stages are first excluded using the dataset layer, then association constraints are verified using the field group layer, and finally the degree of matching between the target field and the applicable context range is calculated using the field layer, and the uncertainty flag is passed to the corresponding matching result.
[0023] Preferably, the rule nodes are layered, including: performing topological decomposition on the rule nodes based on the dependency edges, and dividing rule nodes with the same pre-input constraints and acting on the same set of fields into candidate clusters of the same layer;
[0024] Within each candidate cluster at the same level, a mutually exclusive subgraph is constructed based on the conflicting edges, and a joint subgraph is constructed based on the cooperative edges;
[0025] The rule nodes in the mutually exclusive subgraph are mapped to different path branches, and the rule nodes in the joint subgraph that satisfy the output constraint compatibility condition are merged into composite path nodes.
[0026] The adjacent levels are connected according to the input and output constraints of the composite path nodes to form the retrieval skeleton of the candidate rule path.
[0027] Preferably, establishing the data state copy includes: extracting the target fields, related fields and their reference records involved in the candidate rule path from the dataset to be optimized, forming a local shadow dataset that maintains the original field dependency relationships;
[0028] For each rule node in the path, the state before rule execution and the state after rule execution are recorded in the local shadow dataset, and the state after execution of the previous rule node is used as the input state of the next rule node.
[0029] Independent state chains are generated for each of the mutually exclusive branches, and a shared state chain is generated for the joint execution branch. The constraint violation terms and distribution change terms corresponding to each state chain are used as the calculation inputs for data state changes.
[0030] Preferably, determining the target rule path includes: setting the field constraint satisfaction and field association consistency as primary decision factors, and setting the local distribution offset and the number of times the rule is repeatedly triggered as secondary decision factors;
[0031] First, delete the candidate rule paths that do not meet the preset constraint relationship according to the first-level decision quantity, and then perform a sequence comparison on the remaining candidate rule paths according to the second-level decision quantity;
[0032] When the secondary decision values of two candidate rule paths are the same, compare the context matching uncertainty and the coverage relationship of the applied fields of the corresponding rule nodes, and select the candidate rule path with low uncertainty and whose applied fields do not exceed the abnormal association range composed of the target field and the associated field.
[0033] Preferably, after the uncertainty marker is passed to the matching result, matching confidence intervals are established according to the dataset layer, the field group layer, and the field layer respectively;
[0034] When the rule execution result is consistent with the output constraint of the rule node, the matching confidence interval is shrunk based on the actual data characteristics of the corresponding level; when the rule execution result deviates from the output constraint, the matching confidence interval is expanded while retaining the original boundary.
[0035] When consecutive batches of data context fingerprints drift in the same direction, the data fingerprints of historically stable batches are set as anchor points, and the context application range of the rule nodes is adjusted based on the difference between the anchor point and the current data context fingerprint.
[0036] Preferably, after the target rule path is executed, the actual input state, actual output state, path branch, and subsequent rule triggering status of each rule node are written into the rule execution trajectory.
[0037] Based on the rules, the trajectory is used to identify round-trip correction sequences on the same field set, and the rule node combinations that form the round-trip correction sequences are marked as oscillation combinations;
[0038] During subsequent candidate rule path retrieval, the oscillation combination is written as a new mutual exclusion constraint into the corresponding candidate cluster at the same level, and the new mutual exclusion constraint is released only when the data context fingerprint exceeds the historical applicable range corresponding to the oscillation combination.
[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0040] 1. Field structure characteristics, statistical distribution characteristics, source identifiers, processing stage identifiers, and historical quality status are encoded as data context fingerprints. The applicable conditions, target fields, input constraints, output constraints, and correction directions of rules are organized into a rule scope graph. During rule matching, rules inconsistent with the current data state are first excluded based on the context fingerprint. Then, candidate rule paths are generated based on dependency edges, conflicting edges, and collaborative edges. Field constraint satisfaction, field association consistency, and distribution offset are compared using a copy of the data state. Therefore, rule selection no longer relies on fixed field bindings or fixed priorities, enabling the exclusion of inapplicable rules and conflicting rule combinations before execution. This reduces the possibility of erroneous correction of legitimate distribution changes and limits round-trip corrections on the same field.
[0041] 2. Hierarchical data context fingerprinting organizes matching information according to dataset layer, field group layer, and field layer, enabling layer-by-layer verification of source version, processing stage, joint distribution, and single field status. Uncertainty markers and matching confidence intervals are used to preserve matching boundaries under conditions of missing features or distribution drift. After the target rule path is executed, the actual input state, actual output state, path branch, and subsequent triggering conditions are written into the rule execution trajectory. The applicable range of rule nodes is adjusted according to the difference between the actual output and the expected constraints. For rule combinations that form a round-trip correction sequence, the system converts them into mutual exclusion constraints in subsequent path retrieval, ensuring that the rule application boundaries and relationships between rules are updated with the data status, preventing the recurrence of similar conflicts in consecutive batches. Attached Figure Description
[0042] Figure 1 A flowchart for the overall process of optimizing data quality based on dynamic rule matching;
[0043] Figure 2 This is a graph showing the executableness of candidate rule path structures.
[0044] Figure 3 A composition diagram of the status evaluation of candidate rule path data;
[0045] Figure 4 Update the trajectory map for flexible application of rule nodes to their boundaries;
[0046] Figure 5 Flowchart for data context fingerprint generation and hierarchical rule matching. Detailed Implementation
[0047] In one embodiment, a data quality optimization method based on dynamic rule matching includes: acquiring the dataset to be optimized; extracting field structure features, statistical distribution features, source identifiers, processing stage identifiers, and historical quality states; generating a data context fingerprint; converting preset data quality rules into rule nodes containing applicable conditions, target fields, input constraints, output constraints, and correction directions; constructing a rule scope graph based on field overlap relationships, prerequisite dependencies, and correction conflict relationships; determining an applicable rule set based on the data context fingerprint; generating candidate rule paths in the rule scope graph; deriving the data state changes corresponding to each candidate rule path; calculating field constraint satisfaction, field association consistency, and distribution offset; and selecting a target rule path to perform data quality optimization. The optimization process updates the applicable conditions of rule nodes based on the differences between the actual output state and the corresponding output constraints. The dataset to be optimized is a set of data records in the same quality processing batch with unified field description information. The data context fingerprint is used to characterize the real data state of the current batch of data. The rule scope graph is used to store the structural relationships between quality rules that can change the execution result. The candidate rule path is used to represent the combination of rules with an executable order in the current data state. The data state inference is completed in a copy isolated from the original data. The determination of the target rule path is simultaneously restricted by the field's own constraints, the field association constraints, and the data distribution preservation conditions. This makes rule selection, rule sorting, pre-execution verification, and post-execution boundary revision form a continuous data processing closed loop.
[0048] Specifically, please refer to Figure 1When acquiring the dataset to be optimized, the system reads the dataset identifier, data source version, field name, field type, allowed value range of the field, mapping relationship between fields, current processing node, and previous processing node from the input description of the data processing task. It then establishes a connection between data records, metadata, and historical quality records according to a unified batch identifier. The historical quality records at least store the missing status, exception type, executed rules, pre-correction value, post-correction value, and subsequent rule triggering status of the same field in historical batches. For datasets to be optimized consisting of multiple logical data tables, a cross-table field reference index is established according to the field mapping relationship, allowing fields belonging to different data tables but expressing the same data attribute to be grouped into the same context analysis scope. For fields with the same name but different sources, processing stages, or semantic identifiers, different field identity codes are assigned to prevent incorrect matching of rules to fields with different data meanings based solely on field name. After reading, the system performs consistency verification between metadata and actual data. If a field declared in the metadata does not exist or the actual field lacks descriptive information, the corresponding field is marked as a context field to be supplemented, and the original data is retained; correction rules are not directly invoked.
[0049] Field structure features include field data type, length form, null value distribution, unique value ratio, repetition pattern, character composition, numerical precision form, and inter-field reference relationships. Statistical distribution features include positional distribution, dispersion, skewness, quantile intervals, category frequency, anomaly concentration intervals, and joint value status. Source identifiers include data source identity, data source version, extraction task identifier, and upstream output batch. Processing stage identifiers include the current transformation node, completed transformation type, and subsequent processing target. Historical quality status includes historical anomaly categories, historical rule hit sequences, historical correction stable states, and historical rollback records. Before entering the data context fingerprint, each type of feature is encoded according to a defined data type. Categorical features use discrete encoding and retain unknown category positions. Continuous features undergo scaling transformation based on the reference interval formed by historical stable batches. Set features generate ordered representations according to member identity and relationships between members. Missing features are not replaced with fixed constants but instead written with missing position markers, candidate value ranges, and candidate sources, enabling subsequent matching calculations to distinguish between two states: a true value of zero and a feature that cannot be obtained.
[0050] In this embodiment, the degree of matching between the data context fingerprint and the applicable conditions of the rule node is calculated according to the following formula:
[0051] ;
[0052] in, This represents the matching value between the rule node with number r and the current data context fingerprint, where m represents the number of feature dimensions involved in the matching. This represents the preset weight of the j-th feature dimension in the rule application judgment. This indicates whether the j-th feature has a definite value; it is set to 1 if it has a definite value, and a confidence value between 0 and 1 if it has a candidate representation or is in a missing state. This indicates the degree of consistency between the current feature value and the applicable range of the rule node in the j-th dimension. A value of 1 indicates that the feature falls completely within the applicable range; a value decreasing based on distance indicates that the feature falls within the transition range; and a value of 0 indicates that the feature exceeds the acceptable range. This represents the total uncertainty created by the uncertain features involved in the matching. The weights represent the constraint coefficients of uncertainty on the matching values. These weights are only used to express the strength of the relationship between each feature and the applicable conditions of the rule, and are not used as running parameters that can be adjusted independently of the field semantics.
[0053] For example, a numerical field's source version, processing stage, field type, and joint distribution state correspond to four matching dimensions, with weights of 0.3, 0.25, 0.2, and 0.25 respectively. The current source version, processing stage, and field type are all consistent with the rule's applicable conditions, so the consistency level is 1 for each. The joint distribution state is between the applicable and exclusionary intervals, so the consistency level is 0.6. When all four dimensions have definite values, the corresponding confidence value is 1, and the total uncertainty is 0. Substituting these values yields a matching value of 0.9. When the joint distribution state can only form a candidate representation due to missing upstream statistical information, the corresponding confidence value is 0.5, and a total uncertainty is generated, causing the matching value to decrease. The rule node can enter the candidate set but is marked with an uncertainty tag. Subsequent path selection requires comparing this uncertainty with the state deduction results of other rule paths to avoid directly treating candidate rules as definite applicable rules when features are incomplete.
[0054] When converting preset data quality rules into rule nodes, the detection object, triggering condition, correction object, correction value generation method, pre-execution data requirements, and post-execution data requirements in the rule are parsed. The detection object and correction object are uniformly mapped to the target field set. The triggering condition is decomposed into applicable conditions and input constraints. The field state that must be met after correction is converted into output constraints. The change of field value from the current state to the correction state is represented as the correction direction. The applicable conditions limit the source, stage, structure, and distribution state in which the rule can participate in matching. The input constraints limit the data state that the target field and related fields must have when the rule is executed. The output constraints limit the field conditions and related conditions that must be met after execution. The correction direction includes at least value increase, value decrease, null value filling, format normalization, category mapping, record removal, and relationship reconstruction. Different correction types use discrete direction encoding. Rules involving numerical changes also record the change range. Rules involving category changes record the category migration relationship. Rules involving related fields record the common change relationship between the main field and the subordinate field.
[0055] The rule scope graph uses a directed attribute graph to store rule nodes and the relationships between them. Each rule node has a unique rule identifier and is associated with a target field set, applicable context, input constraints, output constraints, and correction direction. When the target field sets of any two rule nodes have an intersection and the correction direction causes opposite changes to the intersection fields, a conflict edge is established. When the output constraint of one rule node can satisfy the input constraint of another rule node, a dependency edge is established from the former to the latter. When multiple rule nodes act on different fields but jointly maintain the same field relationship, a collaborative edge is established. Rule nodes whose target fields do not have an intersection and there is no constraint transmission between input and output do not establish a direct relationship. Implicit conflicts are also detected during the graph construction process. Implicit conflicts refer to two rule nodes that, although they do not directly act on the same field, modify fields with functional dependencies or value mapping relationships, and the modified fields cannot simultaneously satisfy the relationship. For implicit conflicts, conflict edges with associated field identifiers are established, so that the conflict identification scope covers direct field overlap and indirect effects transmitted through field relationships.
[0056] When determining the applicable rule set based on data context fingerprints, exclusion matching is first performed according to data source identifier, processing stage identifier, and field type. Rule nodes whose source version is incompatible with the rule node's limited version, whose current processing stage is outside the rule's allowed stage, or whose target field type cannot bear the rule's output value are excluded. Then, matching values are calculated based on structural features, distribution features, and historical quality status. For rule nodes that meet the minimum applicability relationship, the matching values, feature dimensions involved in the matching, and uncertainty markers are retained to form the applicable rule set. The minimum applicability relationship is not a separately set fixed numerical condition, but is determined by the necessary applicability conditions in the rule node. If any necessary applicability condition is violated, the rule is directly excluded. If all necessary applicability conditions are met but the extended applicability conditions are uncertain, the rule is retained as a candidate. This makes the source, stage, and field carrying capacity constitute hard applicability boundaries, while the statistical distribution and historical quality status constitute flexible applicability boundaries that can be revised according to the execution results.
[0057] When generating candidate rule paths in the rule scope graph, a rule node without unmet prerequisite dependencies is selected from the applicable rule set as the starting node. Subsequent rule nodes are connected along the dependency edges. When a conflicting edge is encountered, the conflicting rule nodes are placed in different mutually exclusive path branches. When a cooperative edge is encountered, it is checked whether the output constraints of each rule node can be met simultaneously. If they can be met simultaneously, the relevant rule nodes are combined into a joint execution branch. If they cannot be met simultaneously, the cooperative relationship is converted into a conditional mutually exclusive relationship. Each candidate rule path saves the rule execution order, the edge type between rule nodes, the union of the applied fields, and the expected output constraint set. For rule nodes with circular dependencies in the graph, the entry node that can be directly executed is determined based on the strictness of the input constraints and the original state of the target field. Circular dependency combinations for which the entry node cannot be determined are not included in this batch of candidate rule paths and are written into the pending review relationship record for processing when the applicable conditions are updated.
[0058] In this embodiment, the structural executableness of the candidate rule path is calculated according to the following formula:
[0059] ;
[0060] in, This represents the structured executable value of the candidate rule path with the number p. This represents the set of dependency edges contained in the candidate rule path. This indicates the degree to which the output constraints of rule node u satisfy the input constraints of rule node v. A value of 1 indicates complete satisfaction; a value between 0 and 1 indicates partial satisfaction that can be supplemented by subsequent joint nodes; and a value of 0 indicates no satisfaction. This represents the set of conflicting node pairs that still need to be checked within the path. This indicates the degree of conflict between the outputs of rule node x and rule node y. A value of 1 indicates that the outputs cannot coexist, while a value of 0 indicates that there is no conflict. Represents a set of cooperative node pairs. This indicates the degree of compatibility of the output constraints when rule node g and rule node h are executed together. The product operation is used to ensure that any non-executable dependency or completely conflicting relationship can exclude the corresponding path.
[0061] Please refer to Figure 2 , Figure 2 The horizontal axis represents the candidate rule path number, showing 16 candidate paths from PATH-001 to PATH-016. The vertical axis represents the structural executable strength of the candidate rule path, with a value ranging from 0 to 1. The bar textures and the right-side block legend represent different executable intervals: dark solid bars correspond to the priority execution area, diagonal bars correspond to the candidate observation area, and cross-textured bars correspond to the blocking and review area. Figure 2 The long gray dashed line in the diagram represents the candidate entry line, and the gray dotted line represents the priority execution line. These lines are used to determine whether the path meets the conditions for entry state deduction and priority execution sorting.
[0062] from Figure 2 As can be seen, 12 out of the 16 candidate paths have a structurally executable strength exceeding the candidate entry line, indicating that the dependency satisfaction, conflict resolution, and cooperative edge compatibility of these paths can form a basically continuous rule execution chain. The pillar heights of paths PATH-003, PATH-005, PATH-009, PATH-010, PATH-014, and PATH-015 are close to 0.70, significantly higher than the candidate entry line, indicating that the connection between the preceding output constraints and subsequent input constraints of their rule nodes is relatively smooth, and no completely incompatible conflict relationships appear. Paths PATH-001, PATH-006, PATH-011, and PATH-016 fall into the blocking verification zone, indicating that there are cases of insufficient dependency satisfaction, conflict nodes failing to resolve, or low compatibility of cooperative node output constraints within these paths.
[0063] When a candidate rule path contains two dependent edges and one set of collaborating nodes, the satisfaction level of the two dependent edges is 1 and 0.8 respectively. The compatibility level is 1 when the output constraints of the collaborating nodes can be met together, and there are no unresolved conflicts in the path. The structure executable value is 0.8. When the satisfaction level of any dependent edge is 0, the structure executable value is 0, and the path is excluded before state deduction. When the collaborating nodes require the same associated field to maintain two incompatible category states, the compatibility level is 0, and the corresponding path is also excluded. The structure executable value does not directly indicate the quality of the data. It is only used to confirm whether the rule combination has a complete execution chain at the input and output constraint level, and to avoid creating a data state copy for rule combinations that cannot be completed in terms of structure.
[0064] When deriving the data state changes corresponding to each candidate rule path, the target fields involved in the path, the related fields that have a dependency relationship with the target fields, and the records that reference the fields are extracted from the dataset to be optimized to form a local shadow dataset. The local shadow dataset retains the field data type, record identity, field reference relationship and original value, but is isolated from the original data writing channel. Each rule node executes the same value generation logic as the formal correction on the local shadow dataset. Before the rule is executed, an input snapshot is saved, and after the rule is executed, an output snapshot is saved. The output snapshot of the current rule node is passed to the subsequent rule nodes in the path. For mutually exclusive branches, independent state chains are established. For joint execution branches, serial or parallel derivation is determined according to whether the rule action fields overlap. When the action fields overlap, serial derivation is performed according to the dependency relationship. When the action fields do not overlap but jointly maintain the association constraint, the output is calculated separately and then the joint constraint verification is performed.
[0065] Field constraint satisfaction is calculated based on rule output constraints, field metadata constraints, and existing relation constraints within the dataset. Field association consistency is calculated based on functional dependencies, mapping relationships, allowed range of joint values, and cross-table reference relationships. Distribution offset is calculated based on the differences in category frequency, quantile position, and joint distribution before and after execution on the local shadow dataset. During calculation, only the data range actually affected by the rule path and the data range directly related to it are compared to avoid diluting local changes with large-scale unchanged data. The number of times a rule is repeatedly triggered is obtained by re-running the detection conditions in the applicable rule set. When the same rule or a rule with the opposite correction direction is triggered again after state deduction, the corresponding number of triggers is recorded, and the triggering field, triggering condition, and preceding rule node that triggered the trigger are written into the state chain, so that path selection can identify subsequent anomalies and round-trip correction tendencies after a single correction.
[0066] In this embodiment, the data status evaluation value of the candidate rule path is calculated according to the following formula:
[0067] ;
[0068] in, This represents the data status evaluation value of candidate rule path p. Indicates the degree of constraint satisfaction of the field. This indicates consistency in field associations. Indicates the distribution offset. This indicates the number of times the rule is repeatedly triggered. This represents the accumulated uncertainty in path matching. , , , and These represent the constraint coefficients of the corresponding evaluation values within the scope of the current rule. When field metadata constraints and field association constraints are conditions that must be met, they are processed first through path filtering logic. The evaluation values are only compared in order of candidate rule paths that have met hard constraints to prevent small distribution offsets from offsetting field constraint violations.
[0069] Please refer to Figure 3 , Figure 3 The horizontal axis represents the candidate rule path number after structural screening, and the vertical axis represents the data status evaluation value and its component contribution. The stacked bars are indicated by the right-hand box legend: field constraint contribution and association consistency contribution are positive components, while distribution offset deduction, duplicate trigger deduction, and uncertain deduction are negative components; the black dotted solid line represents the comprehensive evaluation value after summarizing all candidate paths. Figure 3 This is used to demonstrate how candidate paths are ranked and compared based on multidimensional state evaluation results after being screened out by hard constraints.
[0070] from Figure 3 As can be seen, the overall evaluation values of the candidate paths are mainly concentrated between 0.49 and 0.57, with little overall difference, but the composition of different components has significant explanatory value. The average contribution of field constraints is about 0.31, which is the main positive source of the overall evaluation value, indicating that the degree to which the path output satisfies constraints such as field type, necessary values, uniqueness, and referencing relationships determines whether the path can maintain a high evaluation. The average contribution of association consistency is about 0.24, indicating that the ability of candidate paths to maintain functional dependencies, mapping relationships, and joint value states is also a key factor in the ranking. The overall magnitude of distribution offset deduction, repeated trigger deduction, and uncertainty deduction is small, but they can still lower the overall evaluation value in some paths. In particular, repeated trigger deduction is used to identify the tendency of round-trip corrections that may trigger reverse rules again after a single correction.
[0071] For example, candidate path A has a field constraint satisfaction of 1, a field association consistency of 0.9, a distribution offset of 0.1, a rule repetition trigger of 0, and an uncertainty of 0.05. Candidate path B has a field constraint satisfaction of 1, a field association consistency of 0.7, a distribution offset of 0.02, a rule repetition trigger of 1, and an uncertainty of 0. When the association consistency is set as a hard constraint and the lower limit is allowed to be higher than 0.7, candidate path B is directly excluded and does not participate in the evaluation value comparison. When both paths meet the hard constraints, the path that maintains the field association state and does not generate repetition triggers is selected according to the evaluation value. This avoids selecting rule combinations that will trigger subsequent corrections simply because the local distribution change is small.
[0072] When selecting a target rule path, candidate rule paths with incompatible field types, invalid necessary output constraints, broken cross-field reference relationships, or deterministic conflicts after rule execution are first deleted. Then, the paths are compared according to field constraint satisfaction, field association consistency, distribution offset, number of repeated rule triggers, and matching uncertainty. When multiple candidate rule paths have the same evaluation order, the path with the range of affected fields not exceeding the abnormal association range and a small number of rule nodes is selected. The abnormal association range consists of the initial abnormal field, fields that have a direct dependency relationship with the initial abnormal field, and fields that are explicitly required to be corrected by the rule output constraints. After the target rule path is determined, the same correction is performed on the original data according to the rule order in the state deduction. For each write, the rule identifier, input state, output state, affected field, context fingerprint, and successor and predecessor rule nodes are saved for verification and updating of applicable conditions after execution.
[0073] When updating the applicability conditions of rule nodes based on the difference between the actual output status and the corresponding output constraints, the field status, associated field status, subsequent rule detection results, and distribution status after the target rule path is formally executed are read. The actual output is compared item by item with the output constraints recorded in the rule node. When the actual output meets the output constraints and does not cause the conflict rule to be retried, the current data context fingerprint is written into the rule node as a stable applicable sample. When the actual output does not meet the output constraints or causes the reverse correction rule to be triggered, the current data context fingerprint is written into the rule node as a boundary sample. The applicability condition update only adjusts the feature dimensions related to the actual difference and does not move the entire applicable range due to the result of a single execution. Hard applicability conditions such as source identifier, field type, and processing stage are only updated when the rule definition changes or the data processing chain changes. Soft applicability conditions such as statistical distribution, historical quality status, and joint value range are updated based on the execution results of consecutive batches.
[0074] In this embodiment, the boundary of the flexible application range of the rule node is updated according to the following formula:
[0075] ;
[0076] in, This represents the applicable boundary of rule node r after updating in the j-th flexible feature dimension. Indicates the applicable boundaries before the update. This represents the step size coefficient for a single boundary update. This indicates the result of rule execution. It is set to 1 if the actual output satisfies the output constraints and does not cause subsequent conflicts, and to a negative value if the actual output deviates from the output constraints. This represents the feature value of the current data context fingerprint in the j-th dimension. This represents the suppression coefficient of the abnormal execution record on the boundary update magnitude. This indicates the proportion of abnormal execution of rule nodes in adjacent historical batches. The update method causes the boundary to move slowly towards stable applicable samples and limits the change of a single batch of data to the applicable range when there are many abnormal executions in history.
[0077] Please refer to Figure 4 , Figure 4 The horizontal axis represents the consecutive batch numbers, and the vertical axis represents the flexible application boundary values of the regular nodes. The upper coordinate system shows the trajectory of the updated boundary as the batch changes; the black solid dotted line represents the updated boundary, and the gray dotted line represents the mean reference line. The lower coordinate system shows the corresponding trajectory of the boundary before the update; the long dashed square dotted line represents the boundary before the update, and the gray dotted line also represents the mean reference line of the updated boundary. The two graphs share the same horizontal axis to display the same batch sequence, making it easy to observe the movement relationship and update magnitude of the boundary before and after the update.
[0078] from Figure 4 As can be seen, the updated boundary is mainly distributed between 0.54 and 0.71, while the unupdated boundary is mainly distributed between 0.58 and 0.68. This indicates that the flexible application boundary did not drift drastically due to the feature values of a single batch, but rather moved to a limited extent under the combined effect of stable sample promotion and abnormal execution suppression. The upper update boundary curve exhibits a small oscillating upward and downward alternating pattern. When the rule execution result satisfies the output constraints and does not trigger subsequent conflicts, the boundary slowly moves towards the fingerprint feature value direction of the current data context. When the execution result deviates from the output constraints or the proportion of historical anomalies is high, the update amplitude is suppressed, and the curve either retracts or remains unchanged. The fluctuation range of the lower unupdated boundary curve is relatively concentrated, indicating that the historical boundary provides a stable anchor point for this batch of updates.
[0079] When the applicable boundary before the update is 0.6, the current feature value is 0.8, the step size coefficient is 0.2, the rule execution result flag is 1, the abnormal execution ratio is 0, and the suppression coefficient is 0.5, the boundary after the update is 0.64. The applicable range of the rule moves to the current stable sample in a limited way. When the rule execution result deviates from the output constraint and the execution result flag is negative, the boundary moves in the direction of excluding the current sample. When there are many abnormal execution records in adjacent historical batches, the update amplitude is suppressed to prevent continuous abnormal batches from driving the applicable range to drift rapidly. The boundary update record and the corresponding data context fingerprint are saved together so that the corresponding historical boundary can be selected in subsequent batches according to the data source version and processing stage.
[0080] In a preferred embodiment, please refer to Figure 5The generation of data context fingerprints includes constructing structural sub-fingerprints, distribution sub-fingerprints, source sub-fingerprints, and stage sub-fingerprints at the field granularity. Field type, null value pattern, uniqueness, value range, distribution skewness, inter-field dependency strength, data source version, and processing node sequence number are encoded separately. Fingerprint association edges are formed based on inter-field dependencies. Each sub-fingerprint and its association edges are combined into a data context fingerprint with a hierarchical index. When some features are missing, candidate representations of the missing features are determined based on data fingerprints from the same source and adjacent processing stages, and the uncertainty markers corresponding to the candidate representations are retained. The structural sub-fingerprint records the field carrying form with fields as basic nodes; the distribution sub-fingerprint records the value state of fields and field groups; the source sub-fingerprint stores version and batch relationships along the data source chain; and the stage sub-fingerprint stores the current data's position in the processing flow. Fingerprint association edges include field dependency edges, field mapping edges, and processing stage propagation edges. The hierarchical index enables rule matching to locate specific target fields layer by layer from the overall state of the dataset.
[0081] The optimal boundary for coarse-grained fields is selected using an adaptive aggregation algorithm based on information gain: Starting with a single field as the initial fine-grained unit, candidate field groups are generated from the bottom up based on the strength of inter-field dependencies, business semantic relevance, and joint distribution entropy. After each aggregation, the ratio of rule matching accuracy gain to computational cost increment is calculated. Aggregation stops when this ratio falls below a preset gain threshold, and the current aggregation result is determined as the coarse-grained field boundary. The information gain calculation formula is as follows:
[0082]
[0083] In the formula, S is the current set of fields, and F is the candidate field to be aggregated. Let S be the joint distribution entropy. For the set of fields after adding F, The number of fields in the field set. For a collection of fields The joint distribution entropy.
[0084] The structural sub-fingerprint is implemented using fixed-length discrete one-hot encoding: the field data type, length level, null value pattern, unique value level, and character category are encoded separately, with encoding bits of 8 bits, 4 bits, 4 bits, 4 bits, and 6 bits respectively, which are then concatenated to form a 26-dimensional structural feature vector; among them, the null value pattern is divided into four levels according to the field missing rate: no missing, low missing, medium missing, and high missing; and the unique value level is divided into four levels according to the proportion of unique values in the field: fully unique, highly unique, medium unique, and low unique.
[0085] The distribution sub-fingerprint is implemented using quantile statistical coding: seven statistical measures are extracted from the continuous numerical field, namely the minimum value, lower quartile, median, upper quartile, maximum value, skewness coefficient, and kurtosis coefficient. After min-max normalization, they are mapped to a 7-dimensional continuous feature vector. For the category field, the top 10 main categories are selected from high to low frequency, and the proportion of each category is concatenated to form a 10-dimensional category feature vector. If the number of categories is less than 10, zero values are used to fill the gaps.
[0086] The source sub-fingerprint is implemented using hierarchical hash encoding: the data source identity, mode version number, and conversion version number are each mapped to a 32-bit fixed-length hash value, which is then concatenated into a 96-bit source feature vector in the hierarchical order of "data source identity-mode version-conversion version". At the same time, the batch number hash value of the upstream output batch is recorded to completely trace the data source chain.
[0087] The stage sub-fingerprint is implemented using topological order coding: a global topological sequence number is assigned to each processing node based on the directed acyclic graph of the data processing chain, and the current processing node sequence number, the completed transformation type code, and the subsequent processing target code are concatenated into a 16-dimensional stage feature vector; among them, the transformation type is mapped to 4-bit binary code according to four categories: format conversion, data cleaning, data aggregation, and field association.
[0088] The strength of inter-field dependencies is determined based on the continuous state of common changes in field values in historical stable batches. For deterministic functional dependencies, the set of determining fields, the determined fields, and allowed mapping relationships are recorded. For non-deterministic associations, the joint value range, conditional distribution state, and abnormal combinations are recorded. The data source version adopts a version representation formed by the combination of source identity, schema version, and transformation version. The processing node sequence number is generated based on the directed relationship in the data processing chain rather than directly sorted based on runtime. When multiple processing nodes are in parallel, the common predecessor node and branch identity are recorded for each parallel node. Candidate representations of missing features are extracted from adjacent processing stages of the same data source version. If the corresponding feature exists in both the preceding and subsequent stages, it is mapped to the current stage in reverse or forward according to the field transformation relationship. If the mapping results are inconsistent, multiple candidate values and their respective sources are retained simultaneously, and a single value is not forcibly selected when generating fingerprints.
[0089] Preferably, the data context fingerprint with hierarchical index is organized sequentially into a dataset layer, a field group layer, and a field layer. The dataset layer records the source version, processing stage, and global distribution status. The field group layer records the set of fields with associated constraints and their joint distribution. The field layer records the structure and value characteristics of a single field. When matching rule nodes, rule nodes with inconsistent sources or stages are first excluded by the dataset layer. Then, the associated constraints are verified by the field group layer. Finally, the matching degree between the target field and the applicable context range is calculated by the field layer, and the uncertainty flag is passed to the corresponding matching result. The dataset layer uses batch identifiers to connect all field group layer nodes. The field group layer divides field groups according to functional dependencies, business code mappings, or joint constraint relationships. The same field can belong to multiple field groups with different relational meanings at the same time. The field layer stores a structural sub-fingerprint and a distribution sub-fingerprint for each field. Rule nodes are associated with the lowest level required for their scope of action. Rules that act on the global distribution are associated with the dataset layer. Rules that maintain the relationships between fields are associated with the field group layer. Rules that correct the format or value of a single field are associated with the field layer.
[0090] The dataset layer exclusion matching uses hard condition comparison. When the source identity is inconsistent and there is no version inheritance relationship, the processing stage is not in the rule-allowed stage set, or the global data form conflicts with the rule input requirements, the associated rule node will not enter the next layer of matching. During the field group layer verification, the field relationship expression is extracted from the input constraints of the rule node, and the joint value state of the current field group is compared with the relationship expression. When some fields in the field group are missing, the missing state is converted into group-level uncertainty, and the association constraint is not directly determined to be valid. After the field layer matching calculation is completed, the uncertainty of the dataset layer, field group layer and field layer is accumulated into the rule node matching result according to the source path, so that the uncertainty caused by the upper layer missing can affect the path selection of the lower layer rule, avoiding the problem of ignoring the data source or the incomplete state of the associated field when the single field state matches the rule condition.
[0091] In another preferred embodiment, constructing a rule scope graph includes configuring a context applicable range, a set of field applications, pre-input constraints, post-execution output constraints, and a correction direction vector for each rule node. A conflict edge is established when the field application sets of two rule nodes intersect and their correction direction vectors are opposite. A dependency edge is established when the output constraint of one rule node satisfies the pre-input constraints of another rule node. A collaborative edge is established when multiple rule nodes jointly constrain the same field association relationship. The rule nodes are layered according to the conflict edges, dependency edges, and collaborative edges to form a directed rule scope graph for candidate rule path retrieval. The correction direction vector consists of the identity of the applied field and the field state change type. For numerical fields, the change direction and allowed change range are recorded. For category fields, the category set before migration and the category set after migration are recorded. For null value processing, the migration from a missing state to a definite state is recorded. For record deletion rules, the impact of deleted records on associated fields and aggregation results is identified.
[0092] A vector data retrieval method is used to achieve rapid matching between rule nodes and data context fingerprints. The correction direction vector is constructed by combining the field identifier vector and the direction encoding vector: first, the field identity encoding of the field to be applied is mapped to a one-hot vector with the same dimensions as the total number of global fields as the field identifier vector; then, the field state change type is mapped to the direction encoding vector of the corresponding dimension; the two are multiplied bitwise to obtain the final correction direction vector. The direction encoding vector of the numeric field includes three dimensions: value increase, value decrease, and state unchanged; the direction encoding vector of the category field includes three dimensions: category mapping, category addition, and category deletion; the direction encoding vector for null value processing includes three dimensions: null value filling, null value retention, and null value removal. The total dimensions of the vector are consistent with the total number of change types involved in the rule node.
[0093] The vector data retrieval in the rule matching stage is implemented using a hierarchical approximate nearest neighbor algorithm: the applicable conditions of all rule nodes are pre-encoded into rule feature vectors consistent with the data context fingerprint dimension, and a hierarchical vector retrieval index is constructed; during matching, the data context fingerprint vector of the current batch is used as the query vector, and a coarse recall is first performed in the dataset layer vector space to filter out candidate rule nodes that match the source and stage, and then fine ranking is performed in the field group layer and field layer vector spaces through cosine similarity calculation to retrieve candidate rule nodes with similarity higher than a preset threshold. Finally, the applicable rule set is obtained through hard condition filtering; during the retrieval process, a weight attenuation coefficient is applied to the feature dimension carrying uncertainty label, and the attenuation coefficient is positively correlated with the feature credibility.
[0094] The pre-input constraints and post-output constraints adopt a unified constraint expression structure, which includes constraint objects, comparison relationships, reference objects, allowed states, and constraint ranges. This ensures that the output constraints of one rule node can be aligned item by item with the input constraints of another rule node. When the output constraint completely covers the input constraints of the subsequent rule, a definite dependency edge is established. When the output constraint only covers part of the input constraints and the remaining conditions can be satisfied by the original data, a conditional dependency edge is established. When the output state is contradictory to the subsequent input state, a conflict edge is established. The establishment of a cooperative edge requires that the fields modified by multiple rule nodes appear together in the same association constraint, and that executing any rule alone cannot make the association constraint valid. After the graph is constructed, relational closure analysis is performed to identify indirect dependencies formed by multi-level dependency propagation and indirect conflicts formed by intermediate fields, and the indirect relationships are saved in the path retrieval index.
[0095] Furthermore, the hierarchical structure of rule nodes includes performing topological decomposition on rule nodes based on dependent edges. Rule nodes with the same pre-input constraints and acting on the same set of fields are divided into candidate clusters at the same level. Within each candidate cluster at the same level, a mutually exclusive subgraph is constructed based on conflicting edges, and a joint subgraph is constructed based on cooperative edges. Rule nodes in the mutually exclusive subgraph are mapped to different path branches. Rule nodes in the joint subgraph that satisfy the output constraint compatibility condition are merged into composite path nodes. Adjacent levels are connected according to the input and output constraints of the composite path nodes to form the retrieval skeleton of candidate rule paths. The topological decomposition is based on whether the input constraints depend on the output of other rules, not on the rule type or rule creation time. Rule nodes in the candidate cluster at the same level can participate in the selection under the same input state. The mutually exclusive subgraph stores node combinations that cannot enter the same path together, and the joint subgraph stores node combinations that need to be executed together to satisfy the association constraints.
[0096] Composite path nodes do not change the execution logic of internal rule nodes. Instead, they use rule nodes with common input states and compatible output states as overall candidates for path retrieval. When internal rules apply to different fields, they can be deduced separately in independent shadow data branches and then merged for output. When internal rules apply to the same field, the order is determined based on the dependency edges. Output constraint compatibility conditions are determined by checking the intersection of allowed values of the same field, whether the cross-field relationship can satisfy the state, and whether the record range overlaps. If some rule outputs in the joint subgraph are incompatible, the joint subgraph is split into multiple combinations of the most compatible nodes, and each combination forms a composite path node. When connecting adjacent levels, the combined output constraints of the composite path nodes of the previous level are compared with the combined input constraints of the nodes of the next level, and only connections with complete constraint transitive relationships are retained.
[0097] In an extended embodiment, generating candidate rule paths and deducing data state changes includes using a rule node without pre-dependent dependencies in the applicable rule set as the starting node, extending the path along the dependent edges, generating mutually exclusive branches when encountering conflicting edges, generating joint execution branches when encountering cooperative edges, establishing a corresponding data state copy for each branch, sequentially applying rule constraints in the path, obtaining field constraint satisfaction, field association consistency, local distribution offset, and rule repetition trigger count, filtering out paths with hard constraint violations according to the multi-dimensional constraint order, and determining the target rule path from the remaining paths. The rule node without pre-dependent dependencies includes nodes that do not depend on the output of other rules and nodes whose preconditions have been satisfied by the original data. During the path extension process, the visited rule nodes and the current field state are saved. If the next rule node has appeared and the current field state is the same as the previous visited state, the extension is stopped to prevent the formation of a loop path without state changes.
[0098] Hard constraints include field data type constraints, non-nullable constraints, uniqueness constraints, deterministic mapping constraints, master-slave reference constraints, and necessary output constraints explicitly stated by rule nodes. Soft constraints include statistical distribution preservation conditions, historical stable state proximity conditions, and rule scope control conditions. The order of multidimensional constraints is established by prioritizing hard constraints over soft constraints and field relationship constraints over single-item distribution proximity conditions. If any hard constraint is violated in the path, it will not be compensated by other evaluation quantities. The remaining paths will first compare field association consistency, then compare local distribution offset and repeated triggering times. When there is an uncertainty marker in the path, the corresponding uncertainty source will be retained. If the uncertainty involves the input required by the hard constraint, the path will be transferred to the verification branch and more stringent output verification will be adopted. It will not be directly treated as a path with a complete input state.
[0099] Furthermore, establishing a data state replica involves extracting the target fields, associated fields, and their referenced records involved in the candidate rule paths from the dataset to be optimized, forming a local shadow dataset that maintains the original field dependencies. For each rule node in the path, the pre-execution state and post-execution state are recorded in the local shadow dataset, and the post-execution state of the previous rule node is used as the input state of the next rule node. Independent state chains are generated for mutually exclusive branches, and shared state chains are generated for joint execution branches. The constraint violation terms and distribution change terms corresponding to each state chain are used as the calculation input for data state changes. The referenced records are determined through field dependency indexes and cross-table reference indexes. When a target field is referenced by an aggregated field, records participating in the same aggregation range are extracted simultaneously. When a target field is used as a mapping key, the dependent records associated with the mapping key are extracted simultaneously, ensuring that the local shadow dataset contains the minimum closure data required to determine the consequences of rule output.
[0100] The state before rule execution includes the original value of the field, the state of the field structure, the values of associated fields, the current constraint satisfaction state, and the current distribution position. The state after rule execution includes the corrected value, the source of the corrected value, the state of the affected associated fields, the newly added constraint violations, and the eliminated constraint violations. The state chain saves adjacent states with rule nodes as connection points, so that any final anomaly can be traced back to the preceding rule that caused the change. The shared state chain in the joint execution branch uses the same initial snapshot. Each rule node generates transient output. Before merging the transient output, field write conflicts are checked. When there are different write values for the same field, the merging order is determined according to rule dependencies and output constraints. When a unique merged state cannot be formed, the joint branch is converted into a mutually exclusive branch.
[0101] In a preferred embodiment, determining the target rule path includes setting field constraint satisfaction and field association consistency as primary decision factors, and setting local distribution offset and rule repetition trigger count as secondary decision factors. First, candidate rule paths whose primary decision factors do not meet the preset constraint relationship are deleted. Then, the remaining candidate rule paths are compared sequentially according to the secondary decision factors. When the secondary decision factors of two candidate rule paths are the same, the context matching uncertainty and the coverage relationship of the corresponding rule nodes are compared. The candidate rule path with lower uncertainty and whose applied field does not exceed the abnormal association range composed of the target field and the associated field is selected. The primary decision factors are recorded separately for each constraint item rather than being merged into a single value. If any necessary constraint item is not met, the path is deleted. Field association consistency is checked at the same time to check the relationship between the corrected field and the uncorrected associated field to prevent unreasonable corrections from being covered up by simultaneously modifying multiple fields.
[0102] Local distribution offsets are calculated based on the same record range before and after the rule takes effect. For continuous fields, the position of the quantile interval and the order of local density are compared. For categorical fields, the structure of the category and the conditional category relationship are compared. For field groups with joint constraints, the joint state set is compared to see if any new combinations that have not appeared in history and are not allowed by the rule are added. The number of repeated rule triggers is distinguished between unidirectional and reverse unidirectional triggers. Unidirectional triggers indicate that a single correction has not reached the output constraint. Reverse triggers indicate that the rule combination has an oscillation tendency. Both are saved separately. When the secondary decision quantities are the same, the coverage relationship of the applied field is preferentially selected from the smaller range limited by the dependency closure of the abnormal field to avoid the rule path from extending the correction to fields that do not have abnormal propagation relationships.
[0103] Furthermore, after passing the uncertainty marker to the matching result, matching confidence intervals are established according to the dataset layer, field group layer, and field layer respectively. When the rule execution result is consistent with the output constraint of the rule node, the matching confidence interval is shrunk based on the actual data characteristics of the corresponding layer. When the rule execution result deviates from the output constraint, the matching confidence interval is expanded while retaining the original boundary. When the data context fingerprints of consecutive batches drift in the same direction, the data fingerprints of historically stable batches are set as anchor points. The context application range of the rule node is adjusted according to the difference between the anchor point and the current data context fingerprint. Among them, the confidence interval of the dataset layer limits the combination range of source version, processing stage, and global distribution state; the confidence interval of the field group layer limits the field association state; and the confidence interval of the field layer limits the single field structure and distribution state. The confidence intervals of each layer are updated respectively and form a complete application range through hierarchical reference.
[0104] When the rule execution results are consistent, only the feature dimensions that generate uncertainty and have been verified by the actual output are shrunk. The feature dimensions that did not participate in the current rule execution judgment are kept in their original range. When the execution result deviates from the output constraint, the current feature position is recorded as an exclusion candidate, but the original applicable range is not directly deleted. A stable exclusion boundary is formed after consecutive batches have the same deviation state. The data fingerprint anchor points of historical stable batches are saved according to the data source version and processing stage. When the current data context fingerprint and anchor point change continuously in the same direction and the rule execution result continues to meet the output constraint, the applicable range is translated along the drift direction. When the rule execution begins to deviate from the output constraint during the drift process, the translation is stopped and the corresponding position is set as the boundary sample to distinguish between legitimate data state migration and data state changes that cause the rule to fail.
[0105] In another extended embodiment, after the target rule path is executed, the actual input state, actual output state, path branch, and subsequent rule triggering status of each rule node are written into the rule execution trajectory. Based on the rule execution trajectory, round-trip correction sequences on the same field set are identified, and the combination of rule nodes forming the round-trip correction sequence is marked as an oscillation combination. During subsequent candidate rule path retrieval, the oscillation combination is written as a new mutual exclusion constraint into the corresponding candidate cluster at the same level, and the new mutual exclusion constraint is released only when the data context fingerprint exceeds the historical applicable range corresponding to the oscillation combination. The rule execution trajectory is indexed according to data batch, record range, and field set, and the field change direction is saved between the actual input state and the actual output state, so that opposite modifications implemented by different rules on the same field can be directly identified.
[0106] The identification of round-trip correction sequences is based on the condition that the field state is restored to a historical state or loops within a finite set of states. When rule node A modifies the field state from state one to state two, and then rule node B restores the same field or a deterministically related field from state two to state one, it is recorded as a round-trip correction. When multiple rule nodes cause the field state to loop between more than three states and eventually return to the existing state, it is recorded as a multi-node oscillation. In addition to storing the set of rule nodes, the oscillation combination also stores the rule order, the fields involved, the triggering context fingerprint range, and the input constraints that cause the oscillation. During subsequent path retrieval, the newly added mutual exclusion constraints only apply to the applicable range of the same context and the relationship of the same field, preventing the oscillation record formed under a certain data state from preventing the rule combination from being executed normally in different sources or different processing stages.
[0107] After adding a mutual exclusion constraint to the candidate cluster at the same level, the path finder places the rule nodes in the oscillation combination on different branches when constructing the mutual exclusion subgraph. If the oscillation combination contains dependencies that cannot be directly split, the preceding rule node is retained and the subsequent connection that causes state reversal is blocked. When the data context fingerprint exceeds the historical applicable range, the mutual exclusion constraint is released but the original combination is not directly allowed to be written into the formal data. Instead, it is restored to the candidate combination to be deduced. The local shadow dataset is re-established and it is checked whether round-trip correction is still formed under the current data state. When the re-deduction does not result in state reversal and all output constraints are valid, the applicable boundary of the oscillation combination is updated. When the re-deduction still results in state reversal, the historical applicable range is expanded, thereby maintaining the correspondence between the oscillation constraint and the specific data context.
[0108] In actual data processing, after the dataset to be optimized enters the current batch, a dataset-level fingerprint is generated. Then, based on field dependencies, field group-level fingerprints and field-level fingerprints are formed. Rule nodes undergo hierarchical applicability matching in sequence. Rule nodes with inconsistent source versions, processing stages, or field carrying formats are excluded. The path retrieval range for rule nodes to enter the rule scope graph is retained. The path retrieval device reads dependent edges, conflicting edges, cooperative edges, and historically added mutual exclusion constraints to construct multiple candidate rule paths. Each path establishes an independent state copy and executes correction logic consistent with the formal rule. After the state inference results are filtered out by hard constraints and compared with multi-dimensional states, the target rule path is determined. After the target rule path is formally written into the original data, a rule execution trajectory is formed. The actual output is used to revise the flexible application boundary, confidence interval, and oscillation combination application range.
[0109] In practical applications, a certain field uses a fixed category code in the historical data source version, and a legal category is added in the new data source version. Static rules may uniformly map the newly added category to the historical category. In this embodiment, the data source version and category distribution status are identified by source sub-fingerprint and distribution sub-fingerprint. Old version rule nodes are excluded because the source application conditions are not met. If there are category integrity rules and field association rules for the new version, the two form a collaborative branch in the rule scope graph based on the field relationship maintained by both. The state inference simultaneously checks the legal mapping of the newly added category in the associated field and the corrected category distribution. Only when the output constraints are compatible and there is no destruction of the association relationship will the target rule path be entered, so that the legal category change and the coding error are distinguished in the rule matching stage.
[0110] When the same numerical field hits both the out-of-bounds correction rule and the missing value compensation rule, the rule node records the input constraints and correction direction respectively. The out-of-bounds correction rule requires the field to have a definite value and exceed the allowed range, while the missing value compensation rule requires the field to be in a missing state. Under normal circumstances, the two will not form a candidate combination under the same input state. When the previous format conversion rule may convert illegal characters into null values, the format conversion rule and the missing value compensation rule form a dependency relationship. Whether the format conversion rule and the out-of-bounds correction rule establish a dependency relationship depends on the output state after conversion. The rule scope graph generates different candidate paths accordingly. The local shadow dataset respectively deduces the state changes of post-conversion compensation and post-conversion range verification to avoid sending fields that have not yet formed values directly into the out-of-bounds correction rule in a fixed order.
[0111] For cases where there is a deterministic mapping relationship between fields, even if a single-field correction rule can meet the format constraints of the target field itself, the mapping consistency must be checked at the field group level. When the rule path only modifies the decided field and not the deciding field, and the correction value does not belong to the allowed mapping set of the current deciding field, the hard constraint of field association consistency does not hold, and the corresponding path is deleted. When another candidate path corrects both the deciding field and the decided field at the same time through a collaborative rule, it is necessary to check whether the correction basis comes from the same original exception and whether the output status falls into the allowed mapping relationship. Only after the conditions are met will the joint execution branch be retained. This ensures that field-level corrections cannot be separated from field group constraints and obtain execution permission independently.
[0112] For cases where the distribution of continuous batch data changes slowly, historical stable fingerprints serve as anchors to preserve the distribution status under each source version and processing stage. The difference between the current batch fingerprint and the anchor enters the matching confidence interval. When the output constraints are continuously satisfied after the rule is executed and no conflicting rules are triggered, the flexible applicable range moves to a limited extent according to the current feature direction. When a sudden change occurs in the current batch but is not continued in subsequent batches, a single boundary sample will not change the overall applicable range. When continuous batches drift in the same direction, the applicable boundary is gradually updated with the stable execution record. Therefore, the rule can adapt to the continuous migration of data state while retaining the hard boundary composed of source, stage, and field type.
[0113] In this embodiment, various rule definitions, data context fingerprints, rule scope diagrams, state deduction results, rule execution trajectories, and applicable condition update records are all stored using traceable data objects. Each data object is associated with at least a data batch identifier, data source version, processing stage, and field identity code. When a rule node definition changes, a new rule version is generated. The old rule version and its historical applicable boundaries are not overwritten. The new rule version reads reusable historical stable samples according to the inheritance relationship, but it needs to be re-deduced before it can be written into the formal data. When the data source mode changes, a new source version is generated and the field identity mapping is re-established to avoid the applicable conditions accumulated under the old field identity being directly migrated to the field whose semantics have changed.
[0114] In a preferred implementation, when rule execution fails, the output state is uncertain, or the local shadow dataset lacks the records required to complete the association verification, the candidate rule path is not executed on the original data. The failure location, the missing reference range, and the completed state chain are saved. After the required data is supplemented, the process is re-deduce from the failure node. During the re-deduce, the previous state unaffected by the missing data is reused, but the state that depends on the missing reference is recalculated. If the same rule node fails to form a definite output repeatedly in multiple batches due to the same input conditions, the input conditions are recorded as applicable boundary gaps, and the corresponding uncertainty is increased during subsequent matching to prevent unverifiable rule paths from repeatedly entering the formal execution stage in consecutive batches.
[0115] Furthermore, when a rule node is removed, disabled, or replaced, the rule scope graph synchronously deletes or redirects its associated dependency edges, conflict edges, and cooperative edges. The composite path nodes involving the rule node are re-expanded, and the input-output relationships between the remaining rules are checked. When the historical oscillation combination contains a removed rule node, the oscillation record is not directly deleted. Instead, the field status return information is retained, and it is re-determined whether the remaining rules still constitute an oscillation. After the rule version is replaced, the new version rule node inherits the field action set and relation edge candidates of the old version. However, the parts where the context application range, output constraints, and correction direction have changed are recalculated so that the rule definition change will not cause the remaining invalid relationships in the rule scope graph.
[0116] Through the above implementation, the rule selection in the data quality optimization process is limited by the data context fingerprint, the executable relationship between rules is limited by the rule scope graph, the pre-execution state verification is provided by the local shadow dataset, the candidate path is screened by the multi-dimensional constraint order, and the rule application boundary and oscillation constraints are revised by the actual execution trajectory. Each processing object is associated with the other through data batch identifiers, field identity codes, rule identifiers and context fingerprints. Any rule correction can trace its matching basis, path location, input state, output state and subsequent triggering state, which constitutes a data quality optimization implementation method that can continuously handle changes in source, stage, field association and distribution migration.
Claims
1. A method for data quality optimization based on dynamic rule matching, characterized in that, include: Obtain the dataset to be optimized, extract field structure features, statistical distribution features, source identifiers, processing stage identifiers, and historical quality status, and generate data context fingerprints, including: constructing structure sub-fingerprints, distribution sub-fingerprints, source sub-fingerprints, and stage sub-fingerprints according to field granularity; Encode the field type, null value pattern, uniqueness, value range, distribution skewness, inter-field dependency strength, data source version, and processing node sequence number respectively; Fingerprint association edges are formed based on the dependencies between fields, and each sub-fingerprint and the fingerprint association edges are combined into a data context fingerprint with a hierarchical index. When some features are missing, candidate representations of the missing features are determined based on data fingerprints from the same source and adjacent processing stages, and the uncertainty markers corresponding to the candidate representations are retained. Field structure features include field data type, length format, null value distribution, unique value ratio, repetition pattern, character composition, numerical precision format, and inter-field referencing relationships; The preset data quality rules are converted into rule nodes that include applicable conditions, target fields, input constraints, output constraints, and correction directions. A rule scope diagram is constructed based on field overlap, prerequisite dependencies, and correction conflict relationships, including configuring the context application range, field scope set, prerequisite input constraints, post-execution output constraints, and correction direction vector for each rule node. A conflict edge is established when the field action sets of two rule nodes intersect and the correction direction vectors are opposite; a dependency edge is established when the output constraint of one rule node satisfies the pre-input constraint of another rule node; and a collaborative edge is established when multiple rule nodes jointly constrain the association relationship of the same field. The rule nodes are layered according to the conflict edges, the dependency edges, and the cooperation edges to form a directed rule scope graph for candidate rule path retrieval. Based on the data context fingerprint, a set of applicable rules is determined, and candidate rule paths are generated in the rule scope graph. The data state changes corresponding to each candidate rule path are deduced, the field constraint satisfaction, field association consistency and distribution offset are calculated, the target rule path is selected to perform data quality optimization, and the applicable conditions of the rule node are updated according to the difference between the actual output state and the corresponding output constraints. The degree of matching between the data context fingerprint and the applicable conditions of the rule node is calculated according to the following formula: ; in, This represents the matching value between the rule node with number r and the current data context fingerprint, where m represents the number of feature dimensions involved in the matching. This represents the preset weight of the j-th feature dimension in the rule application judgment. This indicates whether the j-th feature has a definite value; it is set to 1 if it has a definite value, and a confidence value between 0 and 1 if it has a candidate representation or is in a missing state. This indicates the degree of consistency between the current feature value and the applicable range of the rule node in the j-th dimension. A value of 1 indicates that the feature falls completely within the applicable range; a value decreasing based on distance indicates that the feature falls within the transition range; and a value of 0 indicates that the feature exceeds the acceptable range. This represents the total uncertainty created by the uncertain features involved in the matching. The weight represents the constraint coefficient of uncertainty on the matching value. The weight is only used to express the strength of the relationship between each feature and the applicable conditions of the rule, and is not used as an operating parameter that is adjusted independently of the field semantics.
2. The data quality optimization method based on dynamic rule matching according to claim 1, characterized in that, Generating the candidate rule path and deducing the data state change includes: taking the node with no preceding dependency rule in the applicable rule set as the starting node, extending the path along the dependency edge, generating a mutually exclusive branch when encountering the conflict edge, and generating a joint execution branch when encountering the cooperative edge; For each branch, a corresponding data state copy is created, and rule constraints in the path are applied sequentially to obtain the field constraint satisfaction, field association consistency, local distribution offset, and number of rule repetitions. Paths that violate hard constraints are filtered out according to the order of multidimensional constraints, and the target rule path is determined from the remaining paths.
3. The data quality optimization method based on dynamic rule matching according to claim 2, characterized in that, The data context fingerprint with the hierarchical index is organized sequentially into the dataset layer, field group layer, and field layer. The dataset layer records the source version, processing stage, and global distribution status; the field group layer records the set of fields with associated constraints and their joint distribution; and the field layer records the structure and value characteristics of a single field. When matching rule nodes, rule nodes with inconsistent sources or stages are first excluded using the dataset layer, then association constraints are verified using the field group layer, and finally the degree of matching between the target field and the applicable context range is calculated using the field layer, and the uncertainty flag is passed to the corresponding matching result.
4. The data quality optimization method based on dynamic rule matching according to claim 3, characterized in that, Layering the rule nodes includes: performing topological decomposition on the rule nodes based on the dependency edges, and dividing rule nodes with the same pre-input constraints and acting on the same set of fields into candidate clusters of the same layer; Within each candidate cluster at the same level, a mutually exclusive subgraph is constructed based on the conflicting edges, and a joint subgraph is constructed based on the cooperative edges; The rule nodes in the mutually exclusive subgraph are mapped to different path branches, and the rule nodes in the joint subgraph that satisfy the output constraint compatibility condition are merged into composite path nodes. The adjacent levels are connected according to the input and output constraints of the composite path nodes to form the retrieval skeleton of the candidate rule path.
5. The data quality optimization method based on dynamic rule matching according to claim 4, characterized in that, Establishing the data state copy includes: extracting the target fields, related fields and their referenced records involved in the candidate rule path from the dataset to be optimized, forming a local shadow dataset that maintains the original field dependency relationships; For each rule node in the path, the state before rule execution and the state after rule execution are recorded in the local shadow dataset, and the state after execution of the previous rule node is used as the input state of the next rule node. Independent state chains are generated for each of the mutually exclusive branches, and a shared state chain is generated for the joint execution branch. The constraint violation terms and distribution change terms corresponding to each state chain are used as the calculation inputs for data state changes.
6. The data quality optimization method based on dynamic rule matching according to claim 5, characterized in that, Determining the target rule path includes: setting the field constraint satisfaction and field association consistency as primary decision factors, and setting the local distribution offset and the number of times the rule is repeatedly triggered as secondary decision factors; First, delete the candidate rule paths that do not meet the preset constraint relationship according to the first-level decision quantity, and then perform a sequence comparison on the remaining candidate rule paths according to the second-level decision quantity; When the secondary decision values of two candidate rule paths are the same, compare the context matching uncertainty and the coverage relationship of the applied fields of the corresponding rule nodes, and select the candidate rule path with low uncertainty and whose applied fields do not exceed the abnormal association range composed of the target field and the associated field.
7. The data quality optimization method based on dynamic rule matching according to claim 6, characterized in that, After passing the uncertainty marker to the matching result, matching confidence intervals are established according to the dataset layer, the field group layer, and the field layer respectively; When the rule execution result is consistent with the output constraint of the rule node, the matching confidence interval is shrunk based on the actual data characteristics of the corresponding level; when the rule execution result deviates from the output constraint, the matching confidence interval is expanded while retaining the original boundary. When consecutive batches of data context fingerprints drift in the same direction, the data fingerprints of historically stable batches are set as anchor points, and the context application range of the rule nodes is adjusted based on the difference between the anchor point and the current data context fingerprint.
8. The data quality optimization method based on dynamic rule matching according to claim 7, characterized in that, After the target rule path is executed, the actual input state, actual output state, path branch, and subsequent rule triggering status of each rule node are written into the rule execution trajectory. Based on the rules, the trajectory is used to identify round-trip correction sequences on the same field set, and the rule node combinations that form the round-trip correction sequences are marked as oscillation combinations; During subsequent candidate rule path retrieval, the oscillation combination is written as a new mutual exclusion constraint into the corresponding candidate cluster at the same level, and the new mutual exclusion constraint is released only when the data context fingerprint exceeds the historical applicable range corresponding to the oscillation combination.
Citation Information
Patent Citations
Ontology constraint knowledge graph pipeline self-adaptive arrangement and feedback method and system
CN122596205A
Artificial intelligence-based adaptive big data storage and retrieval optimization method and system thereof
US20260037537A1
