A multi-dimensional data verification automatic execution method based on a rule engine

Abstract

The present application relates to the technical field of data verification, and specifically relates to a multi-dimensional data verification automatic execution method based on a rule engine, comprising the following steps: S1: obtaining a data object to be verified, a technical verification rule deposited by a company, and a business verification rule corresponding to a business requirement, identifying a bottom-layer availability rule, and generating a rule dependency locking relationship; S2: performing a pre-availability detection on the data object to be verified, identifying a verification blocking item, generating a rule influence propagation chain, determining a trusted verification interval and a damaged verification interval, and generating a verification trusted boundary list; S3: automatically generating a Python-encapsulated SQL verification code for the technical verification rule and the business verification rule that are not frozen, and running the code to obtain an effective verification result; outputting a result to be reviewed for the business consistency rule that is frozen, and writing the effective verification result and the result to be reviewed into an Excel version multi-dimensional data verification report. The present application improves verification efficiency, problem positioning efficiency, and the accuracy of data governance.

Description

Technical Field

[0001] This invention relates to the field of data verification technology, and in particular to a method for automatically executing multidimensional data verification based on a rule engine. Background Technology

[0002] With the continuous advancement of data warehouse construction, data governance system construction, and integrated business and financial management, enterprises need to perform quality and consistency verification on a large amount of data during business analysis, financial accounting, regulatory reporting, and business operations. Existing data verification technologies typically generate SQL scripts based on preset rules and complete tasks such as field integrity checks, primary key uniqueness checks, total-to-score reconciliation, business-finance reconciliation, reference reconciliation, status reconciliation, and type reconciliation automatically. In practical applications, technical verification rules and business verification rules often originate from different management systems. Technical verification focuses on data quality requirements such as data structure, field type, field integrity, and field association validity, while business verification focuses on business logic consistency requirements. Existing technologies typically convert both types of rules into verification code for direct execution and output anomaly reports based on the execution results, thereby achieving data quality monitoring and business data verification.

[0003] However, existing technologies generally assume that the data to be verified meets the basic data conditions required for business verification. When the underlying data has problems such as missing fields, mismatched field types, duplicate primary keys, broken association keys, or abnormal data ranges, the business verification rules will still continue to be executed, resulting in a large number of surface business anomalies caused by the propagation of underlying data defects. There is a lack of a dependency analysis mechanism between business consistency rules and underlying availability rules, making it impossible to identify which business verification results are generated on the basis of reliable data, which business verification results have been affected by data quality defects, and also impossible to track the impact process of the propagation of underlying anomalies to business verification results. Summary of the Invention

[0004] This invention provides a method for automatically executing multi-dimensional data verification based on a rule engine. It can identify verification blocking factors that affect the credibility of business rules before business verification is executed, establish a credible verification boundary for business consistency rules, and distinguish between real business anomalies and results to be reviewed caused by data defects.

[0005] An automated method for multidimensional data verification based on a rules engine includes the following steps: S1: Obtain the data object to be verified, the technical verification rules accumulated by the company, and the business verification rules corresponding to the business requirements. Parse the technical verification rules into underlying availability rules for judging the validity of data structure, field type, field integrity, and field association. Parse the business verification rules into business consistency rules for judging total score verification, business and finance verification, reference verification, status verification, rule verification, and type verification. Identify the underlying availability rules that each business consistency rule depends on and generate rule dependency locking relationships. S2: Using the rule dependency locking relationship as input, perform a preliminary availability probe on the data object to be verified, identify verification blocking items that prevent the business consistency rule from being truly verified due to missing fields, mismatched types, duplicate primary keys, broken associations, or abnormal data ranges; based on the participation path of each verification blocking item in the corresponding business consistency rule, trace the impact range of the verification blocking item on the business consistency rule verification process, and generate a rule impact propagation chain; determine the trusted verification interval and damaged verification interval of each business consistency rule based on the rule impact propagation chain, and generate a list of trusted verification boundaries; S3: Based on the verification trust boundary list, automatically generate and run the SQL verification code wrapped in Python for the unfrozen technical verification rules and business verification rules to obtain valid verification results; output the pending verification results triggered by the corresponding verification blocking item for the frozen business consistency rules, and write the valid verification results and pending verification results into the Excel version of the multidimensional data verification report respectively.

[0006] Optionally, S1 specifically includes: S11: Read the metadata and data content of the data object to be verified from the data source interface or data warehouse, load the predefined set of technical verification rules from the rule base, and read the set of business verification rules for six types of organizations: total score verification, business and finance verification, reference verification, status verification, rule verification, and type verification from the business configuration end. S12: For each technical verification rule, extract the judgment conditions for four types of verification items: data structure, field type, field integrity, and field association validity. Encapsulate the judgment conditions into executable underlying availability rules. Each underlying availability rule includes a rule identifier, verification type, verification field, and verification threshold. S13: For each business verification rule, parse out the corresponding business consistency logic expression according to the verification type it belongs to.

[0007] Optionally, the business consistency logical expression includes the data fields involved in the verification, aggregate functions, comparison conditions, and error ranges, forming business consistency rules. Each business consistency rule includes a rule identifier, verification type, business logic expression, and a list of underlying verification items it depends on.

[0008] Optionally, S1 further includes traversing each business consistency rule, starting from the list of dependent underlying verification items, matching the rule identifier of the underlying availability rule, establishing a many-to-many dependency relationship between the business consistency rule and the underlying availability rule, and recording the many dependency relationship as a rule dependency locking relationship in the form of key-value pairs.

[0009] Optionally, S2 specifically includes: S21: Read the rule dependency locking relationship and obtain the list of underlying availability rules that each business consistency rule depends on. The underlying availability rules cover four types of verification indicators: data structure, field type, field integrity, and field association validity. S22: Perform the detection operation corresponding to the underlying availability rule for each data object to be verified, and check whether there are cases of missing fields, mismatched field types, duplicate primary keys, broken association keys, or field values ​​exceeding the preset range. Mark the underlying availability rule that fails the detection as a verification blocking item, and record the blocking type and blocking location. S23: Based on the rule dependency locking relationship, starting from each verification blocking item, trace back all business consistency rules that depend on the verification blocking item, and extract the participation path of the verification blocking item in each business consistency rule; S24: For each participating path, mark the scope of the impact of the blocking item on the business consistency rule verification process, merge all blocking items and their scope of impact associated with the same business consistency rule, and generate a rule impact propagation chain with the business consistency rule as the node; S25: Based on the rules affecting the propagation chain, determine the trusted verification interval, determine the damaged verification interval, and generate a list of trusted verification boundaries.

[0010] Optionally, the participation path includes the specific location and calculation steps of the field or associated condition corresponding to the blocking item in the business logic expression, the scope of influence includes intermediate results that cannot be calculated, aggregate values ​​that cannot be compared, and logical branches that cannot be determined, and the verification trust boundary list is generated according to the rule identifier, the blocking item list, the degree of influence, and the interval to which it belongs.

[0011] Optionally, the underlying availability rules cover four types of verification indicators: data structure, field type, field integrity, and field association validity; the trusted verification interval is the interval in which the business consistency rules are located and the underlying data is fully available without being affected by any verification blocking items; the damaged verification interval is the interval in which the business consistency rules are located and the verification results are unreliable due to being affected by at least one verification blocking item.

[0012] Optionally, S3 specifically includes: S31: Read the verification trusted boundary list, obtain the business consistency rule identifiers marked as trusted verification intervals and damaged verification intervals, mark the business consistency rules in trusted verification intervals and all technical verification rules as executable, and mark the business consistency rules in damaged verification intervals as frozen. S32: For each technical verification rule and business verification rule marked as executable, automatically generate an SQL query statement encapsulated in a Python string based on the verification conditions, field mappings and comparison logic in the rule definition. The SQL query statement includes rule identifier, verification dimension, judgment condition and abnormal record output format.

[0013] Optionally, S3 further includes: The Python execution engine is invoked to run each generated SQL verification code in batches, the result set returned by the execution is captured, and the verification records that meet the expected results are merged and organized with the verification exception records that do not meet the expected results into valid verification results. Each valid verification result is associated with the corresponding rule identifier and verification timestamp. For each business consistency rule marked as frozen, read the corresponding verification blocking item information recorded in the verification trusted boundary list, and generate a review result including rule identifier, blocking type, blocking location, and review suggestion.

[0014] Optionally, S3 further includes creating an Excel version of the multidimensional data verification report, establishing a valid verification result worksheet and a pending verification result worksheet in the Excel version of the multidimensional data verification report, writing the obtained valid verification results into the valid verification result worksheet, writing the obtained pending verification results into the pending verification result worksheet, and marking the generation time and data range.

[0015] The beneficial effects of this invention are: This invention constructs a rule dependency locking relationship between business consistency rules and underlying availability rules. Before executing business verification, it identifies verification blocking items through pre-availability detection and further tracks the participation path of verification blocking items in business logic expressions to generate a rule influence propagation chain. This accurately identifies the actual impact range of verification blocking items on the business consistency rule verification process, enabling the identification of which anomalies are caused by the propagation of underlying data problems before business verification is executed. This avoids misjudging data quality issues as business issues and improves the authenticity, interpretability, and credibility of verification results.

[0016] This invention classifies business consistency rules based on the rule impact propagation chain, placing those unaffected by verification blocking items into a trusted verification interval, and those whose verification results are unreliable due to verification blocking items into a damaged verification interval, generating a trusted verification boundary list. During subsequent execution, only business consistency rules and technical verification rules within the trusted verification interval are automatically generated and executed using Python-encapsulated SQL verification code. For business consistency rules within the damaged verification interval, anomaly detection is paused, and instead, a pending verification result including blocking type, blocking location, and verification suggestions is generated. This not only avoids invalid verification and redundant investigations but also clearly identifies the root cause affecting the credibility of business verification, shifting from anomaly detection to anomaly location, thus improving verification efficiency, problem location efficiency, and the accuracy of data governance. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the verification and cause tracing in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Those skilled in the art may employ other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0020] like Figures 1-2 As shown, an automated method for multidimensional data verification based on a rule engine includes the following steps: S1: Obtain the data object to be verified, the company's accumulated technical verification rules, and the business verification rules corresponding to the business requirements. Parse the technical verification rules into underlying availability rules used to judge the validity of data structure, field type, field integrity, and field association. Parse the business verification rules into business consistency rules used to judge total score verification, business and finance verification, reference verification, status verification, rule verification, and type verification. Identify the underlying availability rules that each business consistency rule depends on and generate rule dependency locking relationships.

[0021] S1 specifically includes: S11: Read the metadata and data content of the data object to be verified from the data source interface or data warehouse. The metadata includes at least the table name, field name, field type, field length, primary key identifier, foreign key identifier, field comments, and field source information. The data content is the actual business data record corresponding to the data object to be verified. Simultaneously, load a predefined set of technical verification rules from the rule base. This set describes the verification requirements for data structure, field type, field integrity, and field association validity. Read the business verification rule set from the business configuration end. This set is organized into six categories: total score verification, business and finance verification, reference verification, status verification, rule verification, and type verification. To facilitate unified analysis later, the set of technical verification rules is represented as follows: ; The set of business verification rules is represented as follows: ; in, This represents the set of technical verification rules. Indicates the first Technical verification rules, This indicates the total number of technical verification rules. This represents the set of business verification rules. Indicates the first Business verification rules This indicates the total number of business verification rules. After the rule set is loaded, the technical verification rule set and the business verification rule set will be used as input for subsequent rule parsing and processing.

[0022] S12: Perform semantic parsing on each technical verification rule, extracting data structure verification conditions, field type verification conditions, field integrity verification conditions, and field association validity verification conditions involved in the rule through rule template matching, keyword recognition, or regular expression matching, including: When constraints such as unique primary key, complete index, and consistent table structure are identified, they are parsed as data structure verification conditions. When constraints such as field type, field length, and field format are detected, they are parsed as field type validation conditions. When constraints such as NOT NULL, missing rate, and complete rate are detected, they are parsed as field integrity verification conditions. When constraints such as primary-foreign key associations, encoding mappings, and cross-table references are identified, they are parsed as field association validity validation conditions. The parsing results are uniformly encapsulated into underlying availability rules, including: S121: For any technical verification rule After parsing, the corresponding underlying availability rules are formed, represented as follows: ; in, Indicates the first The underlying availability rules, Indicates the rule identifier, Indicates the verification type. This represents the validation field. The validation threshold represents the data structure, primary key, required fields, and strongly related fields. These directly determine whether subsequent business consistency rules can be truly verified, therefore strict thresholds should be used. Field integrity and reference-type related fields may experience minor input delays, interface synchronization delays, or missing historical data in actual business systems, thus allowing for a small tolerance. Ordinary descriptive fields or auxiliary analysis fields do not directly determine the main verification conclusions for total score verification, business and financial verification, reference verification, status verification, rule verification, and type verification, therefore relatively lenient thresholds can be set. By setting validation thresholds hierarchically according to validation type and field importance, it is possible to avoid misjudging minor data omissions as verification blocking items, and also to prevent field anomalies from continuing to propagate to business consistency rules and generating false business anomalies. In summary, the validation threshold... According to the verification type Take the following values ​​respectively: When the validation type is data structure validation, the validation threshold is set to the complete consistency threshold, that is, the table structure, primary key, required fields or related fields must be consistent with the rule requirements. When the validation type is field type validation, the validation threshold is set to the type consistency threshold, that is, the actual field type is consistent with the type required by the rule, or the data meaning is not lost after conversion by the preset type compatibility relationship; When the validation type is field integrity validation, the validation threshold is set to the upper limit of the null value rate. The upper limit of the null value rate for core business fields is 0, the upper limit of the null value rate for important analysis fields is 1% to 3%, and the upper limit of the null value rate for ordinary description fields is 5% to 10%. When the validation type is field association validity validation, the validation threshold is set to the upper limit of the association failure rate. The upper limit of the association failure rate for primary and foreign key strong constraint fields is 0, the upper limit of the association failure rate for business reference type association fields is 0.5% to 2%, and the upper limit of the association failure rate for auxiliary analysis type association fields is 2% to 5%.

[0023] S122: All underlying availability rules constitute the underlying availability rule set, represented as: ; in, This represents the underlying set of availability rules. This indicates the total number of underlying availability rules generated. For example, when a technical verification rule requires that a field cannot be empty, the field integrity check type can be parsed out; when a technical verification rule requires that the primary key be unique, the data structure check type can be parsed out; when a technical verification rule requires that a foreign key must have a corresponding record in the main table, the field association validity check type can be parsed out. In this way, the original technical verification rules are uniformly converted into structured underlying availability rules, providing a foundation for subsequent dependency construction.

[0024] In this section, rule template matching, keyword recognition, and regular expression matching are used to perform structured parsing of the natural language descriptions in the technical verification rules, thereby identifying the data structure verification conditions, field type verification conditions, field integrity verification conditions, and field association validity verification conditions corresponding to the technical verification rules, including: Rule template matching refers to the pre-construction of several standard technical verification rule templates. Each template corresponds to a fixed verification logic structure. After reading the technical verification rule, the rule text is compared item by item with the templates in the preset template library. When the text structure matches the semantic structure of a certain template, the predefined field positions, constraint positions, and condition positions in the template are extracted. For example, after the rule text "order number must be unique" matches the primary key uniqueness template, the order number is automatically extracted as the verification field, uniqueness is the constraint condition, and it is identified as a data structure verification condition. After the rule text "customer code cannot be empty" matches the non-empty verification template, the customer code is automatically extracted as the verification field, "cannot be empty" is the constraint condition, and it is identified as a field integrity verification condition. Keyword recognition refers to establishing a technical verification keyword dictionary and determining the corresponding verification category based on the keywords appearing in the rule text. The keyword dictionary includes at least structural keywords, type keywords, integrity keywords, and association keywords. When keywords such as unique, primary key, duplicate, and index are identified, the corresponding rule is determined to involve data structure verification conditions. When keywords such as character type, numeric type, date type, and length are identified, the corresponding rule is determined to involve field type verification conditions. When keywords such as cannot be empty, missing, null value, and required are identified, the corresponding rule is determined to involve field integrity verification conditions. When keywords such as association, reference, corresponding record, foreign key, and mapping relationship are identified, the corresponding rule is determined to involve field association validity verification conditions. Regular expression matching refers to pre-setting regular expressions for common rule expressions in technical verification rules, and extracting field names, constraints, and threshold information through pattern matching. For example, for rule descriptions such as field A cannot be empty, field B's null value rate must not exceed 5%, and field C's length is less than 20 characters, the corresponding regular expressions are used to extract the field names, constraint types, and threshold parameters respectively. For association rules such as field X in table A must exist in field Y in table B, the source field, target field, and association relationship are extracted to form the field association validity verification conditions.

[0025] S13: Perform business logic parsing on each business verification rule. Based on its category (total score verification, business and financial verification, reference verification, status verification, rule verification, or type verification), identify the participating verification fields, aggregate functions, comparison conditions, and error ranges in the business verification rule to form the corresponding business consistency rules, including: S131: For any business verification rule The resulting business consistency rules are represented as follows: ; in, Indicates the first Business consistency rules Indicates the business rule identifier. Indicates the verification type. This represents a business logic expression. Represents a list of underlying validation items that are dependent on; The business logic expression is represented as follows: ; in, This represents the set of data fields that are being verified. Represents a set of aggregate functions. This represents the set of comparison conditions. Represents the set of error ranges. This refers to the business logic construction function, which combines the set of data fields, aggregate functions, comparison conditions, and error ranges involved in the verification according to the preset business verification semantics to generate a business logic expression that reflects the business verification objective. It reads the verification type corresponding to the business consistency rule and determines the data fields to be verified based on the verification type. Then, according to the business rule requirements, it selects the corresponding aggregate function to perform processing on the data fields, such as summarizing, counting, calculating the average, extracting the maximum or minimum value, to form the business result to be compared. Afterwards, it establishes the comparison relationship between different business results based on the comparison conditions in the business rule and forms the final judgment logic based on the allowed error range. When the verification type is total-to-score verification, the business logic construction function establishes the correspondence between the detailed data summary result and the summary data; when the verification type is business-finance verification, it establishes the correspondence between the business system data and the financial system data. The system establishes correspondences between statistical data; when the verification type is reference verification, it establishes a correspondence between target data and reference data source; when the verification type is status verification, it establishes a correspondence between the business status transition sequence; when the verification type is rule verification, it establishes a correspondence between business data and preset business rules; when the verification type is type verification, it establishes a correspondence between data objects and preset classification standards; finally, the business logic construction function outputs a business logic expression that can directly represent the business consistency verification target, and uses it as the core judgment basis for business consistency rules; for example, the business logic expression in the total score verification rule can be composed of summing the detailed amounts and comparing them with the total amount; the business logic expression in the business and finance verification rule can be composed of comparing the business system revenue with the financial system revenue; the business logic expression in the status verification rule can be composed of verifying the business status transition sequence.

[0026] S132: After parsing, a set of business consistency rules is formed, represented as: ; in, Represents the set of business consistency rules. This indicates the total number of business consistency rules.

[0027] S14: Traverse each business consistency rule in the business consistency rule set, starting from the corresponding underlying dependency checklist, match each rule identifier in the underlying availability rule set to establish the dependency mapping relationship between the business consistency rule and the underlying availability rule, including: S141: For any business consistency rule The dependency relationship is represented as: ; in, Indicates the first The set of dependencies corresponding to each business consistency rule. Indicates the business rule identifier. This indicates the underlying availability rule identifier that has a dependency relationship with this business rule. This indicates the number of underlying availability rules that the business consistency rule depends on; S142: Summarize all dependencies to form a rule-based dependency locking relationship, represented as: ; in, This indicates that the rule depends on the locking relationship. This indicates a dependency set merging operation. This indicates the total number of business consistency rules.

[0028] In practical applications, rule dependency locking relationships are stored in key-value pairs, where the key is the business rule identifier and the value is the corresponding set of underlying availability rule identifiers. Through rule dependency locking relationships, it is possible to clarify the underlying availability rules that each business consistency rule depends on before execution, providing basic input for subsequent verification blocking item identification, rule impact propagation chain generation, and verification trusted boundary list construction.

[0029] S2: Taking rule dependency locking relationships as input, perform pre-availability detection on the data objects to be verified, identify verification blocking items that prevent business consistency rules from being truly verified due to missing fields, mismatched types, duplicate primary keys, broken associations, or abnormal data ranges; based on the participation path of each verification blocking item in the corresponding business consistency rule, trace the scope of the impact of the verification blocking item on the business consistency rule verification process, and generate a rule impact propagation chain; determine the trusted verification interval and damaged verification interval of each business consistency rule based on the rule impact propagation chain, and generate a list of trusted verification boundaries.

[0030] S2 specifically includes: S21: Read the rule dependency locking relationship and obtain the list of underlying availability rules that each business consistency rule depends on from the rule dependency locking relationship. Since the rule dependency locking relationship has established a many-to-many mapping relationship between business consistency rules and underlying availability rules, it is possible to directly determine the data availability basis that each business consistency rule depends on before execution. For any business consistency rule The corresponding list of underlying availability rules is represented as follows: ; in, Indicates the first A list of underlying availability rules that correspond to each business consistency rule. Indicates the first The underlying availability rules, This indicates the number of underlying availability rules upon which the business consistency rule depends; The underlying availability rules should cover at least the following four types of validation metrics: ; in, Indicates data structure validation metrics. This indicates the field type validation metric. Indicates the field integrity verification index. This indicates the validity verification index of field associations; by reading the rule dependency locking relationship, the data foundation conditions on which each business consistency rule depends in the subsequent execution process can be clarified, providing input basis for the identification of verification blocking items.

[0031] S22: Perform probe operations corresponding to the underlying availability rules for each data object to be verified to determine whether the data object meets the data foundation conditions required for the execution of business consistency rules; for each underlying availability rule in the dependent underlying availability rule list, perform the corresponding data structure probe, field type probe, field integrity probe, and field association validity probe, including: S221: For any underlying availability rule The detection result is defined as follows: ; in, Indicates the first If any of the following four situations occur in the detection results corresponding to the underlying availability rule, the detection of the corresponding underlying availability rule is determined to have failed: field missing, field type mismatch, primary key duplication, association key breakage, and field value exceeding the preset range. S222: For underlying availability rules that fail to detect, define them as verification blocking items, expressed as: ; in, Represents the set of verification blocking items; S223: Record the blocking type and blocking location corresponding to each verification blocking item, represented as follows: ; in, Indicates the first Location record of each verification blocking item, Indicates the type of blocking. This indicates the blocking location, which must include at least the table name, field name, associated object, and data record location. S224: After all detections are completed, a set of verification blocking items is formed.

[0032] S23: Starting with the set of verification blocking items, perform reverse tracing analysis based on rule dependency locking relationships to identify business consistency rules affected by verification blocking items, including: S231: For any verification blocking item Based on the rule dependency locking relationship, determine the set of all business consistency rules that depend on this verification blocking item, represented as: ; in, Indicates the item being verified and blocked. The set of rules affecting business consistency; S232: Analyze the participation path of verification blocking items in business logic expressions, and business consistency rules. The participation path is represented as: ; in, Indicates verification blocking item In business consistency rules The participation path in the algorithm should include at least the locations where the fields corresponding to the blocking items participate in aggregation calculations, the locations where the fields corresponding to the blocking items participate in association calculations, the locations where the fields corresponding to the blocking items participate in conditional judgments, and the locations where the fields corresponding to the blocking items participate in result output. Indicates the first in the participating path One computing node, This indicates the number of nodes involved in the path; by analyzing the path involved, the propagation process and position of the verification blocking item in the business logic expression can be determined.

[0033] S24: Perform node-by-node analysis on each participating path to identify the scope of impact of verification blocking items during the business consistency rule verification process, including: S241: For each node in the participating path, define the node's influence state, represented as: ; in, Indicates the first The impact state of each computing node.

[0034] A node is considered affected when a verification blocking item results in one of the following situations: Intermediate results cannot be calculated; Aggregation results cannot be generated; Data comparison cannot be completed; Unable to execute logical judgment; Unable to generate a final verification result; S242: Construct the set of impact scopes corresponding to the business consistency rules, represented as: ; in, Indicates business consistency rules The scope of influence; S243: Merge all verification blocking items and their impact scope corresponding to the same business consistency rule, as follows: ; in, Indicates the first The rules corresponding to the business consistency rules affect the propagation chain. This represents the set of associated verification blocking items. This represents the set of impact areas; the rule impact propagation chain reflects the complete impact process of verification blocking items propagating from the underlying availability rules to the business consistency rules.

[0035] In S24, after obtaining the participation path corresponding to the verification blocking item, the operation steps in the business consistency rule are analyzed step by step along the participation path to determine whether the verification blocking item will affect the normal execution of the corresponding operation step. Starting from the field or associated condition where the verification blocking item is located, the subsequent data filtering, data association, aggregation calculation, result comparison, logical judgment and result output processing steps are checked in sequence. When the verification blocking item causes a certain step to be unable to obtain valid input data, unable to complete the expected calculation process or unable to form a credible judgment result, the step is marked as an affected step, and its chain effect on subsequent steps is tracked. All affected steps in the same business consistency rule are summarized to determine the overall impact range of the verification blocking item on the business consistency rule. For business consistency rules with multiple verification blocking items, the impact ranges corresponding to each verification blocking item are further merged to form a complete rule impact propagation chain. The rule impact propagation chain is used to characterize the impact process of the verification blocking item from the underlying availability rule through the business logic expression to the final verification result, providing a basis for the subsequent division of credible verification intervals and damaged verification intervals.

[0036] S25: Based on the rule's impact propagation chain, classify the credibility of business consistency rules to determine the trusted verification interval and the compromised verification interval, including... S251: For any business consistency rule The degree of its influence is defined as follows: ; in, Indicates business consistency rules The extent of the impact This indicates the number of computing nodes affected. This indicates the total number of computation nodes in the business logic expression. S252: When satisfied If this occurs, it indicates that the business consistency rule has not been affected by any verification blocking item, and it is determined to be in the trusted verification range; When satisfied If this occurs, it indicates that the business consistency rule is affected by at least one verification blocking item, and it is determined to be in the damaged verification interval. The trusted verification interval and the compromised verification interval are constructed separately, and represented as follows: ; ; in, Indicates the trusted verification interval. Indicates the damaged verification range; S256: Finally, based on the rules affecting the propagation chain, a list of verifiable trusted boundaries is generated, represented as: ; in, This indicates the list of trusted boundaries for verification. Indicates the rule identifier, This represents a list of blocking items. Indicates the degree of impact. Indicates the interval to which it belongs, when When represents the trusted verification interval, when The time represents the damaged verification range, and the verification trust boundary list serves as an important basis for automatically generating Python-encapsulated SQL verification code and performing differentiated verification in step S3.

[0037] S3: Based on the verification trusted boundary list, automatically generate and run Python-encapsulated SQL verification code for unfrozen technical verification rules and business verification rules to obtain valid verification results; for frozen business consistency rules, do not directly output business anomalies, but output the results to be reviewed triggered by the corresponding verification blocking item, and write the valid verification results and the results to be reviewed into the Excel version of the multidimensional data verification report respectively.

[0038] S3 specifically includes: S31: Read the trusted boundary list for verification. Based on the rule identifiers and their respective interval information recorded in the trusted boundary list, determine the execution status of each business consistency rule. Simultaneously, include all technical verification rules in the execution scope by default, as technical verification rules are used to verify the basic data availability and are not affected by the trustworthiness division results of business consistency rules, including: S311: Read the trusted boundary list for verification For any business consistency rule Define its execution state as follows: ; in, Indicates the execution status of the business consistency rules. Indicates the trusted verification interval. Indicates the range of the verified document. Indicates an executable state. Indicates a frozen state; S312: After the status is marked, an executable rule set and a frozen rule set are formed, represented as follows: ; ; in, Represents a set of executable rules. This represents the set of freeze rules.

[0039] S32: Perform code generation processing on the technical verification rules and business consistency rules in the executable rule set. Based on the verification conditions, field mapping relationships, and comparison logic in the rule definition, automatically generate the corresponding SQL verification code and encapsulate it using Python strings, including: S321: For any executable rule The corresponding SQL verification code is represented as follows: ; in, Indicates the first SQL verification code, The SQL generation function refers to a processing function that automatically concatenates rule validation conditions, field mapping relationships, and comparison logic into an SQL query statement based on the rule type of the executable rule. The SQL generation function first reads the rule validation conditions to determine the fields to be validated, thresholds, filtering conditions, or anomaly judgment conditions. Then, it determines the data table, field name, associated fields, and target comparison fields of the field based on the field mapping relationship. Finally, it selects the corresponding SQL structure based on the comparison logic. For example, non-null validation generates null value filtering statements, type validation generates type conversion or format judgment statements, total score verification generates group summary comparison statements, business and finance verification generates cross-table join comparison statements, and status verification generates status transition judgment statements. In this way, SQL verification code that can directly return pass or abnormal records is generated. This indicates the rule validation conditions, such as fields cannot be empty, the total amount must match the total amount in the summary table, the business status must flow in the order of submitted, approved, and credited, and the business code must exist in the reference table, etc. It represents the field mapping relationship, including the business field name, the actual table name, the actual field name, the associated table name, the associated field name, and the necessary filtering range; The comparison logic refers to the data comparison method determined according to the rule objectives, including equal value comparison, range comparison, aggregation comparison, cross-table comparison, status order comparison, type classification comparison, etc., which are used to determine whether to use direct filtering, grouping summary, join query, difference calculation or condition judgment in SQL to generate the final verification code; S322: The generated Python code encapsulating the SQL verification is represented as follows: ; in, This indicates that Python code encapsulates SQL verification. Indicates the rule identifier, Indicates the verification dimension. This represents an SQL query statement. This indicates the output format for exception records.

[0040] S33: Call the Python execution engine to run the generated Python-encapsulated SQL verification code in batches, and collect the execution return results, including: S331: For any Python wrapper for SQL verification code The execution result is expressed as follows: ; in, Represents the execution result set. The code execution function refers to the processing function used to call the Python execution engine to run the Python-encapsulated SQL verification code and return the verification result set. The code execution function first reads the rule identifier, database connection information, SQL query statement and exception record output format from each Python-encapsulated SQL verification code, connects to the target database through the Python database connection component, submits the SQL query statement to the database for execution, receives the data results returned by the database after execution, and organizes them into a structured result set according to the exception record output format corresponding to the rule.

[0041] During execution, if the SQL verification code runs normally, the code execution function will mark the returned records as verification passed or verification failed, and write the valid verification result. If database connection failure, field non-existence, SQL syntax error, execution timeout, or insufficient permissions occur during execution, it will not be directly judged as a business exception. Instead, the reason for the execution failure, the corresponding rule identifier, and the failure location will be recorded, and the rule will be transferred to the pending review result. Through this code execution function, batch scheduling, sequential execution, exception handling, and standardized output of multiple Python-encapsulated SQL verification codes can be achieved.

[0042] S332: Classify the execution results according to the decision conditions in the rule definition, as follows: ; ; in, This indicates the set of records that have passed verification. This represents the set of verification anomaly records; Judgment conditions are the specific criteria used in rules to distinguish between verification pass and verification fail. The judgment standards are determined according to different rule types. For example, for field integrity rules, the judgment condition could be that the field null value rate does not exceed a preset threshold; for field association validity rules, the judgment condition could be that the association success rate meets preset requirements; for total score verification rules, the judgment condition could be that the detailed summary result is consistent with the summary result; for business-finance verification rules, the judgment condition could be that the difference between the business system data and the financial system data is within the allowable error range; for status verification rules, the judgment condition could be that the business status transition sequence conforms to preset business process requirements. The code execution function judges the SQL execution result based on the above judgment conditions, classifying records that meet the judgment conditions as verification pass records and records that do not meet the judgment conditions as verification fail records.

[0043] S333: Verification pass records and verification exception records are uniformly organized to form a valid verification result set, which is represented as follows: ; ; in, This indicates a valid verification result. Indicates the rule identifier, Indicates the verification timestamp. Represents the set of valid verification results. This indicates the number of valid verification results.

[0044] S34: For business consistency rules in the freeze rule set, actual SQL verification operations will no longer be performed. Instead, a review result will be generated based on the verification blocking item information recorded in the verification trust boundary list, including: S341: Business consistency rules for any frozen state Read the corresponding blocking list, represented as: ; in, This represents the corresponding list of verification blocking items. Indicates the first One verification blocking item, Indicates the number of verification blocking items associated with the corresponding business consistency rule; S342: Construct the results to be reviewed. All results to be reviewed together form a set of results to be reviewed, represented as follows: ; ; in, This indicates that the result is pending review. Indicates the rule identifier, Indicates the type of blocking. Indicates the location of the blockage. Indicates a review recommendation. This represents the set of results to be reviewed. This indicates the total number of freeze rules.

[0045] S35: Create an Excel version of the multidimensional data verification report, and write the valid verification results and the results to be reviewed into the corresponding worksheets according to different dimensions, including: S351: Let the generated Excel version of the multidimensional data verification report be represented as: ; in, This refers to the Excel version of the multidimensional data verification report. The worksheet showing valid verification results This is a worksheet showing the results pending review. Indicates the time the report was generated. Indicates a data range; S352: Write the set of valid verification results to... Write the set of results to be reviewed into The final result is a complete Excel version of the multidimensional data verification report, which can display the verification results and the items to be reviewed separately, and retain the rule identifier, verification timestamp, blocking item information and data range information, providing a basis for subsequent data governance, problem investigation and manual review; in, Represents the set of valid verification results. This represents the set of results to be reviewed.

[0046] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0047] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatically executing multidimensional data verification based on a rule engine, characterized in that, Includes the following steps: S1: Obtain the data object to be verified, the technical verification rules accumulated by the company, and the business verification rules corresponding to the business requirements. Parse the technical verification rules into underlying availability rules for judging the validity of data structure, field type, field integrity, and field association. Parse the business verification rules into business consistency rules for judging total score verification, business and finance verification, reference verification, status verification, rule verification, and type verification. Identify the underlying availability rules that each business consistency rule depends on and generate rule dependency locking relationships. S2: Using the rule dependency locking relationship as input, perform a preliminary availability probe on the data object to be verified, identify verification blocking items that prevent the business consistency rule from being truly verified due to missing fields, mismatched types, duplicate primary keys, broken associations, or abnormal data ranges; based on the participation path of each verification blocking item in the corresponding business consistency rule, trace the impact range of the verification blocking item on the business consistency rule verification process, and generate a rule impact propagation chain; determine the trusted verification interval and damaged verification interval of each business consistency rule based on the rule impact propagation chain, and generate a list of trusted verification boundaries; S3: Based on the verification trust boundary list, automatically generate and run the SQL verification code wrapped in Python for the unfrozen technical verification rules and business verification rules to obtain valid verification results; For frozen business consistency rules, output the pending verification results triggered by the corresponding verification blocking item, and write the valid verification results and pending verification results into the Excel version of the multidimensional data verification report.

2. The automatic execution method for multidimensional data verification based on a rule engine according to claim 1, characterized in that, S1 specifically includes: S11: Read the metadata and data content of the data object to be verified from the data source interface or data warehouse, load the predefined set of technical verification rules from the rule base, and read the set of business verification rules for six types of organizations: total score verification, business and finance verification, reference verification, status verification, rule verification, and type verification from the business configuration end. S12: For each technical verification rule, extract the judgment conditions for four types of verification items: data structure, field type, field integrity, and field association validity. Encapsulate the judgment conditions into executable underlying availability rules. Each underlying availability rule includes a rule identifier, verification type, verification field, and verification threshold. S13: For each business verification rule, parse out the corresponding business consistency logic expression according to the verification type it belongs to.

3. The automatic execution method for multidimensional data verification based on a rule engine according to claim 2, characterized in that, The business consistency logic expression includes the data fields involved in the verification, aggregate functions, comparison conditions, and error ranges, forming business consistency rules. Each business consistency rule includes a rule identifier, verification type, business logic expression, and a list of underlying verification items it depends on.

4. The automatic execution method for multidimensional data verification based on a rule engine according to claim 3, characterized in that, S1 further includes traversing each business consistency rule, starting from the list of dependent underlying verification items, matching the rule identifier of the underlying availability rule, establishing a many-to-many dependency relationship between the business consistency rule and the underlying availability rule, and recording the many dependency relationship as a rule dependency locking relationship in the form of key-value pairs.

5. The automatic execution method for multidimensional data verification based on a rule engine according to claim 1, characterized in that, S2 specifically includes: S21: Read the rule dependency locking relationship and obtain the list of underlying availability rules that each business consistency rule depends on. The underlying availability rules cover four types of verification indicators: data structure, field type, field integrity, and field association validity. S22: Perform the detection operation corresponding to the underlying availability rule for each data object to be verified, and check whether there are cases of missing fields, mismatched field types, duplicate primary keys, broken association keys, or field values ​​exceeding the preset range. Mark the underlying availability rule that fails the detection as a verification blocking item, and record the blocking type and blocking location. S23: Based on the rule dependency locking relationship, starting from each verification blocking item, trace back all business consistency rules that depend on the verification blocking item, and extract the participation path of the verification blocking item in each business consistency rule; S24: For each participating path, mark the scope of the impact of the blocking item on the business consistency rule verification process, merge all blocking items and their scope of impact associated with the same business consistency rule, and generate a rule impact propagation chain with the business consistency rule as the node; S25: Based on the rules affecting the propagation chain, determine the trusted verification interval, determine the damaged verification interval, and generate a list of trusted verification boundaries.

6. The automatic execution method for multidimensional data verification based on a rule engine according to claim 5, characterized in that, The participation path includes the specific location and calculation steps of the field or associated condition corresponding to the blocking item in the business logic expression. The scope of influence includes intermediate results that cannot be calculated, aggregate values ​​that cannot be compared, and logical branches that cannot be determined. The verification trust boundary list is generated according to the rule identifier, the list of blocking items, the degree of influence, and the interval to which it belongs.

7. The automatic execution method for multidimensional data verification based on a rule engine according to claim 5, characterized in that, The underlying availability rules cover four types of verification indicators: data structure, field type, field integrity, and field association validity. The trusted verification interval is the interval in which the underlying data of the business consistency rule is located without being affected by any verification blocking item. The damaged verification interval is the interval in which the verification result of the business consistency rule is located due to being affected by at least one verification blocking item.

8. The automatic execution method for multidimensional data verification based on a rule engine according to claim 1, characterized in that, S3 specifically includes: S31: Read the verification trusted boundary list, obtain the business consistency rule identifiers marked as trusted verification intervals and damaged verification intervals, mark the business consistency rules in trusted verification intervals and all technical verification rules as executable, and mark the business consistency rules in damaged verification intervals as frozen. S32: For each technical verification rule and business verification rule marked as executable, automatically generate an SQL query statement encapsulated in a Python string based on the verification conditions, field mappings and comparison logic in the rule definition. The SQL query statement includes rule identifier, verification dimension, judgment condition and abnormal record output format.

9. The automatic execution method for multidimensional data verification based on a rule engine according to claim 8, characterized in that, S3 further includes: The Python execution engine is invoked to run each generated SQL verification code in batches, the result set returned by the execution is captured, and the verification records that meet the expected results are merged and organized with the verification exception records that do not meet the expected results into valid verification results. Each valid verification result is associated with the corresponding rule identifier and verification timestamp. For each business consistency rule marked as frozen, read the corresponding verification blocking item information recorded in the verification trusted boundary list, and generate a review result including rule identifier, blocking type, blocking location, and review suggestion.

10. The automatic execution method for multidimensional data verification based on a rule engine according to claim 9, characterized in that, The S3 further includes creating an Excel version of the multidimensional data verification report, establishing a valid verification result worksheet and a pending verification result worksheet in the Excel version of the multidimensional data verification report, writing the obtained valid verification results into the valid verification result worksheet, writing the obtained pending verification results into the pending verification result worksheet, and marking the generation time and data range.

Images