Data checking method and device, terminal equipment and storage medium

Abstract

The application discloses a data checking method and device, a terminal equipment and a storage medium, and the method comprises the following steps: in response to a checking execution command, input parameters are acquired; a pre-defined checking rule is matched based on the input parameters to obtain a target checking rule; the target checking rule is analyzed based on the target checking rule to generate and execute a structured query language (SQL) statement to obtain a data checking result. The target checking rule is matched based on the input parameters, and the target checking rule can be adapted to different database environments. The target checking rule is also analyzed, the SQL is executed to obtain a data checking result, the data checking is realized based on the SQL, the data checking effect is improved, and the timeliness of the data checking is enhanced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data verification method, apparatus, terminal device, and storage medium. Background Technology

[0002] In recent years, with the increasing intensity of national supervision of banks, regulatory authorities have required banks to submit more and more types of data, and the requirements for data quality have become increasingly stringent. How to identify data quality issues before data is submitted to regulatory authorities has become a primary problem that needs to be solved for bank reporting products. Currently, the data verification methods of regulatory reporting products on the market are mainly based on rule engines. This data verification method performs rule calculations in memory, requiring substantial computing resources and relying on middleware such as Redis (one purchased verification engine requires a 170GB Redis cluster as its computing resource). Middleware such as Redis, as computing resources for verification engines, has advantages such as high performance, in-memory storage, diverse data structures, persistence support, high availability, and ease of use, which can meet the needs of large-scale data verification and provide fast response and stable, reliable services.

[0003] However, the data reported by banks has a long processing chain before it is submitted. The traditional data verification methods mentioned above can only perform output verification and quality checks at the end of the processing chain. For time-sensitive submissions, they cannot meet the need to detect and handle problems in advance, resulting in low timeliness of data verification. Summary of the Invention

[0004] The main objective of this invention is to provide a data verification method, apparatus, terminal device, and storage medium, aiming to enhance the timeliness of data verification by improving the data verification effect.

[0005] To achieve the above objectives, the present invention provides a data verification method, which is applied in a database environment and includes the following steps:

[0006] In response to the verification execution command, obtain the input parameters;

[0007] The target verification rule is obtained by matching the predefined verification rules based on the input parameters.

[0008] Based on the target validation rules, rule parsing is performed to generate and execute structured query statements (SQL) to obtain data validation results.

[0009] Optionally, the step of filtering validation rules based on the input parameters to obtain the target validation rule includes the following before:

[0010] The validation rules are defined, specifically including:

[0011] Retrieve rule configuration template information;

[0012] The verification rules are obtained by defining rules based on the rule configuration template information.

[0013] Optionally, after the step of defining rules based on the rule configuration template information to obtain the verification rules, the method further includes:

[0014] The verification rules are synchronized to a pre-created verification rule table to obtain a verification rule definition table.

[0015] Optionally, the step of matching the predefined verification rules based on the input parameters to obtain the target verification rule includes:

[0016] Data table information is obtained by parsing the input parameters.

[0017] The target verification rule is obtained by matching the predefined verification rules with the information in the data table.

[0018] Optionally, the target verification rules include at least one or more of the following: single-table verification rules, association verification rules, multi-table verification rules, and partial data verification rules within a table.

[0019] Optionally, the step of parsing the rules based on the target validation rules, generating and executing a structured query statement (SQL), and obtaining the data validation result includes:

[0020] Iterate through the target validation rules and select the rule expression to be parsed;

[0021] SQL is generated and executed based on the rule expression to obtain and update the error count statistics and error details.

[0022] Determine whether there are any unparsed rules in the target verification rules;

[0023] If so, proceed as follows: traverse the target validation rules and select the rule expression to be parsed;

[0024] If not, the data verification result is obtained based on the error count statistics and the error details.

[0025] Optionally, the step of generating and executing SQL based on the rule expression to obtain and update the error count statistics and error details includes:

[0026] Parse the rule expression to generate the first type of SQL and the second type of SQL;

[0027] Execute the first type of SQL to obtain and update the error count statistics;

[0028] Execute the second type of SQL based on the error statistics results to obtain and update the error details results.

[0029] Furthermore, to achieve the above objectives, the present invention also provides a data verification device, the device comprising:

[0030] The data acquisition module is used to acquire input parameters in response to the verification execution command;

[0031] The rule configuration module is used to match predefined verification rules based on the input parameters to obtain the target verification rule;

[0032] The rule parsing module is used to parse the rules based on the target verification rules, generate and execute structured query statements (SQL), and obtain data verification results.

[0033] Optionally, the rule configuration module is further used for:

[0034] The validation rules are defined, specifically including:

[0035] Retrieve rule configuration template information;

[0036] The verification rules are obtained by defining rules based on the rule configuration template information.

[0037] Optionally, the rule configuration module is further used for:

[0038] The verification rules are synchronized to a pre-created verification rule table to obtain a verification rule definition table.

[0039] Optionally, the rule configuration module is further used for:

[0040] Data table information is obtained by parsing the input parameters.

[0041] The target verification rule is obtained by matching the predefined verification rules with the information in the data table.

[0042] The target verification rules include at least one or more of the following: single-table verification rules, association verification rules, multi-table verification rules, and partial data verification rules within a table.

[0043] Optionally, the rule parsing module is further configured to:

[0044] Iterate through the target validation rules and select the rule expression to be parsed;

[0045] SQL is generated and executed based on the rule expression to obtain and update the error count statistics and error details.

[0046] Determine whether there are any unparsed rules in the target verification rules;

[0047] If so, proceed as follows: traverse the target validation rules and select the rule expression to be parsed;

[0048] If not, the data verification result is obtained based on the error count statistics and the error details.

[0049] Optionally, the rule parsing module is further configured to:

[0050] Parse the rule expression to generate the first type of SQL and the second type of SQL;

[0051] Execute the first type of SQL to obtain and update the error count statistics;

[0052] Execute the second type of SQL based on the error statistics results to obtain and update the error details results.

[0053] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device including a memory, a processor, and a data verification program stored in the memory and executable on the processor, wherein the data verification program, when executed by the processor, implements the data verification method as described above.

[0054] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data verification program, which, when executed by a processor, implements the data verification method described above.

[0055] This invention discloses a data verification method, apparatus, terminal device, and storage medium. The method involves acquiring input parameters in response to a verification execution command; matching the input parameters against predefined verification rules to obtain a target verification rule; parsing the target verification rule to generate and execute a structured query statement (SQL) to obtain a data verification result. This invention achieves SQL-based data verification by matching the target verification rule from predefined verification rules based on the input parameters, parsing the target verification rule, and executing the SQL to obtain the data verification result. This improves the effectiveness and timeliness of data verification. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the data verification device of the present invention belongs;

[0057] Figure 2This is a flowchart illustrating a first exemplary embodiment of the data verification method of the present invention;

[0058] Figure 3 This is a system framework diagram for data verification in a Gauss database environment in the first exemplary embodiment of the data verification method of the present invention;

[0059] Figure 4 This is a flowchart illustrating a second exemplary embodiment of the data verification method of the present invention;

[0060] Figure 5 This is a flowchart illustrating a third exemplary embodiment of the data verification method of the present invention;

[0061] Figure 6 This is a flowchart illustrating a fourth exemplary embodiment of the data verification method of the present invention;

[0062] Figure 7 This is a flowchart illustrating the workflow of filtering target verification rules in the fourth exemplary embodiment of the data verification method of the present invention;

[0063] Figure 8 This is a flowchart illustrating a fifth exemplary embodiment of the data verification method of the present invention;

[0064] Figure 9 This is a schematic diagram illustrating the workflow of implementing one-place configuration and multiple-place execution in the fifth embodiment of the data verification method of the present invention;

[0065] Figure 10 This is an example diagram of SQL generated based on verification rules in the fifth exemplary embodiment of the data verification method of the present invention.

[0066] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0067] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0068] The main solution of this invention is as follows: in response to a verification execution command, input parameters are obtained; based on the input parameters, predefined verification rules are matched to obtain target verification rules; based on the target verification rules, rule parsing is performed to generate and execute a structured query statement (SQL) to obtain data verification results.

[0069] This application takes into account that current industry-based data verification methods, which are based on rule engines, perform rule calculations in memory, requiring a large amount of computing resources and relying on middleware such as Redis (a certain external verification engine requires a Redis cluster with 170G of memory as its computing resource). Traditional data verification methods can only perform outgoing verification and quality checks at the end of the processing link, which cannot meet the need for early detection and early handling of problems for time-sensitive reports, resulting in low timeliness of data verification.

[0070] Based on this, this application provides a solution that obtains the target verification rule by matching it with a predefined verification rule based on the input parameters, then parses the target verification rule, executes the SQL to obtain the data verification result, and realizes SQL-based data verification, thereby improving the data verification effect and enhancing the timeliness of data verification.

[0071] Specifically, refer to Figure 1 , Figure 1 This is a functional module diagram of the terminal device to which the data verification device of this application belongs. The data verification device can be an independent device capable of data verification and recommendation, and it can be implemented on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with data processing capabilities, a fixed terminal device or server with data processing capabilities, etc. Furthermore, the data verification device can also be implemented within a data verification system.

[0072] In this embodiment, the terminal device to which the data verification device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0073] The memory 130 stores the operating system and data verification program; the output module 110 may be a display screen, etc. The communication module 140 may include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0074] When the data verification program in memory 130 is executed by the processor, it performs the following steps:

[0075] In response to the verification execution command, obtain the input parameters;

[0076] The target verification rule is obtained by matching the predefined verification rules based on the input parameters.

[0077] Based on the target validation rules, rule parsing is performed to generate and execute structured query statements (SQL) to obtain data validation results.

[0078] Furthermore, when the data verification program in memory 130 is executed by the processor, it also performs the following steps:

[0079] The validation rules are defined, specifically including:

[0080] Retrieve rule configuration template information;

[0081] The verification rules are obtained by defining rules based on the rule configuration template information.

[0082] Furthermore, when the data verification program in memory 130 is executed by the processor, it also performs the following steps:

[0083] The verification rules are synchronized to a pre-created verification rule table to obtain a verification rule definition table.

[0084] Furthermore, when the data verification program in memory 130 is executed by the processor, it also performs the following steps:

[0085] Data table information is obtained by parsing the input parameters.

[0086] The target verification rule is obtained by matching the predefined verification rules with the information in the data table.

[0087] The target verification rules include at least one or more of the following: single-table verification rules, association verification rules, multi-table verification rules, and partial data verification rules within a table.

[0088] Furthermore, when the data verification program in memory 130 is executed by the processor, it also performs the following steps:

[0089] Iterate through the target validation rules and select the rule expression to be parsed;

[0090] SQL is generated and executed based on the rule expression to obtain and update the error count statistics and error details.

[0091] Determine whether there are any unparsed rules in the target verification rules;

[0092] If so, proceed as follows: traverse the target validation rules and select the rule expression to be parsed;

[0093] If not, the data verification result is obtained based on the error count statistics and the error details.

[0094] Furthermore, when the data verification program in memory 130 is executed by the processor, it also performs the following steps:

[0095] Parse the rule expression to generate the first type of SQL and the second type of SQL;

[0096] Execute the first type of SQL to obtain and update the error count statistics;

[0097] Execute the second type of SQL based on the error statistics results to obtain and update the error details results.

[0098] This embodiment, through the above-described scheme, obtains input parameters in response to a verification execution command; matches the input parameters against predefined verification rules to obtain target verification rules; parses the target verification rules to generate and execute structured query statements (SQL) to obtain data verification results. This invention obtains the target verification rules based on the input parameters, and these target verification rules are adaptable to different database environments. Furthermore, this invention parses the target verification rules and executes the SQL to obtain data verification results, thus achieving SQL-based data verification, thereby improving the effectiveness and timeliness of data verification.

[0099] Based on, but not limited to, the terminal device architecture described above, this application proposes method embodiments.

[0100] Reference Figure 2 , Figure 2 This is a flowchart illustrating a first exemplary embodiment of the data verification method of this application. The data verification method includes:

[0101] Step S10: In response to the verification execution command, obtain the input parameters;

[0102] Specifically, this embodiment can be applied to a database environment. The calculation process of the data verification method in this embodiment relies on database resources but not on other computing resources, and does not require the deployment of additional middleware or other computing components, thus saving costs (for example, a purchased verification engine requires nearly 100,000 yuan per year for 170G of Redis resources). This embodiment uses a data verification execution engine as its implementation subject, and Huawei's Gauss database as its application environment. The data verification execution engine includes a GSQL (Structured Query Language) client, a verification engine script (also called a validation engine script), a set of configuration tables, and a result table. Gauss is a distributed relational database management system that provides high-performance, high-reliability, and scalable database solutions. The input parameters in this embodiment can be data date parameters, the data table of the target data verification object, etc. The data verification execution engine can dynamically select the object and content to be verified based on the input parameters in the data verification execution command.

[0103] Step S20: Match the predefined verification rules based on the input parameters to obtain the target verification rule;

[0104] Specifically, in this embodiment, the target validation rule is obtained by matching the header name in the input parameters against predefined validation rules. The validation rules need to be predefined based on a rule configuration template. Furthermore, the validation rules can be matched according to different data types. For example, for numeric data, rules such as number range and regular expressions can be used for matching; for string data, rules such as length and character set can be used. The matching process can be performed using methods such as conditional statements, regular expressions, and comparison operations. Based on the matching results, it can be determined which validation rules are applicable to the current input parameters, and then corresponding data validation can be performed. In this embodiment, the target validation rule can also be one or more of the following: single-table validation rules, relational validation rules, multi-table validation rules, and partial data validation rules within a table.

[0105] Step S30: Based on the target verification rules, perform rule parsing, generate and execute a structured query statement (SQL), and obtain the data verification result.

[0106] Specifically, this embodiment runs the parsing script corresponding to the target verification rule, dynamically configures the parameters according to the target verification rule, and generates rule SQL. Rule SQL is a method for implementing and executing specific business rules using SQL statements. It can be used to verify data validity, execute computational logic, and control application permissions. Rule SQL is commonly used in databases to ensure data integrity, accuracy, and consistency. By writing and executing rule SQL, business rules can be executed automatically, avoiding human error and improving data quality.

[0107] More specifically, refer to Figure 3 , Figure 3 This is a system framework diagram of the data verification method in this application under the Gauss database environment.

[0108] First, generate rule definitions using the rule configuration template, and then generate a validation rule definition table.

[0109] Secondly, synchronize the verification rule definition table to the Gauss library;

[0110] Secondly, the system receives data verification execution commands initiated manually or sent by other system scheduling services through the Gsql client, and responds to the execution command by parsing the script to generate and generating dynamic rule SQL according to the rule configuration.

[0111] Then, the rule SQL is sent to the Gauss database and executed to generate a verification result interface table and an error detail interface table. The verification result table stores the statistical results of the number of data verification errors, and the error detail interface table stores the specific information of the errors detected in the data verification.

[0112] Finally, the verification result interface table and the error details interface table are synchronized to the statistical reporting platform.

[0113] This embodiment, through the above-described scheme, obtains input parameters in response to a verification execution command; matches the input parameters against predefined verification rules to obtain target verification rules; parses the target verification rules to generate and execute structured query statements (SQL) to obtain data verification results. This invention designs and implements a configurable SQL-based verification engine. The engine can be deployed at different stages of the data processing chain, enabling the verification process to be moved upstream, identifying and resolving problems at the source. Furthermore, the data verification engine is adapted to multiple databases, allowing execution in different database environments along the processing chain without relying on other middleware or software deployment environments. This invention implements SQL-based data verification, thereby improving data verification effectiveness and enhancing its timeliness.

[0114] Reference Figure 4 , Figure 4 This is a flowchart illustrating a second exemplary embodiment of the data verification method of this application.

[0115] Based on the first embodiment, a second embodiment of this application is proposed. The difference between the second embodiment and the first embodiment is as follows:

[0116] In this embodiment, before step S20, which involves filtering verification rules based on the input parameters to obtain the target verification rule, the following steps are also included:

[0117] Step S15: Define the verification rules;

[0118] Specifically, the validation rules need to be predefined based on a rule configuration template. Furthermore, the validation rules can match different data types. For example, for numeric data, rules such as numeric range and regular expressions can be used for matching; for string data, rules such as length and character set can be used. The matching process can be performed using methods such as conditional statements, regular expressions, and comparison operations.

[0119] Furthermore, this embodiment further refines step S15, which defines the verification rules.

[0120] In this embodiment, step S15, refining the verification rules, includes:

[0121] Step S151: Obtain rule configuration template information;

[0122] Specifically, the rule configuration template information is pre-configured manually, and the rule configuration is based on various databases. This embodiment uses the Gauss database as an example. This embodiment first determines the rule type, which includes data validation rules, business rules, or other types of rules. The rule configuration template information is stored in the database or configuration file and can be provided through an API (Application Programming Interface) or other interfaces. When calling the API, the corresponding API documentation or interface description is required to obtain the rule configuration template information.

[0123] Step S152: Define rules based on the rule configuration template information to obtain the verification rules.

[0124] Specifically, this embodiment takes a verification execution engine as an example, and the internal structure and functions of the verification execution engine can be referred to in the first embodiment described above. This embodiment first determines the data verification requirements based on the rule template configuration information, such as what aspects of the input data need to be verified, and what conditions need to be checked for compliance. Taking data verification at each stage of the processing chain as an example, the verification rules need to be defined at three points: source processing, data integration, and reporting. The specific rule definition method can be: writing the definition code or configuration of the verification rules based on the specific business requirements in the actual implementation process and the fields in the configuration template. The specific implementation method depends on the programming language, framework, or tool used.

[0125] This embodiment, through the above-described scheme, obtains input parameters in response to a verification execution command; matches the input parameters against predefined verification rules to obtain target verification rules; parses the target verification rules to generate and execute structured query statements (SQL) to obtain data verification results. The verification rules of this invention are adaptable to various databases, and a suitable target verification rule can be selected when matching databases. Therefore, this embodiment can be executed in different database environments along the processing chain without relying on other middleware or software deployment environments. It can also meet the need for early detection and handling of problems in time-sensitive reporting, enhancing the timeliness of data verification.

[0126] Reference Figure 5 , Figure 5 This is a flowchart illustrating a third exemplary embodiment of the data verification method of this application.

[0127] Based on the second embodiment, a third embodiment of this application is proposed. The difference between the third embodiment and the second embodiment is as follows:

[0128] In this embodiment, step S152, after defining the rules based on the rule configuration template information to obtain the verification rules, includes:

[0129] Step S153: Synchronize the verification rules to the pre-created verification rule table to obtain the verification rule definition table.

[0130] Specifically, this embodiment first creates a verification rule table, which can be created in a database or other storage systems. The verification rule table should have appropriate fields and data types to meet data verification requirements. Second, this embodiment defines the table structure of the verification rule table, including determining the fields of each verification rule record and its corresponding data type. Third, this embodiment converts the verification rules into data. Specifically, each rule can be converted into a corresponding data record based on its actual meaning, and the field values ​​of each record are filled to represent the specific verification rule. Then, this embodiment synchronizes the verification rules to the verification rule table, and writes the verification rules into the corresponding fields of the verification rule table through corresponding main database operations (such as built-in insert, upload update, etc.), thus obtaining a verification rule definition table.

[0131] This embodiment, through the above-described scheme, obtains input parameters in response to a verification execution command; matches the predefined verification rules based on the input parameters to obtain target verification rules; parses the target verification rules to generate and execute structured query statements (SQL) to obtain data verification results. The verification rules of this invention are adaptable to various databases. After obtaining the target verification rule based on the input parameters, the target verification rule can be directly configured into the corresponding database, achieving the effect of dynamically generating SQL and performing data verification based on different input parameters after a single rule configuration. Therefore, this embodiment of the invention can also meet the need for early detection and handling of problems in time-sensitive reporting, enhancing the timeliness of data verification.

[0132] Reference Figure 6 , Figure 6 This is a flowchart illustrating the fourth exemplary embodiment of the data verification method of this application.

[0133] Based on the first embodiment, a fourth embodiment of this application is proposed. The difference between the fourth embodiment and the first embodiment is that the fourth embodiment refines step S20, which involves matching the predefined verification rules based on the input parameters to obtain the target verification rules.

[0134] In this embodiment, step 20, matching the predefined verification rules based on the input parameters to obtain the target verification rule, may include:

[0135] Step S201: Based on the input parameters, perform data parsing to obtain data table information;

[0136] Specifically, the input parameters in this embodiment may be data date parameters, data tables of target data verification objects, etc. In this embodiment, the structure and format of the input parameters are first determined to select the corresponding parsing script for data parsing. In this embodiment, data table information is obtained through data parsing, and the data table information is used to query relevant verification rules.

[0137] Step S202: Based on the data table information, match the predefined verification rules to obtain the target verification rule.

[0138] Specifically, this embodiment matches data table information with predefined validation rules. Based on the data table's fields, data types, constraints, and other information, the validation rules are compared with their definitions. Then, based on the matching results, this embodiment filters out target validation rules that match the data table information. These target validation rules can be specific validation rules corresponding to data table fields, or they can be general validation rules applicable to the entire table or specific conditions. The target validation rules include one or more of the following: single-table validation rules, relational validation rules, multi-table validation rules, and partial data validation rules within a table.

[0139] More specifically, refer to Figure 7 , Figure 7 This diagram illustrates the workflow for selecting target verification rules in the data verification method of this application. As shown, in this embodiment, the program parameters I_TABLES and I_STG in the input parameters can be used to select the object and content of the verification when running the data verification engine, thereby achieving the selection of target verification rules. Specifically, the selection of target verification rules can be controlled by parameters to perform inter-table join verification or single-table verification; by parameters to perform verification of a single table or multiple tables; or by parameters to perform verification of a portion of data within a single table. The program parameters I_TABLES and I_STG are explained below:

[0140] I_TABLES: This parameter filters the tables to be validated. Multiple tables are separated by #, e.g.

[0141] 0102_YGB#0101_JGXXB will execute the rule in the rule table where PRM_TBL is 0102_YGB and

[0142] The verification rules for 0101_JGXXB; where 0102_YGB is the verification rule table 1 of this embodiment, 0101_JGXXB is the verification rule table 2 of this embodiment, and PRM_TBL is the main parameter of the verification rule table;

[0143] I_STG: This parameter filters for single-table validation or correlation validation. A value of 0 indicates single-table validation, and a value of 1 indicates correlation validation.

[0144] like Figure 7 As shown, this embodiment takes the commonly used control table association as an example. The association verification between tables A and B can be performed after the data of tables A and B are processed by input parameters; or the association verification between tables A, B, and C can be performed after tables A, B, and C are processed.

[0145] This embodiment, through the above-described scheme, obtains input parameters in response to a verification execution command; matches the input parameters against predefined verification rules to obtain target verification rules; parses the target verification rules to generate and execute structured query statements (SQL) to obtain data verification results. This invention refines the target verification rules, specifically selecting the object and content to be verified when running the data verification engine based on the program parameters I_TABLES and I_STG in the input parameters, thereby achieving the filtering of target verification rules. The target verification rules can adapt to different database environments, saving the cost of repeatedly configuring verification rules and enhancing the timeliness of data verification.

[0146] Reference Figure 8 , Figure 8 This is a flowchart illustrating the fifth exemplary embodiment of the data verification method of this application.

[0147] Based on the first embodiment, a fifth embodiment of this application is proposed. The difference between the fifth embodiment and the first embodiment is that the fifth embodiment refines step S30, which involves parsing the target verification rules, generating and executing a structured query statement (SQL), and obtaining data verification results.

[0148] In this embodiment, step S30 involves parsing the target validation rules, generating and executing a structured query statement (SQL), and obtaining the data validation results, including:

[0149] Step S301: Traverse the target verification rules and select the rule expression to be parsed;

[0150] Specifically, this embodiment first needs to traverse the target verification rules. This traversal process can be based on rule priority, filtering and traversing according to preset conditions, or sorting the verification rules according to their trigger frequencies, thus traversing the target verification rules based on these trigger frequencies. The rule expression is a formalized language, a specific syntax structure of a programming language, or a combination of conditions and operations. The rule expression defines the conditions of a rule and the operations performed after the rule is triggered. It typically includes conditions, actions, and logical rules. More specifically, the condition part of the rule expression describes the preconditions for rule triggering. Conditions can be constructed using comparison operators, logical operators, and functions to determine whether certain specific conditions are met. The action part of the rule expression defines the operations to be performed after the rule is triggered. These operations can be function calls, data storage, output information, etc., used to implement specific business logic; the rule expression can also contain logical relationships between rules, such as AND, OR, NOT, etc.; these logical operators can be used to combine multiple conditions to more accurately describe the triggering conditions of the rule.

[0151] Step S302: Generate and execute SQL based on the rule expression to obtain and update the error count statistics and error details.

[0152] Specifically, this embodiment parses the target validation rule, obtains the rule expression, and parses the rule expression by referring to the rule definition table in the above embodiments to generate the corresponding SQL for the database. This embodiment uses the Gauss database environment as an example, therefore the SQL proposed in this embodiment is applicable to the Gauss database. This embodiment obtains and updates the error count statistics and error detail results by executing the SQL for error count statistics and the SQL for error detail extraction, thereby obtaining the data validation result.

[0153] More specifically, refer to Figure 9 , Figure 9 This is a schematic diagram illustrating the workflow of implementing the data verification method of this application, where one configuration is used in multiple locations for execution; for example... Figure 9 As shown, in the data processing flow for regulatory reporting, there is a long processing chain from the business source system to the final report, involving more than one team. To ensure that the responsible units at each stage of the processing chain are accountable for the quality of their respective outputs, this embodiment adapts to multiple data sources. This allows each processing stage to verify its output data after processing through simple configuration. A single set of rules runs simultaneously at each stage of the processing chain, thereby achieving the goals of forward verification and source governance. It also saves the cost of redundant development by each team.

[0154] Furthermore, referring to Figure 10 , Figure 10 This is an example SQL diagram generated based on the verification rules for the data verification method in this application; such as... Figure 10 As shown, this embodiment can dynamically generate SQL rules adapted to the corresponding database by parsing rule expressions based on input parameters at runtime, achieving one-time configuration and multiple executions, and deploying the verification process at different stages of the data processing chain. Among these, Figure 10 The validation rule is "the employee's name cannot be empty"; Figure 10 In this context, `V_PREFIX` and `V_SUFFIX` are prefix parameters for the employee table. These prefix parameters can be any custom string used to identify the name of a database object, and are named according to specific specifications. They can be used to distinguish different types of database objects, such as tables, views, and stored procedures. During execution, `${V_PREFIX}` needs to be replaced with the specific database schema name, while `${V_SUFFIX}` is used to meet the personalized definition requirements for table names in the processing chain and can be set in the running parameters of the data validation engine. Figure 10 XM in the variable is a user-defined variable that stores the employee's name; Figure 10 In this context, dw_snsh_dt = ${TX_DATE} represents the setting of the data period for running the data validation engine, and serves as one of the triggering conditions for "Employee table name cannot be empty".

[0155] Step S303: Determine whether there are any unparsed rules in the target verification rules;

[0156] Specifically, this embodiment determines whether there are any unparsed rules in the target verification rules to ensure that all target verification rules that should be executed are fully executed.

[0157] Step S304, if yes, then execute the following steps: traverse the target verification rules and select the rule expression to be parsed;

[0158] Specifically, if there are unparsed rules in the target verification rules, the process jumps to step S301, continues to traverse the rules and select the rule expression to be parsed; then steps S302 and S303 are executed until all the rule expressions corresponding to the target verification rules are parsed, and the loop is exited.

[0159] Step S305: If not, obtain the data verification result based on the error count statistics and the error details.

[0160] Specifically, in this embodiment, the updated error count statistics and error details are combined and synchronized to the verification result interface table (which stores the error count statistics) and the error details interface table (which stores the error details) in the statistical reporting platform to obtain the data verification result.

[0161] Furthermore, this embodiment further refines step S302, which involves generating and executing SQL based on the rule expression to obtain and update the error count statistics and error details.

[0162] In this embodiment, step S302, generating and executing SQL based on the rule expression to obtain error count statistics and error details, includes:

[0163] Step S3021: Parse the rule expression to generate the first type of SQL and the second type of SQL;

[0164] Specifically, the first type of SQL is an error count SQL, used to count the number of errors found during the data verification process, and the second type of SQL is an error detail extraction SQL, used to extract the specific details of the errors found during the data verification process.

[0165] Step S3022: Execute the first type of SQL to obtain and update the error count statistics;

[0166] Specifically, in this embodiment, the SQL execution result is obtained by executing the script of the first type of SQL, thereby obtaining the error count result, and the error count result is saved in a temporary storage area.

[0167] Step S3023: Execute the second type of SQL based on the error statistics results to obtain and update the error details results.

[0168] Specifically, this embodiment first requires ensuring a successful connection to the database containing error detail data. Using an appropriate database connection tool, provide the correct database hostname, port number, username, and password to connect to the database server. Then, copy the second type of SQL into an SQL editor. As needed, modify the filter conditions in the SQL statement to obtain error details based on specific date ranges, error types, or other relevant conditions, thus obtaining the error detail results.

[0169] This embodiment, through the above-described scheme, obtains input parameters in response to a verification execution command; matches the input parameters against predefined verification rules to obtain target verification rules; parses the target verification rules to generate and execute structured query statements (SQL) to obtain data verification results. This embodiment adapts to multiple data sources, allowing each processing stage to verify its output data after processing with simple configuration. A single rule configuration runs simultaneously at each stage of the processing chain, achieving the goals of forward verification and source governance. It also saves the cost of redundant development by various teams. This method can be applied to different stages in the data processing chain, realizing forward verification, identifying problems at the source for rapid resolution, thereby improving data verification effectiveness and enhancing the timeliness of data verification.

[0170] It should be noted that the above embodiments can be combined and implemented in a reasonable manner according to the actual situation, and this embodiment will not elaborate on this further.

[0171] Furthermore, embodiments of this application also provide a data verification device, the data verification device comprising:

[0172] The data acquisition module is used to acquire input parameters in response to the verification execution command;

[0173] The rule configuration module is used to match predefined verification rules based on the input parameters to obtain the target verification rule;

[0174] The rule parsing module is used to parse the rules based on the target verification rules, generate and execute structured query statements (SQL), and obtain data verification results.

[0175] The principle and implementation process of data verification in this embodiment are explained in the above embodiments and will not be repeated here.

[0176] Furthermore, this application also proposes a terminal device, which includes a memory, a processor, and a data verification program stored in the memory and executable on the processor. When the data verification program is executed by the processor, it implements the steps of the data verification method described above.

[0177] Since this data verification program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.

[0178] Furthermore, embodiments of this application also provide a computer-readable storage medium, on which a data verification program is stored, and when the data verification program is executed by a processor, it implements the steps of the data verification method described above.

[0179] Since this data verification program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.

[0180] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0181] The order of the above embodiments of the present invention is merely for description and does not represent the superiority or inferiority of the embodiments.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0183] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A data verification method, characterized in that, The method is applied in a database environment, and the data verification method includes the following steps: In response to the verification execution command, obtain the input parameters; Based on the input parameters, the predefined verification rules are matched to obtain the target verification rules. The target verification rules include at least one or more of the following: single table verification rules, association verification rules, multi-table verification rules, and partial data verification rules within a table. The target verification rules can be deployed in the source processing stage, data integration stage, and reporting stage of the data processing chain. Based on the target verification rules, rule parsing is performed to generate and execute structured query statements (SQL) to obtain data verification results; The steps of parsing the target validation rules, generating and executing structured query statements (SQL), and obtaining the data validation results include: Iterate through the target validation rules and select the rule expression to be parsed; SQL is generated and executed based on the rule expression to obtain and update the error count statistics and error details. Specifically, a first type of SQL and a second type of SQL are generated by parsing the rule expression; the first type of SQL is executed to obtain and update the error count statistics; and the second type of SQL is executed based on the error statistics to obtain and update the error details. Determine whether there are any unparsed rules in the target validation rules; if so, proceed to the following steps: traverse the target validation rules and select the rule expression to be parsed; if not, obtain the data validation result based on the error count statistics and the error details.

2. The data verification method as described in claim 1, characterized in that, Before the step of filtering validation rules based on the input parameters to obtain the target validation rule, the following steps are included: The validation rules are defined, specifically including: Retrieve rule configuration template information; The verification rules are obtained by defining rules based on the rule configuration template information.

3. The data verification method as described in claim 2, characterized in that, After the step of defining rules based on the rule configuration template information to obtain the verification rules, the method further includes: The verification rules are synchronized to a pre-created verification rule table to obtain a verification rule definition table.

4. The data verification method as described in claim 1, characterized in that, The step of matching the predefined verification rules based on the input parameters to obtain the target verification rule includes: Data table information is obtained by parsing the input parameters. The target verification rule is obtained by matching the predefined verification rules with the information in the data table.

5. A data verification device, characterized in that, The data verification device includes: The data acquisition module is used to acquire input parameters in response to the verification execution command; The rule configuration module is used to match predefined verification rules based on the input parameters to obtain target verification rules. The target verification rules include at least one or more of the following: single table verification rules, association verification rules, multi-table verification rules, and partial data verification rules within a table. The target verification rules can be deployed in the source processing stage, data integration stage, and reporting stage of the data processing chain. The rule parsing module is used to parse the rules based on the target validation rules, generate and execute structured query statements (SQL), and obtain data validation results. The rule parsing module is further configured to traverse the target validation rules, select the rule expression to be parsed; generate and execute SQL based on the rule expression to obtain and update the error count statistics and error details, wherein, by parsing the rule expression, a first type of SQL and a second type of SQL are generated; the first type of SQL is executed to obtain and update the error count statistics; the second type of SQL is executed based on the error count statistics to obtain and update the error details; determine whether there are any unparsed rules in the target validation rules; if so, execute the following steps: traverse the target validation rules and select the rule expression to be parsed; if not, obtain the data validation result based on the error count statistics and the error details.

6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a data verification program stored in the memory and executable on the processor. When the data verification program is executed by the processor, it implements the data verification method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data verification program, which, when executed by a processor, implements the data verification method as described in any one of claims 1-4.

Images