System and method for describing data verification logic

Abstract

The present invention provides a system and method for describing data verification logic, which relates to the field of data processing technology. The method includes: a data source analysis module: based on the conceptual division of the three elements of data verification, it provides online real-time data source definition, legality check and related analysis functions, and extracts metadata processing into data source knowledge for use by subsequent modules; a compilation module: receiving the data source knowledge provided by the data source analysis module, performing joint analysis, and performing logic optimization work to generate the final logic representation, which is then handed over to a specific system, framework or platform to complete the calculation, thereby realizing the loose coupling of the three elements. The present invention can realize the complete decoupling of the three elements of data source, verification logic and verification output, significantly simplify the development, testing and deployment processes, and improve the maintainability of verification logic.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a system and method for describing data checking logic. Background Art

[0002] Financial supervision has put forward increasingly stringent requirements on the data quality of the regulatory data standardization (EAST) of major banks. EAST data has the characteristics of comprehensive business coverage, large amount of detailed data, strict data quality requirements, and high reporting timeliness. In order to continuously improve the quality of EAST data, it is urgent to build a safe and efficient massive data quality detection system.

[0003] In large-scale data quality detection and monitoring, a logical description system is needed to accurately describe the data inspection and verification logic, referred to as verification logic or verification rules. The common solution in the industry uses SQL to describe the verification logic. Although it is simple to learn and widely used, it has many disadvantages in the specific field of data verification, such as tight coupling of functional elements, strong binding of storage products, lack of data security, and low development and deployment efficiency.

[0004] The existing system has the following problems:

[0005] 1) Using SQL to describe the verification logic makes the three verification elements tightly coupled, leading to inefficient development, testing, deployment, and maintenance: Data verification calculations rely on data source, verification logic, and verification output. SQL mixes the three elements together. For the same data source and verification logic, multiple similar SQL statements need to be written just because the output results are different (statistics, sampling, etc.), resulting in a lot of redundancy. In addition, SQL mixes the three elements of information, which is not very readable. When the verification logic is complex (such as Figure 2 ), the boundaries of the three elements will become very vague, and the entire SQL statement will be difficult to understand and maintain. Using the data verification logic description system of the present invention, the three elements can be completely decoupled. A data source only needs to be defined once. Based on the same data source, any number of verification rules can be defined. Before executing the verification logic, there is no need to consider any issues related to the verification output. The development, testing, deployment and maintenance links have been streamlined to varying degrees.

[0006] 2) Data security issues caused by using SQL: Without a custom SQL compiler to restrict the writing of SQL language, data security will be greatly challenged. In this invention, the self-developed SQL parser is used to constrain the definition of the data source, and the compiler is used to complete various checks such as data type and reference using the verification logic description, which significantly improves the data and computing security.

[0007] 3) The use of SQL brings about repeated development of multiple database backends and logical consistency issues: Enterprise-level data verification requires a large amount of atomic computing logic ( Figure 3 ), the SQL language has limited expressive power and must rely on specific database products for secondary development, which will inevitably lead to the strong binding of the implementation of atomic logic to the database product. When data in multiple databases within an organization need to be verified, the same atomic logic must be repeatedly developed for these products, resulting in a waste of development resources. What is more difficult is that the calculation consistency problems caused by different database products require a lot of testing resources to be invested to solve. Using the data verification logic description system in the present invention, all atomic calculations are supported by the system standard library, do not depend on specific database products, do not need to be repeatedly implemented, and do not have multi-terminal logic consistency problems. The verification logic can be written once and used in multiple places. Summary of the invention

[0008] In view of the defects in the prior art, the present invention provides a system and method for describing data checking logic.

[0009] According to a system and method for describing data checking logic provided by the present invention, the scheme is as follows:

[0010] In a first aspect, a system for describing data checking logic is provided, the system comprising:

[0011] Data source analysis module: Based on the conceptual division of the three elements of data verification, it provides online real-time data source definition, legality check and related analysis functions, and extracts metadata into data source knowledge for use by subsequent modules;

[0012] Compilation module: provides online real-time verification of logic description editing, checking and analysis functions; during the compilation process, it receives the data source knowledge provided by the data source analysis module to conduct joint analysis, and performs logic optimization to generate the final logic representation, which is then handed over to a specific system, framework or platform to complete the calculation and output the verification results, thereby achieving loose coupling of the three elements.

[0013] Preferably, the data source analysis module includes an SQL parser, and the ANSI SQL edited and submitted online is parsed by the SQL parser, and after passing, a legal data source definition is generated;

[0014] The SQL parser is responsible for checking the submitted SQL and performs the following four functions:

[0015] Security check: prohibit any non-select statements from being compiled at the syntax level, ensuring that any SQL statement cannot change or delete existing data in the system, thus ensuring data security;

[0016] Availability check: Check whether the submitted statement complies with the select clause syntax rules and serves as a compliance description of the source definition;

[0017] SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define in subsequent verification rules;

[0018] Data source knowledge extraction: After the above SQL normalization is completed, the data source knowledge is extracted from the statement, including the reference name, reference type and namespace information, to provide metadata information support for the subsequent compilation module.

[0019] Preferably, the SQL parser also includes a metadata extraction unit, which interacts online with the specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and availability checks more complete; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for use by subsequent modules and specific data verification and computing systems.

[0020] Preferably, the metadata extraction unit shields the technical differences of backend database products, unifies the expression form of relational metadata, and provides a consistent interface externally on the basis of performing basic functions.

[0021] Preferably, the compiling module comprises:

[0022] Lexical analysis submodule: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream processing to token stream for analysis by the syntax analysis submodule;

[0023] Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it builds a concrete syntax tree based on the LL(*) algorithm according to the syntax rules described by the check logic, and then generates an abstract syntax tree, and finally submits the abstract syntax tree to the semantic analysis submodule;

[0024] Semantic analysis submodule: receives the data source knowledge provided by the data source analysis module, performs joint analysis, and performs type and logic checks on the name references mentioned in the verification rules;

[0025] Optimization and logic generation submodule: After passing the semantic analysis submodule, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation.

[0026] After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, and trims the data structure to generate the final logical representation, which serves as a general interface to adapt to the specific data verification and computing system.

[0027] In a second aspect, a method for describing data checking logic is provided, the method comprising:

[0028] Data source analysis step: Based on the conceptual division of the three elements of data verification, perform online real-time data source definition, legality check and related analysis, and process and extract metadata into data source knowledge for use in subsequent steps;

[0029] Compilation step: Perform online real-time verification of the logic description editing, checking and analysis. During the compilation process, the data source knowledge provided by the data source analysis step is received for joint analysis, and logic optimization is performed to generate the final logic representation, which is then handed over to a specific system, framework or platform to complete the calculation and output the verification results, thus achieving loose coupling of the three elements.

[0030] Preferably, the data source analysis step includes a SQL parser, and the ANSI SQL edited and submitted online is parsed by the SQL parser, and after passing, a legal data source definition is generated;

[0031] The SQL parser is responsible for checking the submitted SQL and mainly performs the following four functions:

[0032] Security check: prohibit any non-select statements from being compiled at the syntax level, ensuring that any SQL statement cannot change or delete existing data in the system, thus ensuring data security;

[0033] Availability check: Check whether the submitted statement complies with the select clause syntax rules and serves as a compliance description of the source definition;

[0034] SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define in subsequent verification rules;

[0035] Data source knowledge extraction: After the above SQL normalization is completed, the data source knowledge is extracted from the statement, including the reference name, reference type and namespace information, to provide metadata information support for the subsequent compilation steps.

[0036] Preferably, the SQL parser also includes a metadata extraction unit, which interacts online with a specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and availability checks more complete; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for use in subsequent steps.

[0037] Preferably, the metadata extraction unit shields the technical differences of backend database products, unifies the expression form of relational metadata, and provides a consistent interface externally on the basis of performing basic functions.

[0038] Preferably, the compiling step comprises:

[0039] Lexical analysis sub-step: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream to token stream for analysis by the syntax analysis sub-step;

[0040] Syntax analysis sub-step: After receiving the token stream provided by the lexical analysis sub-step, a concrete syntax tree is constructed based on the LL(*) algorithm according to the syntax rules described by the check logic, and then an abstract syntax tree is generated, which is finally submitted to the semantic analysis sub-step;

[0041] Semantic analysis sub-step: Receives the data source knowledge provided by the data source analysis step, performs joint analysis, and performs type and logic checks on the name references mentioned in the verification rules;

[0042] Optimization and logic generation sub-step: After the semantic analysis sub-step, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation.

[0043] After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, trims the data structure, and generates the final logical representation as a general interface to adapt to specific computing systems, frameworks, or platforms.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The data verification logic description system of the present invention adopts online compilation to realize online real-time editing and feedback of data source definition and verification rule definition. In the semantic analysis stage, the data source analysis module is connected with the back-end database product to obtain relevant metadata and improve computing security. The verification logic description design in the system adopts strong type checking at compile time to improve overall security.

[0046] 2. The verification logic description in the present invention appropriately introduces Chinese programming, which improves the usability and expands the user group;

[0047] 3. The final logical representation generated after compiling the verification logic description in the present invention is a data structure that is independent of the specific computing platform and framework. It stores complete and typed verification logic description information, can be adapted to a variety of computing platforms and frameworks, and provides verification execution code generation support for specific platforms and frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1Describe the collaboration relationship between the verification logic and the verification calculation;

[0050] Figure 2 Define Chinese support diagram for verification rules;

[0051] Figure 3 Schematic diagram of using standard library functions for checking logic description;

[0052] Figure 4 Describe the system block diagram for the check logic;

[0053] Figure 5 Define process diagrams for data sources;

[0054] Figure 6 This is a diagram of the feedback for the SQL parser security check;

[0055] Figure 7 This is a diagram of the feedback for the SQL parser security check;

[0056] Figure 8 A diagram showing feedback for SQL parser usability check;

[0057] Fig. 9 This is a diagram of SQL normalization for SQL parser;

[0058] Fig.10 This is a diagram of the SQL parser structure and workflow;

[0059] Fig.11 A schematic diagram of the compilation process for the verification logic description;

[0060] Fig.12 This is an example of a compilation error prompt;

[0061] Fig.13 This is a diagram showing the mismatch of branch expression types;

[0062] Fig.14 This is a diagram showing the mismatch of function parameter types;

[0063] Fig.15 A reference error diagram is provided for the verification logic description;

[0064] Fig.16 It is a schematic diagram of the verification engine module and structure;

[0065] Fig.17 Generate structure and flow diagram for code;

[0066] Fig.18 This is a schematic diagram of the Spark framework embedded in the verification engine. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0068] The present invention provides a system for describing data checking logic, which specifically includes:

[0069] Data source analysis module: Based on the conceptual division of the three elements of data verification, it provides online real-time data source definition, legality check and related analysis functions, and extracts metadata into data source knowledge for use by subsequent modules.

[0070] Compilation module: provides online real-time verification of logic description editing, checking and analysis functions; during the compilation process, it receives the data source knowledge provided by the data source analysis module to conduct joint analysis and perform logic optimization, and then generates the final logic representation, which is then handed over to a specific system, framework or platform to complete the calculation, thereby achieving loose coupling of the three elements.

[0071] The data source analysis module includes an SQL parser. The ANSI SQL edited and submitted online is parsed by the SQL parser. After passing the parsing, a legal data source definition is generated.

[0072] The SQL parser is responsible for the first round of checking of the submitted SQL and performs the following four functions:

[0073] Security check: prohibit any non-select statements from being compiled at the syntax level, ensure that any SQL statement cannot change or delete existing data in the system, and ensure data security.

[0074] Availability check: Checks whether the submitted statement complies with the select clause syntax rules and serves as a compliant description of the source definition.

[0075] SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define in subsequent verification rules.

[0076] Data source knowledge extraction: After the above SQL normalization is completed, the data source knowledge is extracted from the statement, including reference name, reference type, and namespace information, to provide metadata information support for subsequent modules.

[0077] The SQL parser also includes a metadata extraction unit, which interacts online with the specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and availability checks more complete; at the same time, the SQL parser processes and extracts metadata into data source knowledge for use by subsequent modules.

[0078] While performing its basic functions, the metadata extraction unit shields the technical differences of back-end database products, unifies the expression of relational metadata, and provides a consistent interface to the outside world.

[0079] The SQL is replaced by data checking logic description, and the compilation module includes:

[0080] Lexical analysis submodule: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream to token stream for analysis by the syntax analysis submodule.

[0081] Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it builds a concrete syntax tree based on the LL(*) algorithm according to the grammatical rules of the verification language, and then generates an abstract syntax tree, and finally submits the abstract syntax tree to the semantic analysis submodule.

[0082] Semantic analysis submodule: receives the data source knowledge provided by the data source analysis submodule, performs joint analysis, and performs type and logic checks on the name references mentioned in the verification rules.

[0083] Optimization and logic generation submodule: After passing the semantic analysis submodule, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation.

[0084] After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, trims the data structure, and generates the final logical representation as a general interface to adapt to specific computing systems, frameworks, or platforms.

[0085] The present invention also provides a method for describing data checking logic, comprising:

[0086] Data source analysis step: Based on the conceptual division of the three elements of data verification, perform online real-time data source definition, legality check and related analysis, and process and extract metadata into data source knowledge for use in subsequent steps.

[0087] Compilation step: Perform online real-time verification of logical description editing, checking and analysis; during the compilation process, receive the data source knowledge provided by the data source analysis step to conduct joint analysis, and perform logic optimization to generate the final logical representation, which is then handed over to a specific system, framework or platform to complete the calculation and output the verification results, thereby achieving loose coupling of the three elements.

[0088] The data source analysis step includes an SQL parser. The ANSI SQL edited and submitted online is parsed by the SQL parser. Once parsed, a legal data source definition is generated.

[0089] The SQL parser is responsible for checking the submitted SQL and mainly performs the following four functions:

[0090] Security check: prohibit any non-select statements from being compiled at the syntax level, ensure that any SQL statement cannot change or delete existing data in the system, and ensure data security.

[0091] Availability check: Checks whether the submitted statement complies with the select clause syntax rules and serves as a compliance description of the source definition.

[0092] SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define in subsequent verification rules.

[0093] Data source knowledge extraction: After the above SQL normalization is completed, the data source knowledge is extracted from the statement, including the reference name, reference type and namespace information, to provide metadata information support for the subsequent compilation steps.

[0094] The SQL parser also includes a metadata extraction unit, which interacts online with the specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and availability checks more complete; at the same time, the SQL parser processes and extracts metadata into data source knowledge for use in subsequent steps.

[0095] While performing its basic functions, the metadata extraction unit shields the technical differences of back-end database products, unifies the expression of relational metadata, and provides a consistent interface to the outside world.

[0096] The compilation steps include:

[0097] Lexical analysis sub-step: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream to token stream for analysis by the syntax analysis sub-step.

[0098] Syntax analysis sub-step: After receiving the token stream provided by the lexical analysis sub-step, a specific syntax tree is constructed based on the LL(*) algorithm according to the syntax rules described by the verification logic, and then an abstract syntax tree is generated, which is finally submitted to the semantic analysis sub-step.

[0099] Semantic analysis sub-step: Receive the data source knowledge provided by the data source analysis step, perform joint analysis, and perform type and logic checks on the name references mentioned in the verification rules.

[0100] Optimization and logic generation sub-steps: After the semantic analysis sub-step, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation.

[0101] After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, trims the data structure, and generates the final logical representation as a general interface to adapt to specific computing systems, frameworks, or platforms.

[0102] Next, the present invention will be described in more detail.

[0103] The embodiment of the present invention includes a system for describing data check logic, which will be described in detail in the first part below. At the same time, in order to fully and more accurately show the advantages of the check logic description system in this embodiment and a typical application scenario, a check calculation system will be supplemented in the second part to reflect a complete check application scenario. The relationship between the first part (logic description) provided by the present invention and the second part (check calculation) as an additional supplementary explanation is as follows: Figure 1 The following will be described in detail in two parts.

[0104] Part 1: The present invention provides a system for describing data checking logic.

[0105] The description of data verification logic usually contains the following three types of information:

[0106] (1) Data source: the source and data structure information of the data to be audited. For example, a data source definition can be given through SQL: select...from A where date='20211220'. This statement indicates that the data source comes from table A and the date is '20211220'.

[0107] (2) Verification logic: Based on a given data source definition, several verification logics, also called verification rules, can be defined. Each rule can be regarded as the smallest unit of a verification calculation. Usually, a data source will define dozens to hundreds of verification rules to complete a comprehensive and detailed data quality verification. Take a simple verification logic as an example: After adding the verification rules to the data source defined in (1), we can get select...from A where date='20211220'and col is notnull. It can be seen that the core logic of the verification rule is mixed with the data source elements, and is actually just a part of the where clause, that is, "col is not null".

[0108] (3) Verification output: describes the structure and purpose of saving the verification calculation results. Taking the SQL statement in (2) as an example, adding a verification output definition can be obtained: select count(*)from A where date='20211220'andcol is not null, that is, the output is a statistical analysis of the data that meets the conditions.

[0109] Data source, verification logic, and verification output are called the three elements of data verification. From the above examples, we can see that traditional SQL mixes these three elements together to form a tight coupling. The present invention takes this as a breakthrough point and completely decouples the three elements through the design and implementation of a new type of logical description, thereby establishing advantages in many aspects and achieving remarkable results.

[0110] In terms of logical description design, the main focus is on the construction and selection of syntax, semantics, features, etc., so that users can safely, accurately, flexibly and conveniently define data sources and verification logic.

[0111] In terms of the implementation of logical description, based on the design work, from an engineering perspective, a supporting compiler and tool chain are implemented to support common usage scenarios, such as online real-time compilation, error feedback, syntax highlighting, logic checking and representation generation, etc., which mainly include data source analysis module and compilation module.

[0112] The following will describe the design and implementation in detail:

[0113] In the design part, the logic description is dedicated to data verification. On the basis of meeting the basic logic construction capabilities, the learning and usage thresholds are minimized. Therefore, basic Chinese support and some grammatical sugars of Chinese expressions are added at the grammatical level, such as Figure 2 As shown in the rule definition section, branch keywords can be written in the form of "if..then..otherwise.." or "if..then..else..", and comparison symbols can be written in Chinese (such as "equals") or mathematical symbols (such as ">="). In addition, some string calculations can also use Chinese expressions, and the syntax conforms to Chinese usage habits.

[0114] Logic includes basic syntax such as basic branch judgment, function call, logical operation, numerical operation, etc. Data has strict type division, including Boolean, integer (byte, short, int, long), floating point (float, double), character, string, date, timestamp, regular expression, etc. For example, Figure 2 The rule definition in the example constructs a date literal '2020-01-01' using the "date" keyword, "true" and "false" are Boolean literals, and so on.

[0115] In order to improve the determinism of calculations, inspired by functional programming, all grammatical structures are expressions, without statements, and the evaluation process does not contain any side effects. Taking Java as a comparison, "if...else..." is a statement with side effects, while in the logic description design of the present invention, "if...then...else..." is an expression without side effects, and its evaluation result is the calculation result of one of the branches. This requires that the calculation result type of each branch is the same. If the type is different, an error will be prompted. Figure 2 The final evaluation result is equal to the evaluation result of the branch judgment expression.

[0116] In order to improve the safety of calculations, the logical description will be subject to strict strong type checking. If it does not meet the requirements of the calculation type, an error will be quickly fed back during compilation.

[0117] Finally, all functions used in the logic description are from the standard library and can be continuously expanded, which facilitates the efficient writing of verification logic and avoids lengthy logic or repeated definitions. The function call example is as follows Figure 3 shown.

[0118] In general, the logic description design follows the principles of safety, flexible expression, simple writing, and easy understanding.

[0119] Implementation part, such as Figure 4 The sub-modules are described as follows:

[0120] Data source analysis module: Based on the division of the above three elements, this module is responsible for completing the definition of the data source, providing online data source definition, real-time legality check and related analysis functions. In order to maintain universality, flexibility and security, the syntax of data source definition is based on a subset of the ANSI SQL language, which only contains select query clauses. This module contains a SQL parser to support the core functions. The workflow is as follows: Figure 5 As shown in the figure, the SQL edited and submitted online is parsed by the SQL parser. Once parsed, a legal data source definition is generated. The SQL parser implements a complete SQL compiler front end, which is responsible for checking the submitted SQL and mainly performs the following four functions:

[0121] (1) Safety check: Figure 6 , 7 As shown in the figure, any non-select statements are prohibited from being compiled at the syntax level to ensure that SQL statements cannot change or delete existing data in the system, thus ensuring data security.

[0122] (2) Availability check: Figure 8 As shown, check whether the submitted statement complies with the select clause syntax rules and can be used as a legal description of the data source definition;

[0123] (3) SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define subsequent verification rules. For example, Fig. 9 As shown, the asterisk (*) in the select statement is fully expanded to facilitate name references and related checks when compiling the module to check rule definitions later;

[0124] (4) Data source knowledge extraction: After the completion of step (3) above, data source knowledge can be extracted from the statement, including reference name, reference type, namespace, etc. For example, in the data source definition "select a, b, c from table", after being extracted by the SQL parser, the data source knowledge will contain references such as a, b, c, table and their types. This information not only provides metadata information support for subsequent compilation modules, but also provides runtime information support during the verification calculation process. It is worth noting that the data source knowledge follows the reference-based principle to retain all metadata. The reference-based representation has many advantages, enabling the module to support semantic access to metadata related to structured and semi-structured data at the same time, and also to support semantic access to streaming data. As a general data structure, data source knowledge provides a unified standard data interface.

[0125] The internal structure and workflow of the SQL parser are as follows Fig.10 As shown in the figure, after the SQL statement has undergone basic lexical and grammatical analysis, in the semantic analysis stage, the metadata extraction unit will interact online with the specific back-end database product to obtain all metadata related to the SQL statement, thereby supporting more stringent semantic analysis and further improving security and availability checks. At the same time, the SQL parser will process and extract these metadata into data source knowledge for use by subsequent compilation modules.

[0126] It is worth noting that, due to its functional limitations, the SQL parser does not contain any backend functions for SQL compilation and cannot truly "execute" SQL; in addition, as a connection point for multiple database products, the metadata extraction unit, while performing its basic functions, shields the technical differences of backend database products, unifies the expression of relational metadata, and provides a consistent interface to the outside world.

[0127] This module completes the definition and analysis work related to the data source, and provides metadata support for subsequent compilation modules, so that the verification rules can obtain clear computational semantics in a specific context. The data source knowledge generated by the SQL parser is a bridge connecting the data source and the verification logic, and also serves to separate the two.

[0128] Compilation module: focuses on compiling analysis, optimizing and generating final representation of the logic description. The compilation process is as follows: Fig.11 As shown in the figure, the checking rules are analyzed and calculated in turn through the sub-modules of lexical, grammatical, semantic, optimization and logic generation, and finally generate a logical intermediate representation that is independent of the specific computing platform and database product. If an error occurs in any of these links, the compilation will be interrupted, and detailed compilation error information will be fed back in advance, so that users can quickly correct the logic. For example Fig.12 As shown in the figure, the expression on the right is mistakenly written as the string 5. The types on both sides of the inequality sign are inconsistent, so the compilation fails and an error is prompted.

[0129] The following will be described according to the sub-modules:

[0130] Lexical analysis submodule: Starting from the grammatical level of the verification logic, the verification rules are segmented and converted from character streams to token streams for analysis by the grammatical analysis submodule.

[0131] Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it builds a concrete syntax tree (CST) based on the LL(*) algorithm according to the syntax rules of the verification logic, and then generates an abstract syntax tree (AST), and finally submits the abstract syntax tree to the semantic analysis submodule.

[0132] Semantic analysis submodule: receives the data source knowledge provided by the data source analysis module and performs semantic analysis on the verification logic. In order to expose more logical errors in advance, strong type checking can ensure the safety of type-based calculations (such as Fig.12 , 13 , 14); the check must also ensure that the evaluation result type of the verification logic is always Boolean to reflect a clear judgment on the data quality; in addition, some rule-based type conversions can be allowed during the analysis process to improve the language usability while maintaining safety.

[0133] like Fig.15 As shown in the figure, the check analysis also includes the name references and logical consistency constraints mentioned in the check rules to fully ensure that the check rule description is complete and computable. After all the analyses are completed, an intermediate representation with type information (Typed IR) will be generated, referred to as the intermediate representation, and submitted to the subsequent optimization and logic generation submodules.

[0134] Optimization and logic generation submodule: Once the semantic analysis submodule is passed, it means that the current verification rules have passed all the checks. The main purpose of this module is to perform logic optimization work, such as Fig.11As shown in the figure, in each round of optimization, the optimizer accepts an intermediate representation and calculates a logically equivalent but more optimized intermediate representation. For example, an important optimization method is to advance all calculations containing only literals, such as 1+1 can be directly optimized to 2, without having to perform a large number of inefficient repeated calculations during the actual execution of the check rules.

[0135] After the optimizer completes multiple rounds of optimization, it can generate an optimal intermediate representation, remove some process results, trim the data structure, and generate the final logical representation. The logical representation is a data structure with data types and auxiliary calculation information. It contains all the information required for calculation and is independent of the specific database product. It can be used as a general interface to adapt to specific computing engines, frameworks or platforms. For details, see the second part below - data verification calculation system.

[0136] At this point, the data source definition and the verification rule definition have been completely decoupled. Since the verification output is closely related to the data persistence layer, it can be completed by a specific computing engine, framework or platform. For details, see the second part below - the result output module in the data verification calculation system.

[0137] Through appropriate modification, this system can add the following expansions:

[0138] (1) Adding support for analysis of semi-structured data: In the above data source analysis module, structured data is described using ANSI SQL, and the SQL parser is responsible for parsing and ultimately extracting data source knowledge. Similarly, this module adds json to describe the structure and data type of semi-structured data, and generates data source knowledge through the json parser. Since data source knowledge preserves metadata based on the principle of preserving references, the interface layer remains stable, and the entire compilation module can be seamlessly connected. The entire system can implement the verification rule analysis function based on semi-structured data sources, and also provide information support for the subsequent expansion of the data verification calculation system.

[0139] (2) Add support for analysis of streaming data: Similar to the idea in (1), while maintaining the reference principle, add a DSL (such as YAML) for describing the structure and type of streaming data, write a parser, generate standard data source knowledge, and then connect to the compilation module, so that the system can analyze the verification rules based on the streaming data source.

[0140] (3) Continuously optimize the final logical representation: In the above optimization and logic generation submodules, the optimizer performs progressive optimization on the intermediate representation. As the system functions expand, the optimizer can continue to add more layers of analysis and optimization, continuously improve the final logical representation, enhance the computational expression capability, and help the verification calculation link generate higher performance execution code.

[0141] On the whole, the multiple modules of this system work closely together to achieve complete decoupling of the three elements of data source, verification logic, and verification output. With the help of online real-time compilation and feedback, users can quickly complete the definition of data source and verification logic, significantly simplifying the development, testing, and deployment processes. The appropriate amount of Chinese is introduced to achieve complementary advantages between Chinese and ordinary writing, improve the readability and maintainability of the verification logic, lower the threshold for use, and reduce costs and increase efficiency; relying on a large number of strict syntax, semantic checks and strong type checks during the compilation period, the security of data and calculations is significantly improved; because the data source definition adopts the ANSI SQL subset, and the verification logic description only relies on the standard data source knowledge interface provided by the data source analysis module, it is independent of the specific database product, which completely solves the problem of logical consistency of multiple database backends, achieving the effect of writing once and running in multiple places.

[0142] At this point, the description of the system for describing the data check logic provided by the present invention is completed. In order to present a complete picture of the check application, the following second part will provide supplementary description based on the system for describing the data check logic.

[0143] Part II: A data verification computing system.

[0144] Based on the work of the data verification logic description system in the first part, the data verification calculation system aims to complete large-scale verification calculation. The core of this system is the verification engine, which provides a complete verification calculation runtime environment. The basic modules and structure diagram refer to Fig.16 As shown, the calculation core module is responsible for the actual calculation. The required target verification data and verification calculation logic are provided by the data input module and the code generation module respectively. After the calculation is completed, the result output module completes the persistence of the verification result.

[0145] The following is a detailed description of each module:

[0146] Data input module: extracts target verification data based on data source related information and submits it to the computing core module. The data source related information comes from the data source knowledge and standardized SQL ( Fig. 9 , 10 ), using this information, this module can fully perceive the structure and source of the data to be checked, and continuously provide data to the computing core module in a stable and secure manner.

[0147] Code generation module: responsible for converting the verification rules related to the target verification data into executable code in batches and submitting them to the computing core module for execution. The internal structure and workflow of the module are as follows: Fig.17As shown in the figure: The final logic representation and data source knowledge are compiled and generated by the data verification logic description system. After receiving this information, the translator traverses the final logic representation and combines the data source knowledge to generate efficient executable code. Inspired by Lambda calculus, the executable code is composed of several orthogonal basic operators. After the verification logic is compiled, it is expressed as an algebraic combination of basic operators. This set of operators can be considered as the "assembly language" of the entire verification engine.

[0148] Computing core module: receives the executable code compiled from the verification data and verification rules, performs calculations and obtains results. The core component of this module is an interpreter for the above basic operators. In order to improve concurrency and strengthen runtime control, the interpreter has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and executed by different threads. In addition, the interpreter has a built-in execution information collection unit (Metrics Unit), which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization. For example, after collecting the execution information of the branch logic, the hit rate of each branch logic is regularly counted. When the hit rate of a branch reaches the threshold, the optimization logic can be triggered to make the branch and the judgment logic execute concurrently, further improving the calculation concurrency.

[0149] In large-scale data verification and calculation scenarios, the verification of a data source usually involves the calculation of dozens or even hundreds of verification rules. Obviously, making full use of the multi-core characteristics of modern CPUs and merging all verification rules that rely on the same data source can save IO resources and improve computing efficiency. Based on this theory, this module designs and implements an aggregation algorithm that only requires one IO of the data source to complete the calculation of any number of verification rules from the same source, and maintains the independence of the calculation results of each rule, achieving "integration but difference".

[0150] It is worth noting that although the check calculation is interpreted at the check engine level, at the entire JVM level, all structures including the interpreter will eventually be compiled into JVM bytecode for execution, which is no different from ordinary programs.

[0151] Result output module: This module is the specific implementation of the verification output in the three elements of data verification. It is responsible for receiving the verification results given by the computing core module, adapting to various data storage, and completing the persistence of the verification results.

[0152] The verification engine can run on a single machine or connected to a distributed computing framework, easily obtaining computing power and horizontal expansion capabilities that match the data scale. Fig.18As shown in the figure, according to the characteristics of Spark distributed computing, this system embeds the verification engine into the Spark runtime system in modules, and puts the work of the code generation module in the Spark driver. After the executable code is generated, it is distributed to the computing core modules of each executor (Spark Executor). The data input and result output modules only need to be properly adapted to quickly form a distributed verification computing environment with the same functions as the stand-alone verification engine, but with greatly increased verification computing power.

[0153] In addition, since there are standard interfaces between the engine modules and loose coupling, the following expansions can be easily achieved with the metadata support provided by the data verification logic description system:

[0154] (1) Introducing verification and calculation support for semi-structured data: Based on the extended support for semi-structured data analysis provided by the data source analysis module in the data verification logic description system, this system can add an independent semi-structured data extractor to the data input module to implement the verification and calculation function of the computing engine for semi-structured data.

[0155] (2) Introducing support for streaming data verification calculations: Based on the data verification logic description system's support for streaming data verification descriptions, this system's data input module adds a streaming data extraction unit, and the result output module adds a streaming data write-back unit to implement the data verification calculation function of streaming data sources (such as Kafka).

[0156] The above two types of extensions benefit from the general computing characteristics of the computing core module. The interpreter inside the module interprets and executes based on the algebraic combination structure of basic operators. During the execution process, the interpreter obtains specific field data to be checked through reference. For example, in the check rule "name is not empty", name is a reference. The interpreter finally obtains external data to be checked from the data input module through reference. The reference-based data acquisition method is suitable for structured and unstructured data, and can also be adapted to streaming data. The computing core module can maintain interface stability and universality, which depends entirely on the principle of obtaining external data to be checked through reference. Following this principle, even if the data involved in the actual calculation does not exist (semi-structured data) or a runtime error occurs, the interpreter can handle it properly, and the exception is also regarded as a check result. Therefore, the computing core module does not need to be changed.

[0157] Overall, the data verification and computing system provides an independent and complete verification and computing environment, realizes a complete separation of storage and computing, and enables the verification computing power to be independently adjusted according to the workload; in addition, it shields the differences between various back-end database products, making it possible for multiple back-ends to reuse and share the same verification system, avoiding duplication of construction and saving scientific and technological resources; finally, the original aggregation algorithm only requires one IO to calculate all homologous verification rules, which greatly promotes the balanced use of system resources and significantly improves computing efficiency.

[0158] In recent years, financial regulators have put forward increasingly stringent requirements on the data quality of the regulatory data standardization (EAST) of major banks. EAST data has the characteristics of comprehensive business coverage, large amount of detailed data, and high timeliness of reporting. In order to continuously improve the quality of EAST data, it is urgent to build a safe and efficient massive data quality detection system.

[0159] Industrial Bank's EAST big data inspection and analysis system is built based on the data verification logic description system in this invention. The system has more than 4,600 existing verification rules. In order to respond quickly to regulatory changes, the system provides real-time online editing, inspection and submission of verification rules, which take effect immediately without being issued, and can be recalculated immediately with the new rule logic. The average daily verification data scale is 1.3TB, and the verification time is 1-2 hours. During peak periods, 5-8TB of data are verified per day, and the calculation time is 5-9 hours. The verification time is basically linearly related to the workload.

[0160] According to preliminary tests, in the same cluster environment, using SQL to write verification rules, given the same data set and completing the same work tasks, the memory resources required for the verification process (in MB*Sec, that is, megabytes*seconds) are only 8% to 14% of the memory resources required by SQL, and the required computing resources (vcore*Sec, that is, virtual cores*seconds) are 52% to 74% of those required by SQL. The verification computing system has a significant advantage in the utilization of IO resources, and also has certain advantages in the utilization of computing resources; in addition, in the same environment, for the verification of the same data source, the number of verification rules increased from 25 to 250, and the computing time increased to 1.9 times the original. Under the same circumstances, the SQL-based verification triggered the IO bottleneck, and the time did not conform to the linear growth.

[0161] Thanks to the flexible result output module, the verification and calculation system can choose to output statistical, sampling and other results, allowing business personnel to quickly grasp the overall picture of data quality and analyze specific problems in detail; relying on the advantages of the verification system's aggregation algorithm, the system's CPU and IO resource usage are basically balanced, and the largest single data source has more than 240 verification rules, and all verification results can be obtained with only one IO; in addition, the verification system is independent of the specific back-end database, and each branch can quickly transplant and deploy it, connect with its own data back-end, and quickly build a branch-wide personalized data verification system.

[0162] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0163] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A system for describing data checking logic, characterized in that: include: Data source analysis module: Based on the three elements of data verification, it provides online real-time data source definition, legality check and related analysis functions, and extracts metadata into data source knowledge for use by subsequent modules; the three elements of data verification include data source, verification logic and verification output; Compilation module: provides online real-time verification of logic description editing, checking and analysis functions; during the compilation process, it receives the data source knowledge provided by the data source analysis module for analysis, and performs logic optimization through the optimizer to generate the final logic representation, which is then handed over to the specific system, framework or platform to complete the calculation, realizing the loose coupling of the three elements; the final logic representation is a data structure with data types and auxiliary calculation information, which contains all the information required for calculation and is adapted to the specific calculation engine, framework or platform as a general interface; The data source analysis module includes an SQL parser. The ANSI SQL edited and submitted online is parsed by the SQL parser. After passing the analysis, a legal data source definition is generated. The SQL parser is responsible for the first round of checking of the submitted SQL and performs the following four functions: Security check: prohibit any non-select statements from being compiled at the syntax level, ensuring that any SQL statement cannot change or delete existing data in the system, thus ensuring data security; Availability check: Check whether the submitted statement complies with the select clause syntax rules and serves as a compliance description of the source definition; SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define in subsequent verification rules; Data source knowledge extraction: After the above SQL normalization is completed, the data source knowledge is extracted from the statement, including reference name, reference type, and namespace information, to provide metadata information support for subsequent modules; The SQL is replaced by data checking logic description, and the compilation module includes: Lexical analysis submodule: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream processing to token stream for analysis by the syntax analysis submodule; Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it builds a concrete syntax tree based on the LL(*) algorithm according to the syntax rules of the verification language, and then generates an abstract syntax tree, and finally submits the abstract syntax tree to the semantic analysis submodule; Semantic analysis submodule: receives the data source knowledge provided by the data source analysis submodule, performs analysis, and performs type and logic checks on the name references mentioned in the verification rules; Optimization and logic generation submodule: After passing the semantic analysis submodule, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation. The optimization method is to advance all calculations that only contain literals. After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, trims the data structure, and generates the final logical representation as a general interface to adapt to specific computing systems, frameworks, or platforms.

2. The system for describing data checking logic according to claim 1, characterized in that: The SQL parser also includes a metadata extraction unit, which interacts online with a specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and availability checks more complete; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for use by subsequent modules.

3. The system for describing data checking logic according to claim 2, characterized in that: The metadata extraction unit shields the technical differences of back-end database products, unifies the expression form of relational metadata, and provides a consistent interface to the outside world on the basis of performing basic functions.

4. A method for describing data checking logic, characterized in that: include: Data source analysis step: Based on the three elements of data verification, online real-time data source definition, legality check and related analysis are performed, and metadata is processed and extracted into data source knowledge for use in subsequent steps; the three elements of data verification include data source, verification logic and verification output; Compilation step: Perform online real-time verification of logical description editing, checking and analysis; during the compilation process, the data source knowledge provided by the data source analysis step is received for analysis, and the optimizer is used to perform logic optimization to generate the final logical representation, which is then handed over to a specific system, framework or platform to complete the calculation and output the verification results, thereby achieving loose coupling of the three elements; the final logical representation is a data structure with data types and auxiliary calculation information, which contains all the information required for calculation and serves as a general interface to adapt to specific computing engines, frameworks or platforms; The data source analysis step includes a SQL parser, and the ANSI SQL edited and submitted online is parsed by the SQL parser, and after passing, a legal data source definition is generated; The SQL parser is responsible for checking the submitted SQL and mainly performs the following four functions: Security check: prohibit any non-select statements from being compiled at the syntax level, ensuring that any SQL statement cannot change or delete existing data in the system, thus ensuring data security; Availability check: Check whether the submitted statement complies with the select clause syntax rules and serves as a compliance description of the source definition; SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define in subsequent verification rules; Data source knowledge extraction: After the above SQL normalization is completed, the data source knowledge is extracted from the statement, including the reference name, reference type and namespace information, to provide metadata information support for the subsequent compilation steps; The compiling step includes: Lexical analysis sub-step: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream to token stream for analysis by the syntax analysis sub-step; Syntax analysis sub-step: After receiving the token stream provided by the lexical analysis sub-step, a concrete syntax tree is constructed based on the LL(*) algorithm according to the syntax rules described by the check logic, and then an abstract syntax tree is generated, which is finally submitted to the semantic analysis sub-step; Semantic analysis sub-step: Receive the data source knowledge provided by the data source analysis step, analyze it, and perform type and logic checks on the name references mentioned in the verification rules; Optimization and logic generation sub-step: After the semantic analysis sub-step, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation. The optimization method is to bring forward all calculations that only contain literals. After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, trims the data structure, and generates the final logical representation as a general interface to adapt to specific computing systems, frameworks, or platforms.

5. The method for describing data checking logic according to claim 4, characterized in that: The SQL parser also includes a metadata extraction unit, which interacts online with a specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and availability checks more complete; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for use in subsequent steps.

6. The method for describing data checking logic according to claim 5, characterized in that: The metadata extraction unit shields the technical differences of back-end database products, unifies the expression form of relational metadata, and provides a consistent interface to the outside world on the basis of performing basic functions.

Images