Data verification logic description system and data verification calculation system

Abstract

The present invention provides a data verification logic description system and a data verification calculation system, including: a data source analysis module: providing functions such as online real-time data source definition, legality check and related analysis, and processing and extracting metadata into data source knowledge; a compilation module: receiving the data source knowledge provided by the data source analysis module, performing joint analysis and logic optimization work, and generating a final logic representation; a data input module: extracting data to be verified; a code generation module: generating executable code by using the above-mentioned final logic representation and data source knowledge; a calculation core module: completing verification calculation according to the generated code and verification data; a result output module: receiving the verification result, completing result persistence, and finally realizing loose coupling of the three elements. The present invention can simplify the development, testing and deployment processes, improve the maintainability of the verification logic; can avoid duplicate construction, save scientific and technological resources, and greatly promote the balanced utilization of system resources and improve the calculation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data verification logic description system and a data verification calculation system. Background Art

[0002] Financial supervision has put forward increasingly strict requirements on the data quality of the standardized regulatory data (abbreviated as EAST) of major banks. EAST data has the characteristics of comprehensive business coverage, large amount of detailed data, strict data quality requirements, high reporting timeliness, etc. In order to continuously improve the quality of EAST data, it is urgent to build a secure and efficient massive data quality detection system.

[0003] In large-scale data quality detection and monitoring, a logic description system is needed to accurately describe the inspection and verification logic of data, abbreviated as verification logic or verification rules; in addition, a calculation system based on the above verification logic is needed to complete the specific calculations regarding this logic. The common solutions in the industry use SQL and secondary development based on a certain database product to describe the verification logic of data, and rely on the deployment of the database product to provide verification computing power. In the field of data verification, this mode has many drawbacks, which are specifically analyzed as follows:

[0004] 1. Using SQL to describe the verification logic makes the three elements of verification strongly coupled, resulting in problems of low efficiency in development, testing, deployment, and maintenance: Data verification calculation depends on three elements: data source, verification logic, and verification output. SQL mixes these three elements together. For the same data source and verification logic, just because the output results are different (statistics, sampling, etc.), multiple similar SQL statements need to be written, causing a large amount of redundancy; in addition, SQL mixes the information of the three elements, and its readability is not high. When the verification logic is complex (such as Figure 2 ), the boundaries of the three elements will become very blurred, and the entire SQL statement is difficult to understand and maintain. Using the data verification logic description system of the present invention, the three elements can be completely decoupled. A data source only needs to be defined once. Based on the same data source, any number of verification rule definitions can be given. Before executing the verification logic, there is no need to consider any issues related to the verification output, and the development, testing, deployment, and maintenance links are all streamlined to varying degrees.

[0005] 2. Data security issues brought by using SQL: The lack of a custom SQL compiler to restrict the writing of the SQL language poses a great challenge to data security. In the present invention, constraints are imposed on the data source definition based on a self-developed SQL parser, and a verification logic description compiler is used to complete various checks such as data types and references, significantly improving data and calculation security.

[0006] 3. Duplicate development and logical consistency issues brought about by using SQL across multiple database backends: Enterprise-level data verification requires a large number of atomic calculation logics ( Figure 3 ), but the expressive power of the SQL language is limited. Secondary development must be carried out with the help of specific database products, which will inevitably lead to a strong binding between the implementation of atomic logics and database products. When the data in multiple databases within an organization needs to be verified, the same atomic logics must be developed repeatedly for these products, resulting in a waste of development resources. Moreover, due to the calculation consistency issues caused by different database products, a large amount of testing resources need to be invested to solve them. By using the data verification logic description system in the present invention, all atomic calculations are supported by the system's standard library, without relying on specific database products, without the need for repeated implementation, and there are no multi-terminal logical consistency issues. The verification logic can be written once and used in multiple places.

[0007] 4. The tight coupling problem between system storage and computing power resources in the memory-computation hybrid architecture: When the requirements for data storage capacity and verification computing power are inconsistent, the traditional memory-computation hybrid architecture based on the same database deployment will face a dilemma: horizontally expanding to meet computing power, but wasting storage, or maintaining appropriate storage, but the computing power cannot be satisfied in the long term. Standalone databases face an even more difficult situation, as a single server simply cannot meet the growing demand for verification computing power from data. The verification engine creates an independently deployed computing environment, achieving memory-computation separation. Even standalone databases can expand computing power according to the workload to meet business requirements.

[0008] 5. The problem that it is impossible to reuse and share the verification system due to heterogeneous database products: When complex logics need to be implemented for verification calculations, secondary development must rely on data storage products. For the same set of logics, each product needs to develop its own implementation. If the calculation results are required to be consistent, additional logical consistency tests must be carried out, wasting a large amount of development and testing resources. The verification engine, in conjunction with the verification logic description, shields the differences in backend database products and provides a completely consistent compilation and runtime environment. The same verification environment can be connected to multiple completely heterogeneous data storage products simultaneously, maximizing the reuse and sharing of the verification system and saving various resources.

[0009] 6. Difficulty that general database products cannot be fully optimized by leveraging knowledge in the verification field: One of the characteristics of the data verification field is that the same data source often contains a large number of verification rules. If each rule is described using SQL, during calculation, each SQL statement is an independent calculation unit, and the database product lacks the necessary information for global overall optimization. In the case of a large volume of data, the cache is almost ineffective, resulting in a huge waste of IO resources. As the number of verification rules increases, the system IO will quickly reach its bottleneck, affecting the overall throughput. The unique aggregation algorithm of the verification engine can calculate all rules from the same source with only one IO operation, greatly promoting the balanced utilization of system resources. Summary of the Invention

[0010] In view of the deficiencies in the prior art, the present invention provides a data verification logic description system and a data verification calculation system.

[0011] According to a data verification logic description system and a data verification calculation system provided by the present invention, the solution is as follows:

[0012] In the first aspect, a data verification logic description system is provided, and the system includes:

[0013] Data source analysis module: Based on the conceptual division of the three elements of data verification, it provides functions for online real-time data source definition, legality check, and related analysis, and processes and extracts metadata into data source knowledge for use by subsequent modules;

[0014] Compilation module: It provides functions for online real-time verification logic description editing, checking, and analysis. During compilation, it receives the data source knowledge provided by the data source analysis module to conduct joint analysis and perform logic optimization work, generating the final logic representation, and then handing it over to a specific data verification calculation system to complete the calculation and output the verification result, realizing the loose coupling of the three elements.

[0015] Preferably, the data source analysis module includes an SQL parser. The ANSI SQL submitted for online editing is parsed by the SQL parser, and after passing, a legal data source definition is generated;

[0016] The SQL parser is responsible for checking the submitted SQL and performing the following four types of functions:

[0017] Security check: At the syntax level, it prohibits any non-select statement from being compiled, ensuring that no SQL statement can modify or delete the existing data in the system, and guaranteeing data security;

[0018] Usability check: It checks whether the submitted statement conforms to the syntax rules of the select clause and is a compliant description as the source definition;

[0019] SQL normalization: Adjust the submitted statement into a normalized SQL with the same semantics and easy for subsequent verification rule definition;

[0020] Data source knowledge extraction: After the above SQL normalization is completed, extract data source knowledge from the statement, including reference names, reference types, and namespace information, to provide metadata information support for the subsequent compilation module.

[0021] Preferably, the SQL parser further includes a metadata extraction unit, which interacts online with a specific backend database product to obtain all metadata related to the SQL statement, supports semantic analysis, and makes security checks and usability checks more perfect; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for use by subsequent modules and specific data verification calculation systems.

[0022] Preferably, on the basis of fulfilling its basic functions, the metadata extraction unit shields the technical differences of the backend database products, unifies the expression forms of relational metadata, and provides a consistent interface externally.

[0023] Preferably, the compilation module includes:

[0024] Lexical analysis sub-module: Starting from the lexical level of the verification logic description, perform word segmentation on the verification rule definition, and convert the character stream into a token stream for the syntax analysis sub-module to analyze;

[0025] Syntax analysis sub-module: After receiving the token stream provided by the lexical analysis sub-module, construct a specific syntax tree based on the syntax rules of the verification logic description using the LL(*) algorithm, then generate an abstract syntax tree, and finally submit the abstract syntax tree to the semantic analysis sub-module;

[0026] Semantic analysis sub-module: Receive the data source knowledge provided by the data source analysis module, perform joint analysis, and perform type and logic checks on the name references mentioned in the verification rules;

[0027] Optimization and logic generation sub-module: After passing through the semantic analysis sub-module, perform logic optimization work. In each round of optimization, the optimizer accepts an intermediate representation, and after calculation, obtains a logically equivalent but more optimized intermediate representation;

[0028] After the optimizer completes multiple rounds of optimization, generate an optimal intermediate representation, remove the intermediate process results, trim the data structure, and generate the final logical representation, which is adapted to the specific data verification calculation system as a general interface.

[0029] Second, a data verification calculation system is provided, and the system includes:

[0030] Data input module: Extracts verified data according to the relevant information of the data source and submits the verified data to the calculation core module;

[0031] Among them, the compilation support tool chain in the data verification logic description system ensures the security of the data source definition, and the SQL parser comprehensively checks the data source definition;

[0032] Code generation module: Utilizes the final logical representation and data source knowledge output by the data verification logic description system to convert the verification rules related to the verified data into executable code and submit it to the calculation core module;

[0033] Calculation core module: Receives the verified data and the executable code after compiling the verification rules, performs calculations and obtains results;

[0034] Result output module: Receives the verification results given by the calculation core module, adapts to multiple data storages, and completes the persistence of the results.

[0035] Preferably, the data input module uses an independent data structure to describe semi-structured data source information to achieve the extraction of semi-structured verified data.

[0036] Preferably, the data input module adds a streaming data extraction unit, and the result output module adds a stream data write-back unit to achieve streaming data source adaptation.

[0037] Preferably, the code generation module includes: Utilizes the final logical representation and data source knowledge output by the data verification logic description system to generate executable code related to specific platforms and frameworks;

[0038] Places the code generation module in the Spark driver, and after the code is generated, it is distributed to the calculation core modules of each executor.

[0039] Preferably, the core component of the calculation core module is an interpreter. The interpreter has an independent thread pool, disassembles the concurrent parts according to the algebraic structure of the operator combination, and hands them over to different threads for execution respectively;

[0040] The interpreter is built-in with a set of units for collecting execution information, which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. In the data verification logic description system of the present invention, online compilation is adopted to achieve online real-time editing and feedback of data source definition and verification rule definition. In the semantic analysis stage, the data source analysis module is connected to the backend database product to obtain relevant metadata, improving computing security; in the system, the verification logic description design adopts strong type checking during compilation to enhance overall security;

[0043] 2. In the data verification logic description system of the present invention, Chinese programming is appropriately introduced in the verification logic description, improving usability and expanding the user group;

[0044] 3. The final logical representation generated after compiling the verification logic description in the data verification logic description system of the present invention is a data structure independent of specific computing platforms and frameworks. It preserves complete and typed verification logic description information, can be adapted to multiple computing platforms and frameworks, and provides support for generating verification execution codes for specific platforms and frameworks;

[0045] 4. Based on the fact that a large number of verification rules in the data verification computing system of the present invention depend on the same data source, the interpreter of the computing core module implements a unique aggregation algorithm, enabling any multiple verification rules of the same data source to participate in the calculation together, greatly alleviating the IO pressure in the case of high-concurrency execution of a large number of verification rules;

[0046] 5. The data verification computing system of the present invention provides an independent and complete verification computing environment, achieving complete separation of storage and computing, enabling the verification computing power to be adjusted independently according to the workload;

[0047] 6. The data verification computing system of the present invention shields the differences of multiple backend database products, making it possible to reuse and share the same set of verification systems for multiple backends, avoiding duplicate construction and saving scientific and technological resources; the unique aggregation algorithm enables all homologous verification rules to be calculated with only one IO, greatly promoting the balanced utilization of system resources and improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects, and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0049] Figure 1 It is a system module diagram of the present invention;

[0050] Figure 2 It is a schematic diagram of Chinese support for verification rule definition;

[0051] Figure 3 It is a schematic diagram of using standard library functions in the verification logic description;

[0052] Figure 4 It is a system module diagram of the verification logic description;

[0053] Figure 5 Schematic diagram for defining the process of the data source;

[0054] Figure 6 Schematic diagram for the security check feedback of the SQL parser;

[0055] Figure 7 Schematic diagram for the security check feedback of the SQL parser;

[0056] Figure 8 Schematic diagram for the availability check feedback of the SQL parser;

[0057] Figure 9 Schematic diagram for the SQL normalization of the SQL parser;

[0058] Figure 10 Schematic diagram for the structure and working process of the SQL parser;

[0059] Figure 11 Schematic diagram for the compilation process of the verification logic description;

[0060] Figure 12 Example of compilation error prompt;

[0061] Figure 13 Schematic diagram for the mismatch of branch expression types;

[0062] Figure 14 Schematic diagram for the mismatch of function parameter types;

[0063] Figure 15 Schematic diagram for the reference error of the verification logic description;

[0064] Figure 16 Schematic diagram for the verification engine module and structure;

[0065] Figure 17 Schematic diagram for the code generation structure and process;

[0066] Figure 18 Schematic diagram for the verification engine embedded with the Spark framework. Specific implementation manner

[0067] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all belong to the protection scope of the present invention.

[0068] The present invention provides a data verification logic description system, including:

[0069] Data source analysis module: Based on the conceptual division of the three elements of data verification, it provides online real-time data source definition, legality check and related analysis functions, extracts and processes metadata into data source knowledge for subsequent modules to use.

[0070] Compilation module: It provides online real-time verification logic description editing, checking and analysis functions; during the compilation process, it receives the data source knowledge provided by the data source analysis module to conduct joint analysis and perform logic optimization work, and then generates the final logic representation, which is then handed over to the specific data verification calculation system to complete the verification calculation, realizing the loose coupling of the three elements.

[0071] The data source analysis module includes an SQL parser. The ANSI SQL submitted through online editing is parsed by the SQL parser. After passing, a legal data source definition is generated.

[0072] The SQL parser is responsible for checking the submitted SQL and performing the following four types of functions:

[0073] Security check: At the syntax level, it prohibits the compilation of any non-select statements, ensuring that any SQL statement cannot modify or delete the existing data in the system and guaranteeing data security.

[0074] Usability check: Check whether the submitted statement conforms to the syntax rules of the select clause and serves as a compliant description of the source definition.

[0075] SQL normalization: Adjust the submitted statement into a normalized SQL with the same semantics and easy for subsequent verification rule definition.

[0076] Data source knowledge extraction: After the above SQL normalization is completed, extract data source knowledge from the statement, including reference names, reference types, and namespace information, providing metadata information support for subsequent modules.

[0077] The SQL parser also includes a metadata extraction unit, which interacts online with specific backend database products to obtain all metadata related to the SQL statement, supports semantic analysis, and makes the security check and usability check more perfect; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for subsequent modules to use.

[0078] On the basis of performing basic functions, the metadata extraction unit shields the technical differences of backend database products, unifies the expression form of relational metadata, and provides a consistent interface externally.

[0079] Using the data verification logic description to replace SQL, the compilation module includes:

[0080] Lexical analysis submodule: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from the character stream into a token stream for analysis by the syntax analysis submodule;

[0081] Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it constructs a concrete syntax tree based on the LL(*) algorithm according to the grammatical rules described by the verification logic, and then generates an abstract syntax tree. Finally, the abstract syntax tree is submitted to the semantic analysis submodule;

[0082] Semantic analysis submodule: Receives data source knowledge provided by the data source analysis module, performs joint analysis, and performs type and logic checks on name references mentioned in the verification rules.

[0083] Optimization and logic generation submodule: After passing the semantic analysis submodule, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and calculates a logically equivalent but more optimized intermediate representation.

[0084] After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, and trims the data structure to generate the final logical representation, which serves as a general interface to adapt to the specific data verification and calculation system.

[0085] The present invention also provides a data verification calculation system, comprising:

[0086] Data input module: extracts verification data based on relevant information of the data source and submits the verification data to the calculation core module.

[0087] The compilation supporting tool chain in the data verification logic description system is used to ensure the security of the data source definition, and the SQL parser performs a comprehensive check on the data source definition.

[0088] The data source knowledge generated in the data verification logic description system is used to support data extraction in this module.

[0089] Code generation module: uses the final logical representation and data source knowledge output from the data verification logic description system to convert the verification rules related to the verification data into executable code and submit it to the computing core module.

[0090] Calculation core module: receives the verification data and the executable code compiled from the verification rules, performs the calculation and obtains the result.

[0091] Result output module: receives the verification results given by the computing core module, adapts to various data storages, and completes the persistence of the results.

[0092] The data input module uses an independent data structure to describe semi-structured data source information and realizes the extraction of semi-structured verification data.

[0093] The data input module adds a streaming data extraction unit, and the result output module adds a stream data write-back unit to achieve streaming data source adaptation.

[0094] The code generation module includes: generating executable code by using the final logical representation and data source knowledge output in the data verification logic description system.

[0095] Place the code generation module in the Spark driver, and after the code is generated, distribute it to the computing core modules of each executor.

[0096] The core component of the computing core module is an interpreter. The interpreter has an independent thread pool, and disassembles the concurrent parts according to the algebraic structure of the operator combination and executes them by different threads respectively.

[0097] The interpreter builds in a set of units for collecting execution information, which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

[0098] Next, a more specific description of the present invention will be given.

[0099] The embodiment of the present invention includes a verification logic description system and a data verification calculation system, which are respectively used to solve the two most important problems in the field of data verification - logical description and calculation. The following is a detailed description of the two systems. Figure 1 As shown, they are respectively used to solve the two most important problems in the field of data verification - logical description and calculation. The following is a detailed description of the two systems.

[0100] The first part: The present invention provides a data verification logic description system.

[0101] The description of data verification logic usually includes the following three types of information:

[0102] (1) Data source: That is, the source and data structure information of the data to be verified. For example, a data source definition can be given through SQL: select... from A where date = '20211220'. This statement indicates that the data source comes from table A and the date is '20211220'.

[0103] (2) Verification Logic: Based on the defined data source, several verification logics can be defined, also known as verification rules. Each rule can be regarded as the smallest unit of verification calculation. Usually, a data source will define dozens to hundreds of verification rules to complete comprehensive and detailed data quality verification. Taking a simple verification logic as an example: After adding the verification rule to the data source defined in (1), we can get select... from A where date = ‘20211220’ and col is not null. It can be seen that the core logic of the verification rule is mixed with the data source elements, and it is actually just a part of the where clause, that is, “col is not null”.

[0104] (3) Verification Output: Describes how the verification calculation results are saved in terms of structure and purpose. Taking the SQL statement in (2) as an example, adding a verification output definition gives: select count(*) from A where date = ‘20211220’ and col is not null, that is, the output is to count the data that meets the conditions.

[0105] The data source, verification logic, and verification output are called the three elements of data verification. As can be seen from the above example, traditional SQL mixes these three elements together, forming a tight coupling. The present invention takes this as a breakthrough point. Through the design and implementation of a new logical description, the three elements are completely decoupled, thus establishing advantages in many aspects and achieving remarkable results.

[0106] In terms of the design of the logical description, it mainly completes the construction and selection of syntax, semantics, features, etc., enabling users to safely, accurately, flexibly, and conveniently give the definitions of the data source and verification logic.

[0107] In terms of the implementation of the logical description, on the basis of the design work, from an engineering perspective, a supporting compiler and tool chain are implemented to support common usage scenarios, such as online real-time compilation, error feedback, syntax highlighting, logic checking, and representation generation. It mainly includes a data source analysis module and a compilation module.

[0108] The following will elaborate on the design and implementation:

[0109] In the design part, the logical description is dedicated to data verification. On the basis of meeting the basic logical construction ability, it tries to lower the learning and usage thresholds. Therefore, basic Chinese support and some syntactic sugars in Chinese expression forms are added at the syntax level, such as Figure 2As shown in the rule definition section, branch keywords can be written in the form of "if...then...else..." or "if..then..else.." Comparison symbols can be written in Chinese (such as "equals") or mathematical symbols (such as ">="). In addition, some string calculations can also use Chinese expressions, and the syntax conforms to Chinese usage habits.

[0110] Logic includes basic syntax such as basic branch judgment, function call, logical operation, numerical operation, etc. Data has strict type division, including Boolean type, integer type (byte, short, int, long), floating point type (float, double), character, string, date, timestamp, regular expression, etc. For example, Figure 2 The rule definition in the example uses the "date" keyword to construct a date literal '2020-01-01', and "true" and "false" are Boolean literals.

[0111] To improve computational determinism, inspired by functional programming, all grammatical structures are expressions, without statements, and the evaluation process does not contain any side effects. Taking Java as a comparison, "if...else..." is a statement with side effects, while in the logic description design of the present invention, "if...then...else..." is an expression without side effects, and its evaluation result is the calculation result of one of the branches. This requires that the calculation result type of each branch is the same. If the types are different, an error will be prompted. Figure 2 The final evaluation result is equal to the evaluation result of the branch judgment expression.

[0112] To improve the security of calculations, the logical description will undergo strict strong type checking. If it does not meet the requirements of the calculation type, an error will be quickly fed back during compilation.

[0113] Finally, all functions used in the logic description are from the standard library and can be continuously expanded, which facilitates the efficient writing of verification logic and avoids lengthy logic or repeated definitions. The function call example is as follows Figure 3 shown.

[0114] In general, the logic description design follows the principles of safety, flexible expression, simple writing, and easy understanding.

[0115] Implementation part, such as Figure 4 As shown, the modules are described in detail as follows:

[0116] Data Source Analysis Module: Based on the above division of the three verification elements, this module is responsible for defining data sources, providing functions such as online data source definition, real-time legality check, and related analysis. To maintain generality, flexibility, and security, the syntax of data source definition is based on a subset of the ANSI SQL language and only includes the select query clause. This module contains an SQL parser to support core functions, and the workflow is as Figure 5 shown. The SQL edited and submitted online is parsed by the SQL parser. After passing the check, a legal data source definition is generated. The SQL parser implements a complete front-end of the SQL compiler, which is responsible for checking the submitted SQL and mainly performs the following four types of functions:

[0117] (1) Security check: As Figure 6 、 7 shown, at the syntax level, any non-select statement is prohibited from being compiled, ensuring that the SQL statement cannot change or delete the existing data in the system and guaranteeing data security;

[0118] (2) Usability check: As Figure 8 shown, check whether the submitted statement conforms to the syntax rules of the select clause and can be used as a legal description of the data source definition;

[0119] (3) SQL normalization: Adjust the submitted statement to a normalized SQL with the same semantics and easy for subsequent verification rule definition. For example, as Figure 9 shown, expand the asterisk (*) in the select statement completely, which is convenient for name reference and related checks when the compilation module defines verification rules later;

[0120] (4) Data source knowledge extraction: After the above step (3) is completed, data source knowledge can be extracted from the statement, including reference names, reference types, namespaces, etc. For example, in the data source definition "select a, b, c from table", after being extracted by the SQL parser, the data source knowledge will include references such as a, b, c, table and their types. These information not only provide metadata information support for the subsequent compilation module, but also can provide runtime information support during the verification calculation process. It should be noted that the data source knowledge retains all metadata based on the principle of reference (reference). The reference-based notation has many advantages, enabling this module to support semantic access to metadata related to structured and semi-structured data at the same time, and can also support semantic access to streaming data. As a general-purpose data structure, the data source knowledge provides a unified standard data interface.

[0121] The internal structure and workflow of the SQL parser are as Figure 10As shown, after the SQL statement undergoes basic lexical and syntactic analysis, in the semantic analysis phase, the metadata extraction unit will interact online with the specific database product at the backend to obtain all the metadata related to the SQL statement, thereby supporting more rigorous semantic analysis, further improving security and usability checks. At the same time, the SQL parser processes and extracts this metadata into data source knowledge for use by the subsequent compilation module.

[0122] It should be noted that due to functional limitations, the SQL parser does not contain any backend functions for SQL compilation and cannot truly "execute" SQL. In addition, as a connection point for multiple database products, on the basis of fulfilling its basic functions, the metadata extraction unit shields the technical differences of the backend database products, unifies the expression form of relational metadata, and provides a consistent interface externally.

[0123] This module completes the definition and analysis work related to the data source, provides metadata support for the subsequent compilation module, and enables the verification rules to obtain clear computational semantics in a specific context. The data source knowledge generated by the SQL parser is a bridge connecting the data source and the verification logic, and also plays a role in separating the two.

[0124] Compilation module: Focuses on the compilation analysis, optimization, and generation of the final representation of the verification logic description. The compilation process is as Figure 11 shown. The verification rules are sequentially analyzed and calculated through sub-modules such as lexical, syntactic, semantic, optimization, and logic generation, and finally a logic intermediate representation independent of the specific computing platform and database product is generated. If an error occurs in any of these links, the compilation will be interrupted, and detailed compilation error information will be fed back in advance to facilitate the user to quickly correct the logic. For example Figure 12 as shown, the right expression is miswritten as the string 5, and the types on both sides of the inequality sign are inconsistent, resulting in compilation failure and an error prompt.

[0125] The following will be described separately according to the sub-modules:

[0126] Lexical analysis sub-module: Starting from the syntactic level of the verification logic, the verification rules are tokenized, and the character stream is processed and converted into a token stream for analysis by the syntactic analysis sub-module.

[0127] Syntactic analysis sub-module: After receiving the token stream provided by the lexical analysis sub-module, according to the syntactic rules of the verification logic, a concrete syntax tree (CST) is constructed based on the LL(*) algorithm, and then an abstract syntax tree (AST) is generated. Finally, the abstract syntax tree is submitted to the semantic analysis sub-module.

[0128] Semantic analysis sub-module: Receives the data source knowledge provided by the data source analysis module, and checks and analyzes the semantics of the verification logic. To expose more logical errors in advance, strong type checking can ensure type-based calculation security (such as Figure 12 , 13 , 14); The check must also ensure that the evaluation result type of the verification logic is always boolean to reflect a clear judgment on data quality; In addition, some rule-based type conversions can be allowed during the analysis process to improve the usability of the language on the premise of security.

[0129] Such as Figure 15 shown, the check and analysis also include name references, logical consistency constraints, etc. mentioned in the verification rules to comprehensively ensure that the verification rules are completely described and computable. After all the analysis is completed, an intermediate representation with type information (Typed IR), simply referred to as the intermediate representation, will be generated and submitted to the subsequent optimization and logic generation sub-module.

[0130] Optimization and logic generation sub-module: Once passed through the above semantic analysis sub-module, it means that the current verification rule has passed all the checks. The main purpose of this module is to perform logic optimization work. As Figure 11 shown, in each round of optimization, the optimizer accepts an intermediate representation and calculates to obtain a logically equivalent but more optimized intermediate representation. For example, an important optimization method is to advance all calculations that only contain literals. For example, 1 + 1 can be directly optimized to 2 without having to perform a large number of inefficient repeated calculations during the actual execution stage of the verification rule.

[0131] After the optimizer completes multiple rounds of optimization, an optimal intermediate representation can be generated. Then, after removing some process results and trimming the data structure, the final logic representation is generated. This logic representation is a data structure with data types and calculation auxiliary information, containing all the information required for calculation, independent of the specific database product, and can be adapted as a general interface to specific calculation engines, frameworks, or platforms. See the following second part - Data Verification Calculation System for details.

[0132] So far, the data source definition and the verification rule definition have been completely decoupled. Since the verification output is closely related to the data persistence layer, it can be completed by specific calculation engines, frameworks, or platforms. See the result output module in the following second part - Data Verification Calculation System for details.

[0133] Through appropriate transformation, the following expansions can be added to this system:

[0134] (1) Add support for analyzing semi-structured data: In the above data source analysis module, structured data is described by ANSI SQL, and the SQL parser is responsible for parsing and finally extracting data source knowledge. Similarly, this module adds JSON to describe the structure and data type of semi-structured data, and generates data source knowledge through a JSON parser. Since the data source knowledge saves metadata based on the principle of preserving references, the interface layer remains stable, the entire compilation module can be seamlessly connected, and the entire system can implement the function of analyzing verification rules based on semi-structured data sources, and also provide information support for the corresponding expansion of the subsequent data verification calculation system.

[0135] (2) Add support for analyzing streaming data: Similar to the idea in (1), on the basis of maintaining the reference principle, add a DSL (such as YAML) for describing the structure and type of streaming data, write a parser, and generate standard data source knowledge, then it can be docked with the compilation module to enable the system to analyze verification rules based on streaming data sources.

[0136] (3) Continuously optimize the final logical representation: In the above optimization and logic generation sub-module, the optimizer performs progressive optimization on the intermediate representation. As the system function expands, the optimizer can continuously add more layers of analysis and optimization, continuously improve the final logical representation, enhance the calculation expression ability, and help the verification calculation link generate higher-performance execution code.

[0137] Overall, multiple modules of this system cooperate closely to achieve complete decoupling of the three elements of data source, verification logic, and verification output. With the help of online real-time compilation and feedback, users can quickly complete the definition of data sources and verification logic, significantly simplify the development, testing, and deployment processes, appropriately introduce Chinese, realize the complementary advantages of Chinese and ordinary writing methods, improve the readability and maintainability of verification logic, lower the usage threshold, reduce costs and increase efficiency; relying on features such as a large number of strict syntax, semantic checks, and strong type checks during compilation, it significantly improves the security of data and calculations; since the data source definition uses a subset of ANSI SQL, and the verification logic description only depends on the standard data source knowledge interface provided by the data source analysis module, regardless of the specific database product, it completely solves the problem of logical consistency of multiple database backends and achieves the effect of writing once and running multiple times.

[0138] Part Two: The present invention also provides a data verification calculation system.

[0139] Based on the work of the data verification logic description system in the first part, the data verification calculation system aims to complete large-scale verification calculations. The core of this system is the verification engine, which provides a complete runtime environment for verification calculations. The schematic diagram of the basic module and structure is referred to Figure 16As shown in the figure, the computing core module is responsible for actual calculations. The required target verification data and verification calculation logic are provided by the data input module and the code generation module respectively. After the calculation is completed, the result output module persists the verification result.

[0140] The following is a detailed description by module:

[0141] Data input module: Extracts the target verification data according to the relevant information of the data source and submits it to the computing core module. Among them, the relevant information of the data source comes from the data source knowledge and normalized SQL provided by the data verification logic description system ( Figure 9 , 10 ). Using this information, this module can fully perceive the structure and source of the data to be verified and continuously provide data to the computing core module stably and securely.

[0142] Code generation module: Responsible for batch converting the verification rules related to the target verification data into executable code and submitting it to the computing core module for execution. The internal structure and working process of the module are as Figure 17 shown: The final logical representation and data source knowledge are both compiled and generated by the data verification logic description system. After the translator receives this information, it traverses the final logical representation and combines the data source knowledge to generate efficient executable code. Inspired by the Lambda calculus, the executable code consists of several orthogonal basic operators. After the verification logic is compiled, it is represented by the algebraic combination of basic operators. It can be considered that this set of operators is the "assembly language" of the entire verification engine.

[0143] Computing core module: Receives the verification data and the executable code after compiling the verification rules, executes the calculation and obtains the result. The core component of this module is an interpreter for the above basic operators. In order to improve concurrency and strengthen runtime control, the interpreter has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and executed by different threads respectively. In addition, the interpreter has a built-in execution information collection unit (Metrics Unit) responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization. For example, after collecting the execution information of the branch logic, the hit rate of each branch logic is statistically analyzed regularly. When the hit rate of a certain branch reaches the threshold, the optimization logic can be triggered to make this branch execute concurrently with the judgment logic, further improving the calculation concurrency.

[0144] In a large-scale data verification and calculation scenario, usually, the verification of a data source involves the calculation of dozens or even hundreds or thousands of verification rules. Obviously, by making full use of the characteristics of modern multi-core CPUs and combining the calculation of all verification rules that depend on the same data source, the purpose of saving IO resources and improving calculation efficiency can be achieved. Based on this theory, this module has designed and implemented an aggregation algorithm, which can complete the calculation of any number of verification rules with the same source with only one IO of the data source, and maintain the independence of the calculation results of each rule, achieving "combination without confusion".

[0145] It should be noted that although the verification calculation is interpreted and executed at the verification engine level, at the entire JVM level, all constructs including the interpreter will ultimately be compiled into JVM bytecode for execution, which is no different from ordinary programs.

[0146] Result output module: This module is the specific implementation of the verification output among the three elements of data verification, responsible for receiving the verification results given by the calculation core module, adapting to multiple data storages, and completing the persistence of the verification results.

[0147] The verification engine can run stand-alone or be connected to a distributed computing framework to easily obtain computing power and horizontal expansion ability that match the data scale. As Figure 18 shown, according to the characteristics of Spark distributed computing, this system embeds the verification engine into the Spark runtime system in modules, places the work of the code generation module in the Spark Driver, and distributes it to the calculation core modules of each executor (Spark Executor) after the executable code is generated. The data input and result output modules only need to make appropriate adaptations to quickly form a distributed verification calculation environment with the same function as the stand-alone verification engine but with a significantly increased verification computing power.

[0148] In addition, since there are standard interfaces between the engine modules and the connection is loosely coupled, with the support of metadata provided by the data verification logic description system, the following expansions can be easily achieved:

[0149] (1) Introduce support for the verification and calculation of semi-structured data: Based on the extended support for semi-structured data analysis by the data source analysis module in the data verification logic description system, this system can add an independent semi-structured data extractor in the data input module to implement the verification and calculation function of the calculation engine for semi-structured data.

[0150] (2) Introduce support for the verification and calculation of streaming data: Based on the support for streaming data verification description in the data verification logic description system, this system adds a streaming data extraction unit to the data input module and a stream data write-back unit to the result output module to implement the verification and calculation function for streaming data sources (such as Kafka).

[0151] The above two types of expansion benefit from the general computing characteristics of the computing core module. The interpreter inside this module interprets and executes based on the algebraic combination structure of basic operators. During the execution process, the interpreter obtains specific data of the fields to be verified through reference. For example, in the verification rule 'name is not empty', name is a reference, and the interpreter finally obtains external data to be verified from the data input module through the reference. The data acquisition method based on reference is applicable to structured, unstructured data, and can also be adapted to streaming data. The computing core module can maintain the stability and generality of the interface, completely relying on the principle of obtaining external data to be verified through reference. Following this principle, even if the data involved in the actual calculation does not exist (semi-structured data) or a runtime error occurs, the interpreter can handle it properly, and the exception is also regarded as a verification result. Therefore, the computing core module does not need to be modified.

[0152] Overall, the data verification and calculation system provides an independent and complete verification and calculation environment, achieving a complete separation of storage and calculation, enabling the verification computing power to be adjusted independently according to the workload; in addition, it shields the differences of multiple backend database products, making it possible for multiple backends to reuse and share the same set of verification systems, avoiding duplicate construction and saving scientific and technological resources; finally, it has an original aggregation algorithm, which can calculate all homologous verification rules with only one I / O, greatly promoting the balanced utilization of system resources and significantly improving the calculation efficiency.

[0153] In recent years, financial supervision has put forward increasingly strict requirements on the data quality of the standardized supervision data (abbreviation: EAST) of major banks. EAST data has the characteristics of comprehensive business coverage, large volume of detailed data, and high reporting timeliness. In order to continuously improve the quality of EAST data, it is urgent to build a safe and efficient mass data quality detection system.

[0154] The EAST big data inspection and analysis system of Industrial Bank is built based on the data verification logic description system and the data verification and calculation system in this invention. There are more than 4,600 existing verification rules in the system. In order to respond quickly to regulatory changes, the system provides functions such as real-time online editing, inspection, and submission of verification rules, which take effect immediately without distribution, and can recalculate with the new rule logic immediately. The average daily scale of verified data is 1.3TB, and the verification duration is 1 - 2 hours. During peak periods, 5 - 8TB of data is verified in a single day, and the calculation duration is 5 - 9 hours. The verification duration is basically linearly related to the workload.

[0155] According to preliminary tests, in the same cluster environment, when writing verification rules using SQL, given the same data set and completing the same work tasks, the memory resources required for the verification process (in MB*Sec, i.e., megabyte * second) are only 8% to 14% of the memory resources required by SQL, and the computing resources required (vcore*Sec, i.e., virtual core number * second) are 52% to 74% of those required by SQL. The verification computing system has a significant advantage in the utilization rate of IO resources and also has a certain advantage in the utilization rate of computing resources; in addition, in the same environment, for the verification of the same data source, when the number of verification rules increases from 25 to 250, the computing time increases to 1.9 times the original. In the same situation, the verification based on SQL triggers an IO bottleneck and the time does not increase linearly.

[0156] Benefiting from the flexible result output module, the verification computing system can select to output results such as statistics and sampling, which is convenient for business personnel to quickly grasp the overall picture of data quality and analyze specific problems in detail; relying on the advantages of the aggregation algorithm of the verification system, the use of the system's CPU and IO resources is basically balanced. For the verification rules of the largest single data source exceeding 240, all verification results can be obtained with only one IO; in addition, the verification system is independent of the specific backend database, and each branch can quickly transplant and deploy it, dock with its own data backend, and quickly build a personalized data verification system for the local branch.

[0157] Those skilled in the art know that in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same function. Therefore, the system and its various devices, modules, and units provided by the present invention can be regarded as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structure within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or the structure within the hardware component.

[0158] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific implementation manners, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily with each other.

Claims

1. A data verification logic description system, characterized in that, Including: Data source analysis module: Based on the conceptual division of the three elements of data verification, it provides functions such as online real-time data source definition, legality check, and related analysis, and processes and extracts metadata into data source knowledge for subsequent modules to use; The three elements of the data verification include: data source, verification logic, and verification output; Compilation module: Edit, check, and analyze the verification logic description; during the compilation process, receive the data source knowledge provided by the data source analysis module to conduct joint analysis and perform logic optimization work, and then generate the final logic representation, which is then handed over to the specific data verification calculation system to complete the verification calculation, realizing the loose coupling of the three elements; The data source analysis module includes an SQL parser. The ANSI SQL submitted online for editing is parsed by the SQL parser, and after passing, a legal data source definition is generated; The SQL parser is responsible for checking the submitted SQL and performing the following four types of functions: Security check: At the syntax level, prohibit any non-select statement from being compiled, ensuring that any SQL statement cannot modify or delete the existing data in the system and guaranteeing data security; Usability check: Check whether the submitted statement conforms to the syntax rules of the select clause and is a compliant description as the source definition; SQL normalization: Adjust the submitted statement to a normalized SQL with the same semantics and easy for subsequent verification rule definition; Data source knowledge extraction: After the above SQL normalization is completed, extract data source knowledge from the statement, including reference names, reference types, and namespace information, to provide metadata information support for subsequent modules; Using the data verification logic description to replace SQL, the compilation module includes: Lexical analysis sub-module: Starting from the lexical level of the verification logic description, perform word segmentation on the verification rule definition, and process and convert the character stream into a token stream for the syntax analysis sub-module to analyze; Syntax analysis sub-module: After receiving the token stream provided by the lexical analysis sub-module, construct a specific syntax tree based on the syntax rules of the verification logic description using the LL(*) algorithm, and then generate an abstract syntax tree, and finally submit the abstract syntax tree to the semantic analysis sub-module; Semantic analysis sub-module: Receive the data source knowledge provided by the data source analysis module to conduct joint analysis, and perform type and logic checks on the name references mentioned in the verification rules; Optimization and logic generation sub-module: After passing through the semantic analysis sub-module, perform logic optimization work. In each round of optimization, the optimizer accepts an intermediate representation and calculates to obtain an intermediate representation that is logically equivalent but more optimized; After the optimizer completes multiple rounds of optimization, generate an optimal intermediate representation, remove the intermediate process results, and trim the data structure to generate the final logic representation, which is adapted to the specific data verification calculation system as a general interface.

2. The data verification logic description system according to claim 1, wherein The SQL parser also includes a metadata extraction unit, which interacts online with the specific database product at the back end to obtain all metadata related to the SQL statement, supports semantic analysis, and makes the security check and usability check more perfect; at the same time, the SQL parser processes and extracts the metadata into data source knowledge for subsequent modules to use.

3. The data verification logic description system according to claim 2, wherein The metadata extraction unit can shield the technical differences of back-end database products, unify the expression form of relational metadata, and provide a consistent interface externally.

4. A data verification calculation system, based on the data verification logic description system according to any one of claims 1-3, characterized in that, It includes: Data input module: Extract the verified data according to the information related to the data source, and submit the verified data to the calculation core module; Among them, the compilation support tool chain in the data verification logic description system is used to ensure the security of the data source definition, and the SQL parser comprehensively checks the data source definition; Among them, the data source knowledge generated in the data verification logic description system is used to support the data extraction of this module; Code generation module: Use the final logical representation and data source knowledge output in the data verification logic description system to convert the verification rules related to the verified data into executable code and submit it to the calculation core module; Calculation core module: Receive the verified data and the executable code after compiling the verification rules, execute the calculation and obtain the result; Result output module: Receive the verification result given by the calculation core module, adapt to multiple data storages, and complete the persistence of the result.

5. The data verification and calculation system according to claim 4, wherein The data input module uses an independent data structure to describe the semi-structured data source information and realizes the extraction of semi-structured verified data.

6. The data verification and calculation system according to claim 4, wherein The data input module adds a streaming data extraction unit, and the result output module adds a streaming data write-back unit to realize the adaptation of streaming data sources.

7. The data verification and calculation system according to claim 4, wherein The code generation module includes: Generate executable code using the final logical representation and data source knowledge output in the data verification logic description system; Place the code generation module in the Spark driver, and after the code is generated, distribute it to the calculation core modules of each executor.

8. The data verification calculation system according to claim 4, wherein The core component of the calculation core module is an interpreter, which has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and executed by different threads respectively; The interpreter is built-in with a set of units for collecting execution information, which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

Images