Data verification calculation system and method

Abstract

The present invention provides a data verification calculation system and method, which relates to the field of data processing technology. The method includes: a data input module: extracting verification data according to relevant information of a data source, and submitting the verification data to a computing core module; a code generation module: converting verification rules related to the verification data into executable code, and submitting it to the computing core module; a computing core module: receiving the verification data and the executable code compiled from the verification rules, performing calculations and obtaining results; a result output module: receiving the verification results given by the computing core module, adapting to a variety of data storages, and completing the persistence of the results. The present invention enables the verification computing power to be independently adjusted according to the workload, making it possible for a variety of backends to reuse and share the same set of verification systems, avoiding duplication of construction, saving scientific and technological resources, and greatly promoting the balanced use of system resources and improving computing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data verification calculation system and method. Background Art

[0002] Data quality inspection and analysis technology supports the inspection and monitoring of business indicators and data quality, and analyzes data from five dimensions: completeness, validity, timeliness, accuracy, and uniqueness, helping users to promptly discover and correct data quality issues, thereby reducing losses. In today's information explosion, the scale of data is becoming increasingly large, and the existing related technical systems are facing severe challenges such as low execution efficiency.

[0003] In large-scale data quality testing and monitoring, common solutions use SQL and secondary development based on database products to describe data inspection and verification logic (referred to as verification logic or verification rules), and rely on the deployment of database products to provide verification computing power. In the field of data verification, this model has three major drawbacks, which are analyzed as follows:

[0004] 1. Tight coupling of system storage and computing resources under storage-computing hybrid architecture: When the storage capacity of verification data is inconsistent with the demand for verification computing power, the traditional storage-computing hybrid architecture deployed on the same database will face a dilemma: horizontal expansion to meet computing power, but waste storage, or maintain moderate storage, but computing power cannot be met for a long time. Stand-alone databases face an even more difficult situation, and a single server is completely unable to meet the demand for verification computing power for growing data. The verification engine creates an independently deployed computing environment and realizes the separation of storage and computing. Even a stand-alone database can expand computing power according to workload to meet business needs.

[0005] 2. The heterogeneity of database products leads to the problem of being unable to reuse and share the verification system: When the verification calculation needs to implement complex logic, it must rely on data storage products for secondary development. For the same set of logic, each product needs to develop its own implementation. If the calculation results are required to be consistent, a separate logic consistency test must be carried out, wasting a lot of development and testing resources. The verification engine shields the differences between backend database products and provides a completely consistent runtime environment. The same set of verification environment can be connected to multiple completely heterogeneous data storage products at the same time, maximizing the reuse and sharing of the verification system and saving various resources.

[0006] 3. The problem that general database products cannot fully optimize using verification domain knowledge: A major feature of the data verification field is that the same data source often contains a large number of verification rules. If each rule is described in SQL, each SQL statement is an independent calculation unit during calculation. Database products lack the necessary information for global optimization. Under the massive data scale, the cache is almost ineffective, and there is a huge waste of IO resources. As the number of verification rules increases, the system IO will soon reach a bottleneck, affecting the overall throughput. The unique aggregation algorithm of the verification engine can calculate all homologous rules with only one IO, which greatly promotes the balanced use of system resources.

[0007] The invention patent with publication number CN108920256A discloses a verification task execution method and device, including: obtaining verification information corresponding to the verification task to be executed; judging whether there is a target verification task that is the same as the verification task to be executed in the completed verification tasks based on the verification information; and determining that the target verification task exists in the completed verification tasks, then using the execution result corresponding to the target verification task as the execution result of the verification task to be executed. Summary of the invention

[0008] In view of the defects in the prior art, the present invention provides a data verification calculation system and method.

[0009] According to a data verification calculation system and method provided by the present invention, the scheme is as follows:

[0010] In a first aspect, a data verification computing system is provided, the system comprising:

[0011] Data input module: extracts verification data based on relevant information of the data source and submits the verification data to the calculation core module;

[0012] Code generation module: converts the verification rules related to the target verification data into executable code and submits it to the computing core module;

[0013] Calculation core module: receives verification data and verification rules, compiles and converts them into executable code, performs calculations and obtains results;

[0014] Result output module: receives the verification results given by the computing core module, adapts to various data storages, and completes the persistence of the results.

[0015] Preferably, the data input module uses an independent data structure to describe the semi-structured data source information to achieve the extraction of semi-structured verification data.

[0016] Preferably, the data input module adds a streaming data extraction unit, and the result output module adds a streaming data write-back unit to achieve streaming data source adaptation.

[0017] Preferably, the code generation module is placed in the Spark driver, and the code is distributed to the computing core modules of each executor after being generated.

[0018] Preferably, the core module component of the computing core module is an interpreter, which has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and respectively handed over to different threads for execution.

[0019] Preferably, the interpreter has a built-in unit for executing information collection, which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

[0020] In a second aspect, the present invention provides a data verification calculation method, the method comprising:

[0021] Data input step: extract verification data based on relevant information of the data source and submit the verification data to the calculation core step;

[0022] Code generation step: convert the verification rules related to the verification data into executable code and submit it to the calculation core step;

[0023] The core steps of calculation: receiving the verification data and the executable code compiled by the verification rules, performing the calculation and obtaining the result;

[0024] Result output step: Receive the verification results given by the core computing step, adapt to various data storage, and complete the persistence of the results.

[0025] Preferably, the data input step uses an independent data structure to describe the semi-structured data source information to achieve the extraction of semi-structured verification data.

[0026] Preferably, the data input step adds a streaming data extraction unit, and the result output step adds a streaming data write-back unit to achieve streaming data source adaptation.

[0027] Preferably, the code generation step is placed in the Spark driver, and the code is distributed to the computing core steps of each executor after being generated.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Based on the fact that a large number of check rules rely on the same data source, the interpreter design of the computing core module implements a unique aggregation algorithm, so that any number of check rules from the same data source can participate in the calculation together, which greatly alleviates the IO pressure under the high concurrent execution of a large number of check rules;

[0030] 2. The present invention provides an independent and complete verification computing environment, realizes complete storage and computing separation, and enables the verification computing power to be independently adjusted according to the workload;

[0031] 3. The present invention shields the differences between various back-end database products, making it possible for various back-ends to reuse and share the same set of verification systems, avoiding duplication of construction and saving scientific and technological resources;

[0032] 4. The present invention has created an original aggregation algorithm that can calculate all homology check rules with only one IO, which greatly promotes the balanced utilization of system resources and improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 Describe the collaboration relationship between the verification logic and the verification calculation;

[0035] Figure 2 Define Chinese support diagram for verification rules;

[0036] Figure 3 Schematic diagram of using standard library functions for checking logic description;

[0037] Figure 4 Describe the system block diagram for the verification logic;

[0038] Figure 5 Define process diagrams for data sources;

[0039] Figure 6 This is a diagram of the feedback for the SQL parser security check;

[0040] Figure 7 This is a diagram of the feedback for the SQL parser security check;

[0041] Figure 8 A diagram showing feedback for SQL parser usability check;

[0042] Fig. 9 This is a diagram of SQL normalization for SQL parser;

[0043] Fig.10 This is a diagram of the SQL parser structure and workflow;

[0044] Fig.11 A schematic diagram of the compilation process for the verification logic description;

[0045] Fig.12 This is an example of a compilation error prompt;

[0046] Fig.13 This is a diagram showing the mismatch of branch expression types;

[0047] Fig.14 This is a diagram showing the mismatch of function parameter types;

[0048] Fig.15 A reference error diagram is provided for the verification logic description;

[0049] Fig.16 It is a schematic diagram of the verification engine module and structure;

[0050] Fig.17 Generate structure and flow diagram for code;

[0051] Fig.18 This is a schematic diagram of the Spark framework embedded in the verification engine. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0053] The present invention provides a data checking and computing system, which specifically comprises: a data input module, a code generation module, a computing core module and a result output module.

[0054] Specifically, the data input module extracts verification data based on relevant information of the data source and submits the verification data to the computing core module.

[0055] The data input module uses an independent data structure to describe the semi-structured data source information to realize the extraction of semi-structured verification data. The data input module adds a streaming data extraction unit, and the result output module adds a streaming data write-back unit to realize the streaming data source adaptation.

[0056] Code generation module: converts the verification rules related to the target verification data into executable code and submits it to the computing core module. The code generation module is placed in the Spark driver, and the code is distributed to the computing core modules of each executor after code generation.

[0057] Computing core module: receives the verification data and the executable code compiled and converted by the verification rules, performs the calculation and obtains the result. The core component of the computing core module is an interpreter, which has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and executed by different threads. The interpreter has a built-in execution information collection unit, which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

[0058] Result output module: receives the verification results given by the computing core module, adapts to various data storages, and completes the persistence of the results.

[0059] The present invention also provides a data verification calculation method, the method comprising:

[0060] Data input step: extract verification data based on relevant information of the data source, and submit the verification data to the calculation core step.

[0061] The data input step uses an independent data structure to describe the semi-structured data source information to extract the semi-structured verification data. The data input step adds a streaming data extraction unit, and the result output step adds a streaming data write-back unit to adapt the streaming data source.

[0062] Code generation step: convert the verification rules related to the target verification data into executable code and submit it to the computing core step; the code generation step is placed in the Spark driver, and after the code is generated, it is distributed to the computing core step of each executor.

[0063] The core steps of calculation are: receiving the verification data and the compiled and converted executable code of the verification rules, performing the calculation and obtaining the result.

[0064] Result output step: Receive the verification results given by the core computing step, adapt to various data storage, and complete the persistence of the results.

[0065] Next, the present invention will be described in more detail.

[0066] The embodiment of the present invention includes a data verification calculation system. In the calculation process, the system uses the metadata (including the final logic representation, data source knowledge and normalized SQL) generated during the compilation process of the verification logic description. These metadata come from the data verification logic description system. The relationship between the system (verification calculation) and the data verification logic description system (logic description) is as follows: Figure 1As shown, in order to explain the embodiment more completely and accurately, we must first explain the data verification logic description system and the generation of related metadata, as detailed in the following first part - data verification logic description system. On the basis of the first part, the main body of the embodiment of this invention, that is, a data verification calculation system, is described in detail in the second part. The following will be described in detail in two parts:

[0067] Part 1: Data verification logic description system.

[0068] The description of data verification logic usually contains the following three types of information:

[0069] (1) Data source: the source and data structure information of the data to be audited. For example, a data source definition can be given through SQL: select...from A where date='20211220'. This statement indicates that the data source comes from table A and the date is '20211220'.

[0070] (2) Verification logic: Based on a given data source definition, several verification logics, also called verification rules, can be defined. Each rule can be regarded as the smallest unit of a verification calculation. Usually, a data source will define dozens to hundreds of verification rules to complete a comprehensive and detailed data quality verification. Take a simple verification logic as an example: After adding the verification rules to the data source defined in (1), we can get select...from A where date='20211220'and col is notnull. It can be seen that the core logic of the verification rule is mixed with the data source elements, and is actually just a part of the where clause, that is, "col is not null".

[0071] (3) Verification output: describes the structure and purpose of saving the verification calculation results. Taking the SQL statement in (2) as an example, adding a verification output definition can be obtained: select count(*)from A where date='20211220'andcol is not null, that is, the output is a statistical analysis of the data that meets the conditions.

[0072] Data source, verification logic, and verification output are called the three elements of data verification. From the above example, we can see that traditional SQL mixes these three elements together, forming a tight coupling. Taking this as a breakthrough, through the design and implementation of a new type of logical description, the three elements are completely decoupled, thereby establishing advantages in many aspects and achieving remarkable results.

[0073] In terms of logical description design, the main focus is on the construction and selection of syntax, semantics, features, etc., so that users can safely, accurately, flexibly and conveniently define data sources and verification logic.

[0074] In terms of the implementation of logical description, based on the design work, from an engineering perspective, a supporting compiler and tool chain are implemented to support common usage scenarios, such as online real-time compilation, error feedback, syntax highlighting, logic checking and representation generation, etc., which mainly include data source analysis module and compilation module.

[0075] The following will describe the design and implementation in detail:

[0076] In the design part, the logic description is dedicated to data verification. On the basis of meeting the basic logic construction capabilities, the learning and usage thresholds are minimized. Therefore, basic Chinese support and some grammatical sugars of Chinese expressions are added at the grammatical level, such as Figure 2 As shown in the rule definition section, branch keywords can be written in the form of "if..then..otherwise.." or "if..then..else..", and comparison symbols can be written in Chinese (such as "equals") or mathematical symbols (such as ">="). In addition, some string calculations can also use Chinese expressions, and the syntax conforms to Chinese usage habits.

[0077] Logic includes basic syntax such as basic branch judgment, function call, logical operation, numerical operation, etc. Data has strict type division, including Boolean, integer (byte, short, int, long), floating point (float, double), character, string, date, timestamp, regular expression, etc. For example, Figure 2 The rule definition in the example constructs a date literal '2020-01-01' using the "date" keyword, "true" and "false" are Boolean literals, and so on.

[0078] In order to improve the determinism of calculations, inspired by functional programming, all grammatical structures are expressions, without statements, and the evaluation process does not have any side effects. Taking Java as a comparison, "if...else..." is a statement with side effects, while in this design, "if...then...else..." is an expression without side effects, and its evaluation result is the calculation result of one of the branches. This requires that the calculation result type of each branch is the same. If the type is different, an error will be prompted. Figure 2 The final evaluation result of the check rule definition in is equal to the evaluation result of the branch judgment expression.

[0079] In order to improve the safety of calculations, the logical description will be subject to strict strong type checking. If it does not meet the requirements of the calculation type, an error will be quickly fed back during compilation.

[0080] Finally, all functions used in the logic description are from the standard library and can be continuously expanded, which facilitates the efficient writing of verification logic and avoids lengthy logic or repeated definitions. The function call example is as follows Figure 3 shown.

[0081] In general, the logic description design follows the principles of safety, flexible expression, simple writing, and easy understanding.

[0082] Implementation part, such as Figure 4 The sub-modules are described as follows:

[0083] Data source analysis module: Based on the division of the above three elements, this module is responsible for completing the definition of the data source, providing online data source definition, real-time legality check and related analysis functions. In order to maintain universality, flexibility and security, the syntax of data source definition is based on a subset of the ANSI SQL language, which only contains select query clauses. This module contains a SQL parser to support the core functions. The workflow is as follows: Figure 5 As shown in the figure, the SQL edited and submitted online is parsed by the SQL parser. Once parsed, a legal data source definition is generated. The SQL parser implements a complete SQL compiler front end, which is responsible for checking the submitted SQL and mainly performs the following four functions:

[0084] (1) Safety check: Figure 6 , 7 As shown in the figure, any non-select statements are prohibited from being compiled at the syntax level to ensure that SQL statements cannot change or delete existing data in the system, thus ensuring data security.

[0085] (2) Availability check: Figure 8 As shown, check whether the submitted statement complies with the select clause syntax rules and can be used as a legal description of the data source definition;

[0086] (3) SQL normalization: Adjust the submitted statements to standardized SQL with the same semantics and easy to define subsequent verification rules. For example, Fig. 9 As shown, the asterisk (*) in the select statement is fully expanded to facilitate name references and related checks when compiling the module to check rule definitions later;

[0087] (4) Data source knowledge extraction: After the completion of step (3) above, data source knowledge can be extracted from the statement, including reference name, reference type, namespace, etc. For example, in the data source definition "select a, b, c from table", after being extracted by the SQL parser, the data source knowledge will contain references such as a, b, c, table and their types. This information not only provides metadata information support for subsequent compilation modules, but also provides runtime information support during the verification calculation process. It is worth noting that the data source knowledge follows the reference-based principle to retain all metadata. The reference-based representation has many advantages, enabling the module to support semantic access to metadata related to structured and semi-structured data at the same time, and also to support semantic access to streaming data. As a general data structure, data source knowledge provides a unified standard data interface.

[0088] The internal structure and workflow of the SQL parser are as follows Fig.10 As shown in the figure, after the SQL statement has undergone basic lexical and grammatical analysis, in the semantic analysis stage, the metadata extraction unit will interact online with the specific back-end database product to obtain all metadata related to the SQL statement, thereby supporting more stringent semantic analysis and further improving security and availability checks. At the same time, the SQL parser will process and extract these metadata into data source knowledge for use by subsequent compilation modules.

[0089] It is worth noting that, due to its functional limitations, the SQL parser does not contain any backend functions for SQL compilation and cannot truly "execute" SQL; in addition, as a connection point for multiple database products, the metadata extraction unit, while performing its basic functions, shields the technical differences of backend database products, unifies the expression of relational metadata, and provides a consistent interface to the outside world.

[0090] This module completes the definition and analysis work related to the data source, and provides metadata support for the subsequent compilation module, so that the verification rules can obtain clear computational semantics in a specific context. The data source knowledge generated by the SQL parser is a bridge connecting the data source and the verification logic, and also serves to separate the two. In addition, the data source knowledge can also play an important role in the verification calculation link, as detailed in the second part - a data verification calculation system.

[0091] Compilation module: focuses on compiling analysis, optimizing and generating final representation of the logic description. The compilation process is as follows: Fig.11As shown in the figure, the checking rules are analyzed and calculated in turn through the sub-modules of lexical, grammatical, semantic, optimization and logic generation, and finally generate a logical intermediate representation that is independent of the specific computing platform and database product. If an error occurs in any of these links, the compilation will be interrupted, and detailed compilation error information will be fed back in advance, so that users can quickly correct the logic. For example Fig.12 As shown in the figure, the expression on the right is mistakenly written as the string 5. The types on both sides of the inequality sign are inconsistent, so the compilation fails and an error is prompted.

[0092] The following will be described according to the sub-modules:

[0093] Lexical analysis submodule: Starting from the grammatical level of the verification logic, the verification rules are segmented and converted from character streams to token streams for analysis by the grammatical analysis submodule.

[0094] Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it builds a concrete syntax tree (CST) based on the LL(*) algorithm according to the syntax rules of the verification logic, and then generates an abstract syntax tree (AST), and finally submits the abstract syntax tree to the semantic analysis submodule.

[0095] Semantic analysis submodule: receives the data source knowledge provided by the data source analysis module and performs semantic analysis on the verification logic. In order to expose more logical errors in advance, strong type checking can ensure the safety of type-based calculations (such as Fig.12 , 13 , 14); the check must also ensure that the evaluation result type of the verification logic is always Boolean to reflect a clear judgment on the data quality; in addition, some rule-based type conversions can be allowed during the analysis process to improve the language usability while maintaining safety.

[0096] like Fig.15 As shown in the figure, the check analysis also includes the name references and logical consistency constraints mentioned in the check rules to fully ensure that the check rule description is complete and computable. After all the analyses are completed, an intermediate representation with type information (Typed IR) will be generated, referred to as the intermediate representation, and submitted to the subsequent optimization and logic generation submodules.

[0097] Optimization and logic generation submodule: Once the semantic analysis submodule is passed, it means that the current verification rules have passed all the checks. The main purpose of this module is to perform logic optimization work, such as Fig.11As shown in the figure, in each round of optimization, the optimizer accepts an intermediate representation and calculates a logically equivalent but more optimized intermediate representation. For example, an important optimization method is to advance all calculations containing only literals, such as 1+1 can be directly optimized to 2, without having to perform a large number of inefficient repeated calculations during the actual execution of the check rules.

[0098] After the optimizer completes multiple rounds of optimization, it can generate an optimal intermediate representation, remove some process results, trim the data structure, and generate the final logical representation. The logical representation is a data structure with data types and auxiliary computing information. It contains all the information required for the calculation and is independent of the specific database product. It can be used as a general interface to adapt to specific computing engines, frameworks or platforms. For details, see the second part below - a data verification computing system.

[0099] At this point, the data source definition and the verification rule definition have been completely decoupled. Since the verification output is closely related to the data persistence layer, it can be completed by a specific computing engine, framework or platform. For details, see the second part below - a data verification computing system.

[0100] Through appropriate modification, this system can add the following expansions:

[0101] (1) Adding support for analysis of semi-structured data: In the above data source analysis module, structured data is described using ANSI SQL, and the SQL parser is responsible for parsing and ultimately extracting data source knowledge. Similarly, this module adds json to describe the structure and data type of semi-structured data, and generates data source knowledge through the json parser. Since data source knowledge preserves metadata based on the principle of preserving references, the interface layer remains stable, and the entire compilation module can be seamlessly connected. The entire system can implement the verification rule analysis function based on semi-structured data sources, and also provide information support for the subsequent expansion of the data verification calculation system.

[0102] (2) Add support for analysis of streaming data: Similar to the idea in (1), while maintaining the reference principle, add a DSL (such as YAML) for describing the structure and type of streaming data, write a parser, generate standard data source knowledge, and then connect to the compilation module, so that the system can analyze the verification rules based on the streaming data source.

[0103] (3) Continuously optimize the final logical representation: In the above optimization and logic generation submodules, the optimizer performs progressive optimization on the intermediate representation. As the system functions expand, the optimizer can continue to add more layers of analysis and optimization, continuously improve the final logical representation, enhance the computational expression capability, and help the verification calculation link generate higher performance execution code.

[0104] On the whole, the multiple modules of this system work closely together to achieve complete decoupling of the three elements of data source, verification logic, and verification output. With the help of online real-time compilation and feedback, users can quickly complete the definition of data source and verification logic, significantly simplifying the development, testing, and deployment processes. The appropriate amount of Chinese is introduced to achieve complementary advantages between Chinese and ordinary writing, improve the readability and maintainability of the verification logic, lower the threshold for use, and reduce costs and increase efficiency; relying on a large number of strict syntax, semantic checks and strong type checks during the compilation period, the security of data and calculations is significantly improved; because the data source definition adopts the ANSI SQL subset, and the verification logic description only relies on the standard data source knowledge interface provided by the data source analysis module, it is independent of the specific database product, which completely solves the problem of logical consistency of multiple database backends, achieving the effect of writing once and running in multiple places.

[0105] Part 2: The present invention provides a data verification calculation system.

[0106] Based on the final logical representation, data source knowledge and standardized SQL provided by the data verification description system in the first part above, the data verification calculation system provided in the present invention is intended to complete large-scale verification calculation. The core is the verification engine, which provides a complete verification calculation runtime environment. The basic modules and structure schematics refer to Fig.16 As shown, the calculation core module is responsible for the actual calculation. The required target verification data and verification calculation logic are provided by the data input module and the code generation module respectively. After the calculation is completed, the result output module completes the persistence of the verification result.

[0107] The following is a detailed description of each module:

[0108] Data input module: extracts target verification data based on data source related information and submits it to the computing core module. The data source related information comes from the data source knowledge and standardized SQL ( Fig. 9 , 10 ), using this information, this module can fully perceive the structure and source of the data to be checked, and continuously provide data to the computing core module in a stable and secure manner.

[0109] Code generation module: responsible for converting the verification rules related to the target verification data into executable code in batches and submitting them to the computing core module for execution. The internal structure and workflow of the module are as follows: Fig.17As shown in the figure: The final logic representation and data source knowledge are compiled and generated by the data verification logic description system. After receiving this information, the translator traverses the final logic representation and combines the data source knowledge to generate efficient executable code. Inspired by Lambda calculus, the executable code is composed of several orthogonal basic operators. After the verification logic is compiled, it is expressed as an algebraic combination of basic operators. This set of operators can be considered as the "assembly language" of the entire verification engine.

[0110] Computing core module: receives the executable code compiled from the verification data and verification rules, performs calculations and obtains results. The core component of this module is an interpreter for the above basic operators. In order to improve concurrency and strengthen runtime control, the interpreter has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and executed by different threads. In addition, the interpreter has a built-in execution information collection unit (Metrics Unit), which is responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization. For example, after collecting the execution information of the branch logic, the hit rate of each branch logic is regularly counted. When the hit rate of a branch reaches the threshold, the optimization logic can be triggered to make the branch and the judgment logic execute concurrently, further improving the calculation concurrency.

[0111] In large-scale data verification and calculation scenarios, the verification of a data source usually involves the calculation of dozens or even hundreds of verification rules. Obviously, making full use of the multi-core characteristics of modern CPUs and merging all verification rules that rely on the same data source can save IO resources and improve computing efficiency. Based on this theory, this module designs and implements an aggregation algorithm that only requires one IO of the data source to complete the calculation of any number of verification rules from the same source, and maintains the independence of the calculation results of each rule, achieving "integration but difference".

[0112] It is worth noting that although the check calculation is interpreted at the check engine level, at the entire JVM level, all structures including the interpreter will eventually be compiled into JVM bytecode for execution, which is no different from ordinary programs.

[0113] Result output module: This module is the specific implementation of the verification output in the three elements of data verification. It is responsible for receiving the verification results given by the computing core module, adapting to various data storage, and completing the persistence of the verification results.

[0114] The verification engine can run on a single machine or connected to a distributed computing framework, easily obtaining computing power and horizontal expansion capabilities that match the data scale. Fig.18As shown in the figure, according to the characteristics of Spark distributed computing, this system embeds the verification engine into the Spark runtime system in modules, and puts the work of the code generation module in the Spark driver. After the executable code is generated, it is distributed to the computing core modules of each executor (Spark Executor). The data input and result output modules only need to be properly adapted to quickly form a distributed verification computing environment with the same functions as the stand-alone verification engine, but with greatly increased verification computing power.

[0115] In addition, since there are standard interfaces between the engine modules and loose coupling, the following expansions can be easily achieved with the metadata support provided by the data verification logic description system:

[0116] (1) Introducing verification and calculation support for semi-structured data: Based on the extended support for semi-structured data analysis provided by the data source analysis module in the data verification logic description system, this system can add an independent semi-structured data extractor to the data input module to implement the verification and calculation function of the computing engine for semi-structured data.

[0117] (2) Introducing support for streaming data verification calculations: Based on the data verification logic description system's support for streaming data verification descriptions, this system's data input module adds a streaming data extraction unit, and the result output module adds a streaming data write-back unit to implement the data verification calculation function of streaming data sources (such as Kafka).

[0118] The above two types of extensions benefit from the general computing characteristics of the computing core module. The interpreter inside the module interprets and executes based on the algebraic combination structure of basic operators. During the execution process, the interpreter obtains specific field data to be checked through reference. For example, in the check rule "name is not empty", name is a reference. The interpreter finally obtains external data to be checked from the data input module through reference. The reference-based data acquisition method is suitable for structured and unstructured data, and can also be adapted to streaming data. The computing core module can maintain interface stability and universality, which depends entirely on the principle of obtaining external data to be checked through reference. Following this principle, even if the data involved in the actual calculation does not exist (semi-structured data) or a runtime error occurs, the interpreter can handle it properly, and the exception is also regarded as a check result. Therefore, the computing core module does not need to be changed.

[0119] Overall, the data verification and computing system provides an independent and complete verification and computing environment, realizes a complete separation of storage and computing, and enables the verification computing power to be independently adjusted according to the workload; in addition, it shields the differences between various back-end database products, making it possible for multiple back-ends to reuse and share the same verification system, avoiding duplication of construction and saving scientific and technological resources; finally, the original aggregation algorithm only requires one IO to calculate all homologous verification rules, which greatly promotes the balanced use of system resources and significantly improves computing efficiency.

[0120] In recent years, financial regulators have put forward increasingly stringent requirements on the data quality of the regulatory data standardization (EAST) of major banks. EAST data has the characteristics of comprehensive business coverage, large amount of detailed data, and high timeliness of reporting. In order to continuously improve the quality of EAST data, it is urgent to build a safe and efficient massive data quality detection system.

[0121] Industrial Bank's EAST big data inspection and analysis system is built based on the data verification and calculation system in this invention. The system has more than 4,600 existing verification rules. In order to respond quickly to regulatory changes, the system provides real-time online editing, inspection and submission of verification rules, which take effect immediately without being issued, and can be recalculated immediately with the new rule logic. The average daily verification data scale is 1.3TB, and the verification time is 1-2 hours. During peak periods, 5-8TB of data are verified per day, and the calculation time is 5-9 hours. The verification time is basically linearly related to the workload.

[0122] According to preliminary tests, in the same cluster environment, using SQL to write verification rules, given the same data set and completing the same work tasks, the memory resources required for the verification process (in MB*Sec, that is, megabytes*seconds) are only 8% to 14% of the memory resources required by SQL, and the required computing resources (vcore*Sec, that is, virtual cores*seconds) are 52% to 74% of those required by SQL. The data verification computing system has a significant advantage in the utilization of IO resources, and also has certain advantages in the utilization of computing resources; in addition, in the same environment, for the verification of the same data source, the number of verification rules increased from 25 to 250, and the computing time increased to 1.9 times the original. Under the same circumstances, the SQL-based verification triggers the IO bottleneck, and the time does not conform to the linear growth.

[0123] Thanks to the flexible result output module, the data verification calculation system can choose to output statistical, sampling and other results, allowing business personnel to quickly grasp the overall picture of data quality and analyze specific problems in detail; relying on the advantages of the verification system's aggregation algorithm, the system's CPU and IO resource usage are basically balanced, and the largest single data source has more than 240 verification rules, and all verification results can be obtained with only one IO; in addition, the verification system is independent of the specific back-end database, and each branch can quickly transplant and deploy it, connect with its own data back-end, and quickly build a local personalized data verification system for the branch.

[0124] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0125] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data verification computing system, characterized in that: include: Data input module: extracts target verification data based on data source related information and submits it to the computing core module. The data source related information comes from the data source knowledge and standardized SQL provided by the data verification logic description system. Code generation module: converts the verification rules related to the target verification data into executable code and submits it to the computing core module. The workflow is as follows: after the translator receives the final logical representation and data source knowledge, it traverses the final logical representation and combines the data source knowledge to generate executable code. The executable code consists of several orthogonal basic operators. The final logic representation and data source knowledge are compiled and generated by the data verification logic description system, including: The data verification logic description system includes a data source analysis module and a compilation module; The data source analysis module includes an SQL parser, and the SQL parser workflow is as follows: after the SQL statement undergoes basic lexical and grammatical analysis, in the semantic analysis stage, the metadata extraction unit will interact online with the back-end specific database product to obtain all metadata related to the SQL statement, and the SQL parser will process and extract these metadata into data source knowledge; The compilation module includes: Lexical analysis submodule: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream processing to token stream for analysis by the syntax analysis submodule; Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it constructs a syntax tree according to the grammatical rules of the verification language, then generates an abstract syntax tree, and finally submits the abstract syntax tree to the semantic analysis submodule; Semantic analysis submodule: receives data source knowledge provided by the data source analysis submodule, and performs type and logic checks on the name references mentioned in the verification rules; Optimization and logic generation submodule: After passing the semantic analysis submodule, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation. The optimization method is to advance all calculations containing only literals. After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, and trims the data structure before generating the final logical representation. Calculation core module: receives the verification data and the executable code converted by the verification rules, performs the calculation and obtains the result; The core module component of the computing core module is an interpreter for basic operators. The interpreter has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and respectively handed over to different threads for execution. Result output module: receives the verification results given by the computing core module, adapts to various data storages, and completes the persistence of the results.

2. The data verification computing system according to claim 1, characterized in that: The data input module uses an independent data structure to describe semi-structured data source information, thereby realizing the extraction of semi-structured verification data.

3. The data verification computing system according to claim 1, characterized in that: The data input module adds a streaming data extraction unit, and the result output module adds a streaming data write-back unit to achieve streaming data source adaptation.

4. The data verification computing system according to claim 1, characterized in that: The code generation module is placed in the Spark driver, and the code is distributed to the computing core modules of each executor after being generated.

5. The data verification computing system according to claim 1, characterized in that: The interpreter has a built-in set of execution information collection units, which are responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

6. A data verification calculation method, characterized in that: include: Data input step: extract target verification data based on data source related information and submit it to the calculation core step. The data source related information comes from the data source knowledge and standardized SQL provided by the data verification logic description system. Code generation step: convert the verification rules related to the verification data into executable code and submit it to the computing core step. The workflow is as follows: after the translator receives the final logical representation and data source knowledge, it traverses the final logical representation and combines the data source knowledge to generate executable code. The executable code consists of several orthogonal basic operators. The final logic representation and data source knowledge are compiled and generated by the data verification logic description system, including: The data verification logic description system includes a data source analysis module and a compilation module; The data source analysis module includes an SQL parser, and the SQL parser workflow is as follows: after the SQL statement undergoes basic lexical and grammatical analysis, in the semantic analysis stage, the metadata extraction unit will interact online with the back-end specific database product to obtain all metadata related to the SQL statement, and the SQL parser will process and extract these metadata into data source knowledge; The compilation module includes: Lexical analysis submodule: Starting from the lexical level of the verification logic description, the verification rule definition is segmented and converted from character stream processing to token stream for analysis by the syntax analysis submodule; Syntax analysis submodule: After receiving the token stream provided by the lexical analysis submodule, it constructs a syntax tree according to the grammatical rules of the verification language, then generates an abstract syntax tree, and finally submits the abstract syntax tree to the semantic analysis submodule; Semantic analysis submodule: receives data source knowledge provided by the data source analysis submodule, and performs type and logic checks on the name references mentioned in the verification rules; Optimization and logic generation submodule: After passing the semantic analysis submodule, logic optimization is performed. In each round of optimization, the optimizer accepts an intermediate representation and obtains a logically equivalent but more optimized intermediate representation after calculation. The optimization method is to advance all calculations containing only literals. After the optimizer completes multiple rounds of optimization, it generates an optimal intermediate representation, removes the intermediate process results, and trims the data structure before generating the final logical representation. The core calculation step: receiving the check data and the executable code compiled and converted by the check rules, executing the calculation and obtaining the result. The core component of the core calculation step is an interpreter for basic operators. The interpreter has an independent thread pool. According to the algebraic structure of the operator combination, the concurrent parts are disassembled and respectively handed over to different threads for execution. Result output step: Receive the verification results given by the core computing step, adapt to various data storage, and complete the persistence of the results.

7. The data checking calculation method according to claim 6, characterized in that: The data input step uses an independent data structure to describe the semi-structured data source information, thereby realizing the extraction of semi-structured verification data.

8. The data checking calculation method according to claim 6, characterized in that: The data input step adds a streaming data extraction unit, and the result output step adds a streaming data write-back unit to achieve streaming data source adaptation.

9. The data checking calculation method according to claim 6, characterized in that: The code generation step is placed in the Spark driver, and the code is distributed to the computing core steps of each executor after generation.

10. The data checking calculation method according to claim 6, characterized in that: The interpreter has a built-in set of execution information collection units, which are responsible for collecting information during the calculation process and feeding it back to other units for runtime optimization.

Images