Data quality inspection method, system and method based on model indexes

Abstract

The invention provides a data quality inspection system and method based on model indexes. The system comprises a quality inspection modeling module, an SQL generation module, a scheduling management module, a quality inspection execution module and a quality inspection billboard module. The quality inspection modeling module is used for establishing an entity table logic model, quality inspection indexes, index assignment and a data set; the SQL generation module is used for generating a query SQL statement according to the quality inspection indexes, the index assignment and the data set of the quality inspection modeling module; the scheduling management module is used for generating a quality inspection scheduling task according to the query SQL statement; the quality inspection execution module is used for performing quality inspection on the data in the database to be detected by executing the quality inspection scheduling task, and counting and recording data quality inspection information; and the quality inspection board module is used for visualizing quality inspection statistical information. And the accuracy and efficiency of multi-table quality inspection and complex logic quality inspection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data quality inspection system and method based on model indicators. Background Art

[0002] At present, data, as a new type of production factor, is the basis of digitalization, networking, and intelligence. It has been rapidly integrated into various links such as production, distribution, circulation, consumption, and social service management, profoundly changing the mode of production, lifestyle, and social governance. Improving data quality is a major problem that needs to be solved urgently. Quality inspection of data and screening out available, high-quality data from massive amounts of stock data is an effective way to improve data quality. However, current data quality inspection methods consume a lot of resources and are not efficient. They only have good effects on quality inspection of single tables and simple logic. Multi-table quality inspection can only deal with quality inspection of data comparison between two tables. How to find a time-saving, labor-saving, fast and efficient quality inspection method for complex logic of multiple tables is an urgent problem that needs to be solved. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a data quality inspection system and method based on model indicators, the system comprising: a quality inspection modeling module, an SQL generation module, a scheduling management module, a quality inspection execution module and a quality inspection dashboard module;

[0004] Quality inspection modeling module, used to establish entity table logical model, quality inspection indicators, indicator assignment and data set;

[0005] SQL generation module, used to generate query SQL statements according to the quality inspection indicators, indicator assignments and data sets of the quality inspection modeling module;

[0006] A scheduling management module, used to generate quality inspection scheduling tasks according to the query SQL statement;

[0007] A quality inspection execution module is used to perform quality inspection on the data in the database to be inspected by executing the quality inspection scheduling task, and to statistically record data quality inspection information;

[0008] The quality inspection dashboard module is used to visualize quality inspection statistics.

[0009] Furthermore, the quality inspection modeling module is used to establish the entity table logical model, quality inspection indicators, indicator assignment and data set, including:

[0010] For each entity table, remove the fields that are not of interest to quality inspection, and retain the fields of dimension, date, measure, and primary key type;

[0011] Automatically create simple indicators with corresponding names through the fields of the measurement type, and assign values ​​to the simple indicators for simple logical quality inspection of data;

[0012] Creating complex indicators by connecting multi-field mathematical symbols or logical symbols, and assigning values ​​to the complex indicators for complex logical quality inspection of data;

[0013] One or more data sets are formed by the dimension fields, simple indicators and complex indicators.

[0014] Furthermore, the dimension field is a statistical angle.

[0015] Furthermore, the data set is a logically wide table;

[0016] Dimension fields, simple indicators and complex indicators are logical fields of the wide table;

[0017] The wide table converts the multi-table quality inspection into a single-table quality inspection;

[0018] The complex indicators convert complex logic into simple logic.

[0019] Furthermore, the scheduling management module is also used to execute the quality inspection scheduling tasks according to a preset timing cycle.

[0020] The present invention also provides a data quality inspection method based on model indicators, including:

[0021] Establish entity table logical model, quality inspection indicators, indicator assignment and data set;

[0022] Generate a query SQL statement based on the quality inspection indicators, indicator values ​​and data sets;

[0023] Generate a quality inspection scheduling task according to the query SQL statement;

[0024] By executing the quality inspection scheduling task, the data in the database to be inspected is quality inspected, and the data quality inspection information is statistically recorded.

[0025] Furthermore, it also includes:

[0026] Visualize quality inspection statistics.

[0027] Furthermore, the entity table logical model, quality inspection indicators, indicator assignment and data set are established, including:

[0028] For each entity table, remove the fields that are not of interest to quality inspection, and retain the fields of dimension, date, measure, and primary key type;

[0029] Automatically create simple indicators with corresponding names through the fields of the measurement type, and assign values ​​to the simple indicators for simple logical quality inspection of data;

[0030] Creating complex indicators by connecting multi-field mathematical symbols or logical symbols, and assigning values ​​to the complex indicators for complex logical quality inspection of data;

[0031] One or more data sets are formed by the dimension fields, simple indicators and complex indicators.

[0032] Furthermore, the data set is a logically wide table;

[0033] Dimension fields, simple indicators and complex indicators are logical fields of the wide table;

[0034] The wide table converts the multi-table quality inspection into a single-table quality inspection;

[0035] The complex indicators convert complex logic into simple logic.

[0036] Furthermore, after the step of generating a quality inspection scheduling task according to the query SQL statement, the method further includes:

[0037] The quality inspection scheduling tasks are executed according to the preset timing cycle.

[0038] The data quality inspection system and method based on model indicators provided by the present invention realize automatic quality inspection of complex data logic on the one hand, and automatic multi-table quality inspection on the other hand, solving the problem that the accuracy of complex logic quality inspection is not high and multi-table quality inspection can only be done manually. The present invention greatly improves the accuracy and efficiency of multi-table quality inspection and complex logic quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a structural schematic diagram of a data quality inspection system based on model indicators provided by an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of the structure of a quality inspection modeling module involved in an embodiment of the present invention;

[0041] Figure 3 It is a flow chart of a data quality inspection method based on model indicators provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0043] Example 1

[0044] The present invention provides a data quality inspection system based on model indicators, such as Figure 1As shown, it includes: quality inspection modeling module, SQL generation module, scheduling management module, quality inspection execution module and quality inspection dashboard module;

[0045] Quality inspection modeling module, used to establish entity table logical model, quality inspection indicators, indicator assignment and data set;

[0046] SQL generation module, used to generate query SQL statements according to the quality inspection indicators, indicator assignments and data sets of the quality inspection modeling module;

[0047] A scheduling management module, used to generate quality inspection scheduling tasks according to the query SQL statement;

[0048] A quality inspection execution module is used to perform quality inspection on the data in the database to be inspected by executing the quality inspection scheduling task, and to statistically record data quality inspection information;

[0049] The quality inspection dashboard module is used to visualize quality inspection statistics.

[0050] The structure of the quality inspection modeling module is as follows Figure 2 As shown, it is used to remove fields that are not of interest to quality inspection for each entity table, and retain fields of dimension, date, measurement and primary key types; automatically create simple indicators with corresponding names through fields of measurement type, and use them for simple logical quality inspection of data by assigning values ​​to the simple indicators; create complex indicators based on multi-field mathematical symbols or logical symbols, and use them for complex logical quality inspection of data by assigning values ​​to the complex indicators; form one or more data sets through the dimension fields, simple indicators and complex indicators. The dimension fields are from a statistical perspective. The date is a date or timestamp in a narrow sense. The primary key here is different from the primary key in the narrow sense in the database. Any field that can be associated with other models can be called a primary key. The data set is a logically wide table; the dimension fields, simple indicators and complex indicators are the logical fields of the wide table; the wide table converts multi-table quality inspection into single-table quality inspection; the complex indicators convert complex logic into simple logic.

[0051] The quality inspection execution module provides data support for data rectification and quality inspection dashboards.

[0052] Example 2

[0053] Based on the same inventive concept, the present invention also provides a data quality inspection method based on model indicators, such as Figure 3 As shown, including:

[0054] Step S301, establishing entity table logical model, quality inspection indicators, indicator assignment and data set;

[0055] Step S302, generating a query SQL statement according to the quality inspection index, index assignment and data set;

[0056] Step S303, generating a quality inspection scheduling task according to the query SQL statement;

[0057] Step S304: Perform quality inspection on the data in the database to be inspected by executing the quality inspection scheduling task, and statistically record the data quality inspection information.

[0058] Furthermore, it also includes:

[0059] Visualize quality inspection statistics.

[0060] Furthermore, the entity table logical model, quality inspection indicators, indicator assignment and data set are established, including:

[0061] For each entity table, remove the fields that are not of interest to quality inspection, and retain the fields of dimension, date, measure, and primary key type;

[0062] Automatically create simple indicators with corresponding names through the fields of the measurement type, and assign values ​​to the simple indicators for simple logical quality inspection of data;

[0063] Creating complex indicators by connecting multi-field mathematical symbols or logical symbols, and assigning values ​​to the complex indicators for complex logical quality inspection of data;

[0064] One or more data sets are formed by the dimension fields, simple indicators and complex indicators.

[0065] Furthermore, the data set is a logically wide table;

[0066] Dimension fields, simple indicators and complex indicators are logical fields of the wide table;

[0067] The wide table converts the multi-table quality inspection into a single-table quality inspection;

[0068] The complex indicators convert complex logic into simple logic.

[0069] Furthermore, after the step of generating a quality inspection scheduling task according to the query SQL statement, the method further includes:

[0070] The quality inspection scheduling tasks are executed according to the preset timing cycle.

[0071] Example 3

[0072] For example, there is a scenario with two tables. One is the coffee product information table, which contains five fields: product ID, coffee type, cup size, product name, and price (price). The other is the coffee sales table, which contains five fields: order ID, order date, store, product ID, and sales quantity (num). The quality inspection requirement is that data with sales less than 100,000 in half a year is invalid data. We cannot obtain sales from a single table, and cannot directly determine that the data of those stores is invalid. In order to quality inspect these data, we need to establish a quality inspection model. The fields in the coffee product information table are marked as primary key, dimension, dimension, dimension, and dimension in sequence. We do not care about the order ID in the coffee sales table and can remove it. The remaining fields are marked as date, dimension, primary key, and indicator in sequence. The system automatically creates a simple indicator named num for the sales quantity. In order to quality inspect the sales, we first create a complex indicator sales (amount) with price*sales quantity (price*num), and then select coffee type, cup size, product name, price, store, sales quantity, and sales to create a coffee sales data set. This is a logical wide table containing 7 fields. The system will generate a database query SQL based on the above quality inspection model: SELECT store, SUM (sales) AS sales in the last six months FROM coffee data set WHERE data date > = '2024-02-05' AND data date < = '2024-08-05' GROUP BY store; the quality inspection execution module executes the above SQL statement to quality inspect the data in the database and record the data quality inspection information. The complete quality inspection process is now complete.

[0073] The data quality inspection system and method based on model indicators provided by the present invention realize automatic quality inspection of complex data logic on the one hand, and automatic multi-table quality inspection on the other hand, solving the problem that the accuracy of complex logic quality inspection is not high and multi-table quality inspection can only be done manually. The present invention greatly improves the accuracy and efficiency of multi-table quality inspection and complex logic quality inspection.

[0074] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalents that do not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data quality inspection system based on model indicators, characterized in that: include: Quality inspection modeling module, SQL generation module, scheduling management module, quality inspection execution module and quality inspection dashboard module; Quality inspection modeling module, used to establish entity table logical model, quality inspection indicators, indicator assignment and data set; SQL generation module, used to generate query SQL statements according to the quality inspection indicators, indicator assignments and data sets of the quality inspection modeling module; A scheduling management module, used to generate quality inspection scheduling tasks according to the query SQL statement; A quality inspection execution module is used to perform quality inspection on the data in the database to be inspected by executing the quality inspection scheduling task, and to statistically record data quality inspection information; The quality inspection dashboard module is used to visualize quality inspection statistics.

2. The system according to claim 1, characterized in that The quality inspection modeling module is used to establish entity table logical models, quality inspection indicators, indicator assignments, and data sets, including: For each entity table, remove the fields that are not of interest to quality inspection, and retain the fields of dimension, date, measure, and primary key type; Automatically create simple indicators with corresponding names through the fields of the measurement type, and assign values ​​to the simple indicators for simple logical quality inspection of data; Creating complex indicators by connecting multi-field mathematical symbols or logical symbols, and assigning values ​​to the complex indicators for complex logical quality inspection of data; One or more data sets are formed by the dimension fields, simple indicators and complex indicators.

3. The system according to claim 2, characterized in that The dimension field is the statistical angle.

4. The system according to claim 1 or 2, characterized in that: The data set is a logically wide table; Dimension fields, simple indicators and complex indicators are logical fields of the wide table; The wide table converts the multi-table quality inspection into a single-table quality inspection; The complex indicators convert complex logic into simple logic.

5. The system according to claim 1, characterized in that The scheduling management module is also used to execute quality inspection scheduling tasks according to a preset timing cycle.

6. A data quality inspection method based on model indicators, characterized in that: include: Establish entity table logical model, quality inspection indicators, indicator assignment and data set; Generate a query SQL statement based on the quality inspection indicators, indicator values ​​and data sets; Generate a quality inspection scheduling task according to the query SQL statement; By executing the quality inspection scheduling task, the data in the database to be inspected is quality inspected, and the data quality inspection information is statistically recorded.

7. The method according to claim 6, characterized in that Also includes: Visualize quality inspection statistics.

8. The method according to claim 6, characterized in that Establish entity table logical model, quality inspection indicators, indicator assignment and data set, including: For each entity table, remove the fields that are not of interest to quality inspection, and retain the fields of dimension, date, measure, and primary key type; Automatically create simple indicators with corresponding names through the fields of the measurement type, and assign values ​​to the simple indicators for simple logical quality inspection of data; Creating complex indicators by connecting multi-field mathematical symbols or logical symbols, and assigning values ​​to the complex indicators for complex logical quality inspection of data; One or more data sets are formed by the dimension fields, simple indicators and complex indicators.

9. The method according to claim 6 or 8, characterized in that: The data set is a logically wide table; Dimension fields, simple indicators and complex indicators are logical fields of the wide table; The wide table converts the multi-table quality inspection into a single-table quality inspection; The complex indicators convert complex logic into simple logic.

10. The method according to claim 6, characterized in that After the step of generating a quality inspection scheduling task according to the query SQL statement, the method further includes: The quality inspection scheduling tasks are executed according to the preset timing cycle.

Images