Data quality detection method and system for cross-data layer libraries and tables of associated data sources

Abstract

The present invention relates to a data quality detection method and system for cross-data layer libraries and tables of associated data sources, and belongs to the field of data governance technology. The present invention includes: configuring data quality detection indicators, setting table quality detection indicators and field quality detection indicators, and verifying table quality detection indicators and field quality detection indicators; initiating a single batch data quality detection task, extracting and selecting different data layers from multiple data sources to associate tables and fields in the tables; configuring table quality detection indicator weights for a general quality inspection table, and then configuring different weights of field quality detection indicators for all selected fields in the general quality inspection table; setting a data quality detection timed task, performing a batch data quality detection task according to the set timed task, and obtaining a data quality detection report; the present invention can more comprehensively and flexibly realize quality detection of the data table as a whole and accurate to the field level.

Description

Technical Field

[0001] The present invention relates to a data quality detection method and system for cross-data layer libraries and tables of associated data sources, belonging to the technical field of data governance. Background Art

[0002] When designing data quality detection methods and systems in practical applications, we should fully consider that data governance strategies can cover the business logic and data characteristics of most application scenarios, and can flexibly adjust detection tasks and indicators as business development and data changes, to ensure the continued effectiveness and adaptability of data quality detection, and avoid a one-size-fits-all governance approach. At present, most data quality detection methods and systems detect a single data source or target table, and do not fully consider the batch quality inspection requirements of creating a common quality inspection table for multiple data sources at different quality inspection layers for quality inspection and cleaning conversion, and fail to effectively solve the problem of applying the same quality inspection indicators and assigning different weights to different quality inspection requirements for multiple target fields. Data quality detection methods often focus on the diversity of indicator applications, and it is difficult to provide an in-depth and detailed field data quality detection method to cover all key data quality dimensions. At present, there is a lack of relevant technical solutions for field quality weight calculation and detection. There is no clear technical demonstration and experimental proof on how to adjust the weight setting of a single field quality indicator applied to different fields, whether the weight setting for a single field will affect the effect of the field quality indicator applied to the entire data quality detection, and whether the value range of the field indicator weight adjustment interval should be controlled to obtain effective quality detection results. Summary of the invention

[0003] In view of the above problems, the present invention provides a method and system for detecting data quality of associated data sources across data layer libraries and tables. The present invention avoids the deviation that may be caused by a single weight allocation method, and solves the problem of flexibly configuring data quality detection indicators and weights when associated data sources select fields in multiple data layers for batch detection in data quality detection; the method of the present invention can more comprehensively and flexibly realize quality detection of the data table as a whole and accurate to the field level; the system of the present invention can support processing of data from multiple data sources and different data layers.

[0004] The technical solution of the present invention is: in a first aspect, the present invention provides a data quality detection method for cross-data layer libraries and tables of associated data sources, the method comprising:

[0005] S1. Configure data quality detection indicators: set table quality detection indicators and field quality detection indicators, and verify table quality detection indicators and field quality detection indicators;

[0006] S2. Initiate a single batch data quality inspection task, extract and select different data layers from multiple data sources to associate tables and fields in the tables; configure the weights of table quality inspection indicators for the general quality inspection table, and then configure different weights of field quality inspection indicators for all selected fields in the general quality inspection table;

[0007] S3. Set a scheduled task for data quality detection, perform batch data quality detection tasks according to the set scheduled task, and obtain a data quality detection report.

[0008] As a further solution of the present invention, in S1, the data quality detection indicator configuration includes configuration operations of naming and indicator writing the description table quality detection indicator and the field quality detection indicator;

[0009] The naming of table quality detection indicators includes: "Is it a dimension table?", "Is there associated data?", "Is the data table name in compliance with the specification?"

[0010] Among them, "Is it a dimension table" needs to detect whether the fields in a table are in a one-to-one relationship; "Is there related data" needs to detect whether there are fields in the table that can be used for cross-table association analysis. The fields that need to be used for data association in the data quality inspection table should be set in advance;

[0011] The names of field quality inspection indicators include: "Duplicate value check", "Existence of null values", "Whether the field naming complies with the specification", and "Value range inspection";

[0012] Among them, "value range check" is to detect the fields that use code values ​​instead of actual values. If there are abnormal values ​​that do not belong to the value range, a rectification task is generated to let the data source unit confirm the meaning of the abnormal value and add it to the code table;

[0013] Update and maintain data quality detection indicators based on the two dimensions of tables and fields, set the quality indicators corresponding to each indicator, and write the quality indicators into the background; set the task attributes corresponding to each data quality detection indicator, and finally generate corresponding data quality operation and maintenance tasks and data quality rectification tasks based on the detection results.

[0014] As a further solution of the present invention, in S2, when initiating a single batch data quality inspection task, multiple quality inspection data layers are selected, and multiple databases and data tables are synchronously selected and corresponding to each data layer. In each data layer, the selected multiple data tables are merged into a general quality inspection table through data mapping rules, and the general quality inspection table exists in the selected multiple quality inspection data layers. Furthermore, the single quality inspection task performs quality inspection on the same general quality inspection table data stored in different layers. Different table quality inspection indicators and field quality inspection indicators are configured for each layer, and weights are set for each quality inspection indicator. The naming of the quality inspection data layer includes the data detail layer DWD, the data intermediate layer DWM, the data service layer DWS, and the data application layer ADS. In actual quality inspection, a general quality inspection table is created and named in the quality inspection data layer according to governance needs, and the same data will be stored in different quality inspection data layers for different cleaning and quality inspection tasks.

[0015] As a further solution of the present invention, in the single batch data quality inspection task of S2, when configuring the weight of the field quality inspection indicator Q for all selected fields in the general quality inspection table A The configuration method in which the field quality inspection indicator Q A is applied to multiple different fields for detection is as follows: The weight of Q A appearing in multiple fields is the cumulative product of the weights of all occurrence quantities, W A =W A1 W A2 W A3… W Ak... W An ; The specific calculation formula of W A is: W A = , where m is the sum of the quantities for setting weights when a single field quality inspection indicator is applied to multiple different fields for detection;

[0016] When the same field quality inspection indicator is applied to different fields and different weights are assigned, the weight of any field quality inspection indicator Q A appearing in a field is W An = W An is a decimal, 0 < W An < 1. In the single batch data quality inspection task, n is the sum of the quantities of the field quality inspection indicators applied to each field, X is the adjustment interval of the weight value of each field quality inspection indicator, -1 < X < 1; for the weight setting of multiple field quality inspection indicators configured for the same field, it is ensured that the sum of the weights of all field quality inspection indicators is 1.

[0017] ​​As a further solution of the present invention, in said S2, when configuring the weights of the table quality inspection indicators for the general quality inspection table in a single batch data quality inspection task, the general quality inspection table configuration table for table quality inspection indicators Q1 to Q N Assign corresponding weights W1 to W N ,Field quality inspection indicators are applied multiple times to different selected fields.,In one task, the general quality inspection table performs quality inspection based on,fields 1 to N.

[0018] As a further solution of the present invention, in S3, the data quality detection report includes quality detection number, detection time, detection data layer, detection database, detailed information of data fields involved and data source unit name; the data quality detection report lists all the data quality indicators detected, and calculates the single data quality indicator detection score based on the arithmetic average of each data quality indicator. The final total score of the data quality detection report is calculated by the weighted average of each indicator.

[0019] In a second aspect, the present invention provides a data quality detection system for cross-data layer libraries and tables of associated data sources, including:

[0020] Data quality detection index configuration module: used to set table quality detection index and field quality detection index, and perform table quality detection index and field quality detection index verification;

[0021] Data quality inspection index management module: used to initiate a single batch data quality inspection task, extract and select different data layers from multiple data sources to associate tables and fields in the tables; configure table quality inspection index weights for the general quality inspection table, and then configure different weights of field quality inspection indicators for all selected fields in the general quality inspection table;

[0022] Data quality detection module: used to set data quality detection scheduled tasks, perform batch data quality detection tasks according to the set scheduled tasks, and obtain data quality detection reports.

[0023] As a further embodiment of the present invention, the score G of the data quality report is ;in, Indicates the quality detection index of a single field in batch detection score; Indicates the quality detection index of a single field in batch detection The score, Indicates the quality detection index of a single field in batch detection where N is the sum of the number of field quality detection indicators applied in a batch data quality detection task.

[0024] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0026] The beneficial effects of the present invention are:

[0027] 1. The present invention designs a data quality weight calculation formula for batch testing of the same data quality detection indicator in multiple fields. The data quality detection indicator of the same field can be assigned different weights. By reasonably adjusting the weight of the same field quality detection indicator applied between different fields, the final score of the field quality indicator is affected. The existing methods do not consider the problem of flexibly configuring data quality detection indicators and adjusting weights for multiple fields in data quality detection. The quality detection report of the present invention generates a single indicator score and a weighted average overall score of the entire quality detection task for a single indicator weight setting. The generated quality detection report satisfies the evaluation of quality inspection data from the local to the overall. The local embodiment is that the report evaluates data quality by using a single quality detection indicator score as a single dimension, and the overall embodiment is that the report uses a weighted average to calculate the overall score of all quality detection indicators to evaluate the data.

[0028] 2. In the present invention, batch quality inspection tasks can select data layers to associate multiple data source libraries and tables, so as to merge multiple table data into a common quality inspection table and store it in different quality inspection layers, thus solving the problem of different requirements for cross-unit quality inspection of the same batch of data in data quality inspection and the difficulty of cleaning and converting according to different indicators. The above-mentioned data stratification batch quality inspection method greatly improves the efficiency and ease of use of data quality inspection;

[0029] 3. The present invention distinguishes between table quality inspection indicators and field quality inspection indicators, sets corresponding quality operation and maintenance and rectification attributes for specific indicators, and automatically generates quality rectification and operation and maintenance tasks for the corresponding data of indicators that are lower than the quality inspection score threshold set by the system after the quality inspection is completed; provides flexible and configurable data quality inspection indicator management and indicator writing form, which can meet the data quality management needs of different industries and different business scenarios;

[0030] 4. The data quality inspection report generated by this method and system generates a warning data list for the quality inspection task indicator data that is below the quality inspection threshold score, which satisfies data governance personnel to carry out accurate data quality operation and maintenance according to specific warning indicators. At the same time, it displays the warning list data and the data quality inspection ranking of each data source unit according to the data source unit, which can timely urge the data source unit to make data quality rectification and feedback, and help data governance personnel to dynamically monitor data quality abnormal indicators and establish a data update and quality improvement management mechanism with the data source unit through quality inspection tasks;

[0031] 5. The present invention can realize the quality inspection of the data table as a whole and accurately to the field level in a relatively comprehensive and flexible manner; the data quality inspection system supports the processing of data from multiple data sources and different data layers.

[0032] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is an overall flow chart of the data quality detection method of the cross-data layer library table of the associated data source of the present invention;

[0034] Figure 2 It is a schematic diagram of the functional modules of the data quality detection system of the cross-data layer library table of the associated data source of the present invention;

[0035] Figure 3 A schematic diagram of the selection and weight configuration of data quality inspection indicators for a general quality inspection table in batch data quality inspection of the present invention;

[0036] Figure 4 A detailed schematic diagram of a quality inspection report containing individual scores and comprehensive scores of data quality inspection indicators of the present invention;

[0037] Figure 5 This is a schematic diagram of the system interface for performing data quality early warning based on data source units and quality detection tasks in the present invention;

[0038] Figure 6 This is an experimental diagram of the weight calculation formula for single field data quality detection index in batch quality inspection of the present invention. DETAILED DESCRIPTION

[0039] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0040] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0041] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0042] The present invention solves the problem that data quality detection in the data governance process cannot perform single batch data quality detection based on the weights set by the table and field data quality detection indicators. The present invention extracts from multiple data sources and selects different data layers to associate data tables and fields. A general quality inspection table is configured to manage and track data quality. Multiple table data quality detection indicators and weights are configured in a general quality inspection table, and multiple fields are selected to configure corresponding field data quality inspection indicators and assign weights. The same field data quality detection indicator is allowed to be applied to different fields and assigned different proportional weights. The total weight of the field data quality indicator is finally generated by multiplying the weights of all the times the indicator appears.

[0043] First, as Figure 1 The figure is a general flow chart of the batch data quality detection method of the cross-data layer library table of the associated data source of the present invention. The specific steps of the method of the present invention are as follows:

[0044] S1. Configure data quality detection indicators: set table quality detection indicators and field quality detection indicators, and verify table quality detection indicators and field quality detection indicators;

[0045] The data quality detection indicator configuration includes configuration operations of naming and indicator writing for description table quality detection indicators and field quality detection indicators;

[0046] The naming of table quality detection indicators includes: "Is it a dimension table", "Is there related data", "Is the data table name in compliance with the specifications", etc.; among them, "Is it a dimension table" requires detecting whether the fields in a table are in a one-to-one relationship; taking the enterprise basic information table and the enterprise branch registration information table as examples, in the enterprise basic information table, one unified social credit code only corresponds to one enterprise, and the table is identified as a dimension table in the data quality detection; while in the enterprise branch registration information table, one unified social credit code can correspond to multiple branches, and the table is identified as not a dimension table in the data quality detection. For the detection of dimension tables, it is also necessary to clarify the unique identification field. The unified social credit code is a common unique identification field; "Is there related data" requires detecting whether there are fields in the table that can be used for cross-table correlation analysis, and setting the fields in the data quality detection table that need to be used for data association in advance, such as the enterprise unified social credit code, enterprise name, legal person ID card, etc.

[0047] The names of field quality inspection indicators include: "Duplicate value check", "Existence of null values", "Whether the field naming complies with the specification", "Value range check", etc.

[0048] Among them, "value range check" is to detect the fields that use code values ​​instead of actual values. If there are abnormal values ​​that do not belong to the value range, a rectification task is generated to let the data source unit confirm the meaning of the abnormal value and add it to the code table;

[0049] Data governance personnel can update and maintain data quality detection indicators based on the two dimensions of tables and fields, set the quality indicators corresponding to each indicator, and write the quality indicators to the background using SQL statements or execution scripts; set the corresponding task attributes of the indicator based on each data quality detection indicator, and finally generate corresponding data quality operation and maintenance tasks and data quality rectification tasks based on the detection results.

[0050] S2. Data governance personnel initiate a single batch data quality inspection task, extract and select different data layers from multiple data sources to associate tables and fields in the tables; configure table quality inspection indicator weights for the general quality inspection table, and then configure different weights for field quality inspection indicators for all selected fields in the general quality inspection table;

[0051] In S2, when initiating a single batch data quality inspection task, the data governance personnel select multiple quality inspection data layers, and simultaneously select multiple databases and data tables corresponding to each data layer. In each data layer, the selected multiple data tables are merged through data mapping rules to form a general quality inspection table, and the general quality inspection table exists in the selected multiple quality inspection data layers; then the single quality inspection task performs quality inspection on the same general quality inspection table data stored in different layers, and each layer is configured with different table quality inspection indicators and field quality inspection indicators, and a weight is set for each quality inspection indicator; the quality inspection data layer naming includes data detail layer DWD, data middle layer DWM, data service layer DWS, and data application layer ADS. In actual quality inspection, a general quality inspection table is created and named in the quality inspection data layer according to governance needs, and the same data will be stored in different quality inspection data layers for different cleaning and quality inspection tasks.

[0052] The specific implementation method of merging the selected multiple data tables to form a general quality inspection table at each data layer is: "Use data extraction tools to extract the required data tables from multiple data sources, clean the extracted data tables and establish associations and data mapping rules between the tables. At each data layer, according to the data mapping rules, merge the selected multiple data tables to generate a general quality inspection table containing multiple tables and fields. The quality inspection table can contain various information required for data quality assessment, such as data accuracy, completeness, consistency, timeliness and accessibility."

[0053] In S2, the data management personnel name the quality inspection task and the general quality inspection table for this quality inspection. Multiple copies of the general quality inspection table corresponding to each quality inspection task are stored in databases related to different quality inspection data layers and data source units as containers for table objects and inspection fields for this quality inspection. The data management personnel select multiple source layer databases as the data source for quality inspection, and synchronize the data tables and related fields of multiple source layers based on the existing quality inspection data layer association selection. By merging the fields in multiple tables into the general quality inspection table, it is possible to manage the same named general management tables and fields in multiple quality inspection data layers. This helps to manage data from multiple quality inspection layers in one quality inspection task, facilitating subsequent unified cleaning and conversion.

[0054] In a single batch data quality inspection task, S2 performs field quality inspection index Q on all selected fields in the general quality inspection table. A When configuring the weight of the field quality detection indicator Q A The configuration method used for testing in multiple different fields is: Field quality detection indicator Q A The weight of a field that appears in multiple fields is the sum of the weights of all occurrences, W A =W A1 W A2 WA3… W Ak... W An ; W A The specific calculation formula of W is: W A = , where m is the sum of the number of weight settings when a single field quality detection index is applied in multiple different fields for detection;

[0055] When the same field quality detection index is applied to different fields and different weights are assigned, for any field quality detection index Q A The weight when it appears in a field is W An = , W An is a decimal, 0 < W An < 1. In a single batch data quality detection task, n is the sum of the number of field quality detection indexes applied to each field, X is the adjustment range of the weight value of each field quality detection index, -1 < X < 1; for the weight setting of multiple field quality detection indexes configured for the same field, the sum of the weights of all field quality detection indexes is 1.

[0056] As a further solution of the present invention, in the single batch data quality detection task, when configuring the weight of the table quality detection index for the general quality inspection table in S2, in one task, the general quality inspection table configuration table assigns corresponding weights W1 to W N to the table quality detection indexes Q1 to Q N . The field quality detection index is applied multiple times on different selected fields, and in one task, the general quality inspection table performs quality inspection based on fields 1 to N.

[0057] As Figure 3 shown, it is a schematic diagram of the selection and weight configuration of the data quality detection index of the general quality inspection table in the batch data quality detection of the present invention. In the batch data quality detection, the data index and weight configuration index of the general quality inspection table are batch quality inspections that meet the configuration of multiple indexes for one field and different weights for one index applied to different fields. In one task, the general quality inspection table can configure the table data quality detection indexes 1 to N and assign corresponding weights. The field quality detection index can be applied multiple times on different selected fields. In one task, the general quality inspection table can perform quality inspection based on fields 1 to N. Fields 1 and fields 2 to N can simultaneously apply the field index Q A , and the weights are set as W A1 , W A2 to W An . However, Q A may not necessarily be continuously applied to each field according to the actual detection needs. The application of other field quality detection indexes Q B to Q N is the same. The field index Q BApplied to field 1 and field N respectively set W B1 , W B2 , field index Q C Applied to field 1 and field 2, respectively, with weights set to W C1 , W C2 , field index Q D Not used in this detection task, field indicator Q E Applied to field 2 and field N, setting the weight to W respectively E1 , W E2 . Field indicator Q N According to the actual detection needs, whether it is applied continuously or discontinuously to fields 1 to N, the weight W is set respectively. N1 , W N2 To W Nn The weight setting method of the field data quality detection indicator in this method takes into account the different weight effects of a data quality detection indicator being applied to multiple field quality detections.

[0058] S3. Set a scheduled task for data quality detection, perform batch data quality detection tasks according to the set scheduled task, and obtain a data quality detection report.

[0059] As a further solution of the present invention, in S3, the data quality detection report includes quality detection number, detection time, detection data layer, detection database, detailed information of data fields involved and data source unit name; the data quality detection report lists all the data quality indicators detected, and calculates the single data quality indicator detection score based on the arithmetic average of each data quality indicator. The final total score of the data quality detection report is calculated by the weighted average of each indicator.

[0060] The present invention generates a quality inspection report for a batch data quality inspection task, which records the score of a single data quality inspection indicator and the weighted average score of the quality inspection indicators of all tables and fields in the entire report, and generates operation and maintenance and rectification tasks corresponding to the data quality inspection indicators. If the weighted average total score of the quality report is lower than the system set threshold, a quality warning for the inspection data is generated in the data quality warning module.

[0061] like Figure 4 As shown, it is a detailed schematic diagram of the quality inspection report containing single scores and comprehensive scores of data quality inspection indicators of the present invention; the data quality inspection report generated based on a single quality inspection task includes the operator's name, inspection number, inspection time, scheduled task cycle, quality inspection data layer, quality inspection data source, quality inspection data source table, general quality inspection table, quality inspection fields and information on data source units involved, and displays specific data quality inspection indicator definitions, quality inspection conclusions, single indicator scores and overall report scores based on each quality inspection indicator and classification.

[0062] As a further solution of the present invention, the analysis dimensions during data quality testing specifically include: the overall data quality test pass rate of the system, the number statistics of quality inspection libraries, the number statistics of quality inspection tables, the number of data quality test items to be maintained, and the number of data quality test items that have been rectified.

[0063] Data quality early warning indicators, single data quality detection tasks are executed according to the scheduled tasks in seconds, minutes, hours, days, months, weeks and years. The system flexibly configures the algorithm to set the cumulative data quality detection indicator early warning threshold score and the overall report early warning threshold according to the cycle and number of task executions. When the quality detection task scores below the threshold, the quality detection tasks with quality risks are displayed according to the data source unit and the single task list.

[0064] like Figure 5 As shown, it is a schematic diagram of the system interface for performing data quality early warning according to data source units and quality detection tasks of the present invention; Figure 5 Provides statistical analysis data display related to data quality detection, including the overall data quality detection pass rate of the system, the number of quality inspection libraries, the number of quality inspection tables, the number of data quality inspection items to be maintained, and the number of data quality inspection items that have been rectified. Display the unit's data quality warning in the form of a list, slide to display the warning database and warning data table of the top 10 data sources, and use a percentage progress bar to display the proportion of warning data to all data. Display a list of high-risk quality detection results.

[0065] Second, as Figure 2 As shown, the present invention provides a data quality detection system for cross-data layer libraries and tables of associated data sources, which can be used to perform metadata management, cleaning conversion and data quality detection on collected data, and support daily data update iteration and data quality improvement management;

[0066] The data quality detection system for cross-data layer libraries and tables of the associated data source includes:

[0067] Data quality detection index configuration module: used to set table quality detection index and field quality detection index, and perform table quality detection index and field quality detection index verification;

[0068] Data quality inspection index management module: used to initiate a single batch data quality inspection task, extract and select different data layers from multiple data sources to associate tables and fields in the tables; configure table quality inspection index weights for the general quality inspection table, and then configure different weights of field quality inspection indicators for all selected fields in the general quality inspection table;

[0069] Data quality detection module: used to set data quality detection scheduled tasks, perform batch data quality detection tasks according to the set scheduled tasks, and obtain data quality detection reports.

[0070] The data quality detection system for cross-data layer libraries and tables of the associated data source also includes a metadata management module, a data quality early warning module, and a data quality rectification and operation and maintenance module;

[0071] The metadata management module is used to manage metadata basic information and storage information, and manage the data source, collection method, update method and update cycle of the collection and management;

[0072] In the data quality warning module, the task of single data quality detection can be executed in seconds, minutes, hours, days, months, weeks and years through the scheduled task setting. It is used to flexibly configure the algorithm to set the cumulative data quality detection index warning valve score and the overall report warning threshold according to the cycle and number of task executions. When the quality detection task scores below the threshold, the quality detection tasks with quality risks are displayed according to the data source unit and the single task list;

[0073] The data quality rectification and operation and maintenance module is used to process the two states of data to be operated and maintained and to be rectified after data quality detection and the corresponding operations according to the data quality detection index. The data to be operated and maintained state means that after the data quality detection, it is prompted that the data management personnel need to perform manual processing in the system to meet the requirements of a certain data quality detection index; the data to be rectified state means that after the data quality detection, it is prompted that the data source unit needs to be fed back through the system to resubmit the data in order to meet the requirements of a certain data quality detection index. Then the operation corresponding to the above two states is the generation of data quality rectification tasks. If the single data quality detection index detection score or the weighted average score of the overall quality report marked with the operation and maintenance attribute is lower than the minimum threshold set by the system, the system automatically generates the quality operation and maintenance work order and operation entry corresponding to the operation and maintenance index that needs to be processed by the data management personnel in the data quality operation and maintenance module; if the single data quality detection index score or the weighted average score of the overall quality report marked with the rectification attribute is lower than the minimum threshold set by the system, the system automatically generates the quality rectification work order corresponding to the rectification index that needs to be fed back to the data source unit in the data quality rectification module and sends the prompt data source unit to update the data that meets the rectification index to the system through the system message.

[0074] In the data quality rectification and operation and maintenance module, the data quality task attributes are distinguished in the stage of defining data quality indicators. The data quality rectification task refers to the task type in which the quality detection indicator score is lower than the preset threshold and the feedback source unit needs to provide data again; the data quality operation and maintenance task refers to the operation and maintenance task type in which the quality detection indicator score is lower than the preset threshold and the system management personnel need to perform data cleaning and conversion.

[0075] As a further embodiment of the present invention, the score G of the data quality report is ;in, Indicates the quality detection index of a single field in batch detection score; Indicates the quality detection index of a single field in batch detection The score, Indicates the quality detection index of a single field in batch detection where N is the sum of the number of field quality detection indicators applied in a batch data quality detection task.

[0076] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0077] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0078] In this embodiment, Figure 6 This is a schematic diagram of the experimental calculation formula for the weight of a single field data quality inspection indicator in batch quality inspection. It is the result obtained to demonstrate the rationality of the formula design and related restrictions in the present invention. The field quality inspection indicator weight calculation formula is W= is an exponential function. The following experimental comparison is performed. The sum of the number of field quality detection indicators applied to each field is set to [5, 10, 20], and the sum of the number of weight settings for a single field quality detection indicator when applied to different fields is set to [5, 10, 20]. A The specific application of the paper explains the following conclusions:

[0079] Figure 6 In the first row of the figure, when n remains unchanged, as m increases, that is, in a batch quality inspection task, if field 1 to field N are all applied with the field quality inspection index Q A , Q A The more the number of fields is applied to, the higher the field quality detection index Q A The value range of the corresponding weight adjustment value X is more limited each time.

[0080] Figure 6In the first column of the figure, when m remains unchanged and n increases, that is, in a single batch quality inspection task, regardless of the number of field quality inspection indicators applied to each field, only when the value range of the weight adjustment value of a single field quality inspection indicator is 0.2 < X < 1, will it have an effective positive impact on the overall weight W of the field quality inspection indicator. A It will have an effective positive impact.

[0081] When both n and m change, and the value range of the weight adjustment value of a single field quality inspection indicator is -1 < X < -0.75 or 0.5 < X < 1, it will have an effective positive impact on the overall weight of the field quality inspection indicator.

[0082] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made without departing from the spirit of the present invention within the scope of knowledge possessed by those of ordinary skill in the art.

Claims

1. A method for detecting data quality across data layer libraries and tables of an associated data source, characterized by: The method comprises: S1. Configure data quality detection indicators: set table quality detection indicators and field quality detection indicators, and verify table quality detection indicators and field quality detection indicators; S2. Initiate a single batch data quality inspection task, extract and select different data layers from multiple data sources to associate tables and fields in the tables; configure the weights of table quality inspection indicators for the general quality inspection table, and then configure different weights of field quality inspection indicators for all selected fields in the general quality inspection table; In a single batch data quality inspection task, S2 performs field quality inspection index Q on all selected fields in the general quality inspection table. A When configuring the weight of the field quality detection indicator Q A The configuration method used for testing in multiple different fields is: Field quality detection indicator Q A The weight of a field that appears in multiple fields is the sum of the weights of all occurrences, W A =W A1 W A2 W A3… W Ak... W An ; W A The specific calculation formula is: A = , m is the sum of the number of weight settings when a single field quality detection indicator is applied to multiple different fields for detection; The same field quality detection index is applied to different fields and different weights are assigned. Any field quality detection index Q A appears in a field with a weight of W An = , W An is a decimal, 0 < W An < 1. In a single batch data quality detection task, n is the sum of the number of field quality detection indexes applied to each field, X is the weight value adjustment interval of each field quality detection index, -1 < X < 1; for the weight setting of multiple field quality detection indexes configured for the same field, the sum of the weights of all field quality detection indexes satisfies 1; In S2, when initiating a single batch data quality inspection task, multiple quality inspection data layers are selected, multiple databases and data tables are synchronously selected to correspond to each data layer, and the selected multiple data tables are merged into a general quality inspection table in each data layer through data mapping rules. The general quality inspection table exists in the selected multiple quality inspection data layers; then the single quality inspection task performs quality inspection on the same general quality inspection table data stored in different layers, and each layer is configured with different table quality inspection indicators and field quality inspection indicators, and a weight is set for each quality inspection indicator; S3. Set a scheduled task for data quality detection, perform batch data quality detection tasks according to the set scheduled task, and obtain a data quality detection report.

2. The data quality detection method for cross-data layer libraries and tables of associated data sources according to claim 1 is characterized by: In S1, the data quality detection indicator configuration includes configuration operations of naming and indicator writing for the description table quality detection indicator and the field quality detection indicator; The naming of table quality detection indicators includes: "whether it is a dimension table", "whether there is associated data", and "whether the data table name conforms to the specification"; Among them, "Is it a dimension table" needs to detect whether the fields in a table are in a one-to-one relationship; "Is there related data" needs to detect whether there are fields in the table that can be used for cross-table association analysis. The fields that need to be used for data association in the data quality inspection table should be set in advance; The names of field quality inspection indicators include: "Duplicate value check", "Existence of null values", "Whether field naming complies with the specification", and "Value range check"; Among them, "Value range check" is to detect the fields that use code values ​​instead of actual values. If there are abnormal values ​​that do not belong to the value range, a rectification task is generated to let the data source unit confirm the meaning of the abnormal value and add it to the code table; Update and maintain data quality detection indicators based on the two dimensions of tables and fields, set the quality indicators corresponding to each indicator, and write the quality indicators into the background; set the task attributes corresponding to each data quality detection indicator, and finally generate corresponding data quality operation and maintenance tasks and data quality rectification tasks based on the detection results.

3. The data quality detection method for cross-data layer libraries and tables of associated data sources according to claim 1 is characterized by: The names of the quality inspection data layers include data detail layer DWD, data middle layer DWM, data service layer DWS, and data application layer ADS.

4. The data quality detection method for cross-data layer libraries and tables of associated data sources according to claim 1 is characterized by: In the single batch data quality inspection task, when S2 configures the weight of the table quality inspection index for the general quality inspection table, the general quality inspection table configuration table for table quality inspection indexes Q1 to Q N Assign corresponding weights W1 to W N ,Field quality inspection indicators are applied multiple times to different selected fields.,In one task, the general quality inspection table performs quality inspection based on,fields 1 to N.

5. The data quality detection method for cross-data layer libraries and tables of associated data sources according to claim 1 is characterized by: In S3, the data quality detection report includes the quality detection number, detection time, detection data layer, detection database, detailed information on the data fields involved and the name of the data source unit; the data quality detection report lists all the data quality indicators detected, and calculates the single data quality indicator detection score based on the arithmetic average of each data quality indicator. The final total score of the data quality detection report is calculated by the weighted average of each indicator.

6. A data quality detection system for cross-data layer libraries and tables of associated data sources, characterized in that: include: Data quality detection index configuration module: used to set table quality detection index and field quality detection index, and perform table quality detection index and field quality detection index verification; Data quality inspection index management module: used to initiate a single batch data quality inspection task, extract and select different data layers from multiple data sources to associate tables and fields in the tables; configure table quality inspection index weights for the general quality inspection table, and then configure different weights of field quality inspection indicators for all selected fields in the general quality inspection table; In a single batch data quality inspection task, all selected fields in the general quality inspection table are tested for field quality inspection index Q A When configuring the weight of the field quality detection indicator Q A The configuration method used for testing in multiple different fields is: Field quality detection indicator Q A The weight of a field that appears in multiple fields is the sum of the weights of all occurrences, W A =W A1 W A2 W A3… W Ak... W An ; W A The specific calculation formula is: A = , m is the sum of the number of weight settings when a single field quality detection indicator is applied to multiple different fields for detection; The same field quality detection index is applied to different fields and different weights are assigned. Any field quality detection index Q A appearing in a field has a weight of W An = , W An is a decimal, 0 < W An < 1. In a single batch data quality detection task, n is the sum of the number of field quality detection indexes applied to each field, X is the adjustment interval of the weight value of each field quality detection index, -1 < X < 1; for the weight setting of multiple field quality detection indexes configured for the same field, the sum of the weights of all field quality detection indexes satisfies 1; When initiating a single batch data quality inspection task, multiple quality inspection data layers are selected, and multiple databases and data tables are simultaneously selected to correspond to each data layer. In each data layer, the selected multiple data tables are merged through data mapping rules to form a general quality inspection table, and the general quality inspection table exists in the selected multiple quality inspection data layers; then the single quality inspection task performs quality inspection on the same general quality inspection table data stored in different layers, and each layer is configured with different table quality inspection indicators and field quality inspection indicators, and a weight is set for each quality inspection indicator; Data quality detection module: used to set data quality detection scheduled tasks, perform batch data quality detection tasks according to the set scheduled tasks, and obtain data quality detection reports.

7. The data quality detection system for cross-data layer libraries and tables of associated data sources according to claim 6 is characterized in that: The score of the data quality report is G= ; in, Indicates the quality detection index of a single field in batch detection score; Indicates the quality detection index of a single field in batch detection The score, Indicates the quality detection index of a single field in batch detection where N is the sum of the number of field quality detection indicators applied in a batch data quality detection task.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images