A method and apparatus for enterprise data quality assessment

Abstract

The application relates to the technical field of data processing, in particular to an enterprise data quality evaluation method and device. The method comprises the following steps: acquiring a current data quality check rule table; checking current business data of an enterprise based on the current data quality check rule table to generate a current business data check result; acquiring a historical business data check result of the enterprise, and combining the two to generate a business data quality change chart of the enterprise. Current metadata information and historical metadata information of each business system under the enterprise are acquired, the two are compared, table change information and field change information in each business system are determined, current metadata information of a data center is acquired, the current metadata information of each business system is compared with the current metadata information of the data center, metadata information differences between each business system and the data center are determined, and finally a metadata quality monitoring chart of the enterprise is generated.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and device for assessing enterprise data quality. Background Technology

[0002] In today's business environment, enterprise data is growing rapidly. This is especially true for large corporations, which may have multiple subsidiaries, each using different business systems and databases. This results in inconsistent data formats and numerous data issues, posing significant challenges to data management and quality.

[0003] In current technologies, enterprises typically use manual checks and interventions to address data quality issues. However, with the rapid growth of data volume and the diversification of data sources, traditional methods are not only wasteful of manpower but also inefficient and can no longer meet the demands. Summary of the Invention

[0004] To at least partially overcome the problems of wasteful and inefficient manual data quality processing in related technologies, this application provides a method and device for enterprise data quality assessment.

[0005] The proposed solution is as follows:

[0006] According to a first aspect of the embodiments of this application, a method for assessing enterprise data quality is provided, comprising:

[0007] Retrieve the current data quality verification rule table;

[0008] The current business data of the enterprise is verified based on the current data quality verification rule table, and the current business data verification result is generated; the current business data includes the current table data and current field data of the various business systems under the enterprise.

[0009] Obtain the historical business data verification results of the enterprise, and combine the current business data verification results with the historical business data verification results to generate a chart showing the changes in the enterprise's business data quality.

[0010] Obtain the current and historical metadata information of each business system under the enterprise, compare the current metadata information with the historical metadata information, and determine the table change information and field change information in each business system;

[0011] Obtain the current metadata information of the data platform, compare the current metadata information of each business system with the current metadata information of the data platform, and determine the differences between the metadata information of each business system and the data platform.

[0012] Based on the table and field change information in each business system, as well as the differences in metadata information between each business system and the data platform, a metadata quality monitoring chart for the enterprise is generated.

[0013] Preferably, the current business data of the enterprise is validated based on the current data quality validation rule table to generate the current business data validation result, including:

[0014] Based on the verification types configured in the current data quality verification rule table, the current field data is verified sequentially to obtain the score of the current field data under different verification types;

[0015] The scores of the current field data under different validation types are averaged and aggregated to obtain the comprehensive score of the current field data.

[0016] The average score of the current table data is obtained by aggregating the comprehensive scores of all data in the current field.

[0017] Based on the pre-configured weights of each table in the business system, the scores of all current table data in the business system are aggregated by weighted average to obtain the score of the business system.

[0018] Based on the pre-configured weights of each business system under the enterprise, the scores of all business systems are aggregated by weighted average to obtain the current business data quality score of the enterprise.

[0019] Preferably, the verification type includes at least:

[0020] Null value rate validation, enumerated value validation, value type validation, threshold validation, volatility validation, unique value validation, and custom validation;

[0021] The null value rate check is used to verify whether the ratio of the number of null value data in table data or field data to the total amount of data is not less than a preset null value rate threshold.

[0022] The enumeration value validation is used to verify whether the table data or field data is a preset enumeration value;

[0023] The value type validation is used to verify whether table data or field data conforms to a preset value type;

[0024] The threshold check is used to verify whether the actual value of the table data or field data is within the upper or lower range of the preset data volume value.

[0025] The volatility check is used to verify whether the fluctuation between the actual value of the table data or field data and the average value within the period exceeds a preset volatility value.

[0026] The unique value check is used to verify whether table data or field data is unique.

[0027] The custom validation is used to verify whether table data or field data conforms to custom validation rules.

[0028] Preferably, the method further includes:

[0029] Issue alerts when the company's current business data is abnormal or does not conform to the rule definition;

[0030] Issue alerts when table and field changes are detected in the enterprise's metadata;

[0031] The following situations exist where a company's current business data does not conform to the rule definition:

[0032] The ratio of null values ​​in table data or field data to the total data volume is not less than the preset null value rate threshold;

[0033] The table data or field data is not a preset enumerated value;

[0034] The table data or field data does not conform to the preset value type;

[0035] The actual value of the table data or field data is not within the preset data volume range;

[0036] The actual value of the table data or field data fluctuates more than the preset fluctuation value compared to the average value within the period;

[0037] Duplicate values ​​exist in table data or field data;

[0038] The table data or field data does not conform to the custom validation rules.

[0039] Preferably, the business data quality change chart includes:

[0040] Verification time, verified business system, verified table data, verified field data, overall trend of business data quality within the period, trend of business data quality under each verification type within the period, verification results of each verification type, report ranking of the number of alarms within the period, and field ranking of the number of alarms within the period.

[0041] The verification results for each type of verification include: the proportion of business data that does not conform to the rule definition, and a comparison between the current business data quality score and the average business data quality score within the period.

[0042] Preferably, the metadata quality monitoring chart includes:

[0043] Differences between table change information and field change information in each business system and metadata information between each business system and the data platform;

[0044] The table change information in each business system includes: the total number of tables that have been changed, the number of tables under each change type, the change type of the table, the name of the changed table, the business system to which the changed table belongs, and the database type and database name to which the changed table belongs.

[0045] The field change information in each business system includes: the total number of fields that have changed, the number of fields under each change type, the change type of the field, the content of the changed field, the table to which the changed field belongs, the business system to which the changed field belongs, and the database type and database name to which the changed field belongs;

[0046] The differences in metadata information between various business systems and the data platform include: table information not uploaded and field information not uploaded;

[0047] The information on tables that were not uploaded includes: the total number of tables that were not uploaded, the type of tables that were not uploaded, the table name of the tables that were not uploaded, the business system to which the tables belonged, and the database type and database name to which the tables belonged.

[0048] The information on fields not uploaded includes: the total number of fields not uploaded, the type of fields not uploaded, the content of the fields not uploaded, the table to which the fields not uploaded belong, the business system to which the fields not uploaded belong, and the database type and database name to which the fields not uploaded belong.

[0049] Preferably, before obtaining the current data quality verification rule table, the method further includes:

[0050] Determine the current scheduling mode;

[0051] The scheduling modes include: single scheduling mode, batch scheduling mode, specified scheduling mode, and abnormal rescheduling mode;

[0052] The single-execution scheduling mode is used to execute high-priority verification tasks individually.

[0053] The batch scheduling mode is used to execute all verification tasks in batches;

[0054] The specified scheduling mode is used to execute the specified verification task;

[0055] The abnormal re-execution mode is used to re-execute the verification task that has an abnormality.

[0056] Preferably, the current data quality verification rule table is obtained, including:

[0057] Obtain the information of the enterprise to be verified and the verification rules corresponding to each verification type; the information of the enterprise to be verified includes: information of each business system under the enterprise, current table data and current field data of each business system, table weight level and business system weight level;

[0058] Clear the historical data quality verification rule table to obtain a blank template for data quality verification rules;

[0059] The blank template for data quality verification rules is split into a main template for data quality verification rules and multiple sub-templates for data quality verification rules; each sub-template for data quality verification rules corresponds one-to-one with each verification type.

[0060] Enter the enterprise information to be verified, the verification rules corresponding to each verification type, and the current scheduling mode into the split data quality verification rule blank template to obtain the current data quality verification rule table.

[0061] The current data quality verification rule table is reviewed, and if the review is passed, the current data quality verification rule table is archived to the database.

[0062] Preferably, the system monitors whether the verification task is executed normally and issues an alarm when an abnormality occurs in the verification task.

[0063] According to a second aspect of the embodiments of this application, an enterprise data quality assessment device is provided, comprising:

[0064] Processor and memory;

[0065] The processor and memory are connected via a communication bus:

[0066] The processor is used to call and execute the program stored in the memory;

[0067] The memory is used to store a program, which is at least used to execute an enterprise data quality assessment method as described in any of the above.

[0068] The technical solution provided in this application may include the following beneficial effects: The enterprise data quality assessment method in this application includes: obtaining a current data quality verification rule table; verifying the enterprise's current business data based on the current data quality verification rule table to generate current business data verification results; the current business data includes current table data and current field data in various business systems under the enterprise; obtaining the enterprise's historical business data verification results, and generating a business data quality change chart of the enterprise by combining the current business data verification results and historical business data verification results; obtaining the current metadata information and historical metadata information of various business systems under the enterprise, comparing the current metadata information with the historical metadata information to determine the table change information and field change information in each business system; obtaining the current metadata information of the data platform, comparing the current metadata information of each business system with the current metadata information of the data platform to determine the differences in metadata information between each business system and the data platform; and generating a metadata quality monitoring chart of the enterprise based on the table change information and field change information in each business system, as well as the differences in metadata information between each business system and the data platform. The technical solution in this application can efficiently and automatically monitor the quality of an enterprise's business data and metadata according to preset rules, and display the data in chart form, quantifying and visualizing data quality issues. Enterprises can more intuitively identify data quality problems through charts, thereby improving overall data quality and providing strong support for enterprise digital operations and transformation.

[0069] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0071] Figure 1 This is a flowchart illustrating an enterprise data quality assessment method provided in one embodiment of this application;

[0072] Figure 2 This is a schematic diagram of the structure of an enterprise data quality assessment device provided in one embodiment of this application.

[0073] Reference numerals: Processor-21; Memory-22. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] Figure 1 This is a flowchart illustrating an enterprise data quality assessment method according to an embodiment of this application, with reference to... Figure 1 A method for assessing enterprise data quality includes:

[0076] S11: Get the current data quality verification rule table;

[0077] S12: Verify the company's current business data based on the current data quality verification rule table, and generate the current business data verification result; the current business data includes the current table data and current field data in the various business systems under the company;

[0078] S13: Obtain the historical business data verification results of the enterprise, and generate a chart of changes in the enterprise's business data quality by combining the current business data verification results and the historical business data verification results.

[0079] S14: Obtain the current and historical metadata information of each business system under the enterprise, compare the current metadata information with the historical metadata information, and determine the table change information and field change information in each business system;

[0080] S15: Obtain the current metadata information of the data platform, compare the current metadata information of each business system with the current metadata information of the data platform, and determine the differences between the metadata information of each business system and the data platform.

[0081] S16: Generate metadata quality monitoring charts for the enterprise based on table and field change information in each business system, as well as the differences in metadata information between each business system and the data platform.

[0082] It should be noted that in this embodiment, "current" refers to the current day.

[0083] It should be noted that before obtaining the current data quality verification rule table, the method also includes:

[0084] Determine the current scheduling mode;

[0085] The scheduling modes include: single scheduling mode, batch scheduling mode, specified scheduling mode, and abnormal rescheduling mode;

[0086] The single-execution scheduling mode is used to execute high-priority verification tasks individually.

[0087] Batch scheduling mode is used to execute all verification tasks in batches;

[0088] Specifying a scheduling mode is used to execute the specified verification task;

[0089] The exception re-execution mode is used to re-execute verification tasks that have encountered exceptions.

[0090] It should be noted that this embodiment supports four scheduling modes: single scheduling mode, batch scheduling mode, specified scheduling mode, and abnormal rescheduling mode.

[0091] The single-execution scheduling mode is used to execute high-priority verification tasks individually, which is more timely. When an abnormality occurs in the object being verified individually, an alarm can be triggered immediately.

[0092] Batch scheduling mode is used to execute all verification tasks in batches. For tasks with low priority, a unified scheduling time is set after the verification tasks with higher priority are completed on the same day.

[0093] The specified scheduling mode is used to execute the specified verification task, and it supports manual scheduling of any table.

[0094] The anomaly re-execution mode is used to re-execute verification tasks that have anomalies. It supports batch re-execution for tasks that are anomalies of the day, rule configuration errors, system execution errors, or are marked as needing re-execution.

[0095] It should be noted that retrieving the current data quality verification rule table includes:

[0096] Obtain the information of the enterprise to be verified and the verification rules corresponding to each verification type; the information of the enterprise to be verified includes: information of each business system under the enterprise, current table data and current field data in each business system, table weight level and business system weight level;

[0097] Clear the historical data quality verification rule table to obtain a blank template for data quality verification rules;

[0098] The blank template for data quality verification rules is split into a main template for data quality verification rules and multiple sub-templates for data quality verification rules; each sub-template for data quality verification rules corresponds one-to-one with each verification type.

[0099] Enter the information of the enterprise to be verified, the verification rules corresponding to each verification type, and the current scheduling mode into the split blank template of data quality verification rules to obtain the current data quality verification rule table.

[0100] The current data quality verification rule table is reviewed, and if the review is passed, the current data quality verification rule table is archived to the database.

[0101] It should be noted that, since the verification rules are constantly being optimized, the data quality verification rule table in this embodiment needs to be updated daily. When the system starts, the historical data quality verification rule table is cleared first to obtain a blank template for data quality verification rules.

[0102] Then, by entering the blank template of the data quality verification rules based on the obtained enterprise information to be verified, the current data quality verification rule table can be obtained.

[0103] In practice, the blank template for data quality verification rules is in Excel spreadsheet format. In this embodiment, information about the companies to be verified and the current scheduling mode can be entered in batches through an Excel spreadsheet for subsequent batch maintenance.

[0104] It should be noted that the main template for the data quality verification rules for the enterprise information to be verified and the current scheduling mode is used, while the sub-templates for the verification rules for each verification type are used.

[0105] It should be noted that, in this embodiment, the audit content for the current data quality verification rule table is whether the current data quality verification rule table has empty values ​​and whether the filled-in content is a preset value.

[0106] It should be noted that in this embodiment, the current data quality verification rule table is archived to the database upon approval for subsequent reverse lookups. The database stores the data quality verification rule table generated daily.

[0107] It should be noted that this embodiment supports administrators in modifying the verification rules. Administrators can modify individual monitoring rules through the front-end interface to facilitate subsequent monitoring rule maintenance.

[0108] It should be noted that the current business data of the enterprise is validated based on the current data quality validation rule table, and the current business data validation results are generated, including:

[0109] Based on the verification types configured in the current data quality verification rule table, the current field data is verified sequentially to obtain the score of the current field data under different verification types;

[0110] The scores of the current field data under different validation types are averaged and aggregated to obtain the comprehensive score of the current field data.

[0111] The average score of the current table data is obtained by aggregating the comprehensive scores of all data in the current field.

[0112] Based on the pre-configured weights of each table in the business system, the scores of all current table data in the business system are aggregated by weighted average to obtain the score of the business system.

[0113] Based on the pre-configured weights of each business system under the enterprise, the scores of all business systems are aggregated by weighted average to obtain the current business data quality score of the enterprise.

[0114] It should be noted that the calculation logic for the business data quality score is as follows:

[0115] Step 1: For each field of data, score it based on each type of validation. For example, for null value validation, if there are 10 rows of null values ​​and a total of 100 rows, the score is 90%.

[0116] Step 2: Average and aggregate the scores of the current field data under different validation types to obtain the comprehensive score of the current field data.

[0117] Step 3: Aggregate the overall scores of all data in the current field of the current table to obtain the score of the current table data.

[0118] Step 4: In a single business system, determine the weight of the tables. For example, if table A has an importance of 1, table B has an importance of 3, and table C has an importance of 3, then the weight of table A = 1 / (1+3+3), the weight of table B = 3 / (1+3+3), and the weight of table C = 3 / (1+3+3).

[0119] Step 5: Based on the pre-configured weights of each table in the business system, perform a weighted average aggregation of the scores of all current table data in the business system to obtain the score of the business system.

[0120] Step 6: Based on the pre-configured weights of each business system under the enterprise, the scores of all business systems are weighted and averaged to obtain the current business data quality score of the enterprise.

[0121] The validation types should include at least:

[0122] Null value rate validation, enumerated value validation, value type validation, threshold validation, volatility validation, unique value validation, and custom validation;

[0123] The null value rate check is used to verify whether the ratio of the number of null values ​​in table data or field data to the total amount of data is not less than a preset null value rate threshold.

[0124] Enumeration value validation is used to verify whether table data or field data is a preset enumeration value;

[0125] Value type validation is used to verify whether table data or field data conforms to a preset value type.

[0126] Threshold validation is used to verify whether the actual value of table data or field data is within the upper or lower range of a preset data volume value.

[0127] Volatility check is used to verify whether the fluctuation of the actual value of table data or field data exceeds the average value within a period.

[0128] Unique value validation is used to verify whether table data or field data is unique.

[0129] Custom validation is used to verify whether table data or field data conforms to custom validation rules.

[0130] It should be noted that the method also includes:

[0131] Issue alerts when the company's current business data is abnormal or does not conform to the rule definition;

[0132] Issue alerts when table and field changes are detected in the enterprise's metadata;

[0133] The following situations exist where a company's current business data does not conform to the rule definition:

[0134] The ratio of null values ​​in table data or field data to the total data volume is not less than the preset null value rate threshold;

[0135] The table data or field data is not a preset enumerated value;

[0136] The table data or field data does not conform to the preset value type;

[0137] The actual value of the table data or field data is not within the preset data volume range;

[0138] The actual value of the table data or field data fluctuates more than the preset fluctuation value compared to the average value within the period;

[0139] Duplicate values ​​exist in table data or field data;

[0140] The table data or field data does not conform to the custom validation rules.

[0141] It should be noted that the configurable parameters in the null value rate validation include: table, field, and preset null value rate threshold.

[0142] The calculation logic for enumeration value validation is data not in (enumeration value 1, enumeration value 2, enumeration value 3), where configurable parameters include: table, field, and preset enumeration value.

[0143] The calculation logic for value type validation is data<>valueType, where configurable parameters include: table, field, and preset value type (regular expressions are supported).

[0144] In practice, the regular expressions corresponding to commonly used value type validations and commonly used aggregation methods can be maintained in the system in advance for direct use later.

[0145] The threshold validation calculation logic is that the data is within the range of [upper limit value, lower limit value]; the configurable parameters include: table, field, upper limit value, lower limit value, aggregate function (avg, sum, count, count distinct).

[0146] This embodiment also supports threshold verification for different aggregation methods, such as threshold verification for each row, threshold verification after SUM, and threshold verification after COUNT, so as to flexibly monitor data quality.

[0147] The calculation logic for volatility verification is |(current actual value - avg(comparison time period))| / avg(comparison time period), where configurable parameters include: table, field, comparison time period, and aggregate functions (avg, sum, count, countdistinct).

[0148] Volatility verification supports setting the comparison time period, such as comparing the average of the previous 30 / 20 / 10 days.

[0149] The volatility verification in this embodiment supports different aggregation methods, such as SUM aggregation followed by volatility calculation, COUNT aggregation followed by volatility calculation, etc., so as to flexibly monitor data quality.

[0150] The unique value validation calculation logic is count distinct, and the configurable parameters include: table and field.

[0151] Custom validation allows you to define your own validation rules and SQL (Structured Query Language, database language). The SQL must return TRUE or FALSE.

[0152] Based on the above, the scoring logic for each verification type is as follows:

[0153] Null value rate check: 1 - number of rows with null values ​​ / total number of rows;

[0154] Enumeration value validation: 1 - number of rows with invalid enumeration values ​​ / total number of rows;

[0155] Value type validation: 1 - number of rows with invalid value types / total number of rows;

[0156] Custom validation: If it returns 1, you get 100 points; if it returns 0, you get 0 points.

[0157] Threshold validation: If no aggregation function is used, i.e., threshold validation is performed for each row, then the score is 1 - number of rows exceeding the threshold / total number of rows; if an aggregation function is used, and the aggregation result exceeds the threshold, then the score is 0; if it is within the threshold range, then the score is 100.

[0158] Volatility verification: Scoring is conducted across the following volatility ranges:

[0159] Volatility 0%, 100 points;

[0160] Volatility 0%-30% (inclusive), 70 points;

[0161] Volatility 30%-60% (inclusive), 50 points;

[0162] Volatility 60%-100% (inclusive), 30 points;

[0163] Volatility > 100%, 0 points.

[0164] It should be noted that the charts showing changes in business data quality include:

[0165] Verification time, verified business system, verified table data, verified field data, overall trend of business data quality within the period, trend of business data quality under each verification type within the period, verification results of each verification type, report ranking of the number of alarms within the period, and field ranking of the number of alarms within the period.

[0166] The verification results for each type of verification include: the proportion of business data that does not conform to the rule definition, and a comparison between the current business data quality score and the average business data quality score within the period.

[0167] It should be noted that the overall trend of business data quality within the period, and the trend of business data quality under each verification type within the period, are displayed as line graphs. The remaining verification results are displayed as charts.

[0168] In practice, the top 20 reports and fields showing the number of alarms within a period are displayed.

[0169] It should be noted that the validation results for all validation types include: the proportion of business data that currently does not conform to the rule definition, and a comparison between the current business data quality score and the average business data quality score within the period. This provides a clear indication of whether the field data quality has improved.

[0170] It should be noted that the metadata quality monitoring charts include:

[0171] Differences between table change information and field change information in each business system and metadata information between each business system and the data platform;

[0172] The table change information in each business system includes: the total number of tables that have been changed, the number of tables under each change type, the change type of the table, the name of the changed table, the business system to which the changed table belongs, and the database type and database name to which the changed table belongs.

[0173] The field change information in each business system includes: the total number of fields that have changed, the number of fields under each change type, the change type of the field, the content of the changed field, the table to which the changed field belongs, the business system to which the changed field belongs, and the database type and database name to which the changed field belongs;

[0174] The differences in metadata information between various business systems and the data platform include: table information not uploaded and field information not uploaded;

[0175] Information on tables that were not uploaded includes: the total number of tables that were not uploaded, the type of tables that were not uploaded, the table name of the tables that were not uploaded, the business system to which the tables belonged, and the database type and database name to which the tables belonged.

[0176] The information for fields that were not uploaded includes: the total number of fields that were not uploaded, the type of fields that were not uploaded, the content of the fields that were not uploaded, the table to which the fields belonged, the business system to which the fields belonged, and the database type and database name to which the fields belonged.

[0177] It should be noted that the metadata quality monitoring chart includes two parts: table change information and field change information in each business system, and the differences in metadata information between each business system and the data platform.

[0178] The table change types include additions and deletions.

[0179] Number of new tables: The number of new tables added today compared to yesterday within the time frame;

[0180] Deleted Tables: The number of tables deleted today compared to yesterday within the time range.

[0181] The change types for fields include addition, deletion, and modification.

[0182] Number of newly added fields: The number of fields added today compared to yesterday within the time range;

[0183] Deleted Fields: The number of fields deleted today compared to yesterday within the time range;

[0184] Number of changed fields: The number of fields that changed today compared to yesterday within the time frame.

[0185] The total number of tables that were not uploaded refers to the number of tables that existed in the source system but did not exist in the data platform within the specified time frame.

[0186] The total number of fields not uploaded refers to the number of fields that exist in the source system but not in the data platform within a given time frame.

[0187] It should be noted that in cases of changes to tables / fields in the source system, such as adding or deleting tables, the fields should be left blank. Similarly, in cases where tables / fields have not been uploaded, the fields should be left blank if any tables have not been uploaded.

[0188] It should be noted that this embodiment also monitors whether the verification task is executed normally, and issues an alarm when the verification task is abnormal.

[0189] It should be noted that the alarm email address in this embodiment can be uniformly set to the administrator's email address.

[0190] It is understood that the enterprise data quality assessment method in this embodiment includes: obtaining a current data quality verification rule table; verifying the enterprise's current business data based on the current data quality verification rule table to generate current business data verification results; the current business data includes current table data and current field data in various business systems under the enterprise; obtaining the enterprise's historical business data verification results, and generating a business data quality change chart by combining the current business data verification results and historical business data verification results; obtaining the current metadata information and historical metadata information of various business systems under the enterprise, comparing the current metadata information with the historical metadata information to determine the table change information and field change information in each business system; obtaining the current metadata information of the data platform, comparing the current metadata information of each business system with the current metadata information of the data platform to determine the differences in metadata information between each business system and the data platform; and generating a metadata quality monitoring chart for the enterprise based on the table change information and field change information in each business system, as well as the differences in metadata information between each business system and the data platform. The technical solution in this embodiment can efficiently and automatically monitor the enterprise's business data quality and metadata quality according to preset rules, and display the data in chart form, making data quality issues quantifiable and visual. Enterprises can use charts to more intuitively identify data quality issues, thereby improving overall data quality and providing strong support for enterprise digital operations and transformation.

[0191] Example 2

[0192] Figure 2 This is a schematic diagram of the structure of an enterprise data quality assessment device provided in one embodiment of this application, with reference to... Figure 2 An enterprise data quality assessment device, comprising:

[0193] Processor 21 and memory 22;

[0194] Processor 21 and memory 22 are connected via a communication bus:

[0195] The processor 21 is used to call and execute the program stored in the memory 22;

[0196] The memory 22 is used to store a program, which is used to execute at least one of the enterprise data quality assessment methods in the above embodiments.

[0197] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0198] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0199] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0200] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0201] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0203] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0204] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0205] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for assessing enterprise data quality, characterized in that, include: Retrieve the current data quality verification rule table; The current business data of the enterprise is verified based on the current data quality verification rule table, and the current business data verification result is generated. The current business data includes the current table data and current field data of various business systems under the enterprise; Obtain the historical business data verification results of the enterprise, and combine the current business data verification results with the historical business data verification results to generate a chart showing the changes in the enterprise's business data quality. Obtain the current and historical metadata information of each business system under the enterprise, compare the current metadata information with the historical metadata information, and determine the table change information and field change information in each business system; Obtain the current metadata information of the data platform, compare the current metadata information of each business system with the current metadata information of the data platform, and determine the differences between the metadata information of each business system and the data platform. Based on the table and field change information in each business system, as well as the differences in metadata information between each business system and the data platform, generate metadata quality monitoring charts for the enterprise. Based on the current data quality verification rule table, the enterprise's current business data is verified, and the current business data verification results are generated, including: Based on the verification types configured in the current data quality verification rule table, the current field data is verified sequentially to obtain the score of the current field data under different verification types; The scores of the current field data under different validation types are averaged and aggregated to obtain the comprehensive score of the current field data. The average score of the current table data is obtained by aggregating the comprehensive scores of all data in the current field. Based on the pre-configured weights of each table in the business system, the scores of all current table data in the business system are aggregated by weighted average to obtain the score of the business system. Based on the pre-configured weights of each business system under the enterprise, the scores of all business systems are aggregated by weighted average to obtain the current business data quality score of the enterprise.

2. The method according to claim 1, characterized in that, The verification types include at least: Null value rate validation, enumerated value validation, value type validation, threshold validation, volatility validation, unique value validation, and custom validation; The null value rate check is used to verify whether the ratio of the number of null values ​​in table data or field data to the total amount of data is not less than a preset null value rate threshold. The enumeration value validation is used to verify whether the table data or field data is a preset enumeration value; The value type validation is used to verify whether table data or field data conforms to a preset value type; The threshold check is used to verify whether the actual value of the table data or field data is within the upper or lower range of the preset data volume value. The volatility check is used to verify whether the fluctuation between the actual value of the table data or field data and the average value within the period exceeds a preset volatility value. The unique value check is used to verify whether table data or field data is unique. The custom validation is used to verify whether table data or field data conforms to custom validation rules.

3. The method according to claim 2, characterized in that, The method further includes: Issue alerts when the company's current business data is abnormal or does not conform to the rule definition; Issue alerts when table and field changes are detected in the enterprise's metadata; The following situations exist where a company's current business data does not conform to the rule definition: The ratio of null values ​​in table data or field data to the total data volume is not less than the preset null value rate threshold; The table data or field data is not a preset enumerated value; The table data or field data does not conform to the preset value type; The actual value of the table data or field data is not within the preset data volume range; The actual value of the table data or field data fluctuates more than the preset fluctuation value compared to the average value within the period; Duplicate values ​​exist in table data or field data; The table data or field data does not conform to the custom validation rules.

4. The method according to claim 3, characterized in that, The charts showing changes in the quality of the business data include: Verification time, verified business system, verified table data, verified field data, overall trend of business data quality within the period, trend of business data quality under each verification type within the period, verification results of each verification type, report ranking of the number of alarms within the period, and field ranking of the number of alarms within the period. The verification results for each type of verification include: the proportion of business data that does not conform to the rule definition, and a comparison between the current business data quality score and the average business data quality score within the period.

5. The method according to claim 1, characterized in that, The metadata quality monitoring chart includes: Differences between table change information and field change information in each business system and metadata information between each business system and the data platform; The table change information in each business system includes: the total number of tables that have been changed, the number of tables under each change type, the change type of the table, the name of the changed table, the business system to which the changed table belongs, and the database type and database name to which the changed table belongs. The field change information in each business system includes: the total number of fields that have changed, the number of fields under each change type, the change type of the field, the content of the changed field, the table to which the changed field belongs, the business system to which the changed field belongs, and the database type and database name to which the changed field belongs; The differences in metadata information between various business systems and the data platform include: table information not uploaded and field information not uploaded; The information on tables that were not uploaded includes: the total number of tables that were not uploaded, the type of tables that were not uploaded, the table name of the tables that were not uploaded, the business system to which the tables belonged, and the database type and database name to which the tables belonged. The information on fields not uploaded includes: the total number of fields not uploaded, the type of fields not uploaded, the content of the fields not uploaded, the table to which the fields not uploaded belong, the business system to which the fields not uploaded belong, and the database type and database name to which the fields not uploaded belong.

6. The method according to claim 1, characterized in that, Before obtaining the current data quality verification rule table, the method further includes: Determine the current scheduling mode; The scheduling modes include: single scheduling mode, batch scheduling mode, specified scheduling mode, and abnormal rescheduling mode; The single-execution scheduling mode is used to execute high-priority verification tasks individually. The batch scheduling mode is used to execute all verification tasks in batches; The specified scheduling mode is used to execute the specified verification task; The abnormal re-execution mode is used to re-execute the verification task that has an abnormality.

7. The method according to claim 6, characterized in that, Retrieve the current data quality verification rule table, including: Obtain the information of the enterprise to be verified and the verification rules corresponding to each verification type; the information of the enterprise to be verified includes: information of each business system under the enterprise, current table data and current field data of each business system, table weight level and business system weight level; Clear the historical data quality verification rule table to obtain a blank template for data quality verification rules; The blank template for data quality verification rules is split into a main template for data quality verification rules and multiple sub-templates for data quality verification rules; each sub-template for data quality verification rules corresponds one-to-one with each verification type. Enter the enterprise information to be verified, the verification rules corresponding to each verification type, and the current scheduling mode into the split data quality verification rule blank template to obtain the current data quality verification rule table. The current data quality verification rule table is reviewed, and if the review is passed, the current data quality verification rule table is archived to the database.

8. The method according to claim 1, characterized in that, Monitor whether the verification task is executed normally, and issue an alarm when the verification task is abnormal.

9. A device for assessing enterprise data quality, characterized in that, include: Processor and memory; The processor and memory are connected via a communication bus: The processor is used to call and execute the program stored in the memory; The memory is used to store a program, which is at least used to execute an enterprise data quality assessment method according to any one of claims 1-8.

Images