Data quality supervision method

Abstract

The invention belongs to the technical field of data supervision, and discloses a data quality supervision method, which specifically comprises the following steps of: 1, creating a quality rule for checking a data table and a field according to a service quality rule through a unified scheduling platform; according to the method, the quality rules for verifying the data tables and the fields are created through the unified scheduling platform, flexible creation and automatic verification of the service quality rules are achieved, and therefore the data quality is improved, the automation level of data processing is enhanced, secondly, the task execution condition is monitored in real time through SLA monitoring, it is ensured that the data processing task is ready on time, and the data processing efficiency is improved. According to the method and the system, the business continuity is enhanced, then the table information, the field information and the index caliber are comprehensively managed through the metadata management platform, the data structure and meaning can be more accurately understood, so that the analysis efficiency and accuracy are improved, and finally, the data development efficiency is improved by establishing a perfect data blood relationship analysis function.

Description

Technical Field

[0001] The present invention belongs to the technical field of data supervision, and specifically relates to a data quality supervision method. Background Art

[0002] With the continuous development and popularization of information technology, data has become an indispensable and important resource in modern society. In organizations, data is widely used in various fields such as decision-making, business process optimization, and customer relationship management. However, with the continuous increase in the amount of data, the quality of data has become a highly concerned issue.

[0003] Data quality issues include data accuracy, consistency, completeness, reliability, etc., which may lead to inaccurate decisions, inefficient business processes and customer dissatisfaction. Especially in the fields of finance, medicine, law, etc., poor data quality may lead to significant risks and legal liabilities. Therefore, data quality governance supervision becomes crucial. Regulatory authorities have formulated regulations and standards for data quality governance to supervise the collection, storage, processing and transmission of data to ensure the high quality and security of data, which not only helps to protect the public interest, but also helps to maintain fair competition in the market and the reputation of enterprises. In addition, with the continuous increase in global data exchanges, cross-border data flows have also become an important issue. International organizations and agreements such as the EU’s The General Data Protection Regulation (GDPR) and other laws provide guiding principles for cross-border data transmission and stipulate minimum standards for data protection. In summary, data quality governance and supervision play a key role in the current digital era. It is not only related to the quality and security of data, but also has a profound impact on the economic, social and legal fields. The data quality supervision methods in the existing technology generally have the guarantee of data SLA ready time and the guarantee of data execution order accuracy by relying on scheduling. However, of these two methods, the former cannot guarantee the accuracy of data statistical values, and the latter has different data caliber specifications and is more difficult to manage. Therefore, it is necessary to take active measures to ensure the effective implementation and compliance of data quality governance and supervision to cope with the increasingly complex data environment and challenges. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a data quality supervision method, which has the advantage of improving data management quality.

[0005] To achieve the above purpose, the present invention provides the following technical solution: a data quality supervision method, the specific steps are as follows:

[0006] Step 1: Business Quality Rules

[0007] Through the unified scheduling platform, quality rules for verifying data tables and fields are created. When the quality rules are triggered, the system will send an alarm message to detect and handle exceptions in a timely manner.

[0008] Step 2: SLA Monitoring

[0009] Configure SLA monitoring to ensure that tasks are ready on time. If a task is not ready on time, the system will send an alarm so that the exception can be handled in time.

[0010] Step 3: Metadata Management

[0011] Through the metadata management platform, table information, field information and indicator caliber can be managed, which is conducive to data asset implementation and data analysis;

[0012] Step 4: Bloodline Analysis

[0013] Establishing a comprehensive data lineage analysis function can track the data flow path, improve data development efficiency and data flow query capabilities.

[0014] Preferably, in step 1, ensure that all newly created tables follow a unified naming convention, such as prefixes representing data hierarchies ODS, DIM, DWD, DWA, APP, etc., and table names are lowercase and start with a letter, and do not contain special characters except underscores.

[0015] Preferably, the metadata described in step three needs to be comprehensively reviewed and updated every quarter or every six months to ensure its accuracy and completeness. Ensuring the accuracy and timeliness of the metadata, especially whether the indicator processing logic has been maintained, is an important part of ensuring data quality.

[0016] Preferably, when the metadata in step 3 is updated, it is necessary to update the table in a timely manner, so as to control the version of the metadata management table.

[0017] Preferably, the SLA monitoring in step 2 needs to be reasonably set according to the actual situation of the task and business needs to ensure that all key tasks are included in the SLA monitoring scope.

[0018] Preferably, the alarm information described in step 2 can be notified to the management personnel by means of email notification, voice notification, Webhook notification, etc.

[0019] Preferably, the tasks described in step 2 include key tasks such as data extraction, conversion, loading process, data service interface, etc. It is necessary to ensure that these key tasks are monitored in the SLA to avoid the situation where the SLA monitoring does not cover the key tasks.

[0020] Preferably, when establishing the data lineage analysis function as described in step 4, it is necessary to provide an intuitive and easy-to-use lineage information visualization tool to help managers quickly understand the dependencies and flow paths between data.

[0021] Preferably, when establishing the data lineage analysis function as described in step 4, a strict permission control mechanism needs to be implemented in the lineage analysis system, and corresponding access rights need to be allocated according to user roles and responsibilities. At the same time, sensitive lineage information is encrypted for storage and transmission to ensure data security.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] By creating quality rules for verifying data tables and fields through a unified scheduling platform, flexible creation and automatic verification of business quality rules are achieved, thereby improving data quality and enhancing the level of automation in data processing. Secondly, real-time monitoring of task execution is carried out through SLA monitoring to ensure that data processing tasks are ready on time and enhance business continuity. Then, comprehensive management of table information, field information and indicator caliber is carried out through the metadata management platform, which enables a more accurate understanding of data structure and meaning, thereby improving analysis efficiency and accuracy. Finally, by establishing a complete data lineage analysis function, this not only improves the efficiency of data development, but also enhances the query capability of data flow and improves the traceability of data. The above steps provide a comprehensive, standardized and efficient solution for data quality management of Internet companies, which can greatly improve Internet companies' insight and understanding of business data, and help promote the company's sustainable development and innovation. DETAILED DESCRIPTION

[0024] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0025] The embodiment of the present invention provides a data quality supervision method, and the specific steps are as follows:

[0026] Step 1: Business Quality Rules

[0027] Through the unified scheduling platform, quality rules for verifying data tables and fields are created. When the quality rules are triggered, the system will send an alarm message to detect and handle exceptions in a timely manner.

[0028] Step 2: SLA Monitoring

[0029] Configure SLA monitoring to ensure that tasks are ready on time. If a task is not ready on time, the system will send an alarm so that the exception can be handled in time.

[0030] Step 3: Metadata Management

[0031] Through the metadata management platform, table information, field information and indicator caliber can be managed, which is conducive to data asset implementation and data analysis;

[0032] Step 4: Bloodline Analysis

[0033] Establishing a comprehensive data lineage analysis function can track the data flow path, improve data development efficiency and data flow query capabilities.

[0034] Through the unified scheduling platform, flexible creation and automatic verification of business quality rules are realized, which improves data quality and enhances the automation level of data processing. Secondly, by introducing SLA monitoring, it ensures that data processing tasks are ready on time, thereby ensuring the continuity and stability of data processing. Then, the metadata management platform is used to comprehensively manage table information, field information and indicator caliber. Through clear metadata descriptions, data analysts can more accurately understand the data structure and meaning, thereby improving analysis efficiency and accuracy. Finally, by establishing a complete data lineage analysis function, when data problems occur, the source of the problem can be quickly located and corresponding solutions can be taken to ensure data integrity and consistency.

[0035] In step 1, ensure that all newly created tables follow a unified naming convention, such as prefixes indicating data stratification ODS, DIM, DWD, DWA, APP, etc. The table name is lowercase and starts with a letter, and does not contain special characters except underscores.

[0036] The validation data table rules are:

[0037]

[0038] Among them, the metadata in step three needs to be comprehensively reviewed and updated every quarter or every six months to ensure its accuracy and completeness. Ensuring the accuracy and timeliness of the metadata, especially whether the indicator processing logic has been maintained, is an important link in ensuring data quality.

[0039] If new data tables or fields are added or the existing data model is modified, the metadata maintenance cycle should be shortened accordingly to ensure that the metadata remains synchronized with the actual data environment.

[0040] Among them, when the metadata in step three is updated, it needs to be updated in the table in a timely manner to control the version of the metadata management table.

[0041] The metadata management platform table is as follows:

[0042]

[0043] Among them, the SLA monitoring in step 2 needs to be reasonably set according to the actual situation of the task and business needs to ensure that all key tasks are included in the SLA monitoring scope.

[0044] Through reasonable configuration, we can avoid unnecessary pressure and waste of resources caused by too high SLA targets, while too low SLA may not be able to guarantee service quality.

[0045] The alarm information in step 2 can be notified to the administrator via email, voice notification, Webhook notification, etc.

[0046] Through a variety of alarm methods, we can ensure that when abnormal tasks occur, relevant managers can be notified in time to ensure the continuity of the tasks.

[0047] Among them, the tasks in step 2 include key tasks such as data extraction, conversion, loading process, data service interface, etc. It is necessary to ensure that these key tasks are monitored by SLA to avoid the situation where SLA monitoring does not cover key tasks.

[0048] Through SLA monitoring, we can ensure that these tasks are ready on time and that any abnormalities in the tasks can be discovered in a timely manner.

[0049] Among them, when establishing the data lineage analysis function in step 4, it is necessary to provide an intuitive and easy-to-use lineage information visualization tool to help managers quickly understand the dependencies and flow paths between data.

[0050] By providing a bloodline information visualization tool, managers can quickly locate and analyze data problems when using it, thereby improving the efficiency of data processing by managers and making it more convenient for managers to use.

[0051] Among them, in step 4, when establishing the data lineage analysis function, it is necessary to implement a strict permission control mechanism in the lineage analysis system, assign corresponding access rights according to user roles and responsibilities, and at the same time, encrypt the storage and transmission of sensitive lineage information to ensure data security.

[0052] By strictly managing the permissions of the bloodline analysis system, data security can be enhanced, sensitive data can be prevented from being leaked to unauthorized personnel, and the core data assets of the enterprise can be protected from malicious use or tampering.

[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0054] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data quality supervision method, characterized in that: The specific steps are as follows: Step 1: Business Quality Rules Through the unified scheduling platform, quality rules for verifying data tables and fields are created. When the quality rules are triggered, the system will send an alarm message to detect and handle exceptions in a timely manner. Step 2: SLA Monitoring Configure SLA monitoring to ensure that tasks are ready on time. If a task is not ready on time, the system will send an alarm so that the exception can be handled in time. Step 3: Metadata Management Through the metadata management platform, table information, field information and indicator caliber can be managed, which is conducive to data asset implementation and data analysis; Step 4: Bloodline Analysis Establishing a comprehensive data lineage analysis function can track the data flow path, improve data development efficiency and data flow query capabilities.

2. The data quality supervision method according to claim 1, characterized in that: Ensure that all newly created tables follow the same naming conventions as described in step 1, such as prefixes indicating data stratification ODS, DIM, DWD, DWA, APP, etc., and that table names are lowercase and start with a letter and do not contain special characters except underscores.

3. The data quality supervision method according to claim 1, characterized in that: The metadata described in step three needs to be comprehensively reviewed and updated every quarter or every six months to ensure its accuracy and completeness. Ensuring the accuracy and timeliness of metadata, especially whether the indicator processing logic has been maintained, is an important part of ensuring data quality.

4. The data quality supervision method according to claim 1, characterized in that: When the metadata described in step 3 is updated, it is necessary to update the table in a timely manner to control the version of the metadata management table.

5. The data quality supervision method according to claim 1, characterized in that: The SLA monitoring described in step 2 needs to be reasonably set up according to the actual situation of the task and business needs to ensure that all key tasks are included in the SLA monitoring scope.

6. The data quality supervision method according to claim 1, characterized in that: The alarm information described in step 2 can be notified to the administrator via email notification, voice notification, Webhook notification, etc.

7. The data quality supervision method according to claim 1, characterized in that: The tasks described in step 2 include key tasks such as data extraction, conversion, loading process, data service interface, etc. It is necessary to ensure that these key tasks are monitored in the SLA to avoid the situation where SLA monitoring does not cover key tasks.

8. The data quality supervision method according to claim 1, characterized in that: When establishing the data lineage analysis function as described in step 4, it is necessary to provide an intuitive and easy-to-use lineage information visualization tool to help managers quickly understand the dependencies and flow paths between data.

9. The data quality supervision method according to claim 1, characterized in that: As described in step 4, when establishing the data lineage analysis function, it is necessary to implement a strict permission control mechanism in the lineage analysis system, assign corresponding access rights according to user roles and responsibilities, and at the same time, encrypt the storage and transmission of sensitive lineage information to ensure data security.

Images