A data quality monitoring method based on business entities

Abstract

The present invention discloses a data quality monitoring method based on business entities, including: a) determining the data scope that needs to be monitored; b) identifying core business entities; c) building a base map for data quality monitoring; d) formulating data monitoring rules, and monitoring data quality according to the rules. The present invention proposes a universal data quality monitoring method based on core business entity statistical indicators, which examines data quality from a global business perspective and supports data governance.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data quality monitoring method based on a business entity. Background Art

[0002] Data quality monitoring is a global and visual display of data quality. At present, mainstream data quality tools mainly discover defective data through methods such as data profiling, rule verification based on SQL scripts, and outlier analysis based on algorithms, and quantify data in multiple dimensions by combining data quality indicators such as timeliness, completeness, compliance, and consistency. Among them, data profiling is an analysis method based on the principle of database structure, which is mainly used to understand information such as data structure, numerical validity, and distribution, and to discover general data problems; rule verification based on SQL scripts requires manual conversion of business requirements into data verification scripts, which are executed regularly to discover internal logical problems in the data; outlier analysis based on algorithms has received widespread attention in recent years, and some institutions are studying the realization of intelligent verification through machine learning classification algorithms, but there are still problems such as high modeling requirements and limited adaptability. At the same time, the monitoring model based on traditional multidimensional data quality indicators has become increasingly difficult to meet the needs of more efficient and sophisticated management. Summary of the invention

[0003] The technical problem to be solved by the present invention is to propose a universal data quality monitoring method based on statistical indicators of core business entities, to examine data quality from a global business perspective and to support data governance.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A data quality monitoring method based on a business entity, comprising:

[0006] a) Determine the scope of data that needs to be quality monitored;

[0007] b) Identify core business entities;

[0008] c) Build a base map for data quality monitoring;

[0009] d) Formulate data monitoring rules and monitor data quality based on the rules.

[0010] As a further improvement of the above technical solution:

[0011] In step a),

[0012] The method for determining the data range comprises the following steps:

[0013] 1) Determine the organization, analyze and determine the organization related to the data to be monitored, which is generally also the party responsible for data quality;

[0014] 2) Analyze business capabilities, evaluate the basic business capabilities involved in specific business needs, and determine the information systems to be monitored based on business capabilities with reference to the information architecture;

[0015] 3) Focus on data objects and determine the scope of data tables to be monitored by referring to the database design of each system;

[0016] 4) Within a specified time range, use methods including incremental, full or specified time periods to clarify the data scope of each data table involved in quality monitoring.

[0017] In step b),

[0018] The identified core business entities include organization entities, person entities, location entities and event entities;

[0019] The organizational entity is an organizational unit involved in business activities, which can be any level in the organizational hierarchy and has specific business functions;

[0020] The person-type entities are people and things that initiate business activities or are affected by business activities;

[0021] The location entity is used to describe the geographical location where a business activity or a specific event occurs;

[0022] The event entity is data used to describe business activities and business events, and its key attributes include person, time, place and event description.

[0023] In step c),

[0024] The process of constructing the base map includes the following steps:

[0025] 1) Analyze key business processes, draw business process diagrams based on the data scope of step a), and analyze the relationships between business activities and the flow of business information;

[0026] 2) Map core business entities, match the identified business entities with the business links in the business process, and clarify the corresponding relationship between each business link and business entity;

[0027] 3) Draw a base map for data quality monitoring, wherein the base map draws a business entity distribution map with business links as the horizontal axis and organizational units as the vertical axis, and uses arrows to describe the logical relationship between data generation between business entities to form an arrow diagram;

[0028] 4) Matching monitoring data, analyzing the business entity data in the base map, matching it with the data table determined in step a), and establishing a mapping relationship between the business entity and the physical data table.

[0029] In step d),

[0030] The process of formulating data monitoring rules includes the following steps:

[0031] 1) Analyze the logical dependencies between business entities. The business entity data that are interrelated in business have a certain logical relationship;

[0032] 2) Prepare business data quality inspection rules based on the logical dependency relationship between business entities. Subsequent business entity data that does not meet the logical dependency relationship is considered as problem data, and the link where the problem occurs is the subsequent business link;

[0033] 3) Develop data quality monitoring functions, convert data quality inspection rules into data monitoring inspection scripts, build a data monitoring rule library, and develop information technology functions to automatically check data during the cycle.

[0034] Furthermore, the logical relationship includes existence dependency, counting dependency and index dependency;

[0035] The existence dependency relationship is that the subsequent business entity has certain characteristic attributes of the preceding business entity;

[0036] The counting dependency relationship is that under certain restrictions, the number of service instances in the leading service entity should be greater than or equal to the number of service instances of certain service entities in the subsequent service;

[0037] The indicator dependency relationship is that under certain restrictions, the key statistical indicators of the leading business entity should be greater than or equal to the key statistical indicators of certain business entity instances in the subsequent business.

[0038] Furthermore, the monitoring function includes monitoring rule base management, business data quality monitoring and abnormal data analysis and display;

[0039] The monitoring rule library management is used to realize the centralized management of monitoring rules and scripts, and can configure the monitoring rule execution cycle and execution parameters to perform monitoring task scheduling.

[0040] The business data quality monitoring is used to intuitively display the data quality monitoring results in combination with the data base map, issue abnormality alarm information, highlight the data entities with data abnormalities, and use dynamic text to prompt the violated monitoring rules and the number of related data entries;

[0041] The abnormal data analysis display is used to view the data that violates the rules by clicking on the abnormal alarm information. The data includes the leading business entity instance identifier, number of instances, key statistical indicators, subsequent business entity instance identifier, number of instances, key statistical indicators, and rule description.

[0042] Furthermore, the process of monitoring data quality includes:

[0043] 1) Periodically check the logical dependencies between business entity data according to quality inspection rules;

[0044] 2) For existing dependencies, check whether non-dependencies are found; for count dependencies or indicator dependencies, check whether statistical data differences are found; if not, repeat step 1); if yes, issue an abnormal alarm and conduct abnormal data analysis.

[0045] The beneficial effects of the present invention are:

[0046] A general data quality monitoring model based on business entities suitable for complex data architecture environments is proposed. It has the characteristics of being oriented to the internal logic of the business, simple and efficient, and intuitive and easy to understand. It is convenient for business users to globally control the data quality level from a business perspective and intuitively display the business links where data quality problems arise. Compared with traditional methods, the present invention closely combines data governance work with business, making data quality intuitive and easy to understand, and easier to locate the adverse effects of data problems, winning recognition and support for data governance work, and can serve as an important support means for data governance and data asset operation management. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a preferred embodiment.

[0048] Figure 2 It is a logic block diagram for monitoring data quality in a preferred embodiment.

[0049] Figure 3 This is a display diagram of the quality monitoring base map in the preferred embodiment. DETAILED DESCRIPTION

[0050] The business entity-based data quality monitoring method of this embodiment has been put into practical application in the construction of the supply chain operation center system of State Grid Corporation of China. The data monitoring and management function has been developed to achieve quasi-real-time monitoring of the quality of the entire supply chain data such as planning, bidding, contracts, suppliers, warehousing, etc. The system users cover the State Grid headquarters and 27 provincial and municipal companies, which strongly supports the daily operation of the supply chain operation center of State Grid Corporation of China.

[0051] refer to Figure 1-3 , this embodiment includes the following five steps:

[0052] 1. Determine the scope of monitoring

[0053] Use the functional decomposition method to analyze the data scope that needs to be quality monitored from specific business needs, which is divided into the following steps:

[0054] 1. Determine the organizational structure, analyze and determine the organizational structure related to the data to be monitored, which is generally also the party responsible for data quality.

[0055] 2. Analyze business capabilities, evaluate the basic business capabilities involved in specific business needs, and determine the information systems to be monitored based on business capabilities with reference to the information architecture. This may involve complex monitoring environments such as multiple information systems and data warehouses.

[0056] 3. Focus on data objects and refer to the database design of each system to determine the scope of the data table to be monitored.

[0057] 4. Within a certain time range, use incremental / full / specified time periods and other methods to clarify the data scope of each data table involved in quality monitoring.

[0058] (II) Identification of core business entities

[0059] Entity is an abstract description of business elements, used to describe objects or things that exist objectively and can be distinguished from each other in the real world. This invention uses information modeling to identify core business entities in combination with business needs and monitoring scope, and uses business entities as the basis and basis for data quality monitoring to monitor data quality from a business perspective:

[0060] 1. Organizational entities: organizational units involved in business activities, which can be at any level in the organizational hierarchy and have specific business functions.

[0061] 2. Human entities: people and things that initiate business activities or are affected by business activities. They generally belong to specific organizations and are core elements that can appear repeatedly in the business process.

[0062] 3. Location entities describe the geographical location where business activities or specific events occur, and can provide different levels of geographical location accuracy in a step-by-step manner.

[0063] 4. Event entities are used to describe data on business activities and business events. They generally include key attributes such as people, time, location, and event description, and have the characteristic of continuing to grow over time.

[0064] (III) Building a data monitoring base map

[0065] The present invention proposes a data diagram method based on a business perspective, which intuitively presents the inherent business logic relationship of the data and can be used as a base map for data quality monitoring based on business entities. The construction process is divided into the following steps:

[0066] 1. Analyze key business processes, draw business process diagrams based on the business capabilities determined in the "Determine Monitoring Scope" step, and analyze the relationships between business activities and the flow of business information.

[0067] 2. Map core business entities, match the business entities identified in the "Identify core business entities" step with the business links in the business process, and clarify the corresponding relationship between each business link and business entity.

[0068] 3. Draw a data base map with the business link as the horizontal axis (i.e. Figure 3 "Procurement Plan" and "Bidding Management" in...), the organizational unit is the vertical axis (i.e. Figure 3 "State Grid Corporation of China" "State Grid Beijing Company" ...), draw a business entity distribution map (i.e. Figure 3 Small boxes connected by arrows) are used to describe the connection between a specific organizational unit and a business entity in a specified business link, and arrows are used to describe the logical relationship between data generated between business entities to form an arrow diagram.

[0069] 4. Match the monitoring data, analyze the business entity data in the data base map, match it with the system data table determined in the "Determine the monitoring scope" step, and establish a mapping relationship between the business entity and the physical data table.

[0070] (IV) Formulate data monitoring rules

[0071] Based on the data base map, the logical dependencies between business entities are analyzed. Based on the logical dependencies, business data verification rules are proposed from a global perspective. With the help of information technology, data quality monitoring functions are developed. The main steps are as follows:

[0072] 1. Analyze the logical dependencies between business entities. The business entity data that are interrelated in business have a certain logical relationship, mainly including:

[0073] (1) There is a dependency relationship, that is, the subsequent business entity has certain characteristic attributes of the preceding business entity. For example, the payment application number in the payment entity must exist in the payment application entity, that is, the payment application comes first and then the payment behavior.

[0074] (2) Count dependency, that is, under certain restrictions (generally based on organizational structure and time constraints), the number of business instances in the leading business entity should be greater than or equal to the number of business instances in certain business entities in the subsequent business. For example, under the same restrictions, the number of subcontracts in the bidding sub-bid entity should be greater than or equal to the number of subcontracts in the winning bid entity.

[0075] (3) Indicator dependency, that is, under certain constraints (generally based on organizational structure and time constraints), the key statistical indicators of the leading business entity should be greater than or equal to the key statistical indicators of certain business entity instances in the subsequent business. For example, under the same constraints, the total bid price of the bidding entity should be greater than or equal to the total bid amount of the winning entity.

[0076] 2. Formulate business data verification rules and compile business data quality inspection rules based on the logical dependency relationship between business entities. Subsequent business entity data that does not meet the logical dependency relationship is regarded as problem data, and the link where the problem occurs is the subsequent business link.

[0077] 3. Develop data quality monitoring functions, convert data quality inspection rules into data monitoring inspection scripts, build a data monitoring rule library, and develop information technology functions to automatically check data in cycles. The main functions are as follows:

[0078] (1) Monitoring rule library management, realizing centralized management of monitoring rules and scripts, configuring monitoring rule execution cycle and execution parameters, and scheduling monitoring tasks.

[0079] (2) Business data quality monitoring: Combined with the data base map, the data quality monitoring results are intuitively displayed, and abnormal change alarm information is issued to highlight the data entities with data changes, and the monitoring rules that are violated and the number of related data entries are prompted with dynamic text.

[0080] (3) Analysis and display of abnormal data: Click on the abnormal alarm information to view the data that violates the rules, including the preceding business entity instance ID, number of instances, key statistical indicators, the subsequent business entity instance ID, number of instances, key statistical indicators, rule description and other information.

[0081] 5. Monitoring data quality

[0082] Relying on data quality monitoring tools, data quality is monitored periodically from the overall business perspective. When suspicious data quality issues are found, immediate warning information is generated to prompt management personnel to check and verify in a timely manner to promote problem data governance. The process of monitoring data quality includes: 1) According to the quality inspection rules, periodically inspect the logical dependencies between business entity data; 2) For existing dependencies, check whether non-dependencies are found; for count dependencies or indicator dependencies, check whether statistical data differences are found; if not, repeat step 1); if yes, issue an abnormal alarm and conduct abnormal data viewing and analysis.

Claims

1. A data quality monitoring method based on business entities, characterized in that: include: a) Determine the scope of data that needs to be quality monitored; b) Identify core business entities; c) Build a base map for data quality monitoring; d) Formulate data monitoring rules and monitor data quality according to the rules; The process of formulating data monitoring rules includes the following steps: 1) Analyze the logical dependencies between business entities. The business entity data that are interrelated in business have a certain logical relationship; 2) Prepare business data quality inspection rules based on the logical dependency relationship between business entities. Subsequent business entity data that does not meet the logical dependency relationship is considered as problem data, and the link where the problem occurs is the subsequent business link; 3) Develop data quality monitoring functions, convert data quality inspection rules into data monitoring inspection scripts, build a data monitoring rule library, and develop information function cycles to automatically inspect data; The logical relationship includes existence dependency, counting dependency and index dependency; The existence dependency relationship is that the subsequent business entity has certain characteristic attributes of the preceding business entity; The counting dependency relationship is that under certain restrictions, the number of service instances in the leading service entity should be greater than or equal to the number of service instances of certain service entities in the subsequent service; The indicator dependency relationship is that under certain restrictions, the key statistical indicators of the leading business entity should be greater than or equal to the key statistical indicators of certain business entity instances in the subsequent business; The monitoring functions include monitoring rule base management, business data quality monitoring and abnormal data analysis and display; The monitoring rule library management is used to realize the centralized management of monitoring rules and scripts, and can configure the monitoring rule execution cycle and execution parameters to schedule monitoring tasks; The business data quality monitoring is used to intuitively display the data quality monitoring results in combination with the data base map, issue abnormality alarm information, highlight the data entities with data abnormalities, and use dynamic text to prompt the violated monitoring rules and the number of related data entries; The abnormal data analysis display is used to click on the abnormal alarm information to view the data that violates the rules, which includes the leading business entity instance identifier, number of instances, key statistical indicators, subsequent business entity instance identifier, number of instances, key statistical indicators, and rule description; The process of monitoring data quality includes: 1) Periodically check the logical dependencies between business entity data according to quality inspection rules; 2) For existing dependencies, check whether non-dependencies are found; for count dependencies or indicator dependencies, check whether statistical data differences are found; if not, repeat step 1); if yes, issue an abnormal alarm and conduct abnormal data analysis.

2. The data quality monitoring method according to claim 1, characterized in that: The method for determining the data range comprises the following steps: 1) Determine the organization, analyze and determine the organization related to the data to be monitored, which is generally also the party responsible for data quality; 2) Analyze business capabilities, evaluate the basic business capabilities involved in specific business needs, and determine the information systems to be monitored based on business capabilities with reference to the information architecture; 3) Focus on data objects and refer to the database design of each system to determine the scope of the data table to be monitored; 4) Within a specified time range, use methods including incremental, full or specified time periods to clarify the data scope of each data table involved in quality monitoring.

3. The data quality monitoring method according to claim 1, characterized in that: The identified core business entities include organization entities, person entities, location entities and event entities; The organizational entity is an organizational unit involved in business activities, any level in the organizational hierarchy, and has specific business functions; The person-type entities are people and things that initiate business activities or are affected by business activities; The location entity is used to describe the geographical location where a business activity or a specific event occurs; The event entity is data used to describe business activities and business events, and its key attributes include person, time, place and event description.

4. The data quality monitoring method according to claim 1, characterized in that: The process of constructing the base map includes the following steps: 1) Analyze key business processes, draw business process diagrams based on the data scope of step a), and analyze the relationships between business activities and the flow of business information; 2) Map core business entities, match the identified business entities with the business links in the business process, and clarify the corresponding relationship between each business link and business entity; 3) Draw a base map for data quality monitoring, which uses business links as the horizontal axis and organizational units as the vertical axis to draw a business entity distribution map, and uses arrows to describe the logical relationship between data generation between business entities, forming an arrow diagram; 4) Match the monitoring data, analyze the business entity data in the base map, match it with the data table determined in step a), and establish a mapping relationship between the business entity and the physical data table.

Images