A high-quality data management system based on data governance
By designing a high-quality data management system based on data governance, it solves the problem that traditional data management systems are difficult to cope with large-scale data and multi-source heterogeneous data, and achieves high-quality and reliability management of data.
Patent Information
- Application Number
- CN202411393337.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-10-08
AI Technical Summary
Traditional data management systems are difficult to cope with the growing amount and complexity of data. Multi-source heterogeneous data leads to uneven data quality and lack of effective data governance mechanisms, which affects the reliability and value of data.
A high-quality data management system based on data governance was designed, including business data acquisition module, business data conversion module, data resource integration module, data quality basic evaluation module, comprehensive data quality evaluation index calculation module, data quality judgment module and data The governance report output module uses these modules to collect, convert, integrate, evaluate and manage data to ensure high quality and reliability.
By standardizing the unified data format, integrating multi-source data, comprehensively evaluating data quality and timely monitoring data problems, effectively managing multi-source heterogeneous data, improving the reliability and value of data, solving the uneven data quality in traditional data management systems question.
Smart Images

Figure CN118897837B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data governance, and more specifically, to a high-quality data management system based on data governance. Background Art
[0002] In today's digital age, data has become the core asset of enterprises and organizations. With the continuous expansion of business and the deepening of informatization, the amount of data has exploded and the types of data have become increasingly diversified.
[0003] Relational database management systems and enterprise-level data warehouses have provided a solid foundation for many industries over the past few decades. By providing reliable data storage solutions, efficient data query capabilities, and preliminary data integration functions, they have promoted the development of data analysis and business intelligence and provided strong support for the decision-making process of enterprises.
[0004] However, in actual use, it still has some shortcomings, such as difficulty in coping with the growing demand for data volume and complexity, and multi-source heterogeneous data, resulting in uneven data quality that is difficult to meet; traditional data management systems often lack effective data governance mechanisms, resulting in frequent data quality problems, affecting the reliability and value of the data. Summary of the invention
[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides a high-quality data management system based on data governance, which solves the problems raised in the above-mentioned background technology through the following scheme.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A high-quality data management system based on data governance, including:
[0008] Business data collection module: used to collect data in various data formats from different business data sources, and encapsulate the collected data into data sets and transmit them to the business data conversion module;
[0009] Business data conversion module: used to pre-process the transmitted data sets, use conditional filtering and regular expression matching methods to convert the data in each data set into the existing data set of the target business field in a unified format, and transmit it to the data resource integration module;
[0010] Data resource integration module: used to integrate existing data sets and data sources in the target business field, classify them according to the nature and purpose of the data, select appropriate storage technologies to store the classified data sets, and send the data sets in different storage technologies to the data quality basic assessment module;
[0011] Data quality basic evaluation module: used to analyze the transmitted data set and calculate the consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value, and transmit them to the comprehensive data quality evaluation index calculation module;
[0012] The data quality basic assessment module includes a data quality receiving unit, a data feature extraction unit, a consistency score calculation unit, an accuracy score calculation unit, a completeness score calculation unit, a timeliness score calculation unit, and a data quality output unit. The specific processing process of the basic assessment is as follows:
[0013] Data quality receiving unit: receiving the target data set transmitted by the data integration module;
[0014] Data feature extraction unit: Verify that the target data set has been preprocessed, extract the numerical data in the target data set, record the total number of data points in the target data set, the value of the jth data point in the target data set, the time when the jth data point in the target data set was recorded, the time set for analysis, and the time range covered by the target data set, which are denoted as n, x, respectively. j ,t j , t, and Δt; use machine learning algorithms to count the total number of missing values and the total number of outliers in the target data set, denoted as ITn and IFn respectively; calculate the mean value of all data points in the target data set, the standard deviation of all data points in the target data set, the data trend index in the target data set, and the data change rate in the target data set, denoted as μ, σ, β respectively 0 , β 1 ; Further, the specific calculation formula for the average value of all data points in the target data set is:
[0015] ;
[0016] The specific calculation formula for the standard deviation of all data points in the target data set is:
[0017] ;
[0018] The specific calculation formula for the data trend indicator in the target data set is:
[0019] ;
[0020] The specific calculation formula for the data change rate in the target data set is:
[0021] ;
[0022] Consistency score calculation unit: The total number of data points n in the target data set, the value x of the jth data point in the target data set j, the average value μ of all data points in the target data set, and the standard deviation σ of all data points in the target data set, calculate the data cohesion index MO of the target data set. The specific calculation formula is:
[0023] ,
[0024] Where j represents the index of the data set; the data trend indicator β in the target data set 0 , the data change rate β in the target data set 1 , set the analysis time t, and the data cohesion index MO of the target data set, and calculate the time correlation trend index TR of the data in the target data set. The specific calculation formula is:
[0025] TR=β 0 +β 1 ×t+MO;
[0026] The data cohesion index MO of the target data set, the time correlation trend index TR of the data in the target data set, and the value x of the jth data point in the target data set j , calculate the consistency score function value C, the specific calculation formula is:
[0027] ,
[0028] Where n represents the total number of data points in the target data set, and j represents the index of the data point;
[0029] Accuracy score calculation unit: The data cohesion index MO of the target data set, the F1 score, and the value x of the jth data point in the target data set j , calculate the accuracy score function value A, the specific calculation formula is:
[0030] ,
[0031] The F1 score refers to a statistical indicator, with a maximum value of 1 and a minimum value of 0;
[0032] Completeness score calculation unit: Calculate the proportion of missing values IT in the target data set from the total number of data points n in the target data set and the total number of missing values ITn in the target data set. The specific calculation formula is:
[0033] IT = ITn / n;
[0034] The proportion IF of data outliers in the target data set is calculated by the total number of data points n in the target data set and the total number of outliers IFn in the target data set. The specific calculation formula is:
[0035] IF=IFn / n;
[0036] The integrity score function value I is calculated by the time correlation trend index TR of the data in the target data set, the proportion of missing values IT of the data in the target data set, and the proportion of abnormal values IF of the data in the target data set. The specific calculation formula is:
[0037] ;
[0038] Timeliness score calculation unit: The time t of the set analysis and the time t of the jth data point in the target data set j, And the time range Δt covered by the target data set, calculate the data time fraction Tj of the target data set. The specific calculation formula is:
[0039] Tj=((tt j ) / Δt);
[0040] The time t of the analysis is set, and the time t of the jth data point in the target data set is recorded. j, As well as the time range Δt covered by the target data set, calculate the time factor Tf in the target data set. The specific calculation formula is:
[0041] Tf=exp(-Δt×(tt j ));
[0042] The timeliness score function value T is calculated by the time correlation trend index TR of the data in the target data set, the data time score Tj of the target data set, and the time factor Tf in the target data set. The specific calculation formula is:
[0043] ,
[0044] Where n represents the total number of data points in the target data set, and j represents the index of the data point;
[0045] Data quality output unit: transmits the calculated consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value to the comprehensive data quality assessment index calculation module.
[0046] Comprehensive data quality assessment index calculation module: calculates comprehensive data quality assessment index based on consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value;
[0047] The specific calculation formula of the comprehensive data quality evaluation index calculation module to calculate the comprehensive data quality evaluation index CAIT based on the consistency score function value C, the accuracy score function value A, the integrity score function value I, and the timeliness score function value T is:
[0048] ,
[0049] Among them, V represents the value coefficient of data, DMC represents the data maintenance cost, α C , α A , α I , α T They are respectively expressed as the penalty factor for consistency score, the penalty factor for accuracy score, the penalty factor for completeness score, and the penalty factor for timeliness score.
[0050] Data quality judgment module: used to judge and monitor data quality, detect data quality issues and trigger data iteration strategies;
[0051] Data governance report output module: used to output processed and analyzed data to users in report format through data visualization tools, so as to quickly make data governance decisions;
[0052] Database: used to store high-quality data, store instructions, manage collected data, and provide data retrieval and update functions;
[0053] Processor: used to execute the instructions stored in the database.
[0054] Preferably, the business data collection module includes a structured data collection unit, a semi-structured data collection unit, an unstructured data collection unit, a metadata collection unit, and a business data output unit, wherein the structured data collection unit uses ETL tools to collect structured data in the enterprise database; the semi-structured data collection unit comprehensively uses API interfaces and crawler technology to collect semi-structured business information and interim results in business processes; the unstructured data collection unit uses a streaming data processing framework to collect unstructured data without a fixed structure on social media; the metadata collection unit records the source of data by establishing metadata fields; and the business data output unit encapsulates the data collected by different units into data sets and transmits them to the business data conversion module.
[0055] Preferably, the business data conversion module preprocesses including removing duplicates, filling missing values, and format standardization; further, using an exact matching algorithm to quickly identify duplicate records in a data set and remove them according to system-set rules; using mean filling, median filling, and mode filling methods to process missing values in a data set; formulating a unified data format standard for various data types in a data set; wherein the system-set rules include determining a unique identifier, a comparison method, and a removal strategy; further, for different types of data, clearly defining a field combination that can uniquely identify a record; when using an exact matching algorithm, strictly comparing whether the values of the unique identifier fields are completely consistent, if the unique identifier field values of two records are exactly the same, they are marked as duplicates; when duplicates are determined, merging the information of the duplicates, and merging the different field values of the duplicates.
[0056] Preferably, the data resource integration module includes a data resource integration receiving unit, a data combing and classification unit, a data integration unit and a data resource integration output unit; wherein the data resource integration receiving unit is used to receive the existing data set of the target business field in a unified format; the data combing and classification unit integrates and combs the existing data set of the target business field with the data source, and classifies the data according to the nature and purpose of the data; the data integration unit selects a suitable storage technology for storage of the classified data set according to business needs; the data resource integration output unit sends the target data set in different storage technologies to the data quality basic assessment module.
[0057] Preferably, the data quality judgment module determines the specific processing process of data quality as follows: determine the data quality threshold based on the target business needs, data usage scenarios and historical data quality performance, and divide the comprehensive data quality assessment indicators into five levels: excellent, good, medium, poor and extremely poor; and establish a data quality monitoring mechanism. As the data environment changes and business needs evolve, the threshold range of the comprehensive data quality assessment indicators is regularly adjusted, checked and evaluated to promptly discover and handle new data quality problems; once the data quality judgment module detects that the comprehensive data quality assessment indicator is lower than the set minimum threshold, the data iteration strategy is initiated; it should be noted that the data iteration strategy includes iterative analysis of the data, determines the specific type and scope of data quality problems, and formulates data cleaning and repair strategies and performs automated processing according to the type and severity of the problems.
[0058] Preferably, the data governance report output module and the specific output process of the data governance report are as follows: based on the judgment results of the data quality judgment module, a detailed data quality report is generated, including comprehensive data quality assessment indicators and specific values of each sub-indicator, which displays the quality status of the data and points out the type, scope and severity of the data quality problems; at the same time, high-quality data is output to the database to improve the reliability and value of the data.
[0059] Technical effects and advantages of the present invention:
[0060] 1. The present invention effectively manages multi-source heterogeneous data by formulating unified data format standards for various data types and integrating existing data sets and data sources in the target business field;
[0061] 2. The present invention analyzes the data set and calculates the consistency, accuracy, completeness, and timeliness score function values, and calculates the comprehensive data quality evaluation index based on each score function value, comprehensively evaluates the data quality, and promptly discovers data quality problems, thereby coping with the problem of uneven data quality;
[0062] 3. The present invention establishes a data quality monitoring mechanism, regularly adjusts, checks and evaluates the threshold range, promptly discovers and handles new data quality issues, and improves the reliability and value of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 It is a system flow chart of the present invention.
[0064] Figure 2 It is a diagram of the method steps of the present invention.
[0065] Figure 3 It is a schematic diagram of the system structure of the present invention.
[0066] Figure 4 It is a schematic diagram of the structure of the business data collection module of the present invention.
[0067] Figure 5 It is a schematic diagram of the structure of the data quality basic assessment module of the present invention. DETAILED DESCRIPTION
[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0069] As attached Figure 1A high-quality data management system based on data governance is shown, including a business data collection module, a business data conversion module, a data resource integration module, a data quality basic assessment module, a comprehensive data quality assessment index calculation module, a data quality judgment module, a data governance report output module, a database, and a processor.
[0070] The business data collection module is used to collect data in various data formats from different business data sources, and encapsulate the collected data into data sets for transmission to the business data conversion module.
[0071] Furthermore, the business data collection module includes a structured data collection unit, a semi-structured data collection unit, an unstructured data collection unit, a metadata collection unit, and a business data output unit. It should be noted that the structured data collection unit uses ETL tools to collect structured data in the enterprise database; the semi-structured data collection unit comprehensively uses API interfaces and crawler technology to collect semi-structured business information and interim results in business processes; the unstructured data collection unit uses a streaming data processing framework to collect unstructured data without a fixed structure on social media; the metadata collection unit records the data source by establishing metadata fields; and the business data output unit encapsulates the data collected by different units into different data sets and transmits them to the business data conversion module.
[0072] It should be noted that structured data has a clear data structure and format, and ETL tools can efficiently extract, transform and load it from the data source into the data management system; semi-structured data provides enterprises with market trends and competitor dynamics through API interfaces with partners and crawler technology, so as to understand the progress of the business and adjust strategies in time; unstructured data is processed and captured in real time on social media through a streaming data processing framework, so as to understand consumers' evaluation and demand for products and provide enterprises with the latest information; data sources are marked by establishing metadata fields, and the metadata fields are recorded as m f , where f represents the type of data source, thereby ensuring the quality and security of the data.
[0073] In this embodiment, the structured data includes transaction record data and customer information data. Furthermore, the transaction record data includes transaction time, transaction amount, and fields that clearly indicate the information of both parties to the transaction; the customer information data includes customer name, contact information, and purchase history.
[0074] In this embodiment, the semi-structured data includes business information and interim results. Furthermore, the business information includes XML formatted data on web pages, including product inventory information and market price trends; the interim results include project progress reports, meeting minutes, news information and industry reports in the form of semi-structured documents.
[0075] In this embodiment, the unstructured data includes text, pictures, and video content on social media.
[0076] In this embodiment, the metadata field includes the source information of the data, the creation time of the data, the modification time and the data owner. The metadata field of the social media platform is denoted as m A , the metadata field of the sensor is denoted as m B , the metadata field of the enterprise database is denoted as m C .
[0077] The business data conversion module is used to pre-process the transmitted data sets, use conditional filtering and regular expression matching methods to convert the data in each data set into an existing data set in the target business field in a unified format, and transmit it to the data resource integration module.
[0078] It should be noted that preprocessing includes removing duplicates, filling missing values, and format standardization; further, an exact matching algorithm is used to quickly identify duplicate records in the data set and remove them according to the rules set by the system; mean filling, median filling, and mode filling methods are used to process missing values in the data set; and unified data format standards are established for various data types in the data set.
[0079] It should be noted that the rules set by the system include determining the unique identifier, comparison method, and removal strategy; further, for different types of data, the field combination that can uniquely identify a record is clearly defined; when using the exact matching algorithm, the values of the unique identifier field are strictly compared to see if they are completely consistent. If the unique identifier field values of two records are exactly the same, they are marked as duplicates; when the duplicates are determined, the information of the duplicates is merged, and the different field values of the duplicates are merged.
[0080] In this embodiment, for the customer information data set, duplicate customer records are removed based on the customer's unique identification data information to ensure that each customer appears only once in the data set.
[0081] In this embodiment, for numerical data, if the data is normally distributed, the mean is used for filling; if the data is skewed, the median is used for filling; for categorical data, the mode filling method is used for prediction filling.
[0082] In this embodiment, for date fields, they are unified into a specific year-month-day format; for currency fields, the currency symbol and decimal places are unified; for text fields, the case is unified and special characters are removed; string type data is converted to numeric type for mathematical operations; and date strings are converted to date type for time series analysis.
[0083] The data resource integration module is used to integrate the existing data sets and data sources in the target business field, classify them according to the nature and purpose of the data, select appropriate storage technologies to store the classified data sets, and send the data sets in different storage technologies to the data quality basic assessment module.
[0084] Furthermore, the data resource integration module includes a data resource integration receiving unit, a data combing and classification unit, a data integration unit and a data resource integration output unit; it should be noted that the data resource integration receiving unit is used to receive the existing data sets of the target business field in a unified format; the data combing and classification unit integrates and combs the existing data sets of the target business field with the data sources, and classifies them according to the nature and purpose of the data; the data integration unit selects appropriate storage technology for storage of the classified data sets according to business needs; the data resource integration output unit sends the target data sets in different storage technologies to the data quality basic assessment module.
[0085] It should be noted that a business data resource integration method is now provided, comprising the following steps:
[0086] A1: Identify the sources and uses of existing data sets based on the target business domain. The data sets involved include structured data sets, semi-structured data sets, unstructured data sets, and data sources marked with metadata fields;
[0087] A2: For data with clear data sources and uses, a target data set based on business needs is constructed. The target data set consists of a series of data lists, each of which contains specific information on data sources, data content, and data uses.
[0088] A3: Establish a data classification framework according to its nature and purpose, and divide the target data set into different categories;
[0089] A4: According to different categories of target data sets, select a suitable storage technology combination for storage. The storage technology combination includes relational database storage, distributed file system storage, and NoSQL database storage.
[0090] It should be noted that the data classification framework includes basic data sets, business data sets and thematic data sets; further, basic data sets represent common and widely used data; business data sets represent data directly related to specific business processes; and thematic data sets represent data organized around specific topics.
[0091] In this embodiment, the basic data includes basic customer information and product catalog; the business data includes order details and sales records; and the subject data includes marketing activity data or customer service data.
[0092] In this embodiment, if the proportion of structured data in the target data set exceeds 50%, a relational database is used for storage; if the proportion of unstructured data in the target data set exceeds 50%, a distributed file system is used; if the proportion of semi-structured data in the target data set exceeds 50%, a NoSQL database is used for storage.
[0093] The basic data quality assessment module is used to analyze the transmitted data set and calculate the consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value, and transmit them to the comprehensive data quality assessment indicator calculation module.
[0094] The data quality basic assessment module includes a data quality receiving unit, a data feature extraction unit, a consistency score calculation unit, an accuracy score calculation unit, a completeness score calculation unit, a timeliness score calculation unit, and a data quality output unit. The specific processing process of the basic assessment is as follows:
[0095] Data quality receiving unit: receiving the target data set transmitted by the data integration module;
[0096] Data feature extraction unit: Verify that the target data set has been preprocessed, extract the numerical data in the target data set, record the total number of data points in the target data set, the value of the jth data point in the target data set, the time when the jth data point in the target data set was recorded, the time set for analysis, and the time range covered by the target data set, which are denoted as n, x, respectively. j ,t j , t, and Δt; use machine learning algorithms to count the total number of missing values and the total number of outliers in the target data set, denoted as ITn and IFn respectively; calculate the mean value of all data points in the target data set, the standard deviation of all data points in the target data set, the data trend index in the target data set, and the data change rate in the target data set, denoted as μ, σ, β respectively 0 , β 1 ,Furthermore, the specific calculation formula for the average value of all data points in the target data set is:
[0097] ;
[0098] The specific calculation formula for the standard deviation of all data points in the target data set is:
[0099] ;
[0100] The specific calculation formula for the data trend indicator in the target data set is:
[0101] ;
[0102] The specific calculation formula for the data change rate in the target data set is:
[0103] ;
[0104] Consistency score calculation unit: The total number of data points n in the target data set, the value x of the jth data point in the target data set j , the average value μ of all data points in the target data set, and the standard deviation σ of all data points in the target data set, calculate the data cohesion index MO of the target data set. The specific calculation formula is:
[0105] ,
[0106] Where j represents the index of the data set; the data trend indicator β in the target data set 0 , the data change rate β in the target data set 1 , set the analysis time t, and the data cohesion index MO of the target data set, and calculate the time correlation trend index TR of the data in the target data set. The specific calculation formula is:
[0107] TR=β 0 +β 1 ×t+MO;
[0108] The data cohesion index MO of the target data set, the time correlation trend index TR of the data in the target data set, and the value x of the jth data point in the target data set j , calculate the consistency score function value C, the specific calculation formula is:
[0109] ,
[0110] Where n represents the total number of data points in the target data set, and j represents the index of the data point;
[0111] Accuracy score calculation unit: The data cohesion index MO of the target data set, the F1 score F1, and the value x of the jth data point in the target data set j , calculate the accuracy score function value A, the specific calculation formula is:
[0112] ,
[0113] The F1 score refers to a statistical indicator, with a maximum value of 1 and a minimum value of 0;
[0114] Completeness score calculation unit: Calculate the proportion of missing values IT in the target data set from the total number of data points n in the target data set and the total number of missing values ITn in the target data set. The specific calculation formula is:
[0115] IT = ITn / n;
[0116] The proportion IF of data outliers in the target data set is calculated by the total number of data points n in the target data set and the total number of outliers IFn in the target data set. The specific calculation formula is:
[0117] IF=IFn / n;
[0118] The integrity score function value I is calculated by the time correlation trend index TR of the data in the target data set, the proportion of missing values IT of the data in the target data set, and the proportion of abnormal values IF of the data in the target data set. The specific calculation formula is:
[0119] ;
[0120] Timeliness score calculation unit: The time t of the set analysis and the time t of the jth data point in the target data set j, And the time range Δt covered by the target data set, calculate the data time fraction Tj of the target data set. The specific calculation formula is:
[0121] Tj=((tt j ) / Δt);
[0122] The time t of the analysis is set, and the time t of the jth data point in the target data set is recorded. j, As well as the time range Δt covered by the target data set, calculate the time factor Tf in the target data set. The specific calculation formula is:
[0123] Tf=exp(-Δt×(tt j ));
[0124] The timeliness score function value T is calculated by the time correlation trend index TR of the data in the target data set, the data time score Tj of the target data set, and the time factor Tf in the target data set. The specific calculation formula is:
[0125] ,
[0126] Where n represents the total number of data points in the target data set, and j represents the index of the data point;
[0127] Data quality output unit: transmits the calculated consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value to the comprehensive data quality assessment index calculation module.
[0128] The comprehensive data quality assessment index calculation module calculates the comprehensive data quality assessment index based on the consistency score function value, the accuracy score function value, the completeness score function value, and the timeliness score function value.
[0129] The specific calculation formula of the comprehensive data quality evaluation index calculation module for CAIT is calculated based on the consistency score function value C, the accuracy score function value A, the integrity score function value I, and the timeliness score function value T:
[0130] ,
[0131] Among them, V represents the value coefficient of data, DMC represents the data maintenance cost, α C , α A , α I , α T They are respectively expressed as the penalty factor for consistency score, the penalty factor for accuracy score, the penalty factor for completeness score, and the penalty factor for timeliness score.
[0132] It should be noted that the value coefficient of data is determined based on the potential benefits of the data; data maintenance costs include the costs of data cleaning, verification, updating and protection; the penalty factors for consistency score, accuracy score, completeness score and timeliness score reflect the negative impact on the comprehensive indicators when the data quality score is low, and are determined based on the business's tolerance for data quality and the severity of data quality issues.
[0133] The data quality judgment module is used to judge and monitor the quality of data, detect data quality issues and trigger data iteration strategies.
[0134] It should be noted that the specific processing process of the data quality judgment module for judging data quality is as follows: determine the data quality threshold based on the target business needs, data usage scenarios and historical data quality performance, and divide the comprehensive data quality assessment indicators into five levels: excellent, good, medium, poor and extremely poor; and establish a data quality monitoring mechanism. As the data environment changes and business needs evolve, the threshold range of the comprehensive data quality assessment indicators is regularly adjusted, checked and evaluated to promptly discover and handle new data quality issues; once the data quality judgment module detects that the comprehensive data quality assessment indicator is lower than the set minimum threshold, the data iteration strategy is initiated; it should be noted that the data iteration strategy includes iterative analysis of the data, determining the specific type and scope of data quality issues, and formulating data cleaning and repair strategies and performing automated processing based on the type and severity of the problem.
[0135] In this embodiment, the comprehensive data quality assessment index is divided into different levels, and a comprehensive data quality assessment index value above 0.8 is excellent, a comprehensive data quality assessment index value between 0.6-0.8 is good, a comprehensive data quality assessment index value between 0.4-0.6 is medium, a comprehensive data quality assessment index value between 0.2-0.4 is poor, and a comprehensive data quality assessment index value below 0.2 is extremely poor.
[0136] It should be noted that if the comprehensive data quality assessment index of the target dataset is in a good range, but the sub-indicators are low, it means that there are problems with the sub-indicators of the dataset and it needs to be focused on and improved; the sub-indicators represent consistency, accuracy, completeness, and timeliness.
[0137] In this embodiment, specific types of data quality problems include missing values, duplicate values, erroneous values, and outdated data; for missing value problems, mean filling, median filling, and regression filling methods are used; for duplicate value problems, an exact matching algorithm is used to remove them; for erroneous value problems, manual inspection and correction are performed; for outdated data problems, a data update mechanism is established to update the data regularly.
[0138] Data governance report output module, the specific output process of the data governance report is as follows: based on the judgment results of the data quality judgment module, a detailed data quality report is generated, including comprehensive data quality assessment indicators and specific values of each sub-indicator, showing the quality status of the data and data quality issues, pointing out the type, scope and severity of the problem; at the same time, high-quality data is output to the database to improve the reliability and value of the data.
[0139] It should be noted that the output report is presented in the form of tables, charts, and text, so that the data quality information is clear at a glance; for data integrity issues, a bar chart is used to show the complete record ratio coefficient of different data sets; for timeliness issues, a line chart is used to show the change in data update frequency over time.
[0140] It should be noted that the data governance report output module will feed back data quality information to data governance staff to understand the effectiveness of data governance measures and adjust governance strategies in a timely manner; if data quality issues are not improved within a period of time, the staff will analyze the reasons based on the report of the data governance report output module and take more effective governance measures.
[0141] In this embodiment, the sales department uses the high-quality customer data provided by the data governance report output module to conduct market analysis and sales strategy formulation, while the finance department uses the same data to prepare and analyze financial statements, thereby improving the overall operational efficiency of the enterprise.
[0142] The processor is configured to execute instructions stored in the database.
[0143] In this example, both the user interface and the network interface are used to communicate with other devices.
[0144] The database is used to store high-quality data and manage collected data, and provide data retrieval and update functions.
[0145] As attached Figure 2 A high-quality data management method based on data governance is shown, comprising the following steps:
[0146] S1: used to collect data in various data formats from different business data sources, and encapsulate the collected data into data sets and transmit them to the business data conversion module;
[0147] S2: used to pre-process the transmitted data sets, use conditional filtering and regular expression matching methods to convert the data in each data set into an existing data set in the target business field in a unified format, and transmit it to the data resource integration module;
[0148] S3: It is used to integrate the existing data sets and data sources in the target business field, classify the data according to its nature and purpose, select appropriate storage technologies to store the classified data sets, and send the data in different storage technologies to the data analysis module;
[0149] S4: used to analyze the transmitted data set and calculate the consistency score function value, the accuracy score function value, the completeness score function value, and the timeliness score function value, and transmit them to the comprehensive data quality assessment indicator calculation module;
[0150] S5: Calculate the comprehensive data quality assessment index based on the consistency score function value, the accuracy score function value, the completeness score function value, and the timeliness score function value;
[0151] S6: used to judge and monitor data quality, detect data quality issues and trigger data iteration strategies;
[0152] S7: Used to output processed and analyzed data to users in a report format through data visualization tools, so as to quickly make data governance decisions.
[0153] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;
[0154] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A high-quality data management system based on data governance, characterized in that: include: Business data collection module: used to collect data in various data formats from different business data sources, and encapsulate the collected data into data sets and transmit them to the business data conversion module; Business data conversion module: used to pre-process the transmitted data sets, use conditional filtering and regular expression matching methods to convert the data in each data set into the existing data set of the target business field in a unified format, and transmit it to the data resource integration module; Data resource integration module: used to integrate existing data sets and data sources in the target business field, classify them according to the nature and purpose of the data, select appropriate storage technologies to store the classified data sets, and send the data sets in different storage technologies to the data quality basic assessment module; Data quality basic evaluation module: used to analyze the transmitted data set and calculate the consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value, and transmit them to the comprehensive data quality evaluation index calculation module; The data quality basic assessment module includes a data quality receiving unit, a data feature extraction unit, a consistency score calculation unit, an accuracy score calculation unit, a completeness score calculation unit, a timeliness score calculation unit, and a data quality output unit. The specific processing process of the basic assessment is as follows: Data quality receiving unit: receiving the target data set transmitted by the data integration module; Data feature extraction unit: Verify that the target data set has been preprocessed, extract the numerical data in the target data set, record the total number of data points in the target data set, the value of the jth data point in the target data set, the time when the jth data point in the target data set was recorded, the time set for analysis, and the time range covered by the target data set, which are denoted as n, x, respectively. j ,t j , t, and Δt; the total number of missing values and the total number of outliers in the target data set are counted by machine learning algorithms, which are recorded as ITn and IFn respectively; Calculate the average value of all data points in the target data set, the standard deviation of all data points in the target data set, the data trend index in the target data set, and the data change rate in the target data set, which are denoted as μ, σ, β0, and β1 respectively; further, the specific calculation formula for the average value of all data points in the target data set is: ; The specific calculation formula for the standard deviation of all data points in the target data set is: ; The specific calculation formula for the data trend indicator in the target data set is: ; The specific calculation formula for the data change rate in the target data set is: ; Consistency score calculation unit: The total number of data points n in the target data set, the value x of the jth data point in the target data set j , the average value μ of all data points in the target data set, and the standard deviation σ of all data points in the target data set, calculate the data cohesion index MO of the target data set. The specific calculation formula is: , Where j represents the index of the data set; the time correlation trend index TR of the data in the target data set is calculated by the data trend index β0 in the target data set, the data change rate β1 in the target data set, the set analysis time t, and the data cohesion index MO of the target data set. The specific calculation formula is: TR = β0 + β1 × t + MO; The data cohesion index MO of the target data set, the time correlation trend index TR of the data in the target data set, and the value x of the jth data point in the target data set j , calculate the consistency score function value C, the specific calculation formula is: , Where n represents the total number of data points in the target data set, and j represents the index of the data point; Accuracy score calculation unit: The data cohesion index MO of the target data set, the F1 score, and the value x of the jth data point in the target data set j , calculate the accuracy score function value A, the specific calculation formula is: , The F1 score refers to a statistical indicator, with a maximum value of 1 and a minimum value of 0; Completeness score calculation unit: Calculate the proportion of missing values IT in the target data set from the total number of data points n in the target data set and the total number of missing values ITn in the target data set. The specific calculation formula is: IT = ITn / n; The proportion IF of data outliers in the target data set is calculated by the total number of data points n in the target data set and the total number of outliers IFn in the target data set. The specific calculation formula is: IF=IFn / n; The integrity score function value I is calculated by the time correlation trend index TR of the data in the target data set, the proportion of missing values IT of the data in the target data set, and the proportion of abnormal values IF of the data in the target data set. The specific calculation formula is: ; Timeliness score calculation unit: The time t of the set analysis and the time t of the jth data point in the target data set j, And the time range Δt covered by the target data set, calculate the data time fraction Tj of the target data set. The specific calculation formula is: Tj=((t-t j ) / Δt); The time t of the analysis is set, and the time t of the jth data point in the target data set is recorded. j, As well as the time range Δt covered by the target data set, calculate the time factor Tf in the target data set. The specific calculation formula is: Tf=exp(-Δt×(t-t j )); The timeliness score function value T is calculated by the time correlation trend index TR of the data in the target data set, the data time score Tj of the target data set, and the time factor Tf in the target data set. The specific calculation formula is: , Where n represents the total number of data points in the target data set, and j represents the index of the data point; Data quality output unit: transmits the calculated consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value to the comprehensive data quality assessment index calculation module; Comprehensive data quality assessment index calculation module: calculates comprehensive data quality assessment index based on consistency score function value, accuracy score function value, completeness score function value, and timeliness score function value; The specific calculation formula of the comprehensive data quality evaluation index calculation module to calculate the comprehensive data quality evaluation index CAIT based on the consistency score function value C, the accuracy score function value A, the integrity score function value I, and the timeliness score function value T is: , Among them, V represents the value coefficient of data, DMC represents the data maintenance cost, α C , α A , α I , α T They are respectively expressed as the penalty factor of consistency score, the penalty factor of accuracy score, the penalty factor of completeness score, and the penalty factor of timeliness score; Data quality judgment module: used to judge and monitor data quality, detect data quality issues and trigger data iteration strategies; Data governance report output module: used to output processed and analyzed data to users in the form of reports through data visualization tools, so as to quickly make data governance decisions; Database: used to store high-quality data, store instructions, manage collected data, and provide data retrieval and update functions; Processor: used to execute the instructions stored in the database.
2. A high-quality data management system based on data governance according to claim 1, characterized in that: The business data collection module includes a structured data collection unit, a semi-structured data collection unit, an unstructured data collection unit, a metadata collection unit, and a business data output unit, wherein the structured data collection unit uses ETL tools to collect structured data in the enterprise database; the semi-structured data collection unit comprehensively uses API interfaces and crawler technology to collect semi-structured business information and interim results in business processes; the unstructured data collection unit uses a streaming data processing framework to collect unstructured data without a fixed structure on social media; the metadata collection unit records the data source by establishing metadata fields; and the business data output unit encapsulates the data collected by different units into data sets and transmits them to the business data conversion module.
3. A high-quality data management system based on data governance according to claim 1, characterized in that: The business data conversion module preprocesses, including removing duplicates, filling missing values, and format standardization; further, using an exact matching algorithm to quickly identify duplicate records in a data set and remove them according to system-set rules; using mean filling, median filling, and mode filling methods to process missing values in a data set; formulating a unified data format standard for various data types in a data set; wherein the system-set rules include determining a unique identifier, a comparison method, and a removal strategy; further, for different types of data, clearly defining a field combination that can uniquely identify a record; when using an exact matching algorithm, strictly comparing whether the values of the unique identifier fields are completely consistent, if the unique identifier field values of two records are exactly the same, they are marked as duplicates; when duplicates are determined, merging the information of the duplicates, and merging different field values of the duplicates.
4. A high-quality data management system based on data governance according to claim 1, characterized in that: The data resource integration module includes a data resource integration receiving unit, a data combing and classification unit, a data integration unit and a data resource integration output unit; wherein the data resource integration receiving unit is used to receive the existing data set of the target business field in a unified format; the data combing and classification unit integrates and combs the existing data set of the target business field with the data source, and classifies the data according to the nature and purpose of the data; the data integration unit selects a suitable storage technology for storage of the classified data set according to business needs; the data resource integration output unit sends the target data set in different storage technologies to the data quality basic assessment module.
5. A high-quality data management system based on data governance according to claim 1, characterized in that: The data quality judgment module determines the specific processing process of data quality as follows: determine the data quality threshold based on the target business needs, data usage scenarios and historical data quality performance, and divide the comprehensive data quality assessment indicators into five levels: excellent, good, medium, poor and extremely poor; and establish a data quality monitoring mechanism. As the data environment changes and business needs evolve, the threshold range of the comprehensive data quality assessment indicators is regularly adjusted, checked and evaluated to promptly discover and handle new data quality issues; once the data quality judgment module detects that the comprehensive data quality assessment indicator is lower than the set minimum threshold, the data iteration strategy is initiated; it should be noted that the data iteration strategy includes iterative analysis of the data, determining the specific type and scope of data quality issues, and formulating data cleaning and repair strategies and performing automated processing based on the type and severity of the problem.
6. A high-quality data management system based on data governance according to claim 1, characterized in that: The data governance report output module and the specific output process of the data governance report are as follows: based on the judgment results of the data quality judgment module, a detailed data quality report is generated, including comprehensive data quality assessment indicators and specific values of each sub-indicator, which displays the quality status of the data and points out the type, scope and severity of the data quality issues; at the same time, high-quality data is output to the database to improve the reliability and value of the data.
Citation Information
Patent Citations
Data quality management method and system based on self-optimization subsystem
CN118427187A
Method and system for data quality assessment
WO2015039693A1
