Full-life-cycle data governance method and system based on data assets

Abstract

The invention discloses a full life cycle data governance method and system based on data assets. Comprising a data asset inventory and identification module, a data architecture design and organization module, a data standard formulation and management module, a data model design and management module, a data quality management and auditing module, a data security management module, a data asset panoramic display module and a data standard definition and quality auditing tool module. The data asset checking and identifying module is used for comprehensively checking all data assets in an enterprise and identifying the source, the type, the use condition and the storage mode of the data; the data assets are checked and organized, the functions of data asset panorama, data map, data standard, data model, data quality and the like are provided, effective management of the data assets is achieved, in addition, a data standard definition tool is further provided, visual modeling and configuration rules are provided for quality auditing, and the data assets can be effectively managed. And the data quality is further improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for full lifecycle data governance based on data assets. Background Technology

[0002] Data assets refer to valuable data resources owned by an organization or individual. These include various types of data, such as customer information, sales data, financial data, product information, and market research data. Data assets are of significant value to organizations because they can be used to gain insights into business trends, support decision-making, improve efficiency, and create business value.

[0003] Data assets are valuable data resources owned by organizations or individuals that can support business decisions and create commercial value. For organizations, effectively managing and protecting data assets is crucial.

[0004] With the acceleration of digital transformation, data has become one of the most important strategic assets for enterprises. However, the explosive growth of data volume, the diversification of data sources, and the uneven quality of data have brought enormous challenges to enterprise data management. Traditional data management methods are often limited to local or specific scenarios and lack systematic governance of the entire lifecycle of data assets, resulting in problems such as data silos, low data quality, and data security risks. Moreover, traditional data governance methods are usually targeted at specific business scenarios or systems, lacking a global perspective and failing to cover the entire lifecycle of data assets. The identification, classification, and standardization of data assets often rely on manual operations, which are inefficient and prone to errors. To address these issues, we propose a data governance method and system based on the entire lifecycle of data assets. Summary of the Invention

[0005] The purpose of this invention is to provide a data governance method and system based on the entire lifecycle of data assets, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, a data governance system based on the entire lifecycle of data assets is provided, including a data asset inventory and identification module, a data architecture design and organization module, a data standard formulation and management module, a data model design and management module, a data quality management and audit module, a data security management module, a data asset panoramic display module, and a data standard definition and quality audit tool module.

[0007] The data asset inventory and identification module is used to conduct a comprehensive inventory of all data assets within the enterprise, and to identify the source, type, usage and storage method of the data;

[0008] The data architecture design and organization module is used to build the enterprise's data infrastructure and integrate all data assets into a unified data architecture;

[0009] The data standard setting and management module is used to define and implement data standards, and to standardize data formats, content, and naming conventions;

[0010] The data model design and management module is used to design data models;

[0011] The data quality management and auditing module is used for data quality monitoring and auditing.

[0012] The data security management module is used to implement data security management;

[0013] The data asset panoramic display module is used to provide a panoramic display of data assets, show the overall situation of various data assets, and support real-time monitoring and analysis of data assets.

[0014] The data standard definition and quality audit tool module provides a tool platform that supports visual modeling and rule configuration.

[0015] Preferably, the data asset inventory and identification module includes an automatic data asset scanning unit and a data asset metadata management unit;

[0016] The automatic data asset scanning unit is used to automatically scan all data sources and identify the type, source, storage location, and usage of the data;

[0017] The data asset metadata management unit is used to record metadata for each data asset and provide detailed information about the data asset.

[0018] Preferably, the data architecture design and organization module includes a data storage structure design unit and a data access and integration unit;

[0019] The data storage structure design unit is used to design and organize the data storage structure, supporting efficient data management and querying.

[0020] The data access and integration unit is used to design the data access layer.

[0021] Preferably, the data standard setting and management module includes a data format and naming rules unit and a data standard implementation and review unit;

[0022] The data format and naming rules unit is used to define data format specifications and naming conventions;

[0023] The data standard implementation and review unit is used to implement and review compliance with data standards.

[0024] Preferably, the data model design and management module includes a data model creation and optimization unit and a data model version management unit;

[0025] The data model creation and optimization unit is used to design and optimize the data model;

[0026] The data model version management unit is used to manage the versions and updates of the data model.

[0027] Preferably, the data quality management and audit module includes a data quality monitoring and reporting unit and a data quality problem tracking and repair unit;

[0028] The data quality monitoring and reporting unit is used to monitor data quality in real time and generate quality reports;

[0029] The data quality problem tracking and repair unit tracks and repairs data quality problems.

[0030] Preferably, the data security management module includes a data access control and permission management unit and a data encryption and backup management unit;

[0031] The data access control and permission management unit is used to manage data access permissions;

[0032] The data encryption and backup management unit is used to encrypt and store data, transmit data in an encrypted manner, and perform regular backups.

[0033] Preferably, the data asset panoramic display module includes a data asset health status display unit and a data asset usage display unit;

[0034] The data asset health status display unit is used to display the health status of data assets, including data availability, update frequency, and quality.

[0035] The data asset usage display unit is used to display information on the frequency of use, users, and purposes of data assets.

[0036] Preferably, the data standard definition and quality audit tool module includes a standard definition and visualization modeling tool unit and a quality audit rule configuration and execution unit;

[0037] The standard definition and visual modeling tool unit is used to provide visual modeling tools to help define data standards;

[0038] The quality audit rule configuration and execution unit is used to support users in configuring and executing data quality audit rules.

[0039] The method of using a data asset-based full lifecycle data governance system as described in any of the above includes the following steps:

[0040] S1. Inventory data assets, identify data assets, and classify them;

[0041] S2. Build a data infrastructure and organize data assets into the data architecture;

[0042] S3. Establish data standards to standardize data format and content;

[0043] S4. Design a data model to support the storage and management of data assets;

[0044] S5. Implement data quality management and conduct data quality audits;

[0045] S6. Implement data security management to protect the security of data assets;

[0046] S7 provides a panoramic view of data assets, showcasing the overall status of data assets;

[0047] S8. Provides a data map to show the distribution of data assets;

[0048] S9 provides data standard definition tools, supporting visual modeling and rule configuration for quality auditing.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. This invention, through a data asset panoramic display module, enables enterprises to intuitively understand the distribution, usage, and health status of all data assets, breaking down data silos and improving data visibility and transparency.

[0051] 2. Through the data quality monitoring and reporting unit, the system can detect data quality problems (such as missing values, duplicate data, inconsistent data, etc.) in real time and generate quality reports to help enterprises fix problems in a timely manner; the quality audit rule configuration and execution unit supports user-defined audit rules and executes them through automated processes, which significantly improves the reliability and consistency of data quality.

[0052] 3. Through the data format and naming rules unit, the system can formulate unified data standards to ensure data consistency across different systems and departments. The data standard implementation and review unit can automatically check the compliance with data standards to ensure their implementation.

[0053] 4. The data storage structure design unit of this invention supports enterprises in designing flexible data architectures (such as data lakes, data warehouses, etc.) according to business needs, thereby improving the efficiency of data storage and retrieval; the data model creation and optimization unit helps enterprises design data models that meet business needs, and realizes dynamic updates and optimization of models through the version management unit.

[0054] 5. Through the data access control and permission management unit, the system can finely manage data access permissions and prevent unauthorized access; the data encryption and backup management unit ensures the security of data during storage and transmission, and prevents data loss through regular backups; the system can help enterprises meet the requirements of data security and privacy protection regulations such as GDPR, and reduce compliance risks.

[0055] 6. The automatic data asset scanning unit of this invention can automatically identify and classify data assets, reduce manual operations, and improve efficiency; through AI and machine learning technologies, the system can automatically detect data quality problems and provide repair suggestions, thereby improving the level of intelligence in data governance.

[0056] 7. By integrating functions such as data asset inventory, standardization, quality management, and security management, this invention provides a unified governance platform, reducing the complexity and cost of integrating multiple tools; the application of automated tools and intelligent technologies reduces the need for manual intervention and lowers the human resource costs of data governance.

[0057] 8. The standard definition and visualization modeling tool unit of this invention provides a user-friendly interface, supporting users to quickly define data standards and models through drag and drop, thus lowering the technical threshold; the data asset panoramic display module and data map function provide an intuitive visualization interface to help users quickly understand the overall situation of data assets. Attached Figure Description

[0058] Figure 1 This is a system flowchart of the present invention;

[0059] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Please see Figure 1This invention provides a technical solution: a data governance system based on the entire lifecycle of data assets, including a data asset inventory and identification module, a data architecture design and organization module, a data standard formulation and management module, a data model design and management module, a data quality management and audit module, a data security management module, a data asset panoramic display module, and a data standard definition and quality audit tool module.

[0062] The Data Asset Inventory and Identification module is used to conduct a comprehensive inventory of all data assets within the enterprise, identifying the data's source, type, usage, and storage method; the Data Architecture Design and Organization module is used to build the enterprise's data infrastructure, integrating all data assets into a unified data architecture; the Data Standards Formulation and Management module is used to define and implement data standards, standardizing data formats, content, and naming conventions; the Data Model Design and Management module is used to design data models; the Data Quality Management and Audit module is used to monitor and audit data quality; the Data Security Management module is used to implement data security management; the Data Asset Panoramic Display module provides a panoramic view of data assets, showcasing the overall status of various data assets and supporting real-time monitoring and analysis; the Data Standards Definition and Quality Audit Tool module provides a tool platform that supports visual modeling and rule configuration.

[0063] It should be noted that the data asset inventory and identification module comprehensively scans and identifies data assets to form a data asset catalog. The data architecture design and organization module builds a unified data infrastructure to integrate data assets. The data standard setting and management module defines and implements data standards to ensure data consistency. The data model design and management module designs and optimizes data models to ensure data consistency. The data quality management and audit module can monitor data quality in real time and fix data problems. The data asset panoramic display module displays the health status and usage of data assets. The data security management module implements data access control and encrypted backup to ensure data security. The data asset panoramic display module displays the health status and usage of data assets. The data standard definition and quality audit tool module provides visualization tools to support data standard definition and quality audit.

[0064] The data asset inventory and identification module includes an automatic data asset scanning unit and a data asset metadata management unit. The automatic data asset scanning unit is used to automatically scan all data sources and identify the type, source, storage location, and usage of the data. The data asset metadata management unit is used to record metadata for each data asset and provide detailed information about the data asset.

[0065] It should be noted that during operation, the system automatically scans all data sources within the enterprise to identify the type, source, storage location, and usage of data assets. It retrieves basic information about the data assets from the automatic data asset scanning unit and supplements metadata such as description, owner, creation time, and update time through manual input or automated rules. This metadata is then stored in the metadata management system, supporting rapid querying and updates. The system establishes relationships between data assets (such as parent-child relationships and dependencies), forming a panoramic view of the data assets. The automatic data asset scanning unit connects to the data source, scans and extracts basic data information, and automatically identifies the data type, source, and purpose based on the scan results. The data asset metadata management unit records metadata for each data asset, forming a data asset catalog. Through log analysis, it statistically analyzes the frequency and popularity of data usage, stores the metadata in the metadata management system, and displays detailed information about the data assets through the panoramic display module.

[0066] For structured data (such as table data), the data volume can be calculated using the number of records (rows) and the number of fields (columns), as follows:

[0067] Data volume = number of rows × number of columns × average field size

[0068] For unstructured data (such as files), the data size is calculated directly from the file size.

[0069] Calculation using frequency:

[0070]

[0071] The popularity of data is calculated by combining access frequency and data volume:

[0072] Data popularity = usage frequency × log (data volume);

[0073] Metadata completeness is calculated as the ratio of the number of populated metadata fields to the total number of fields, using the following formula:

[0074]

[0075] The update frequency of metadata is calculated by the update time interval, using the following formula:

[0076]

[0077] The data architecture design and organization module includes a data storage structure design unit and a data access and integration unit. The data storage structure design unit is used to design and organize the data storage structure to support efficient data management and querying. The data access and integration unit is used to design the data access layer.

[0078] It should be noted that the workflow is as follows: Based on the type of data assets and business needs, select an appropriate data storage model (such as relational databases, NoSQL databases, data lakes, etc.); based on data access patterns and data volume, design data partitioning strategies (such as time-based partitioning, region-based partitioning); based on query requirements, design data indexes (such as primary key indexes, unique indexes, composite indexes); and based on the characteristics of the data, select an appropriate data compression algorithm. In the data access and integration unit, based on business needs, design data access interfaces; based on data access patterns, design data caching strategies; based on data sources and uses, design data integration schemes; and based on data security policies, design data access permission control mechanisms.

[0079] Data storage capacity is calculated using data volume, compression ratio, and replication factor:

[0080] Storage capacity = Data volume × Compression ratio × Replication factor

[0081] The compression ratio is the ratio of the compressed data size to the original data size, and the replication factor is the number of data replicas.

[0082] Query performance is estimated using the data scan range and index coverage:

[0083]

[0084] Among them, the data scan range is the amount of data that needs to be scanned in the query, and the index coverage rate is the proportion of query conditions covered by the index;

[0085] Cache hit rate is calculated by combining the number of cache hits with the total number of accesses.

[0086]

[0087] Data access latency is calculated using cache hit rate, database query time, and network latency.

[0088] Access latency = (1 - cache hit rate) × database query time + network latency;

[0089] Data throughput is calculated using the number of concurrent accesses and the time per access:

[0090]

[0091] The data standard development and management module includes a data format and naming rules unit and a data standard implementation and review unit. The data format and naming rules unit is used to develop data format specifications and naming conventions. The data standard implementation and review unit is used to implement and review compliance with data standards.

[0092] It should be noted that the data format and naming rules unit formulates data format specifications and naming conventions to form data standard documents. The data standard implementation and review unit publishes the data standard documents to the data governance platform and implements the standards through automated tools. It regularly reviews data assets to check whether the data format and naming comply with the standards, corrects data that does not comply with the standards, ensures the consistency of data format and naming, and generates a data standard compliance report, which shows the format compliance rate, naming compliance rate, and correction rate.

[0093] Data format compliance is calculated as the ratio of data volume conforming to the format specifications to the total data volume:

[0094]

[0095] Data naming compliance is calculated as the ratio of data volume conforming to the naming conventions to the total data volume:

[0096]

[0097] Data standard compliance rate is calculated as the ratio of the amount of data that conforms to the standard to the total amount of data.

[0098]

[0099] The data standard correction rate is calculated as the ratio of the amount of corrected data to the amount of non-compliant data.

[0100]

[0101] The data model design and management module includes a data model creation and optimization unit and a data model version management unit; the data model creation and optimization unit is used to design and optimize data models; the data model version management unit is used to manage the versions and updates of data models.

[0102] The data quality management and audit module includes a data quality monitoring and reporting unit and a data quality problem tracking and repair unit. The data quality monitoring and reporting unit is used to monitor data quality in real time and generate quality reports. The data quality problem tracking and repair unit tracks and repairs data quality problems.

[0103] It should be noted that data quality indicators include data accuracy, completeness, consistency, timeliness, and uniqueness. The data quality monitoring and reporting unit acquires data in real time, checks data quality using preset quality standards, and generates quality reports periodically or as needed based on the results of real-time monitoring. The data quality problem tracking and repair unit tracks data quality problems found in monitoring and reports in detail, identifies the location, type, and possible root causes of problematic data items. Once a problem is identified, the repair process will be carried out according to different repair methods. After repair, the repair results must be verified to ensure that the problem has been completely resolved and to prevent the problem from recurring.

[0104] In data quality monitoring, the formula for data accuracy is:

[0105]

[0106] The accurate data count is used to measure the correctness of data. The accurate data count refers to the number of data items that are consistent with the true value. The data count is the total number of monitored data.

[0107] To measure whether data is missing or incomplete, the number of complete data items refers to the number of valid data items that are not missing.

[0108] Measuring whether data is updated and acquired on time;

[0109] To determine if there are duplicate data;

[0110]

[0111] To measure the effectiveness of the repair, reflecting the resolution of the problem after the repair;

[0112]

[0113] This reflects the improvement in accuracy after data repair;

[0114]

[0115] Measure the improvement effect after fixing data consistency issues.

[0116] The data security management module includes a data access control and permission management unit and a data encryption and backup management unit. The data access control and permission management unit is used to manage data access permissions. The data encryption and backup management unit is used to encrypt and store data and transmit it, and to perform regular backups.

[0117] It's important to note that the data security management module ensures data security through two key units. First, the data access control and permissions management unit uses authentication and permission allocation mechanisms to control who can access the data and their authorized scope, ensuring that only authorized users can access and manipulate sensitive data. Second, the data encryption and backup management unit uses encryption technology to protect the confidentiality of data during storage and transmission, preventing data leakage or tampering, and regularly backs up data to ensure recovery in case of loss or damage. Together, these two components, through preventing unauthorized access, encryption protection, and backup and recovery mechanisms, ensure the security, integrity, and availability of data.

[0118] The data asset panoramic display module includes a data asset health status display unit and a data asset usage display unit. The data asset health status display unit is used to display the health status of data assets, including data availability, update frequency, and quality. The data asset usage display unit is used to display the usage frequency, users, and purpose information of data assets.

[0119] It's important to note that the Data Asset Health Status Display unit provides visualization of data health status by monitoring data availability, update frequency, and quality, helping managers promptly identify potential quality issues or data aging. The Data Asset Usage Display unit focuses on data usage, showcasing the frequency of data asset use, users, and specific applications, providing decision-makers with data utilization efficiency and value assessments. Together, these two units help organizations comprehensively understand the health status and actual usage of their data assets, thereby optimizing the management and application of data resources.

[0120] The data standard definition and quality audit tool module includes a standard definition and visual modeling tool unit and a quality audit rule configuration and execution unit. The standard definition and visual modeling tool unit provides visual modeling tools to help define data standards. The quality audit rule configuration and execution unit supports users in configuring and executing data quality audit rules.

[0121] It should be noted that the Standards Definition and Visual Modeling Tools unit provides visual modeling tools to help users intuitively define data standards, including data formats, naming rules, and structural requirements, to ensure data consistency and standardization. The Quality Audit Rule Configuration and Execution unit allows users to configure and execute data quality audit rules, automatically checking data integrity, accuracy, consistency, and other quality indicators to help identify and correct data quality issues. These two units work together to ensure the effective implementation of data standards and continuous quality monitoring, improving data reliability and usability.

[0122] Please see Figure 2The method of using a data asset-based full lifecycle data governance system as described in any of the above includes the following steps:

[0123] S1. Inventory data assets, identify data assets, and classify them;

[0124] S2. Build a data infrastructure and organize data assets into the data architecture;

[0125] S3. Establish data standards to standardize data format and content;

[0126] S4. Design a data model to support the storage and management of data assets;

[0127] S5. Implement data quality management and conduct data quality audits;

[0128] S6. Implement data security management to protect the security of data assets;

[0129] S7 provides a panoramic view of data assets, showcasing the overall status of data assets;

[0130] S8. Provides a data map to show the distribution of data assets;

[0131] S9 provides data standard definition tools, supporting visual modeling and rule configuration for quality auditing.

[0132] In summary, this invention provides functions such as data asset overview, data map, data standards, data model, data quality, and data security by inventorying and organizing data assets, thus achieving effective management of data assets. In addition, it provides a data standard definition tool, offering visual modeling and configuration rules for quality auditing, further improving data quality.

[0133] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data governance system based on the entire lifecycle of data assets, characterized in that, It includes modules for data asset inventory and identification, data architecture design and organization, data standard setting and management, data model design and management, data quality management and auditing, data security management, data asset panoramic display, and data standard definition and quality auditing tools. The data asset inventory and identification module is used to conduct a comprehensive inventory of all data assets within the enterprise, and to identify the source, type, usage and storage method of the data; The data architecture design and organization module is used to build the enterprise's data infrastructure and integrate all data assets into a unified data architecture; The data standard setting and management module is used to define and implement data standards, and to standardize data formats, content, and naming conventions. The data model design and management module is used to design data models; The data quality management and audit module is used for data quality monitoring and auditing. The data security management module is used to implement data security management; The data asset panoramic display module is used to provide a panoramic display of data assets, show the overall situation of various data assets, and support real-time monitoring and analysis of data assets. The data standard definition and quality audit tool module provides a tool platform that supports visual modeling and rule configuration.

2. The data governance system based on the full lifecycle of data assets according to claim 1, characterized in that, The data asset inventory and identification module includes an automatic data asset scanning unit and a data asset metadata management unit. The automatic data asset scanning unit is used to automatically scan all data sources and identify the type, source, storage location, and usage of the data; The data asset metadata management unit is used to record metadata for each data asset and provide detailed information about the data asset.

3. A data governance system based on the full lifecycle of data assets according to claim 1, characterized in that, The data architecture design and organization module includes a data storage structure design unit and a data access and integration unit. The data storage structure design unit is used to design and organize the data storage structure, supporting efficient data management and querying. The data access and integration unit is used to design the data access layer.

4. A data governance system based on the full lifecycle of data assets according to claim 1, characterized in that, The data standard setting and management module includes a data format and naming rules unit and a data standard implementation and review unit; The data format and naming rules unit is used to define data format specifications and naming conventions; The data standard implementation and review unit is used to implement and review compliance with data standards.

5. A data governance system based on the full lifecycle of data assets according to claim 1, characterized in that, The data model design and management module includes a data model creation and optimization unit and a data model version management unit; The data model creation and optimization unit is used to design and optimize the data model; The data model version management unit is used to manage the versions and updates of the data model.

6. A data governance system based on the full lifecycle of data assets according to claim 1, characterized in that, The data quality management and audit module includes a data quality monitoring and reporting unit and a data quality problem tracking and repair unit; The data quality monitoring and reporting unit is used to monitor data quality in real time and generate quality reports; The data quality problem tracking and repair unit tracks and repairs data quality problems.

7. A data governance system based on the entire lifecycle of data assets according to claim 1, characterized in that, The data security management module includes a data access control and permission management unit and a data encryption and backup management unit; The data access control and permission management unit is used to manage data access permissions; The data encryption and backup management unit is used to encrypt and store data, transmit data in an encrypted manner, and perform regular backups.

8. A data governance system based on the entire lifecycle of data assets according to claim 1, characterized in that, The data asset panoramic display module includes a data asset health status display unit and a data asset usage display unit; The data asset health status display unit is used to display the health status of data assets, including data availability, update frequency, and quality. The data asset usage display unit is used to display information on the frequency of use, users, and purposes of data assets.

9. A data governance system based on the entire lifecycle of data assets according to claim 1, characterized in that, The data standard definition and quality audit tool module includes a standard definition and visualization modeling tool unit and a quality audit rule configuration and execution unit; The standard definition and visual modeling tool unit is used to provide visual modeling tools to help define data standards; The quality audit rule configuration and execution unit is used to support users in configuring and executing data quality audit rules.

10. A method of using a data asset-based full lifecycle data governance system according to any one of claims 1-9, characterized in that, Includes the following steps: S1. Inventory data assets, identify data assets, and classify them; S2. Build a data infrastructure and organize data assets into the data architecture; S3. Establish data standards to standardize data format and content; S4. Design a data model to support the storage and management of data assets; S5. Implement data quality management and conduct data quality audits; S6. Implement data security management to protect the security of data assets; S7 provides a panoramic view of data assets, showcasing the overall status of data assets; S8. Provides a data map to show the distribution of data assets; S9 provides data standard definition tools, supporting visual modeling and rule configuration for quality auditing.

Images