Integrated data management method

Abstract

The invention provides an integrated data management method. The method comprises the following steps: S1, data access; s2, data organization; s3, data processing; s4, data management; and S5, data service. The integrated data management method provided by the invention is flexible: by providing an open data storage capability and opening and adapting a data storage component, a user can optimize according to the characteristics and requirements of each data warehouse level, so that the performance, expandability and cost effectiveness are improved, and the data management efficiency is improved. The flexible architecture design can meet diversified requirements in a data warehouse environment, by providing two data processing modes of dragging graphical processing and Sql writing, the requirements of a user can be better met, the flexibility and efficiency of data processing are improved, and the user selects a proper mode to carry out data processing according to own requirements and skills.

Description

Technical Field

[0001] The present invention relates to the field of data governance, and in particular to an integrated data governance method. Background Art

[0002] Data governance is a comprehensive methodology, which aims to ensure the effective management and utilization of internal and external data assets of an organization. If there is a lack of effective overall data governance architecture design or some data governance capabilities are missing, it will be difficult for the business layer to directly use big data to provide invisible value for the enterprise. At this time, big data and business are out of touch, resulting in a series of problems in the process of using big data, such as data unknowable, difficult to meet needs, and difficult to share data. This method builds an integrated data governance platform, accesses data assets, measures data quality, establishes a data resource directory, builds a data lineage map, and provides data service capabilities to comprehensively govern and understand data, formulate data governance and usage specifications, ensure data availability, security, and data quality, continuously monitor and evaluate data, measure data quality, and conduct risk assessment and processing. In addition, a technical system based on specifications is established, such as data standard management, data resource directory, metadata management, data quality management, data security management, etc., to strengthen the governance of data sources and data resource pools, and ensure data reliability, security control, and auditing of analytical applications.

[0003] The existing technology currently has the following disadvantages:

[0004] 1. Traditional data governance platforms use built-in data warehouses or data lakes to complete data collection and aggregation by connecting to external data sources. The built-in data storage components have certain limitations and performance issues. At the same time, if users specify data storage components, the system cannot adapt and meet the needs;

[0005] 2. During the data access process, there is a lack of complex processes such as early data exploration, automated table creation, incremental full access, and real-time reconciliation;

[0006] 3. Data processing is the process of achieving data standardization, including data extraction, cleaning, association, comparison, identification, objectification and other operations. The data processing function of traditional data governance platforms is single and closed, and customized development is required for specific data processing scenarios;

[0007] 4. The data quality verification rule settings cannot be open or some data quality verification rules cannot be built in, which cannot meet the ever-changing business and data needs.

[0008] Therefore, it is necessary to provide an integrated data governance method to solve the above technical problems. Summary of the invention

[0009] The present invention provides an integrated data governance method, which solves a series of problems encountered by enterprises in the process of using big data, such as data unknowability, difficult demand realization, and difficult data sharing.

[0010] In order to solve the above technical problems, the present invention provides an integrated data governance method, comprising the following steps:

[0011] S1, data access;

[0012] S2, data organization;

[0013] S3, data processing;

[0014] S4. Data governance;

[0015] S5. Data services.

[0016] Preferably, the data access in S1 accesses multi-source heterogeneous data in a standardized and modular manner through a unified data access mode; by exploring the data source, the basic information of the data source, data scale, data quality and other information are obtained, the data to be accessed is mastered in advance, and a dynamically configurable data access mode is provided to realize data access functions such as data policy configuration, task configuration, task scheduling, and breakpoint continuation; when the data is accessed, the access data source and the written target library are reconciled in real time. When the access task is completed, the data reconciliation can be used to check whether the amount of read data and the amount of accessed data are consistent; at the same time, the data resource directory is maintained through the configured data access task to realize the monitoring of data lineage;

[0017] Data access supports access to multiple data sources based on the diversity of data collection, multi-source heterogeneity, and complexity of access paths. It supports common relational databases, non-relational databases, interface data sources, and message middleware systems. It supports access to structured data files, semi-structured data files, and unstructured files from file systems such as FTP, SFTP, and HDFS.

[0018] Data access supports CDC for relational databases, including insert, update, and delete. It also supports full and incremental data collection based on logical incremental fields.

[0019] The data access system supports multiple data access methods including real-time streaming data access, full offline data access, and periodic batch data entry.

[0020] Preferably, the specific process of data access in S1 is as follows:

[0021] S11. First, configure the data source to be accessed through the system, including various types of data sources such as databases, interfaces, message middleware, and files;

[0022] S12. For database-type data sources, data exploration can be configured to obtain basic information of the data source, data scale, data quality, etc. in advance, and the explored data table can be established in advance in the accessed data warehouse;

[0023] S13. Configure the corresponding data access task through the system. You can select the target data warehouse and the corresponding target table to be accessed. It supports configuring whether the data access task performs data reconciliation.

[0024] S14. After the data access task is configured, you can configure the task scheduling strategy in the task list, supporting strategies such as single operation and periodic operation;

[0025] S15. After the task is successfully executed, the data access status can be reconciled through the data reconciliation module.

[0026] Preferably, the data access system verifies the accessed structured data, semi-structured data and unstructured data according to the data access strategy and transmission strategy, and outputs and stores the data into the original repository or distributes the data to other data warehouses according to the access output distribution strategy.

[0027] Preferably, the data organization module in S2 provides hierarchical management, subject management, and model management.

[0028] Preferably, the hierarchical management is constructed for data warehouse stratification, and flexible storage capacity requirements are achieved by binding different data storage components to the hierarchical levels;

[0029] Topic management supports the establishment of topics at each data level for thematic division of data;

[0030] Model management supports maintenance and management of the constructed data warehouse hierarchy, and supports online creation of new data models, binding of data tables, and association of data standards.

[0031] Preferably, the data processing in S3 is a process of achieving data standardization, including data extraction, cleaning, association, comparison, identification, objectification and other operations, supporting real-time and offline computing, supporting batch processing operations, and providing two data processing methods: drag-and-drop graphical processing and SQL writing. The system supports real-time and offline computing, and by providing two data processing methods: drag-and-drop graphical processing and SQL writing, it can help users choose the appropriate data processing method according to their preferences and skill levels;

[0032] Data processing complies with relevant standards and achieves standardization and deep integration of heterogeneous data through standardized processing procedures such as data extraction, cleaning, association, comparison, identification, objectification, and knowledge graph construction. Data processing adopts an open architecture and can dynamically orchestrate data processing processes in a unified, standard, and easy-to-expand manner.

[0033] Preferably, the data governance in S4 includes data standards, metadata management, data asset management, data quality, data security, data operation and maintenance management, etc.

[0034] Preferably, the effect of data aggregation and fusion is guaranteed by managing data resource catalogs, metadata, classification, blood relationship and other information, standardizing the data organization form, and controlling data quality at the same time, ensuring high-quality operation of the data throughout its life cycle through operation and maintenance means;

[0035] Data standard management is responsible for managing and maintaining the standards of the entire data center and providing standardized service support for other management activities;

[0036] Metadata management provides metadata support for lineage management and data asset management. It establishes lineage relationships, conducts lineage analysis, metadata-level hierarchical definitions, and resource catalog planning based on metadata management. Data asset management is responsible for data asset sharing, distribution, use, and management, including data resource catalogs, hierarchical classification, lifecycle management, and lineage management.

[0037] Data quality provides necessary guarantees for data quality by establishing assessment standards and management specifications, providing closed-loop processes and management functions such as data quality problem discovery, monitoring and tracking, and analysis and application. Data operation and maintenance management comprehensively grasps the overall situation and usage of data access, data flow, and data resources.

[0038] Data quality provides custom and configurable data quality rules and validation logic. Through open data validation, organizations can define data quality standards based on their own business needs and standards and evaluate the accuracy and completeness of data.

[0039] Preferably, the data service in S5 mainly provides data services and API services to the outside world for the internal data of the platform;

[0040] It mainly includes functional modules such as mounting and publishing, data subscription, subscription approval, service management, and credential management;

[0041] The main purpose of mounting and publishing is to mount internal data to the data resource directory in the form of API interface, so that users can subscribe to it on the data portal.

[0042] The data subscription function mainly allows users to log in to the data portal and subscribe to relevant data according to their own permissions;

[0043] The data administrator approves the subscription through the subscription approval module;

[0044] The service management function is mainly for subscription data, which is published as API services and supports the online and offline operation of API services.

[0045] Credential management mainly involves granting users authorization credentials and validity periods, and implementing user data permission control through AK and SK two-way verification.

[0046] Compared with related technologies, the integrated data governance method provided by the present invention has the following beneficial effects:

[0047] The present invention provides an integrated data governance method with flexibility: by providing open data storage capabilities and opening up data storage components for adaptation, users can optimize according to the characteristics and requirements of each data warehouse level to improve performance, scalability and cost-effectiveness. This flexible architecture design can meet the diverse needs in the data warehouse environment. By providing two data processing methods, namely, drag-and-drop graphical processing and SQL writing, it can better meet the needs of users and improve the flexibility and efficiency of data processing. Users can choose the appropriate method for data processing according to their needs and skills.

[0048] Efficiency: When data is accessed, data reconciliation is performed in real time between the access data source and the written target database. When the access task is completed, data reconciliation can be used to check whether the amount of data read and the amount of data accessed are consistent. This allows for fast and efficient data access and management, improving the efficiency and quality of data processing.

[0049] Openness: Providing custom and configurable data quality rules and validation logic, allowing users to freely add new data quality rules and validation logic as needed to adapt to changing business and data needs. This approach can help organizations better respond to data quality challenges, discover and solve data quality problems in a timely manner, and improve the flexibility and efficiency of data quality management. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a preferred embodiment of an integrated data management method provided by the present invention;

[0051] Figure 2 Flowchart for data access. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0053] Please refer to Figure 1 and Figure 2,in, Figure 1 A schematic diagram of a preferred embodiment of an integrated data management method provided by the present invention; Figure 2 A flow chart of data access. An integrated data governance method includes the following steps:

[0054] S1, data access;

[0055] S2, data organization;

[0056] S3, data processing;

[0057] S4. Data governance;

[0058] S5. Data services.

[0059] The data access in S1 uses a unified data access mode to access multi-source heterogeneous data in a standardized and modular manner; by exploring the data source, basic information of the data source, data scale, data quality and other information are obtained, the data to be accessed is mastered in advance, and a dynamically configurable data access mode is provided to realize data access functions such as data policy configuration, task configuration, task scheduling, and breakpoint continuation; when data is accessed, data reconciliation is performed on the access data source and the written target library in real time. When the access task is completed, the data reconciliation can be used to check whether the amount of read data is consistent with the amount of accessed data; at the same time, the data resource directory is maintained through the configured data access task to realize the monitoring of data lineage;

[0060] Data access supports access to multiple data sources based on the diversity of data collection, multi-source heterogeneity, and complexity of access paths. It supports common relational databases, non-relational databases, interface data sources, and message middleware systems. It supports access to structured data files, semi-structured data files, and unstructured files from file systems such as FTP, SFTP, and HDFS.

[0061] Data access supports CDC for relational databases, including insert, update, and delete. It also supports full and incremental data collection based on logical incremental fields.

[0062] The data access system supports multiple data access methods including real-time streaming data access, full offline data access, and periodic batch data entry.

[0063] The specific process of data access in S1 is as follows:

[0064] S11. First, configure the data source to be accessed through the system, including various types of data sources such as databases, interfaces, message middleware, and files;

[0065] S12. For database-type data sources, data exploration can be configured to obtain basic information of the data source, data scale, data quality, etc. in advance, and the explored data table can be established in advance in the accessed data warehouse;

[0066] S13. Configure the corresponding data access task through the system. You can select the target data warehouse and the corresponding target table to be accessed. It supports configuring whether the data access task performs data reconciliation.

[0067] S14. After the data access task is configured, you can configure the task scheduling strategy in the task list, supporting strategies such as single operation and periodic operation;

[0068] S15. After the task is successfully executed, the data access status can be reconciled through the data reconciliation module.

[0069] The data access system: checks the accessed structured data, semi-structured data, and unstructured data according to the data access strategy and transmission strategy, and outputs and stores them into the original library or distributes them to other data warehouses according to the access output distribution strategy.

[0070] The data organization module in S2 provides hierarchical management, subject management, and model management.

[0071] The hierarchical management is constructed for data warehouse stratification, and flexible storage capacity requirements are achieved by binding different data storage components to the hierarchical levels;

[0072] Topic management supports the establishment of topics at each data level for thematic division of data;

[0073] Model management supports maintenance and management of the constructed data warehouse hierarchy, and supports online creation of new data models, binding of data tables, and association of data standards.

[0074] The data processing in S3 is the process of achieving data standardization, including data extraction, cleaning, association, comparison, identification, objectification and other operations. It supports real-time and offline computing, supports batch processing operations, and provides two data processing methods: drag-and-drop graphical processing and SQL writing. The system supports real-time and offline computing, and by providing two data processing methods: drag-and-drop graphical processing and SQL writing, it can help users choose the appropriate data processing method according to their preferences and skill levels;

[0075] Data processing complies with relevant standards and achieves standardization and deep integration of heterogeneous data through standardized processing procedures such as data extraction, cleaning, association, comparison, identification, objectification, and knowledge graph construction. Data processing adopts an open architecture and can dynamically orchestrate data processing processes in a unified, standard, and easy-to-expand manner.

[0076] The data governance in S4 includes data standards, metadata management, data asset management, data quality, data security, data operation and maintenance management, etc.

[0077] By managing data resource catalogs, metadata, classification, blood relationships and other information, we can ensure the effect of data aggregation and integration, standardize the data organization form, and control data quality, and ensure high-quality operation of the data throughout its life cycle through operation and maintenance measures;

[0078] Data standard management is responsible for managing and maintaining the standards of the entire data center and providing standardized service support for other management activities;

[0079] Metadata management provides metadata support for lineage management and data asset management. It establishes lineage relationships, conducts lineage analysis, metadata-level hierarchical definitions, and resource catalog planning based on metadata management. Data asset management is responsible for data asset sharing, distribution, use, and management, including data resource catalogs, hierarchical classification, lifecycle management, and lineage management.

[0080] Data quality provides necessary guarantees for data quality by establishing assessment standards and management specifications, providing closed-loop processes and management functions such as data quality problem discovery, monitoring and tracking, and analysis and application. Data operation and maintenance management comprehensively grasps the overall situation and usage of data access, data flow, and data resources.

[0081] Data quality provides custom and configurable data quality rules and validation logic. Through open data validation, organizations can define data quality standards based on their own business needs and standards and evaluate the accuracy and completeness of data.

[0082] The data service in S5 mainly provides data services and API services to the external parties for the internal data of the platform;

[0083] It mainly includes functional modules such as mounting and publishing, data subscription, subscription approval, service management, and credential management;

[0084] The main purpose of mounting and publishing is to mount internal data to the data resource directory in the form of API interface, so that users can subscribe to it on the data portal.

[0085] The data subscription function mainly allows users to log in to the data portal and subscribe to relevant data according to their own permissions;

[0086] The data administrator approves the subscription through the subscription approval module;

[0087] The service management function is mainly for subscription data, which is published as API services and supports the online and offline operation of API services.

[0088] Credential management mainly involves granting users authorization credentials and validity periods, and implementing user data permission control through AK and SK two-way verification.

[0089] Key point 1: Open data storage

[0090] By providing open data storage capabilities and making data storage components open and adaptable, users can bind and use different data storage components in the data warehouse according to the needs of each data warehouse level;

[0091] Key point 2: Unified data access mode

[0092] By exploring the data source, establishing exploration tasks, automatically establishing the data tables that need to be accessed to the data access layer, and accessing multi-source heterogeneous data in a standardized and modular manner through a unified data access mode; by exploring the data source, obtaining basic information about the data source, data scale, data quality and other information, understanding the data to be accessed in advance, providing a dynamically configurable data access mode, and realizing data access functions such as data increment and full volume strategy configuration, task configuration, task scheduling, and breakpoint resumption; when data is accessed, real-time data reconciliation is performed between the access data source and the write target database. When the access task is completed, data reconciliation can be used to check whether the amount of data read and the amount of data accessed are consistent;

[0093] Key point 3: Support real-time and offline computing

[0094] By providing two data processing methods, drag-and-drop graphical processing and SQL writing, users can choose the appropriate data processing method according to their preferences and skill levels. Drag-and-drop graphical processing is usually more suitable for users who are not familiar with SQL language. They can complete data processing tasks through simple operations, while SQL writing is more suitable for users who are familiar with SQL language. They can write complex SQL statements to process data according to their needs. By providing these two data processing methods, users' needs can be better met, the flexibility and efficiency of data processing can be improved, and users can choose the appropriate method for data processing according to their needs and skills.

[0095] Key Point 4: Provide custom and configurable data quality rules and validation logic

[0096] Provides customizable and configurable data quality rules and validation logic. Through open data validation, organizations can define data quality standards based on their own business needs and standards, and evaluate the accuracy and completeness of data. Open data validation allows users to freely add new data quality rules and validation logic as needed to adapt to changing business and data needs. This approach can help organizations better respond to data quality challenges, discover and resolve data quality issues in a timely manner, and improve the flexibility and efficiency of data quality management.

[0097] Compared with related technologies, the integrated data governance method provided by the present invention has the following beneficial effects:

[0098] The present invention provides an integrated data governance method with flexibility: by providing open data storage capabilities and opening up data storage components for adaptation, users can optimize according to the characteristics and requirements of each data warehouse level to improve performance, scalability and cost-effectiveness. This flexible architecture design can meet the diverse needs in the data warehouse environment. By providing two data processing methods, namely, drag-and-drop graphical processing and SQL writing, it can better meet the needs of users and improve the flexibility and efficiency of data processing. Users can choose the appropriate method for data processing according to their needs and skills.

[0099] Efficiency: When data is accessed, data reconciliation is performed in real time between the access data source and the written target database. When the access task is completed, data reconciliation can be used to check whether the amount of data read and the amount of data accessed are consistent. This allows for fast and efficient data access and management, improving the efficiency and quality of data processing.

[0100] Openness: Providing custom and configurable data quality rules and validation logic, allowing users to freely add new data quality rules and validation logic as needed to adapt to changing business and data needs. This approach can help organizations better respond to data quality challenges, discover and solve data quality problems in a timely manner, and improve the flexibility and efficiency of data quality management.

[0101] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An integrated data governance method, characterized in that: The following steps are involved: S1, data access; S2, data organization; S3, data processing; S4. Data governance; S5. Data services.

2. The integrated data governance method according to claim 1, characterized in that: The data access in S1 uses a unified data access mode to access multi-source heterogeneous data in a standardized and modular manner; by exploring the data source, basic information of the data source, data scale, data quality and other information are obtained, the data to be accessed is mastered in advance, and a dynamically configurable data access mode is provided to realize data access functions such as data policy configuration, task configuration, task scheduling, and breakpoint continuation; when data is accessed, data reconciliation is performed on the access data source and the written target library in real time. When the access task is completed, the data reconciliation can be used to check whether the amount of read data is consistent with the amount of accessed data; at the same time, the data resource directory is maintained through the configured data access task to realize the monitoring of data lineage; Data access supports access to multiple data sources based on the diversity of data collection, multi-source heterogeneity, and complexity of access paths. It supports common relational databases, non-relational databases, interface data sources, and message middleware systems. It supports access to structured data files, semi-structured data files, and unstructured files from file systems such as FTP, SFTP, and HDFS. Data access supports CDC for relational databases, including insert, update, and delete. It also supports full and incremental data collection based on logical incremental fields. The data access system supports multiple data access methods including real-time streaming data access, full offline data access, and periodic batch data entry.

3. The integrated data governance method according to claim 1, characterized in that: The specific process of data access in S1 is as follows: S11. First, configure the data source to be accessed through the system, including various types of data sources such as databases, interfaces, message middleware, and files; S12. For database-type data sources, data exploration can be configured to obtain basic information of the data source, data scale, data quality, etc. in advance, and the explored data table can be established in advance in the accessed data warehouse; S13. Configure the corresponding data access task through the system. You can select the target data warehouse and the corresponding target table to be accessed. It supports configuring whether the data access task performs data reconciliation. S14. After the data access task is configured, you can configure the task scheduling strategy in the task list, supporting strategies such as single operation and periodic operation; S15. After the task is successfully executed, the data access status can be reconciled through the data reconciliation module.

4. The integrated data governance method according to claim 2, characterized in that: The data access system: checks the accessed structured data, semi-structured data, and unstructured data according to the data access strategy and transmission strategy, and outputs and stores them into the original library or distributes them to other data warehouses according to the access output distribution strategy.

5. The integrated data governance method according to claim 1, characterized in that: The data organization module in S2 provides hierarchical management, subject management, and model management.

6. The integrated data governance method according to claim 5, characterized in that: The hierarchical management is constructed for data warehouse stratification, and flexible storage capacity requirements are achieved by binding different data storage components to the hierarchical levels; Topic management supports the establishment of topics at each data level for thematic division of data; Model management supports maintenance and management of the constructed data warehouse hierarchy, and supports online creation of new data models, binding of data tables, and association of data standards.

7. The integrated data governance method according to claim 1, characterized in that: The data processing in S3 is the process of achieving data standardization, including data extraction, cleaning, association, comparison, identification, objectification and other operations. It supports real-time and offline computing, supports batch processing operations, and provides two data processing methods: drag-and-drop graphical processing and SQL writing. The system supports real-time and offline computing, and by providing two data processing methods: drag-and-drop graphical processing and SQL writing, it can help users choose the appropriate data processing method according to their preferences and skill levels; Data processing complies with relevant standards and achieves standardization and deep integration of heterogeneous data through standardized processing procedures such as data extraction, cleaning, association, comparison, identification, objectification, and knowledge graph construction. Data processing adopts an open architecture and can dynamically orchestrate data processing processes in a unified, standard, and easy-to-expand manner.

8. The integrated data governance method according to claim 1, characterized in that: The data governance in S4 includes data standards, metadata management, data asset management, data quality, data security, data operation and maintenance management, etc.

9. The integrated data governance method according to claim 8, characterized in that: By managing data resource catalogs, metadata, classification, blood relationships and other information, we can ensure the effect of data aggregation and integration, standardize the data organization form, and control data quality, and ensure high-quality operation of the data throughout its life cycle through operation and maintenance measures; Data standard management is responsible for managing and maintaining the standards of the entire data center and providing standardized service support for other management activities; Metadata management provides metadata support for lineage management and data asset management. It establishes lineage relationships, conducts lineage analysis, metadata-level hierarchical definitions, and resource catalog planning based on metadata management. Data asset management is responsible for data asset sharing, distribution, use, and management, including data resource catalogs, hierarchical classification, lifecycle management, and lineage management. Data quality provides necessary guarantees for data quality by establishing assessment standards and management specifications, providing closed-loop processes and management functions such as data quality problem discovery, monitoring and tracking, and analysis and application. Data operation and maintenance management comprehensively grasps the overall situation and usage of data access, data flow, and data resources. Data quality provides custom and configurable data quality rules and validation logic. Through open data validation, organizations can define data quality standards based on their own business needs and standards and evaluate the accuracy and completeness of data.

10. The integrated data governance method according to claim 1, characterized in that: The data service in S5 mainly provides data services and API services to the external parties for the internal data of the platform; It mainly includes functional modules such as mounting and publishing, data subscription, subscription approval, service management, and credential management; The main purpose of mounting and publishing is to mount internal data to the data resource directory in the form of API interface, so that users can subscribe to it on the data portal. The data subscription function mainly allows users to log in to the data portal and subscribe to relevant data according to their own permissions; The data administrator approves the subscription through the subscription approval module; The service management function is mainly for subscription data, which is published as API services and supports the online and offline operation of API services. Credential management mainly involves granting users authorization credentials and validity periods, and implementing user data permission control through AK and SK two-way verification.

Images