A data governance system for developing data standards based on knowledge management

Abstract

The present invention provides a data governance system for formulating data standards based on knowledge management, which relates to the field of data management technology, including: a data governance subsystem, which is configured to supervise the acquisition, processing and use of data, and is independently deployed in each project; a data standard knowledge base, which is configured to store the data standards of each project and provide data standard recommendations, wherein the data standards include code standards and data item standards; an implementation workbench, which is configured to interact with the data governance subsystem and the data standard knowledge base for data and knowledge, provide independent tenants for each project, and formulate data standards when implementing the project. The present invention realizes knowledge sharing of the implementation work of each project by adding an implementation workbench and a data standard knowledge base, automatically constructs a standardized data model based on data standards, automatically constructs data standardization tasks based on master data standardization relationships, and improves the utilization efficiency of data standards.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data governance system for formulating data standards based on knowledge management. Background Art

[0002] Data standards include code standards and data item standards, which are essential for establishing a sound data governance system. Data modeling in data governance is the process of applying these standards to describe the data model in detail. The way to formulate data standards in existing data governance projects is that project implementation engineers first conduct in-depth and comprehensive research and analysis of existing business data. They carefully study the data meaning, data type, and data length of each table and field, and combine national standards and industry standards to formulate a data standard framework applicable to this project.

[0003] To this end, implementation engineers will take a comprehensive and in-depth approach to conduct detailed research on each business area and related workflows. They will classify and summarize the role of data in each node and identify various factors that may affect data quality. At the same time, they will also refer to national and industry standards to develop a more detailed data standard system based on the actual situation of the project. This set of standards will contain the exact meaning and attribute requirements of each data item to meet the various requirements of the country and the industry.

[0004] Finally, this set of data standards, which has been carefully studied and formulated, will be input into the project's data governance system. It will provide guidance for future data modeling work and supervise the work in each link of data collection, storage and use to ensure the accuracy, timeliness and traceability of data. By implementing this set of standards, the project can ensure the consistency, integrity and reliability of data, thereby establishing an efficient and standardized data governance system.

[0005] However, manually formulating a standard is very time-consuming and labor-intensive, and the reuse of data standards is still limited, making it impossible to maximize the use of data standards.

[0006] Therefore, how to effectively use data standards to help manage data governance systems is an urgent problem to be solved. Summary of the invention

[0007] In view of this, the present invention provides a data governance system for formulating data standards based on knowledge management. By adding an implementation workbench and a data standard knowledge base, knowledge sharing of the implementation work of each project is realized, standardized data models are automatically constructed based on data standards, and data standardization tasks are automatically constructed based on master data standardization relationships, thereby improving the utilization efficiency of data standards.

[0008] The technical purpose of the present invention is achieved in this way:

[0009] The present invention provides a data governance system for formulating data standards based on knowledge management, comprising:

[0010] A data governance subsystem, which is configured to regulate the acquisition, processing, and use of data and is deployed independently in each project;

[0011] A data standard knowledge base, which is configured to store data standards for various projects and provide data standard recommendations, wherein data standards include code standards and data item standards;

[0012] The implementation workbench is configured to interact with the data governance subsystem and the data standards knowledge base for data and knowledge, provide independent tenants for each project, and develop data standards when implementing projects.

[0013] Based on the above technical solution, preferably, the data governance subsystem includes a metadata management module, a data standard management module, a master data management module, a data quality management module and a data security management module.

[0014] Based on the above technical solution, preferably, the data standard knowledge base is preset with:

[0015] Supplier system table, including system number, supplier, system, version, data standard code and master data code;

[0016] Data item standard table, including data standard code, data item code, data item name, data item Chinese abbreviation, data type, data length and standard type;

[0017] Code standard table, including data standard code, code number, code name and standard type;

[0018] A standard table of code values, including code numbers, codes, and code names;

[0019] Master data table, including master data code, data table code, database mode, table English name and table Chinese name;

[0020] Standardized table of master data items, including master data code, data table code, field English name, field Chinese name and data item code;

[0021] The master data code standardization table includes the master data code, data table code, field English name, field Chinese name and code number.

[0022] Based on the above technical solution, preferably, the process of formulating data standards is:

[0023] The first step is to connect the data source of the current project to the data governance subsystem and use the metadata management module to collect metadata of the data source.

[0024] The second step is to conduct data research and identify master data from data sources;

[0025] Step 3: Obtain data standard recommendation results from the data standard knowledge base;

[0026] Step 4: Obtain the recommended results of master data standardization from the data standard knowledge base;

[0027] Step 5: The implementation engineer adjusts and supplements the data standard recommendation results and the master data standardization recommendation results. For fields with the same meaning but inconsistent data items and codes, the data items and codes are adjusted until they are unified. For unstandardized master data fields, the national standard or custom standard is selected according to the meaning of the field, and the supplemented data standard recommendation results and the supplemented master data standardization recommendation results are obtained.

[0028] Step 6: Save the supplemented data standard recommendation results and the supplemented master data standardization recommendation results to the data standard knowledge base;

[0029] Step 7: Synchronously save the supplemented data standard recommendation results to the data standard management module of the data governance subsystem, and synchronously store the supplemented master data standardization recommendation results to the master data management module of the data governance subsystem;

[0030] Step 8: The master data management module builds a data model in the data warehouse standard layer according to the supplemented master data standardization recommendation results;

[0031] In the ninth step, the master data management module forms a data standardization conversion relationship from the ODS layer data to the standard layer data, and sets a timed task for the data standardization conversion relationship in the data scheduling module.

[0032] On the basis of the above technical solution, preferably, the second step includes:

[0033] Obtaining a master data recommendation result from a data standard knowledge base according to a first recommendation method, the master data recommendation result including a schema name, a table English name, and a table Chinese name;

[0034] According to the data source and master data recommendation results, the master data identification results of the data source are obtained according to the screening method. The master data identification results include the schema name, table English name, table Chinese name, field English name, field Chinese name, field data type and field length.

[0035] On the basis of the above technical solution, preferably, the first recommended method includes:

[0036] According to the supplier, system and version of the data source of the current project, query the supplier system table to obtain the target master data code corresponding to the data source;

[0037] The master data recommendation result is obtained by querying the master data table according to the target master data code. The master data recommendation result is a collection of all tables in the system's schema.

[0038] On the basis of the above technical solution, preferably, the screening method includes:

[0039] Get the data volume, table Chinese name and field Chinese name in the data source report;

[0040] Setting a data volume threshold, filtering the metadata according to the data volume threshold, filtering out metadata below the data volume threshold as redundant data, and obtaining a first filtering result;

[0041] The first screening results are screened according to the core business keywords based on the table Chinese name and the field Chinese name, and the core business data is identified as the second screening result;

[0042] The second screening result and the master data recommendation result are combined to obtain candidate master data;

[0043] The candidate master data is sent to the implementation engineer for final screening to obtain the master data identification results of the data source.

[0044] Based on the above technical solution, preferably, the third step includes:

[0045] According to the supplier, system and version of the current project data source, query the supplier system table to obtain the target data standard code;

[0046] In the data item standard table, query according to the target data standard code to obtain the data item standard recommendation result, the data item standard includes the data item code, data item name, data item Chinese abbreviation, data type, data length and standard type;

[0047] In the code standard table, query the target data standard code to obtain the code standard recommendation result, and the code standard includes code number, code name and standard type;

[0048] In the code value standard table, according to the code number in the code standard recommendation result, the corresponding code value standard is searched to obtain the code value standard recommendation result, where the code value standard includes a code and a code name;

[0049] The data item standard recommendation results, the code standard recommendation results and the code value standard recommendation results are used as the data standard recommendation results.

[0050] Based on the above technical solution, preferably, the fourth step includes:

[0051] According to the supplier, system and version of the data source of the current project, query the supplier system table to obtain the target master data code corresponding to the data source;

[0052] The master data standardization recommendation results are obtained by querying the master data table, the master data data item standardization table and the master data code standardization table according to the target master data code. The master data standardization recommendation results include data items and codes of all fields and form a master data standardization relationship.

[0053] Based on the above technical solution, preferably, the ninth step includes:

[0054] The master data management module establishes the data standardization conversion relationship between the English names of the master data tables and fields at the ODS layer of the data warehouse and the English names of the physical tables and fields at the standard layer of the data warehouse based on the master data standardization relationship;

[0055] Configure the DataX tool and create a data conversion task in the DataX tool to convert ODS layer data into standard layer data.

[0056] Set the scheduled task of data standardization conversion relationship in the data scheduling module and execute the data standardization task within the specified time.

[0057] The method of the present invention has the following beneficial effects compared with the prior art:

[0058] (1) The data governance system provided by the present invention centrally deploys the data standard knowledge base, so that data standards can be shared by all projects, avoiding repeated construction and maintenance, and improving the efficiency of data management; the implementation workbench is provided to each project as a SaaS service, providing a unified working environment and tools for implementation engineers, which can reduce duplication of work and improve work efficiency;

[0059] (2) The implementation workbench in the present invention provides independent tenants for each project, realizes the isolation of the implementation work of each project, and ensures the independence and security of project data; the data standard knowledge base provides services for all data governance projects, realizes the knowledge sharing of the implementation work of each project, and promotes team collaboration and experience sharing;

[0060] (3) The present invention pre-sets a rich standard knowledge base, and the system can quickly match the data standard templates related to the current project. These standard templates usually include standard definitions of the main data domains and fields. It can help the project quickly identify which are the key data areas and fields of the project, and provide standard suggestions for these areas and fields, greatly reducing the workload of the project to formulate standards from scratch. At the same time, the system can also automatically screen and recommend important data areas and fields based on project data characteristics, further reducing the intensity of manual work;

[0061] (4) The present invention adopts the same set of standards to ensure that different projects can maintain consistency in data expression and definition, which is not only conducive to the future exchange and integration of data between different systems, but also avoids the problem of inconsistent standard definitions due to human factors;

[0062] (5) In the process of formulating standards, the project of the present invention may produce some new and valuable standards, which can be shared in the standard knowledge base of the system to provide reference for other subsequent projects. The continuous enrichment of the content of the standard knowledge base will also help improve the quality and scope of application of the standards. In this way, the sharing and utilization of standard knowledge is realized, duplication of work is avoided, and the overall standard level is improved;

[0063] (6) The present invention can unify the data structure and meaning of different systems by establishing a data model of the standard layer, which is beneficial to the data exchange and integration of different systems in the later stage;

[0064] (7) The present invention automatically implements the standard conversion task, replacing the tedious work that originally needed to be completed manually. This greatly improves work efficiency. At the same time, the automatic task can also effectively avoid data problems caused by manual operation errors, thereby improving work quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 is a system structure diagram of an embodiment of the present invention;

[0067] Figure 2 The figure is a flowchart of an implementation of an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] like Figure 1 As shown, the present invention provides a data governance system for formulating data standards based on knowledge management, comprising:

[0070] A data governance subsystem, which is configured to regulate the acquisition, processing, and use of data and is deployed independently in each project;

[0071] A data standard knowledge base, which is configured to store data standards for various projects and provide data standard recommendations, wherein data standards include code standards and data item standards;

[0072] The implementation workbench is configured to interact with the data governance subsystem and the data standards knowledge base for data and knowledge, provide independent tenants for each project, and develop data standards when implementing projects.

[0073] In this embodiment, the data governance system consists of a data standard knowledge base, an implementation workbench, and a data governance subsystem.

[0074] Specifically, in one embodiment of the present invention, the data governance subsystem includes a metadata management module, a data standard management module, a master data management module, a data quality management module and a data security management module.

[0075] The data governance subsystem consists of modules such as metadata management, data standard management, master data management, data quality management, and data security management. Users use the data governance subsystem to supervise the acquisition, processing, and use of data. The implementation workbench includes the formulation of data standards. During the project implementation phase, implementation engineers use the implementation workbench to formulate data standards for this project. The data standard knowledge base stores the data standards of each project and provides data standard recommendations. Data standards include code standards and data item standards.

[0076] The data governance subsystem is deployed independently in each project. The implementation workbench and data standard knowledge base are deployed centrally to provide SaaS services for implementation engineers. The implementation workbench provides independent tenants for each project to achieve isolation of the implementation work of each project. The data standard knowledge base provides services for all data governance projects to achieve knowledge sharing of the implementation work of each project.

[0077] Specifically, the data standard knowledge base presets include:

[0078] Supplier system table, including system number, supplier, system, version, data standard code and master data code;

[0079] Data item standard table, including data standard code, data item code, data item name, data item Chinese abbreviation, data type, data length and standard type;

[0080] Code standard table, including data standard code, code number, code name and standard type;

[0081] A standard table of code values, including code numbers, codes, and code names;

[0082] Master data table, including master data code, data table code, database mode, table English name and table Chinese name;

[0083] Standardized table of master data items, including master data code, data table code, field English name, field Chinese name and data item code;

[0084] The master data code standardization table includes the master data code, data table code, field English name, field Chinese name and code number.

[0085] In the present invention, the process of formulating data standards is as follows:

[0086] The first step is to connect the data source of the current project to the data governance subsystem and use the metadata management module to collect metadata of the data source.

[0087] The second step is to conduct data research and identify master data from data sources;

[0088] The second step includes:

[0089] Obtaining a master data recommendation result from a data standard knowledge base according to a first recommendation method, the master data recommendation result including a schema name, a table English name, and a table Chinese name;

[0090] According to the data source and master data recommendation results, the master data identification results of the data source are obtained according to the screening method. The master data identification results include the schema name, table English name, table Chinese name, field English name, field Chinese name, field data type and field length.

[0091] The first recommended method includes:

[0092] According to the supplier, system and version of the data source of the current project, query the supplier system table to obtain the target master data code corresponding to the data source;

[0093] The master data recommendation result is obtained by querying the master data table according to the target master data code. The master data recommendation result is a collection of all tables in the system's schema.

[0094] Screening methods include:

[0095] Get the data volume, table Chinese name and field Chinese name in the data source report;

[0096] Setting a data volume threshold, filtering the metadata according to the data volume threshold, filtering out metadata below the data volume threshold as redundant data, and obtaining a first filtering result;

[0097] The first screening results are screened according to the core business keywords based on the table Chinese name and the field Chinese name, and the core business data is identified as the second screening result;

[0098] The second screening result and the master data recommendation result are combined to obtain candidate master data;

[0099] The candidate master data is sent to the implementation engineer for final screening to obtain the master data identification results of the data source.

[0100] Step 3: Obtain data standard recommendation results from the data standard knowledge base;

[0101] The third step includes:

[0102] According to the supplier, system and version of the current project data source, query the supplier system table to obtain the target data standard code;

[0103] In the data item standard table, query according to the target data standard code to obtain the data item standard recommendation result, the data item standard includes the data item code, data item name, data item Chinese abbreviation, data type, data length and standard type;

[0104] In the code standard table, query the target data standard code to obtain the code standard recommendation result, and the code standard includes code number, code name and standard type;

[0105] In the code value standard table, according to the code number in the code standard recommendation result, the corresponding code value standard is searched to obtain the code value standard recommendation result, where the code value standard includes a code and a code name;

[0106] The data item standard recommendation results, the code standard recommendation results and the code value standard recommendation results are used as the data standard recommendation results.

[0107] Step 4: Obtain the recommended results of master data standardization from the data standard knowledge base;

[0108] The fourth step includes:

[0109] According to the supplier, system and version of the data source of the current project, query the supplier system table to obtain the target master data code corresponding to the data source;

[0110] The master data standardization recommendation results are obtained by querying the master data table, the master data data item standardization table and the master data code standardization table according to the target master data code. The master data standardization recommendation results include data items and codes of all fields and form a master data standardization relationship.

[0111] Step 5: The implementation engineer adjusts and supplements the data standard recommendation results and the master data standardization recommendation results. For fields with the same meaning but inconsistent data items and codes, the data items and codes are adjusted until they are unified. For unstandardized master data fields, the national standard or custom standard is selected according to the meaning of the field, and the supplemented data standard recommendation results and the supplemented master data standardization recommendation results are obtained.

[0112] Step 6: Save the supplemented data standard recommendation results and the supplemented master data standardization recommendation results to the data standard knowledge base;

[0113] Step 7: Synchronously save the supplemented data standard recommendation results to the data standard management module of the data governance subsystem, and synchronously store the supplemented master data standardization recommendation results to the master data management module of the data governance subsystem;

[0114] Step 8: The master data management module builds a data model in the data warehouse standard layer according to the supplemented master data standardization recommendation results;

[0115] In the ninth step, the master data management module forms a data standardization conversion relationship from the ODS layer data to the standard layer data, and sets a timed task for the data standardization conversion relationship in the data scheduling module.

[0116] The ninth step includes:

[0117] The master data management module establishes the data standardization conversion relationship between the English names of the master data tables and fields at the ODS layer of the data warehouse and the English names of the physical tables and fields at the standard layer of the data warehouse based on the master data standardization relationship;

[0118] Configure the DataX tool and create a data conversion task in the DataX tool to convert ODS layer data into standard layer data.

[0119] Set the scheduled task of data standardization conversion relationship in the data scheduling module and execute the data standardization task within the specified time.

[0120] See also Figure 2 , described with a specific embodiment:

[0121] (1) Collect metadata. Call the metadata collection capability in the metadata management of the data governance subsystem. The data governance subsystem manages the vendor, system, version, data source type, IP, port, user, and password of the data source. After connecting to the data source, collect the metadata of the database, including the schema name, table English name, table comment, field English name, field comment, data type, whether it is required, and whether it is a primary key.

[0122] (2) Identify master data. Conduct data research and identify master data from the data source. Master data information includes: schema name, table English name, table Chinese name, field English name, field Chinese name, field data type, and field length. Obtain standardized relationships of master data from the knowledge base.

[0123] Specifically, first, the target master data code corresponding to the data source is obtained by querying the supplier system table from the data standard knowledge base according to the supplier, system and version of the data source of the current project. The master data recommendation result is obtained by querying the target master data code in the master data table, and the master data recommendation result is a collection of all tables in the system's schema.

[0124] After that, confirm the master data. According to the data source report, the implementation engineer selects a table with a relatively large amount of data as the master data, and determines the core business data as the master data based on the Chinese name of the table and the Chinese name of the field. Combined with the master data recommendation results obtained in the second step. Determine the master data of the data source. Among them, a large amount of data means that there are many data items in the table. Some empty tables, temporary tables, and backup tables can be excluded through the data source report. Data with obviously too low a number of data items can be excluded first. After that, the implementation engineer comprehensively considers the Chinese name of the table and the Chinese name of the field, and the master data recommendation results to select the core data of the implementation project as the master data.

[0125] (3) Obtain data standard recommendation results from the data standard knowledge base.

[0126] The data standard knowledge base presets supplier system tables, data item standard tables and code standard tables.

[0127] The supplier system table is as follows:

[0128] SYSID PROVIDER_CODE SYSTEM_CODE VERSION STANDARD_CODE MAINDATA_CODE 100001 Yunqi Datamiddleplatform V1.0.0 700001 900001 100002 Yunqi Smartform V2.0.0 700002 900002

[0129] Among them, SYSID is the system number, PROVIDER_CODE is the supplier, SYSTEM_CODE is the system, VERSION is the version, MAINDATA_CODE is the master data code, and STANDARD_CODE is the data standard code.

[0130] The standard representation of data items is as follows:

[0131]

[0132] Among them, STANDARD_CODE is the data standard code, DATAITEM is the data item code, DATAITEM_NAME is the data item name, DATAITEM_CHN is the Chinese abbreviation, DATAITEM_TYPE is the data type, DATAITEM_LENGTH is the length, and STD_TYPE is the standard type.

[0133] The following is an example of a code standard:

[0134] STANDARD_CODE ITEM ITEM_NAME STD_TYPE 700001 XXXZ School Type Code National Standards 700002 JbXj Faculty and staff category code National Standards 700003 XQ Semester Code Custom Standards

[0135] Among them, STANDARD_CODE is the data standard code, ITEM is the code number, ITEM_NAME is the code name, and STD_TYPE is the standard type.

[0136] The following are examples of standard code values:

[0137]

[0138]

[0139] Where ITEM is the code number, CODE is the code, and CODE_NAME is the name.

[0140] According to the supplier, system, and version of the current project data source, query the knowledge base supplier system table to obtain STANDARD_CODE. Query the knowledge base data item standard table according to STANDARD_CODE to obtain the recommended data item standard results. The data item standards include: data item coding, data item name, Chinese abbreviation, data type, length, and standard type. Query the knowledge base code standard table according to STANDARD_CODE to obtain the recommended code standard results. The code standards include: code number, code name, and standard type. Query the knowledge base code value standard table according to ITEM to obtain the recommended code standard code value results. The code value standards include: code and name.

[0141] (4) Obtain the recommended results of master data standardization from the knowledge base.

[0142] The knowledge base presets master data tables, master data item standardization tables, and master data code standardization tables.

[0143] The master data table example is as follows:

[0144] MAINDATA_CODE TABLE_ID SCHEMA TABLENAME_EN TABLENAME_CHN 900001 800001 I Staffinfo Faculty and Staff Information Sheet 900001 800002 I Userinfo User Information Table 900001 800003 I Schoolinfo School Information Sheet

[0145] Among them, MAINDATA_CODE is the main data code, TABLE_ID is the data table code, SCHEMA is the database mode, TABLENAME_EN is the English name of the table, and TABLENAME_CHN is the Chinese name of the table.

[0146] The following is an example of a standardized representation of a master data item:

[0147] MAINDATA_CODE TABLE_ID COLUMN_EN COLUMN_CHN DATAITEM 900001 800001 NAME Name JCTB020101 900001 800001 ENGLISHNAME English name JCTB020102 900001 800001 PINYIN Name Pinyin JCTB020103

[0148] Among them, MAINDATA_CODE is the main data code, TABLE_ID is the data table code, COLUMN_EN is the English name of the field, COLUMN_CHN is the Chinese name of the field, and DATAITEM is the data item code.

[0149] The following is an example of the standardized representation of the master data code:

[0150]

[0151]

[0152] Among them, MAINDATA_CODE is the master data code, TABLE_ID is the data table code, COLUMN_EN is the English name of the field, COLUMN_CHN is the Chinese name of the field, and ITEM is the code number.

[0153] According to the supplier, system, and version of the current project data source, query the knowledge base supplier system table to obtain MAINDATA_CODE, and query the master data standardization recommendation results according to MAINDATA_CODE in the knowledge base master data table, master data item standardization table, and master data code standardization table. The recommendation results include data items and codes for all fields in the master data table.

[0154] (5) Adjust and supplement data standards. The second step identified the master data of this project. The fourth step obtained the recommended results of data standardization for the master data. The third step obtained the recommended results of data standards. The implementation engineer observed the standardized results of the master data of this project based on the knowledge base recommendation on the implementation workbench. For fields with the same meaning, if there were inconsistent data items and codes, the implementation engineer adjusted the data items and codes until they were unified. For unstandardized master data fields, the implementation engineer selected national standards or custom standards based on the meaning of the fields.

[0155] (6) Knowledge base storage data standards.

[0156] Save the finalized data items and codes of this project to the knowledge base data item standard table, code standard table and code value standard table. Data item knowledge data includes: data standard code, data item code, data item name, Chinese abbreviation, data type, length, standard type. Code knowledge data includes: data standard code, code number, code name, standard type, code, name. The above knowledge is used to recommend data standards for the project next time.

[0157] The master data standardized relationship finalized for this project is saved in the master data table, master data item standardized table and master data code standardized table of the knowledge base. The master data standardized knowledge data includes: master data code, data table code, database mode, table English name, table Chinese name, data table code, field English name, field Chinese name, data item code, data table code, field English name, field Chinese name, code number. The above knowledge is used to recommend the master data standardized relationship for the project next time.

[0158] (7) Synchronize data standards to the data governance subsystem.

[0159] Save the finalized data items and codes of this project to the data standard module of the data governance subsystem. Data item data includes: data item code, data item name, Chinese abbreviation, data type, length, standard type. Code data includes: code number, code name, standard type, code, name. Data standards are used to build data models in the data governance subsystem.

[0160] The master data standardized relationship finalized for this project is saved to the master data module of the data governance subsystem. The master data standardized data includes: master data code, data table code, database mode, table English name, table Chinese name, field English name, field Chinese name, data item code, field English name, field Chinese name, code number. The master data standardized relationship is used to build the standard layer data model in the master data module of the data governance subsystem.

[0161] (8) Build a standard layer data model.

[0162] The master data management module creates physical tables in the standard layer of the data warehouse based on the English name of the table, the Chinese name of the table, the English name of the field, the Chinese name of the field, the English name of the data item field, the Chinese name of the data item field, the data item data type, and the data item length to achieve data standardization.

[0163] (9)Data standardization.

[0164] The master data management module establishes a standardized conversion relationship between the English names of the master data tables and fields at the ODS layer of the data warehouse and the English names of the physical tables and fields at the standard layer of the data warehouse based on the standardized relationship of the master data. This relationship is maintained in the DataX tool, forming a data conversion task from the ODS layer data to the standard layer data. In the data scheduling module, a scheduled task for this data standardized relationship is set to execute the data standardization task within the specified time.

[0165] Specifically, in this embodiment, by establishing a data standard knowledge base and an implementation workbench, data standards and recommended results are systematically managed, the data standardization process is accelerated, manual intervention and errors are reduced, the efficiency and accuracy of data management are improved, and the consistency and reliability of data are ensured. The data standardization system provides unified data standards and recommended results, promotes data sharing and collaboration between different projects, reduces the phenomenon of data islands, improves the value of data utilization and the effect of business decision-making, and promotes the smooth flow of information and collaborative work within the organization. Through the data quality management module and the data security management module, data quality is monitored and maintained, the confidentiality and integrity of data are protected, data compliance and security are ensured, data leakage and damage are prevented, data credibility and confidentiality are improved, and the security of organizational data assets is guaranteed. Data standardization conversion relationships are established to ensure the consistency and traceability of data at different levels and links, optimize data processes and quality control, improve data processing efficiency and accuracy, reduce data processing costs and risks, and promote continuous improvement and enhancement of data management. Through the implementation of the data standardization system, the complexity and cost of data management can be reduced, errors and duplication in data processing can be reduced, data management risks and uncertainties can be reduced, the sustainable utilization and management efficiency of data resources can be improved, and costs and resources can be saved for the organization, and competitiveness can be enhanced. Establish data standardization conversion relationships, ensure the consistency and standardization of data at different levels, strengthen the formulation and implementation of data standards, improve the effectiveness of data integration and analysis, promote the accuracy and timeliness of data-driven decision-making, and promote the construction of organizational data-driven culture and digital transformation.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data governance system for formulating data standards based on knowledge management, characterized in that: include: A data governance subsystem, which is configured to regulate the acquisition, processing, and use of data and is deployed independently in each project; A data standard knowledge base, which is configured to store data standards for various projects and provide data standard recommendations, wherein data standards include code standards and data item standards; The implementation workbench is configured to interact with the data governance subsystem and the data standards repository for data and knowledge, provide independent tenants for each project, and develop data standards when implementing projects; The data governance subsystem includes metadata management module, data standard management module, master data management module, data quality management module and data security management module; The data standard knowledge base presets include: Supplier system table, including system number, supplier, system, version, data standard code and master data code; Data item standard table, including data standard code, data item code, data item name, data item Chinese abbreviation, data type, data length and standard type; Code standard table, including data standard code, code number, code name and standard type; A standard table of code values, including code numbers, codes, and code names; Master data table, including master data code, data table code, database mode, table English name and table Chinese name; Standardized table of master data items, including master data code, data table code, field English name, field Chinese name and data item code; Master data code standardization table, including master data code, data table code, field English name, field Chinese name and code number; The process of developing data standards is: The first step is to connect the data source of the current project to the data governance subsystem and use the metadata management module to collect metadata of the data source. The second step is to conduct data research and identify master data from data sources; Step 3: Obtain data standard recommendation results from the data standard knowledge base; Step 4: Obtain the recommended results of master data standardization from the data standard knowledge base; Step 5: The implementation engineer adjusts and supplements the data standard recommendation results and the master data standardization recommendation results. For fields with the same meaning but inconsistent data items and codes, the data items and codes are adjusted until they are unified. For unstandardized master data fields, the national standard or custom standard is selected according to the meaning of the field, and the supplemented data standard recommendation results and the supplemented master data standardization recommendation results are obtained. Step 6: Save the supplemented data standard recommendation results and the supplemented master data standardization recommendation results to the data standard knowledge base; Step 7: Synchronously save the supplemented data standard recommendation results to the data standard management module of the data governance subsystem, and synchronously store the supplemented master data standardization recommendation results to the master data management module of the data governance subsystem; Step 8: The master data management module builds a data model in the data warehouse standard layer according to the supplemented master data standardization recommendation results; In the ninth step, the master data management module forms a data standardization conversion relationship from the ODS layer data to the standard layer data, and sets a timed task for the data standardization conversion relationship in the data scheduling module.

2. A data governance system for formulating data standards based on knowledge management as claimed in claim 1, characterized in that: The second step includes: Obtaining a master data recommendation result from a data standard knowledge base according to a first recommendation method, the master data recommendation result including a schema name, a table English name, and a table Chinese name; According to the data source and master data recommendation results, the master data identification results of the data source are obtained according to the screening method. The master data identification results include the schema name, table English name, table Chinese name, field English name, field Chinese name, field data type and field length.

3. A data governance system for formulating data standards based on knowledge management as claimed in claim 2, characterized in that: The first recommended method includes: According to the supplier, system and version of the data source of the current project, query the supplier system table to obtain the target master data code corresponding to the data source; The master data recommendation result is obtained by querying the target master data code in the master data table. The master data recommendation result is a collection of all tables in the system's schema.

4. A data governance system for formulating data standards based on knowledge management as claimed in claim 2, characterized in that: Screening methods include: Get the data volume, table Chinese name and field Chinese name in the data source report; Setting a data volume threshold, filtering the metadata according to the data volume threshold, filtering out metadata below the data volume threshold as redundant data, and obtaining a first filtering result; The first screening results are screened according to the core business keywords based on the table Chinese name and the field Chinese name, and the core business data is identified as the second screening result; The second screening result and the master data recommendation result are combined to obtain candidate master data; The candidate master data is sent to the implementation engineer for final screening to obtain the master data identification results of the data source.

5. A data governance system for formulating data standards based on knowledge management as claimed in claim 1, characterized in that: The third step includes: According to the supplier, system and version of the current project data source, query the supplier system table to obtain the target data standard code; In the data item standard table, query according to the target data standard code to obtain the data item standard recommendation result, the data item standard includes the data item code, data item name, data item Chinese abbreviation, data type, data length and standard type; In the code standard table, query the target data standard code to obtain the code standard recommendation result, and the code standard includes code number, code name and standard type; In the code value standard table, according to the code number in the code standard recommendation result, the corresponding code value standard is searched to obtain the code value standard recommendation result, where the code value standard includes a code and a code name; The data item standard recommendation results, the code standard recommendation results and the code value standard recommendation results are used as the data standard recommendation results.

6. A data governance system for formulating data standards based on knowledge management as claimed in claim 1, characterized in that: The fourth step includes: According to the supplier, system and version of the data source of the current project, query the supplier system table to obtain the target master data code corresponding to the data source; The master data standardization recommendation results are obtained by querying the master data table, the master data data item standardization table and the master data code standardization table according to the target master data code. The master data standardization recommendation results include data items and codes of all fields and form a master data standardization relationship.

7. A data governance system for formulating data standards based on knowledge management as claimed in claim 6, characterized in that: The ninth step includes: The master data management module establishes the data standardization conversion relationship between the English names of the master data tables and fields at the ODS layer of the data warehouse and the English names of the physical tables and fields at the standard layer of the data warehouse based on the master data standardization relationship; Configure the DataX tool and create a data conversion task in the DataX tool to convert ODS layer data into standard layer data. Set the scheduled task of data standardization conversion relationship in the data scheduling module and execute the data standardization task within the specified time.

Images