Metadata treatment platform system and method and computer readable storage medium

Abstract

The invention is suitable for the technical field of data, provides a management platform system and method of metadata and a computer readable storage medium, standardizes a storage structure, a data type and a naming rule of data from the source, ensures consistency and normalization of the data, and has practicability and usability. According to the management platform system for the metadata, the data standard definition module, the standardization processing module and the metadata processing module are integrated, setting of field attributes is standardized according to published data standards in a data table building link of a data warehouse, the effect of controlling the metadata is achieved, automation and intellectualization of management of the metadata are achieved, and the management efficiency of the metadata is improved. The cost of manual intervention is reduced, the efficiency and accuracy of metadata management are improved, and the problem that data standards are difficult to be effectively executed in metadata management is solved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a metadata governance platform system, method, and computer-readable storage medium. Background Art

[0002] In the current digital age, enterprises are faced with the challenge of collecting, storing, analyzing and utilizing massive amounts of data. As the core infrastructure for data storage and analysis, data warehouses are becoming increasingly important. However, with the explosive growth of enterprise data and the diversification of data sources, metadata management in data warehouses has become increasingly complex and difficult.

[0003] In the process of building and managing data warehouses, metadata, as "data of data", plays a vital role. Metadata not only describes the structure, content, source and conversion process of data, but also provides important information such as data quality, access rights, and security level. However, in existing metadata management solutions, data standards and metadata management are often two independent processes, lacking an effective linkage mechanism, making it difficult to effectively implement data standards in actual metadata management. Summary of the invention

[0004] The embodiments of the present application provide a metadata governance platform system, method, and computer-readable storage medium, which can solve the problem that data standards are difficult to be effectively implemented in actual metadata management.

[0005] In a first aspect, an embodiment of the present application provides a metadata governance platform system, including:

[0006] A data standard definition module, used to define attribute information of a first data standard field;

[0007] A standardization processing module is used to create a first data standard field according to the attribute information, and to create a unique identifier for the first data standard field; and to determine whether there is a second data standard field with the same attribute information as the first data standard field in the standard code table; if not, to create a subscription channel for the first data standard field, the subscription channel is used to support the first data standard field to establish a first mapping relationship with the first data warehouse table field of the data warehouse;

[0008] A metadata processing module is used to establish a second mapping relationship between the metadata and the first data standard field through a subscription channel when a first mapping relationship is established between the first data standard field and the first data warehouse table field, and store the metadata in the metadata table; and to obtain a first storage position of the first data warehouse table field corresponding to the first mapping relationship in the data warehouse; and store the metadata in a second storage position of the metadata table according to the first storage position.

[0009] In a second aspect, an embodiment of the present application provides a method for a metadata governance platform system as described above, which is applied to the governance platform system, and the method includes:

[0010] The data standard definition module defines the attribute information of the first data standard field;

[0011] The standardization processing module creates a first data standard field according to the attribute information, and creates a unique identifier for the first data standard field;

[0012] The standardization processing module determines whether there is a second data standard field having the same attribute information as the first data standard field in the standard code table; if not, the standardization processing module creates a subscription channel for the first data standard field, the subscription channel being used to support the first data standard field to establish a first mapping relationship with the first data warehouse table field of the data warehouse respectively;

[0013] When a first mapping relationship is established between the first data standard field and the first data warehouse table field, the metadata processing module establishes a second mapping relationship between the metadata and the first data standard field through the subscription channel, and stores the metadata in the metadata table;

[0014] The metadata processing module obtains a first storage position of a first data warehouse table field corresponding to a first mapping relationship in the data warehouse; the metadata processing module stores the metadata into a second storage position of the metadata table according to the first storage position.

[0015] In a third aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the metadata governance platform system described in the first aspect above.

[0016] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0017] Compared with the prior art, the beneficial effects of the embodiment of the present application are as follows: the governance platform system of metadata of the present application includes a data standard definition module for defining the attribute information of the first data standard field; a standardization processing module for creating the first data standard field according to the attribute information and creating a unique identifier for the first data standard field; and for judging whether there is a second data standard field with the same attribute information as the first data standard field in the standard code table; if not, a subscription channel is created for the first data standard field, and the subscription channel is used to support the first data standard field to establish a first mapping relationship with the first data warehouse table field of the data warehouse respectively; a metadata processing module for establishing a second mapping relationship between the metadata and the first data standard field through the subscription channel when the first data standard field establishes a first mapping relationship with the first data warehouse table field, and storing the metadata in the metadata table; and for obtaining the first storage location of the first data warehouse table field corresponding to the first mapping relationship in the data warehouse; according to the first storage location, the metadata is stored in the second storage location of the metadata table. The present application standardizes the setting of field attributes according to the published data standards in the data table building link of the data warehouse, and achieves the effect of controlling metadata, ensuring the consistency and standardization of data, and the present application is practical and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a first schematic diagram of a metadata governance platform system provided by an embodiment of the present application;

[0020] Figure 2 This is a second schematic diagram of a metadata governance platform system provided by an embodiment of the present application;

[0021] Figure 3 It is a flowchart of a metadata governance method provided by an embodiment of the present application; DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0023] 1. Data warehouse is a system used for reporting and data analysis. It is an important part of enterprise-level data management. Data warehouse is usually a place where data is collected, cleaned, integrated and stored in a way suitable for analysis from various source systems.

[0024] 2. Data standards are a set of rules and guidelines that define how to create, manage, and use data. Data standards ensure the consistency, accuracy, and reliability of data, facilitate data sharing across departments or systems, reduce data inconsistencies, and improve the overall quality of data.

[0025] 3. Metadata is data that describes information resources or data objects, and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0026] 4. BDBJE (Berkeley DB JE, ​​Berkeley DB Java Edition) is a distributed key-value (KV) structured embedded database engine that supports multi-threaded access, transactions and data backup, and can efficiently support millions of data volumes. BDBJE is a high-performance, scalable embedded database engine that supports transaction processing and is written in Java. It allows applications to use database functions directly in their code without connecting to a separate database server over the network. The key-value pairs in BDBJE can be very flexible data types, such as simple strings or numbers, or more complex data structures. Because BDBJE supports distributed features, data can be replicated between multiple nodes, providing high availability and fault tolerance, and is suitable for building large-scale distributed systems that require high reliability.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] The metadata governance solution based on data standards aims to standardize the storage, use and management of data in the data warehouse through data standards and improve the quality and consistency of data. However, the existing metadata governance solutions based on data standards often have the following shortcomings:

[0033] (1) Disconnection between data standards and metadata management: Data standards and metadata management are often two independent processes, lacking an effective linkage mechanism, which makes it difficult for data standards to be effectively implemented in actual data management;

[0034] (2) Lack of intelligent standard monitoring and abnormal alarm: Most existing metadata management systems lack intelligent analysis and alarm functions and are unable to timely discover and handle abnormal situations in the process of data standard execution;

[0035] (3) Lack of automatic generation of standard optimization suggestions: With the development of business and the accumulation of data, data standards may need to be adjusted and optimized according to actual conditions. However, existing metadata management systems often cannot provide intelligent standard optimization suggestions and require manual intervention and adjustment.

[0036] The data warehouse is the core of the data platform, and the standardization problem in its table building process has always been a challenge. Traditional data warehouse table building methods often lack unified standards, resulting in chaotic data structure, difficult maintenance, and even affecting the accuracy and efficiency of data analysis. Therefore, this application proposes a metadata governance platform system, method and computer-readable storage medium to achieve efficient, intelligent and flexible metadata governance in the data warehouse.

[0037] Figure 1 This is a first schematic diagram of a metadata governance platform system provided in one embodiment of the present application. The system includes a data standard definition module 10, a standardization processing module 11 and a metadata processing module 12.

[0038] The data standard definition module 10 is used to define the attribute information of the first data standard field.

[0039] The standardization processing module 11 is used to create a first data standard field according to the attribute information and create a unique identifier for the first data standard field; and to determine whether there is a second data standard field in the standard code table with the same attribute information as the first data standard field; if not, a subscription channel is created for the first data standard field, and the subscription channel is used to support the first data standard field to establish a first mapping relationship with the first data warehouse table field of the data warehouse.

[0040] The metadata processing module 12 is used to establish a second mapping relationship between the metadata and the first data standard field through a subscription channel when a first mapping relationship is established between the first data standard field and the first data warehouse table field, and store the metadata in the metadata table; and to obtain a first storage position of the first data warehouse table field corresponding to the first mapping relationship in the data warehouse; and store the metadata in a second storage position of the metadata table according to the first storage position.

[0041] For example, in some specific implementations, the following may be used: Figure 3 The metadata governance platform system shown in the figure has the following specific implementation scheme:

[0042] 1. Data governance platform development: First, a data governance platform needs to be developed to manage and maintain data standards. The platform will also provide a user-friendly interface that allows administrators to define, modify, and delete data standards. The platform should also support version management of data standards to ensure the stability and traceability of data standards.

[0043] 2. Data standard definition and management (data standard definition module): On the data governance platform, administrators can define data standards, including field names, data types, lengths, constraints and other information. Administrators can also add documentation for each data standard so that users can understand and apply the standards. The definition and management of data standards should have an authority control mechanism to ensure that only authorized users can perform operations. Data standard definition: In the data governance module, a field-level standardization management mechanism is established to implement data standards. The scope of data standard management includes field names, data types, lengths, constraints and other information, that is, by defining the attribute type and range of the field, the field content is standardized.

[0044] According to the standard_fields table, the field name, data type and other fields are used as the table header, and the corresponding input items are brought into the page to complete the creation. At the same time, the standard solution will also set enumeration items according to the standard code table to standardize the value range of the field value. Through interface operations or API calls, administrators can create, modify and delete data standards.

[0045] 2. Standardized field management (standardization processing module): When a new standardized field is created, a unique key is generated. At this time, the standard_fields table will be traversed to confirm whether there are duplicates. For the newly created fields, a subscription channel will be opened, and mapping with data warehouse table fields and metadata table fields is supported.

[0046] 3. Metadata management (metadata processing module): Metadata generation: After selecting the standard new field in the data warehouse, the monitoring mechanism will be triggered, and the subscribed data standard field will be synchronously added to the dmm_databases metadata table of doris, and the library table location of the field in the data warehouse will be automatically obtained, and the dmm_databases will be automatically graded and stored in the corresponding location. The metadata is elected as a leader through BDBJE, and one of the follower nodes becomes the leader node, responsible for the metadata write operation. When the leader node goes down, the other follower nodes will re-elect a leader to ensure the high availability of the service. Metadata storage and management system: such as Apache Atlas, Apache Amundsen, etc., used to manage and maintain data metadata, including data lineage, data map, data dictionary, etc. Metadata collection and injection tools: such as Apache Falcon, Apache NiFi, etc., used to collect and inject metadata information of data to realize automatic management and tracking of data.

[0047] The metadata governance platform system regulates the setting of field attributes of data standards according to data standards in the data table creation process of the data warehouse, and achieves the effect of controlling metadata. The system creates table fields of the data warehouse (data warehouse) by associating data standard solutions, and realizes real-time monitoring of data warehouse data by data standards; and, for newly created data standards, deduplication verification is performed by traversing the standard code table (main table), and a subscription interface is designed to support active association of the data warehouse; after the data warehouse table is created, metadata is automatically created, and metadata and data standards subscribed by the data warehouse fields are also associated with each other, which increases the dimension of data monitoring and realizes the automation and intelligence of metadata governance. Based on standardized design, the system is easier and more efficient in future expansion and maintenance; new data standards and fields can be easily integrated into the existing system without large-scale reconstruction or modification of the system to improve system scalability and maintainability. And, promote team collaboration and communication: unified data standards and metadata management mechanisms help different departments or teams to collaborate and communicate in data processing, analysis and use, and reduce misunderstandings and conflicts caused by inconsistent data or misunderstandings.

[0048] In some embodiments, the attribute information includes an attribute type and an attribute value range corresponding to the attribute type. Optionally, the attribute type includes one or more of a field name, a data type, a length, and a constraint condition.

[0049] For example, the data standard definition module: The platform will also provide a user-friendly interface that allows administrators to define, modify, and delete data standards. The system should also support version management of data standards to ensure the stability and traceability of data standards. In the data governance module (data standard definition module), a field-level standardization management mechanism is established to implement data standards. The scope of data standard management includes information such as field name, data type, length, and constraints, that is, by defining the attribute type and range of the field, the field content is standardized.

[0050] Specifically, according to the standard_fields table, the field name, data type and other fields are used as table headers and brought into the corresponding input items on the page to complete the creation. At the same time, the standard solution will also set enumeration items according to the standard code table to standardize the value range of the field value. Through interface operations or API calls, administrators can create, modify and delete data standards. The metadata governance platform system enhances data security by defining field constraints and enumeration items, limiting the value range of data, effectively preventing the input of illegal or unreasonable data, and enhancing data security.

[0051] The metadata governance platform system ensures data consistency and standardization: by directly selecting the maintained data standard solution before creating the data warehouse, it aims to standardize the data storage structure, data type and naming rules from the source, thereby ensuring data consistency and standardization and reducing problems caused by data redundancy and inconsistency. In addition, the system improves data quality: by using data standards as field attributes at the beginning of table construction, it ensures that the data meets the established governance standards from the source, greatly reducing data irregularities and errors, and significantly improving data accuracy and consistency.

[0052] Figure 2 This is a second schematic diagram of a metadata governance platform system provided by an embodiment of the present application, which is provided as an example and not as a limitation. In the system, the standardization processing module 11 further includes a permission control submodule 111, and / or the metadata processing module 12 includes a monitoring submodule 121, and / or the metadata processing module 12 further includes a grading submodule 122. Figure 2 As shown, in the system, the standardization processing module 11 includes a permission control submodule 111 , the metadata processing module 12 includes a monitoring submodule 121 , and the metadata processing module 12 also includes a grading submodule 122 .

[0053] In one embodiment, the standardization processing module 11 further includes an authority control submodule 111 , and the authority control submodule 111 is used to set the authority for the standardization processing module 11 to create, modify, and delete the first data standard field.

[0054] Optionally, the permission control submodule 111 is used to determine whether the current system account is an authorized account; if so, the permission control submodule 111 is used to set the permission for creating and modifying the standardization processing module 11 to be enabled. Also, the permission control submodule 111 is used to determine whether the current system account is an administrative account; if so, the permission control submodule 111 is used to set the permission for creating, modifying, and deleting the standardization processing module 11 to be enabled, and the permission for adding a description document as a first data standard field is enabled.

[0055] In some embodiments, the metadata processing module 12 includes a monitoring submodule 121, and the monitoring submodule 121 is used to monitor whether a first mapping relationship is established between the first data standard field and the first data warehouse table field.

[0056] In some embodiments, the metadata processing module 12 further includes a grading submodule 122, which is used to configure a preset level for the metadata according to the first storage position of the first data warehouse table field in the data warehouse. The metadata processing module 12 is used to search for a third storage position corresponding to the preset level in the metadata table according to the preset level, and use the third storage position as the second storage position.

[0057] In some embodiments, the metadata table is stored in a distributed storage system, and the distributed storage system includes more than three nodes, all of which can communicate with each other, wherein the first node is a leader node and the remaining nodes are follower nodes; the leader node is used to store metadata, and the follower node is used to copy the data stored in the lead node. Also, the metadata processing module 12 also includes a high-availability processing submodule, which is used to store metadata in the metadata table corresponding to the leader node; and, when all follower nodes do not receive the heartbeat signal of the leader node within a preset time, the first node is judged to be in a downtime state, and a second node is re-selected from the remaining nodes, and the second node is used as the leader node. The embodiment adopts the leader election mechanism of BDBJE, and on this basis, uses the metadata grading technology to ensure the high availability of the metadata service.

[0058] The metadata governance platform system simplifies the data management process: the automated metadata generation and synchronization mechanism reduces manual intervention and management costs, while improving the efficiency and accuracy of data management. The association between metadata and data standards makes data management and governance more systematic and standardized.

[0059] In one embodiment, the metadata governance platform system also includes intelligent diagnosis and NLP suggestion modules. In the data platform, a classification mechanism is set in advance for problem triggers, and an external NLP vertical model is connected. Based on the report data as input, the model intelligently recognizes semantics and provides generative responses, which are displayed on the front end through HTML. Through intelligent diagnosis, the set monitoring mechanism is combined with the NLP model, and a generative solution is given while warning of real-time data anomalies.

[0060] In one embodiment, the metadata governance platform system also includes a data standard monitoring module. Real-time monitoring and alarming: real-time monitoring of data standard changes and the actual situation of the data, timely discovery and processing of data anomalies, timely notification of relevant personnel through the alarm mechanism, and improved response speed and efficiency of problem handling. In addition, support for continuous optimization of data standards: automatic generation of standard optimization suggestions based on data usage and standard popularity, providing a scientific basis for the continuous optimization of data standards and promoting continuous improvement of data governance.

[0061] In one embodiment, the metadata governance platform system also includes a data compliance detection tool module, a compliance report and an audit tool module. Improve business decision-making capabilities: High-quality data provides solid support for business decisions. Based on standardized metadata management, business teams can obtain accurate and consistent data more quickly, thereby making more scientific and reasonable decisions.

[0062] For example, in some specific implementations, the following may be used: Figure 2 The metadata governance platform system shown in the figure has the following specific implementation scheme:

[0063] 1. Data governance platform development: First, a data governance platform needs to be developed to manage and maintain data standards. The platform will also provide a user-friendly interface that allows administrators to define, modify, and delete data standards. The platform should also support version management of data standards to ensure the stability and traceability of data standards.

[0064] 2. Data standard definition and management: On the data governance platform, administrators can define data standards, including field names, data types, lengths, constraints, and other information. Administrators can also add documentation for each data standard so that users can understand and apply the standard. The definition and management of data standards should have a permission control mechanism to ensure that only authorized users can perform operations.

[0065] Data standard definition: In the data governance module, a field-level standardization management mechanism is established to implement data standards. The scope of data standard management includes information such as field name, data type, length, and constraints, that is, by defining the attribute type and range of the field, the field content is standardized.

[0066] According to the standard_fields table, the field name, data type and other fields are used as the table header, and the corresponding input items are brought into the page to complete the creation. At the same time, the standard solution will also set enumeration items according to the standard code table to standardize the value range of the field value. Through interface operations or API calls, administrators can create, modify and delete data standards.

[0067] 3. Standardized field management: A new standardized field will generate a unique key. At this time, the standard_fields table will be traversed to confirm whether there are any duplications. For the newly created fields, the subscription channel will be opened to support mapping with data warehouse table fields and metadata table fields. Standardization construction in the process of building data warehouse tables: In the process of building data warehouse tables, the concept of data standards is introduced. When building data warehouse tables through tools or scripts, the table can be subscribed to the fields of data standards to establish mapping. After the data warehouse field is created, the subscribed data standard will be bound to the newly created field.

[0068] Create new data warehouse table fields through the associated data standard solution to achieve real-time monitoring of data warehouse data by data standards. For new data standards, perform deduplication verification by traversing the main table, and design a subscription interface to support active association of data warehouses.

[0069] 4. Data compliance management:

[0070] Data compliance detection tools: such as Apache Atlas and Collibra, which are used to define and enforce data compliance policies and monitor compliance during data processing.

[0071] Compliance reporting and auditing tools: such as Compliance Sheriff and OpenDLP, which are used to generate compliance reports and support data auditing and tracing functions.

[0072] 5. Metadata Management:

[0073] Metadata storage and management systems: such as Apache Atlas, Apache Amundsen, etc., are used to manage and maintain data metadata, including data lineage, data maps, data dictionaries, etc.

[0074] Metadata collection and injection tools: such as Apache Falcon and Apache NiFi, which are used to collect and inject metadata information of data to achieve automated management and tracking of data.

[0075] Metadata generation: After selecting the standard new field in the data warehouse, the monitoring mechanism will be triggered, and the subscribed data standard field will be synchronously added to the dmm_databases metadata table of doris, and the library table location of the field in the data warehouse will be automatically obtained, and dmm_databases will be automatically graded and stored in the corresponding location. The metadata is elected as a leader through BDBJE, and one of the follower nodes becomes the leader node, responsible for the metadata write operation. When the leader node goes down, the other follower nodes will re-elect a leader to ensure high availability of the service.

[0076] After the data warehouse creates a table, metadata is automatically created. The metadata and the data standards subscribed by the data warehouse fields are also related to each other, which increases the dimension of data monitoring. The leader election mechanism of BDBJE is adopted, and the metadata grading technology is used on this basis to ensure the high availability of metadata services.

[0077] 6. Data governance platform: such as Collibra, Alation, etc., provide comprehensive data governance functions, including data quality management, data security management, data compliance management and metadata management, to help organizations achieve comprehensive management and control of data governance.

[0078] 7. Data standard monitoring: Use database triggers, message queue listeners, or scheduled tasks (including Cron jobs, Spring's @Scheduled annotation, etc.) to trigger the automated verification process. Through the built-in indicator data integration of the data platform, the use of data standards, standard inconsistencies, standard changes, etc. are recorded, and statistics are compiled into data reports to ensure the consistency of data warehouse standards.

[0079] 8. Intelligent diagnosis and NLP suggestions: In the data platform, a classification mechanism is set up in advance for problem triggers, and an external NLP vertical model is connected. Based on the report data as input, the model intelligently recognizes the semantics and provides a generative response, which is displayed on the front end through HTML. Through intelligent diagnosis, the set monitoring mechanism is combined with the NLP model, and a generative solution is given while warning of real-time data anomalies.

[0080] The metadata governance platform system of this solution integrates the data standard definition module, the standardization processing module and the metadata processing module to build a comprehensive metadata governance platform system. In the data table building process of the data warehouse, the field attribute settings are standardized according to the published data standards, and the effect of controlling metadata is achieved, so as to realize the automation and intelligence of metadata governance, reduce the cost of manual intervention, improve the efficiency and accuracy of metadata governance, and solve the problem that data standards are difficult to be effectively implemented in metadata management. Specifically, the objectives of the present invention include but are not limited to the following aspects: (1) Ensure the consistency and standardization of data: By directly selecting the maintained data standard scheme in the table building process, the present invention aims to standardize the storage structure, data type and naming rules of data from the source, so as to ensure the consistency and standardization of data and reduce the problems caused by data redundancy and inconsistency. (2) Realize dynamic management and real-time synchronization of metadata: The scheme proposed by the present invention closely combines the management of metadata with data standards, realizes the real-time synchronization and update of metadata in an automated manner, ensures that metadata can accurately reflect the actual status of data, and improves the timeliness and accuracy of metadata. (3) Enhance the monitoring and exception alarm capabilities of data standards: By introducing an intelligent standard monitoring mechanism, the present invention can monitor the application of data standards in real time, promptly discover and handle data anomalies that do not conform to the standards, and promptly notify relevant personnel through the alarm system to reduce the risks brought by data errors. (4) Provide intelligent standard optimization suggestions: Based on information such as data usage and standard popularity, the present invention can automatically generate standard optimization suggestions to help data management personnel adjust and optimize data standards in a timely manner according to business development and data changes, thereby improving the flexibility and adaptability of data management. (5) Improve the efficiency and automation level of data governance: By integrating multiple functional modules such as data standard management, metadata management, anomaly detection and alarm, and standard optimization suggestions, the present invention aims to build a comprehensive data governance system to achieve automation and intelligence in data governance, reduce the cost of manual intervention, and improve the efficiency and accuracy of data governance.

[0081] When creating a table in the data warehouse, select the maintained data standard scheme as the attribute of the new field, so that the fields of the data warehouse meet the governance standards at the beginning of creation, ensuring the standardization and consistency of the data. After the table is created, the system will automatically synchronize a copy of the metadata to the metadata management system, and the metadata will be associated with the data standard. By monitoring the changes in data standards and the actual situation of the data in real time, the system can automatically detect and analyze anomalies, and issue alarms to notify relevant personnel in a timely manner when anomalies are found. In addition, the system can also automatically generate standard optimization suggestions based on information such as data usage and standard popularity, providing strong support for the continuous optimization of data standards.

[0082] Figure 3FIG. 1 is a flow chart of a metadata governance method provided by an embodiment of the present application. The method is applied to a governance platform system. Figure 3 As shown, the method includes:

[0083] S301: The data standard definition module defines attribute information of a first data standard field;

[0084] S302: The standardization processing module creates a first data standard field according to the attribute information, and creates a unique identifier for the first data standard field;

[0085] S303: The standardization processing module determines whether there is a second data standard field with the same attribute information as the first data standard field in the standard code table; if not, the standardization processing module creates a subscription channel for the first data standard field, and the subscription channel is used to support the first data standard field to establish a first mapping relationship with the first data warehouse table field of the data warehouse;

[0086] S304: When a first mapping relationship is established between the first data standard field and the first data warehouse table field, the metadata processing module establishes a second mapping relationship between the metadata and the first data standard field through the subscription channel, and stores the metadata in the metadata table;

[0087] S305: The metadata processing module obtains a first storage position of a first data warehouse table field corresponding to the first mapping relationship in the data warehouse; the metadata processing module stores the metadata into a second storage position of the metadata table according to the first storage position.

[0088] In one embodiment, the authority control submodule sets the authority for the standardization processing module to create, modify, and delete the first data standard field.

[0089] Optionally, the permission control submodule determines whether the current system account is an authorized account; if so, the permission for the standardization processing module to create and modify the standardization processing module is set to be enabled;

[0090] as well as,

[0091] The permission control submodule determines whether the current system account is a management account; if so, the permission for the standardization processing module to create, modify, and delete the standardization processing module is set to on, and the permission for adding a description document as the first data standard field is turned on.

[0092] In one embodiment, the metadata processing module includes a monitoring submodule. Before the first data standard field establishes a first mapping relationship with the first data warehouse table field, the monitoring submodule monitors whether the first data standard field establishes a first mapping relationship with the first data warehouse table field.

[0093] In one embodiment, the metadata processing module further includes a grading submodule. The grading submodule configures a preset grade for the metadata according to the first storage location of the first data warehouse table field in the data warehouse;

[0094] The metadata processing module searches the metadata table for a third storage location corresponding to the preset level according to the preset level, and uses the third storage location as the second storage location.

[0095] In one embodiment, the metadata table is stored in a distributed storage system, and the distributed storage system includes more than three nodes, all of which can communicate with each other, and all of the nodes communicate with each other through heartbeat signals. Among them, the first node is a leader node, and the remaining nodes are follower nodes. The leader node is used to store metadata, and the follower node is used to copy the data stored in the lead node. The metadata processing module also includes a high-availability processing submodule.

[0096] The high availability processing submodule stores the metadata in the metadata table corresponding to the leader node, and copies the metadata stored in the leader node to all follower nodes. Also, when all follower nodes do not receive the heartbeat signal of the leader node within a preset time, the first node is judged to be down, and a second node is re-selected from the remaining nodes and the second node is used as the leader node.

[0097] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0098] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0099] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0100] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the embodiments of the method of the metadata governance platform system as described above.

[0101] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0103] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0105] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A metadata governance platform system, characterized in that: include: A data standard definition module, used to define attribute information of a first data standard field; a standardization processing module, configured to create the first data standard field according to the attribute information, and to create a unique identifier for the first data standard field; and to determine whether there is a second data standard field having the same attribute information as the first data standard field in the standard code table; if not, to create a subscription channel for the first data standard field, the subscription channel being used to support the first data standard field to establish a first mapping relationship with a first data warehouse table field of the data warehouse; a metadata processing module, configured to establish a second mapping relationship between metadata and the first data standard field through the subscription channel when the first data standard field and the first data warehouse table field establish the first mapping relationship, and store the metadata in the metadata table; and, for obtaining a first storage position of the first data warehouse table field corresponding to the first mapping relationship in the data warehouse; The metadata is stored in a second storage location of the metadata table according to the first storage location.

2. The metadata governance platform system according to claim 1, characterized in that: The standardization processing module also includes an authority control submodule, and the authority control submodule is used to set the authority for the standardization processing module to create, modify, and delete the first data standard field.

3. The metadata governance platform system according to claim 2, characterized in that: The permission control submodule is used to determine whether the current system account is an authorized account; if so, the permission control submodule is used to set the permission for creating and modifying the standardization processing module to be enabled; as well as, The permission control submodule is used to determine whether the current system account is an administrative account; if so, the permission control submodule is used to set the permission for creating, modifying, and deleting the standardization processing module to be turned on, and the permission for adding a description document to the first data standard field is turned on.

4. The metadata governance platform system according to claim 1, characterized in that: The metadata processing module includes a monitoring submodule, and the monitoring submodule is used to monitor whether the first data standard field establishes the first mapping relationship with the first data warehouse table field.

5. The metadata governance platform system according to claim 1, characterized in that: The attribute information includes an attribute type and an attribute value range corresponding to the attribute type.

6. The metadata governance platform system according to claim 5, characterized in that: The attribute type includes one or more of field name, data type, length, and constraint conditions.

7. The metadata governance platform system according to claim 1, characterized in that: The metadata processing module further includes a grading submodule, the grading submodule being used to configure a preset grade for the metadata according to the first storage location of the first data warehouse table field in the data warehouse; The metadata processing module is used to search the metadata table for a third storage position corresponding to the preset level according to the preset level, and use the third storage position as the second storage position.

8. The metadata governance platform system according to claim 1, characterized in that: The metadata table is stored in a distributed storage system, which includes more than three nodes, all of which can communicate with each other, wherein the first node is a leader node and the remaining nodes are follower nodes; the leader node is used to store the metadata, and the follower node is used to copy the data stored in the lead node; The metadata processing module further includes a high availability processing submodule, and the high availability processing submodule is used to store the metadata into the metadata table corresponding to the leader node; and When all the follower nodes do not receive the heartbeat signal of the leader node within a preset time, the first node is determined to be in a downtime state, and a second node is reselected from the remaining nodes and the second node is used as the leader node.

9. A method for the metadata governance platform system as claimed in claim 1, characterized in that: Applied to the governance platform system, the method includes: The data standard definition module defines the attribute information of the first data standard field; The standardization processing module creates the first data standard field according to the attribute information, and creates a unique identifier for the first data standard field; The standardization processing module determines whether there is a second data standard field in the standard code table with the same attribute information as the first data standard field; if not, the standardization processing module creates a subscription channel for the first data standard field, and the subscription channel is used to support the first data standard field to establish a first mapping relationship with the first data warehouse table field of the data warehouse respectively; When the first data standard field and the first data warehouse table field establish the first mapping relationship, the metadata processing module establishes a second mapping relationship between the metadata and the first data standard field through the subscription channel, and stores the metadata in the metadata table; The metadata processing module obtains a first storage location of the first data warehouse table field corresponding to the first mapping relationship in the data warehouse; the metadata processing module stores metadata into a second storage location of the metadata table according to the first storage location.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to claim 9 is implemented.

Images