A metadata analysis processing method and device

Abstract

The application provides a metadata analysis processing method and device, and relates to the technical field of big data analysis processing.The method comprises the following steps: collecting metadata in a data source; acquiring a business process, declaring the granularity of a metadata model according to the business process, and constructing a dimension table and a fact table according to the granularity; taking the business process as a modeling driver, and establishing a metadata model according to the metadata, the dimension table and the fact table; when a data analysis request is acquired, performing data analysis by using the metadata model to obtain an analysis result.The metadata model is established by collecting metadata, the metadata of enterprises scattered in multiple application systems is centrally managed, a complete and comprehensive data structure relationship is presented for the enterprises, the enterprises are assisted in centrally and efficiently managing data assets, the metadata analysis processing of different business requirements can be met, data value can be mined from the analysis result to guide business or product innovation, and the production of enterprises is empowered.

Description

Technical Field

[0001] This invention relates to the field of big data analysis and processing technology, and more particularly to a metadata analysis and processing method and apparatus. Background Technology

[0002] As enterprises grow, the amount of data collected and used each year increases rapidly, and the volume of data also grows accordingly. Metadata formats are becoming increasingly diverse and inconsistent, posing challenges to the collection, dissemination, and sharing among various data sources. Metadata management, as the core of big data governance, is the foundation and prerequisite for effectively managing this data and plays a crucial role in enterprise IT infrastructure development.

[0003] Building a metadata management system is challenging, yet it's a crucial step in the implementation of a data warehouse project. Understanding and effectively managing metadata to maximize its value, and constructing a stable and robust metadata management system, is the primary task of data warehouse management.

[0004] In summary, there is an urgent need for a technical solution that can overcome the above difficulties and help enterprises manage metadata stably and efficiently. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a metadata analysis and processing method and apparatus that can assist enterprises in managing metadata stably and efficiently, and provide metadata analysis and processing services to meet different business needs.

[0006] In a first aspect of the present invention, a metadata analysis and processing method is proposed, the method comprising:

[0007] Collect metadata from the data source;

[0008] Obtain the business process, declare the granularity of the metadata model based on the business process, and construct the dimension table and fact table based on the granularity;

[0009] Using the business process as the modeling driver, a metadata model is built based on the metadata, dimension table, and fact table.

[0010] When a data analysis request is received, the metadata model is used to perform data analysis and obtain the analysis results.

[0011] Furthermore, metadata from the data source is collected, including:

[0012] Determine the scope of data collection based on business needs;

[0013] The metadata within the data collection scope from the data source is unloaded into a data file, and the data file is loaded into the data warehouse.

[0014] Furthermore, the metadata within the data collection scope from the data source is unloaded into data files, and the data files are loaded into the data warehouse, including:

[0015] Transfer data files to a specified storage server using FTP tools or the mv command;

[0016] Use a data loading tool to reload the data files from the storage server into the data warehouse.

[0017] Furthermore, data loading tools are used to reload data files from the storage server into the data warehouse, including:

[0018] ETL tools are used to extract data files, and the metadata in the data files is cleaned and transformed. The transformed metadata is then loaded into the data warehouse.

[0019] Furthermore, the metadata stored in the data warehouse includes one or more of the following: metadata of the data source, metadata of the data warehouse, and manually entered metadata.

[0020] Furthermore, the business process is obtained, and the granularity of the metadata model is declared based on the business process. Dimension tables and fact tables are then constructed based on the granularity, including:

[0021] Select the business process based on the aforementioned business requirements;

[0022] Based on the granularity of the metadata model declared in the business process, starting from the finest level of granularity, construct dimension tables and fact tables corresponding to multiple levels of granularity from fine to coarse.

[0023] Set the dimension fields in the dimension table according to the granularity;

[0024] The data in the fact table is set according to the metrics of the business process, where the metrics in the same fact table have the same granularity.

[0025] Furthermore, the metadata model includes at least first-level topics such as basic information, event management, and relationship mapping; wherein,

[0026] The basic information includes at least the static attribute information of the individual objects;

[0027] The event management includes at least information on data processing activities and data warehouse operations.

[0028] The relationship mapping includes information on at least multiple connections, mappings, or transformations between objects.

[0029] Furthermore, the basic information includes at least second-level topics such as data structure, data organization, and measurement logic;

[0030] The data structure includes at least a table structure, field descriptions, and field types; the data organization includes at least the architecture and schema for storing data in a data warehouse; and the measurement logic includes at least the logical operation relationships between measurements.

[0031] The event management includes at least a second-level topic, including operation logs, access logs, and ETL processes.

[0032] The operation record includes at least records of operations on the data warehouse; the access record includes at least information about the data accesser and the access time; and the ETL process includes at least rules and procedures for data cleaning and data transformation.

[0033] The relationship mapping includes at least a second-level topic, including kinship, data distribution, and aggregation rules.

[0034] The lineage relationship includes at least the lineage relationship between data, between metadata, and between data and metadata; the data distribution includes at least the data distribution in the data warehouse; and the aggregation rules include at least the aggregation rules for each level of data in the data warehouse.

[0035] Furthermore, the method includes:

[0036] Regularly check the metadata model. If there is any inconsistency between the metadata model and the metadata of the data source, use ETL tools to extract the metadata and update the metadata model.

[0037] Furthermore, when a data analysis request is received, the metadata model is used to perform data analysis to obtain analysis results, including:

[0038] The metadata model is used to perform full-chain analysis, hot / cold analysis, lineage analysis, and / or correlation analysis.

[0039] Furthermore, full-chain analysis is performed using the aforementioned metadata model, including:

[0040] Based on the data structure, data organization, and measurement logic described above, the data quality, data standards, and data security of the entire chain data are analyzed to obtain the full-chain analysis results.

[0041] Furthermore, the aforementioned metadata model is used to perform hot / cold index analysis, including:

[0042] The hotness / coldness of the data is analyzed based on the operation records, access records, and ETL process to obtain the hotness / coldness analysis results.

[0043] Furthermore, kinship analysis is performed using the aforementioned metadata model, including:

[0044] Based on the aforementioned lineage relationships, data distribution, and aggregation rules, lineage tracing is performed on the data to be analyzed. Starting from the data to be analyzed, the relevant metadata objects and their lineage relationships are traced to obtain the lineage analysis results. Among these, lineage tracing includes reverse lineage tracing, forward lineage tracing, and / or full-chain lineage tracing.

[0045] Furthermore, the aforementioned metadata model is used to perform correlation analysis, including:

[0046] Based on the metadata model, the relationship between the data and other data and the processing procedures involved in the data are analyzed to obtain the data usage information, and the correlation analysis results are obtained based on the data usage information.

[0047] Furthermore, the method also includes:

[0048] A physical model and ETL program script are generated based on the metadata, and the ETL process is automatically managed based on the lineage.

[0049] In a second aspect of the present invention, a metadata analysis and processing apparatus is provided, the apparatus comprising:

[0050] The data acquisition module is used to collect metadata from the data source.

[0051] The module is used to obtain the business process, declare the granularity of the metadata model based on the business process, and construct the dimension table and fact table based on the granularity.

[0052] The modeling module is used to build a metadata model based on the metadata, dimension table, and fact table, using the business process as the modeling driver.

[0053] The analysis module is used to perform data analysis using the metadata model when a data analysis request is received, and to obtain the analysis results.

[0054] In a third aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a metadata analysis and processing method.

[0055] In a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements a metadata analysis and processing method.

[0056] The metadata analysis and processing method and apparatus proposed in this invention establishes a metadata model by collecting metadata, centrally manages the metadata of an enterprise that is scattered across multiple application systems, presents a complete and comprehensive data structure relationship to the enterprise, assists the enterprise in managing data assets in a centralized and efficient manner, can meet the metadata analysis and processing needs of different business needs, and can extract data value from the analysis results to guide business or product innovation, thereby empowering enterprise production. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic flowchart of a metadata analysis and processing method according to an embodiment of the present invention.

[0059] Figure 2 This is a detailed flowchart illustrating the process of collecting metadata according to a specific embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram illustrating the relationship between unloading, collecting, transmitting, and loading data according to a specific embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram illustrating the method for collecting metadata according to an embodiment of the present invention.

[0062] Figure 5 This is a detailed flowchart illustrating the model creation process according to an embodiment of the present invention.

[0063] Figure 6 This is a schematic diagram of the first-level topic of the metadata model according to an embodiment of the present invention.

[0064] Figure 7 This is a schematic diagram illustrating the relationship between two levels of topics in a metadata model according to an embodiment of the present invention.

[0065] Figure 8 This is a schematic diagram of the metadata analysis and processing device architecture according to an embodiment of the present invention.

[0066] Figure 9 This is a schematic diagram of the architecture of the analysis module according to an embodiment of the present invention.

[0067] Figure 10 This is a schematic diagram of the architecture of a metadata analysis and processing device according to another embodiment of the present invention.

[0068] Figure 11 This is a schematic diagram of a computer device structure according to an embodiment of the present invention. Detailed Implementation

[0069] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0070] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0071] According to an embodiment of the present invention, a metadata analysis and processing method and apparatus are proposed, relating to the field of big data analysis and processing technology.

[0072] In the embodiments of the present invention, the following terms need to be explained:

[0073] ETL (Extract-Transform-Load) describes the process of extracting, transforming, and loading data from a source to a destination.

[0074] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0075] Figure 1 This is a schematic flowchart of a metadata analysis and processing method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0076] S101, collect metadata from the data source;

[0077] S102, Obtain the business process, declare the granularity of the metadata model according to the business process, and construct the dimension table and fact table according to the granularity;

[0078] S103, Using the business process as the modeling driver, establish a metadata model based on the metadata, dimension table and fact table;

[0079] S104, When a data analysis request is received, the data analysis is performed using the metadata model to obtain the analysis results.

[0080] This invention establishes a metadata model by collecting metadata, centrally managing the metadata of an enterprise that is scattered across multiple application systems. It presents a complete and comprehensive data structure relationship to the enterprise, assists the enterprise in centrally managing data assets, and unifies the supervision of business workflows, data flows, and information flows. It can meet the metadata analysis and processing needs of different business requirements, and extract data value from the analysis results to guide business or product innovation, thereby empowering enterprise production.

[0081] To provide a clearer explanation of the above metadata analysis and processing methods, each step will be described in detail below.

[0082] refer to Figure 2 This is a detailed flowchart illustrating the process of collecting metadata according to a specific embodiment of the present invention. Figure 2 As shown, the detailed process for collecting metadata from the data source is as follows:

[0083] S201, Determine the data collection scope based on business needs;

[0084] S202, unload the metadata in the data source that is within the data collection scope into a data file, and load the data file into the data warehouse.

[0085] Enterprises typically have multiple application systems, such as business systems for various operations and OA office systems. These systems generate a large amount of metadata every day, and the scope of metadata collection can be determined based on the specific business needs of the enterprise.

[0086] After determining the scope of data collection, use ETL tools to import the metadata from these systems into a database. The data needs to be cleaned and transformed before importation to ensure its usability.

[0087] After metadata is collected, integrated, and processed, it is stored in a unified metadata repository to achieve unified management of metadata. Data collection generally involves three steps: data unloading, collection and transmission, and data loading.

[0088] For details, please refer to Figure 3 This is a schematic diagram illustrating the relationship between unloading, collecting, transmitting, and loading data according to a specific embodiment of the present invention. Figure 3 As shown, the specific process is as follows:

[0089] S301 unloads metadata from the data source that is within the data collection scope into a data file.

[0090] The data source can be an upstream system database, where the required data is unloaded into data files.

[0091] S302, transfer data files to a specified storage server via FTP tools or by using the mv command;

[0092] FTP (File Transfer Protocol) is a protocol used on the Internet to transfer files. FTP tools can be used to upload or download files.

[0093] The `mv` command can be used to move files from one directory to another.

[0094] S303 uses a data loading tool to reload data files from the storage server into the data warehouse.

[0095] In practical applications, there are two ways to store metadata: one is to use a separate database for metadata management, and the other is to store it in the same database as business data. Enterprises can choose according to their internal and external environments.

[0096] In one specific embodiment, the specific process of S303 is as follows:

[0097] ETL tools are used to extract data files, and the metadata in the data files is cleaned and transformed. The transformed metadata is then loaded into the data warehouse. ETL tools can extract data files at fixed intervals.

[0098] Since the access method, data scale, business meaning, dataset tables, fields, etc. of the data source may change, data exploration can be performed first. The ETL standardization process is defined based on the data exploration results. The data exploration results can include the null value situation, standardization situation, value range situation, and problem data situation of the fields.

[0099] Then, it is determined whether there is a suitable operator in the ETL process; if there is no suitable operator in the ETL process, a suitable operator is customized through the built-in tool operators and scalar operators; if there is a suitable operator in the ETL process, the ETL process is defined to clean and transform the data, and the cleaned and transformed data is loaded into the data warehouse.

[0100] refer to Figure 4 This is a schematic diagram illustrating the method for collecting metadata according to an embodiment of the present invention. Figure 4 As shown, the metadata stored in the data warehouse includes one or more of the following: metadata of the data source, metadata of the data warehouse, and manually entered metadata.

[0101] Specifically, external data sources include source report metadata, ETL tool metadata, and report tool metadata.

[0102] The metadata of a data warehouse includes physical model metadata and fact table information metadata.

[0103] Manually entered metadata includes metadata such as mapping documents, task configurations, business rules, and business terminology.

[0104] In specific application scenarios, the scope of metadata collection can be defined based on the enterprise's business needs. Although, based on the definition of metadata, any data that can describe data can be managed as metadata, enterprises have multiple application systems, resulting in a large volume and complex structure of metadata. If all metadata is managed, it will lead to an extremely complex metadata model and a waste of database resources. Therefore, this invention collects metadata that can benefit the enterprise based on business needs. The types of metadata can include data business metadata, data operation and maintenance metadata, and data management metadata, etc.

[0105] The detailed process of building the model is explained below in conjunction with S102 and S103.

[0106] refer to Figure 5 This is a detailed flowchart illustrating the model building process according to an embodiment of the present invention. Figure 5 As shown, the specific process is as follows:

[0107] S501, Select a business process based on the business requirements.

[0108] First, it's necessary to identify which business processes the data warehouse should cover. Business process selection is crucial; this step forms the foundation for modeling, and all subsequent steps are based on this business data. Therefore, the first step in modeling is selecting the business processes.

[0109] Dimensional modeling is closely tied to business operations, so it must be based on business principles. Selecting a business process involves choosing the business processes that need to be modeled within the entire business workflow, based on operational requirements and considerations such as future scalability.

[0110] S502, based on the granularity of the metadata model declared in the business process, starting from the finest level of granularity, construct dimension tables and fact tables corresponding to multiple levels of granularity from fine to coarse.

[0111] After defining the business process, declare the granularity of the model. This granularity determines what is represented in the facts. Granularity must be declared before selecting dimensions and facts, as each candidate dimension or fact must be consistent with the defined granularity. Enforcing granularity consistency across all dimensions corresponding to a fact is crucial for ensuring the performance and usability of data warehouse applications. When data is retrieved from a given business process, the raw granularity is the lowest level. It is recommended to start designing with raw granularity data, as raw records can satisfy unpredictable user queries. Aggregated data granularity is important for optimizing query performance, but such granularity often cannot meet the needs for querying detailed data. Different facts can have different granularities, but multiple different granularities should not be mixed within the same fact. After the model is built, it may be necessary to return to this step to modify the granularity level due to the acquisition of new information.

[0112] S503, Set the dimension fields in the dimension table according to the granularity.

[0113] After declaring the granularity, confirm the dimensions of the model. The dimension table is the foundation of the fact table, indicating where the data for the fact table is collected from. The dimension table stores all relevant data for a particular dimension.

[0114] Dimension tables serve as entry points and descriptive identifiers for business analysis. To ensure that no duplicate data appears in dimension tables, the primary keys of dimensions should be unique.

[0115] S504 sets the data in the fact table according to the metrics of the business process, where the metrics in the same fact table have the same granularity.

[0116] After defining the dimension tables, the fact tables are further defined. These are formed by identifying digitized metrics. The fact tables are closely related to the business users of the enterprise's application systems, allowing users to access the data stored in the data warehouse.

[0117] In practical applications, most measures in fact tables are numerical, representing quantitative values ​​that are cumulative and calculable, such as cost, quantity, and amount. Each row in a fact table corresponds to a measure, and the data in each row represents a specific level of detail, known as granularity. One of the core principles of dimensional modeling is that all measures in the same fact table must have the same granularity to ensure that there is no problem of duplicate calculation of measures.

[0118] Within the same fact table, data must have the same granularity. Do not mix different granularities within the same fact table; create separate fact tables for data with different granularities. Furthermore, when retrieving data from a given business process, it is recommended to start designing from the finest granularity to withstand unpredictable user queries. However, the roll-up aggregation granularity is crucial for improving query performance. Therefore, for data with clear requirements, create roll-up aggregation granularities tailored to those requirements; for data with unclear requirements, create atomic granularities.

[0119] S505, using the business process as the modeling driver, establish a metadata model based on the metadata, dimension table, and fact table.

[0120] In one specific embodiment, the metadata model can be checked periodically. If inconsistencies arise between the metadata model and the data source's metadata, an ETL tool is used to extract and update the metadata model. If the data structure of the data source changes, the metadata model is rebuilt. This allows the determination of the metadata update frequency. For example, if long-term or large-scale data analysis is required, it may not be necessary to load the metadata daily, but rather weekly or monthly.

[0121] Metadata modeling is an iterative process that can be flexibly applied according to the enterprise's business and data situation, referring to modeling standards to transform complex underlying data into a simple, complete, and orderly data matrix. Model design needs to consider the relationships between various business processes, effectively combining business data with technical data to form a unified understanding of business definitions and metadata identification. This ensures that metadata has cross-departmental and neutral characteristics, enabling the model to express and cover all business processes, and quickly forming a complete metadata management system.

[0122] refer to Figure 6 This is a schematic diagram of a first-level topic in a metadata model according to an embodiment of the present invention. Figure 6 As shown, the metadata model includes at least first-level topics such as basic information, event management, and relationship mapping; wherein,

[0123] The basic information includes at least the static attribute information of individual objects; for example, a description of the data warehouse structure, and descriptions of data tables and fields.

[0124] The event management includes at least information on data processing behavior and data warehouse operation behavior; for example, information on database operations such as adding, deleting, and modifying data.

[0125] The relational mapping includes information on at least multiple connections, mappings, or transformations between objects. For example, it includes mappings and transformation rules between application systems, between models, and between tables.

[0126] To further refine the segmentation of business data, each topic is expanded into second-level topics. For details, please refer to... Figure 7 This is a schematic diagram illustrating the relationship between two levels of topics in a metadata model according to an embodiment of the present invention. Figure 7 As shown, the basic information includes at least a second-level topic, including data structure, data organization, and measurement logic;

[0127] The data structure includes at least a table structure, field descriptions, and field types; the data organization includes at least the architecture and schema for storing data in a data warehouse; and the measurement logic includes at least the logical operation relationships between measurements.

[0128] The event management includes at least a second-level topic, including operation logs, access logs, and ETL processes.

[0129] The operation record includes at least records of operations on the data warehouse; the access record includes at least information about the data accesser and the access time; and the ETL process includes at least rules and procedures for data cleaning and data transformation.

[0130] The relationship mapping includes at least a second-level topic, including kinship, data distribution, and aggregation rules.

[0131] The lineage relationship includes at least the lineage relationship between data, between metadata, and between data and metadata; the data distribution includes at least the data distribution in the data warehouse; and the aggregation rules include at least the aggregation rules for each level of data in the data warehouse.

[0132] It should be noted that the specific information contained under each topic level is only an example and can be adjusted as needed in actual application.

[0133] The establishment of a metadata model requires the accumulation and iteration of data. By describing, locating, classifying, and summarizing business data with large differences in storage structure, a metadata model is abstracted. Then, based on business relationships, composition relationships and dependency relationships are abstracted, and the subject domains of the metadata model are established, so that the metadata model can express and cover a variety of businesses.

[0134] The data analysis process will be explained in detail below, referring to S104.

[0135] When a data analysis request is received, the metadata model can be used to perform full-chain analysis, hot / cold analysis, lineage analysis, and / or correlation analysis.

[0136] The specific process for performing full-chain analysis using the aforementioned metadata model is as follows:

[0137] Based on the data structure, data organization, and measurement logic described above, the data quality, data standards, and data security of the entire chain data are analyzed to obtain the full-chain analysis results.

[0138] The specific process for performing hot / cold index analysis using the aforementioned metadata model is as follows:

[0139] The hotness / coldness of the data is analyzed based on the operation records, access records, and ETL process to obtain the hotness / coldness analysis results.

[0140] Hot and cold data analysis can identify which data is frequently used by enterprises. Its value lies in visualizing data activity levels, allowing business and management personnel to clearly see the activity level of the data, enabling them to better manage the data and thus supporting self-service data analysis.

[0141] The specific process for performing kinship analysis using the aforementioned metadata model is as follows:

[0142] Based on the aforementioned lineage relationships, data distribution, and aggregation rules, lineage tracing is performed on the data to be analyzed. Starting from the data to be analyzed, the relevant metadata objects and their lineage relationships are traced to obtain the lineage analysis results. Among these, lineage tracing includes reverse lineage tracing, forward lineage tracing, and / or full-chain lineage tracing.

[0143] Lineage analysis can comprehensively track the data processing process, such as where the data comes from, what processing processes it has undergone, and find all related metadata and their lineage relationships.

[0144] The specific process for performing correlation analysis using the aforementioned metadata model is as follows:

[0145] Based on the metadata model, the relationship between the data and other data and the processing procedures involved in the data are analyzed to obtain the data usage information, and the correlation analysis results are obtained based on the data usage information.

[0146] Relationship analysis can analyze the relationships between data and other data, and how these relationships are established. It examines the usage of specific data from two perspectives: the other entities associated with a particular entity, and the processing processes that the entity participates in. This forms a network of entities and the processes they participate in, thereby further understanding the importance of the entity. For example, the relationships between tables and ETL programs, tables and analytical applications, and tables and other tables. Relationship analysis can also support the impact assessment of requirement changes.

[0147] This invention centralizes and manages metadata. When analyzing a specific business process or stage, or when changes occur in that process, the metadata model can be used to analyze in real time the affected business functions, enterprise application systems, involved personnel, and whether regulatory oversight is involved. Metadata analysis and processing enable rapid identification of data sources and processing steps, helping data analysts quickly pinpoint data issues. Metadata lineage analysis allows understanding the relationships between different data metrics and analyzing the impact of fluctuations in the data sources that generate those metrics.

[0148] In one embodiment, metadata can also be used to achieve automated ETL management, specifically as follows:

[0149] A physical model and ETL program script are generated based on the metadata, and the ETL process is automatically managed based on the lineage.

[0150] In practical applications, metadata models can be used to establish lineage, influence, and association relationships among metadata, enabling impact analysis, synchronization checks, indicator consistency analysis, and entity association queries.

[0151] Specifically, impact analysis can trace the impact of metadata objects on downstream processes; synchronization checks can check whether the data structure from the source table to the target table has changed; through indicator consistency analysis, it is possible to periodically analyze whether the indicator definitions are consistent with the actual situation; and during entity association queries, the surrogate keys of the fact table and dimension table are automatically associated to query the association between entities.

[0152] This invention can establish a mature metadata model, unify metadata management, divide the metadata model into multiple themes, meet various data analysis requests, support full-chain analysis, hot and cold analysis, lineage analysis, and correlation analysis, and assist enterprises in managing data assets in a centralized and efficient manner.

[0153] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0154] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 8 The metadata analysis and processing apparatus of exemplary embodiments of the present invention will be described.

[0155] The implementation of the metadata analysis and processing apparatus can refer to the implementation of the methods described above, and will not be repeated here. The terms "module" or "unit" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0156] Based on the same inventive concept, this invention also proposes a metadata analysis and processing device, such as... Figure 8 As shown, the device includes:

[0157] The acquisition module 810 is used to acquire metadata from the data source;

[0158] The construction module 820 is used to obtain the business process, declare the granularity of the metadata model according to the business process, and construct the dimension table and fact table according to the granularity.

[0159] Modeling module 830 is used to build a metadata model based on the metadata, dimension table and fact table, using the business process as the modeling driver.

[0160] The analysis module 840 is used to perform data analysis using the metadata model when a data analysis request is received, and to obtain analysis results.

[0161] It should be noted that although several modules of the metadata analysis and processing apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0162] In one embodiment, the acquisition module 810 is specifically used to: determine the data acquisition scope according to business needs; unload the metadata in the data acquisition scope from the data source into a data file, and load the data file into the data warehouse.

[0163] Specifically, data files can be transferred to a specified storage server using FTP tools or the mv command; and data loading tools can be used to reload the data files from the storage server into the data warehouse.

[0164] Data loading tools can be ETL tools. ETL tools extract data files, perform data cleaning and transformation on the metadata in the data files, and load the transformed metadata into the data warehouse.

[0165] In one embodiment, the metadata stored in the data warehouse includes one or more of the following: metadata of the data source, metadata of the data warehouse, and manually entered metadata.

[0166] In one embodiment, the construction module 820 is specifically used for: selecting a business process according to the business requirements; constructing dimension tables and fact tables corresponding to multiple levels of granularity from fine to coarse, starting from the finest level of granularity, according to the granularity of the metadata model declared by the business process; setting the dimension fields in the dimension tables according to the granularity; and setting the data in the fact tables according to the metrics of the business process, wherein the metrics in the same fact table have the same granularity.

[0167] In one embodiment, the metadata model established by the modeling module 830 has at least three first-level topics and nine second-level topics.

[0168] Specifically, the metadata model includes at least first-level topics such as basic information, event management, and relationship mapping; among which,

[0169] The basic information includes at least the static attribute information of the individual objects;

[0170] The event management includes at least information on data processing activities and data warehouse operations.

[0171] The relationship mapping includes information on at least multiple connections, mappings, or transformations between objects.

[0172] The second-level topics are explained in detail below:

[0173] The basic information includes at least a second-level topic, including data structure, data organization, and measurement logic.

[0174] The data structure includes at least a table structure, field descriptions, and field types; the data organization includes at least the architecture and schema for storing data in a data warehouse; and the measurement logic includes at least the logical operation relationships between measurements.

[0175] The event management includes at least a second-level topic, including operation logs, access logs, and ETL processes.

[0176] The operation record includes at least records of operations on the data warehouse; the access record includes at least information about the data accesser and the access time; and the ETL process includes at least rules and procedures for data cleaning and data transformation.

[0177] The relationship mapping includes at least a second-level topic, including kinship, data distribution, and aggregation rules.

[0178] The lineage relationship includes at least the lineage relationship between data, between metadata, and between data and metadata; the data distribution includes at least the data distribution in the data warehouse; and the aggregation rules include at least the aggregation rules for each level of data in the data warehouse.

[0179] In one embodiment, the analysis module 840 is specifically used to: perform full-chain analysis, hot / cold analysis, lineage analysis, and / or correlation analysis using the metadata model.

[0180] refer to Figure 9 This is a schematic diagram of the architecture of an analysis module according to an embodiment of the present invention. Figure 9 As shown, the analysis module 840 includes:

[0181] The full-chain analysis unit 841 is used to analyze the data quality, data standards and data security of the full-chain data according to the data structure, data organization and measurement logic, and obtain the full-chain analysis results.

[0182] The hot / cold index analysis unit 842 is used to analyze the hot / cold index of the data based on the operation record, access record and ETL process, and obtain the hot / cold index analysis result.

[0183] The lineage analysis unit 843 is used to perform lineage tracing on the data to be analyzed according to the lineage relationship, data distribution and aggregation rules. Starting from the data to be analyzed, it tracks the relevant metadata objects and the lineage relationship of the relevant metadata objects to obtain the lineage analysis result. The lineage tracing includes reverse lineage tracing, forward lineage tracing and / or full-chain lineage tracing.

[0184] The correlation analysis unit 844 is used to analyze the relationship between the data and other data and the processing process in which the data participates based on the metadata model, to obtain the data usage, and to obtain the correlation analysis result based on the data usage.

[0185] refer to Figure 10 This is a schematic diagram of the architecture of a metadata analysis and processing device according to another embodiment of the present invention. Figure 10 As shown, the device also includes:

[0186] The inspection module 850 is used to periodically check the metadata model. If there is an inconsistency between the metadata model and the metadata of the data source, the metadata model is updated by extracting metadata using ETL tools.

[0187] See again Figure 10 The system also includes:

[0188] ETL management module 860 is used to generate physical models and ETL program scripts based on the metadata, and automatically manage the ETL process based on the lineage relationship.

[0189] Based on the aforementioned inventive concept, such as Figure 11 As shown, the present invention also proposes a computer device 1100, including a memory 1110, a processor 1120, and a computer program 1130 stored in the memory 1110 and executable on the processor 1120. When the processor 1120 executes the computer program 1130, it implements the aforementioned metadata analysis and processing method.

[0190] Based on the aforementioned inventive concept, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned metadata analysis and processing method.

[0191] The metadata analysis and processing method and apparatus proposed in this invention establishes a metadata model by collecting metadata, centrally manages the metadata of an enterprise that is scattered across multiple application systems, presents a complete and comprehensive data structure relationship to the enterprise, assists the enterprise in managing data assets in a centralized and efficient manner, can meet the metadata analysis and processing needs of different business needs, and can extract data value from the analysis results to guide business or product innovation, thereby empowering enterprise production.

[0192] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0193] This invention is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0194] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a processFigure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0195] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0196] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A metadata analysis and processing method, characterized in that, The method includes: Collect metadata from the data source; Obtain the business process, declare the granularity of the metadata model based on the business process, and construct the dimension table and fact table based on the granularity; Using the business process as the modeling driver, a metadata model is built based on the metadata, dimension table, and fact table. When a data analysis request is received, the metadata model is used to perform data analysis and obtain analysis results. Specifically, the business process is used as the modeling driver, and a metadata model is established based on the metadata, dimension tables, and fact tables, including: Through data accumulation and iteration, a metadata model is abstracted by describing, locating, classifying, and summarizing business data with large differences in storage structure. Composition and dependency relationships are abstracted for business relationships, and subject areas of the metadata model are established. In particular, according to the enterprise's business and data situation, the complex underlying data is transformed into a data cube. When designing the model, the relationships between businesses are considered, and business data is combined with technical data to form a unified understanding of business definition and metadata identification. This gives the metadata cross-departmental and neutral characteristics, enabling the model to express and cover all business, forming a complete metadata management system. When a data analysis request is received, the metadata model is used to perform data analysis and obtain analysis results, including: The metadata model is used for full-chain analysis, hot / cold analysis, lineage analysis, and correlation analysis. Metadata is centrally maintained and managed. When analyzing a specific business or process, or when changes occur in a business or process, the metadata model is used to analyze in real time the affected business functions, enterprise application systems, involved personnel, and whether regulatory oversight is involved. Metadata analysis and processing pinpoint data sources and processing steps, identifying data issues. Metadata lineage analysis helps understand the relationships between different data indicators and analyze the impact of fluctuations in the data source that generates these indicators. The hot / cold data analysis is used to analyze commonly used enterprise data, making data activity levels visible and providing support for self-service data analysis. The correlation analysis is used to analyze the relationship between data and other data and how these relationships are established. When viewing the usage of specific data from other entities associated with an entity and the processing processes that the entity participates in, a network of entities and the processing processes they participate in is formed to determine the importance of entities. This includes the correlation between tables and ETL programs, tables and analytical applications, and tables and other tables. The correlation analysis is used to support the impact assessment of requirement changes. Based on the metadata model, the lineage, influence, and association relationships between metadata are established. Impact analysis, synchronization checks, indicator consistency analysis, and entity association queries are performed on the metadata. For impact analysis, the influence of metadata objects on downstream processes is traced downwards. For synchronization checks, the data structure from the source table to the target table is checked for changes. For indicator consistency analysis, the consistency between indicator definitions and actual situations is analyzed periodically. For entity association queries, the surrogate keys of the fact table and dimension table are automatically associated to query the associations between entities.

2. The metadata analysis and processing method according to claim 1, characterized in that, Collect metadata from the data source, including: Determine the scope of data collection based on business needs; The metadata within the data collection scope from the data source is unloaded into a data file, and the data file is loaded into the data warehouse.

3. The metadata analysis and processing method according to claim 2, characterized in that, Unloading metadata from the data source that falls within the data collection scope into data files, and loading these data files into the data warehouse, including: Transfer data files to a specified storage server using FTP tools or the mv command; Use a data loading tool to reload the data files from the storage server into the data warehouse.

4. The metadata analysis and processing method according to claim 3, characterized in that, Using data loading tools, data files from the storage server are reloaded into the data warehouse, including: ETL tools are used to extract data files, and the metadata in the data files is cleaned and transformed. The transformed metadata is then loaded into the data warehouse.

5. The metadata analysis and processing method according to claim 2, characterized in that, The metadata stored in the data warehouse includes one or more of the following: metadata from the data source, metadata from the data warehouse, and manually entered metadata.

6. The metadata analysis and processing method according to claim 2, characterized in that, The metadata model includes at least first-level topics such as basic information, event management, and relationship mapping; among which... The basic information includes at least the static attribute information of the individual objects; The event management includes at least information on data processing activities and data warehouse operations. The relationship mapping includes information on at least multiple connections, mappings, or transformations between objects.

7. The metadata analysis and processing method according to claim 6, characterized in that, The basic information includes at least a second-level topic, including data structure, data organization, and measurement logic. The data structure includes at least a table structure, field descriptions, and field types; the data organization includes at least the architecture and schema for storing data in a data warehouse; and the measurement logic includes at least the logical operation relationships between measurements. The event management includes at least a second-level topic, including operation logs, access logs, and ETL processes. The operation record includes at least records of operations on the data warehouse; the access record includes at least information about the data accesser and the access time; and the ETL process includes at least rules and procedures for data cleaning and data transformation. The relationship mapping includes at least a second-level topic, including kinship, data distribution, and aggregation rules. The lineage relationship includes at least the lineage relationship between data, between metadata, and between data and metadata; the data distribution includes at least the data distribution in the data warehouse; and the aggregation rules include at least the aggregation rules for each level of data in the data warehouse.

8. The metadata analysis and processing method according to claim 7, characterized in that, The method includes: Regularly check the metadata model. If there is any inconsistency between the metadata model and the metadata of the data source, use ETL tools to extract the metadata and update the metadata model.

9. The metadata analysis and processing method according to claim 7, characterized in that, Full-chain analysis using the aforementioned metadata model includes: Based on the data structure, data organization, and measurement logic described above, the data quality, data standards, and data security of the entire chain data are analyzed to obtain the full-chain analysis results.

10. The metadata analysis and processing method according to claim 7, characterized in that, The hot / cold index analysis is performed using the aforementioned metadata model, including: The hotness / coldness of the data is analyzed based on the operation records, access records, and ETL process to obtain the hotness / coldness analysis results.

11. The metadata analysis and processing method according to claim 7, characterized in that, Using the aforementioned metadata model for kinship analysis includes: Based on the aforementioned lineage relationships, data distribution, and aggregation rules, lineage tracing is performed on the data to be analyzed. Starting from the data to be analyzed, the relevant metadata objects and their lineage relationships are traced to obtain the lineage analysis results. Among these, lineage tracing includes reverse lineage tracing, forward lineage tracing, and full-chain lineage tracing.

12. The metadata analysis and processing method according to claim 7, characterized in that, The association analysis is performed using the aforementioned metadata model, including: Based on the metadata model, the relationship between the data and other data and the processing procedures involved in the data are analyzed to obtain the data usage information, and the correlation analysis results are obtained based on the data usage information.

13. The metadata analysis and processing method according to claim 8, characterized in that, The method also includes: A physical model and ETL program script are generated based on the metadata, and the ETL process is automatically managed based on the lineage.

14. A metadata analysis and processing device, characterized in that, The device includes: The data acquisition module is used to collect metadata from the data source. The module is used to obtain the business process, declare the granularity of the metadata model based on the business process, and construct the dimension table and fact table based on the granularity. The modeling module is used to build a metadata model based on the metadata, dimension table, and fact table, using the business process as the modeling driver. The analysis module is used to perform data analysis using the metadata model when a data analysis request is received, and to obtain analysis results. Specifically, the modeling module is used for: Through data accumulation and iteration, a metadata model is abstracted by describing, locating, classifying, and summarizing business data with large differences in storage structure. Composition and dependency relationships are abstracted for business relationships, and subject areas of the metadata model are established. In particular, according to the enterprise's business and data situation, the complex underlying data is transformed into a data cube. When designing the model, the relationships between businesses are considered, and business data is combined with technical data to form a unified understanding of business definition and metadata identification. This gives the metadata cross-departmental and neutral characteristics, enabling the model to express and cover all business, forming a complete metadata management system. Specifically, the analysis module is used for: The metadata model is used for full-chain analysis, hot / cold analysis, lineage analysis, and correlation analysis. Metadata is centrally maintained and managed. When analyzing a specific business or process, or when changes occur in a business or process, the metadata model is used to analyze in real time the affected business functions, enterprise application systems, involved personnel, and whether regulatory oversight is involved. Metadata analysis and processing pinpoint data sources and processing steps, identifying data issues. Metadata lineage analysis helps understand the relationships between different data indicators and analyze the impact of fluctuations in the data source that generates these indicators. The hot / cold data analysis is used to analyze commonly used enterprise data, making data activity levels visible and providing support for self-service data analysis. The correlation analysis is used to analyze the relationship between data and other data and how these relationships are established. When viewing the usage of specific data from other entities associated with an entity and the processing processes that the entity participates in, a network of entities and the processing processes they participate in is formed to determine the importance of entities. This includes the correlation between tables and ETL programs, tables and analytical applications, and tables and other tables. The correlation analysis is used to support the impact assessment of requirement changes. Based on the metadata model, the lineage, influence, and association relationships between metadata are established. Impact analysis, synchronization checks, indicator consistency analysis, and entity association queries are performed on the metadata. For impact analysis, the influence of metadata objects on downstream processes is traced downwards. For synchronization checks, the data structure from the source table to the target table is checked for changes. For indicator consistency analysis, the consistency between indicator definitions and actual situations is analyzed periodically. For entity association queries, the surrogate keys of the fact table and dimension table are automatically associated to query the associations between entities.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

Images