Data blood relationship management method, device, equipment, medium and product
By dynamically registering and synchronizing lineage information in the metadata management module, and combining it with the engine's lineage relationship for full-link parsing, the problem of insufficient accuracy in metadata lineage analysis is solved, and efficient data lineage relationship management is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2025-06-24
- Publication Date
- 2026-08-04
AI Technical Summary
Existing metadata lineage analysis technologies suffer from inconsistent data source formats, making it difficult to accurately identify lineage relationships and impacting data problem identification and business decision-making.
The business application installation package is generated based on metadata. The lineage relationship is dynamically registered in the metadata management module through the lineage relationship installation package, and the lineage information is registered synchronously when the data application interface is launched. The lineage relationship is combined with the engine for full-link parsing.
It enables automatic deduction of lineage relationships when users access data, improving the accuracy and efficiency of parsing, reducing manual adaptation and parsing workload, and meeting rapidly changing business needs.
Smart Images
Figure CN120763123B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a data lineage management method, apparatus, equipment, storage medium, and program product. Background Technology
[0002] With the advent of the big data era and the surge in data volume, metadata management, as a core concept of data governance, has gained significant attention. Service metadata lineage technology has been introduced and developed to better manage and understand data. Service metadata lineage describes and tracks the flow and changes of data within services. This technology can be implemented through metadata-based approaches, data flow graph-based approaches, and data warehouse-based approaches. These technologies help enterprises streamline data usage chains, optimize data asset management, and improve data quality and application value.
[0003] Metadata lineage analysis refers to establishing relationships between metadata by tracing their source, processing, and destination, thereby analyzing the scope of influence and dependencies of metadata. This helps in understanding issues related to data lifecycle, data quality, data security, and data governance.
[0004] Existing metadata lineage analysis techniques primarily determine the relationships between data objects by reverse-analyzing the data flow logic. However, due to the different systems and formats of data sources, it is difficult to standardize the data source formats, making it difficult to accurately identify lineage relationships. This affects the identification of data problems and business decisions, thus the technique suffers from accuracy issues. Summary of the Invention
[0005] This application provides a data lineage management method, apparatus, device, storage medium, and program product, which solves the problem of insufficient accuracy in determining the relationship between data objects using existing reverse data stream parsing logic.
[0006] Firstly, to achieve the above objectives, embodiments of this application provide a data lineage management method, comprising:
[0007] The business application installation package is generated based on metadata, and the business application installation package includes lineage installation packages.
[0008] In runtime, the lineage installation package is deployed in the metadata management module, and the metadata lineage is dynamically registered in the metadata management module.
[0009] When the data application programming interface (API) is launched, the API lineage information is synchronously registered to the metadata management module.
[0010] The lineage installation package is distributed to the data management-related subsystem, and the engine lineage generated by the subsystem during task execution is registered to the metadata management module. The engine lineage is related to the lineage installation package.
[0011] When a user accesses target data through a data application programming interface (API), the metadata management module is controlled to perform end-to-end lineage analysis on the target data based on the API definition, metadata lineage, API lineage information, and engine lineage, and then displays the analysis results.
[0012] The process of generating a business application installation package based on metadata includes:
[0013] In the process of modeling user data and orchestrating data processing logic, the attributes and relationships of the data model are described through metadata definitions;
[0014] Define the data processing logic using a domain-specific language.
[0015] Based on the description of attributes and relationships in the data model and the definition of data processing logic, field-level lineage parsing is performed to obtain lineage relationship data;
[0016] The bloodline data is compiled to obtain a bloodline installation package, and the bloodline installation package is packaged into the business application installation package.
[0017] Specifically, in runtime, the lineage relationship installation package is deployed in the metadata management module, and the metadata lineage relationship is dynamically registered in the metadata management module, including:
[0018] Based on the business application installation package, execute the business application installation process;
[0019] During the installation of the business application, the lineage installation package is installed and deployed to the metadata management module. The metadata management module performs deployment representation of the lineage installation package and dynamically registers the metadata lineage to the metadata management module.
[0020] The method further includes at least one of the following:
[0021] In the event of an update to the business application, the metadata lineage in the metadata management module is updated dynamically.
[0022] When the business application is uninstalled, the metadata lineage in the metadata management module is automatically deleted.
[0023] Specifically, when the data application programming interface (API) is launched, the API lineage information is synchronously registered with the metadata management module, including at least one of the following:
[0024] After the user completes the development of the data application interface in the data application interface open module and puts the data application interface online to the application interface gateway, the data application interface lineage information is synchronously registered to the metadata management module.
[0025] When a user explores data on the target interface, the system redirects to the data application programming interface (API) open module based on the developed structured query language (SQL) to create the API. If the created API is then uploaded to the API management platform, the API lineage information is synchronously registered to the metadata management module.
[0026] The method further includes:
[0027] When the data application interface is taken offline, the metadata management module is notified to unregister the lineage information of the data application interface.
[0028] Secondly, to achieve the above objectives, embodiments of this application provide a data lineage management device, comprising:
[0029] The generation module is used to generate business application installation packages based on metadata. The business application installation packages include lineage-related installation packages.
[0030] The first registration module is used to dynamically register metadata lineage relationships in the metadata management module by deploying the lineage relationship installation package in the metadata management module;
[0031] The second registration module is used to synchronously register the data application interface lineage information to the metadata management module when the data application interface is launched.
[0032] The processing module is used to distribute the lineage installation package to the data management-related subsystem and register the engine lineage generated by the subsystem during task execution to the metadata management module. The engine lineage is related to the lineage installation package.
[0033] The control module is used to control the metadata management module to perform end-to-end lineage parsing on the target data based on the interface definition of the data application interface, the metadata lineage, the data application interface lineage information, and the engine lineage when the user accesses the target data through the data application interface, and to display the parsing results.
[0034] Thirdly, to achieve the above objectives, embodiments of this application provide a data lineage management device, including a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; the transceiver is used to send and receive data under the control of the processor, and the processor implements the data lineage management method as described in the first aspect when executing the program.
[0035] Fourthly, to achieve the above objectives, embodiments of this application provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the data lineage management method as described in the first aspect.
[0036] Fifthly, to achieve the above objectives, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the data lineage management method as described in the first aspect.
[0037] The beneficial effects of the above technical solution in this application are as follows:
[0038] In the embodiments of this application, firstly, a business application installation package is generated based on metadata, and the business application installation package includes a lineage relationship installation package; thus, a data model is built based on metadata. Secondly, the lineage relationship installation package is deployed in the metadata management module, and the metadata lineage relationship is dynamically registered in the metadata management module. Thirdly, when the data application interface goes online, the data application interface lineage information is synchronously registered to the metadata management module. Then, the lineage relationship installation package is distributed to the subsystems related to data management, and the engine lineage relationship generated by the subsystem during task execution is registered to the metadata management module. The engine lineage relationship is related to the lineage relationship installation package. Thus, the installation of relevant tools / models for deriving lineage relationships is realized. Finally, when a user accesses target data through the data application interface, the metadata management module is controlled to perform full-link lineage parsing on the target data based on the interface definition of the data application interface, the metadata lineage relationship, the data application interface lineage information, and the engine lineage relationship, and the parsing results are displayed. In this way, when users access data, the lineage relationship can be automatically deduced using a data model built on metadata, improving the accuracy and efficiency of the analysis. Attached Figure Description
[0039] Figure 1 This is one of the flowcharts illustrating the data lineage management method according to an embodiment of this application;
[0040] Figure 2 This is a second schematic flowchart of the data lineage management method according to an embodiment of this application;
[0041] Figure 3 This is a schematic diagram of the processing flow in the design state in an embodiment of this application;
[0042] Figure 4 This is a schematic diagram of the data lineage management device according to an embodiment of this application;
[0043] Figure 5 This is a schematic diagram of the data lineage management device according to an embodiment of this application. Detailed Implementation
[0044] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.
[0045] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.
[0046] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0047] In addition, the terms "system" and "network" are often used interchangeably in this article.
[0048] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.
[0049] Before describing the embodiments of this application, the relevant technical points will be explained first:
[0050] The realization of metadata and data lineage is not just about metadata collection and lineage resolution. To truly realize metadata and data lineage and put them into practical application, a series of processes must be followed, including metadata collection and lineage resolution, construction of meta-models and graph models, storage in databases, construction of data services, and implementation of data applications based on metadata and data lineage.
[0051] The underlying data sources for metadata and lineage primarily consist of data components commonly used in data development. These include:
[0052] Databases, such as MySQL / Hive, are used to retrieve metadata information such as tables, fields, schemas, indexes, and total counts of data.
[0053] Data storage: such as HDFS / OSS, to obtain metadata information such as the storage path and number of files of the data.
[0054] Computing engines such as Flink / Spark can provide task information for data development jobs. They can also retrieve FlinkSQL and SparkSQL, and resolve corresponding table-level dependency relationships.
[0055] Message queues, such as Kafka / RocketMQ, can provide metadata information such as the message queue, topics, and subscribers of the current data. For data lineage, message queues can record the data transmission path nodes.
[0056] Visual reports: such as Tableau / FineReport, can provide metadata information such as reports, dashboards, and charts. For data lineage, it can record the data source and relationships of the report.
[0057] Scheduling systems, such as DolphinScheduler / Airflow, provide metadata information such as scheduling time, scheduling rate, and triggering conditions. Lineage analysis can be performed by obtaining the SQL scripts inside the scheduling system.
[0058] However, existing kinship analysis techniques primarily rely on reverse parsing of data processing stream logic to obtain kinship relationships between data objects. The main problems with this approach include the following:
[0059] The accuracy of data lineage information is a concern: The current architecture, due to data sources from different systems and formats, makes it difficult to unify data and accurately identify lineage relationships. This results in an inability to accurately reflect data flow paths and relationships. Inaccurate lineage information, such as incorrect data flow diagrams or incomplete data transmission paths, makes it difficult for organizations to identify and correct data quality issues, thereby impacting the effectiveness of business decisions and data management.
[0060] Data lineage coverage and timeliness issues: The limitation of the current architecture is that many organizations' data lineages only cover a portion of the data flow, rather than the entire data lifecycle. This limitation prevents organizations from fully understanding the data flow path and its impact, restricting the effectiveness of data governance; data lineage information should be updated in real-time or near real-time to reflect the latest state of current data flow. The current architecture cannot update lineage information in real time, resulting in an inability to meet rapidly changing business needs.
[0061] The complex processes make it impossible to view the data processing chain from a global perspective: In open data scenarios, data flow via data application programming interfaces (APIs) lacks lineage analysis capabilities, leading to breakpoints in the data lineage chain. This makes it impossible to comprehensively track the data flow and, when encountering data issues or changes in data processes, it's impossible to determine the specific impact and affected points. Here, "Application Programming Interface" (API) is used interchangeably with "Data Application Programming Interface."
[0062] In view of this, embodiments of this application provide a data lineage management method, such as... Figure 1 As shown, it includes:
[0063] Step 101: In design mode, generate a business application installation package based on metadata. The business application installation package includes a lineage relationship installation package. Specifically, this step can be implemented in design mode on a data development platform, i.e., generating a business application installation package including a lineage relationship installation package based on metadata on the data development platform. The lineage relationship installation package may include, for example, a lineage relationship resolver used to parse the lineage relationships of data. The application can also be referred to as an Application, or simply APP; the terms APP and Application are interchangeable in the following text.
[0064] Step 102, by deploying the bloodline installation package in the metadata management module (e.g., Figure 2 The second step, the metadata / lineage installation package, dynamically registers metadata lineage relationships in the metadata management module. Specifically, this step can be a runtime step executed on devices (such as servers or systems) in a production environment (e.g., a customer's production environment). That is, the business application is installed in the production environment based on the business application installation package, and the lineage installation package is deployed in the metadata management module of the production environment. This step, by deploying the lineage installation package in the metadata management module, dynamically registers metadata lineage relationships in the metadata management module, enabling the metadata management module to automatically analyze data lineage relationships.
[0065] Step 103: When the data application programming interface (API) goes online, synchronize and register the API lineage information to the metadata management module (e.g., ...). Figure 2 (④ API lineage collection and analysis). This enables lineage analysis capabilities even when data flows through data application programming interfaces, preventing breakpoints in the data lineage chain and allowing for comprehensive tracking of the data chain. This allows for the identification of specific impact areas and points when data issues or data process changes occur.
[0066] Step 104: Distribute the lineage installation package to the subsystem related to data management, and register the engine lineage generated by the subsystem during task execution with the metadata management module (e.g., ...). Figure 2 ③ In the section on "Collection of Shadow Bloodline Relationships," the engine bloodline relationship is related to the bloodline relationship installation package; wherein, the subsystem is, for example, a subsystem related to data management, specifically... Figure 2 For example, a subsystem may include one or more of a control subsystem, a computing subsystem, and a storage subsystem, but is not limited thereto. Specifically, the step of "distributing the lineage installation package to the data management-related subsystem" is implemented within the business application installation and deployment process.
[0067] In other words, during the execution of tasks from application installation and distribution to big data computing and storage engines, the task relationship lineage of physical engines is generated. This lineage is then reported to the metadata management module (or metadata management service) via a reporting interface, thereby enabling the registration and management of the task relationship lineage of physical engines.
[0068] It should be noted that steps 102, 103 and 104 in the embodiments of this application are all steps executed in the running state, and the order of execution is not limited. That is, at least two of the above three steps can be executed simultaneously or sequentially, and the order of execution is not limited.
[0069] Step 105: When a user accesses target data through a data application programming interface (API), the metadata management module performs full-link lineage analysis on the target data based on the API definition, metadata lineage, API lineage information, and engine lineage, and displays the analysis results. That is, for API metadata information accessed by upper-layer services through the API, the metadata management module / service collects the API definition to deduce and analyze the full-link lineage of the entire data lifecycle from API to model to task (task scheduling in lineage management, tasks during application installation and deployment), thus achieving API lineage management.
[0070] In the data lineage management method of this application embodiment, firstly, a business application installation package is generated based on metadata, and the business application installation package includes a lineage installation package; secondly, by deploying the lineage installation package in the metadata management module, metadata lineage relationships are dynamically registered in the metadata management module, enabling the metadata management module to automatically analyze data lineage relationships using the deployed lineage installation package; when the data application interface goes online, the data application interface lineage information is synchronously registered to the metadata management module; the lineage installation package is distributed to subsystems related to data management, and the engine lineage relationships generated by the subsystems during task execution are registered to the metadata management module, the engine lineage relationships being related to the lineage installation package; finally, when a user accesses target data through the data application interface, the metadata management module is controlled to perform full-link lineage parsing on the target data based on the interface definition of the data application interface, the metadata lineage relationships, the data application interface lineage information, and the engine lineage relationships, and the parsing results are displayed. In this way, a business application installation package, including a lineage relationship installation package, is designed based on metadata on the data development platform. The business application installation package is then installed and the lineage relationship installation package is deployed in the customer's production environment. This enables the automatic deduction of the lineage relationship of data accessed by users through the data application interface. As a result, forward parsing of data lineage relationships based on metadata is achieved, reducing the workload of manual adaptation and parsing, and improving the accuracy and efficiency of parsing.
[0071] As an optional implementation, step 101 includes:
[0072] In the process of modeling user data and orchestrating data processing logic, the attributes and relationships of the data model are described through metadata definitions; among them, such as Figure 2 As shown, the mathematical model obtained by modeling user data is a three-layer model, including, for example, a business model, a logical model, and a physical model; that is, user data modeling and data processing logic orchestration are implemented in the design state (data development platform); in this process, the attributes and relationships of the three-layer model (business model, logical model, and physical model) can be described by defining metadata.
[0073] The data processing logic is defined using a domain-specific language (DSL).
[0074] Based on the description of attributes and relationships in the data model and the definition of data processing logic, field-level lineage parsing is performed to obtain lineage relationship data; among which, based on the above definitions, it is possible to provide, for example... Figure 2The data lineage parser shown derives field-level lineage relationships through model relationships and computational relationships, thereby obtaining lineage relationship data.
[0075] The bloodline data is compiled to obtain a bloodline installation package, and the bloodline installation package is then packaged into the business application installation package. For example, this step can be performed by... Figure 2 The compiler implementation in [the context].
[0076] Here, it should be noted that, Figure 2 The implementation process of ① in the design state is as follows: Figure 3 As shown, where, Figure 3 Corresponding to Figure 2 The blood relation resolver in the system includes:
[0077] Step 1: Dimensional Modeling: Based on the multidimensional modeling theory of Data Warehouse Master, we construct a dimensional system, primarily consisting of dimension tables and fact tables. The fact table is the central element, with all dimensions directly linked to it in a star-shaped distribution. Typically, a multidimensional data model is built from a fact table (containing data describing specific events within the business) and multiple associated dimension tables (containing the data upon which the dimensions are based). A dimension generally has one dimension table (or possibly multiple), and the fact table has a field that links to the dimension tables. For example, sales data might have a "customer" dimension, with a "customers" dimension table containing attributes such as customer name, gender, and region. The fact table would have a field called "customer_id," linked to the "customers" dimension table.
[0078] Step Two: Metric Modeling: Atomic metrics and measures have the same meaning. A metric is based on a specific business event behavior and is an indivisible indicator in the business definition. It is a term with a clear business meaning, such as payment amount. Typically, a metric / atomic indicator is a data value, where the intersection of the dimensions of a multidimensional data model is the metric.
[0079] Step 3: Basic Aggregation Layer: Based on the dimensional system and metric calculation definition, a basic aggregation layer model is created to aggregate heterogeneous and diverse data from various systems to form a basic data center.
[0080] Step 4: Metric Modeling: Metrics are a recombination calculation of measures / atomic metrics, which usually includes key attributes such as metric definition, business rules, calculation scope, and business constraints.
[0081] Step 5: Define indicator relationships: Define the relationships between different types of indicators in the indicator system, such as anomaly drill-down relationships, similar indicator relationships, etc.
[0082] Step Six: Generate Cube Aggregation Model: Based on the dimensional and indicator systems, create Cube models according to scenarios to enable multi-dimensional use of Online Analytical Processing (OLAP).
[0083] Based on the above steps, step seven is also included: convergence computation operator logic.
[0084] Using the data generated from steps one through seven above Figure 2 Here is an example of an application of the bloodline resolver in the database:
[0085] First, create a data source (such as a Hadoop data source): You can define a business logic model based on business rules, and then define an entity physical model based on the business logic model, including column information and storage information, as well as the aging cycle of the physical model.
[0086] Secondly, data processing: Through visual process configuration, batch data streams are created. Using a visual drag-and-drop interface, operators used in the data stream processing, such as extraction, union, search, routing, grouping, and transformation, are orchestrated. Basic information, including the data source connection, custom SQL, associated physical model, and corresponding mapping relationships, is configured on the basic configuration page for each operator.
[0087] Secondly, based on the large model, knowledge extraction and identification are performed on the text related to the physical model, and model components are extracted to create a model mapping. The resulting physical model is then presented in the form of a graph. Specifically, this includes:
[0088] 1) Text preprocessing, including: word segmentation: splitting the text into words or phrases; stop word removal: removing common words that are not helpful for analysis; and part-of-speech tagging: tagging the part of speech of each word (such as noun, verb, etc.).
[0089] 2) Knowledge extraction, including: Named Entity Recognition (NER): identifying key entities in the text (such as model names, parameters, variables, etc.); Relationship extraction: identifying relationships between entities (such as dependencies between model parameters); and Topic modeling: extracting the main topics or concepts from the text.
[0090] 3) Information extraction, including: Keyword extraction: Identify and extract keywords or phrases that are crucial to understanding the model; Model component identification: Identify the main components of the model and their attributes (such as inputs, outputs, and parameters).
[0091] 4) Structured data generation, including: building knowledge graphs: organizing the extracted information into a graph format to display model components and relationships; generating summaries: automatically generating a brief description and key features of the model.
[0092] 5) Model applications: including: leveraging large models (such as GPT-4): utilizing their natural language understanding and generation capabilities to improve the accuracy and comprehensiveness of knowledge extraction; training and optimization: fine-tuning large models using domain-specific datasets to improve their ability to understand physical model text.
[0093] The above example, through platform tools, establishes data lineage by configuring data sources, creating physical models, orchestrating processes, and mapping relationships. This facilitates subsequent storage in the metadata target database and then displays the data on a visual interface via an interface. In other words, by systematically modeling O-domain data in the telecommunications field and describing complete data object relationships through metadata, users only need to describe the business model during big data application development, without needing to concern themselves with lineage definitions. During system operation, the lineage component automatically resolves lineage relationships, improving the accuracy and efficiency of lineage resolution.
[0094] As an optional approach, step 102 includes:
[0095] Based on the business application installation package, execute the business application installation process; Figure 2 For example, this step can be implemented by application installation management (model);
[0096] During the installation of the business application, the lineage installation package is installed and deployed to the metadata management module. The metadata management module performs deployment representation of the lineage installation package and dynamically registers the metadata lineage to the metadata management module.
[0097] In other words, during runtime, the lineage relationship installation packages in the business application installation package can be installed and deployed to the metadata management module (or metadata management service) through the business application installation and deployment process, so as to realize the dynamic registration of lineage relationships.
[0098] Furthermore, as an optional implementation, the method also includes at least one of the following:
[0099] In the event of an update to the business application, the metadata lineage in the metadata management module is updated dynamically.
[0100] When the business application is uninstalled, the metadata lineage in the metadata management module is automatically deleted.
[0101] In other words, the lineage relationships dynamically registered in the metadata management module are related to the running business applications. When the business applications are updated, the lineage relationships dynamically registered in the metadata management module are also updated accordingly. When the business applications are uninstalled, the lineage relationships dynamically registered in the metadata management module are also deleted accordingly. In this way, dynamic management of lineage relationships is achieved. This allows data lineage information to be updated in real time or near real time to reflect the latest state of current data flow, thereby meeting rapidly changing business needs.
[0102] As an optional implementation, step 103 involves synchronously registering the application interface lineage information to the metadata management module when the data application interface goes online, including at least one of the following:
[0103] After a user completes the development of a data application interface in the data application interface open module and uploads the data application interface to the application interface gateway, the data application interface lineage information is synchronously registered to the metadata management module. In this way, the user can view the application interface lineage data on the physical model page of the metadata management module (or metadata management tool).
[0104] When a user explores data on the target interface, they are redirected to the data application programming interface (API) open module using the developed Structured Query Language (SQL) to create a data API. Once the created API is uploaded to the API management platform, its lineage information is synchronously registered with the metadata management module. In this way, the user can view the API lineage data on the physical model page of the metadata management module. For example, the target interface is the Yacht interface, and the API management platform is, for example, the API Fabric.
[0105] Furthermore, as an optional implementation, the method also includes:
[0106] When the data application programming interface (API) is taken offline, the metadata management module is notified to deregister the API lineage information. That is, when a user takes the API offline on the API opening module's page, the metadata management module is simultaneously notified to deregister the API information (such as API lineage data).
[0107] Additionally, it should be noted that in the physical model lineage diagram of the metadata management module, if there are related data application interfaces, the page will display the relevant data application interface lineage diagram.
[0108] The method described in the above embodiments of this application establishes an end-to-end lineage chain from the data model to the data application programming interface (API), enabling automatic derivation of the API lineage. Specifically, this method enhances API lineage analysis by integrating the API lineage of the dataset into the end-to-end lineage characteristics to support comprehensive data processing. The implementation includes: reporting the mapping relationship between the API and the model to the metadata management module when the data API is launched; providing an interface to support querying the data API lineage; and displaying the API lineage information on the physical model page of the metadata management module.
[0109] In this embodiment, the data orchestration service is integrated as a module into the metadata management tool. The table storing lineage data in data orchestration and the table storing lineage data in metadata management are not merged. That is, the lineage information generated by data orchestration (batch processing) and the lineage information generated by multidimensional aggregation are stored in two separate tables within the metadata management tool. The original lineage query interface provided by DL would return the union of the data from the two tables to the caller. In this specification, lineage data needs to be merged as the basis for lineage injection and reporting. This achieves several advantages: First, it enables forward parsing processing, building a data model based on metadata, and automatically deriving lineage relationships, reducing the workload of manual adaptation and parsing, and improving the accuracy and efficiency of parsing. Second, through enhanced lineage analysis capabilities, users can more clearly understand the history and trajectory of data, improving the efficiency and quality of data management and utilization. Third, through systematic data modeling and automated lineage relationship derivation, this technology can significantly improve the efficiency and accuracy of data management, reduce operating costs, and enhance the ability to drive data-driven decision-making. Fourth, the methods described above also have the capability to integrate with big data platforms such as Hadoop, providing support for data link analysis, quality management, and operation and maintenance, thereby enhancing the platform's data processing and analysis capabilities. As data security and compliance become increasingly important, this method helps enterprises better track and manage data flows, ensuring compliance and reducing the risk of violations.
[0110] Additionally, it should be noted that in the long-term planning of data lineage, the generation, storage, querying, and application of lineage are all included. Figure 2 This is reflected in, specifically as follows:
[0111] Bloodline collection and registration (corresponding to) Figure 2 ③): Provides a graphical bloodline injection capability to supplement missing bloodline information;
[0112] Bloodline formation (corresponding to) Figure 2②): The lineage generated by the DataMold pattern is written into the metadata package by Dolas and then installed into the lineage module through the application; among them, DataMold is a tool for user data modeling and data management.
[0113] Lineage tracing: DAMS / SEQ queries the entire lineage through its query interface;
[0114] Bloodline registration (corresponding to) Figure 2 (④): Provide a lineage registration interface, cancel the reverse parsing capability in the DataPipeline scenario, and the lineage generated in the DataPipeline mode is reported by the engine;
[0115] Quality Application: Call the data quality interface to query quality information and display it in the lineage view;
[0116] Security Applications: Provides a model lineage query interface to assist in data installation for sensitive attribute identification.
[0117] Embodiments of this application also provide a data lineage management device, such as... Figure 4 As shown, it includes:
[0118] The generation module 401 is used to generate a business application installation package based on metadata. The business application installation package includes a lineage installation package.
[0119] The first registration module 402 is used to dynamically register metadata lineage relationships in the metadata management module by deploying the lineage relationship installation package in the metadata management module.
[0120] The second registration module 403 is used to synchronously register the data application interface lineage information to the metadata management module when the data application interface is launched.
[0121] Processing module 404 is used to distribute the lineage installation package to the data management-related subsystem, and register the engine lineage generated by the subsystem during task execution to the metadata management module, wherein the engine lineage is related to the lineage installation package;
[0122] The control module 405 is used to control the metadata management module to perform full-link lineage parsing on the target data based on the interface definition of the data application interface, the metadata lineage, the data application interface lineage information, and the engine lineage when the user accesses the target data through the data application interface, and to display the parsing results.
[0123] The generation module 401 includes:
[0124] The Acquisition submodule is used to obtain the attribute and relationship descriptions of the data model through metadata definitions during the process of modeling user data and orchestrating data processing logic.
[0125] Define submodules to define data processing logic using a domain-specific language (DSL).
[0126] The parsing submodule is used to perform field-level lineage parsing based on the description of attributes and relationships in the data model and the definition of data processing logic to obtain lineage data;
[0127] The compilation submodule is used to compile the bloodline data to obtain the bloodline installation package, and to package the bloodline installation package into the business application installation package.
[0128] The first registration module 402 includes:
[0129] The execution submodule is used to execute the business application installation process based on the business application installation package.
[0130] The deployment submodule is used to install and deploy the lineage installation package to the metadata management module during the installation process of the business application. The metadata management module performs deployment representation of the lineage installation package and dynamically registers the metadata lineage to the metadata management module.
[0131] The device further includes at least one of the following:
[0132] The update module is used to update the metadata lineage in the metadata management module dynamically when the business application is updated;
[0133] The deletion module is used to automatically delete the metadata lineage in the metadata management module when the business application is uninstalled.
[0134] The second registration module 403 includes at least one of the following:
[0135] The first registration submodule is used to synchronously register the data application interface lineage information to the metadata management module when the user has completed the development of the data application interface in the data application interface open module and put the data application interface online to the application interface gateway.
[0136] The second registration submodule is used to synchronously register the data application interface lineage information to the metadata management module when the user explores data on the target interface, jumps to the data application interface opening module based on the developed structured query language SQL to create the data application interface, and uploads the created data application interface to the application interface management platform.
[0137] The device further includes:
[0138] The unregistration module is used to notify the metadata management module to unregister the lineage information of the data application interface when the data application interface is taken offline.
[0139] It should be noted that the data lineage management device provided in this application embodiment can implement all the method steps implemented in the above data lineage management method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.
[0140] An embodiment of this application also provides a data lineage management device, including a transceiver 510, a processor 500, a memory 520, and a program stored in the memory 520 and executable on the processor 500; wherein, when the processor 500 executes the program, it implements the data lineage management method as described above.
[0141] The transceiver 510 is used to receive and send data under the control of the processor 500.
[0142] Among them, Figure 5 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 can be multiple elements, including a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium.
[0143] The processor 500 is responsible for managing the bus architecture and general processing, while the memory 520 can store the data used by the processor 500 when performing operations.
[0144] This application also provides a readable storage medium storing a program. When executed by a processor, this program implements the data lineage management method described above and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions for executing the methods described in the various embodiments of this application.
[0146] Therefore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the data lineage management method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.
[0147] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.
[0148] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.
[0149] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.
[0150] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values includes the upper and lower limits of the range and any subranges in between.
[0151] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data lineage management method, characterized by, include: The business application installation package is generated based on metadata, and the business application installation package includes lineage installation packages. By deploying the lineage installation package in the metadata management module, the metadata lineage is dynamically registered in the metadata management module; When the data application programming interface (API) is launched, the API lineage information is synchronously registered to the metadata management module. The lineage installation package is distributed to the data management-related subsystem, and the engine lineage generated by the subsystem during task execution is registered to the metadata management module. The engine lineage is related to the lineage installation package. When a user accesses target data through a data application programming interface (API), the metadata management module is controlled to perform end-to-end lineage analysis on the target data based on the API definition, metadata lineage, API lineage information, and engine lineage, and then displays the analysis results. The process of generating a business application installation package based on metadata includes: In the process of modeling user data and orchestrating data processing logic, the attributes and relationships of the data model are described through metadata definitions; Define the data processing logic using a domain-specific language. Based on the description of attributes and relationships in the data model and the definition of data processing logic, field-level lineage parsing is performed to obtain lineage relationship data; The bloodline data is compiled to obtain a bloodline installation package, and the bloodline installation package is packaged into the business application installation package.
2. The method of claim 1, wherein, By deploying the lineage installation package in the metadata management module, and dynamically registering metadata lineage relationships in the metadata management module, including: Based on the business application installation package, execute the business application installation process; During the installation of the business application, the lineage installation package is installed and deployed to the metadata management module. The metadata management module performs deployment representation of the lineage installation package and dynamically registers the metadata lineage to the metadata management module.
3. The method according to claim 1 or 2, characterized in that, The method further includes at least one of the following: In the event of an update to the business application, the metadata lineage in the metadata management module is updated dynamically. When the business application is uninstalled, the metadata lineage in the metadata management module is automatically deleted.
4. The method of claim 1, wherein, When a data application programming interface (API) is launched, the API lineage information is synchronously registered with the metadata management module, including at least one of the following: After the user completes the development of the data application interface in the data application interface open module and puts the data application interface online to the application interface gateway, the data application interface lineage information is synchronously registered to the metadata management module. When a user explores data on the target interface, the system redirects to the data application programming interface (API) open module based on the developed structured query language to create the API. The created API is then uploaded to the API management platform, and the API lineage information is synchronously registered to the metadata management module.
5. The method according to claim 1 or 4, characterized in that, The method further includes: When the data application interface is taken offline, the metadata management module is notified to unregister the lineage information of the data application interface.
6. A data lineage management apparatus characterized by comprising: include: The generation module is used to generate business application installation packages based on metadata. The business application installation packages include lineage-related installation packages. The first registration module is used to dynamically register metadata lineage relationships in the metadata management module by deploying the lineage relationship installation package in the metadata management module; The second registration module is used to synchronously register the data application interface lineage information to the metadata management module when the data application interface is launched. The processing module is used to distribute the lineage installation package to the data management-related subsystem and register the engine lineage generated by the subsystem during task execution to the metadata management module. The engine lineage is related to the lineage installation package. The control module is used to control the metadata management module to perform end-to-end lineage parsing on the target data based on the interface definition of the data application interface, the metadata lineage, the data application interface lineage information, and the engine lineage when the user accesses the target data through the data application interface, and to display the parsing results; The generation module includes: The Acquisition submodule is used to obtain the attribute and relationship descriptions of the data model through metadata definitions during the process of modeling user data and orchestrating data processing logic. Define submodules to define data processing logic using a domain-specific language (DSL). The parsing submodule is used to perform field-level lineage parsing based on the description of attributes and relationships in the data model and the definition of data processing logic to obtain lineage data; The compilation submodule is used to compile the bloodline data to obtain the bloodline installation package, and to package the bloodline installation package into the business application installation package.
7. A data lineage management device comprising a transceiver, a processor, a memory, and a program stored on the memory and executable on the processor; wherein, The transceiver is used to send and receive data under the control of the processor, and when the processor executes the program, it implements the data lineage management method as described in any one of claims 1 to 5.
8. A readable storage medium, on which a program or instructions are stored, characterized in that, When the program or instructions are executed by the processor, they implement the data lineage management method as described in any one of claims 1 to 5.
9. A computer program product, characterised in that, It includes computer instructions that, when executed by a processor, implement the data lineage management method as described in any one of claims 1 to 5.