Metadata-based Data Lineage Analysis Method and System

Abstract

This invention discloses a data lineage analysis method and system based on metadata. The invention establishes a metadata registration table for each database to record metadata information of each data table and field value, including data source information and data retrieval logic. In real-world business activities, this effectively analyzes and displays data lineage and enables unified data management. Then, in response to a client-initiated execution request for a target task, the system executes the target task based on data in the target database. When anomalies occur in the results after executing the target task, the system searches the metadata registration table of the target database for the source information of the data corresponding to the target task, generates a data lineage diagram based on the source information, and displays it, thus providing a visually intuitive understanding of the data's origin.

Description

Technical Field

[0001] This invention relates to the field of data traceability technology for nuclear power data, and in particular to a data lineage analysis method and system based on metadata. Background Technology

[0002] In the field of nuclear power technology, due to differences in data management and utilization methods among various departments, the data storage and distribution mechanisms also differ. In data synchronization scenarios, due to the high complexity of system interaction, data quality issues are often difficult to locate, some of which are management issues and others are technical issues.

[0003] In traditional methods, data tracing is usually done based on the source address of the data. However, for data with data dependencies, such as retrieving target data from a database based on address information, where part of the target data is obtained from other databases, usually only the first database is found, while the other database is ignored. This results in a flaw in the tracing process, leading to inaccurate tracing. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data lineage analysis method and system based on metadata, addressing at least one deficiency in the existing technology.

[0005] The technical solution adopted by this invention to solve its technical problem is: to construct a data lineage analysis method based on metadata and apply it to a server, including the following steps:

[0006] S1: Establish a metadata registry table for each database; the metadata registry table records the metadata information of each data table and field value in the database, and the metadata information includes the data source information and data retrieval logic;

[0007] S2: In response to the client's request to execute the target task, execute the target task based on the data in the target database;

[0008] S3: When the result information obtained after executing the target task is abnormal, search the metadata registration table of the target database for the source information of the data corresponding to the target task;

[0009] S4: Generate a data lineage diagram based on the source information and display it.

[0010] Preferably, in the metadata-based data lineage analysis method of the present invention, the method further includes:

[0011] S5: Determine the source of each data point based on the data lineage diagram to trace the data back to its source.

[0012] Preferably, in the metadata-based data lineage analysis method of the present invention, step S5 includes:

[0013] The data stored in the database is compared with the actual data. If they are inconsistent, the source of each data is determined according to the data lineage diagram, and the data in the database is updated.

[0014] Preferably, in the metadata-based data lineage analysis method of the present invention, the data stored in the database includes the original stored data and the associated data obtained from other databases.

[0015] Preferably, in the metadata-based data lineage analysis method of the present invention, obtaining data from other databases includes:

[0016] In response to a data synchronization request, the data retrieval logic is parsed from the data synchronization request;

[0017] Based on the data retrieval logic, data is synchronized from other databases to the target database, and the source information and data retrieval logic of the data are added to the metadata registration table of the target database to form a metadata registration table containing the data content.

[0018] Preferably, in the metadata-based data lineage analysis method of the present invention, step S3, which involves searching for the source information of the data corresponding to the target task in the metadata registration table of the target database, includes:

[0019] S31: Search the metadata registration table of the target database for the source information of the original data corresponding to the target task and the source information of the associated data;

[0020] S32: Locate the first database storing the associated data based on the source information of the associated data;

[0021] S33: If the associated data includes first associated data and second associated data, and the second associated data does not belong to the data of the first database, then obtain the metadata registration table of the first database and search for the source information of the second associated data therein.

[0022] This invention also constructs a metadata-based data lineage analysis system, applied to a server, comprising:

[0023] A creation module is used to establish a metadata registry table for each database; the metadata registry table records metadata information of each data table and field value in the database, and the metadata information includes data source information and data retrieval logic;

[0024] The execution module is used to respond to the execution request for the target task initiated by the client and execute the target task based on the data in the target database;

[0025] The lookup module is used to look up the source information of the data corresponding to the target task in the metadata registration table of the target database when the result information obtained after executing the target task is abnormal.

[0026] The generation module is used to generate a data lineage diagram based on the source information and then display it.

[0027] Preferably, in the metadata-based data lineage analysis system of the present invention, the system further includes:

[0028] The determination module is used to determine the source of each data point based on the data lineage diagram in order to trace the source of the data.

[0029] Preferably, in the metadata-based data lineage analysis system of the present invention, the determining module is further used to compare the data stored in the database with the actual data. If they are inconsistent, the source of each data is determined according to the data lineage diagram, and the data in the database is updated.

[0030] Preferably, in the metadata-based data lineage analysis system of the present invention, the data stored in the database includes the original stored data and the associated data obtained from other databases.

[0031] Preferably, in the metadata-based data lineage analysis system of the present invention, the system further includes:

[0032] The parsing module is used to parse the data retrieval logic from the data synchronization request in response to the data synchronization request;

[0033] The synchronization module is used to synchronize data from other databases to the target database based on the data retrieval logic, and add the data source information and data retrieval logic to the metadata registration table of the target database to form a metadata registration table containing data content.

[0034] Preferably, in the metadata-based lineage analysis system of the present invention, the search module includes:

[0035] The target lookup module is used to search the metadata registration table of the target database for the source information of the original data corresponding to the target task and the source information of the associated data;

[0036] The first association lookup module is used to search for the first database storing the association data based on the source information of the association data.

[0037] The second association lookup module is used to obtain the metadata registration table of the first database and search for the source information of the second association data if the associated data includes the first associated data and the second associated data, and the second associated data does not belong to the data of the first database.

[0038] By implementing this invention, the following beneficial effects are achieved:

[0039] This invention establishes a metadata registry table for each database, recording metadata information for each data table and field value within the database. This metadata information includes data source information and data retrieval logic. In real-world business activities, this effectively analyzes and displays data lineage and enables unified data management. Then, in response to a client's request to execute a target task, the invention executes the target task based on data in the target database. When an anomaly occurs in the result information after executing the target task, the invention searches the target database's metadata registry table for the source information of the data corresponding to the target task, generates a data lineage diagram based on the source information, and displays it, thus providing a visually intuitive understanding of the data's origin. Attached Figure Description

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0041] Figure 1 This is a flowchart of the data lineage analysis method based on metadata of the present invention;

[0042] Figure 2 This is a block diagram of the data lineage analysis system based on metadata according to the present invention. Detailed Implementation

[0043] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0044] It should be noted that the flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0046] like Figure 1 As shown, this invention discloses a data lineage analysis method based on metadata, applied to a server, comprising the following steps:

[0047] Step S1: Create a metadata registry table for each database; the metadata registry table records the metadata information of each data table and field value in the database, including the data source information and data retrieval logic.

[0048] Metadata refers to attribute data describing data in a database, specifically including attribute data of raw data and related data. Source information refers to the source of the data, indicating which database it originates from. Data retrieval logic refers to the data retrieval logic code, which is used to retrieve business data from the corresponding database or to process the corresponding business data. The data retrieval logic code and its corresponding explanation are shown in Table 1 below.

[0049] Table 1

[0050]

[0051]

[0052] In this embodiment, the data stored in the database includes the original data stored in the database itself and related data obtained from other databases. For example, in addition to storing its own data, database c also stores some data from database b. Furthermore, database b also stores some data from database a.

[0053] Retrieving data from other databases, specifically including:

[0054] In response to a data synchronization request, the data retrieval logic is parsed from the data synchronization request;

[0055] Based on this data retrieval logic, data is synchronized from other databases to the target database, and the source information and data retrieval logic of the data are added to the metadata registration table of the target database to form a metadata registration table containing the data content.

[0056] Step S2: In response to the client's request to execute the target task, execute the target task based on the data in the target database.

[0057] In some embodiments, the target task can be to update and display materials in the material management system based on data in the target database. Specifically, if department A (such as the purchasing department) purchases materials required for construction and updates the material information and corresponding purchase quantity in the target database, and department B (the engineering department) submits a material requirement and updates the materials used and usage in other databases, and synchronizes the materials used and usage to the target database for storage, when a client initiates an execution request for the target task, the system retrieves the material information, corresponding purchase quantity, materials used, and usage from the target database, and updates the materials in the material management system based on the retrieved data.

[0058] Step S3: When the result information obtained after executing the target task is abnormal, determine the metadata registration table of the target database where the data is stored, and look up the source information of the data corresponding to the target task in the metadata registration table.

[0059] In some embodiments, abnormal results may occur when the results do not match reality, such as when the updated material inventory in the material management system does not match the actual inventory.

[0060] In this embodiment, step S3 involves searching the metadata registration table of the target database for the source information of the data corresponding to the target task, specifically including:

[0061] Step S31: Search the metadata registration table of the target database for the source information of the original data corresponding to the target task and the source information of the associated data;

[0062] Step S32: Locate the first database storing the associated data based on the source information of the associated data;

[0063] Step S33: If the associated data includes first associated data and second associated data, and the second associated data does not belong to the data of the first database, then obtain the metadata registration table of the first database, and search for the source information of the second associated data in the metadata registration table of the first database, thereby obtaining the source information of the original data and the associated data.

[0064] Step S4: Generate and display a data lineage diagram based on the source information to help identify the root cause of data quality issues.

[0065] Among them, the data lineage diagram is used to represent the source of each data and the relationship between the data. The data lineage analysis algorithm is used to represent the data transmission from the source system to the target system or the reverse operation to represent the source link of the data from the target system to the source system. It can also be expressed in a graphical way.

[0066] In some embodiments, a management page may be displayed in the material management system, on which a lineage diagram is displayed.

[0067] In this embodiment, the method further includes:

[0068] Step S5: Determine the source of each data point based on the data lineage diagram to trace the data's origin.

[0069] In some embodiments, after determining the source of each data, the accuracy of the data is checked in the database of the respective source, the data stored in the database is compared with the actual data, and if they are inconsistent, the source of each data is determined according to the data lineage diagram, and the data in the database is updated, for example, the latest related data from other databases can be updated to the target database.

[0070] like Figure 2 As shown, this invention also discloses a data lineage analysis system based on metadata, applied to a server, including: a creation module, an execution module, a search module, and a generation module. Specifically:

[0071] In this embodiment, the creation module is used to establish a metadata registration table for each database; the metadata registration table records the metadata information of each data table and field value in the database, including the data source information and data retrieval logic.

[0072] Metadata refers to attribute data describing data in a database, specifically including attribute data of raw data and related data. Source information refers to the source of the data, indicating which database it originates from. Data retrieval logic refers to the data retrieval logic code, which is used to retrieve business data from the corresponding database or to process the corresponding business data. The data retrieval logic code and its corresponding explanation are shown in Table 1 below.

[0073] Table 1

[0074]

[0075]

[0076] In this embodiment, the data stored in the database includes the original data stored in the database itself and related data obtained from other databases. For example, in addition to storing its own data, database c also stores some data from database b. Furthermore, database b also stores some data from database a.

[0077] The system also includes functions for retrieving data from other databases:

[0078] The parsing module is used to respond to data synchronization requests and parse the data retrieval logic from the data synchronization requests;

[0079] The synchronization module is used to synchronize data from other databases to the target database based on the data retrieval logic, and add the data source information and data retrieval logic to the metadata registration table of the target database to form a metadata registration table containing the data content.

[0080] In this embodiment, the execution module is used to respond to the execution request for the target task initiated by the client and execute the target task based on the data in the target database.

[0081] In some embodiments, the target task can be to update and display materials in the material management system based on data in the target database. Specifically, if department A (such as the purchasing department) purchases materials required for construction and updates the material information and corresponding purchase quantity in the target database, and department B (the engineering department) submits a material requirement and updates the materials used and usage in other databases, and synchronizes the materials used and usage to the target database for storage, when a client initiates an execution request for the target task, the system retrieves the material information, corresponding purchase quantity, materials used, and usage from the target database, and updates the materials in the material management system based on the retrieved data.

[0082] In this embodiment, the lookup module is used to determine the metadata registration table of the target database where the data is stored when the result information obtained after executing the target task is abnormal, and to look up the source information of the data corresponding to the target task in the metadata registration table.

[0083] In some embodiments, the lookup module includes:

[0084] The target lookup module is used to search the metadata registration table of the target database for the source information of the original data corresponding to the target task and the source information of the associated data when the result information obtained after executing the target task is abnormal.

[0085] The first association lookup module is used to search the first database storing association data based on the source information of the association data.

[0086] The second association lookup module is used to obtain the metadata registration table of the first database and search for the source information of the second association data in the metadata registration table of the first database if the associated data includes the first associated data and the second associated data, and the second associated data does not belong to the data of the first database.

[0087] In this embodiment, the generation module is used to generate a data lineage diagram based on the source information and display it in order to find the root cause of data quality problems.

[0088] Among them, the data lineage diagram is used to represent the source of each data and the relationship between the data. The data lineage analysis algorithm is used to represent the data transmission from the source system to the target system or the reverse operation to represent the source link of the data from the target system to the source system. It can also be expressed in a graphical way.

[0089] In some embodiments, a management page may be displayed in the material management system, on which a lineage diagram is displayed.

[0090] In this embodiment, the system further includes:

[0091] The determination module is used to identify the source of each data point based on the data lineage diagram, enabling data tracing. Specifically, after determining the source of each data point, the module checks the accuracy of the data in its respective source database, compares the data stored in the database with the actual data, and if discrepancies are found, determines the source of each data point based on the data lineage diagram and updates the data in the database accordingly. For example, the latest related data from other databases can be updated to the target database.

[0092] By implementing this invention, the following beneficial effects are achieved:

[0093] This invention establishes a metadata registry table for each database, recording metadata information for each data table and field value within the database. This metadata information includes data source information and data retrieval logic. In real-world business activities, this effectively analyzes and displays data lineage and enables unified data management. Then, in response to a client's request to execute a target task, the invention executes the target task based on data in the target database. When an anomaly occurs in the result information after executing the target task, the invention searches the target database's metadata registry table for the source information of the data corresponding to the target task, generates a data lineage diagram based on the source information, and displays it, thus providing a visually intuitive understanding of the data's origin.

[0094] It is understood that the above embodiments only illustrate preferred embodiments of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can freely combine the above technical features without departing from the concept of the present invention, and can also make several modifications and improvements, all of which fall within the protection scope of the present invention. Therefore, all equivalent transformations and modifications made with respect to the scope of the claims of the present invention should fall within the scope of the claims of the present invention.

Claims

1. A data lineage analysis method based on metadata, applied to a server, characterized in that, The following steps are involved: S1: Establish a metadata registry table for each database; the metadata registry table records the metadata information of each data table and field value in the database, and the metadata information includes the data source information and data retrieval logic; S2: In response to the client's request to execute the target task, execute the target task based on the data in the target database; S3: When the result information obtained after executing the target task is abnormal, search the metadata registration table of the target database for the source information of the data corresponding to the target task; S4: Generate a data lineage diagram based on the source information and display it; The data stored in the database includes the original data stored in the database and related data obtained from other databases; The process of obtaining data from other databases includes: In response to a data synchronization request, the data retrieval logic is parsed from the data synchronization request; Based on the data retrieval logic, data is synchronized from other databases to the target database, and the source information and data retrieval logic of the data are added to the metadata registration table of the target database to form a metadata registration table containing the data content.

2. The data lineage analysis method based on metadata according to claim 1, characterized in that, The method further includes: S5: Determine the source of each data point based on the data lineage diagram to trace the data back to its source.

3. The data lineage analysis method based on metadata according to claim 2, characterized in that, Step S5 includes: The data stored in the database is compared with the actual data. If they are inconsistent, the source of each data is determined according to the data lineage diagram, and the data in the database is updated.

4. The data lineage analysis method based on metadata according to claim 1, characterized in that, Step S3, which involves searching the metadata registration table of the target database for the source information of the data corresponding to the target task, includes: S31: Search the metadata registration table of the target database for the source information of the original data corresponding to the target task and the source information of the associated data; S32: Locate the first database storing the associated data based on the source information of the associated data; S33: If the associated data includes first associated data and second associated data, and the second associated data does not belong to the data of the first database, then obtain the metadata registration table of the first database and search for the source information of the second associated data therein.

5. A metadata-based data lineage analysis system, applied to a server, characterized in that, include: Create a module to establish a metadata registry table for each database; The metadata registration table records metadata information for each data table and field value in the database. The metadata information includes data source information and data retrieval logic. The execution module is used to respond to the execution request for the target task initiated by the client and execute the target task based on the data in the target database; The lookup module is used to look up the source information of the data corresponding to the target task in the metadata registration table of the target database when the result information obtained after executing the target task is abnormal. The generation module is used to generate a data lineage diagram based on the source information and then display it. The data stored in the database includes the original data stored in the database and related data obtained from other databases; The system also includes: The parsing module is used to parse the data retrieval logic from the data synchronization request in response to the data synchronization request; The synchronization module is used to synchronize data from other databases to the target database based on the data retrieval logic, and add the data source information and data retrieval logic to the metadata registration table of the target database to form a metadata registration table containing data content.

6. The metadata-based data lineage analysis system according to claim 5, characterized in that, The system also includes: The determination module is used to determine the source of each data point based on the data lineage diagram in order to trace the source of the data.

7. The metadata-based data lineage analysis system according to claim 6, characterized in that, The determining module is further used to compare the data stored in the database with the actual data. If they are inconsistent, the source of each data is determined according to the data lineage diagram, and the data in the database is updated.

8. The metadata-based data lineage analysis system according to claim 5, characterized in that, The search module includes: The target lookup module is used to search the metadata registration table of the target database for the source information of the original data corresponding to the target task and the source information of the associated data; The first association lookup module is used to search for the first database storing the association data based on the source information of the association data. The second association lookup module is used to obtain the metadata registration table of the first database and search for the source information of the second association data if the associated data includes the first associated data and the second associated data, and the second associated data does not belong to the data of the first database.

Images