Data lifecycle management method and system based on data lineage graph

Abstract

The present invention provides a data lifecycle management method and system based on a data lineage graph, including: Step 1: Parse data processing records to obtain data upstream and downstream relationships; Step 2: Construct a data lineage graph based on data upstream and downstream relationships; Step 3: Establish a data access permission information record table, including user, data, permission information, permission status and validity period fields; Step 4: Add new data usage permissions, automatically record relevant information in the data access permission information record table, and mark the permission status as effective. The present invention stores the relationships parsed by the script and the entities after knowledge fusion in a graph database through a knowledge storage module, displays and stores data lineage in a graph, and provides all derived evolution subsets of a certain directly authorized data when permissions are subsequently recovered, so as to facilitate accurate management and control.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data lifecycle management method and system based on a data lineage graph. Background Art

[0002] In the data analysis environment, for sensitive data, such as credit data, data analysts need to apply for permissions to carry out analysis work. According to the requirements of data lifecycle management, relevant data permissions must be reclaimed after the work is completed. The current data permission recovery is mainly for directly authorized data, but the authorized data will generate a lot of derived data after passing through complex data pipelines.

[0003] Patent document CN112434071A (application number: CN202011477301.5) discloses a metadata lineage relationship and impact analysis platform based on data atlas, including: obtaining the data to be analyzed and identifying the metadata in the data to be analyzed; identifying the category attributes of the metadata and classifying the metadata based on the category attributes, and at the same time, performing lineage analysis and impact analysis on the metadata based on the category attributes, and constructing the upper and lower logical relationship of the metadata; displaying the upper and lower logical relationship of the metadata in the form of a data atlas. However, this patent cannot avoid the need to perform intermediate data conversion every time data is processed and lineage tracing is performed. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a data lifecycle management method and system based on a data lineage graph.

[0005] The data lifecycle management method based on the data lineage graph provided by the present invention includes:

[0006] Step 1: Analyze data processing records and obtain upstream and downstream relationships of data;

[0007] Step 2: Build a data lineage graph based on the upstream and downstream relationships of the data;

[0008] Step 3: Create a data access permission information record table, including user, data, permission information, permission status and validity period fields;

[0009] Step 4: Add data usage permissions, automatically record the relevant information in the data access permission information record table, and mark the permission status as effective.

[0010] Preferably, step 4 includes: the data administrator directly grants the authority, or the user processes and backs up the authorized data to obtain the use rights of the derived data, which are granted to the user by default.

[0011] Preferably, the method further includes step 5: reclaiming the data usage rights, which may be directly reclaimed by the data administrator or automatically reclaimed after the data rights expire;

[0012] First, the data usage rights are directly revoked, and then the derived data information of the data is obtained through the data lineage graph. Based on the derived data information, the user's rights to the data are revoked, and the permission status in the relevant records of the data access permission information record table is marked as invalid.

[0013] Preferably, the method further includes step 6: re-granting permissions to the data whose permissions have been reclaimed, first directly granting the permission to use the data, then obtaining the derived data information of the data through the data lineage graph, obtaining the user's permission recovery record for the derived data from the data access permission information record table, and re-granting the user permission to the data based on the permission recovery record, and marking the permission status in the relevant records of the data access permission information record table as effective.

[0014] The data lifecycle management system based on the data lineage graph provided by the present invention includes:

[0015] Data processing analysis module: data processing process recording and storage, script analysis, data upstream and downstream relationship extraction and relationship information storage;

[0016] Data lineage graph construction module: ontology construction, knowledge fusion, knowledge storage and knowledge operation and maintenance;

[0017] Data lineage retrieval module: obtains the derived data information of given data;

[0018] Data permission management module: adding data permissions, recording permission information, reclaiming permissions, and re-empowering permissions.

[0019] Preferably, the data processing process recording and storage includes: recording the commands, script contents and operation log information used in data processing;

[0020] The script parsing includes: identifying the start and end points of data processing and processing rules and the transferred fields through a preset rule model, renaming statements and field matching statements in the parsing statements, and automatically identifying entities with the same name and data flow.

[0021] Preferably, the ontology construction includes: combing the empowered data and derived data to construct the ontology structure required for the data lineage map;

[0022] The knowledge fusion includes: using the same-name entity information obtained by script analysis and the manually summarized prior knowledge to align the multi-source heterogeneous data entities, then entity linking the entities with the ontology structure, cleaning the data, removing expired and invalid noise, and finally expressing the data processing and derivation process in the form of entity-relationship-entity and entity-attribute-attribute value;

[0023] The knowledge storage includes: importing the fused knowledge into a graph database and setting a unique identifier for the entity;

[0024] The knowledge operation and maintenance includes: finding isolated nodes in the graph and completing missing information.

[0025] Preferably, the data lineage retrieval module includes: for given data, based on the constructed data lineage graph, starting from the node where the data is located, obtaining information of all downstream nodes along the data flow, and traversal ends if a closed loop is encountered or there are no more downstream nodes.

[0026] Preferably, the data authority addition includes: direct authorization of data by a data administrator or the user directly obtaining the authority of derived data of the authorized data;

[0027] The permission information record includes: recording the scope of the user's permission to use the data, including read-only and full control options, and the permission status includes effective and invalid options.

[0028] Preferably, the permission recovery includes: first directly recovering the use permission of the part of data, then obtaining the derived data information of the data through the constructed data lineage graph, recovering the user's permission to the part of data based on the derived data information, and updating the data permission status in the data access permission information record table;

[0029] The re-empowerment includes: first directly granting the permission to use the part of data, then obtaining the derived data information of the data through the constructed data lineage graph, obtaining the user's permission recovery record for the above-mentioned derived data from the data access permission information record table, and based on the permission recovery record, re-granting the user the permission to this part of data, and updating the data permission status in the data access permission information record table.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) The script parsing module automatically extracts processing rules from the data processing code, summarizes the upstream and downstream of the data, obtains the data derivation path and provides it to the data lineage graph for integration; the knowledge fusion module combines the results of script parsing and the manually summarized rules, aggregates multi-source heterogeneous data, and integrates concepts belonging to the same entity, avoiding the need for intermediate data conversion each time data is processed and lineage traced;

[0032] (2) The relationships parsed by the script and the entities after knowledge fusion are stored in the graph database through the knowledge storage module, and the data lineage is displayed and stored in the form of a graph. When the authority is subsequently recovered, all derived evolutionary subsets of a certain directly authorized data are provided to facilitate accurate management and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0036] Example:

[0037] like Figure 1 The present invention proposes a method for data lifecycle management based on data lineage graph, which is used to manage the rights of empowered data and its derived data, including the following steps:

[0038] 1. Obtain the upstream and downstream relationships of data by parsing data processing records;

[0039] 2. Build a data lineage graph through the upstream and downstream relationships of the acquired data;

[0040] 3. Establish a data access permission information record table, which mainly includes fields such as user, data, permission information, permission status, and validity period. The permission information records the scope of the user's permission to use the data in the record, including options such as read-only and full control. The permission status records whether the permission information in the record is effective, including effective and invalid options.

[0041] 4. Adding data usage permissions: For a specific user, adding permissions to specific data can be obtained in the following two ways: (1) Direct authorization by the data administrator; (2) The user is granted the permission to use data derived from operations such as processing and backup of authorized data by the user by default. When adding data usage permissions, the relevant information is automatically recorded in the data access permission information record table, and the permission status is marked as effective.

[0042] 5. Reclaiming data usage rights: The user's data usage rights can be reclaimed in the following two ways: (1) directly by the data administrator, and (2) automatically after the data rights expire. When reclaiming the rights of a specific user to specific data, the specific process is to first directly revoke the usage rights of the data, then obtain the derived data information of the data through the constructed data lineage graph, and then revoke the user's rights to the data based on the derived data information. After the above rights are reclaimed, the rights status in the relevant records of the data access rights information record table is marked as invalid.

[0043] 6. Re-granting permissions to reclaimed data: When re-granting permissions to specific data to a specific user, the specific process is to first directly grant the permission to use the data, then obtain the derived data information of the data through the constructed data lineage graph, obtain the permission recovery record of the user for the above derived data from the data access permission information record table, and re-grant the user the permission for this part of the data based on the permission recovery record. After the above permissions are re-granted, the permission status in the relevant records of the data access permission information record table is marked as effective.

[0044] The present invention also provides a data lifecycle management system based on a data lineage graph, including: a data processing process analysis module, a data lineage graph construction module, a data lineage retrieval module, and a data authority management module.

[0045] The data processing process analysis module includes processes such as data processing process recording and storage, script analysis, data upstream and downstream relationship extraction, and relationship information storage;

[0046] The data lineage graph construction module includes processes such as ontology construction, knowledge fusion, knowledge storage and knowledge operation and maintenance;

[0047] The data lineage retrieval module is mainly used to obtain the derived data information of given data;

[0048] The data authority management module includes functions such as adding data authority, recording authority information, reclaiming authority, and re-assigning authority.

[0049] The data processing process recording and storage is to record the commands, script contents, operation logs and other information used in data processing.

[0050] The processing process analysis analyzes the processing process information, generally database script statements or operation logs, and identifies the start and end points of data processing, processing rules, and transferred fields through the rule model. Rename statements and field matching statements in the parsing statements automatically identify entities with the same name and data flow.

[0051] The ontology construction refers to sorting out the empowered data and derived data, summarizing the ontologies such as tables, source systems, fields in the data, the attributes of the ontologies, the upstream and downstream, inclusion, derivation and other relationships between ontologies and relationship attributes, and constructing the ontology structure required for the data lineage map.

[0052] The knowledge fusion refers to the use of the same-name entity information obtained by script analysis and the manually summarized prior knowledge to align the multi-source heterogeneous data entities, and then link the entities with the ontology structure. In addition, the data needs to be cleaned to remove expired and invalid noise. Finally, the data processing and derivation process is expressed in the form of entity-relationship-entity and entity-attribute-attribute value.

[0053] The knowledge storage is the process of importing the fused knowledge into the graph database. The unique identifier of the entity is set to ensure that different entities have different identifiers. The graph structure is set according to the settings of the ontology building module, and the fused knowledge is stored in the database according to the graph structure.

[0054] The knowledge operation and maintenance mentioned above refers to automatically finding isolated abnormal nodes in the graph and manually completing the missing information. New relationships and entities are updated in a timely manner and outdated information is corrected.

[0055] The data lineage retrieval module is mainly used to obtain the derived data information of given data. For given data, the data lineage graph constructed based on the data lineage graph construction module takes the node where the data is located as the starting point, and obtains the information of all downstream nodes along the data flow. If a closed loop is encountered (i.e., it eventually flows back to the origin) or there are no more downstreams, the traversal ends.

[0056] The new permission distribution refers to the direct authorization of data by data administrators or the direct authorization of users to obtain derived data of the data they have permission to.

[0057] The data access permission information record table mainly includes fields such as user, data, permission information, permission status, and validity period. The permission information records the scope of use permissions of the user shown in the record for the data shown, including options such as read-only and full control. The permission status records whether the permission information shown in the record is effective, including effective and invalid options.

[0058] The permission recovery is for a specific user. When recovering the permission of a specific user to a specific data, the permission to use the data is first directly recovered, and then the derived data information of the data is obtained through the data lineage graph constructed by the data lineage graph construction module. Based on the derived data information obtained, the user's permission to this part of the data is recovered. The permission recovery operation will update the data permission status in the data access permission information record table.

[0059] The re-authorization is to re-grant the permission to the specific data to the specific user after the permission to the specific data has been revoked. When re-authorizing the permission to the specific data to the specific user, the specific process is to first directly grant the permission to use the part of the data, then obtain the derived data information of the data through the constructed data lineage graph, obtain the permission recovery record of the user to the above-mentioned derived data from the data access permission information record table, and re-authorize the user to the part of the data based on the above-mentioned permission recovery record. The re-authorization operation will update the data permission status in the data access permission information record table.

[0060] As highly sensitive data, credit data needs to be strictly controlled in terms of data permissions based on the “minimum necessary” principle. When using credit data for data analysis, it is often necessary to apply for usage permissions for a certain period of time. However, it is difficult for data administrators to track the derived data retained by data users after processing and saving the above data. When permissions are reclaimed, only directly authorized data can be reclaimed. Therefore, through the method for data lifecycle management based on a data lineage map described in the present invention, data usage permissions can be completely controlled to effectively reduce the risk of data leakage.

[0061] In sensitive data management, it is relatively easy to control directly authorized data. However, when directly authorized data undergoes complex processing to generate a multi-layer derived data network, it is more difficult to comprehensively and without omission control all derived data.

[0062] The data lineage map can accurately trace the flow and derivative evolution of data, making it possible to solve data positioning and accurate permission recovery, and providing a powerful tool for sensitive data management and data life cycle control.

[0063] When the data processing link is long enough, using traditional relational databases to record data requires a table for each level of processing, and joint searches require connecting the tables in series, which makes data tracing difficult and inefficient, and it is difficult to avoid omissions. The data lineage graph construction module stores the processing link directly in a graph format, which can be directly searched in depth and breadth, and can be fully tracked regardless of deep processing or multiple derivations at the same level.

[0064] The present invention combines data lineage and data authority management related technologies to achieve comprehensive control of data and its derived data permissions.

[0065] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0066] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data lifecycle management method based on data lineage graph, characterized in that: include: Step 1: Analyze data processing records and obtain upstream and downstream relationships of data; Step 2: Build a data lineage graph based on the upstream and downstream relationships of the data; Step 3: Create a data access permission information record table, including user, data, permission information, permission status and validity period fields; Step 4: Add data usage permissions, automatically record relevant information in the data access permission information record table, and mark the permission status as effective; The step 4 includes: the data administrator directly grants the right, or the user processes and backs up the authorized data to obtain the right to use the derived data, and grants the user the right to use the derived data by default; It also includes step 5: reclaiming the data usage rights, which is directly reclaimed by the data administrator or automatically reclaimed after the data rights expire; First, directly revoke the data usage rights, then obtain the derived data information of the data through the data lineage graph, and based on the derived data information, revoke the user's rights to the derived data, and mark the rights status in the relevant records of the data access rights information record table as invalid; It also includes step 6: re-granting permissions to the data whose permissions have been reclaimed, first directly granting the permission to use the data, then obtaining the derived data information of the data through the data lineage graph, obtaining the user's permission recovery record for the derived data from the data access permission information record table, and based on the permission recovery record, re-granting the user permission for the derived data, and marking the permission status in the relevant records of the data access permission information record table as effective.

2. A data lifecycle management system based on a data lineage graph, characterized in that: include: Data processing analysis module: data processing process recording and storage, script analysis, data upstream and downstream relationship extraction and relationship information storage; Data lineage graph construction module: ontology construction, knowledge fusion, knowledge storage and knowledge operation and maintenance; Data lineage retrieval module: obtains the derived data information of given data; Data permission management module: adding data permissions, recording permission information, reclaiming permissions, and re-assigning permissions; The data processing process recording and storage includes: recording the commands, script content and operation log information used in data processing; The script parsing includes: identifying the start and end points of data processing and processing rules and the transferred fields through a preset rule model, parsing the renaming statements and field matching statements in the statements, and automatically identifying entities with the same name and data flow direction; The ontology construction includes: sorting out the empowered data and derived data to construct the ontology structure required for the data lineage map; The knowledge fusion includes: using the same-name entity information obtained by script analysis and the manually summarized prior knowledge to align the multi-source heterogeneous data entities, then entity linking the entities with the ontology structure, cleaning the data, removing expired and invalid noise, and finally expressing the data processing and derivation process in the form of entity-relationship-entity and entity-attribute-attribute value; The knowledge storage includes: importing the fused knowledge into a graph database and setting a unique identifier for the entity; The knowledge operation and maintenance includes: finding isolated nodes in the graph and completing the missing information; The data lineage retrieval module includes: for given data, based on the constructed data lineage graph, starting from the node where the data is located, obtaining the information of all downstream nodes along the data flow, and traversing ends if a closed loop is encountered or there are no more downstream nodes; The new additions to data permissions include: direct authorization of data by data administrators or the permission of users to directly obtain derived data of their authorized data; The permission information record includes: recording the scope of the user's permission to use the data, including read-only and full control options, and the permission status including effective and invalid options; The permission recovery includes: first directly recovering the use permission of the data, then obtaining the derived data information of the data through the constructed data lineage graph, recovering the user's permission to the derived data based on the derived data information, and updating the data permission status in the data access permission information record table; The re-empowerment includes: first directly granting the right to use the data, then obtaining the derived data information of the data through the constructed data lineage graph, obtaining the user's permission recovery record for the derived data from the data access permission information record table, and based on the permission recovery record, re-granting the user the permission for the derived data, and updating the data permission status in the data access permission information record table.

Images