Data kinship management method, system and storage medium

Abstract

An embodiment of the present invention discloses a data lineage relationship management method, system and storage medium, which can reduce the management complexity of data lineage relationships and improve query efficiency. The method may include: a data construction parser; a syntax tree construction parser connected to the data construction parser; a graph database module connected to the syntax tree construction parser; wherein the data construction parser is used to parse the data source change data sent from the business system to obtain data lineage relationship structure data; the syntax tree construction parser is used to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module; the graph database module is used to store and manage graph structure data corresponding to the multi-layer syntax tree instance, the graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and in particular to a data lineage relationship management method, system and storage medium. Background Art

[0002] With the rapid development of the internet industry, big data has become a new trend and the norm. Many large websites and enterprises generate vast quantities of data daily. In fields such as transportation, logistics, e-commerce, municipal administration, and industrial production, vast amounts of data are constantly accumulating. Analyzing and processing this vast amount of data and extracting useful information requires processes such as data collection, data cleaning, data preprocessing, data storage, data analysis, and data mining. Throughout this process, data flows across various storage media and is transferred or stored in various forms within these media. Recording, managing, and displaying the relationships within this data flow has become a crucial requirement. These relationships within this data flow represent the kinship of big data.

[0003] Existing technology stores and manages data relationships during data flow by creating a data warehouse and numerous data relationship tables within it. Data warehouses are built using traditional relational databases. As data volumes and data source types grow, more and more relationship tables must be created within the relational database to record data relationships. This leads to high complexity in managing data relationships and low query efficiency. Summary of the Invention

[0004] To solve the above technical problems, embodiments of the present invention hope to provide a data lineage management method, system and storage medium, which can reduce the management complexity of data lineage and improve query efficiency.

[0005] The technical solution of the present invention is achieved as follows:

[0006] In an embodiment of the present invention, a data management system is provided, the system comprising:

[0007] Data structure parser;

[0008] a syntax tree construction parser connected to the data construction parser;

[0009] A graph database module connected to the syntax tree construction parser; wherein,

[0010] The data structure parser is used to parse the data source change data sent from the business system to obtain data lineage relationship structure data;

[0011] The syntax tree construction parser is used to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module;

[0012] The graph database module is used to store and manage the graph structure data corresponding to the multi-layer syntax tree instance, and the graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

[0013] In the above system, the system further comprises: a processor; the processor is connected to the data construction parser, the syntax tree construction parser and the graph database module respectively;

[0014] The processor is configured to hierarchically traverse each node of the multi-layer syntax tree instance and collect graph structure data corresponding to each node at each node of the multi-layer syntax tree instance.

[0015] In the above system, the system further comprises: a data lineage writing interface module connected to the data structure parser;

[0016] The data lineage write interface module is used to receive the data source change data from the business system and transmit the data source change data to the data structure parser. The data source change data is the data obtained by the business system recording and encapsulating the storage structure change data according to a preset data format. The storage structure change data is the data generated when the data storage structure and / or data source relationship of the data source in the business system changes.

[0017] In the above system, the system further comprises: a data lineage reading interface module connected to the processor;

[0018] The data lineage reading interface module is used to receive a data lineage reading request sent by the business system and determine a data node identifier from the data lineage reading request;

[0019] The processor is used to call the query statement corresponding to the data node identifier and search for target blood relationship information corresponding to the data node identifier from the graph database module; the target blood relationship information is the blood relationship path between the data nodes determined by the data node identifier.

[0020] In the above system, the syntax tree construction parser is also used to construct the data lineage structure corresponding to the target lineage relationship information;

[0021] The data structure parser is further used to encapsulate the target data blood relationship structure according to the data format corresponding to the business system to obtain target blood relationship display data.

[0022] In the above system, the data blood relationship reading interface module is further used to send the target blood relationship display data to the business system, so that the business system can display the target blood relationship display data on a visual interface.

[0023] An embodiment of the present invention provides a data lineage relationship management method, the method comprising:

[0024] Receive data source change data from the business system, the data source change data being data obtained by the business system recording and encapsulating storage structure change data in a preset data format, the storage structure change data being data generated when the data storage structure and / or data source relationship of a data source in the business system changes;

[0025] Calling a data structure parser to parse the data source change data to obtain data lineage relationship structure data;

[0026] Calling a syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module;

[0027] The graph structure data corresponding to the multi-layer syntax tree instance is stored in a graph database module so that the graph structure data can be managed by using the graph database module. The graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

[0028] In the above method, after calling the syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module, and before storing the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module, the method further includes:

[0029] Each node of the multi-layer syntax tree instance is traversed hierarchically, and graph structure data corresponding to each node is collected at each node of the multi-layer syntax tree instance.

[0030] In the above method, after storing the graph structure data in the graph database module, the method further includes:

[0031] When receiving a data lineage reading request sent by the business system, determining a data node identifier from the data lineage reading request;

[0032] Using the query statement corresponding to the data node identifier, searching the graph database module for target blood relationship information corresponding to the data node identifier;

[0033] Calling the syntax tree construction parser to construct a target data blood relationship structure corresponding to the target blood relationship information; the target blood relationship information is a blood relationship path between data nodes determined by the data node identifiers;

[0034] The target data blood relationship structure is encapsulated according to the data format corresponding to the business system to obtain target blood relationship display data.

[0035] In the above method, after encapsulating the target data blood relationship structure according to the data format corresponding to the business system to obtain the target blood relationship display data, the method further includes:

[0036] The target blood relationship display data is sent to the business system so that the business system can display the target blood relationship display data on a visual interface.

[0037] An embodiment of the present invention provides a data management system, the system comprising:

[0038] A data receiving unit, configured to receive data source change data from a business system, wherein the data source change data is data obtained by the business system recording and encapsulating storage structure change data in accordance with a preset data format, and the storage structure change data is data generated when the data storage structure and / or data source relationship of a data source in the business system changes;

[0039] A data parsing unit, configured to parse the data source change data to obtain data lineage relationship structure data;

[0040] A data conversion unit, configured to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module;

[0041] A data storage unit is used to store the graph structure data corresponding to the multi-layer syntax tree instance in a graph database module, so as to use the graph database module to manage the graph structure data, wherein the graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

[0042] An embodiment of the present invention provides a storage medium storing a computer program for use in a data management system. When the computer program is executed by a processor, the method described above is implemented.

[0043] The embodiment of the present invention provides a data lineage management method, system and storage medium, which can include: a data structure parser; a syntax tree structure parser connected to the data structure parser; a graph database module connected to the syntax tree structure parser; wherein the data structure parser is used to parse the data source change data sent from the business system to obtain data lineage structure data; the syntax tree structure parser is used to convert the data lineage structure data into a multi-layer syntax tree instance compatible with the graph database module; the graph database module is used to store and manage the graph structure data corresponding to the multi-layer syntax tree instance, the graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node. Using the above system implementation scheme, the data management system converts the data lineage structure data into a multi-layer syntax tree instance, and stores the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module. In the graph database module, the data lineage structure of each node can be obtained by accessing each node, thereby reducing the management complexity of the data lineage and improving the query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the structure of a data management system provided by an embodiment of the present invention Figure 1 ;

[0045] Figure 2 A schematic diagram of the system composition of an exemplary data management system provided by an embodiment of the present invention;

[0046] Figure 3 The process of a data lineage relationship management method provided by the embodiment of the present invention Figure 1 ;

[0047] Figure 4 A schematic diagram of a process for storing data lineage relationships in an AgensGraph according to an exemplary data management system provided by an embodiment of the present invention;

[0048] Figure 5 The process of a data lineage relationship management method provided by the embodiment of the present invention Figure 2 ;

[0049] Figure 6 A schematic diagram of a process for reading data lineage relationships from an AgensGraph in an exemplary data management system provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the structure of a data management system provided by an embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0052] Example 1

[0053] The embodiment of the present invention provides a data management system 1, such as Figure 1 As shown, the data management system 1 includes:

[0054] Data structure parser 10;

[0055] A syntax tree construction parser 11 connected to the data construction parser 10;

[0056] A graph database module 12 connected to the syntax tree construction parser 11; wherein,

[0057] The data structure parser 10 is used to parse the data source change data sent from the business system to obtain data lineage relationship structure data;

[0058] The syntax tree construction parser 11 is used to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module;

[0059] The graph database module 12 is used to store and manage the graph structure data corresponding to the multi-layer syntax tree instance, and the graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

[0060] A data management system provided by an embodiment of the present invention is suitable for managing the association relationship between multiple data sources in a business system.

[0061] In an embodiment of the present invention, a business system includes multiple types of data sources. These data sources can store the same data in different dimensions based on their own data structures. In this case, data lineage relationships exist between different types of data sources that store the same data. When a business system adds, modifies, or deletes data structures and / or data source relationships within a data source, the business system sends data source change data to the data management system.

[0062] In the embodiment of the present invention, the data source includes but is not limited to Mysql, Hive, Vertica, Oracle, Postgresql, etc., and is selected based on actual needs and is not specifically limited in the embodiment of the present invention.

[0063] In an embodiment of the present invention, the business system sends data source change data in json format to the data management system, and the data management system calls the data construction parser to parse the data source change data in json format to obtain data lineage relationship structure data; then, the data management system calls the syntax tree construction parser to construct a multi-layer syntax tree instance corresponding to the data lineage relationship structure data; the data management system writes the graph structure data corresponding to the multi-layer syntax tree instance into the graph database module (AgensGraph) for storage.

[0064] Optional, such as Figure 1 As shown, the system further includes: a processor 13; the processor 13 is connected to the data construction parser 10, the syntax tree construction parser 11 and the graph database module 12 respectively;

[0065] The processor 13 is configured to hierarchically traverse each node of the multi-layer syntax tree instance and collect graph structure data corresponding to each node at each node of the multi-layer syntax tree instance.

[0066] In an embodiment of the present invention, after the data relationship system calls the syntax tree construction parser to convert the data lineage relationship into a multi-layer syntax tree instance, the processor hierarchically traverses each node of the multi-layer syntax tree instance and collects graph structure data corresponding to each node.

[0067] In a specific embodiment, the processor 13 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present invention do not specifically limit this.

[0068] Optionally, the system 1 further comprises: a data lineage writing interface module connected to the data structure parser 10;

[0069] The data lineage write interface module is used to receive the data source change data from the business system and transmit the data source change data to the data structure parser. The data source change data is the data obtained by the business system recording and encapsulating the storage structure change data according to a preset data format. The storage structure change data is the data generated when the data storage structure and / or data source relationship of the data source in the business system changes.

[0070] In an embodiment of the present invention, when a business system performs change operations such as creation, modification and deletion of data tables, creation, modification and deletion of fields in a data source during data flow; or performs change operations such as creation, modification or deletion of data source relationships, the business system generates storage structure change data. The business system generates data source change data according to a preset data format, and the data management system calls the data lineage write interface module to receive the data source change data.

[0071] In the embodiment of the present invention, the preset data format may be a data format pre-agreed upon by the data source system and the data management system.

[0072] Exemplarily, the preset data format is the json data format.

[0073] It can be understood that a preset data format is pre-agreed between the business system and the data management system, and the business system can convert the storage structure change data format into data source change data in the preset data format according to the preset data format. When the types of data source systems continue to increase, the data management system in the embodiment of the present invention has good scalability for storage structure change data in different formats.

[0074] Optionally, the system further comprises: a data lineage reading interface module connected to the processor;

[0075] The data lineage reading interface module is used to receive a data lineage reading request sent by the business system and determine a data node identifier from the data lineage reading request;

[0076] The processor 13 is used to call the query statement corresponding to the data node identifier and search for target blood relationship information corresponding to the data node identifier from the graph database module; the target blood relationship information is the blood relationship path between the data nodes determined by the data node identifier.

[0077] In an embodiment of the present invention, a user can query the lineage relationship of data on the visualization interface of the business system. Specifically, the user enters the data node identifier on the visualization interface and clicks the data lineage relationship query button. At this time, the visualization interface adds the data node identifier to the data lineage reading request and sends the data lineage reading request to the data management system. The data management system calls the data lineage reading interface module to receive the data lineage request and determines the data node identifier from the data lineage reading request.

[0078] In the embodiment of the present invention, a data lineage reading request calls a query statement to search for target lineage relationship information corresponding to a data node identifier in AgensGraph.

[0079] In the embodiment of the present invention, the query statement may be a Cypher query statement or a Postgresql query statement, and the specific selection is made according to actual conditions, and the embodiment of the present invention does not make any specific limitation.

[0080] Optionally, the syntax tree construction parser 11 is further used to construct a target data blood relationship structure corresponding to the target blood relationship information;

[0081] The data structure parser 10 is further configured to encapsulate the target data blood relationship structure according to the data format corresponding to the business system to obtain target blood relationship display data.

[0082] In an embodiment of the present invention, the data management system calls a syntax number construction parser to construct the target blood relationship information into a target data blood relationship structure. Afterwards, the data management system calls a data construction parser to construct the target blood relationship display data in the json data format required for visual interface display.

[0083] Optionally, the data blood relationship reading interface module is further configured to send the target blood relationship display data to the business system, so that the business system can display the target blood relationship display data on a visual interface.

[0084] In an embodiment of the present invention, the data management system calls the data lineage reading interface module to send the target lineage relationship display data to the business system. The business system draws the data lineage relationship graph corresponding to the data node based on the target lineage relationship display data, and displays the data lineage relationship graph on the visual interface of the business system.

[0085] For example, Figure 2This is a system composition diagram for the data management system. The data management system includes a data lineage write interface, a JSON data construction parser, a graph structure syntax tree construction parser, AgensGraph, and a data lineage read interface. The data management system interacts with the data source through the data lineage write interface, and with the visualization interface through the data lineage read interface. The JSON data construction parser is responsible for parsing the JSON data passed in by the data source system and encapsulating the data retrieved from AgensGraph. The syntax tree construction parser is responsible for constructing a graph structure data syntax tree for the data passed in by the data source and parsed by the JSON data construction parser, and for constructing a graph structure data syntax tree for the data lineage relationships retrieved from AgensGraph. AgensGraph stores the graph structure data of data lineage and provides writing and query capabilities.

[0086] It can be understood that the data management system converts the data lineage relationship into a multi-layer syntax tree instance, and stores the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module. In the graph database module, the data lineage relationship of the node can be obtained by accessing each node, thereby reducing the management complexity of the data lineage relationship and improving query efficiency.

[0087] Example 2

[0088] The embodiment of the present invention provides a data lineage relationship management method, such as Figure 3 As shown, the method may include:

[0089] S101. Receive data source change data from a business system. The data source change data is data obtained by the business system recording and packaging storage structure change data according to a preset data format. The storage structure change data is data generated when the data storage structure and / or data source relationship of a data source in the business system changes.

[0090] It should be noted that the data lineage management method proposed in the embodiment of the present invention is implemented by the data management system of embodiment one. The specific internal connection relationship of the data management system and the specific execution content of each part are as shown in embodiment one and will not be repeated here.

[0091] In an embodiment of the present invention, when a business system performs change operations such as creation, modification and deletion of data tables, creation, modification and deletion of fields in a data source during data flow; or performs change operations such as creation, modification or deletion of data source relationships, the business system generates storage structure change data. The business system generates data source change data according to a preset data format, and the data management system calls the data lineage write interface module to receive the data source change data.

[0092] In the embodiment of the present invention, the data source includes but is not limited to Mysql, Hive, Vertica, Oracle, Postgresql, etc., and is selected based on actual needs and is not specifically limited in the embodiment of the present invention.

[0093] In the embodiment of the present invention, the preset data format may be a data format pre-agreed upon by the data source system and the data management system.

[0094] Exemplarily, the preset data format is the json data format.

[0095] S102: Calling a data structure parser to parse the data source change data to obtain data lineage relationship structure data.

[0096] After the data management system receives data source change data from the business system, the data management system calls the data structure parser to parse the data source change data to obtain data lineage relationship structure data.

[0097] In the embodiment of the present invention, the storage format of the data source change data in json data format is<key,value> , the data construction parser obtains the value it carries (data lineage relationship) by parsing the key.

[0098] S103. Call the syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance that is compatible with the graph database module.

[0099] After the data management system parses the data source change data and obtains the data lineage relationship structure data, the data management system calls the syntax tree construction parser to convert the data lineage relationship into a multi-layer syntax tree instance that is compatible with the graph database module.

[0100] In an embodiment of the present invention, the data management system calls a syntax tree construction parser to convert the data lineage structure data of the number of json data into a multi-layer syntax tree instance corresponding to AgensGraph. Afterwards, the data management system hierarchically traverses each node of the multi-layer syntax tree instance and collects the graph structure data corresponding to each node at each node of the multi-layer syntax tree instance.

[0101] S104. Store the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module, and use the graph database module to manage the graph structure data. The graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table fields corresponding to each node.

[0102] After the data management system converts the data lineage relationship into a multi-layer syntax tree instance that is compatible with the graph database module, the data management system stores the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module, and the data management system uses the graph database module to manage the graph structure data.

[0103] In the embodiment of the present invention, the data management system writes the collected graph structure data corresponding to the multi-layer syntax tree instance into the graph database module (AgensGraph) for storage.

[0104] For example, Figure 4 As shown in the figure, the process of the data management system writing data lineage relationships into AgensGraph is as follows:

[0105] 1. The data management system calls the lineage data writing interface create_lineage to receive the lineage data in JSON format;

[0106] 2. The data management system parses the data lineage relationship in JSON format and generates a multi-layer syntax tree instance;

[0107] 3. The data management system hierarchically traverses each node of the multi-layer syntax tree instance and collects the corresponding node data at each node;

[0108] 4. The data management system stores the collected node data in AgensGraph.

[0109] It can be understood that the data management system converts the data lineage relationship into a multi-layer syntax tree instance, and stores the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module. In the graph database module, the data lineage relationship of the node can be obtained by accessing each node, thereby reducing the management complexity of the data lineage relationship and improving query efficiency.

[0110] Furthermore, after the data management system stores the graph structure data in the graph database module, that is, after S104, the data management system can also read the process in the graph database module, such as Figure 5 As shown, specifically including:

[0111] S201. When receiving a data lineage reading request sent by a business system, determine a data node identifier from the data lineage reading request.

[0112] In an embodiment of the present invention, the user enters the data node identifier on the visual interface of the business system and clicks the data lineage relationship query button. At this time, the visual interface adds the data node identifier to the data lineage reading request and sends the data lineage reading request to the data management system. The data management system calls the data lineage reading interface module to receive the data lineage request and determines the data node identifier from the data lineage reading request.

[0113] S202. Use the query statement corresponding to the data node identifier to search for target blood relationship information corresponding to the data node identifier from the graph database module; the target blood relationship information is the blood relationship path between the data nodes determined by the data node identifier.

[0114] After the data management system determines the data node identifier from the data lineage reading request, the data management system uses a query statement to search for target lineage relationship information corresponding to the data node identifier from the graph database module.

[0115] In the embodiment of the present invention, the query statement may be a Cypher query statement or a Postgresql query statement, and the specific selection is made according to actual conditions, and the embodiment of the present invention does not make any specific limitation.

[0116] S203: Calling a syntax tree construction parser to construct a target data blood relationship structure corresponding to the target blood relationship information.

[0117] After the data management system finds the target blood relationship information corresponding to the data node identifier from the graph database module, the data management system calls the syntax tree construction parser to construct the target data blood relationship structure corresponding to the target blood relationship information.

[0118] S204. Encapsulate the target data blood relationship structure according to the data format corresponding to the visualization interface to obtain target blood relationship display data.

[0119] After the data management system constructs the target data blood relationship structure corresponding to the target blood relationship information, the data management system encapsulates the target data blood relationship structure according to the data format corresponding to the visualization interface to obtain the target blood relationship display data.

[0120] In the embodiment of the present invention, the data management system calls the data construction parser to construct the target data blood relationship, and constructs the target blood relationship display data in the json data format required for visual interface display.

[0121] Furthermore, the data management system sends the target blood relationship display data to the business system so that the business system can display the target blood relationship display data on a visual interface.

[0122] In an embodiment of the present invention, the data management system calls the data lineage reading interface module to send the target lineage relationship display data to the visualization interface. The visualization interface draws the data lineage relationship graph corresponding to the data node according to the target lineage relationship display data, and displays the data lineage relationship graph on the visualization interface.

[0123] For example, Figure 6 As shown in the figure, the process of the data management system reading data lineage relationships from AgensGraph is as follows:

[0124] 1. The data management system calls the data lineage reading interface read_lineage to receive the data node identifier;

[0125] 2. The data management system queries the target blood relationship information corresponding to the data node identifier from AgensGraph;

[0126] 3. The data management system calls the syntax tree parser to construct the target blood relationship information into abstract path structure data and returns it to the visualization interface;

[0127] 4. The visualization interface draws a page blood relationship diagram based on the returned json data.

[0128] The embodiment of the present invention provides a data management system 7, such as Figure 7 Shown, including:

[0129] A data receiving unit 70 is configured to receive data source change data from a business system. The data source change data is data obtained by the business system recording and encapsulating storage structure change data in a preset data format. The storage structure change data is data generated when the data storage structure and / or data source relationship of a data source in the business system changes.

[0130] The data parsing unit 71 is used to parse the data source change data to obtain data lineage relationship structure data;

[0131] A data conversion unit 72 is used to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module;

[0132] The data storage unit 73 is used to store the graph structure data corresponding to the multi-layer syntax tree instance into the graph database module, so as to use the graph database module to manage the graph structure data. The graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

[0133] In the above embodiment, the system further includes: a node traversal unit;

[0134] The node traversal unit is used to hierarchically traverse each node of the multi-layer syntax tree instance, and collect graph structure data corresponding to each node at each node of the multi-layer syntax tree instance.

[0135] In the above embodiment, the system further comprises: a determination unit, a search unit, a construction unit, and a packaging unit;

[0136] The determining unit is configured to determine a data node identifier from a data lineage reading request when receiving the data lineage reading request sent by the business system;

[0137] The search unit is configured to use a query statement corresponding to the data node identifier to search the graph database module for target blood relationship information corresponding to the data node identifier; the target blood relationship information is a blood relationship path between the data nodes determined by the data node identifier;

[0138] The construction unit is configured to call the syntax tree construction parser to construct a data lineage structure corresponding to the target lineage relationship information;

[0139] The encapsulation unit is used to encapsulate the target data blood relationship structure according to the data format corresponding to the business system to obtain target blood relationship display data.

[0140] In the above embodiment, the system further includes: a sending unit;

[0141] The sending unit is used to send the target blood relationship display data to the business system, so that the business system can display the target blood relationship display data on a visual interface.

[0142] An embodiment of the present invention provides a storage medium storing one or more programs. The one or more programs can be executed by one or more processors and are applied to a data management system. When the programs are executed by the processors, the following method is implemented:

[0143] Receive data source change data from the business system, the data source change data being data obtained by the business system recording and encapsulating storage structure change data in a preset data format, the storage structure change data being data generated when the data storage structure and / or data source relationship of a data source in the business system changes;

[0144] Calling a data structure parser to parse the data source change data to obtain data lineage relationship structure data;

[0145] Calling a syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module;

[0146] The graph structure data corresponding to the multi-layer syntax tree instance is stored in a graph database module so that the graph structure data can be managed by using the graph database module. The graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

[0147] Optionally, the calling of a syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module; thereafter, before storing the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module, the method further comprises:

[0148] Each node of the multi-layer syntax tree instance is traversed hierarchically, and graph structure data corresponding to each node is collected at each node of the multi-layer syntax tree instance.

[0149] Optionally, after storing the graph structure data in a graph database module, the method further includes:

[0150] When receiving a data lineage reading request sent by the business system, determining a data node identifier from the data lineage reading request;

[0151] Using the query statement corresponding to the data node identifier, searching the graph database module for target blood relationship information corresponding to the data node identifier;

[0152] Calling the syntax tree construction parser to construct a data blood relationship structure corresponding to the target blood relationship information; the target blood relationship information is a blood relationship path between data nodes determined by the data node identifiers;

[0153] The target data blood relationship structure is encapsulated according to the data format corresponding to the business system to obtain target blood relationship display data.

[0154] Optionally, after encapsulating the target data blood relationship structure according to the data format corresponding to the business system to obtain target blood relationship display data, the method further includes:

[0155] The target blood relationship display data is sent to the business system so that the business system can display the target blood relationship display data on a visual interface.

[0156] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0157] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a network function deployment server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data management system, characterized in that: The system comprises: Data structure parser; a syntax tree construction parser connected to the data construction parser; A graph database module connected to the syntax tree construction parser; the graph database module is AgensGraph, wherein, The data structure parser is used to parse the data source change data sent from the business system to obtain data lineage relationship structure data; the data source change data is the data obtained by the business system recording and encapsulating the storage structure change data according to a preset data format. The storage structure change data is the data generated when the data storage structure and / or data source relationship of the data source in the business system changes; the preset data format is JSON format; The syntax tree construction parser is used to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module; The graph database module is used to store and manage graph structure data corresponding to the multi-layer syntax tree instance. The graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

2. The system according to claim 1, wherein: The system further comprises: a processor; the processor is connected to the data construction parser, the syntax tree construction parser and the graph database module respectively; The processor is configured to hierarchically traverse each node of the multi-layer syntax tree instance and collect graph structure data corresponding to each node at each node of the multi-layer syntax tree instance.

3. The system according to claim 1, wherein: The system further comprises: a data lineage writing interface module connected to the data structure parser; The data lineage write interface module is used to receive the data source change data from the business system and transmit the data source change data to the data structure parser. The data source change data is the data obtained by the business system recording and encapsulating the storage structure change data according to a preset data format. The storage structure change data is the data generated when the data storage structure and / or data source relationship of the data source in the business system changes.

4. The system according to claim 2, wherein: The system further comprises: a data lineage reading interface module connected to the processor; The data lineage reading interface module is configured to receive a data lineage reading request sent by the business system and determine a data node identifier from the data lineage reading request; The processor is used to call the query statement corresponding to the data node identifier and search for target blood relationship information corresponding to the data node identifier from the graph database module; the target blood relationship information is the blood relationship path between the data nodes determined by the data node identifier.

5. The system according to claim 4, characterized in that The syntax tree construction parser is further used to construct a data blood relationship structure corresponding to the target blood relationship information; The data structure parser is further used to encapsulate the data blood relationship structure according to the data format corresponding to the business system to obtain target blood relationship display data.

6. The system according to claim 5, characterized in that The data blood relationship reading interface module is further used to send the target blood relationship display data to the business system, so that the business system can display the target blood relationship display data on a visual interface.

7. A data lineage relationship management method, characterized in that: The method comprises: Receive data source change data from the business system, the data source change data being data obtained by the business system recording and encapsulating storage structure change data in a preset data format. The storage structure change data is data generated when the data storage structure and / or data source relationship of the data source in the business system changes; the preset data format is JSON format; Calling a data structure parser to parse the data source change data to obtain data lineage relationship structure data; Calling a syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module; The graph structure data corresponding to the multi-layer syntax tree instance is stored in a graph database module so that the graph structure data can be managed by using the graph database module. The graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node; the graph database module is AgensGraph.

8. The method according to claim 7, characterized in that After calling the syntax tree construction parser to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to the graph database module, and before storing the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module, the method further includes: Each node of the multi-layer syntax tree instance is traversed hierarchically, and graph structure data corresponding to each node is collected at each node of the multi-layer syntax tree instance.

9. The method according to claim 7, characterized in that After storing the graph structure data corresponding to the multi-layer syntax tree instance in the graph database module, the method further includes: When receiving a data lineage reading request sent by the business system, determining a data node identifier from the data lineage reading request; Using the query statement corresponding to the data node identifier, searching the graph database module for target blood relationship information corresponding to the data node identifier; the target blood relationship information is the blood relationship path between the data nodes determined by the data node identifier; Calling the syntax tree construction parser to construct a data lineage structure corresponding to the target lineage relationship information; The target data blood relationship structure is encapsulated according to the data format corresponding to the business system to obtain target blood relationship display data.

10. The method according to claim 9, characterized in that After encapsulating the target data blood relationship structure according to the data format corresponding to the business system to obtain target blood relationship display data, the method further includes: The target blood relationship display data is sent to the business system so that the business system can display the target blood relationship display data on a visual interface.

11. A data management system, characterized in that: The system comprises: A data receiving unit, configured to receive data source change data from a business system, wherein the data source change data is data obtained by the business system recording and encapsulating storage structure change data in accordance with a preset data format, and the storage structure change data is data generated when the data storage structure and / or data source relationship of a data source in the business system changes; A data parsing unit, configured to parse the data source change data to obtain data lineage relationship structure data; A data conversion unit, configured to convert the data lineage relationship structure data into a multi-layer syntax tree instance adapted to a graph database module; A data storage unit is used to store the graph structure data corresponding to the multi-layer syntax tree instance in a graph database module, so as to use the graph database module to manage the graph structure data, wherein the graph structure data is used to represent the graph attribute data of each node in the multi-layer syntax tree instance and the data lineage relationship path between different nodes; the graph attribute data includes the data table and / or data table field corresponding to each node.

12. A storage medium having a computer program stored thereon, characterized in that: Applied to a data management system, the computer program implements the method according to any one of claims 7 to 10 when executed by a processor.

Images