Data blood relationship map generation method and device, equipment, storage medium and program product

Abstract

The invention relates to a data blood relationship map generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring to-be-processed original data and metadata corresponding to the original data from a plurality of data sources, wherein the plurality of data sources are respectively associated with a plurality of data life cycles involved in a target service; analyzing the original data corresponding to the plurality of data sources, and determining data flow direction information corresponding to the original data; and according to the data flow direction information corresponding to the original data and the metadata corresponding to the original data, constructing a data blood relationship map of the target business. By adopting the method, the traceability of the business data can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, device, equipment, storage medium and program product for generating a data lineage graph. Background Art

[0002] Data lineage plays a vital role in data management and data governance. Data lineage not only helps enterprises understand the source, flow process and final use of data, but also improves the efficiency of data operation and maintenance and data governance, and ensures the accuracy, consistency and reliability of data.

[0003] In the related technology, mainly during the data processing flow, the system tools are responsible for collecting data lineage information when sending data mapping, so as to collect relatively accurate and fine-grained data lineage information. However, it can only support the data lineage collection of a single system. When processing large-scale, cross-system business, the collected data lineage information cannot meet the traceability requirements of the business data. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, equipment, storage medium and program product for generating a data lineage graph that can improve the traceability of business data in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for generating a data lineage graph, comprising:

[0006] Acquire raw data to be processed and metadata corresponding to the raw data from multiple data sources, wherein the multiple data sources are respectively associated with multiple data lifecycles involved in the target business;

[0007] Analyze the original data corresponding to the multiple data sources respectively to determine the data flow information corresponding to the original data;

[0008] A data lineage graph of the target business is constructed based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

[0009] In one embodiment, respectively parsing the original data corresponding to the multiple data sources to determine the data flow information corresponding to the original data includes:

[0010] Decomposing the original data to obtain a plurality of tag information, and identifying key element information corresponding to the original data from the tag information;

[0011] Constructing an abstract syntax tree corresponding to the original data according to the key element information and preset grammar rule information;

[0012] The abstract syntax tree is recursively parsed to determine data flow information corresponding to the original data.

[0013] In one embodiment, the original data includes a structured query language; if the structured query language includes nested subqueries, recursively parsing the abstract syntax tree to determine data flow information corresponding to the original data includes:

[0014] Recursively parse each layer of nested subqueries according to the abstract syntax tree to determine data source information and data destination information of each layer of nested subqueries;

[0015] The data flow information corresponding to the structured query language is determined according to the data source information and data destination information of each nested sub-query.

[0016] In one embodiment, constructing a data lineage graph of the target business according to the data flow information corresponding to the original data and the metadata corresponding to the original data includes:

[0017] Create a blank data lineage graph;

[0018] Adding metadata corresponding to each original data as a node in the data lineage graph to the data lineage graph;

[0019] Connecting each node in the data lineage graph according to the data flow information corresponding to the original data;

[0020] Using a recursive fusion function, the metadata of the associated nodes connected to each node in the data lineage graph is fused into the node as the data lineage information of the node.

[0021] In one of the embodiments, after obtaining the raw data to be processed and metadata corresponding to the raw data from multiple data sources, the method further includes:

[0022] Using a preprocessing operation, preprocessing the original data and metadata corresponding to the original data;

[0023] The preprocessing operation includes at least one of the following: field type mapping, duplicate record removal, missing value processing, data error correction, data type conversion, data format standardization, data aggregation grouping, and business rule verification.

[0024] In one of the embodiments, the data life cycle involved in the target business includes at least one of the following: data collection, data storage, data processing, and data application.

[0025] In a second aspect, the present application provides a device for generating a data lineage graph, comprising:

[0026] A collection module, used to obtain raw data to be processed and metadata corresponding to the raw data from multiple data sources, wherein the multiple data sources are respectively associated with multiple data life cycles involved in the target business;

[0027] A parsing module, used to parse the original data corresponding to the multiple data sources respectively, and determine the data flow information corresponding to the original data;

[0028] A generation module is used to construct a data lineage graph of the target business based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

[0029] In one of the embodiments, the parsing module is also used to decompose the original data to obtain multiple tag information, and identify key element information corresponding to the original data from the tag information; construct an abstract syntax tree corresponding to the original data based on the key element information and preset grammar rule information; recursively parse the abstract syntax tree to determine the data flow information corresponding to the original data.

[0030] In one of the embodiments, the original data includes structured query language; if the structured query language includes nested subqueries, the parsing module is further used to recursively parse each layer of nested subqueries according to the abstract syntax tree to determine data source information and data destination information of each layer of nested subqueries; and determine data flow information corresponding to the structured query language based on the data source information and data destination information of each layer of nested subqueries.

[0031] In one of the embodiments, the generation module is also used to create a blank data lineage graph; add the metadata corresponding to each original data as a node in the data lineage graph; connect each node in the data lineage graph according to the data flow information corresponding to the original data; use a recursive fusion function to fuse the metadata of the associated nodes connected to each node in the data lineage graph into the node as the data lineage information of the node.

[0032] In one of the embodiments, the acquisition module is further used to pre-process the original data and metadata corresponding to the original data using a pre-processing operation;

[0033] The preprocessing operation includes at least one of the following: field type mapping, duplicate record removal, missing value processing, data error correction, data type conversion, data format standardization, data aggregation grouping, and business rule verification.

[0034] In one of the embodiments, the data life cycle involved in the target business includes at least one of the following: data collection, data storage, data processing, and data application.

[0035] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0038] The above-mentioned data lineage graph generation method, device, equipment, storage medium and program product first obtain the raw data to be processed and the metadata corresponding to the raw data from multiple data sources, and the multiple data sources are respectively associated with multiple data life cycles involved in the target business. Subsequently, the raw data corresponding to the multiple data sources are parsed respectively to determine the data flow information corresponding to the raw data. Finally, a data lineage graph for the target business is constructed based on the data flow information corresponding to the raw data and the metadata corresponding to the raw data. Since the collected raw data covers each data source associated with multiple data life cycles involved in the target business, the data flow information of each raw data is obtained through automatic parsing of the data to automatically generate a data lineage graph for the target business, thereby breaking down data silos and achieving seamless integration and unified view of the raw data from each data source, and thereby improving the traceability of business data through the generated data lineage graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 An application environment diagram for generating a data lineage diagram provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a flow chart of a method for generating a data lineage graph provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a flow chart of another method for generating a data lineage graph provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a flow chart of another method for generating a data lineage graph provided in an embodiment of the present application;

[0044] Figure 5 A structural block diagram of a device for generating a data lineage graph provided in an embodiment of the present application;

[0045] Figure 6 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] First, the related technology will be described below.

[0048] Data lineage plays a vital role in data management and data governance. Data lineage not only helps enterprises understand the source, flow process and final use of data, but also improves the efficiency of data operation and maintenance and data governance, and ensures the accuracy, consistency and reliability of data.

[0049] In related technologies, data lineage information can be generated in the following ways.

[0050] The first method is that during the data processing flow, the system tool is responsible for collecting data lineage when sending data mapping, thereby collecting relatively accurate and fine-grained data lineage information. However, it can only support data lineage collection in a single system. When processing large-scale, cross-system business, the collected data lineage information cannot meet the traceability requirements of business data.

[0051] The second method is to manually record the source and flow of data to generate data lineage information. However, although the data lineage information generated manually is more accurate, it is inefficient, prone to errors, and difficult to adapt to the processing needs of large-scale, cross-system business.

[0052] The third method mainly uses machine learning to calculate the similarity of data through the dependencies between data sets, and then generate data lineage information. However, although machine learning is not dependent on tools and businesses, the accuracy of the generated data lineage information still needs to be manually confirmed, which is also difficult to adapt to the processing needs of large-scale, cross-system businesses.

[0053] To solve the above problems, the embodiments of the present application provide a method, device, equipment, storage medium and program product for generating a data lineage graph. Since the collected original data covers various data sources associated with multiple data life cycles involved in the target business, the data flow information of each original data is obtained through automatic data analysis to automatically generate a data lineage graph for the target business, thereby breaking down data silos and achieving seamless integration and unified view of the original data from various data sources, and thereby improving the traceability of business data through the generated data lineage graph.

[0054] The application environment of the method for generating a data lineage graph is described below.

[0055] The data lineage graph generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with multiple servers 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Each server 104 can be a data source associated with multiple data life cycles involved in the target business.

[0056] The terminal 102 can obtain the raw data to be processed and the metadata corresponding to the raw data from the multiple servers 104, and the multiple data sources are respectively associated with multiple data life cycles involved in the target business. Subsequently, the terminal 102 parses the raw data corresponding to the multiple data sources respectively to determine the data flow information corresponding to the raw data. Finally, the terminal 102 constructs a data lineage graph of the target business based on the data flow information corresponding to the raw data and the metadata corresponding to the raw data.

[0057] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0058] In an exemplary embodiment, Figure 2 As shown, a method for generating a data lineage graph is provided, and the method is applied to Figure 1 The following example describes the terminals in Figure 1, including S201-S203:

[0059] S201. Obtaining raw data to be processed and metadata corresponding to the raw data from multiple data sources, where the multiple data sources are respectively associated with multiple data lifecycles involved in a target business.

[0060] In the present application, when it is necessary to generate a data lineage graph for a target business, the original data to be processed and metadata corresponding to the original data can be obtained from multiple data sources.

[0061] Among them, the above-mentioned target business can be any business, which is not limited in the embodiment of the present application, such as data financial business, data query business, etc.

[0062] The raw data refers to the unprocessed business data of the target business, and the metadata refers to the data describing the raw data.

[0063] In some embodiments, the data life cycle involved in the target business includes at least one of the following: data collection, data storage, data processing, and data application.

[0064] For example, the terminal can collect raw data and metadata from multiple data sources through automated scripts, thereby ensuring the comprehensiveness and accuracy of the collected data in multiple data life cycles involved in the target business. The automated script may include a metadata extraction tool to automatically obtain metadata such as table structure and field attributes from different data sources.

[0065] In some embodiments, after obtaining the raw data to be processed and the metadata corresponding to the raw data from multiple data sources, the terminal may also use a preprocessing operation to preprocess the raw data and the metadata corresponding to the raw data.

[0066] Among them, the preprocessing operation includes at least one of the following: field type mapping, duplicate record removal, missing value processing, data error correction, data type conversion, data format standardization, data aggregation grouping, and business rule verification.

[0067] For example, the terminal's automated script can record detailed information about each preprocessing operation, including the processing logic and rules corresponding to each preprocessing operation, so as to ensure that the original data can be cleaned, formatted and optimized as necessary during the collection process from the source system to the target system. For example, the automated script can map the source system field type to the target system, remove duplicate records, handle missing values, correct data errors, perform data type conversion, standardize data formats, aggregate and group data, and verify data to comply with business rules.

[0068] Exemplarily, the terminal's automation script may include a metadata model, so that the collected metadata is converted into a unified format and stored in a metadata repository. The scheduling system on the terminal may be responsible for managing and scheduling the execution of data collection and preprocessing operations on the automation script. The scheduling system may also record the execution order, execution time, and dependencies between the data collection and preprocessing operations.

[0069] The metadata model is a standardized framework for defining a piece of metadata information, which is used to record the structure and attributes of metadata. The metadata model includes a series of metadata entities, attributes, relationships, and constraints, which together constitute a description of the original data, including information such as the source, structure, quality, and usage of the data.

[0070] For example, a metadata model usually contains information such as entities, attributes, relationships, and constraints. Entities represent the main objects in the metadata model, such as tables, fields, data flows, transformation processes, etc. Attributes describe the characteristics of entities, such as the name of the table, the data type of the field, the source and target of the data flow, etc. Relationships define the associations between entities, such as the relationship between fields and tables, the relationship between data flows and transformation steps, etc. Constraints specify the rules and restrictions of metadata. In addition, the metadata model can also be enriched with other dimensions as needed, including business term descriptions, business rules, versions, security and permission descriptions, etc.

[0071] S202: Analyze the original data corresponding to the multiple data sources respectively to determine the data flow information corresponding to the original data.

[0072] In this step, after the terminal obtains the original data to be processed and the metadata corresponding to the original data from multiple data sources, the original data corresponding to the multiple data sources can be analyzed respectively to determine the data flow information corresponding to the original data.

[0073] In some embodiments, the terminal can use predefined parsing rules or parsing methods to parse the raw data corresponding to multiple data sources respectively, determine the data flow information corresponding to the raw data, and thus ensure the accuracy and completeness of the data lineage relationship. During the parsing process, the terminal will also record key information such as the source, destination, and conversion logic of the raw data, laying the foundation for subsequent lineage fusion.

[0074] In some embodiments, the terminal may first decompose the original data to obtain multiple tag information, and identify key element information corresponding to the original data from the tag information. Subsequently, the terminal constructs an abstract syntax tree corresponding to the original data according to the key element information and preset grammar rule information. Finally, the terminal may recursively parse the abstract syntax tree to determine the data flow information corresponding to the original data.

[0075] Among them, the raw data includes structured query language (SQL).

[0076] Exemplarily, the terminal may decompose the SQL statement into a series of token information, such as keywords, identifiers, operators, constants, etc. Then, the key element information corresponding to the original data is identified from the token information, and the key element information can be used to determine the source and destination of the original data.

[0077] For example, if the SQL statement is SELECT a FROM b WHERE c = d, it can be decomposed into tag information such as SELECT, a, FROM, b, WHERE, c, =, d, etc. Then, key element information such as SELECT, FROM, JOIN, INSERT and WHERE is identified therefrom.

[0078] Exemplarily, after the key element information corresponding to the original data is identified, syntax analysis may be performed to construct an abstract syntax tree (AST) using the key element information and preset syntax rules.

[0079] Among them, AST is a hierarchical structure that represents the logical structure of SQL statements in the form of a tree. Each node in AST represents an SQL construct, such as table name, field name, query condition, etc.

[0080] For example, after constructing the abstract syntax tree corresponding to the original data, the AST can be recursively parsed to determine the data flow information corresponding to the structured query language. During the traversal of the recursive parsing, each node of the AST can be accessed to collect information about the data source (such as the table in the FROM clause) and the data target (such as the table in the INSERT INTO clause). If the stored procedure or data warehouse contains calls to other stored procedures or Extract, Transform, Load (ETL) tasks, these calls need to be recursively parsed until the original source of the data is traced back to determine the data flow information.

[0081] In some embodiments, if the structured query language includes nested subqueries, the abstract syntax tree is recursively parsed to determine the data flow information corresponding to the original data, including: first, the terminal recursively parses each layer of nested subqueries according to the abstract syntax tree to determine the data source information and data destination information of each layer of nested subqueries. Subsequently, the terminal determines the data flow information corresponding to the structured query language according to the data source information and data destination information of each layer of nested subqueries.

[0082] Exemplarily, the key element information in the SQL statement can be used to identify whether the SQL statement contains nested subqueries. For SQL statements containing nested subqueries, the above recursive parsing can be repeated to ensure that each layer of nested subqueries can correctly identify their data source and destination, thereby determining the data flow information corresponding to the structured query language, so as to map the data flow between the subquery and the external query into the data lineage graph.

[0083] S203: Construct a data lineage graph of the target business based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

[0084] In this step, after the terminal parses the original data corresponding to multiple data sources respectively and determines the data flow information corresponding to the original data, it can build a data lineage graph of the target business based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

[0085] Among them, the above-mentioned data lineage graph can be a full-link data lineage graph of the target business.

[0086] It should be understood that the generation of a data lineage graph is to integrate and associate multiple data life cycles involved in the target business, such as data collection, data storage, data processing, data application, data warehouse, interface, reporting system and other raw data as lineage data to form a unified and comprehensive lineage view. In this way, large-scale and highly complex lineage data can be processed.

[0087] In some embodiments, the terminal can retrieve the original data scattered in different links as the lineage data through the fusion engine, and then integrate the lineage data according to the data flow and dependency relationship through the lineage fusion algorithm.

[0088] Among them, the lineage fusion algorithm can realize the fusion association of multiple system lineage data based on the depth-first search (DFS) strategy to generate full-link data lineage.

[0089] The lineage data graph includes nodes, edges, and graphs. A node represents a node in the lineage data graph, and contains the metadata of the node (such as table name, field name, etc.). An edge represents the connection relationship between nodes in the lineage data graph, that is, the data flow direction. A graph represents the entire data lineage graph, which includes multiple nodes and edges between nodes.

[0090] In some embodiments, the terminal may first create a blank data lineage graph. Secondly, the terminal may add the metadata corresponding to each raw data as a node in the data lineage graph to the data lineage graph. Thirdly, the terminal may connect each node in the data lineage graph according to the data flow information corresponding to the raw data. Finally, the terminal may use a recursive fusion function to fuse the metadata of the associated nodes connected to each node in the data lineage graph into the node as the data lineage information of the node.

[0091] Exemplarily, the terminal creates a blank data lineage graph, which can be understood as initializing the data lineage graph, thereby creating a blank data lineage graph to store all lineage data. Subsequently, the terminal adds the metadata corresponding to the original data collected from each data source as a node in the data lineage graph, and then constructs the edges between each node through the data flow information corresponding to each metadata, thereby connecting each node in the data lineage graph. Finally, a recursive fusion function is used to fuse the data lineage information between each node.

[0092] Exemplarily, the input of the recursive fusion function can be the node currently being processed or the entire data lineage graph. The fusion purpose of the recursive fusion function is to recursively fuse all relevant data lineage information starting from the node currently being processed. The output of the recursive fusion function is the constructed data lineage graph.

[0093] Exemplarily, the processing process of the recursive fusion function includes: if the node currently being processed has been visited, return to avoid a loop. Mark the node currently being processed as visited. Traverse all the associated nodes of the outgoing edges of the node currently being processed, that is, the nodes to which the data corresponding to the node currently being processed flows. For each associated node of the outgoing edge, use a recursive fusion function. After recursively returning, the metadata of all the associated nodes of the outgoing edges are merged into the node currently being processed as the data lineage information of the node. If the node currently being processed is the final node, end the recursion.

[0094] In some embodiments, after constructing the data lineage graph of the target business, the data lineage graph of the target business can be displayed on the display component of the terminal, and the user can intuitively view the data lineage graph through the visual interface to determine the origin, flow and final output of the business data. In addition, the data lineage graph can also be maintained and updated by the lineage data to ensure the timeliness and accuracy of the lineage data.

[0095] The data lineage graph generation method provided in the embodiment of the present application uses automated technology to collect raw data and metadata and perform lineage analysis, which greatly improves operational efficiency and accuracy and reduces reliance on manual intervention. This is in contrast to traditional technologies, which are often limited to a single link and have deficiencies in automation, requiring more manual operations. The data lineage graph generation method provided in the present application provides more powerful and flexible support in data management and governance.

[0096] The method for generating a data lineage graph provided in an embodiment of the present application first obtains the raw data to be processed and the metadata corresponding to the raw data from multiple data sources, and the multiple data sources are respectively associated with multiple data life cycles involved in the target business. Subsequently, the raw data corresponding to the multiple data sources are respectively parsed to determine the data flow information corresponding to the raw data. Finally, a data lineage graph for the target business is constructed based on the data flow information corresponding to the raw data and the metadata corresponding to the raw data. Since the collected raw data covers each data source associated with multiple data life cycles involved in the target business, the data flow information of each raw data is obtained through automatic parsing of the data to automatically generate a data lineage graph for the target business, thereby breaking down data silos, achieving seamless integration and unified views of the raw data from each data source, and improving the traceability of business data through the generated data lineage graph.

[0097] The following describes how to determine the data flow information corresponding to the original data. Figure 3 A schematic diagram of a flow chart of another method for generating a data lineage graph provided in an embodiment of the present application, wherein the method for generating a data lineage graph is applied to a terminal device, such as Figure 3As shown, the method for generating the data lineage graph includes S301-S305:

[0098] S301. Obtaining original data to be processed and metadata corresponding to the original data from multiple data sources, where the multiple data sources are respectively associated with multiple data lifecycles involved in a target business.

[0099] Among them, the above-mentioned target business can be any business, which is not limited in the embodiment of the present application, such as data financial business, data query business, etc.

[0100] The raw data refers to the unprocessed business data of the target business, and the metadata refers to the data describing the raw data.

[0101] In some embodiments, the data life cycle involved in the target business includes at least one of the following: data collection, data storage, data processing, and data application.

[0102] S302: Decompose the original data to obtain a plurality of tag information, and identify key element information corresponding to the original data from the tag information.

[0103] Among them, the raw data includes structured query language (SQL).

[0104] Exemplarily, the terminal may decompose the SQL statement into a series of token information, such as keywords, identifiers, operators, constants, etc. Then, the key element information corresponding to the original data is identified from the token information, and the key element information can be used to determine the source and destination of the original data.

[0105] S303: construct an abstract syntax tree corresponding to the original data according to the key element information and the preset syntax rule information.

[0106] Exemplarily, after the key element information corresponding to the original data is identified, syntax analysis may be performed to construct an abstract syntax tree (AST) using the key element information and preset syntax rules.

[0107] Among them, AST is a hierarchical structure that represents the logical structure of SQL statements in the form of a tree. Each node in AST represents an SQL construct, such as table name, field name, query condition, etc.

[0108] S304: recursively parse the abstract syntax tree to determine data flow information corresponding to the original data.

[0109] For example, after constructing the abstract syntax tree corresponding to the original data, the AST can be recursively parsed to determine the data flow information corresponding to the structured query language. During the traversal of the recursive parsing, each node of the AST can be accessed to collect information about the data source (such as the table in the FROM clause) and the data target (such as the table in the INSERT INTO clause). If the stored procedure or data warehouse contains calls to other stored procedures or Extract, Transform, Load (ETL) tasks, these calls need to be recursively parsed until the original source of the data is traced back to determine the data flow information.

[0110] In some embodiments, if the structured query language includes nested subqueries, the abstract syntax tree is recursively parsed to determine the data flow information corresponding to the original data, including: first, the terminal recursively parses each layer of nested subqueries according to the abstract syntax tree to determine the data source information and data destination information of each layer of nested subqueries. Subsequently, the terminal determines the data flow information corresponding to the structured query language according to the data source information and data destination information of each layer of nested subqueries.

[0111] S305: Construct a data lineage graph of the target business based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

[0112] The following describes how to construct a data lineage graph for the target business. Figure 4 A flowchart of another method for generating a data lineage graph provided in an embodiment of the present application is provided. The method for generating a data lineage graph is applied to a terminal device, such as Figure 4 As shown, the method for generating the data lineage graph includes S401-S406:

[0113] S401. Obtaining raw data to be processed and metadata corresponding to the raw data from multiple data sources, wherein the multiple data sources are respectively associated with multiple data lifecycles involved in a target business.

[0114] S402: Analyze the original data corresponding to the multiple data sources respectively to determine the data flow information corresponding to the original data.

[0115] S403. Create a blank data lineage diagram.

[0116] S404: Add the metadata corresponding to each original data as a node in the data lineage graph.

[0117] S405. Connect each node in the data lineage graph according to the data flow information corresponding to the original data.

[0118] S406: Use a recursive fusion function to fuse metadata of associated nodes connected to each node in the data lineage graph into the node as data lineage information of the node.

[0119] The method for generating a data lineage graph provided in an embodiment of the present application first obtains the raw data to be processed and the metadata corresponding to the raw data from multiple data sources, and the multiple data sources are respectively associated with multiple data life cycles involved in the target business. Subsequently, the raw data corresponding to the multiple data sources are respectively parsed to determine the data flow information corresponding to the raw data. Finally, a data lineage graph for the target business is constructed based on the data flow information corresponding to the raw data and the metadata corresponding to the raw data. Since the collected raw data covers each data source associated with multiple data life cycles involved in the target business, the data flow information of each raw data is obtained through automatic parsing of the data to automatically generate a data lineage graph for the target business, thereby breaking down data silos, achieving seamless integration and unified views of the raw data from each data source, and improving the traceability of business data through the generated data lineage graph.

[0120] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0121] Based on the same inventive concept, the embodiment of the present application also provides a device for generating a data lineage graph for implementing the method for generating a data lineage graph involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of the device for generating a data lineage graph provided below can refer to the limitations of the method for generating a data lineage graph above, and will not be repeated here.

[0122] In an exemplary embodiment, Figure 5 As shown, a data lineage graph generation device 500 is provided, comprising: a collection module 501, a parsing module 502 and a generation module 503, wherein:

[0123] The acquisition module 501 is used to obtain the raw data to be processed and the metadata corresponding to the raw data from multiple data sources, and the multiple data sources are respectively associated with multiple data life cycles involved in the target business.

[0124] The parsing module 502 is used to parse the original data corresponding to multiple data sources respectively to determine the data flow information corresponding to the original data.

[0125] The generation module 503 is used to construct a data lineage graph of the target business according to the data flow information corresponding to the original data and the metadata corresponding to the original data.

[0126] In one of the embodiments, the parsing module 502 is also used to decompose the original data to obtain multiple tag information, and identify key element information corresponding to the original data from the tag information; construct an abstract syntax tree corresponding to the original data based on the key element information and preset grammar rule information; recursively parse the abstract syntax tree to determine the data flow information corresponding to the original data.

[0127] In one of the embodiments, the original data includes structured query language; if the structured query language includes nested subqueries, the parsing module is further used to recursively parse each layer of nested subqueries according to the abstract syntax tree to determine data source information and data destination information of each layer of nested subqueries; based on the data source information and data destination information of each layer of nested subqueries, determine the data flow information corresponding to the structured query language.

[0128] In one of the embodiments, the generation module 503 is also used to create a blank data lineage graph; add the metadata corresponding to each original data as a node in the data lineage graph; connect each node in the data lineage graph according to the data flow information corresponding to the original data; use a recursive fusion function to fuse the metadata of the associated nodes connected to each node in the data lineage graph into the node as the data lineage information of the node.

[0129] In one embodiment, the acquisition module 501 is further used to pre-process the original data and metadata corresponding to the original data using a pre-processing operation;

[0130] Among them, the preprocessing operation includes at least one of the following: field type mapping, duplicate record removal, missing value processing, data error correction, data type conversion, data format standardization, data aggregation grouping, and business rule verification.

[0131] In one of the embodiments, the data life cycle involved in the target business includes at least one of the following: data collection, data storage, data processing, and data application.

[0132] The above modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating a data bloodline graph is implemented.

[0134] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned method for generating a data lineage graph when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating the data lineage graph is implemented.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned method for generating a data lineage graph when executed by a processor.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a data lineage graph, characterized in that: The method comprises: Acquire raw data to be processed and metadata corresponding to the raw data from multiple data sources, wherein the multiple data sources are respectively associated with multiple data lifecycles involved in the target business; Analyze the original data corresponding to the multiple data sources respectively to determine the data flow information corresponding to the original data; A data lineage graph of the target business is constructed based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

2. The method according to claim 1, characterized in that The step of respectively parsing the original data corresponding to the multiple data sources to determine the data flow information corresponding to the original data includes: Decomposing the original data to obtain a plurality of tag information, and identifying key element information corresponding to the original data from the tag information; Constructing an abstract syntax tree corresponding to the original data according to the key element information and preset grammar rule information; The abstract syntax tree is recursively parsed to determine data flow information corresponding to the original data.

3. The method according to claim 2, characterized in that The original data includes a structured query language; if the structured query language includes a nested subquery, recursively parsing the abstract syntax tree to determine data flow information corresponding to the original data includes: Recursively parse each layer of nested subqueries according to the abstract syntax tree to determine data source information and data destination information of each layer of nested subqueries; The data flow information corresponding to the structured query language is determined according to the data source information and data destination information of each nested sub-query.

4. The method according to claim 1, characterized in that: The step of constructing a data lineage graph of the target business according to the data flow information corresponding to the original data and the metadata corresponding to the original data includes: Create a blank data lineage graph; Adding metadata corresponding to each original data as a node in the data lineage graph to the data lineage graph; Connecting each node in the data lineage graph according to the data flow information corresponding to the original data; Using a recursive fusion function, the metadata of the associated nodes connected to each node in the data lineage graph is fused into the node as the data lineage information of the node.

5. The method according to any one of claims 1 to 4, characterized in that: After obtaining the raw data to be processed and metadata corresponding to the raw data from the multiple data sources, the method further includes: Using a preprocessing operation, preprocessing the original data and metadata corresponding to the original data; The preprocessing operation includes at least one of the following: field type mapping, duplicate record removal, missing value processing, data error correction, data type conversion, data format standardization, data aggregation grouping, and business rule verification.

6. The method according to any one of claims 1 to 4, characterized in that: The data life cycle involved in the target business includes at least one of the following: data collection, data storage, data processing, and data application.

7. A device for generating a data lineage graph, characterized in that: The device comprises: A collection module, used to obtain raw data to be processed and metadata corresponding to the raw data from multiple data sources, wherein the multiple data sources are respectively associated with multiple data life cycles involved in the target business; A parsing module, used to parse the original data corresponding to the multiple data sources respectively, and determine the data flow information corresponding to the original data; A generation module is used to construct a data lineage graph of the target business based on the data flow information corresponding to the original data and the metadata corresponding to the original data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images