Method, apparatus, device and storage medium for generating cross-engine data lineage relationships

Abstract

This application provides a method, apparatus, device, and storage medium for generating cross-engine data lineage relationships. The method includes: acquiring target data and engine identifiers of multiple engines performing data processing on the target data in a data platform; determining sub-target data corresponding to each engine identifier in the target data; generating corresponding lexical and grammar files based on the engine identifiers, the lexical and grammar files being used to perform data lineage analysis on the corresponding sub-target data to obtain data sub-lineage relationships; and integrating the data sub-lineage relationships obtained from multiple engine identifiers to generate a global data lineage relationship for the target data. The method of this application improves the accuracy of cross-engine data lineage relationship generation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for generating cross-engine data lineage. Background Technology

[0002] The concept of data lineage is likened to a "dragon vein" for data, its importance self-evident. As data applications become increasingly diverse, the problem of data lacking a clear lineage is becoming more prominent. From a macro perspective, data is like pearls scattered everywhere, gradually forming data silos; from a micro perspective, data is like DNA with breaks, making it impossible to quickly identify data sources, processing logic, or computational methods. Without a network, without a "dragon vein," data cannot become a living, transferable, traceable, assessable, and quantifiable ecosystem. In today's open-source era, numerous different types of computing engines have emerged. These engines are applied to different use cases due to their unique characteristics. While they generally support or have been adapted to SQL, and there is an international SQL standard, their syntactic dialects vary somewhat.

[0003] In existing technologies, traditional ETL (Extract-Transform-Load) tools such as Informatica, Kettle, and Datastage can be used to obtain the source table, target table, or field relationships by using their hooks to extract the execution plan, or data lineage can be obtained through open-source metadata management components.

[0004] However, the existing methods of obtaining lineage do not take into account the grammatical dialects between different engines, which can lead to broken data lineage and inaccurate generated lineage. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for generating cross-engine data lineage relationships, in order to solve the problem of inaccurate generation of cross-engine data lineage relationships.

[0006] Firstly, this application provides a method for generating cross-engine data lineage, including:

[0007] Obtain the target data and the engine identifiers of multiple engines in the data platform that perform data processing on the target data;

[0008] Determine the sub-target data corresponding to each engine identifier in the target data;

[0009] Based on the engine identifier, a corresponding lexical file and a syntax file are generated. The lexical file and the syntax file are used to perform data lineage analysis on the corresponding sub-target data to obtain the data sub-lineage.

[0010] The data sub-lineage relationships obtained from multiple engine identifiers are integrated to generate the global data lineage relationship of the target data.

[0011] Secondly, this application provides a cross-engine data lineage generation apparatus, comprising:

[0012] The acquisition module is used to acquire target data and engine identifiers of multiple engines in the data platform that perform data processing on the target data;

[0013] The determination module is used to determine the sub-target data corresponding to each engine identifier in the target data;

[0014] The generation module is used to generate corresponding lexical files and syntax files based on the engine identifier. The lexical files and syntax files are used to perform data lineage analysis on the corresponding sub-target data to obtain data sub-lineage.

[0015] The generation module is also used to integrate the data sub-lineage relationships obtained based on multiple engine identifiers to generate the global data lineage relationship of the target data.

[0016] Thirdly, this application provides a cross-engine data lineage generation device, comprising:

[0017] Processor, memory, communication interface;

[0018] The memory is used to store the executable instructions of the processor;

[0019] The processor is configured to execute the cross-engine data lineage generation method described in the first aspect above by executing the executable instructions.

[0020] Fourthly, this application provides a readable storage medium, including: a computer program stored thereon, wherein when executed by a processor, the computer program implements the cross-engine data lineage generation method described in the first aspect above.

[0021] The cross-engine data lineage generation method, apparatus, device, and storage medium provided in this application acquire target data and engine identifiers of multiple engines performing data processing on the target data in a data platform. They determine sub-target data corresponding to each engine identifier within the target data, generate corresponding lexical and grammar files based on the engine identifiers, and use these files to perform data lineage analysis on the corresponding sub-target data to obtain sub-lineage relationships. The sub-lineage relationships obtained from multiple engine identifiers are then integrated to generate the global data lineage relationship of the target data. By dividing the target data into multiple sub-target data based on engine identifiers and generating corresponding lexical and grammar files for data lineage analysis of each sub-target data, the accuracy of the data lineage relationship analysis for each sub-target data is improved, thus enhancing the accuracy of the global data lineage relationship of the target data and improving the overall accuracy of the cross-engine data lineage generation method. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A flowchart illustrating the cross-engine data lineage generation method provided in this application embodiment;

[0024] Figure 2 This is a flowchart illustrating the process of performing data lineage analysis on corresponding sub-target data to obtain the data sub-lineage relationship of target data, as provided in an embodiment of this application.

[0025] Figure 3 This is a flowchart illustrating the process of traversing the syntax tree using the visitor pattern to obtain the sub-data lineage of the target data, as provided in an embodiment of this application.

[0026] Figure 4 A schematic diagram of a cross-engine data lineage generation device provided in this application embodiment;

[0027] Figure 5 This is a schematic diagram of the structure of a cross-engine data lineage generation device provided in an embodiment of this application.

[0028] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] In existing technologies, data lineage can be obtained using traditional ETL (Extract-Transform-Load) tools such as Informatica, Kettle, and DataStage, by using their hooks to retrieve the execution plan to obtain the source table, target table, or field relationships, or through open-source metadata management components. However, existing methods of obtaining lineage do not consider the syntactic dialects between different engines, which can lead to broken data lineages and inaccurate generated lineages.

[0031] This application obtains target data and engine identifiers of multiple engines performing data processing on the target data in a data platform. It then identifies sub-target data corresponding to each engine identifier within the target data, generates corresponding lexical and grammar files based on the engine identifiers, and uses these files to perform data lineage analysis on the corresponding sub-target data to obtain data sub-lineage relationships. These sub-lineage relationships obtained from multiple engine identifiers are then integrated to generate a global data lineage relationship for the target data. By dividing the target data into multiple sub-target data based on engine identifiers and generating corresponding lexical and grammar files for data lineage analysis of each sub-target data, the accuracy of data lineage analysis for each sub-target data is improved, thus enhancing the accuracy of the global data lineage relationship for the target data and improving the accuracy of the cross-engine data lineage relationship generation method.

[0032] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating the cross-engine data lineage generation method provided in the first embodiment of this application. The executing entity is a cross-engine data lineage generation device, which can generally be implemented by software, hardware, or a combination of software and hardware.

[0034] like Figure 1 As shown, the cross-engine data lineage generation method in this embodiment may include the following steps:

[0035] Step S101: Obtain the target data and the engine identifiers of multiple engines in the data platform that perform data processing on the target data.

[0036] Specifically, it can obtain the target data and the engine identifiers of multiple engines in the data platform that perform data processing on the target data. An engine refers to a component that flows through the target data during data processing. Specifically, the set of components, i.e., the set of engines, varies in different data service scenarios. Optionally, the types of engines, i.e., components, include, but are not limited to: Hive, Spark, Flink, HBase, Redis, ClickHouse, MySQL, and Redis.

[0037] Step S102: Determine the sub-target data corresponding to each engine identifier in the target data.

[0038] Specifically, in the target data obtained in step S101, sub-target data corresponding to each engine identifier can be determined for data lineage analysis, thereby improving the accuracy of data lineage analysis.

[0039] Step S103: Generate the corresponding lexical file and syntax file according to the engine identifier. The lexical file and syntax file are used to perform data lineage analysis on the corresponding sub-target data to obtain the data sub-lineage.

[0040] Specifically, based on the engine identifier obtained in step S10, corresponding lexical and grammar files can be generated. These files are used to perform data lineage analysis on the corresponding sub-target data to obtain the data sub-lineage relationships. Optionally, the methods for generating the lexical and grammar files include, but are not limited to, generating them using a parser generation tool.

[0041] Step S104: Integrate the data sub-lineage relationships obtained from multiple engine identifiers to generate the global data lineage relationship of the target data.

[0042] Specifically, the data sub-lineage relationships generated in step S103 based on multiple engine identifiers can be integrated to generate the global data lineage relationship of the target data. Optionally, this application does not limit the process of integrating multiple data sub-lineage relationships.

[0043] The cross-engine data lineage generation method provided in this embodiment obtains target data and engine identifiers of multiple engines performing data processing on the target data in the data platform. It then determines the sub-target data corresponding to each engine identifier within the target data, generates corresponding lexical and grammar files based on the engine identifiers, and uses these files to perform data lineage analysis on the corresponding sub-target data to obtain sub-lineage relationships. The sub-lineage relationships obtained from multiple engine identifiers are then integrated to generate the global data lineage relationship of the target data. By dividing the target data into multiple sub-target data based on engine identifiers and generating corresponding lexical and grammar files for data lineage analysis of each sub-target data, the accuracy of the data lineage relationship analysis for each sub-target data is improved, thus enhancing the accuracy of the global data lineage relationship of the target data and improving the overall accuracy of the cross-engine data lineage generation method.

[0044] Figure 2 This is a flowchart illustrating the process of performing data lineage analysis on corresponding sub-target data to obtain the data sub-lineage relationship of target data, as provided in the second embodiment of this application. Figure 1 Based on the illustrated embodiment, this embodiment elaborates on the process of performing data lineage analysis on the corresponding sub-target data.

[0045] like Figure 2 As shown, the data lineage analysis performed on the corresponding sub-target data in this embodiment to obtain the data sub-lineage of the target data may include the following steps:

[0046] Step S201: Based on the lexical file, grammar file, and parser generation tool, generate the lexical analyzer and parser corresponding to the engine identifier.

[0047] Specifically, based on the lexical and grammar files corresponding to the engine identifier generated in step S103, and the syntax analysis generation tool, a lexical analyzer and a syntax analyzer corresponding to the engine identifier can be generated. The syntax analyzer generation tool refers to a language recognition tool that can generate a lexical analyzer and a syntax analyzer based on given lexical and grammar rules to handle various grammar types, including context-free grammars.

[0048] Optionally, a lexical file is used to define lexical rules, i.e., how to divide the input character sequence into different lexical units (tokens). Optionally, lexical files typically have a .g4 extension and contain the definitions of lexical rules. Optionally, the basic structure of a lexical file includes: import statements, lexical rules, special lexical rules, and lexical modes. Import statements are used to import other lexical files; lexical rules define how to identify different lexical units. Each lexical rule consists of an identifier and a regular expression used to match the input character sequence. For example, if the target data is a simple parsing expression, the syntax rules might include: "NUMBER:[0-9]+; PLUS:'+'; MINUS:'-'; MULTIPLY:'*'; DIVIDE:' / ';"; special rules include whitespace rules, comment rules, etc.; lexical modes define different lexical modes, each of which can have different lexical rules. Optionally, the syntax file is used to define syntax rules and semantic actions. For example, if the target data is a simple parsed expression, the defined syntax rules are expression rules, which include addition, subtraction, multiplication, and division operations. For example, in the rule `expression:term|expression PLUS term|expression MINUSterm|expression MULTIPLY term|expression DIVIDE term; term:NUMBER;`, the `expression` rule represents an expression, which can be a term (a number), an expression plus a term, or an expression minus a term, and so on. The semantic actions following each rule can be used to handle field-level lineage.

[0049] Specifically, firstly, a lexical analyzer corresponding to the engine identifier can be generated based on the lexical file and the parser generation tool. Secondly, a parser corresponding to the engine identifier can be generated based on the syntax file and the parser generation tool.

[0050] Step S202: According to the lexical analyzer, the characters of the sub-target data are divided into lexical units.

[0051] Specifically, based on the lexical analyzer generated in step S201, the characters of the target data can be divided into lexical units. Optionally, the target data includes: system-level data, table-level data, and field-level data. Optionally, the characters of the sub-target data can be divided into lexical units, including: dividing the characters of the sub-target data into field-level lexical units to achieve the analysis of the data lineage of the target data at the field level.

[0052] Step S203: Generate a syntax tree for the sub-target data based on the parser and the sub-target data that divides characters into lexical units.

[0053] Specifically, a syntax tree for sub-target data can be generated based on the lexical analyzer generated in step S201 and the target data in step S202, which divides characters into lexical units.

[0054] Step S204: Traverse the syntax tree using the visitor pattern to obtain the sub-data lineage of the target data. The visitor pattern is a traversal pattern that allows for customization of the access method type as needed.

[0055] Specifically, the syntax tree of the sub-target data produced in step S203 can be traversed using the visitor pattern to obtain the sub-data lineage of the target data. Optionally, the data sub-lineage is a field-level data lineage reflecting the relationship between different fields. The visitor pattern is a traversal pattern whose access method type can be customized as needed. Optionally, the visitor pattern can also customize the number of traversals as needed.

[0056] This embodiment provides a process for performing data lineage analysis on corresponding sub-target data to obtain the data sub-lineage of target data. This involves generating a lexical analyzer and a syntax analyzer corresponding to the engine identifier based on lexical files, syntax files, and a parser generation tool. Using the lexical analyzer, the characters of the sub-target data are divided into lexical units. Based on the syntax analyzer and the sub-target data with lexical units, a syntax tree for the sub-target data is generated. The syntax tree is traversed using a visitor pattern to obtain the sub-data lineage of the target data. The visitor pattern allows for customization of the access method type. The method of generating lexical analyzers and syntax analyzers for each sub-target data using a parser generation tool and the lexical and syntax files corresponding to each engine identifier improves the accuracy of the syntax tree generation, further enhancing the accuracy of the cross-engine data lineage generation method. The visitor pattern, which allows for customization of the access method type, further improves the accuracy of the generated sub-data lineage and enhances the accuracy of the cross-engine data lineage generation method.

[0057] Figure 3 This is a flowchart illustrating the process of traversing the syntax tree using the visitor pattern to obtain the sub-data lineage of the target data, as provided in the third embodiment of this application. Figure 2 Based on the illustrated embodiment, the syntax file includes syntax rules. This embodiment elaborates on the process of traversing the syntax tree through the visitor pattern to obtain the sub-data lineage of the target data.

[0058] like Figure 3 As shown, the process of traversing the syntax tree using the visitor pattern in this embodiment to obtain the subdata lineage of the target data may include the following steps:

[0059] Step S301: Determine the access method type of the visitor pattern according to the syntax rules. The syntax rules include multiple sub-syntax rules.

[0060] Specifically, the access method type of the visitor pattern described in step S204 can be determined based on the syntax rules included in the syntax file. The syntax rules in the syntax file include multiple sub-syntax rules, that is, the visitor method type of the visitor pattern includes multiple sub-visitor method types corresponding to the sub-syntax rules. The determined access method will be called when traversing the syntax tree.

[0061] Step S302: Traverse the syntax tree according to the access method type to obtain the data sub-lineage of the target data.

[0062] Specifically, based on the access method type determined in step S302, the syntax tree generated in step S203 can be traversed to obtain the data sub-lineage of the target data.

[0063] Specifically, firstly, the combination of syntax tree nodes corresponding to each sub-syntax rule can be determined according to the syntax rules. Each node combination includes a current node and multiple child nodes. Secondly, the corresponding syntax tree node combination can be accessed according to the access method to obtain the data sub-lineage relationship corresponding to the syntax tree node combination. Here, the access method is the access method corresponding to the sub-access method type described in step S301, and the syntax tree node combination is the corresponding syntax tree node combination determined according to the corresponding sub-syntax rule. Finally, all syntax tree nodes can be traversed recursively to obtain the data sub-lineage relationship of the target data. The data sub-lineage relationship of the target data includes the data sub-lineage relationships corresponding to all syntax tree node combinations.

[0064] The recursive approach allows for a deep traversal of the entire syntax tree, from left to right and from inner to outer layers, to identify databases, system tables, table aliases, field names, field processing logic, field aliases, and the mapping relationships between them. As seen above, the lineage network identification requires more than just left-to-right, inner-to-outer layer analysis (parse the innermost structure first, then progressively outwards); it also needs to identify databases, system tables, table aliases, field names, field processing logic, field aliases, and the mapping relationships between them. Optionally, the recursive traversal can include: first, determining the data structure design (<level key, LogicField collection>) for each layer from inner to outer, such as 1, 2, 3, etc., where each level is a logical field object (which can correspond to multiple fields), and each logical field object contains multiple field objects. Secondly, the system iterates through the fields at each level, storing the field alias, source table (including subqueries or others), database, and cluster of the table into a Field object, forming a collection of field objects. Next, it iterates through the list of Field objects contained within the LogicField (which contains multiple Field objects). Finally, it iterates through the Field objects of the recursive LogicField objects within the hierarchical relationship from the innermost layer, continuously accumulating and processing logic upwards to trace back to the target field—that is, flattening the logic from the innermost field to the target field. The difference between a LogicField object and a Field object is that they are at the same level, with one LogicField corresponding to multiple Field objects. Field object 1: Field 1. The existence of cross-level relationships between LogicField objects and Field objects only occurs due to different calling levels. A LogicField object is a collection of Field objects, containing multiple Field objects (processed from multiple databases and tables). The LogicField object iterates through and stores the fields, table names, table aliases, and database names of multiple Field objects in a list. The Field object is the finest-grained single field object, containing attributes such as field alias, field name, table name, table alias, database name, whether it is a subquery, and whether it has a clear database and table source (some fields do not have table aliases or table references, and need to be matched in the metadata).

[0065] Optional, in Figure 1 or Figure 2 or Figure 3Based on any of the embodiments shown, after generating the global data lineage of the target data, the data lineage can be stored in a graph database. Graph databases are novel databases implemented based on graph theory, adept at handling complex relationship networks composed of vertices and edges, and offer unparalleled advantages in execution and query efficiency compared to traditional relational databases. Optionally, this embodiment does not limit the type of graph database.

[0066] This embodiment provides a method for traversing a syntax tree using a visitor pattern to obtain the sub-data lineage of the target data. It determines the access method type of the visitor pattern based on syntax rules, which include multiple sub-syntax rules. By traversing the syntax tree according to the access method type, the sub-data lineage of the target data is obtained. Specifically, by using different syntax rules included in different syntax files corresponding to different engine identifiers, different access method types of the visitor pattern are determined to traverse the syntax trees generated for the sub-target data corresponding to different engine identifiers. This improves the accuracy of the syntax tree traversal and further enhances the accuracy of the cross-engine data lineage generation method.

[0067] Figure 4 This is a schematic diagram of a cross-engine data lineage generation device provided in the fourth embodiment of this application.

[0068] like Figure 4 As shown, the cross-engine data lineage generation device 40 of this embodiment includes an acquisition module 41, a determination module 42, and a generation module 43.

[0069] The acquisition module 41 is used to acquire the target data and the engine identifiers of multiple engines in the data platform that perform data processing on the target data.

[0070] The determination module 42 is used to determine the sub-target data corresponding to each engine identifier in the target data.

[0071] The generation module 43 is used to generate corresponding lexical files and syntax files based on the engine identifier. The lexical files and syntax files are used to perform data lineage analysis on the corresponding sub-target data to obtain the data sub-lineage.

[0072] The generation module 43 is also used to integrate the data sub-lineage relationships obtained from multiple engine identifiers to generate the global data lineage relationship of the target data.

[0073] The apparatus provided in this embodiment can be used to execute the above-described method embodiments. Figures 1 to 3 The technical solution is similar in principle and effect, and will not be described again in this embodiment.

[0074] Figure 5This is a schematic diagram of a cross-engine data lineage generation device provided in the fifth embodiment of this application.

[0075] like Figure 5 As shown, the cross-engine data lineage generation device 50 of this embodiment includes: a processor 51, a memory 52, and a communication interface 53.

[0076] Memory 52 is used to store the processor's executable instructions;

[0077] The processor 51 is configured to execute the above method embodiments by executing executable instructions. Figures 1 to 3 A method for generating cross-engine data lineage for any item.

[0078] In the above Figure 5 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0079] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0080] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0081] This application also provides a readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, performs the above-described method embodiments. Figures 1 to 3A method for generating cross-engine data lineage for any item.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0083] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0084] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for generating data lineage across engines, the method comprising: receiving a data lineage request; identifying a data lineage graph for the data lineage request; and generating the data lineage graph for the data lineage request. The method comprises: acquiring target data and engine identifiers of multiple engines in a data platform performing data processing on the target data; the engines refer to components in which the target data flows in a data processing process; determining sub-target data corresponding to each of the engine identifiers in the target data; generating corresponding lexical files and syntax files according to the engine identifiers, the lexical files and the syntax files being used for data blood relationship analysis on the corresponding sub-target data to obtain data sub-blood relationships; the lexical files are used for defining lexical rules, and the syntax files are used for defining syntax rules and semantic actions; integrating the data sub-blood relationships obtained according to the multiple engine identifiers to generate a global data blood relationship of the target data.

2. The method of claim 1, wherein, The basic structure of the lexical file comprises: an import statement, lexical rules, special lexical rules, and a lexical pattern.

3. The method of claim 2, wherein, The method further comprises: generating a lexical analyzer corresponding to the engine identifier according to the lexical file and a syntax analyzer generation tool; generating a syntax analyzer corresponding to the engine identifier according to the syntax file and the syntax analyzer generation tool.

4. The method of claim 3, wherein, The data blood relationship analysis on the corresponding sub-target data to obtain the data sub-blood relationships of the target data comprises: dividing characters of the sub-target data into lexical units according to the lexical analyzer; generating a syntax tree of the sub-target data according to the syntax analyzer and the target data in which characters are divided into the lexical units; obtaining the data sub-blood relationships of the target data by traversing the syntax tree through a visitor pattern, the visitor pattern being a traversal pattern in which an access method type can be defined according to requirements.

5. The method of claim 4, wherein, The syntax file comprises syntax rules, and the traversal of the syntax tree through the visitor pattern to obtain the data sub-blood relationships of the target data comprises: determining an access method type of the visitor pattern according to the syntax rules, the syntax rules comprising multiple sub-syntax rules; traversing the syntax tree according to the access method type to obtain the data sub-blood relationships of the target data.

6. The method of claim 5, wherein, The traversal of the syntax tree according to the access method type comprises: determining a syntax tree node combination corresponding to each of the sub-syntax rules according to the syntax rules, the node combination comprising a current node and multiple child nodes; accessing the corresponding syntax tree node combination according to the access method to obtain data sub-blood relationships corresponding to the syntax tree node combination; traversing all the syntax tree nodes through a recursive mode to obtain the data sub-blood relationships of the target data, the data sub-blood relationships of the target data comprising data sub-blood relationships corresponding to all the syntax tree node combinations.

7. The method of claim 6, wherein, The target data comprises system-level data, table item-level data, and field-level data; and the division of characters of the sub-target data into lexical units comprises: dividing characters of the sub-target data into field-level lexical units, and the data sub-blood relationships being field-level data blood relationships reflecting relationships between different fields.

8. The method of claim 7, wherein, The method further comprises: The data lineage is stored into a graph database.

9. An apparatus for generating data lineage across engines, the apparatus comprising: a data lineage generation module configured to generate data lineage across engines. The method comprises the steps of: an acquisition module, configured to acquire target data and engine identifiers of a plurality of engines in a data platform for performing data processing on the target data; the engine refers to a component in which the target data flows in a data processing process; a determination module, configured to determine, in the target data, sub-target data corresponding to each of the engine identifiers; a generation module, configured to generate corresponding lexical files and syntax files according to the engine identifiers, the lexical files and the syntax files being used for data lineage analysis on the corresponding sub-target data to obtain data sub-lineage; the lexical files are used for defining lexical rules, and the syntax files are used for defining syntax rules and semantic actions; the generation module is further configured to integrate the data sub-lineage obtained according to the plurality of engine identifiers to generate a global data lineage of the target data.

10. A data lineage generation device across engines, the device comprising: a data lineage generation engine configured to generate a data lineage for a data flow; and a data lineage storage configured to store the data lineage. The method comprises the steps of: a processor, a memory, and a communication interface; the memory is configured to store executable instructions of the processor; wherein the processor is configured to execute the executable instructions to perform the cross-engine data lineage generation method according to any one of claims 1 to 8.

11. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the cross-engine data lineage generation method according to any one of claims 1 to 8.

Images