Spark engine-based static parsing method and apparatus for data lineage, medium, and product

Abstract

The present invention provides a Spark engine-based static parsing method for data lineage, comprising: replacing logical plan nodes with virtual nodes that only read and write metadata, and registering a listener on a Spark engine, to achieve static parsing of data lineage with zero intrusion into service code. In addition, a logical node replacement rule can be added by means of a rule interface of the Spark engine, thereby achieving relative stability of the static parsing method during Spark engine version updates.

Description

A Spark engine-based data lineage static analysis method, device, medium and product Technical Field

[0001] The present invention relates to the field of computer data processing, and in particular to a Spark engine-based data lineage static analysis method and device, medium and product. Background Art

[0002] In the age of intelligence, the demand for data processing is increasing. Accurately tracking and tracing data can effectively identify data issues in big data systems and improve data quality. Data lineage technology is a method for tracking and recording the flow and changes of data throughout its lifecycle. Currently, the following methods are commonly used to generate data lineage:

[0003] (1) Manually inputting data lineage: Data developers write documents to record the lineage relationships between data tables, conversion logic, and other information. Data entities and their lineage relationships are manually created and managed through drag-and-drop operations and form input. This method relies on manual operations, is prone to errors and omissions, and is difficult to maintain and update.

[0004] (2) Leveraging the logging and auditing capabilities of data warehouses: Some data warehouses and database management systems (such as Snowflake, Redshift, and MySQL) provide logging and auditing capabilities that can record data operations and change histories. These logs and audit records can also be analyzed by scripting or tools to extract data lineage information. However, this approach is limited to specific systems that support logging and auditing capabilities and has limited compatibility.

[0005] (3) Using full-link monitoring, a traceID is transparently transmitted from the business layer to identify data lineage. However, the disadvantage of this method is that it intrudes into the business code.

[0006] (4) Construct an abstract syntax tree for the business code and analyze it. However, the disadvantage of this method is that the business code must be visible, and the compiled project needs to be decompiled, and it needs to be adapted to multiple programming languages. In addition, the business code may contain a large amount of logic that is not related to data lineage and needs to be excluded, making this method very complex to implement. Summary of the Invention

[0007] To address the above-mentioned shortcomings, the present invention provides a method and device, medium and product for static data lineage analysis based on the Spark engine, which aims to achieve efficient and convenient data lineage analysis without invading the business code of the target task. At the same time, it can avoid reading and writing "real" data other than metadata, save the expenses required for the data lineage analysis process, and greatly shorten the time required for the data lineage analysis process.

[0008] In order to achieve the above object, a technical solution adopted by the present invention is: a data lineage static analysis method based on the Spark engine, the static analysis method comprising:

[0009] Step S1: receiving a target task, wherein the target task is a table-based data processing task;

[0010] Step S2: registering a listener on the Spark engine;

[0011] Step S3: Add a logical plan node replacement rule in the Spark engine's parser, wherein the logical plan node replacement rule is used to replace each logical plan node involving at least one of data reading, creation, writing, or deletion with a corresponding virtual node, and is set to be executed after the parsing phase of the parser is completed.

[0012] The virtual node only reads and / or creates metadata when performing read and create operations, and only writes and / or deletes metadata other than the metadata of the original data source of the target task when performing write and delete operations. Each virtual node has the same impact on data lineage as the logical plan node it replaces.

[0013] Step S4: starting execution of the target task on the Spark engine;

[0014] Step S5: extracting blood relationship related information based on the logical plan events obtained by the listener, constructing a table relationship tree, an attribute tree, and optionally a function tree;

[0015] Step S6: traversing the table relationship tree, the attribute tree, and optionally traversing the function tree when the function tree has been constructed, to establish data lineage of the target task, where the data lineage of the target task is used to reflect the lineage relationship between the tables and / or fields involved in the target task;

[0016] Among them, step S2 and step S3 can be exchanged in order, and either one of them can be advanced before step S1.

[0017] In one embodiment, each virtual node is obtained by rewriting the code of the logical plan node it replaces, and the rewriting conditions include:

[0018] Table read statements are rewritten to return only the metadata of the table being read.

[0019] The table creation statement is rewritten to create only the metadata of the table that should be created when it is needed.

[0020] The table write statement is rewritten to write only the metadata that would have been written over the metadata created based on the rewritten table creation statement, and

[0021] A table delete statement is rewritten to delete only the metadata created based on the rewritten table create statement.

[0022] In one embodiment, the rewriting condition of each virtual node also includes a new storage path record statement corresponding to each rewritten table creation statement, and the storage path record statement records the storage path of all metadata created in this virtual node based on the rewritten table creation statement; the static parsing method also includes step S7: deleting all metadata under the storage path recorded based on the storage path statement, and step S7 is after step S5 or step S6.

[0023] In one embodiment, the logical plan node replacement rule replaces at least one, two, three, four, five, or all six of the following logical plan nodes:

[0024] (a)CreateDataSourceTableAsSelectCommand;

[0025] (b)CreateHiveTableAsSelectCommand;

[0026] (c)OptimizedCreateHiveTableAsSelectCommand;

[0027] (d)InsertIntoHadoopFsRelationCommand;

[0028] (e)InsertIntoHiveTable; and

[0029] (f)DropTableCommand.

[0030] In one embodiment, in step S3, the logical plan node replacement rule is added through the injectPostHocResolutionRule interface under the SparkSessionExtensions function of the Spark engine.

[0031] In one embodiment, the established data lineage is stored in a graph database.

[0032] Another aspect of the present invention provides a method for automatically notifying task dependency configuration errors, comprising obtaining a data lineage relationship diagram of a target task based on the static parsing method of the aforementioned embodiment, comparing the dependencies of the target task's current configuration, and notifying the differences;

[0033] The differences include: the upstream related task displayed in the data lineage relationship diagram is not configured, the data lineage relationship diagram does not display any upstream related task, or the data lineage relationship diagram does not display a configured upstream related task.

[0034] Another aspect of the present invention provides a computer device / equipment / system, which is used to implement the steps of the method described in one of the aforementioned embodiments. The computer device / equipment / system includes a task execution module, a lineage information collection module, a logical node replacement module, and a lineage information analysis module.

[0035] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the method described in one of the aforementioned embodiments are implemented.

[0036] Another aspect of the present invention provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in one of the aforementioned embodiments are implemented.

[0037] Compared with the prior art, the advantage of the present invention is that it can achieve efficient and convenient data lineage analysis without intruding the business code of the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] FIG1 is a flow chart of a Spark engine-based data lineage static analysis method according to an embodiment of the present invention;

[0040] FIG2 is a flowchart of a method for automatically prompting task dependency configuration errors provided by another embodiment of the present invention.

[0041] FIG3 : Another embodiment of the present invention provides a data lineage static analysis device based on the Spark engine. DETAILED DESCRIPTION

[0042] Unless otherwise specified, all or some of the steps, devices, systems, equipment, and modules in the methods disclosed herein may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0043] Where appropriate, the component used to implement all or some steps or achieve specific functions may be a processor (such as a central processing unit, a digital signal processor or a microprocessor) that executes corresponding instructions, an integrated circuit (such as an application-specific integrated circuit) or other physical component.

[0044] Where appropriate, such software or instructions may be deployed on a computer-readable medium, which typically includes RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be accessed by a computer.

[0045] Where appropriate or necessary, communication may be required within or between components implementing the methods disclosed herein, which may be accomplished via communication media. Such communication media typically include computer-readable signals, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] As used herein, "at least one" means one or more than one.

[0048] In this article, "first", "second", etc. are only used to distinguish different objects, rather than to describe a specific order.

[0049] In this document, terms such as "include", "comprising", "containing" and "having" are open-ended and indicate the presence of the described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0050] In this article, "Spark engine" refers to an open source big data processing engine development project initiated by the AMP Laboratory of the University of California, Berkeley; in terms of applicable versions, for example, the open source project spark with version number 3.3.0 uploaded by the Apache Spark community on the GitHub platform on June 16, 2022; other versions or Spark engines further developed based on the aforementioned versions may also be applicable to the present invention without changing the structure, function or operation mode of the necessary components / modules involved in the present invention.

[0051] In this document, "data source" refers to a source or carrier that provides data or information, including an accessible dataset, database, file, sensor, web service or other data provider.

[0052] In this article, "dependency" refers to the relationship between a task or operation and other tasks, where the execution or completion of one task depends on the status, results, or execution order of other tasks; common dependencies include task-based dependencies, time window-based dependencies, state-based dependencies, etc.

[0053] In this article, "metadata" refers to structured data specifically used to describe the characteristics and properties of certain data or resources.

[0054] FIG1 is a Spark engine-based data lineage static parsing method provided by an embodiment of the present invention. The static parsing method includes: Step S1: receiving a target task, where the target task is a table-based data processing task.

[0055] In line with the static parsing of data lineage based on the Spark engine, those skilled in the art will understand that the "target task" is and should be a task that can be parsed using the Spark engine. Therefore, the target task should use a language that can be parsed by the Spark engine (i.e., a development language supported by the Spark engine). For example, a task programmed in the scala language. Another example is a task programmed in one of the languages ​​​​such as sql, java, and python. In addition, it should be meaningful to obtain data lineage for the "target task". Specifically, the target task is a table-based data processing task. For example, the target task can be selected from but not limited to table-based data analysis tasks and data import / export tasks.

[0056] In order to obtain the data lineage of the target task through static parsing, it is necessary to receive the target task. In addition, in some cases, in order to complete the static parsing of the data lineage during the (virtual) execution of the target task by the Spark engine of this embodiment, it may be necessary to add another data source or another data / code package that solidifies the static parsing method. These can also be considered as other dependencies necessary for static parsing of the target task.

[0057] The reception of the target task and, in some cases, the addition of other dependencies necessary for the static parsing method to the target task can all be completed by the task execution module. Unless otherwise specified, the task execution module, lineage information collection module, logical node replacement module, and lineage information parsing module introduced below can all be individually or optionally combined into one of the following or a combination thereof: a computer program / instruction, a computer-readable storage medium storing corresponding instructions, a processor that executes corresponding instructions, an integrated circuit or other physical component, or a computer program product containing corresponding instructions. The reception of the target task and / or the addition of dependencies can be automatically completed by the task execution module, or semi-automatically completed by the task execution module after advance configuration by the program developer / user (user). Under appropriate circumstances, before each static parsing, the user can also cooperate to complete the reception of the target task by uploading / configuring one by one, and / or configure / add each dependency one by one by the user.

[0058] Before a target task is accepted, it may go through other steps. For example, a target task developed in Scala undergoes steps such as compilation, packaging and uploading, receiving the package file path, configuring execution parameters, and publishing before being accepted by the task execution module.

[0059] The static parsing method of this embodiment further includes step S2: registering a listener on the Spark engine.

[0060] A listener is an event monitoring mechanism that can be registered on the Spark engine. The "listener" specifically referred to in this article can monitor events generated by the Spark engine, including events containing logical plans (referred to as "logical plan events" in this article). Specifically, the listener used in this article can monitor events containing query plans (called QueryExecution in Spark), and the "query plan" includes a complete parsed logical plan (also called a "logical execution plan"), and the event containing the query plan is the logical plan event. Registering a listener on the Spark engine allows data lineage to be established in subsequent steps based on the logical plan events obtained through monitoring.

[0061] Listeners can be manually written, configured, and registered on the Spark engine by users based on custom listening, or registered by the lineage information collection module. The code and registration methods for listeners that can monitor logical plan events generated by the Spark engine are standard, for example, as disclosed in the official Spark documentation at the following link: https: / / spark.apache.org / docs / 3.3.0 / api / java / org / apache / spark / sql / util / QueryExecutionListener.html. Users can use these code and methods, or make non-substantial modifications to them, to implement their monitoring of logical plan events.

[0062] By obtaining logical plan events through listeners and then using them for static analysis to obtain data lineage, we can overcome the defect that the Spark engine itself cannot directly output logical execution plans and provide a new static analysis method for data lineage based on the Spark engine.

[0063] The static parsing method of this embodiment also includes step S3: adding a logical plan node replacement rule in the parser of the Spark engine, wherein the logical plan node replacement rule is used to replace each logical plan node involving at least one of data reading, creation, writing or deletion with a corresponding virtual node, and is set to be executed after the parsing phase of the parser is completed. When performing read and create operations, the virtual node only reads and / or creates metadata, and when performing write and delete operations, only writes and / or deletes metadata other than the metadata of the original data source of the target task, and each of the virtual nodes has the same impact on data lineage as the logical plan node it replaces.

[0064] The parser is one of the necessary modules / processes / stages for executing tasks in the Spark engine. The parser in the Spark engine will first add a corresponding logical plan node for each operation in the task. Each logical plan node represents a specific logical operation. The nodes are connected through parent-child relationships to form a complete logical execution plan with a tree structure, which is used to represent the sequence of logical operations (i.e., the dependency relationship between logical operations). Typical logical plan node types include data reading / creating / writing / deleting nodes, data computing nodes, and type conversion nodes. Among them, logical plan nodes involving at least one of data reading, creating, writing or deleting include but are not limited to CreateDataSourceTableAsSelectCommand; CreateHiveTableAsSelectCommand; OptimizedCreateHiveTableAsSelectCommand; InsertIntoHadoopFsRelationCommand; InsertIntoHiveTable; and DropTableCommand.

[0065] For the avoidance of doubt, unless otherwise specified, in this document, the term "write" is used in its narrow sense, referring to the operation of overwriting existing similar data with new data at a specific storage location, thereby replacing the meaning of the existing similar data with the meaning of the new data. Although they may all involve the overwriting of data, operations such as "create" and "delete" data in this document do not replace the meaning of existing similar data. Instead, they treat the data at the existing storage location as meaningless, regardless of whether it is similar to the new data. Therefore, in this document, they are not considered to be subordinate concepts of "write" data, but rather concepts parallel to "write" data. In such cases, the meaning of operations such as "read," "create," and "delete" data are self-explanatory.

[0066] The analyzer is also an essential module, process, or phase for executing tasks in the Spark engine. It uses predefined rules and other information to transform the unresolved logical plan generated by the parser into a resolved logical plan. This phase is also known as the "parsing phase." Subsequent static analysis of data lineage requires the use of the resolved logical plan, which is included in the monitored logical plan events.

[0067] To achieve correct parsing, the Spark engine's parser comes with a variety of rules, including data source parsing, metadata parsing, and so on.

[0068] According to this embodiment, the logical plan node replacement rule is a custom rule applicable to the Spark engine's parser. This rule is set to be executed after the parsing phase of the parser is completed, and is used to replace each logical plan node involving at least one of data reading, creation, writing or deletion with a corresponding virtual node.

[0069] The "virtual node" referred to in this article is a node type. Each virtual node can represent a specific logical operation to build a logical execution plan tree together with other logical plan nodes or virtual nodes. However, compared with the logical plan nodes it replaces, virtual nodes do not actually perform read, create, write, or delete operations, and can therefore be considered "virtual".

[0070] For example, to replace the logical plan node CreateDataSourceTableCommand with the corresponding virtual node VirtrualCreateDataSourceTableCommand, the logical plan node replacement rule (named SparkRuler) can use the following code: case class SparkRuler(session:SparkSession) extendsRule[LogicalPlan]{ override def apply(plan:LogicalPlan):LogicalPlan=plan resolveOperators{ case CreateDataSourceTableCommand(tableDesc,ignoreIfExists)if DDLUtils.isDatasourceTable(tableDesc)=> VirtrualCreateDataSourceTableCommand(tableDesc,ignoreIfExists)

[0071] If a logical node replacement rule also replaces other logical plan nodes outside the example, its code can imitate or continue the above code and will not be repeated here.

[0072] Logical plan node replacement rules can be inserted into the rules applicable to the Spark engine's parser by rewriting the Spark engine code. They can also be inserted into the postHocResolutionRules property of the parser Analyzer to ensure that these rules are executed after the parser's parsing phase is completed. In addition, logical plan node replacement rules can also be added through the injectPostHocResolutionRule interface under the Spark engine's SparkSessionExtensions function. Rules added using this interface will be executed after the parser's parsing phase is completed. For the avoidance of doubt, since logical plan node replacement rules are rules added to the parser, they are executed before the optimizer phase after the parser.

[0073] The logical plan node replacement rule can be manually written / configured / added to the Spark engine parser by the user, or automatically added to the Spark engine parser by the logical node replacement module based on pre-set settings.

[0074] By adding custom rules to the parser to replace the previously generated logical plan nodes, static parsing of data lineage can be achieved. This can avoid direct development and intrusion into the target task code (i.e., business code) (especially avoiding the management and development of specific database fields in the business code) and eliminate the need for visibility into the business code.

[0075] By adding custom rules for logical plan node replacement to the parser and setting them to be executed after the parsing phase is completed, it can be ensured that replacing virtual nodes will not affect the generation of other nodes, thereby avoiding affecting the correctness and integrity of the data lineage obtained by static parsing.

[0076] In this embodiment, as a limitation, the virtual node only reads metadata when performing a read operation, and only creates, writes, and deletes metadata other than the metadata of the original data source of the target task when performing a create, write, and delete operation, and each virtual node has the same impact on data lineage as the logical plan node it replaces.

[0077] For example, if the logical plan node requires reading (or creating) Table A, the corresponding virtual node used for replacement only reads (or creates) and returns the metadata of Table A.

[0078] For another example, if the logical plan node requires writing (or deleting) table B, and table B is not from the original data source, the corresponding virtual node used for replacement only writes (or deletes) the metadata of table B.

[0079] For another example, if a logical plan node requires writing (or deleting) table C, and table C is from the original data source (or one of the original data sources), the corresponding virtual node used for replacement can create table C' if it determines that table C' does not exist, and only write the metadata that should be written to table C to the metadata of table C' (or delete the metadata of table C' or keep it empty). At the same time, it ensures that the operations on table C in subsequent logical plan nodes are replaced by operations on table C' to avoid affecting data lineage. Since the fixed table C' is used to clearly replace table C in the original data source when table C is first written or deleted, from the user's perspective, although this replacement occurs, it can be considered to have the same impact on data lineage. Of course, table C' can use the same name as table C (if it does not affect data processing) or a similar name. If a visual display of data lineage is required later (such as a data lineage relationship diagram), table C and table C' can also be displayed together to prevent users from misinterpreting the true data lineage due to this replacement relationship.

[0080] For the avoidance of doubt, the term "original data source" does not include tables created after the target task is executed, regardless of whether the table is subsequently read, written, and / or deleted in another virtual node that replaces the logical plan node (i.e., as an intermediate table). Specifically, a scenario may exist where metadata for a first table (Table D) is created by a first virtual node, and based on the metadata of the first table (Table D) read, a second table (Table E) is created and the metadata is written by a second virtual node.

[0081] While maintaining the same impact on data lineage, replacing each logical plan node involving data read, creation, writing, and deletion with a virtual node that only reads, creates, writes, and deletes metadata can avoid processing data other than metadata (i.e., "real" data reading and writing), thereby reducing the cost and time required for static data lineage resolution. Furthermore, by further avoiding the writing and deletion of data in the original data source, the negative impact of the static data lineage resolution process on the original data source can be avoided, including negative impacts on further development and use based on the original data source.

[0082] For the avoidance of doubt, “only metadata is read and / or created when performing read and create operations, and only metadata other than the metadata of the original data source of the target task is written and / or deleted when performing write and delete operations” is only an improved limitation on the read, create, write, and delete operations in the logical plan node to be replaced, which is to avoid the reading, creation, writing, and deletion of data other than metadata and the writing and deletion of data of the original data source, rather than the limitation or exclusion of operations other than reading, creating, writing, and deleting; operations other than reading, creating, writing, and deleting (regardless of whether they originally existed in the logical plan node to be replaced) can be added, retained, modified, or deleted in the corresponding virtual node. In some scenarios, it may be beneficial to delete these other operations in the virtual node, for example, it can further save costs and shorten parsing time. Operations other than reading, creating, writing, and deleting can also be retained or added in the virtual node to achieve other effects. As in the example above, when the logical plan node requires writing to the original data source table C, the replaced virtual node needs to create table C' first in order to write the data to the replacement table C'. However, the step of creating table C' is a precursor to the writing step, not the writing step itself, and therefore does not fall within the scope limited and / or excluded by such expressions.

[0083] The static parsing method of this embodiment further includes step S4: starting execution of the target task on the Spark engine.

[0084] After the Spark engine starts executing the target task, the parser generates an unparsed logical plan, which the parser further transforms to obtain a parsed logical plan. However, before entering the optimizer stage, the parser also replaces all logical plan nodes in the parsed logical plan that involve at least one of data reading, creation, writing, or deletion with corresponding virtual nodes based on the added logical plan node replacement rules. It then enters other stages such as the optimizer. Since each logical plan node that involves reading, creating, writing, or deleting has been replaced with a virtual node, the Spark engine will only generate a logical execution plan (i.e., the parsed logical plan) during execution, and will not read, create, write, or delete data other than metadata, nor will it write or delete the original data source, that is, it will not "actually" execute the target task.

[0085] In this embodiment, the execution of the target task and the initiation of the execution may be automatically completed by the task execution module; or the user may manually initiate the execution of the target task.

[0086] The static parsing method of this embodiment further includes step S5: extracting blood relationship related information based on the logical plan events obtained by the listener, constructing a relationship tree, an attribute tree, and optionally a function tree.

[0087] Based on the listener defined in this article, during the parser phase, the listener can listen to events emitted / generated by the Spark execution engine containing query plans (i.e., logical plan events). The query plan contains the parsed logical plan (i.e., logical execution plan). Of course, depending on the specific listener used, the listener may also listen to other events besides logical plan events. However, these events do not contain query plans related to data reading and writing, and are therefore unrelated to the logical plan, and therefore cannot be used to extract lineage-related information.

[0088] Because the logical plan event obtained through listening contains the contents of the entire logical execution plan (parsed logical plan), by extracting the logical plan portion from the query plan, the logical execution plan (parsed logical plan) generated during the parser phase can be further obtained based on the logical plan event obtained through listening. In this embodiment, because the logical node replacement rule is executed during the parser phase, specific logical plan nodes in the logical execution plan (parsed logical plan) further obtained based on the logical plan event obtained through listening have been replaced with corresponding virtual nodes.

[0089] Further obtaining the logical execution plan (parsed logical plan) based on the logical plan events obtained by the listener can be completed automatically or semi-automatically by the lineage information collection module, or can be completed manually by the user.

[0090] Furthermore, this embodiment extracts lineage-related information from the logical execution plan (parsed logical plan) obtained based on monitoring, constructs a table relationship tree, an attribute tree, and optionally a function tree.

[0091] Lineage-related information refers to data (metadata) related to building data lineage, such as data source, data source type, table name, field name, etc. The table relationship tree refers to a multi-branch tree that records the mapping relationship between tables. The attribute tree refers to a multi-branch tree that records the mapping relationship between attributes and attributes, and the attributes include field information in the table. The function tree refers to a multi-branch tree that records the mapping relationship between functions and attributes, where functions include various calculation and conversion functions used in the data processing process.

[0092] Specifically, in this embodiment, to construct a table relationship tree, it may be necessary to base it on the following lineage-related information: source / destination table name, type, and storage path; to construct an attribute tree, it may be necessary to base it on the following lineage-related information: source / destination table name, attribute name, and type; to construct a function tree, it may be necessary to base it on the following lineage-related information: function id, name, and attribute id.

[0093] An example of building a table relationship tree based on blood relationship information is:

[0094] Step 1: Use topological sorting to start from table T with out-degree 0 (generally speaking, only writes and no reads), and take all tables read when writing (generally speaking, including creation and deletion) table T as its child nodes.

[0095] Step 2: If the table being read is also a write table (broadly speaking, including creation and deletion), repeat step 1 based on the read table T2 until no new writes occur.

[0096] Similarly, an example of building a function tree based on blood relationship information is:

[0097] Step 1: Through topological sorting, start from the function F with out-degree 0 (not used as input for other functions), and use all input parameters of function F as its child nodes;

[0098] Step 2: If the input parameter is also a function, repeat step 1 based on the output function F2 until no new functions appear.

[0099] Similarly, an example of building an attribute tree based on blood relationship information is:

[0100] Step 1: Based on the table relationship tree, start building based on the attribute A of each root node, and take all upstream inputs that output A as its child nodes;

[0101] Step 2: If the input is a function, traverse the function tree until the upstream attribute A2 is obtained. Repeat step 1 based on A2 until no new attributes or functions appear.

[0102] The relationship between the constructed table relationship tree, attribute tree, and function tree is: the function tree is used to assist in building the attribute tree; the table relationship tree itself can be used to build table-level lineage and also assist in building the attribute tree; the attribute tree will ultimately be used to build field-level lineage.

[0103] Because functions may not necessarily be used in tasks, the constructor tree is optional for establishing the target task's data lineage. That is, if functions (such as count and sum) are used, the constructor tree can be selected and used when establishing data lineage. Alternatively, if functions are not used, the constructor tree can be omitted and data lineage can be established solely based on the table relationship tree and attribute tree.

[0104] Extracting the bloodline-related information from the logical execution plan (parsed logical plan) obtained based on monitoring may be automatically or semi-automatically performed by the bloodline information collection module, or may be manually performed by the user.

[0105] The construction of table relationship tree, attribute tree and function tree based on blood relationship related information can be automatically completed by the blood relationship information collection module.

[0106] Compared with other ways of recording mapping relationships, using a tree format is more intuitive.

[0107] Optionally, in this embodiment, since the logical execution plan (parsed logical plan) also includes nodes that are not related to data lineage, such as type conversion, calculation, etc., which do not involve at least one of data reading or writing, before extracting lineage-related information, these nodes can be removed manually by the user or automatically by the lineage information collection module or other components to improve the efficiency and accuracy of extracting lineage-related information.

[0108] The static parsing method of this embodiment also includes step S6: traversing the table relationship tree, the attribute tree, and optionally traversing the function tree when the function tree has been constructed, to establish the data lineage of the target task, where the data lineage of the target task is used to reflect the lineage relationship between the tables and / or fields involved in the target task.

[0109] In this embodiment, by traversing the table relationship tree and attribute tree constructed in the previous example, the blood relationship between tables and tables, fields and fields involved in the target task (that is, the blood relationship between tables and / or fields involved in the target task) can be obtained.

[0110] In the case of constructing a function tree, the blood relationship between the fields involved in the target task (ie, the blood relationship between the fields involved in the target task) is obtained by using the construction of its auxiliary attribute tree.

[0111] However, as mentioned above, when no functions are used, it is not necessary to construct or traverse the function tree. The data lineage of the target task can be established by simply traversing the table relationship tree and the attribute tree. Therefore, traversing the function tree is optional for establishing data lineage, even if the function tree has been constructed in the previous steps.

[0112] Traversing the table relationship tree, the attribute tree, and the optional function tree and establishing data lineage can be automatically completed by the lineage information parsing module.

[0113] Since both registering the listener and adding the logical plan node replacement rule only take effect when the Spark engine executes the target task, and since registering the listener and adding the logical plan node replacement rule are independent of each other and independent of receiving the target task, there is no necessary precedence relationship. Therefore, in the static parsing method of this embodiment, step S2 and step S3 can be swapped in order, and either one can be advanced to step S1. For example, a feasible swapping scheme is to swap the order of steps S2 and S3 and advance both to step S1. According to this scheme, the order before step S4 after adjustment is step S3 first, then step S2, and then step S1.

[0114] In a static parsing method provided in an optional embodiment, each virtual node is obtained by code rewriting based on the logical plan node it replaces, and the rewriting conditions include: a table read statement is rewritten to only return metadata of the table being read, a table creation statement is rewritten to only create metadata of the table that should be created when creation is required, a table write statement is rewritten to only write metadata that should be written on metadata created based on the rewritten table creation statement, and a table delete statement is rewritten to only delete metadata created based on the rewritten table creation statement.

[0115] Rewrite conditions refer to the rules that need to be applied when rewriting each statement in the code. It can be understood that when applying the rewrite conditions, each rule in the rewrite conditions needs to be applied one by one, including (when the rule sets the application scope or application object) judging whether the application object meets the application scope of the rule (whether it belongs to the application object of the rule), and when it falls into the application scope (application object) (such as the application scope or application object is set), executing the rewrite requirements in the rule; when it falls into the application scope (application object) of multiple specific rules, the rewrite requirements in each of the rules should be executed. Since the code of a logical plan node (virtual node) can include table read statements, table creation statements, table write statements, table deletion statements and other types of statements, therefore, when applying the rewrite conditions to rewrite the code of the logical plan node into a virtual node, (when the rewrite scope and object are set) it is necessary to judge whether each statement in the code of the logical plan node to be replaced belongs to the statement range and statement object that need to be rewritten according to the rules in the rewrite conditions, and execute the corresponding statement rewrite requirements.

[0116] For example, if the logical plan node to be replaced includes both a table read statement and a table creation statement, when applying the rewrite condition to rewrite it into a virtual node, the table read statement needs to be rewritten to only return the metadata of the read table, and the table creation statement needs to be rewritten to only create the metadata of the table that should be created when creation is required.

[0117] For another example, if the logical plan node to be replaced includes a statement that is both a table read statement and a table creation statement, when applying the rewrite condition to rewrite it into a virtual node, the part that implements the table read should be rewritten to only return the metadata of the read table, and the part that implements the table creation should be rewritten to only create the metadata of the table that should be created when creation is required, so as to meet the rewrite condition.

[0118] Table read statements, table create statements, table write statements, and table delete statements refer to statements in the code of a logical plan node (virtual node) that implement table read functions, table create functions, table write functions, and table delete functions, respectively. Although create and delete operations are a special form of write operations, as defined above, the functions implemented by table write statements only include other table write functions besides table creation and deletion. In the code of a node, a code segment that cannot be further divided and implements a single read, create, write, delete, etc. function is generally a statement; multiple adjacent or non-adjacent paragraphs that jointly implement a single function can be regarded as a statement.

[0119] In this embodiment, the table read statement is rewritten to return only the metadata of the table being read. The specific example is the same as the example of Table A above and will not be repeated here.

[0120] In this embodiment, table creation statements are rewritten to create only the metadata of the table that should have been created when creation is required. Table write statements are rewritten to write only the metadata that should have been written to the metadata created based on the rewritten table creation statement. Table delete statements are rewritten to delete only the metadata created based on the rewritten table creation statement. It is clear to those skilled in the art which situations fall under the "need to create" situation. One situation is when the logical plan node to be replaced already requires table creation (including creation for the purpose of writing corresponding data into the newly created table) or requires table creation under certain conditions.

[0121] For example, if the logical plan node to be replaced requires creating Table F using the data from Table A, the corresponding statement is rewritten to use the metadata from Table A to create Table F, which only includes the metadata (that is, only the metadata of the table that should have been created is created). Even if the "using the data from Table A" portion is understood as a table read statement, rewriting "using the data from Table A" to "using the metadata from Table A" also meets the requirement of rewriting the table read statement to return only the metadata of the table being read.

[0122] In addition, since this embodiment requires that the virtual node only write and / or delete metadata other than the metadata of the original data source of the target task when performing write and delete operations, when the table write (or delete) statement in the logical plan node to be replaced indicates that the original data source needs to be written (or deleted), consistent with the above description, (when there is no such table) creating a table that only includes metadata to replace the original data source for data writing (or deletion) is also a "need to create" situation; in this case, the table write (or delete) statement is rewritten so that, if it is determined that a table to replace the original data source has not been created, a replacement table is first created based on the rewritten table creation statement, and then only the metadata of the data that should be written to the newly created replacement table is written (or the metadata of the newly created replacement table is deleted); if it is determined that a table to replace the original data source has been created, since the table was created based on the rewritten table creation statement, only the metadata of the data that should be written to the metadata of the table needs to be written (or the metadata of the table needs to be deleted).

[0123] An example is similar to the example of Table C above: the logical plan node to be replaced requires writing (or deleting) Table C, and Table C is the original data source. In this case, the rewritten table write (or delete) statement in the corresponding virtual node used for replacement creates Table C' if it is determined that Table C' has not been created, and only the metadata that should have been written to Table C is written to the metadata of Table C' (or the metadata of Table C' is deleted). If it is determined that Table C' has already been created, there is no need to create it again, and only the metadata that should have been written to Table C is written to the metadata of Table C' (or the metadata of Table C' is deleted).

[0124] When rewriting table write and delete statements, the "rewritten table creation statement" based on the object table / metadata to be written and deleted after the rewrite can be used in various situations. For example, if the logical plan node replaced by this virtual node (or a previous virtual node) includes a table creation statement, the original table creation statement needs to be rewritten based on the rewrite conditions. In this case, the rewrite results in one case of "rewritten table creation statement." For another example, the logical plan node replaced by this virtual node (or other virtual nodes before) indicates that the original data source needs to be written (constituting a table write statement), but under the restriction that the original data source cannot be written, it is judged that no table has been created to replace the original data source, so it meets the situation of "needing to create" a replacement table, and then a new statement is added to create a table to replace the original data source with the metadata of the original data source (that is, only the metadata of the table that should be created is created), and then the original table write statement is modified to obtain a statement for writing the metadata of the original table write statement that needs to be written to the original data source; the statement used to create a table to replace the original data source here meets the requirement of "only creating the metadata of the table that should be created when creation is needed", and therefore also belongs to a situation of "rewritten table creation statement".

[0125] That is, the expression "rewritten table creation statement" used when rewriting table write statements and table delete statements in the rewrite conditions includes both statements for table creation that are rewritten based on the rewrite conditions because the code of a logical plan node to be replaced includes a table creation statement; and a statement for table creation that meets the requirement of "only creating metadata of the table that should be created when creation is needed", without the requirement that the existence of this statement must mean the existence of a prerequisite for the application of the rewrite rule (that is, the code of the logical plan node to be replaced includes a table creation statement).

[0126] In addition, metadata / table that is "created based on the rewritten table creation statement" means that the metadata / table is created based on the "rewritten table creation statement" during the initial creation step, regardless of whether the metadata / table has undergone one or more other operations such as reading, writing, or deletion, or a combination thereof, after being initially created based on the "rewritten table creation statement".

[0127] Of course, when the table write (or delete) statement in the logical plan node to be replaced indicates that other tables outside the original data source need to be written (or deleted), since the tables to be written (or deleted) must have been created based on the rewritten table creation statement, the rewritten table write (or delete) statement can be directly rewritten to write the metadata of the data that should have been written to the tables to be written (or delete the corresponding metadata of the table).

[0128] An example is the same as the example of Table B above: the logical plan node to be replaced requires writing (or deleting) Table B, and Table B is not the original data source. In this case, the corresponding virtual node used for replacement only writes the metadata of the data that should have been written to the metadata of Table B (or only deletes the metadata of Table B), regardless of whether Table B has been read, written, and / or deleted since it was created.

[0129] The code of the logical plan node to be replaced may or may not include other types of statements besides table read statements, table creation statements, table write statements, and table delete statements. Typical other types of statements include, but are not limited to, statements used for conditional judgments and statements for pruning based on conditional judgments. Similar to the above, for the avoidance of doubt, "table read statements are rewritten to only return the metadata of the table being read, table creation statements are rewritten to only create the metadata of the table that should be created when creation is required, table write statements are rewritten to only write the metadata that should be written on the metadata created based on the rewritten table creation statement, and table delete statements are rewritten to only delete the metadata created based on the rewritten table creation statement" is only a rewrite restriction for table read statements, table creation statements, table write statements, and table delete statements in the logical plan node to be replaced, which is to avoid the reading, creation, writing, and deletion of data other than metadata, rather than a restriction or exclusion of statements other than table read statements, table creation statements, table write statements, and table delete statements.

[0130] Those skilled in the art will understand that, under the premise of ensuring that the logical execution plan (parsed logical plan) after being replaced with a virtual node can be statically parsed to obtain the correct data lineage, other types of statements other than table read statements, table create statements, table write statements, and table delete statements (regardless of whether they originally existed in the logical plan node to be replaced) can be added, retained, modified, or deleted. When such other types of statements are necessary for the logical execution plan (parsed logical plan) after being replaced with a virtual node to be correctly parsed and the data lineage to be correctly established, such other types of statements should be added, retained, or retained after corresponding modifications when rewriting and replacing with a virtual node. Such modifications can also be easily completed by those skilled in the art based on the content of the logical plan node to be replaced. For example, when the logical plan node to be replaced includes a statement that uses ifnotExist for pruning judgment, the statement should be retained or retained after being rewritten, and when it is rewritten, it should be ensured that the return value is consistent with the rewrite condition and the semantic requirements of the logical plan node to be replaced. In addition, as in the previous example, when the logical plan node requires writing to the original data source table C through a table write statement, the replaced virtual node needs to create table C' based on the rewritten table creation statement in order to write the data to be written in the replacement table C'. The step of creating table C' based on the rewritten table creation statement is a predecessor step of the write step, not the write step itself, and therefore does not fall within the scope excluded by the limitation on the "write" operation.

[0131] Rewriting the logical plan nodes to be replaced into virtual nodes based on the rewriting conditions can be implemented by the user himself or automatically or in combination with manual review (semi-automatically) using specific modules / components.

[0132] By establishing unified virtual node writing conditions, including setting them to rewrite logical plan nodes that need to be replaced and obtaining and clarifying unified rewriting conditions, it is helpful to simplify the time and difficulty of virtual node code writing and improve the degree of automation in virtual code writing and application.

[0133] It should be noted that if the data / metadata / table that a logical plan node or table creation statement requires to be created has already been created and currently still exists, or if the data / metadata / table that a logical plan node or table deletion statement requires to be deleted has already been deleted and currently does not exist or has never existed, these are all defects in the target task itself or problems that should be solved by the target task / Spark engine itself. They are not defects in the various data lineage parsing methods themselves, and naturally do not belong to the new problems and improvement directions brought about by the static parsing method in the present invention. Various known or newly invented solutions to such problems can be combined with the static parsing method in the present invention without the need for creative labor.

[0134] In a static parsing method provided in an optional embodiment, the rewriting condition of each virtual node also includes a new storage path record statement corresponding to each rewritten table creation statement, and the storage path record statement records the storage path of all metadata created in this virtual node based on the rewritten table creation statement; the static parsing method also includes step S7: deleting all metadata under the storage path recorded based on the storage path statement, and step S7 is after step S5 or step S6.

[0135] Since, in this embodiment, based on the above, all logical plan nodes involving read, write, create, and delete operations in the logical execution plan (parsed logical plan) have been replaced with corresponding virtual nodes obtained by rewriting based on the rewrite conditions, the table creation statements in each virtual node have been rewritten, and the write locations in the table write statements in each virtual node have also been rewritten to the tables created based on the rewritten table creation statements (regardless of whether the table is newly created in this virtual node or has already been created in other virtual nodes), therefore, all read, write, and delete objects are not the paths and contents already existing in the data source itself, but are newly created paths and contents outside the data source. In such cases, new rules can be added to the rewrite conditions so that when the rewritten table creation statement appears in each virtual node, a corresponding statement can be added to record (also referred to as save) the storage path of all metadata (tables) created in this virtual node based on the rewritten table creation statement, that is, the storage path of all newly added process data (metadata) generated to complete this static parsing method can be obtained. On this basis, by adding a step of deleting all metadata under the storage path recorded based on the storage path statement in this embodiment, as step S7 after step S5 or step S6, all new process data (metadata) generated to complete this static parsing method can be deleted.

[0136] If the storage path has been recorded (and saved) when the metadata is initially created based on the rewritten table creation statement, even if such metadata is read, written, or deleted thereafter, its storage path will still be maintained. Therefore, there is no need to repeatedly add a statement to record (and save) the storage path in the subsequent rewritten table read statement, table write statement, and table delete statement.

[0137] Similar to the interpretation of "rewritten table creation statements" when rewriting table write statements, "each rewritten table creation statement" corresponding to the newly added storage path record statement not only includes the "rewritten table creation statements" in the code of the virtual node generated by rewriting because the code of the logical plan node that needs to be replaced by this virtual node (or other previous virtual nodes) already has a table creation statement; it also includes the statements (paragraphs) that are newly added when rewriting the table write (or delete) statement to create an alternative table when the alternative table of the original data source is missing because the code of the logical plan node that needs to be replaced by this virtual node (or other previous virtual nodes) has a table write (or delete) statement that requires writing (or deleting) the original data source. That is, the expression "rewritten table creation statement" used when adding a storage path record statement in the rewrite condition includes a statement for table creation that is rewritten based on the rewrite condition because the code of a logical plan node to be replaced includes a table creation statement; it also includes a statement for table creation that meets the requirement of "only creating the metadata of the table that should be created when creation is needed", without the requirement that the existence of this statement must mean the existence of a prerequisite for the application of the rewrite rule (that is, the code of the logical plan node to be replaced includes a table creation statement).

[0138] The rewriting is implemented in the same manner as above. Step S7 of deleting all metadata under the storage path recorded based on the storage path statement can be implemented by the user, or can be implemented automatically or in combination with manual review (semi-automatically) using specific modules / components.

[0139] By adding a storage path record statement corresponding to the (rewritten) table creation statement, the storage path of all newly added process data (metadata) generated to complete this static parsing method can be recorded (also referred to as saved), providing users with an intermediary for centralized management of data added based on this static parsing method, and providing a basis for batch review, analysis, modification, deletion, and other types of management of newly added data and their storage paths. In addition, by deleting all newly added data generated to complete this static parsing method according to the recorded storage path, it is possible to release all storage space occupied by process data after completing the static parsing of data lineage, reduce the system burden of this static parsing method in reuse scenarios, and conveniently meet the storage requirements of other data processing tasks when storage resources are limited.

[0140] In a static parsing method provided in an optional embodiment, the logical plan node replacement rule replaces at least any one, two, three, four, five or all six of the following logical plan nodes: CreateDataSourceTableAsSelectCommand; CreateHiveTableAsSelectCommand; OptimizedCreateHiveTableAsSelectCommand; InsertIntoHadoopFsRelationCommand; InsertIntoHiveTable; and DropTableCommand.

[0141] The logical plan nodes listed are all common logical plan nodes when Spark executes read, create, write, and delete tasks. Usually, at least one of them appears in a table-based data processing task.

[0142] The CreateDataSourceTableAsSelectCommand node handles the CTAS (Create Table by Query) statement for the specified data source. It reads data from one table for a specific field and writes it to another. A more detailed description and common code are available at the following official link: https: / / github.com / apache / spark / blob / master / sql / core / src / main / scala / org / apache / spark / sql / execution / command / createDataSourceTables.scala.

[0143] For example, the SQL code steps of a virtual node VirtualCreateDataSourceTableAsSelectCommand rewritten based on the CreateDataSourceTableAsSelectCommand node may be as follows: Based on a custom metadataExists statement, determine whether the target table's metadata exists. If so, determine whether the ifnotExist statement is set. If so, return directly; if not, throw an exception. (The ifnotExist statement and the behavior of returning directly if the table exists are common SQL pruning judgments and processing, and are already part of the original CreateDataSourceTableAsSelectCommand. Therefore, they are retained or modified when rewritten as a virtual node.) If the table's metadata does not exist, the original table write statement is rewritten with the custom buildTempMetadata statement, which is used to generate a table containing only metadata when the table does not exist, while preserving the original data lineage. In addition, the custom setTempMetadataPath statement is used to retain the storage path of the newly created table for subsequent unified and centralized management.

[0144] The CreateHiveTableAsSelectCommand node is responsible for creating a Hive table based on query results. It uses the DatasourceV1 interface to write the query results to the new table. A more detailed description and common code are available in the following link / book / patent document: https: / / spark.apache.org / docs / 2.0.1 / api / java / org / apache / spark / sql / hive / execution / CreateHiveTableAsSelectCommand.html. Since both nodes use CTAS (Create Table As Select) syntax, the data writing logic is the same as for the CreateDataSourceTableAsSelectCommand node. For details on how to rewrite the command, refer to the examples above.

[0145] The OptimizedCreateHiveTableAsSelectCommand node is responsible for creating a Hive table based on query results. It uses the DatasourceV2 interface to write the query results to the new table. A more detailed description and common code are available at the following official link: https: / / spark.apache.org / docs / 3.0.1 / api / java / org / apache / spark / sql / hive / execution / OptimizedCreateHiveTableAsSelectCommand.html. Since both handle CTAS-like syntax, the data writing logic is the same as for the CreateDataSourceTableAsSelectCommand node. For details on how to rewrite it, refer to the examples above.

[0146] The InsertIntoHadoopFsRelationCommand node writes data to the Hadoop file system. A more detailed description and common code are available at the following official link: https: / / github.com / apache / spark / blob / master / sql / core / src / main / scala / org / apache / spark / sql / execution / datasources / InsertIntoHadoopFsRelationCommand.scala. Since this node only inserts data (and does not read, create, write, or delete metadata), the dummy node is simply returned.

[0147] The InsertIntoHiveTable node handles insert statements, inserting data into a Hive table. A more detailed description and common code are available at the following official link: https: / / spark.apache.org / docs / 2.4.3 / api / java / org / apache / spark / sql / hive / execution / InsertIntoHiveTable.html. Since this node only inserts data (and does not read, create, write, or delete metadata), it is similar to the InsertIntoHadoopFsRelationCommand node. For its rewrite, refer to the previous example.

[0148] The DropTableCommand node is responsible for deleting a table, removing its metadata and data. A more detailed description and common code are available at the following official link: https: / / github.com / apache / spark / blob / master / sql / core / src / main / scala / org / apache / spark / sql / execution / command / ddl.scala. Since virtual nodes do not affect the original data source, you only need to clear the table metadata if you determine that the table being deleted was created based on a rewritten table creation statement.

[0149] The Spark engine also provides an interface (API) for adding custom rules to the parser to improve the flexible configuration of parsing rules.

[0150] In a static parsing method provided in an optional embodiment, the logical plan node replacement rule is added through the injectPostHocResolutionRule interface under the SparkSessionExtensions function of the Spark engine.

[0151] For example, when the logical plan node replacement rule requires that the CreateDataSourceTableAsSelectCommand, CreateHiveTableAsSelectCommand, OptimizedCreateHiveTableAsSelectCommand, InsertIntoHadoopFsRelationCommand, InsertIntoHiveTable, and DropTableCommand nodes are all replaced with corresponding virtual nodes with the name prefix added with Virtual, referring to the previous example, you can call the injectPostHocResolutionRule interface to add code written based on the previous SparkRuler rule.

[0152] By adding custom rules to the original rule interface of the Spark engine to achieve static analysis of data lineage, the user's development and management difficulty can be reduced, and direct development and intrusion into the target task code (i.e., business code) can be avoided; and because the rule interface of the Spark engine is long-term stable, this static analysis method is not sensitive to the version iteration of the Spark engine, and can avoid the situation where the corresponding code of the rule interface must be modified, re-developed or re-developed due to Spark engine version updates or other component updates.

[0153] In a static parsing method provided in an optional embodiment, the established data lineage is stored in a graph database.

[0154] A graph database is a database system that stores and queries graph data and is suitable for processing large-scale graph-structured data. A graph database can present the data lineage of a data processing task by displaying data / tasks as nodes (vertices) and their lineage relationships as edges (edges).

[0155] In this embodiment, by storing the established data lineage in a graph database, intuitive display and efficient processing of data lineage for larger-scale and more complex target tasks can be achieved, and it is beneficial to represent and process many-to-many lineage relationships.

[0156] FIG3 is a method for automatically prompting task dependency configuration errors provided by another embodiment of the present invention.

[0157] This embodiment provides a method for automatically prompting task dependency configuration errors, including obtaining a data lineage relationship diagram of a target task based on the aforementioned static parsing method, comparing the dependencies of the target task's current configuration, and prompting the differences; the differences include: upstream related tasks displayed in the data lineage relationship diagram are not configured, the data lineage relationship diagram does not display any upstream related tasks, or the data lineage relationship diagram does not display configured upstream related tasks.

[0158] In this embodiment, the data lineage relationship graph can be obtained by storing the data lineage of the target task obtained based on the aforementioned static analysis method in a graph database. The obtained data lineage relationship graph is used to determine the dependency relationship between tasks.

[0159] Continuing with reference to Figure 2, the exemplary method of this embodiment includes obtaining a data lineage relationship diagram of the target task based on the aforementioned static parsing method, comparing the dependencies of the current configuration of the target task, and highlighting the differences in the comparison results; these differences may be at least one of the following: the upstream related tasks displayed in the data lineage relationship diagram of the target task are not configured, the data lineage relationship diagram of the target task does not display any upstream related tasks, or the data lineage relationship diagram of the target task does not display configured upstream related tasks.

[0160] In other examples, the difference may be prompted to the user through other common forms or a combination thereof, including optical (visual), acoustic (auditory), vibration (tactile), etc.

[0161] In the method for automatically prompting task dependency configuration errors provided by this embodiment, obtaining the data lineage relationship diagram of the target task based on the aforementioned static analysis method can be completed by the method introduced in the aforementioned static analysis method, wherein establishing the data lineage relationship diagram based on data lineage can be automatically completed by the aforementioned lineage information analysis module or other modules / programs; comparing the dependencies of the current configuration of the target task and prompting the differences can be automatically completed by the aforementioned lineage information analysis module or other modules / programs.

[0162] The above method can automatically prompt task dependency configuration errors and achieve the various technical effects of the aforementioned static parsing method.

[0163] FIG3 is a computer device / equipment / system provided by another embodiment of the present invention.

[0164] 3 , the computer device / equipment / system of this embodiment is used to implement the steps of the aforementioned embodiment. The computer device / equipment / system includes a task execution module, a lineage information collection module, a logical node replacement module, and a lineage information analysis module.

[0165] The method of realizing static analysis of data lineage through the task execution module, lineage information collection module, logical node replacement module, and lineage information analysis module, as well as the method of automatically prompting task dependency configuration errors have been introduced in the previous article and will not be repeated here.

[0166] Another embodiment provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment are implemented.

[0167] Another embodiment provides a computer program product, comprising a computer program / instruction, which implements the steps of the method of the above embodiment when executed by a processor.

[0168] The method of using computer programs / instructions to implement the data lineage static analysis and the method of automatically prompting task dependency configuration errors in the aforementioned embodiments have been introduced above and will not be repeated here.

[0169] The present invention has been introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The above implementation description is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. Changes and improvements to the present invention will be possible without exceeding the concept and scope specified in the appended claims. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A Spark engine-based data lineage static analysis method, the static analysis method comprising: Step S1: receiving a target task, wherein the target task is a table-based data processing task; Step S2: registering a listener on the Spark engine; Step S3: Add a logical plan node replacement rule in the Spark engine's parser, wherein the logical plan node replacement rule is used to replace each logical plan node involving at least one of data reading, creation, writing, or deletion with a corresponding virtual node, and is set to be executed after the parsing phase of the parser is completed. The virtual node only reads and / or creates metadata when performing read and create operations, and only writes and / or deletes metadata other than the metadata of the original data source of the target task when performing write and delete operations. Each virtual node has the same impact on data lineage as the logical plan node it replaces. Step S4: starting execution of the target task on the Spark engine; Step S5: extracting blood relationship related information based on the logical plan events obtained by the listener, constructing a table relationship tree, an attribute tree, and optionally a function tree; Step S6: traversing the table relationship tree, the attribute tree, and optionally traversing the function tree when the function tree has been constructed, to establish data lineage of the target task, where the data lineage of the target task is used to reflect the lineage relationship between the tables and / or fields involved in the target task; Among them, step S2 and step S3 can be exchanged in order, and either one of them can be advanced before step S1.

2. The static analysis method according to claim 1, characterized in that Each virtual node is obtained by rewriting the code of the logical plan node it replaces, and the rewriting conditions include: Table read statements are rewritten to return only the metadata of the table being read. The table creation statement is rewritten to create only the metadata of the table that should be created when it is needed. The table write statement is rewritten to write only the metadata that would have been written over the metadata created based on the rewritten table creation statement, and A table delete statement is rewritten to delete only the metadata created based on the rewritten table create statement.

3. The static analysis method according to claim 2, characterized in that The rewriting conditions of each virtual node also include a new storage path record statement corresponding to each rewritten table creation statement, and the storage path record statement records the storage path of all metadata created in this virtual node based on the rewritten table creation statement; the static parsing method also includes step S7: deleting all metadata under the storage path recorded based on the storage path statement, and step S7 is after step S5 or step S6.

4. The static analysis method according to claim 1, characterized in that The logical plan node replacement rule replaces at least one, two, three, four, five or all six of the following logical plan nodes: (a)CreateDataSourceTableAsSelectCommand; (b)CreateHiveTableAsSelectCommand; (c)OptimizedCreateHiveTableAsSelectCommand; (d)InsertIntoHadoopFsRelationCommand; (e)InsertIntoHiveTable; and (f)DropTableCommand.

5. The static analysis method according to claim 1, characterized in that: In step S3, the logical plan node replacement rule is added through the injectPostHocResolutionRule interface under the SparkSessionExtensions function of the Spark engine.

6. The static analysis method according to claim 1, characterized in that: The established data lineage is stored in a graph database.

7. A method for automatically notifying task dependency configuration errors, comprising obtaining a data lineage relationship diagram of a target task based on the static parsing method of claim 6, comparing the dependencies of the target task's current configuration, and noting the differences; The differences include: The upstream related tasks displayed in the data lineage relationship diagram are not configured, the data lineage relationship diagram does not display any upstream related tasks, or the data lineage relationship diagram does not display configured upstream related tasks.

8. A computer device / equipment / system, characterized in that: The computer device / equipment / system is used to implement the steps of the method described in one of claims 1 to 7, and the computer device / equipment / system includes a task execution module, a lineage information collection module, a logical node replacement module, and a lineage information analysis module.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images