Real-time data consanguinity analysis method and system based on Spark logic execution plan

Through the real-time data blood relationship analysis method based on Spark logic execution plan, the problem that the existing technology cannot adapt to multiple computing engines and complex data warehouse environments is solved, cross-engine, comprehensive and accurate data blood relationship tracking is achieved, and the system complexity is reduced and maintenance costs ensure business continuity and the accuracy of blood relationships.

CN118939678AActive Publication Date: 2024-11-12HENAN ZHONGYUAN CONSUMER FINANCE CO LTD

Patent Information

Application Number
CN202411118296.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-12
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The existing technology cannot cover multiple computing engines in a comprehensive and accurate manner, and it is difficult to adapt to the complex and changeable data warehouse environment, and cannot achieve cross-engine, comprehensive, accurate and real-time data ties tracking.

Method used

Using real-time data blood relationship analysis method based on Spark logical execution plan, SQL capture and message delivery, SQL syntax conversion, SparkSession construction and logical execution plan generation, blood relationship analysis and data storage, to realize SQL statement analysis and multiple computing engines. Real-time tracking and storage of blood ties.

Benefits of technology

It achieves wide compatibility and accuracy of different computing engines, reduces system deployment and maintenance costs, ensures business continuity, and ensures the accuracy and comprehensiveness of blood relationships through a unified logical execution plan format.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939678A_ABST
    Figure CN118939678A_ABST
Patent Text Reader

Abstract

The invention discloses a real-time data consanguinity analysis method and system based on a Spark logic execution plan, and the method comprises the steps: SQL capture and message transmission: capturing SQL statements which are being executed in real time from a plurality of calculation engines, and sending the captured SQL statements and source engine information thereof as messages to a message queue; sQL grammar conversion: receiving SQL statements from the message queue and converting the SQL statements; the method comprises the following steps of: constructing a SparkSession and generating a logic execution plan, and constructing and configuring the SparkSession in an analysis service; the constructed SparkSession is utilized, and the converted SQL statement is analyzed into a logic execution plan; blood relationship analysis: performing traversal analysis on the generated logic execution plan to obtain complete blood relationship data; and storing blood relationship data. Through high decoupling with a specific calculation engine, installation of an analysis plug-in or modification of configuration of the engine in the engine is not needed, so that potential stability hazards possibly introduced by system change are avoided, and the compatibility problem caused by dependence on an internal mechanism of a specific engine is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing and data management, and relates to a real-time data lineage analysis method and system based on Spark logic execution plan. Background Art

[0002] With the rapid development of big data technology, data warehouse (DW) has become a core component of enterprise data management and analysis. In a complex data warehouse environment, data is organized in multiple hierarchical structures, such as the source layer, history layer, model layer, and market layer. These layers are interconnected through a series of data processing and conversion processes. Although this multi-level and multi-source data structure improves the flexibility and scalability of data management, it also brings the complexity of data lineage tracking.

[0003] Data lineage refers to all the transformations and dependencies that data undergoes during its lifecycle, including associations between tables and fields. Accurate and real-time acquisition of this lineage information is crucial for scenarios such as data tracing, change impact analysis, data quality monitoring, and data governance. However, in current data warehouse practices, tracking and managing data lineage still faces many challenges.

[0004] First, traditional data lineage tracing methods often rely on manual parsing of SQL scripts or query statements. This method builds an Abstract Syntax Tree (AST) and traverses the parsing to identify dependencies between tables and fields. Although this method can theoretically cover a variety of computing engines, it faces many limitations in practical applications: differences in SQL dialects between different computing engines lead to poor parsing compatibility, complex SQL or special syntax (such as Spark SQL's WITH clause) is difficult to parse, and the parsing results may be inaccurate or miss important information.

[0005] Secondly, another common method is to integrate lineage tracking plug-ins in the computing engine, such as LineageLogger in Apache Hive. This method captures lineage information in real time when SQL is executed, with high accuracy and real-time performance. However, this method also has significant limitations: it is highly dependent on a specific computing engine, and requires additional development work for non-Hive environments (such as Spark, Impala, Trino, etc.), which is difficult. In addition, the integration of plug-ins usually requires modifying the environment configuration, which increases the complexity and maintenance cost of the system.

[0006] In summary, the existing technology has many shortcomings in data lineage tracking, cannot fully and accurately cover a variety of computing engines, and is difficult to adapt to the complex and changing data warehouse environment. Therefore, a new technical solution is urgently needed to overcome the shortcomings of the existing technology and achieve cross-engine, comprehensive, accurate and real-time data lineage tracking. Summary of the invention

[0007] The purpose of the present invention is to solve the problem that the prior art cannot comprehensively and accurately cover a variety of computing engines in data lineage tracking and is difficult to adapt to the complex and changeable data warehouse environment, and to provide a real-time data lineage analysis method and system based on Spark logical execution plan.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: A real-time data lineage analysis method based on Spark logical execution plan includes the following steps: SQL capture and message delivery: Capture the SQL statements being executed from multiple computing engines in real time, and send the captured SQL statements and their source engine information as messages to the message queue; SQL syntax conversion: receiving and converting SQL statements from the message queue; SparkSession construction and logical execution plan generation: Build and configure SparkSession in the parsing service, set it to Local mode, enable Hive support, and associate Hive metadata by configuring the metadata storage address; use the built SparkSession to parse the converted SQL statement into a logical execution plan; Blood relationship analysis: traverse and analyze the generated logical execution plan, recursively traverse the specific type of nodes in the logical execution plan, identify and extract the relationship between tables and fields in the SQL statement, and obtain complete blood relationship data; Bloodline data storage: save the analyzed bloodline relationship data into the graph database.

[0009] The computing engines include Spark, Hive, Impala and Trino.

[0010] When receiving the SQL statement from the message queue, the source engine of the SQL statement is determined. If the SQL statement does not come from the Spark engine, a SQL conversion tool or a regular expression is used to convert the SQL statement into a syntax compatible with Spark SQL.

[0011] The SparkSession construction and logical execution plan generation are specifically as follows: Build a SparkSession, build a SparkSession instance running in local mode, and configure its application name as SparkLineageAnalyser; at the same time, specify the connection address of Hive metadata to enable support for Hive, allowing SparkSession to access Hive metadata when generating a logical execution plan; Get SessionState, get the internal SessionState object from the constructed SparkSession. The SessionState object is responsible for managing all states of the Spark SQL session; Parse SQL statements and generate logical execution plans. Use the SQL parser in SessionState to parse the input SQL statements and generate an abstract syntax tree. Combine Hive metadata to perform metadata binding and map the table name and column name identifiers in the SQL statements to specific metadata objects. Finally, based on the results of parsing and binding, a logical execution plan for the SQL statement is generated.

[0012] The blood relationship analysis specifically includes: Initialize the parsing environment, set up parsing tools and libraries; Load the logical execution plan, submit the SQL statement to Spark SQL, and obtain the logical execution plan generated by Spark SQL, which represents the logical representation of the SQL query; Traverse the logical execution plan tree and use a recursive traversal method to traverse the logical execution plan tree; during the traversal process, identify the data processing steps according to the type of the node.

[0013] Identify the relationship between tables. During the traversal process, identify the association between tables based on the connection nodes; Extract the information of the connection node, including the name of the connected table and the connection condition, to determine the blood relationship between the source table and the target table; Identify the relationship between fields, traverse projections and select nodes; extract the source and destination information of fields, determine the value relationship, filtering relationship, and calculation or conversion logic between fields; For a join node, the join key fields are analyzed to determine the association relationship between the fields across tables.

[0014] The blood relationship data obtained by parsing is saved in a graph database, specifically using a graph data structure to store the extracted blood relationship data, including an adjacency list, an adjacency matrix or a relational database.

[0015] In the graph data structure, the nodes of the graph represent tables or fields, the edges represent the relationships between tables or between fields, and the weights or attributes of the edges contain detailed information about the relationships.

[0016] A real-time data lineage analysis system based on Spark logical execution plan, including the following modules: An SQL capture and message delivery module, which is used to capture the SQL statements being executed from multiple computing engines in real time, and send the captured SQL statements and their source engine information as messages to a message queue; An SQL syntax conversion module, wherein the SQL syntax conversion module is used to receive and convert SQL statements from a message queue; SparkSession construction and logical execution plan generation module, the SparkSession construction and logical execution plan generation module is used to build and configure SparkSession in the parsing service, set it to Local mode, and enable Hive support, and associate Hive metadata by configuring the metadata storage address; use the constructed SparkSession to parse the converted SQL statement into a logical execution plan; A blood relationship analysis module, which is used to traverse and analyze the generated logical execution plan, identify and extract the relationship between tables and fields in the SQL statement by recursively traversing the specific type of nodes in the logical execution plan, and obtain complete blood relationship data; A blood relationship data storage module is used to store the blood relationship data obtained through analysis in a graph database.

[0017] A device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method as described in any one of the preceding items when executing the computer program.

[0018] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the preceding items.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The real-time data lineage analysis method based on the Spark logical execution plan in the present invention is highly decoupled from specific computing engines, and there is no need to install parsing plug-ins in these engines or modify their configurations, thereby avoiding stability risks that may be introduced due to system changes and reducing compatibility issues caused by reliance on the internal mechanisms of specific engines. There is no need to change the configuration of the existing computing engine, making system deployment and maintenance more convenient, while reducing the impact on the existing production environment and ensuring the continuity of business operations. In addition, the highly decoupled design also facilitates subsequent upgrades and iterations of the parsing logic, because the upgrade is mainly concentrated in the parsing module of the system without involving changes to the underlying computing engine.

[0020] Furthermore, by uniformly converting the captured SQL statements into Spark SQL syntax, wide compatibility with different computing engines is achieved, ensuring that no matter which computing engine the SQL statement originally comes from, it can be accurately parsed and processed by the system. Since the SQL statement is obtained directly from the computing engine, there is no need to pay attention to the SQL submission entry and method, thus avoiding SQL omissions caused by monitoring blind spots and ensuring the comprehensiveness of data lineage analysis.

[0021] The logical execution plan in the method is generated by the Spark engine and follows a unified specification, which ensures that the generated logical execution plan has a unified format and accurate content. This accuracy is the basis for the subsequent blood relationship analysis. The logical execution plan contains all the metadata information of the library tables used, providing rich data support for the accurate analysis of blood relationships. The richness of metadata enables the system to have a deeper understanding of the data structure and data flow behind the SQL statement. In addition, the differences caused by the special syntax of different computing engines are eliminated. This processing method of eliminating differences simplifies the analysis logic and improves the versatility and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flow chart of the real-time data lineage analysis method based on Spark logical execution plan of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1 , which is a real-time data lineage analysis method based on Spark logical execution plan in the present invention, specifically comprising the following steps: S1, SQL capture and message delivery, captures the SQL statements being executed from multiple computing engines in real time, and sends the captured SQL statements and their source engine information as messages to the message queue; computing engines include but are not limited to Spark, Hive, Impala, Trino, etc., and the SQL statements of each computing engine are identified with their source engine before sending to facilitate subsequent processing.

[0028] S2, SQL syntax conversion, receives and converts SQL statements from the message queue; determines its source engine. If the SQL statement does not come from the Spark engine, uses SQL conversion tools (such as sqlglot) or regular expressions to convert the SQL statement into syntax compatible with Spark SQL to ensure that a unified logical execution plan can be generated later.

[0029] S3, SparkSession construction and logical execution plan generation, build and configure SparkSession in the parsing service, set it to Local mode, and enable Hive support, and associate Hive metadata by configuring the metadata storage address; use the built SparkSession to parse the converted SQL statement into a logical execution plan.

[0030] S3.1, build SparkSession, build a SparkSession instance running in local mode, and configure its application name as SparkLineageAnalyser; at the same time, specify the connection address of Hive metadata to enable support for Hive, allowing SparkSession to access Hive metadata when generating a logical execution plan.

[0031] S3.2, get SessionState, get the internal SessionState object from the constructed SparkSession. The SessionState object is responsible for managing all states of the Spark SQL session.

[0032] S3.3, parses SQL statements and generates a logical execution plan. Use the SQL parser in SessionState to parse the input SQL statement and generate an abstract syntax tree. Combine Hive metadata for metadata binding and map the table name and column name identifiers in the SQL statement to specific metadata objects.

[0033] S3.4, finally, based on the results of parsing and binding, a logical execution plan of the SQL statement is generated.

[0034] S4, blood relationship analysis, traverses and analyzes the generated logical execution plan, recursively traverses the specific type of nodes in the logical execution plan, identifies and extracts the relationship between tables and fields in the SQL statement, and obtains complete blood relationship data; S4.1, initialize the parsing environment and set up parsing tools and libraries.

[0035] S4.2, load the logical execution plan, submit the SQL statement to Spark SQL, and obtain the logical execution plan generated by it, which represents the logical representation of the SQL query.

[0036] S4.3, traverse the logical execution plan tree, and use a recursive traversal method to traverse the logical execution plan tree; during the traversal process, identify the data processing steps according to the type of the node.

[0037] S4.4, identifying the relationship between tables. During the traversal process, the association relationship between tables is identified based on the connection nodes.

[0038] S4.5, extract the information of the connection node, including the name of the connected table and the connection condition, to determine the blood relationship between the source table and the target table.

[0039] S4.6, identify the relationship between fields, traverse the projection and select nodes; extract the source and destination information of the fields, determine the value relationship, filtering relationship and calculation or conversion logic between fields.

[0040] S4.7, for the connection node, analyze the connection key field to determine the association relationship between the cross-table fields.

[0041] S5, blood relationship data storage, saves the analyzed blood relationship data into the graph database.

[0042] Graph databases can effectively represent and store complex network data structures, making it easier to understand and analyze the relationship between data. Graph data structures are used to store extracted relationship data, including adjacency lists, adjacency matrices, or specialized graph databases. Graph nodes can represent tables or fields, edges can represent relationships between tables or between fields, and edge weights or attributes can contain detailed information about the relationship, such as connection conditions, conversion logic, etc.

[0043] An embodiment of the present invention is a real-time data lineage analysis system based on Spark logical execution plan, comprising the following modules: The SQL capture and message delivery module is used to capture the SQL statements being executed from multiple computing engines in real time, and send the captured SQL statements and their source engine information as messages to the message queue; the message queue is used to temporarily store and forward these SQL statement messages to ensure the real-time and sequential nature of the data.

[0044] The SQL syntax conversion module is used to receive and convert SQL statements from the message queue, and ensure that all SQL statements are compatible with the subsequent Spark SQL processing engine.

[0045] The SparkSession construction and logical execution plan generation module is used to build and configure the SparkSession in the parsing service, set it to Local mode, and enable Hive support, and associate the Hive metadata by configuring the metadata storage address; use the constructed SparkSession to parse the converted SQL statement into a logical execution plan.

[0046] A blood relationship analysis module is used to traverse and parse the generated logical execution plan, identify and extract the relationships between tables and fields in the SQL statement by recursively traversing the specific type of nodes in the logical execution plan, and obtain complete blood relationship data.

[0047] The blood relationship data storage module is used to store the blood relationship data obtained by the analysis into a graph database. Through the graph database, users can easily query, visualize and further analyze blood relationships.

[0048] The system of the present invention ensures the real-time and accuracy of data lineage analysis by real-time capture of SQL statements, rapid generation of logical execution plans, and accurate analysis of lineage relationships. At the same time, through the use of SparkSession and message queues in Local mode, the system reduces the impact on the production environment while maintaining high performance.

[0049] An embodiment of the present invention provides a terminal device. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0050] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0051] The device / terminal equipment may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The device / terminal equipment may include, but is not limited to, a processor and a memory.

[0052] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0053] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the apparatus / terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0054] If the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0055] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time data lineage analysis method based on Spark logical execution plan, characterized in that: The following steps are involved: SQL capture and message delivery: Capture the SQL statements being executed from multiple computing engines in real time, and send the captured SQL statements and their source engine information as messages to the message queue; SQL syntax conversion: receiving and converting SQL statements from the message queue; SparkSession construction and logical execution plan generation: Build and configure SparkSession in the parsing service, set it to Local mode, enable Hive support, and associate Hive metadata by configuring the metadata storage address; Use the built SparkSession to parse the converted SQL statements into a logical execution plan; Blood relationship analysis: traverse and analyze the generated logical execution plan, recursively traverse the specific type of nodes in the logical execution plan, identify and extract the relationship between tables and fields in the SQL statement, and obtain complete blood relationship data; Bloodline data storage: save the analyzed bloodline relationship data into the graph database.

2. A real-time data lineage analysis method based on Spark logical execution plan as described in claim 1, characterized in that: The computing engines include Spark, Hive, Impala and Trino.

3. The real-time data lineage analysis method based on Spark logical execution plan as claimed in claim 1, characterized in that: When receiving the SQL statement from the message queue, the source engine of the SQL statement is determined. If the SQL statement does not come from the Spark engine, a SQL conversion tool or a regular expression is used to convert the SQL statement into a syntax compatible with Spark SQL.

4. The real-time data lineage analysis method based on Spark logical execution plan as claimed in claim 1, characterized in that: The SparkSession construction and logical execution plan generation are specifically as follows: Build a SparkSession, build a SparkSession instance running in local mode, and configure its application name as SparkLineageAnalyser; at the same time, specify the connection address of Hive metadata to enable support for Hive, allowing SparkSession to access Hive metadata when generating a logical execution plan; Get SessionState, get the internal SessionState object from the constructed SparkSession. The SessionState object is responsible for managing all states of the Spark SQL session; Parse SQL statements and generate logical execution plans. Use the SQL parser in SessionState to parse the input SQL statements and generate an abstract syntax tree. Combine Hive metadata to perform metadata binding, and map the table name and column name identifiers in the SQL statement to specific metadata objects; Finally, based on the results of parsing and binding, a logical execution plan for the SQL statement is generated.

5. The real-time data lineage analysis method based on Spark logical execution plan as claimed in claim 1, characterized in that: The blood relationship analysis specifically includes: Initialize the parsing environment and set up parsing tools and libraries; Load the logical execution plan, submit the SQL statement to Spark SQL, and obtain the logical execution plan generated by Spark SQL, which represents the logical representation of the SQL query; Traverse the logical execution plan tree and use a recursive traversal method to traverse the logical execution plan tree; during the traversal process, identify the data processing steps according to the type of the node; Identify the relationship between tables. During the traversal process, identify the association between tables based on the connection nodes; Extract the information of the connection node, including the name of the connected table and the connection condition, to determine the blood relationship between the source table and the target table; Identify the relationship between fields, traverse projections and select nodes; extract the source and destination information of fields, determine the value relationship, filtering relationship, and calculation or conversion logic between fields; For a join node, the join key fields are analyzed to determine the association relationship between the fields across tables.

6. A real-time data lineage analysis method based on Spark logical execution plan as claimed in claim 1, characterized in that: The blood relationship data obtained by parsing is saved in a graph database, specifically using a graph data structure to store the extracted blood relationship data, including an adjacency list, an adjacency matrix or a relational database.

7. A real-time data lineage analysis method based on Spark logical execution plan as described in claim 6, characterized in that: In the graph data structure, the nodes of the graph represent tables or fields, the edges represent the relationships between tables or between fields, and the weights or attributes of the edges contain detailed information about the relationships.

8. A real-time data lineage analysis system based on Spark logical execution plan, characterized in that: Includes the following modules: An SQL capture and message delivery module, which is used to capture the SQL statements being executed from multiple computing engines in real time, and send the captured SQL statements and their source engine information as messages to a message queue; An SQL syntax conversion module, wherein the SQL syntax conversion module is used to receive and convert SQL statements from a message queue; SparkSession construction and logic execution plan generation module, which is used to build and configure SparkSession in the analysis service, set it to Local mode, and enable Hive support, and associate Hive metadata by configuring the metadata storage address; Use the built SparkSession to parse the converted SQL statements into a logical execution plan; A blood relationship analysis module, which is used to traverse and analyze the generated logical execution plan, identify and extract the relationship between tables and fields in the SQL statement by recursively traversing the specific type of nodes in the logical execution plan, and obtain complete blood relationship data; A blood relationship data storage module is used to store the blood relationship data obtained through analysis in a graph database.

9. A device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Data consanguinity analysis method and device, electronic equipment and storage medium

    CN113032362A

  • Full-link field-level blood relationship analysis method, system and equipment and storage medium

    CN115934855A

  • Analysis method for FlinkSQL field blood relationship

    CN116303429A

  • Spark SQL-based blood relationship data acquisition method

    CN116975089A

  • Data blood relationship analysis method and device, equipment and storage medium

    CN117131230A

Cited By

  • Concurrent scheduling and fault-tolerant optimization method and system for distributed jobs of Spark application layer

    CN121681075A

  • Data analysis method supporting lake and warehouse system, electronic equipment and program product

    CN121705300A

  • A data bloodline collection and display method, a big data application system and equipment

    CN122795867A