Multi-source field-level blood relationship tracking method and system

Abstract

The invention discloses a multi-source field-level blood relationship tracking method and system. The method comprises the following steps: extracting features from a multi-source heterogeneous system, matching templates, triggering a corresponding acquisition strategy, and outputting an original blood relationship log; reading an original blood relationship log to generate an abstract syntax tree, extracting a source / target field set to output a field-level mapping relation, further writing the field-level mapping relation into a graph database to form a blood relationship edge, writing metadata into a relational database, establishing a two-level index, and outputting a blood relationship graph capable of being quickly positioned; and taking the blood relationship map as input, returning a target path with the maximum sum of confidence coefficients by adopting a weighted shortest path algorithm, outputting a link-level traceability result, and displaying the link-level traceability result. According to the method, complete tracking of the field-level blood relationship is realized, the query response time is shortened to a millisecond level, rapid access to multiple types of data sources is supported, and the problems of insufficient blood relationship analysis precision, low query efficiency and weak multi-source adaptability in the prior art are solved.

Description

Technical Field

[0001] This application relates to the fields of big data processing and data governance technology, and in particular to a multi-source field-level lineage tracing method and system. Background Technology

[0002] As enterprise data grows exponentially and data processing chains become increasingly complex (such as cross-database ETL, distributed computing, cloud-native data lake storage, etc.), data lineage has become the core support for data governance. It is not only a key basis for tracing the source of data quality anomalies and compliance audits (such as GDPR and the Data Security Law), but also the foundation for data asset inventory.

[0003] While current mainstream open-source tools can achieve basic lineage management, they have three major flaws:

[0004] Low lineage parsing accuracy: Field-level lineage parsing for complex SQL (multi-table joins, nested subqueries) and custom data processing scripts is incomplete, which can easily lead to "link breakage" and make it impossible to accurately locate the mapping relationship between "source field → target field".

[0005] Poor query efficiency: When using a traditional relational database to store lineage relationships, path queries need to traverse a large number of irrelevant nodes when dealing with a lineage graph of millions of nodes, resulting in a response time of more than a second, which cannot meet the needs of real-time tracing.

[0006] Weak multi-source adaptability: Integration with emerging data frameworks (such as Flink CDC and ClickHouse) requires the development of dedicated parsing logic for each framework, resulting in long access cycles and high costs for adding new data sources.

[0007] Therefore, there is an urgent need for a big data lineage management system that combines high precision, high efficiency, and high adaptability to address the pain points of existing technologies. Summary of the Invention

[0008] Based on this, the embodiments of this application provide a multi-source field-level lineage tracing method and system. By realizing complete field-level lineage tracing, the query response time is shortened to the millisecond level, and it supports rapid access to multiple types of data sources, solving the problems of insufficient lineage parsing accuracy, low query efficiency, and weak multi-source adaptability in the prior art.

[0009] Firstly, a multi-source field-level lineage tracing method is provided, which includes:

[0010] S1 lineage collection: Extract features from multi-source heterogeneous systems, match templates and trigger corresponding collection strategies, and output raw lineage logs with data source identifiers;

[0011] S2 lineage parsing: Reads the original lineage log, generates an abstract syntax tree and extracts the source / target field set, and outputs the field-level mapping relationship with confidence after calculating the three-dimensional correlation.

[0012] S3 lineage storage: The field-level mapping relationship is written into a graph database to form lineage edges, the metadata is written into a relational database, and a two-level index is established to output a lineage graph that can be quickly located.

[0013] S4 lineage lookup: Using the lineage map as input, the weighted shortest path algorithm is used to return the target path with the largest sum of confidence, and the link-level tracing result is output;

[0014] S5 Visualization: Based on the link-level tracing results, the graph nodes are clustered in two dimensions and a collapsible topology is generated to complete the end-to-end lineage display.

[0015] Optionally, the S1 bloodline collection specifically includes:

[0016] Read the system tables and configuration files of each data platform, extract three types of features: data source type, log format, and interface support, and generate feature vectors;

[0017] The cosine similarity between the feature vector and the locally preset template vector is calculated to determine the matching degree between the data source features and the template; wherein, the preset template vector includes at least MySQL template vector and Spark template vector;

[0018] The template with the highest matching degree is selected as the collection strategy, and the original lineage log with a unique identifier of the data source is output.

[0019] Optionally, the S2 lineage analysis specifically includes:

[0020] The ANTLR parser is used to convert the SQL text or data processing script in the original lineage log into an abstract syntax tree, and the source field set is extracted by traversing the tree nodes.

[0021] In the source field set, for each pair of source and target fields, calculate the field name edit distance similarity, whether the data type enumeration values ​​are equal, and the absolute value of the column number offset, and sum the three scores according to the preset weights.

[0022] If the weighted score exceeds the preset value, a mapping relationship flow is formed.

[0023] Optionally, the S3 lineage storage specifically includes:

[0024] In the graph database, a first-level index is created based on node type to filter irrelevant nodes; wherein, the node type includes table / field / job;

[0025] Within each node type partition, create a secondary index based on data domain and update time to narrow the query scope.

[0026] Optionally, the S4 bloodline query specifically includes:

[0027] The confidence level of bloodline relationship mapping is converted into path weight to ensure that highly reliable links are selected first;

[0028] Starting from the target node, traverse the predecessor nodes in ascending order of weight, and dynamically update the shortest path between nodes.

[0029] Return the shortest path when the node to be queried is found or all relevant nodes have been traversed.

[0030] Optionally, the S5 visualization specifically includes:

[0031] Each node is labeled with two attributes: the data domain and the processing stage.

[0032] Group nodes with the same data domain and the same processing stage into one category and display them in aggregated form.

[0033] The display interface supports expanding / collapsed cluster nodes, and clicking on a node allows you to view detailed lineage links and field mapping relationships.

[0034] Optionally, the graph database is Neo4j, and the relational database is MySQL; the lineage edges are stored as directed edges from the source field to the target field, and the edge attributes retain the mapping confidence, SQL hash, and operation timestamp.

[0035] Secondly, a multi-source, field-level kinship tracing system is provided, which includes:

[0036] The lineage acquisition module is used to extract features from multi-source heterogeneous systems, match templates, trigger corresponding acquisition strategies, and output raw lineage logs with data source identifiers.

[0037] The lineage resolution module is used to read the original lineage log, generate an abstract syntax tree and extract the source / target field set, and output the field-level mapping relationship with confidence after calculating the three-dimensional correlation.

[0038] The lineage storage module is used to write the field-level mapping relationship into a graph database to form lineage edges, write the metadata into a relational database, establish a two-level index, and output a lineage graph that can be quickly located.

[0039] The kinship query module is used to take the kinship map as input, use a weighted shortest path algorithm to return the target path with the largest sum of confidence, and output the link-level tracing result;

[0040] The visualization module is used to perform two-dimensional clustering of the graph nodes and generate a collapsible topology based on the link-level tracing results, thereby completing the end-to-end lineage display.

[0041] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the multi-source field-level lineage tracing method described in any of the first aspects above.

[0042] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the multi-source field-level lineage tracing method described in any of the first aspects above.

[0043] The beneficial effects of the technical solutions provided in this application include at least the following:

[0044] (1) Through the three-step collection strategy, the system does not need to write dedicated collection code for new data sources. It can directly use existing MySQL and Spark templates to complete log capture, shorten the access cycle, reduce access costs, and truly realize the "zero code" extension of multi-source heterogeneous systems.

[0045] (2) ANTLR is used to generate an abstract syntax tree and score it in three dimensions. Field names, data types and column numbers are included in the mapping judgment at the same time, which effectively improves the accuracy of field-level lineage parsing under complex SQL (multi-table JOIN, nested subqueries), completely eliminates the "link break" phenomenon, and provides a unique and reliable field-level basis for subsequent quality traceability and compliance audit.

[0046] (3) Through a two-level indexing mechanism, the query range is first filtered by table / field / job type, and then trimmed again by business domain and time window, so that the number of nodes that need to be scanned in the graph database is reduced. Attached Figure Description

[0047] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart illustrating the steps of a multi-source field-level lineage tracing method provided in this application embodiment;

[0049] Figure 2 A system architecture diagram of a multi-source field-level lineage tracing method provided in this application embodiment;

[0050] Figure 3This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0053] This invention aims to solve the problems of insufficient accuracy in bloodline analysis, low query efficiency, and weak multi-source adaptability in existing technologies. It provides a big data bloodline management system based on multi-dimensional features, which realizes complete tracking of bloodlines at the field level, shortens the query response time to the millisecond level, and supports rapid access to multiple types of data sources.

[0054] Please refer to Figure 1 The diagram illustrates a flowchart of a multi-source field-level lineage tracing method provided in an embodiment of this application. This method may include the following steps:

[0055] S1 lineage collection: Extracts features from multi-source heterogeneous systems, matches templates, triggers corresponding collection strategies, and outputs raw lineage logs with data source identifiers.

[0056] In this step, the system tables and configuration files of each data platform are first read via JDBC, REST API, or a local proxy to extract three key features: "data source type, log format, and interface support," generating 128-bit feature vectors. Then, the feature vectors are compared with six pre-set template vectors (MySQL, Hive, Spark, Flink, ClickHouse, and Custom) using cosine similarity calculation. The template with the highest score is used as the current data source's acquisition strategy—for example, automatically enabling an "event listener + execution plan parsing" thread for Spark clusters, and injecting an SDK callback JAR file for custom tools without APIs, capturing SQL text and execution plans in real time. Finally, a unified output of raw lineage log files containing "unique data source identifier + timestamp + original SQL" is provided, enabling zero-code integration of new data sources within 5 days, providing complete and lossless input for subsequent parsing.

[0057] S2 lineage parsing: Reads the original lineage log, generates an abstract syntax tree and extracts the source / target field set, and outputs the field-level mapping relationship with confidence after calculating the three-dimensional correlation.

[0058] In this step, after receiving the raw lineage log, the parsing layer calls the ANTLR4 pre-built syntax file to convert the SQL or data processing script into an abstract syntax tree. It traverses the tree nodes from top to bottom to extract the SELECT column expression and the INSERT / UPDATE target column, forming the "source field set" and the "target field set". Then, for each pair of source field s and target field t, it starts the "three-dimensional correlation calculation" in parallel - ① field name edit distance similarity, ② whether the data type enumeration values ​​are equal, ③ the absolute value of the column number offset - and sums them with weights of 0.4 / 0.3 / 0.3. When the comprehensive score is ≥0.8, a directed edge from s to t is established in memory and the confidence score is written. If there are multiple incoming edges for the same target field, only the one with the highest confidence is retained. Finally, the field-level mapping relationship flow of "source field → target field + confidence" is output, which improves the parsing accuracy in complex SQL scenarios from 70% to over 95% and completely eliminates link breaks.

[0059] S3 lineage storage: Field-level mapping relationships are written into a graph database to form lineage edges, metadata is written into a relational database, and two-level indexes are established to output a lineage graph that can be quickly located.

[0060] In this step, the storage layer adopts a hybrid architecture of "graph database Neo4j + relational database MySQL": the field-level mapping relationship obtained from S2 is converted into Cypher statements in real time and written to Neo4j to form "table / field / job" nodes and "lineage dependency" edges. The edge attributes carry mapping confidence, SQL hash, and operation timestamp. At the same time, structured metadata such as data source configuration, user permissions, and data domain classification are written to MySQL in three-table normal form, and a unique index is created on (data_source_id, db_name). Furthermore, a first-level index is created in Neo4j according to "node type" to filter nodes that are irrelevant to the table / field / job. Then, a second-level index is created in each type partition according to "data domain + update time", which reduces the number of nodes that need to be scanned for queries by more than 60%, laying the storage foundation for millisecond-level queries.

[0061] S4 lineage lookup: Taking the lineage graph as input, it uses a weighted shortest path algorithm to return the target path with the largest sum of confidence and outputs the link-level tracing results.

[0062] In this step, the query layer takes the lineage graph in Neo4j as input, converts the edge mapping confidence c into path weight w = 1 - c, forming a weighted directed graph. Then, using the user-specified starting node as the root, it uses the Fibonacci heap-optimized Dijkstra algorithm to traverse predecessor / successor nodes in ascending order of weight, dynamically updating the shortest path. The search stops when the target node is hit for the first time or when the subgraph with confidence > 0.7 is traversed, and the highly reliable link with the largest sum of confidence (i.e., the smallest sum of weight) is returned. The query result is serialized into JSON and written to Redis. The key consists of "starting ID_ending ID_version number", with TTL = 300s. Subsequent identical requests directly return to the cache, achieving real-time traceability capability with a response time of within 50ms for a scale of millions of nodes.

[0063] S5 Visualization: Based on the link-level tracing results, the graph nodes are clustered in two dimensions and a collapsible topology is generated to complete the end-to-end lineage display.

[0064] In this step, the visualization layer first reads the Neo4j node attributes and automatically injects dual labels of "data domain" and "processing stage" into each table / field. The data domain is matched by regular expression using table name prefixes, and the processing stage is mapped by SQL keywords (SELECT / INSERT / CREATE, etc.). Then, in the front-end force-oriented layout, nodes with the same labels are calculated for convex hulls and aggregated into super nodes, which are displayed by default as folded clusters such as "e-commerce-collection" and "finance-storage," solving the visual congestion of thousands of nodes. When the user clicks on a super node, the front-end sends an "expand" request to the back-end. The back-end highlights and renders the link-level tracing results returned by S4 onto the canvas and drills down to display field-level mapping details, achieving one-click expansion / collapse and an 80% improvement in end-to-end lineage readability of the interactive topology.

[0065] This system consists of a five-layer architecture: a bloodline collection layer, a bloodline parsing layer, a bloodline storage layer, a bloodline query layer, and a visualization layer. Each layer collaborates through specific algorithms, and the core principles are as follows:

[0066] 1. Bloodline Acquisition Layer: Multi-source Adaptive Acquisition Mechanism

[0067] Function: Automatically identify data source types (such as MySQL, Hive, Spark, custom tools) and select the optimal strategy to collect metadata (table structure, field definition) and lineage logs (data flow records).

[0068] Core algorithm: Multi-source adaptation decision mechanism.

[0069] Principle: Adaptive data collection from multiple data sources is achieved through three steps: feature extraction, template matching, and strategy selection.

[0070] Feature extraction: Extract key features from the data source, including: data source type (relational / non-relational / computing framework), log format (JSON / text / binary), and interface support (whether API / JDBC is provided).

[0071] Template matching: The system pre-sets collection templates for multiple data sources (such as MySQL templates and Spark templates) and calculates the matching degree between the data source features and the templates (comprehensive feature similarity score, 0~1 points).

[0072] Strategy selection: Select the template with the highest matching degree as the collection strategy. For example, automatically enable the "log listening + execution plan parsing" strategy for Spark clusters and enable the "SDK active reporting" strategy for custom tools.

[0073] Advantages: No need to develop dedicated collection logic for new data sources; it can be quickly adapted through template matching, improving access efficiency by 60%.

[0074] 2. Lineage Analysis Layer: Field-level Precise Lineage Mapping Algorithm

[0075] Function: Parses the original lineage data into field-level lineage relationships, repairs broken links, and ensures complete mapping of "source field → target field".

[0076] Core algorithm: Three-dimensional field mapping and parsing.

[0077] Principle: Based on syntax parsing and multi-dimensional feature matching, accurate field-level lineage identification is achieved. The steps are as follows:

[0078] Syntax Tree Parsing: The ANTLR parser is used to convert SQL or data processing code into an Abstract Syntax Tree (AST), extracting the source field set (such as order_id, amount) and the target field set (such as order_id, final_amount).

[0079] Multi-dimensional matching: Calculates the correlation between fields from three dimensions:

[0080] Field name matching: The similarity of field names is calculated using the edit distance algorithm (e.g., the similarity between "amount" and "final_amount" is 0.6).

[0081] Data type matching: Validates whether the data types of the source field and the target field are consistent (e.g., INT→INT is a match, INT→VARCHAR is a mismatch).

[0082] Positional relationship matching: Analyze the position of fields in SQL / code (e.g., the first field in the SELECT clause corresponds to the first field in the INSERT clause);

[0083] Mapping determination: Based on the combined scores of the three dimensions (weights are configurable), when the total score is greater than or equal to the threshold (default 0.8), it is determined that there is a direct mapping relationship (such as "amount→final_amount").

[0084] Advantages: Solves the problem of field-level lineage breakage in complex SQL (such as multi-table JOIN, nested subqueries), improving parsing accuracy from 70% to over 95%.

[0085] 3. Lineage Storage Layer: Graph Structure and Hierarchical Index Design

[0086] Functionality: Stores lineage nodes (tables, fields, jobs) and relationships, and optimizes query efficiency through indexes.

[0087] Core design: Hybrid storage and hierarchical indexing.

[0088] Storage structure: Adopts a hybrid architecture of "graph database + relational database".

[0089] Graph databases (such as Neo4j): store lineage graphs, where nodes represent data entities (tables, fields, jobs), edges represent relationships between entities (such as "data flow" and "processing dependency"), and edge attributes include mapping confidence, operation time, etc.

[0090] Relational databases (such as MySQL): store structured metadata (such as data source configuration, user permissions, and data domain classification).

[0091] Hierarchical indexing mechanism:

[0092] First-level index: Partition by node type (table / field / job) and filter irrelevant nodes (e.g., when querying "table node", directly exclude "job node");

[0093] Secondary index: For nodes within each type of partition, create an index based on "data domain + update time" (e.g., "e-commerce domain + updates in the last 7 days") to further narrow down the query scope.

[0094] Advantages: Compared to traditional storage, the number of nodes that need to be scanned during a query is reduced by more than 60%, laying the foundation for efficient querying.

[0095] 4. Lineage Query Layer: Weighted Shortest Path Query Algorithm

[0096] Features: Supports upstream lineage query (to check data source), downstream lineage query (to check data flow direction), and shortest path query, returning accurate results.

[0097] Core algorithm: Confidence-based path optimization query.

[0098] Principle: Improve the traditional Dijkstra algorithm by incorporating the "mapping confidence" of kinship relationships to optimize path lookup.

[0099] (1) Weight definition: The "mapping confidence" (0~1) of blood relationship is converted into path weight (the higher the confidence, the smaller the weight) to ensure that high reliability links are selected first;

[0100] (2) Path search: Starting from the target node, traverse the predecessor nodes in ascending order of weight, and dynamically update the shortest path between nodes;

[0101] (3) Termination condition: When the node to be queried is found or all related nodes have been traversed, return the shortest path (i.e. the most reliable lineage link).

[0102] Advantages: Compared with traditional algorithms, the query efficiency is improved by more than 5 times, and the response time for bloodline path queries of millions of nodes is shortened to less than 50ms.

[0103] 5. Visualization Layer: Data Domain - Stage Two-Dimensional Clustering Display

[0104] Function: Displays kinship relationships in the form of a topological graph, solving the problem of "visual congestion" in large-scale graphs, and supporting interaction and tracing.

[0105] Core algorithm: Two-dimensional node clustering.

[0106] Principle: Cluster nodes according to "data domain + processing stage", the steps are as follows:

[0107] (1) Dimensional division: Mark two attributes for each node - data domain (such as "e-commerce" and "finance") and processing stage (such as "collection", "cleaning", "storage" and "consumption");

[0108] (2) Clustering rules: Group nodes with the same data domain and the same processing stage into one category, such as clustering and displaying nodes (business database tables) of "e-commerce domain + collection stage";

[0109] (3) Interaction optimization: Supports the expansion / collapse of cluster nodes. Clicking on a node allows you to view detailed lineage links and field mapping relationships.

[0110] Advantages: Improves the readability of lineage maps containing thousands of nodes by 80%, allowing users to quickly locate target links.

[0111] like Figure 2A system architecture diagram for a multi-source field-level lineage tracing method is presented. Specifically, the bottom layer is the "data input source layer," which includes heterogeneous systems from multiple sources such as Hive, Sqoop, Falco, Stor, and Hbas. Above it, the "application layer" implements data injection and export through the Kafka message component and HTTP / REST interface services. The "classification system" on the right performs real-time classification of business data and writes the results to the JanusGraph distributed graph database engine. At the same time, metadata and indexes are stored in HBase and Solr respectively, forming a high-performance lineage storage and retrieval system of "graph database engine - metadata storage - index data storage."

[0112] System deployment architecture: Adopting a cloud-native architecture, containerized deployment is implemented based on Kubernetes, and the deployment methods of each layer of components are as follows:

[0113] (1) Lineage collection layer: Deployed in the sidecar mode on the data processing node (such as Spark / Flink cluster), listens to component logs in real time, and is non-intrusive;

[0114] (2) Lineage resolution layer: Deployed in the form of microservices, with more than 3 replicas to ensure high availability and support elastic scaling;

[0115] (3) Lineage storage layer: The graph database (Neo4j) adopts a master-slave architecture (1 master and 2 slaves), and the relational database (MySQL) adopts master-slave replication to ensure that data is not lost;

[0116] (4) Lineage query layer: Microservice deployment, combined with Redis caching of high-frequency query results, further reduces response time;

[0117] (5) Visualization layer: The front-end application is deployed on the Nginx server, and static resources are accelerated through CDN, supporting access from PC and mobile terminals.

[0118] It can be seen that the system value of this application includes:

[0119] (1) Improved accuracy: Through three-dimensional field mapping and parsing, the accuracy of field-level lineage parsing has been improved from 70% in the existing technology to over 95%, solving the "link break" problem;

[0120] (2) Efficiency optimization: Based on graph storage, hierarchical index and weighted path algorithm, the bloodline query response time is shortened from seconds to less than 50ms, meeting the real-time tracing requirements;

[0121] (3) Enhanced adaptability: The multi-source adaptation decision mechanism supports shortening the access cycle of new data sources from 15 days to 5 days, reducing access costs by 60%;

[0122] (4) Compliance support: The complete end-to-end lineage meets the requirements of GDPR and the Data Security Law for "data traceability", improving compliance audit efficiency by 80%.

[0123] This application also provides a multi-source field-level lineage tracing system, which may include:

[0124] The lineage acquisition module is used to extract features from multi-source heterogeneous systems, match templates, trigger corresponding acquisition strategies, and output raw lineage logs with data source identifiers.

[0125] The lineage resolution module is used to read the original lineage log, generate an abstract syntax tree and extract the source / target field set, and output the field-level mapping relationship with confidence after calculating the three-dimensional correlation.

[0126] The lineage storage module is used to write the field-level mapping relationship into a graph database to form lineage edges, write the metadata into a relational database, establish a two-level index, and output a lineage graph that can be quickly located.

[0127] The kinship query module is used to take the kinship map as input, use a weighted shortest path algorithm to return the target path with the largest sum of confidence, and output the link-level tracing result;

[0128] The visualization module is used to perform two-dimensional clustering of the graph nodes and generate a collapsible topology based on the link-level tracing results, thereby completing the end-to-end lineage display.

[0129] For specific limitations regarding the multi-source field-level kinship tracing system, please refer to the limitations of the multi-source field-level kinship tracing method mentioned above, which will not be repeated here. Each module in the aforementioned multi-source field-level kinship tracing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0130] In one embodiment, an electronic device is provided, which may be a computer, and its internal structure diagram may be as follows: Figure 3As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for multi-source field-level lineage tracing data. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-source field-level lineage tracing method.

[0131] Those skilled in the art will understand that, Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described multi-source field-level lineage tracing method.

[0133] In one embodiment of this application, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implements the steps of the above-described multi-source field-level lineage tracing method.

[0134] The computer-readable storage medium and computer program product provided in this embodiment are similar in implementation principle and technical effect to the above method embodiments, and will not be repeated here.

[0135] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-source field-level kinship tracking method, characterized in that, The method comprises: S1 blood collection: extracting features, matching templates and triggering corresponding collection strategies for multi-source heterogeneous systems, and outputting original blood log with data source identification; S2 blood analysis: reading the original blood log, generating an abstract syntax tree and extracting a source / target field set, and outputting a field-level mapping relationship with confidence after three-dimensional correlation degree calculation; S3 blood storage: writing the field-level mapping relationship into a graph database to form a blood edge, writing metadata into a relational database, and establishing two indexes to output a blood atlas that can be quickly located; S4 blood query: inputting the blood atlas, using a weighted shortest path algorithm to return a target path with the maximum confidence sum, and outputting a link-level traceability result; S5 visualization: based on the link-level traceability result, performing two-dimensional clustering on the atlas nodes and generating a collapsible topology to complete end-to-end blood display.

2. The multi-source field-level bloodline tracking method according to claim 1, wherein, The S1 blood collection specifically comprises: Read the system table and configuration file of each data platform, extract three types of features including data source type, log format and interface support, and generate a feature vector; Calculate the cosine similarity between the feature vector and the pre-set template vector, and calculate the matching degree of the data source features and the template; wherein the pre-set template vector at least includes a MySQL template vector and a Spark template vector; Select the template with the highest matching degree as the collection strategy, and output the original blood log with the unique identification of the data source.

3. The multi-source field-level bloodline tracking method according to claim 1, wherein, The S2 blood analysis specifically comprises: Use ANTLR grammar parser to convert SQL text or data processing script in the original blood log into an abstract syntax tree, and extract the source field set by traversing the tree nodes; In the source field set, for each pair of source field and target field, calculate the field name edit distance similarity, whether the data type enumeration value is equal, and the absolute value of the column sequence offset, and weight the sum of the three scores according to the preset weight; If the weighted score exceeds the preset value, a mapping relationship flow is formed.

4. The multi-source field-level bloodline tracking method according to claim 1, wherein, The S3 blood storage specifically comprises: Establish a first index according to node type in the graph database to filter irrelevant nodes; wherein the node type includes table / field / job; Establish a second index according to data domain+update time in each node type partition to narrow the query range.

5. The multi-source field-level bloodline tracking method according to claim 1, wherein, The S4 blood query specifically comprises: Convert the mapping confidence of the blood relationship into path weight to ensure that high-reliability links are preferentially selected; Starting from the target node, traverse the predecessor nodes in ascending order of weight, and dynamically update the shortest path between nodes; When the node to be queried is found or all related nodes are traversed, the shortest path is returned.

6. The multi-source field-level bloodline tracking method according to claim 1, wherein, The S5 visualization specifically comprises: Label each node with two attributes, including data domain and processing stage; Nodes with the same data domain and processing stage are classified into one category for node aggregation display; In the display interface, the expansion / folding of clustered nodes is supported, and clicking a node can view detailed blood link and field mapping relationship.

7. The multi-source field-level bloodline tracking method according to claim 1, wherein, The graph database is Neo4j, and the relational database is MySQL; the blood relationship edge is stored in the form of a directed edge from a source field to a target field, and the edge attribute retains a mapping confidence, an SQL hash, and an operation timestamp.

8. A multi-source field-level lineage tracking system, comprising: The system comprises: a blood relationship collection module configured to extract features, match templates, and trigger corresponding collection strategies for a multi-source heterogeneous system, and output original blood relationship logs with data source identifiers; a blood relationship analysis module configured to read the original blood relationship logs, generate an abstract syntax tree, and extract a source / target field set, and output a field-level mapping relationship with a confidence after three-dimensional correlation degree calculation; a blood relationship storage module configured to write the field-level mapping relationship into a graph database to form a blood relationship edge, write metadata into a relational database, establish two-level indexes, and output a blood relationship graph that can be quickly located; a blood relationship query module configured to take the blood relationship graph as input, return a target path with the maximum confidence sum by using a weighted shortest path algorithm, and output a link-level traceability result; a visualization module configured to perform two-dimensional clustering on graph nodes based on the link-level traceability result, generate a foldable topology, and complete end-to-end blood relationship display.

9. An electronic device, comprising: The computer program is stored in the memory and is executed by the processor to implement the multi-source field-level blood relationship tracking method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and is executed by the processor to implement the multi-source field-level blood relationship tracking method according to any one of claims 1 to 7.

Images