A method, device, medium and equipment for data blood relationship link detection

Abstract

The application relates to the technical field of data detection, and specifically provides a data blood relationship link detection method, device, medium and equipment, which comprises the following steps: constructing a data query object based on a current batch of to-be-detected node set; performing multi-hop query in a graph database by using the data query object to obtain a query subgraph; filtering a predecessor node set and a successor node set associated with temporary nodes in the temporary node set in the query subgraph by means of the temporary node set in the query subgraph; modifying the query subgraph based on the predecessor node set and the successor node set to obtain a reconnected subgraph; detecting the reconnected subgraph by using a pre-determined data detection rule to obtain a data consistency detection result; and the data consistency detection result representing whether the source field metadata and the target field metadata of each edge in the reconnected subgraph are consistent. The application can improve the efficiency of data blood relationship link consistency detection and reduce the labor cost.

Description

Technical Field

[0001] This application relates to the field of data detection technology, and more specifically, to a method, apparatus, medium, and device for detecting data lineage links. Background Technology

[0002] Graph databases are the optimal medium for storing and querying data lineage, while lineage consistency testing is a core quality control step to ensure the authenticity and usability of lineage maps. Only by combining the two can we achieve reliable, verifiable, and traceable data governance.

[0003] Currently, existing technologies, after storing field-level lineage relationships in graph databases, typically employ a single-point triggered lineage traversal method for data lineage link detection. Specifically, a field (or table) is manually or systematically designated as the starting point, a depth-first traversal is performed to obtain its upstream and downstream subgraphs, and then consistency checks are performed between the source fields and target fields (such as name / type / length). It can be seen that current data lineage link detection methods mostly rely on single-point queries, and their efficiency cannot be guaranteed when dealing with hundreds of thousands or tens of millions of fields. Summary of the Invention

[0004] The purpose of some embodiments of this application is to provide a method, apparatus, medium and device for data lineage detection. The technical solutions of the embodiments of this application can realize batch data detection and improve the efficiency of data lineage detection.

[0005] In a first aspect, some embodiments of this application provide a method for detecting data lineage links, comprising: constructing a data query object based on a set of nodes to be detected in the current batch; performing a multi-hop query in a graph database using the data query object to obtain a query subgraph; wherein the query subgraph includes lineage edges associated with nodes in the set of nodes to be detected and edge attribute values ​​of the lineage edges; filtering a set of predecessor nodes and a set of successor nodes associated with temporary nodes in the set of temporary nodes from the query subgraph; modifying the query subgraph based on the set of predecessor nodes and the set of successor nodes to obtain a reconnected subgraph; and detecting the reconnected subgraph using predetermined data detection rules to obtain a data consistency detection result; wherein the data consistency detection result characterizes whether the source field metadata and target field metadata of each edge in the reconnected subgraph are consistent.

[0006] Some embodiments of this application obtain a query subgraph from a graph database by constructing a data query object from the current batch of nodes to be detected; then, the query subgraph is modified by relevant nodes in the query subgraph to obtain a reconnected subgraph; finally, data detection rules are used to detect the reconnected subgraph to obtain a data consistency detection result. Embodiments of this application can achieve batch data detection through batch detection, thereby improving the efficiency of data lineage link detection; at the same time, processing the subgraph can improve the accuracy of data lineage link detection.

[0007] In some embodiments, before constructing a data query object based on the set of nodes to be detected in the current batch, the method further includes: determining a batch value under memory constraints based on environmental running memory parameters; wherein the environmental running memory parameters include running memory value, preset ratio and average memory occupied by nodes; dividing the node data to be detected into batches according to the batch value to obtain the set of nodes to be detected in each of the multiple batches; wherein the current batch is one of the multiple batches.

[0008] Some embodiments of this application divide the nodes to be detected into batches based on the batch values ​​determined by the environment's running memory parameters, obtaining a set of nodes to be detected in each batch, which provides support for subsequent batch detection. At the same time, determining the batch values ​​according to the environment's running memory parameters can reduce data processing pressure and improve detection efficiency.

[0009] In some embodiments, constructing a data query object based on the current batch of nodes to be detected includes: obtaining multiple fields from the set of nodes to be detected; wherein the multiple fields include node identifier, traversal direction, traversal depth or lineage identifier; and constructing the data query object using the multiple fields.

[0010] Some embodiments of this application construct a data query object using multiple fields in the set of nodes to be detected, which enables fast and accurate data querying.

[0011] In some embodiments, the step of filtering the set of predecessor nodes and the set of successor nodes associated with temporary nodes in the query subgraph from the set of temporary nodes in the query subgraph includes: determining the set of temporary nodes by reading the temporary label of each node in the query subgraph; and determining the set of predecessor nodes and the set of successor nodes in the node relationship set of the query subgraph based on the connection relationship of each temporary node in the set of temporary nodes.

[0012] Some embodiments of this application determine the set of temporary nodes by reading the temporary labels in the query subgraph, and then filter out the set of predecessor nodes and the set of successor nodes, providing reliable data support for modifying the query subgraph.

[0013] In some embodiments, modifying the query subgraph based on the predecessor node set and the successor node set to obtain a reconnected subgraph includes: creating multiple virtual edges in the query subgraph based on the node correspondence in the predecessor node set and the successor node set; deduplicating the multiple virtual edges and deleting temporary nodes in the temporary node set in the query subgraph to obtain the reconnected subgraph.

[0014] Some embodiments of this application process the query subgraph by using the node correspondence in the predecessor node set and the successor node set to obtain a reconnected subgraph, which can retain only the important detection nodes and ensure the accuracy and efficiency of subsequent detection.

[0015] In some embodiments, the step of using predetermined data detection rules to detect the reconnected subgraph and obtain a data consistency detection result includes: constructing a bidirectional indexing mechanism in memory based on the reconnected subgraph; wherein the bidirectional indexing mechanism includes an indexing mechanism between source nodes and target nodes and an indexing mechanism between node identifiers and lineage file identifiers; retrieving data to be detected in the reconnected subgraph according to the bidirectional indexing mechanism; and using the data detection rules to detect the data to be detected to determine the data consistency detection result.

[0016] Some embodiments of this application use a bidirectional indexing mechanism to retrieve reconnected subgraphs to achieve subsequent data consistency detection, thereby improving the efficiency and reliability of data detection.

[0017] In some embodiments, retrieving the data to be detected in the reconnected subgraph according to the bidirectional indexing mechanism includes: dividing all nodes in the reconnected subgraph according to a preset value to obtain multiple node partitions; determining the data to be detected in each of the multiple node partitions according to the bidirectional indexing mechanism; and detecting the data to be detected using the data detection rules to determine the data consistency detection result, including: performing parallel detection on the data to be detected in each node partition using the data detection rules to obtain the data consistency detection result.

[0018] Some embodiments of this application can improve the efficiency of data lineage detection by partitioning multiple nodes and detecting them in parallel.

[0019] Secondly, some embodiments of this application provide an apparatus for detecting data lineage links, comprising: a construction module for constructing a data query object based on a set of nodes to be detected in the current batch; a query module for performing a multi-hop query in a graph database using the data query object to obtain a query subgraph; wherein the query subgraph includes lineage edges associated with nodes in the set of nodes to be detected and edge attribute values ​​of the lineage edges; a filtering module for filtering a set of predecessor nodes and a set of successor nodes associated with temporary nodes in the query subgraph using a set of temporary nodes in the query subgraph; a modification module for modifying the query subgraph based on the set of predecessor nodes and the set of successor nodes to obtain a reconnected subgraph; and a detection module for detecting the reconnected subgraph using predetermined data detection rules to obtain a data consistency detection result; wherein the data consistency detection result characterizes whether the source field metadata and target field metadata of each edge in the reconnected subgraph are consistent.

[0020] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0021] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0022] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 One of the flowcharts for a data lineage detection method provided in some embodiments of this application; Figure 2 Schematic diagrams of reconnected subgraphs provided for some embodiments of this application; Figure 3A second flowchart illustrating a method for detecting data lineage links provided for some embodiments of this application; Figure 4 Block diagrams of apparatus for detecting data lineage links provided for some embodiments of this application; Figure 5 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0025] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0026] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] In related technologies, lineage queries are typically initiated starting with a single field or table, and rule comparisons are performed on the edges or paths obtained through traversal. For large-scale enterprise-level lineage graphs, in order to cover all fields, it is often necessary to repeatedly trigger single-point traversals, making it difficult to form a stable batch detection pipeline. Furthermore, the size of the subgraph returned by multi-hop queries will expand rapidly as the traversal depth increases, which can easily lead to problems such as increased memory usage, query timeouts, or memory overflows.

[0028] Furthermore, lineage graphs typically contain temporary nodes such as temporary table nodes, intermediate result nodes, unarchived nodes, or temporary resource placeholder nodes. Existing technologies generally handle these temporary nodes in two ways: one is to retain all temporary nodes, which, while maintaining the integrity of the lineage path, introduces a large number of intermediate nodes and edges unrelated to detection, leading to increased subgraph size, increased detection noise, and decreased detection efficiency; the other is to directly delete temporary nodes, which, while simplifying the graph structure, disrupts the lineage connectivity between source and target nodes, causing path breaks, missed detections, and the inability to correctly trace back to the corresponding lineage file or script.

[0029] Therefore, existing technologies lack a graph structure processing mechanism that can eliminate interference from temporary nodes while maintaining the connectivity of lineage paths and the integrity of attribution, and also lack a consistency batch detection scheme suitable for large-scale lineage graphs.

[0030] In view of this, embodiments of this application provide a method, apparatus, storage medium, and device for data lineage link detection, which solves the technical problems in the prior art where retaining temporary nodes in the process of large-scale lineage graph detection leads to subgraph expansion and increased detection noise, while directly deleting temporary nodes causes lineage path breakage, missed detection, and inability to correctly attribute causes.

[0031] The method in this application embodiment includes: constructing a data query object based on the current batch of nodes to be detected; performing a multi-hop query in a graph database using the data query object to obtain a query subgraph; identifying a set of temporary nodes from the query subgraph, and determining the set of predecessor nodes and the set of successor nodes associated with each temporary node in the set of temporary nodes; creating virtual edges in the query subgraph based on the node correspondence in the set of predecessor nodes and the set of successor nodes, and merging the lineage file identifiers on the original lineage edges corresponding to the temporary nodes and writing them into the virtual edges; deleting the temporary nodes and the original associated edges corresponding to the temporary nodes to obtain a reconnected subgraph; performing consistency detection on the reconnected subgraph based on predetermined data detection rules to obtain a data consistency detection result, wherein the data consistency detection result is used to characterize whether the source field metadata and the target field metadata in the reconnected subgraph are consistent.

[0032] For example, in a lineage subgraph, there exists the following path: source field A reaches target field B after passing through temporary fields Temp1 and Temp2. ​​The corresponding original lineage edges are A→Temp1, Temp1→Temp2, and Temp2→B, respectively, and the three original lineage edges carry the lineage file identifier sets {L1}, {L2}, and {L3}, respectively.

[0033] In this embodiment, Temp1 and Temp2 are first identified as temporary nodes; then, their predecessor and successor nodes are determined step by step based on the temporary nodes, and virtual edges are established after penetration; finally, virtual edges from A to B are generated. The edge attributes of the virtual edges from A to B are written with the merged and deduplicated lineage file identifier set {L1, L2, L3}, and Temp1, Temp2 and their corresponding original associated edges are deleted.

[0034] Based on this, a consistency check is then performed on the source field metadata and target field metadata corresponding to A→B. Therefore, this embodiment does not simply delete temporary nodes, but performs a penetrating reconnection between the predecessor and successor nodes before deleting the temporary nodes, and retains the lineage file identifiers corresponding to the original lineage edges. This reduces the subgraph size, query pressure, and detection noise while maintaining the lineage reachability between the source and target nodes and subsequent attribution capabilities. Therefore, this embodiment can support batch detection of large-scale lineage links under limited memory constraints, and improve detection accuracy, detection efficiency, and problem localization efficiency.

[0035] The following is in conjunction with the appendix Figure 1 The implementation process of data lineage link detection provided in some embodiments of this application is illustrated by way of example. This data lineage link detection implementation process can be executed by a terminal device, which can be a mobile terminal, a non-portable computer terminal, a server terminal, etc., and is not specifically limited to these embodiments in this application.

[0036] like Figure 1 As shown, some embodiments of this application provide a flowchart of a method for detecting data lineage links.

[0037] To improve the efficiency of data lineage detection, this application proposes a batch detection method. Batch detection requires first dividing all the data of the nodes to be detected into batches. Therefore, before performing the following operation S110, the data lineage detection method may further include: determining a batch value under memory constraints based on environmental running memory parameters; wherein the environmental running memory parameters include running memory value, a preset ratio, and the average memory occupied by the nodes; dividing the data of the nodes to be detected into batches according to the batch value, obtaining the set of nodes to be detected in each of multiple batches; wherein the current batch is one of multiple batches.

[0038] For example, in a specific embodiment of this application, a memory-constrained cursor paging streaming batch processing method is used to batch the data of the nodes to be detected. Specifically, the batch size (batchSize) is determined based on the JVM heap memory (as a specific example of runtime memory value), a preset ratio memoryRatio, and the average memory usage of nodes (avgNodeMemorySize) (as a specific example of the average memory usage of nodes). batchSize = floor(heapMemory×memoryRatio / avgNodeMemorySize).

[0039] The data to be tested is paginated according to batch size, and a pagination cursor structure is used to record the pagination progress. A set of nodes to be tested within one batch constitutes one page. The cursor in the pagination cursor structure includes fields such as namespace, resourceType, pageNum, pageSize, and lastSeenId (the last processed ID, i.e., the cursor position). Alternatively, keyset pagination can also be used. During subsequent data lineage detection, the set of nodes to be tested under each batch can be retrieved iteratively to execute subsequent detection processes. After each batch is processed, object references are released to avoid memory overflow caused by a single load. Releasing object references refers to releasing temporary object or collection references associated with each batch in the application process's JVM memory, allowing GC (Garbage Collection) to reclaim them and preventing memory accumulation leading to memory overflow as the number of pages increases.

[0040] In some embodiments of this application, the method for detecting data lineage may include: S110: Construct a data query object based on the set of nodes to be detected in the current batch.

[0041] For example, in a specific embodiment of this application, all the data of the nodes to be detected are divided into batches using the above method to achieve batch detection. Each batch is then retrieved cyclically to perform data lineage link detection. To facilitate the explanation of the detection process, we will take any one batch (i.e., the current batch) as an example. Specifically, when detecting the set of nodes to be detected in the current batch, it is necessary to construct a corresponding Data Transfer Object (DTO).

[0042] Optionally, S110 may include: obtaining multiple fields from the set of nodes to be detected; wherein the multiple fields include node identifier, traversal direction, traversal depth or lineage identifier; and constructing the data query object through the multiple fields.

[0043] For example, in a specific embodiment of this application, a query DTO is constructed for the fields in the set of nodes to be detected. These fields include the query ID set `queryIds` (as a specific example of a node identifier) ​​in the current batch of node IDs, the traversal direction, the traversal depth, and the lineage identifier `hasLineageId`. The DTO may contain some or all of these fields; this embodiment of the application does not impose specific limitations on this.

[0044] S120, using the data query object to perform a multi-hop query in the graph database to obtain a query subgraph; wherein, the query subgraph includes the bloodline edges associated with the nodes in the set of nodes to be detected and the edge attribute values ​​of the bloodline edges.

[0045] For example, in a specific embodiment of this application, a multi-hop query is performed in a graph database using a DTO, outputting a query subgraph. The multi-hop query starts from the set of initial nodes and proceeds continuously along the bloodline edges for N hops (N = depth / maxDepth), retrieving all bloodline edges traversed on the 1st, 2nd, ..., Nth hops, rather than only querying directly adjacent one-hop relationships. When the depth limit is reached during the multi-hop query process, it is forcibly terminated, and a set of truncationPoints is recorded for subsequent incremental completion or incremental detection expansion.

[0046] For example, using a GO statement structure, the hop count is written in the query (e.g., a range like 1..depth, or a loop executing GO), and the starting point is the queryIds (a batch of node IDs) in the DTO. The lineage edges in the final output query subgraph represent the lineage relationship connection between "upstream field / table -> downstream field / table". `properties(edge).lineageId` is a property value in the edge properties (i.e., the edge property value), also known as the lineage file identifier `lineageId`. Subsequently, the lineageId can be used to look up the lineage entity `LineageEntity` to locate the corresponding ETL script path.

[0047] S130, using the set of temporary nodes in the query subgraph, filter the set of predecessor nodes and the set of successor nodes associated with the temporary nodes in the set of temporary nodes in the query subgraph.

[0048] For example, in a specific embodiment of this application, temporary nodes are identified from the query subgraph, and the predecessor node set is obtained by filtering dstId==tempNode.id based on their connection relationship with other nodes, and the successor node set is obtained by filtering srcId==tempNode.id.

[0049] Optionally, S130 may include: S131, the set of temporary nodes is determined by reading the temporary label of each node in the query subgraph.

[0050] For example, in a specific embodiment of this application, the temporary tag of the node is first obtained. This temporary tag is built into the system (builtInCode="temp") and can be manually labeled; the temporary node can be used to identify content in the data lineage such as temporary entities, unarchived lineage nodes, and temporary resource placeholders. At runtime, the tag attribute tags (i.e., comma-separated tagId strings) of each node is read, and its existence of a temporary tag ID tempTagId is determined by parsing it; if it exists, the node is treated as a temporary node tempNode. Otherwise, the node is confirmed as a field node ColumnNode, and it is necessary to further check whether the tags of its parent table node contain tempTagId; if it does, the parent table is marked as a temporary table, and its field node is also treated as a temporary node. By identifying the temporary nodes, a set of temporary nodes can be obtained.

[0051] S132, based on each temporary node in the temporary node set, determine the predecessor node set and the successor node set in the node relationship set of the query subgraph according to the connection relationship.

[0052] For example, in a specific embodiment of this application, the temporary nodes identified above are filtered one by one in subgraph.relationships (as a specific example of a set of node relationships): dstId == tempNode.id to obtain the set of predecessor nodes, and srcId == tempNode.id to obtain the set of successor nodes.

[0053] S140, Modify the query subgraph based on the predecessor node set and the successor node set to obtain a reconnected subgraph.

[0054] For example, in a specific embodiment of this application, the connection relationships in the query subgraph can be processed (e.g., deleted and / or merged) through the node correspondence in the predecessor node set and the successor node set to obtain a reconnected subgraph.

[0055] Optionally, S140 may include: creating multiple virtual edges in the query subgraph based on the node correspondence in the predecessor node set and the successor node set; deduplicating the multiple virtual edges and deleting temporary nodes in the temporary node set in the query subgraph to obtain the reconnected subgraph.

[0056] For example, in a specific embodiment of this application, a virtual edge `isVirtual=true` is created for each pair of predecessor-successor nodes in the predecessor node set and successor node set; the virtual edge includes a predecessor edge and a successor edge (as a specific example of multiple virtual edges). Then, the `lineageId` of the predecessor edge and the successor edge are merged to preserve the tracing chain. Specifically, during the merging process, when merging the `lineageId` of the predecessor edge and the `lineageId` of the successor edge (and `lineageIds` accumulated over potentially longer links) into a single set, duplicate `lineageId` values ​​are removed (i.e., the same lineage file ID may appear repeatedly on different edges), avoiding duplicate recording of the same ID in the virtual edge attributes. Finally, temporary nodes and the original associated edges are deleted from the query subgraph after creating the virtual edges, resulting in a reconnected subgraph.

[0057] For example, such as Figure 2 As shown, the subgraph with the temporary node TempColumn represents the connectivity before the traversal; the connectivity after deleting the temporary node and the original associated edges represents the connectivity after the traversal. After the traversal, the connectivity from Src to Dst is maintained, and the set of lineage file IDs required for attribution is preserved, facilitating automatic location of the corresponding ETL script.

[0058] S150, the reconnected subgraph is inspected using a pre-determined data inspection rule to obtain a data consistency inspection result; wherein, the data consistency inspection result indicates whether the source field metadata and target field metadata of each edge in the reconnected subgraph are consistent.

[0059] For example, in a specific embodiment of this application, the data detection rules include the starting namespace, the set of detection dimensions (such as name, type, or length), the maximum traversal depth (maxDepth), the traversal direction (direction), and whether temporary tables (includeTempTables) are included, which can be adjusted as needed. The reconnected subgraph is traversed using these data detection rules, and a consistency comparison check is performed on the data involved in each reconnected bloodline edge to confirm whether the source data and the target data are consistent. If they are consistent, the data consistency detection result is consistent; if they are inconsistent, the reason for the inconsistency is recorded.

[0060] Specifically, S150 may include: constructing a bidirectional indexing mechanism in memory based on the reconnected subgraph; wherein the bidirectional indexing mechanism includes an indexing mechanism between source nodes and target nodes and an indexing mechanism between node identifiers and lineage file identifiers; retrieving data to be detected in the reconnected subgraph according to the bidirectional indexing mechanism; and using the data detection rules to detect the data to be detected, thereby determining the data consistency detection result.

[0061] For example, in a specific embodiment of this application, a bidirectional hash index is constructed (as a specific example of a bidirectional indexing mechanism): srcToDstMap: source node ID → target node ID set; dstToSrcMap: target node ID → source node ID set (as a specific example of an indexing mechanism between source and target nodes); nodeToLineageIdsMap: node ID → lineageId / lineageIds set (as a specific example of an indexing mechanism between lineage file identifiers).

[0062] Examples of bidirectional indexing mechanisms are shown in Tables 1 to 3: Table 1

[0063] Table 2

[0064] Table 3

[0065] Subsequently, the aforementioned bidirectional indexing mechanism is used to index the data on the reconstructed subgraph, and the indexed data to be tested is then tested to obtain data consistency test results. For example, for each bloodline edge srcId -> dstId, the source field metadata corresponding to srcId and the target field metadata corresponding to dstId are retrieved, and the names, types, lengths, etc., are compared item by item according to the rules to obtain data consistency test results.

[0066] For example, the data consistency check rules include name consistency ignoring case. If lower(srcName) == lower(dstName) for a lineage edge, the data consistency check result is consistent. If lower(srcName) and lower(dstName) are different, the data consistency check result is inconsistent, and the reason for inconsistency is recorded as inconsistent field names (ignoring case).

[0067] Optionally, S150 may include: dividing all nodes in the reconnected subgraph according to a preset value to obtain multiple node partitions; determining the data to be detected in each of the multiple node partitions according to the bidirectional indexing mechanism; and performing parallel detection on the data to be detected in each node partition using the data detection rules to obtain the data consistency detection result.

[0068] For example, in a specific embodiment of this application, all nodes in the reconnected subgraph are partitioned according to the partition size (as a specific example of a preset value, which can be 50~200), resulting in multiple node partitions. Then, consistency checks are performed in parallel on each node partition to obtain the final data consistency check result. The HashMap constructed above is used for relationship lookup, while parallel partitioning improves throughput.

[0069] If inconsistencies exist in the data consistency check results, the `lineageId` carried by the lineage edge can be used to look up the `LineageEntity` table to obtain the ETL script path information that caused the inconsistency. This script path information is then written into the check results, for example, into the `taskPath` field, to achieve automatic attribution. Finally, the data consistency check results are written to a memory cache. When the number of data consistency check results in the memory cache reaches a set threshold (e.g., 500), they are written in batches to reduce the number of input / output operations.

[0070] The following is in conjunction with the appendix Figure 3 The present application provides an exemplary description of the specific process for data lineage detection in some embodiments.

[0071] Please see the appendix Figure 3 , Figure 3 A flowchart of a method for detecting data lineage links provided for some embodiments of this application.

[0072] The above process is illustrated below by example.

[0073] S310: Divide the data of the nodes to be tested into batches according to the batch value to obtain the set of nodes to be tested in each batch of multiple batches.

[0074] S320 constructs a data query object based on multiple fields in the set of nodes to be detected in the current batch.

[0075] S330: Utilize the data query object to perform a multi-hop query in the graph database and obtain the query subgraph.

[0076] S340: Determine the set of temporary nodes by reading the temporary label of each node in the query subgraph.

[0077] S350: Based on each temporary node in the temporary node set, filter the node relationship set of the query subgraph to determine the predecessor node set and the successor node set.

[0078] S360 modifies the queried subgraph based on the predecessor node set and the successor node set to obtain the reconnected subgraph.

[0079] S370 constructs a bidirectional indexing mechanism based on reconnected subgraphs and divides all nodes in the reconnected subgraphs according to preset values ​​to obtain multiple node partitions.

[0080] S380 uses data detection rules and a bidirectional indexing mechanism to perform parallel detection on the data to be detected in each node partition, and obtains the data consistency detection results.

[0081] It is understood that the specific implementation process of S310~S380 can be referred to the method embodiment provided above. To avoid repetition, detailed descriptions are omitted here.

[0082] It should be noted that the system iteratively reads the set of nodes to be detected from each batch across multiple batches, using a paging cursor to load and process batches sequentially before releasing references, achieving streaming processing. This allows for comprehensive detection of all node data, improving detection efficiency. Because only the "current batch" data is retained in memory at any given time, the peak memory usage changes from "O(N) related to the total number of nodes N" to "O(batchSize) related to the batch size." This allows for stable scanning of tens of millions of nodes within the same heap memory, meaning this application can complete full batch detection of tens of millions of fields with limited memory, avoiding memory overflow.

[0083] Furthermore, this application constructs a parameterized query object (DTO), sets a maximum traversal depth (maxDepth), and forcibly terminates the query when the maximum is reached. It also records truncationPoints for subsequent incremental query completion. Multi-hop statements are used to return src / dst and lineageId, reducing additional expansion. Because the traversal is limited to maxDepth, the worst-case size is limited to O(b^maxDepth), preventing infinite expansion. Simultaneously, the truncation point recording ensures that the "limited depth" does not permanently lose its extension path, allowing for subsequent completion from the truncation point. maxDepth can be flexibly set according to actual runtime memory usage or specific needs.

[0084] This application identifies ephemeral nodes based on node label attributes and parent node label attributes, obtains their predecessor and successor node sets, creates virtual edges for each predecessor-successor pair and sets an isVirtual flag, and merges the IDs of multiple lineage files associated with the ephemeral node into the virtual edge attributes. Finally, the ephemeral node and the original edge are removed. By reconnecting nodes in the predecessor and successor node sets, the reachability from the source to the target is preserved (avoiding missed detections due to broken links caused by deletion); by "removing ephemeral nodes," the subgraph size and noisy nodes are reduced, which can reduce false alarms and query pressure.

[0085] This application constructs a bidirectional relationship index from source to target and from target to source, as well as a lineage file index. Batch data is divided into fixed partition sizes, and multi-core parallel detection is performed using parallel streams. Hash indexing makes the expected time complexity of relationship lookup O(1), and the overall construction is O(E) (where E is the number of edges), avoiding redundant scanning during the comparison phase. Partition parallelism distributes the detection tasks of independent nodes across multiple cores, improving CPU utilization. Finally, this application can also extract lineageId from the lineage relationship and perform a reverse query on the relational database LineageEntity to obtain the ETL script path write results. The results are first written to a memory buffer, and then batch-added to the database when a threshold (e.g., 500 records) is reached. This method creates a direct mapping between "edge → lineageId → script path," eliminating the need for manual backtracking and reducing the attribution and localization time from hours to seconds. Buffered batch writing reduces the number of writes from N to N / threshold times, improving overall detection efficiency.

[0086] Please refer to Figure 4 , Figure 4 The diagram illustrates a block diagram of a data lineage detection apparatus provided in some embodiments of this application. It should be understood that this data lineage detection apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this data lineage detection apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0087] Figure 4 The data lineage detection device includes at least one software functional module that can be stored in a memory or embedded in the data lineage detection device in the form of software or firmware. The device includes: a construction module 410, used to construct a data query object based on the current batch of nodes to be detected; a query module 420, used to perform a multi-hop query in a graph database using the data query object to obtain a query subgraph; wherein the query subgraph includes lineage edges associated with nodes in the node set to be detected and the edge attribute values ​​of the lineage edges; and a filtering module 430. The system is used to filter the set of predecessor nodes and the set of successor nodes associated with the temporary nodes in the query subgraph using the set of temporary nodes in the query subgraph; the modification module 440 is used to modify the query subgraph based on the set of predecessor nodes and the set of successor nodes to obtain a reconnected subgraph; the detection module 450 is used to detect the reconnected subgraph using a predetermined data detection rule to obtain a data consistency detection result; wherein, the data consistency detection result characterizes whether the source field metadata and the target field metadata of each edge in the reconnected subgraph are consistent.

[0088] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0089] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0090] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0091] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement the methods of any of the above embodiments.

[0092] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0093] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0094] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0096] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for detecting data lineage, characterized in that, include: Construct a data query object based on the set of nodes to be detected in the current batch; The data query object is used to perform a multi-hop query in the graph database to obtain a query subgraph; wherein, the query subgraph includes the bloodline edges associated with the nodes in the set of nodes to be detected and the edge attribute values ​​of the bloodline edges; Using the set of temporary nodes in the query subgraph, filter the set of predecessor nodes and the set of successor nodes that are associated with the temporary nodes in the set of temporary nodes in the query subgraph. The query subgraph is modified based on the predecessor node set and the successor node set to obtain a reconnected subgraph; The reconnected subgraph is inspected using predetermined data inspection rules to obtain data consistency inspection results; wherein, the data consistency inspection results characterize whether the source field metadata and target field metadata of each edge in the reconnected subgraph are consistent.

2. The method as described in claim 1, characterized in that, Before constructing the data query object based on the current batch of nodes to be detected, the method further includes: Batch values ​​under memory constraints are determined based on environmental running memory parameters; wherein, the environmental running memory parameters include running memory value, preset ratio, and average memory usage value per node; The data of the nodes to be detected are divided into batches according to the batch value to obtain the set of nodes to be detected in each batch of multiple batches; wherein, the current batch is one of the multiple batches.

3. The method as described in claim 1 or 2, characterized in that, The process of constructing a data query object based on the set of nodes to be detected in the current batch includes: Obtain multiple fields from the set of nodes to be detected; wherein, the multiple fields include node identifier, traversal direction, traversal depth or lineage identifier; The data query object is constructed using the aforementioned multiple fields.

4. The method as described in claim 1 or 2, characterized in that, The step of filtering the set of predecessor nodes and the set of successor nodes associated with temporary nodes in the query subgraph from the set of temporary nodes in the query subgraph includes: The set of temporary nodes is determined by reading the temporary label of each node in the query subgraph; Based on each temporary node in the temporary node set, the predecessor node set and the successor node set are determined according to the connection relationship in the node relationship set of the query subgraph.

5. The method as described in claim 1 or 2, characterized in that, The step of modifying the query subgraph based on the predecessor node set and the successor node set to obtain a reconnected subgraph includes: Based on the node correspondence in the predecessor node set and the successor node set, multiple virtual edges are created in the query subgraph; The multiple virtual edges are deduplicated, and the temporary nodes in the temporary node set of the query subgraph are deleted to obtain the reconnected subgraph.

6. The method as described in claim 1 or 2, characterized in that, The step of detecting the reconnected subgraph using pre-determined data detection rules to obtain data consistency detection results includes: A bidirectional indexing mechanism is constructed in memory based on the reconnected subgraph; wherein, the bidirectional indexing mechanism includes an indexing mechanism between the source node and the target node and an indexing mechanism between the node identifier and the lineage file identifier; The data to be detected is retrieved in the reconnected subgraph according to the bidirectional indexing mechanism described above; The data to be detected is tested using the data detection rules, and the data consistency detection result is determined.

7. The method as described in claim 6, characterized in that, The step of retrieving the data to be detected in the reconnected subgraph according to the bidirectional indexing mechanism includes: All nodes in the reconnected subgraph are divided according to preset values ​​to obtain multiple node partitions; The data to be detected for each node partition in the plurality of node partitions is determined according to the bidirectional indexing mechanism; The step of using the data detection rules to detect the data to be detected and determining the data consistency detection result includes: The data detection rules are used to perform parallel detection on the data to be detected in each node partition to obtain the data consistency detection result.

8. A device for detecting data lineage, characterized in that, include: The construction module is used to build data query objects based on the set of nodes to be detected in the current batch; The query module is used to perform multi-hop queries in the graph database using the data query object to obtain a query subgraph; wherein, the query subgraph includes the bloodline edges associated with the nodes in the set of nodes to be detected and the edge attribute values ​​of the bloodline edges; The filtering module is used to filter the set of predecessor nodes and the set of successor nodes associated with the temporary nodes in the set of temporary nodes in the query subgraph. The modification module is used to modify the query subgraph based on the predecessor node set and the successor node set to obtain a reconnected subgraph; The detection module is used to detect the reconnected subgraph using predetermined data detection rules to obtain data consistency detection results; wherein, the data consistency detection results characterize whether the source field metadata and target field metadata of each edge in the reconnected subgraph are consistent.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-7.

Images