Data management method based on data blood relationship tracking

Abstract

The invention relates to the technical field of big data processing and data security, and discloses a data management method based on data consanguinity tracking, which comprises the following steps: intercepting input data at a data reading operator and initializing a source data taint tag; in an instruction execution process, performing explicit propagation of stain labels and implicit flow dependency processing based on context stacks; identifying a high-polymerization-degree or desensitization operation, and calling a label degradation function to generate a coarse-grained or desensitization identifier; analyzing the final stain label at a data write-in operator, and constructing a global blood relationship map; and positioning a source based on the final stain label and generating an influence path. According to the method, through instruction-level dynamic analysis and a dynamic label degradation mechanism, fine-grained tracking of the full life cycle of the data is realized, and the problem of metadata expansion caused by implicit dependency tracking missing and aggregation operation is effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of big data processing and data security technology, specifically a data governance method based on data lineage tracing. Background Technology

[0002] With the continuous growth of data volume, data lineage analysis has become a core tool for enterprises to conduct data governance, compliance audits, and root cause analysis of failures. Currently, the conventional technical means of constructing data lineage mainly relies on static analysis of SQL scripts, stored procedures, or task execution plans. This static analysis method focuses on the syntactic structure and logical definition of the code, but it has limitations when facing the complex real-world production environment. Because static analysis is detached from the actual runtime environment of the code, when data processing tasks involve dynamic parameter passing, complex custom function calls, or external dependencies that can only be determined at runtime, static analysis often fails to accurately capture the actual data flow, easily leading to missing or misjudged lineage relationships.

[0003] To address the accuracy issues of static analysis, some technical solutions attempt to employ dynamic taint tracing techniques. However, existing dynamic tracing solutions often struggle to balance tracing granularity with system performance when handling large-scale data computations. Especially in scenarios involving highly aggregated operations such as counting, summing, or averaging, traditional fine-grained tracing mechanisms directly pass labels to all input data, leading to a dramatic increase in the size of the metadata labels for the output data. This metadata overload not only consumes significant memory and I / O resources but can also cause computational tasks to crash in severe cases. Furthermore, existing lineage tracing techniques typically focus only on explicit data flows such as assignment and computation, often neglecting implicit data dependencies arising from conditional branches. This results in incomplete lineage graphs in scenarios involving complex business logic control, failing to accurately reflect the causal paths of data generation. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a data governance method based on data lineage tracing. This method solves the problems that existing data lineage technologies rely heavily on static parsing and lack accuracy in dynamic operating environments, while conventional dynamic tracing technologies face issues such as metadata bloat caused by aggregation operations and ignoring implicit control flow dependencies.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data governance method based on data lineage tracing, comprising the following steps:

[0006] S1. Intercept input data at the data reading operator, determine the smallest traceable unit and generate a source identifier, and initialize a source data taint label containing the source identifier for the input data;

[0007] S2. During the execution of instructions by the computing engine, intercept data operation instructions and propagate taint labels according to preset rules. The propagation includes propagation of explicit data dependencies and implicit flow dependency processing for control flow instructions.

[0008] S3. Identify the type of the current data operation instruction. When the data operation instruction is a preset high-aggregation operation or desensitization operation, call the label degradation function to degrade the taint label of the input data and generate a degraded taint label containing a coarse-grained identifier or a desensitization identifier.

[0009] S4. Intercept the output data at the data writing operator, parse the final taint label of the output data, convert the final taint label into a lineage tuple and send it to the graph database to construct a global lineage graph.

[0010] S5. Receive a reverse tracing query for the target data, obtain the final taint label of the target data, locate the source data based on the identifier set in the final taint label, and generate an influence path in combination with the global lineage graph.

[0011] Preferably, in step S1, the source data taint tags are stored in memory using a bit vector structure;

[0012] The initialization of the source data taint label containing the source identifier includes: establishing a mapping function from the source identifier to the bit vector index, generating a binary sequence as the source data taint label, and setting the bit value of the index corresponding to the source identifier to 1;

[0013] A weak reference mapping table is established between the memory address of the original data object and the memory address of the source data taint tag, and the source data taint tag is managed as shadow metadata.

[0014] Preferably, in step S2, the implicit flow dependency processing for control flow instructions includes:

[0015] Maintain a thread-private context stack in memory;

[0016] When a control flow instruction is intercepted, the conditional variables on which the control flow instruction depends are identified, the taint labels of the conditional variables are obtained and pushed onto the context stack.

[0017] When generating new data within a code block defined by the control flow instructions, the union of all tags in the context stack is calculated as an environmental taint tag, and the environmental taint tag is injected into the taint tag of the new data;

[0018] When the execution flow leaves the scope of the code block, a pop operation is performed on the context stack.

[0019] Preferably, in step S2, the propagation of explicit data dependencies includes:

[0020] Retrieve taint labels for all input operands of the current data manipulation instruction;

[0021] Calculate the union of taint labels for all input operands, and associate the calculated union with the output data generated by the current data manipulation instruction;

[0022] When the taint tags are stored as bit vectors, the union of the taint tags for all input operands is calculated using a bitwise OR operation.

[0023] Preferably, in step S3, before calling the label degradation function to downgrade the tainted labels of the input data, it further includes determining whether the degradation trigger condition is met:

[0024] Determine whether the size of the input data set for the current data operation instruction exceeds a preset threshold;

[0025] If so, then the operation of calling the tag degradation function is triggered;

[0026] The degradation process includes collecting taint tags from all data units in the input data set to form an input tag set.

[0027] Preferably, the step of calling the label degradation function to downgrade the taint labels of the input data specifically includes:

[0028] When the data operation instruction is a high-aggregation operation, an aggregation degradation strategy is executed: a coarse-grained identifier representing the current operation or data batch is generated, and the coarse-grained identifier is used to replace all source identifiers in the input label set.

[0029] When the data operation instruction is a de-identification operation, a de-identification and degradation strategy is executed: a de-identification identifier is generated, and the de-identification identifier is used to replace all source identifiers in the input tag set;

[0030] The generated downgraded taint label after replacement is associated with the output data of the data manipulation instruction, and the shadow metadata mapping in memory is updated.

[0031] Preferably, in step S4, converting the final blemish label into a kinship tuple and sending it to the graph database to construct a global kinship graph includes:

[0032] Generate a target identifier for the output data;

[0033] Extract all source identifiers, coarse-grained identifiers, and desensitized identifiers contained in the final taint label, and combine them with the target identifier to generate a lineage tuple;

[0034] The lineage relationship tuple is sent to the graph database processing service using an asynchronous communication mechanism, and the graph database processing service writes the lineage relationship tuple into the graph database.

[0035] Preferably, the construction of the global pedigree map further includes:

[0036] The granularity change information generated in step S3 is stored as metadata in the nodes or edges of the global lineage graph;

[0037] The granularity change information includes the downgrade operation type, downgrade timestamp, and the granularity level before downgrade.

[0038] The global lineage graph is periodically topologically sorted, and redundant lineage paths in the graph are cleaned up.

[0039] Preferably, in step S5, locating the source data based on the identifier set within the final blemish label includes:

[0040] Parse the final taint label and extract the identifier set;

[0041] Traverse the set of identifiers: if the identifier is a source identifier, obtain the physical storage path and offset based on the mapping relationship established in step S1; if the identifier is a coarse-grained identifier or anonymized identifier generated in step S3, locate the dataset range or data batch information based on the corresponding aggregated metadata.

[0042] A data governance system based on data lineage tracing includes:

[0043] The source data interception module is used to intercept input data at the data reading operator, establish a source identifier, and initialize source data taint labels;

[0044] The taint propagation module is used to intercept data manipulation instructions, perform taint label propagation for explicit data dependencies, and maintain a context stack to handle implicit stream dependencies.

[0045] The label degradation module is used to identify high-aggregation operations or desensitization operations, and when the conditions are met, it calls the label degradation function to generate a downgraded taint label containing a coarse-grained identifier or a desensitization identifier.

[0046] The graph construction module is used to intercept output data at the data writing operator, convert the final taint label into a lineage tuple and construct a global lineage graph.

[0047] The reverse analysis module is used to receive reverse tracing queries, locate the source data using the final taint label, and generate the influence path by combining it with the global lineage map.

[0048] This invention provides a data governance method based on data lineage tracing. It has the following beneficial effects:

[0049] 1. This invention intercepts data operations at the instruction execution layer of the computing engine and maintains the context stack to handle implicit flow dependencies, which can accurately capture logical branches and indirect data dependencies during dynamic runtime. Compared with conventional techniques that rely solely on static SQL parsing, this method achieves fine-grained tracing at the instruction level, solving the problem of broken or lost lineage relationships caused by complex code logic or runtime condition judgments, thereby significantly improving the completeness and accuracy of data lineage tracing.

[0050] 2. This invention introduces a tag degradation mechanism for high aggregation and desensitization operations, which can automatically convert massive fine-grained tags into coarse-grained or desensitized identifiers according to the semantics of the operation. This processing method effectively avoids the problem of excessive memory consumption caused by the rapid expansion of metadata when performing large-scale data aggregation calculations. At the same time, it enables the generated lineage relationship to accurately reflect the granularity changes and semantic transformations of data during the processing, achieving a balance between tracking depth and system performance.

[0051] 3. This invention adopts a strategy of storing shadow metadata based on bit vectors and asynchronously constructing the genealogy, which supports efficient reverse tracing queries. When it is necessary to locate the source of problematic data, the system only needs to parse the identifier set in the target data tag to directly index the physical source data. There is no need to execute complex path traversal algorithms in the global genealogy, which greatly improves the response speed of genealogy analysis. At the same time, the asynchronous writing mechanism ensures that the data governance process will not block the execution of core computing tasks. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall system architecture of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example:

[0056] Please see the appendix Figure 1 This invention provides a data governance method based on data lineage tracing, comprising the following steps:

[0057] S1. Intercept input data at the data reading operator, establish a source identifier, and initialize the source data taint label corresponding to the input data unit;

[0058] S2. During the execution of instructions by the computing engine, intercept data operation instructions, propagate taint labels according to the set operation rules of operand labels, and handle implicit flow dependencies for control flow instructions.

[0059] S3. Identify the current operation instruction type. When the operation instruction is a preset high-aggregation operation or desensitization operation, call the label degradation function to downgrade the tainted label.

[0060] S4. Intercept the output data at the data writing operator, parse the finally generated taint labels, convert the taint labels into lineage tuples and send them to the graph database to build a global lineage graph.

[0061] S5. Receive the reverse tracing query for the target data, obtain the final taint label of the target data, use the source identifier set in the label to locate the source data, and generate the influence path by combining the global lineage graph.

[0062] S6. Perform topological sorting and redundant path cleanup on the global lineage graph, and store granularity change information as metadata in the graph nodes.

[0063] In this embodiment, data lineage tracing begins the moment the data enters the computing environment. The source data interception and taint label initialization mechanism can be implemented through the following sub-steps:

[0064] The dynamic runtime instrumentation module intercepts data read operations and identifies the data source access point. It monitors the underlying execution flow of the computing engine and identifies the source operators used to load external data. These source operators include, but are not limited to, file system read interfaces, database connector result set read interfaces, or message queue consumer interfaces. When the computing engine calls these interfaces to load data, the dynamic runtime instrumentation module triggers interception logic to obtain the metadata context of the current read operation. This metadata context includes the physical path of the data source, table name, partition information, and the data mode being read. For the specific implementation techniques of the underlying bytecode instrumentation, such as modifying class definitions using JavaAgent technology or rewriting function entry points using Python decorators, those skilled in the art can configure them according to the language characteristics of the computing engine.

[0065] Define the smallest traceable unit and generate a source identifier. Based on a pre-defined data governance granularity strategy, determine the smallest traceable unit for the input data. The smallest traceable unit It can be configured at the file level, table level, data shard level, or row record level. The system maintains a globally unique source identifier space. For each intercepted smallest traceable unit The system generates or assigns a unique source identifier based on its metadata context. ,in In row-level tracing mode, the source identifier It can be generated by combining the unique code of the data table with the offset of the row in the storage medium.

[0066] Construct and initialize taint tags for source data. The core of taint tag propagation is each smallest traceable unit. Instantiate taint tags The stained label Defined as a set data structure containing source identifiers. During initialization, this set contains only the source identifier of the current data unit itself. Its initialization relation is mathematically expressed as follows:

[0067] ;

[0068] In the formula, Indicates input data unit The blemish label in its initial state. This represents the unique source identifier assigned to this data unit. This represents set operations.

[0069] To optimize memory usage and processing speed for subsequent propagation calculations, this embodiment implements the aforementioned set structure as a bit vector. The system establishes a source identifier. Position vector subscript mapping function Initial stain label Represented as a binary sequence, where the first... Bit 1 is set to 1, and the remaining bits are set to 0.

[0070] Establish shadow metadata storage mapping. Initialize taint tags. Do not directly modify the original data Instead of storing the binary content, it stores it as shadow metadata in memory. For object-based computing engines, the system establishes a weak reference mapping table in heap memory between the addresses of original data objects and taint tag objects; for column-oriented computing engines, the system allocates a memory region parallel to the data columns for contiguous storage of the corresponding tag sequences. This storage method ensures the binding of taint tags to the data lifecycle. When the original data is garbage collected, the corresponding taint tag storage space is released simultaneously, preventing memory leaks.

[0071] In this embodiment, the construction of data lineage relies on real-time monitoring and tag status updates of each data operation instruction during the runtime of the computing engine. This process can be implemented through the following sub-steps:

[0072] Configure instruction-level instrumentation probes and establish operation mappings. The dynamic runtime instrumentation module establishes interception points at the instruction execution layer of the computing engine. For Java Virtual Machine-based computing frameworks, it uses JavaAgent technology to modify bytecode during the class loading phase or uses the JVMTI interface to listen for instruction events. For interpreted language frameworks, it uses metaclass programming or interpreter hook mechanisms to intercept function calls. The system maintains an instruction operation mapping table, dividing the computing engine's instruction set into data movement instructions, arithmetic and logical operation instructions, and control flow instructions. Instrumentation probes are configured to trigger only when executing instructions of the above types, obtaining the memory address of the input data object by directly accessing the operand stack or registers, and then indexing the corresponding shadow metadata, i.e., taint tags.

[0073] The taint label propagation core performs taint label propagation calculations based on the display data flow dependency rules. When the computation engine performs data transformation operations that generate new data objects, the taint label propagation core calculates the taint labels for the output data according to these rules. Let the currently executing instruction be... Its set of input operands is The generated output data object is The system retrieves each input operand. (in The corresponding current blemish label Output data Stain label It is calculated as the union of the taint labels of all input operands. Its propagation logic is mathematically expressed as follows:

[0074] ;

[0075] In the formula, Indicates taint labels in the output data. Indicates the first A taint label for each input operand. This represents the union operation of sets.

[0076] When taint tags are stored using a bit vector structure, the above set union operation is mapped to the underlying bitwise OR instruction. For each input operand's bit vector, the system performs a bitwise OR operation to generate a result bit vector and associates it with the output data object. This calculation process is completed directly in memory, avoiding the overhead of complex graph traversal or object creation.

[0077] Implicit control flow dependencies are captured and a context stack is maintained. To address indirect data dependencies caused by conditional branching, an implicit flow processing mechanism is introduced. The taint label propagation core maintains a thread-private context stack in memory. .

[0078] When the instrumentation module detects a control flow instruction during dynamic runtime, it identifies the conditional variables that the instruction depends on. The system retrieves the taint labels of the conditional variables. and push it onto the context stack. .

[0079] Within the code block defined by control flow instructions, new data is generated for each instruction. The instruction's label calculation logic, in addition to explicit propagation, requires the further integration of path dependency information from the current context stack. The system calculates the union of all labels in the context stack as an environmental taint. This data is then injected into the labels of the new data. The logical expression is as follows:

[0080] ;

[0081] ;

[0082] In the formula, This indicates the environmental taint labels accumulated in the current execution path. Represents the context stack Each tag element in the text, This represents the union of explicit operand labels obtained from the calculation. This indicates the final taint label that incorporates implicit flow dependencies. When the execution flow leaves the scope of the current control flow code block, the system performs a stack pop operation, removing the corresponding condition variable label from the stack. The data is removed from the context, thus restoring the context state of the previous level. This mechanism ensures that data lineage not only reflects numerical calculation relationships but also accurately reflects the impact of logical judgments on the data generation path.

[0083] In this embodiment, a dynamic tag reduction mechanism is introduced for high aggregation or desensitization operations to address the issues of inaccurate lineage semantics and tag redundancy in these specific scenarios of native taint propagation. This process ensures that the granularity of lineage tracing can adapt to semantic changes in data processing. This process can be implemented through the following sub-steps:

[0084] Establish an operation semantic classification library and identify specific operation types. The system pre-establishes an operation semantic classification library. This classification library records data manipulation functions with high aggregation or desensitization effects. The identifier. High-cohesion operations refer to operations whose output data has significantly reduced dependence on any single element in the input set. De-identification operations refer to operations that drastically change data content through irreversible functions or data replacement methods to achieve privacy protection. When executing instruction-level interception, the dynamic runtime instrumentation module first queries... When the intercepted command Corresponding operations belong Recorded in When this occurs, the downlabeling processing logic is triggered.

[0085] Determine the trigger condition for label downgrading and collect the input label set. The core of taint label propagation is to determine the current... Does it meet the preset tag downgrade trigger conditions? For The operation and trigger conditions can be set to the size of the input dataset. Exceeding a certain threshold If the triggering condition is met, the system will retrieve data from the input dataset. The taint labels of all data units are collected to form the input label set. :

[0086] ;

[0087] In the formula, This represents the set of taint labels for all input data units. This represents the input dataset for the operation. Represents the dataset Middle Data Unit The stain label.

[0088] Execute the tag degradation function based on operation semantics. The core of taint tag propagation involves calling the tag degradation function. right Process the issue to generate a new downgraded taint label. The processing logic of the degradation function depends on... Specific semantic meaning:

[0089] Aggregation degradation strategy: applicable Equal-height aggregation operation. In this case, the output result... Semantically no longer strongly dependent on the input set Each fine-grained source identifier in The system will All fine-grained source identifiers Replace with coarse-grained dataset identifiers Or aggregate batch identifier ,in It is a set of coarse-grained identifiers, and Degradation function The definition is as follows:

[0090] ;

[0091] In the formula, It is a newly generated, unique identifier used to represent all inputs of this aggregation operation.

[0092] Desensitization and downgrade strategies: applicable to Desensitization operations are performed. In this case, although the output... Still with input A functional relationship exists, but the original source identifier... It has lost its direct personal traceability significance. The system will The original source identifier in Replace with desensitized identifier This identifier refers to the de-identification result of the original data or a special marker indicating the source of the de-identification. Degradation function The definition is as follows:

[0093] ;

[0094] In the formula, Indicates that the label All of them Mapping Replacement The function.

[0095] The downgraded tags are associated with the output data, and the memory mapping is updated. The generated downgraded taint tags... Replaces the explicit stain label calculated in step S2 And is associated with the output data object. Above. The core of taint label propagation synchronously updates the shadow metadata storage mapping, ensuring that subsequent instruction propagation continues based on this semantically corrected label. The label downsizing mechanism enables the lineage graph to accurately reflect semantic and granular changes in data while reflecting the data source, avoiding the infinite expansion of the label set after high aggregation operations.

[0096] In this embodiment, the dynamic kinship information generated by the data processing task is solidified during the data writing phase and used to construct and maintain the global kinship map. This process can be implemented through the following sub-steps:

[0097] Intercept data write operations and obtain the final taint label. The dynamic runtime instrumentation module monitors the underlying execution flow of the compute engine, identifying data write operator operations, including the process of outputting data to persistent storage. When a write operation is triggered, the dynamic runtime instrumentation module intercepts the output data unit. And retrieve its final taint label from the shadow metadata storage map in memory. The final taint label contains all the dependency information for the data unit from its source to its final destination, including explicit propagation, implicit flow, and traces of degradation processing.

[0098] Parse taint tags and generate kinship tuples. The kinship graph construction module receives... and The system first provides the target data unit. Generate a unique target identifier The pedigree mapping module parses blemish tags. Extract all source identifiers contained therein. Whether it is fine-grained Still coarse-grained The system maps these identifiers to a set of lineage tuples RR, where each tuple represents a lineage edge from the source to the target:

[0099] ;

[0100] In the formula, This represents the set of bloodline tuples generated by this write operation. From The source identifier extracted from it. It is the target data unit The identifier.

[0101] Asynchronous writing of kinship tuples and kinship graph construction. The kinship graph construction module uses an asynchronous communication mechanism, such as a message queue, to write the generated kinship tuples. The data is pushed to the backend graph database processing service. The graph database processing service is responsible for processing the tuples. Write to a distributed graph database to build a global lineage graph. Among them, data identifiers (including , , (a set of nodes in a graph) Blood relation tuple As a set of directed edges The asynchronous write mechanism ensures that the performance of data processing tasks is not affected by the I / O latency of pedigree map write operations.

[0102] Regular topological sorting and redundancy cleanup of phylogenetic graph data. The phylogenetic graph construction module periodically performs topological sorting and redundancy cleanup on the global phylogenetic graph. Maintenance is performed. Maintenance operations include topological sorting of the graph and verifying its acyclicity. Simultaneously, to optimize query efficiency and storage space, the system identifies and cleans up redundant lineage paths: if a path exists... and direct edge ,and right The contribution can be fully achieved through and This demonstrates that the system can selectively delete or mark redundant edges according to a preset strategy. For the maintenance and optimization of the graph database, those skilled in the art can use well-known graph algorithms.

[0103] Store granularity changes and semantic metadata. The kinship map construction module stores the granularity change information generated by the dynamic downlabeling process in step S3. As metadata, it is stored in the corresponding target node. Or on the edge of blood relations. This metadata includes the downgrade operation type. The downgrade timestamp and the granularity level before the downgrade.

[0104] ;

[0105] In the formula, Represents the relationship between graph nodes Associated metadata It contains specific information about the granularity changes that have occurred within the data unit. This mechanism provides semantic context for upper-level data governance strategies, allowing users to distinguish between fine-grained lineage and aggregated, anonymized lineage.

[0106] Process and store intermediate lineage fragments without downgraded tags. For intermediate computation results that are destroyed before data is written. If its tainted label Containing important lineage information, the instrumentation module will... Before the end of its life cycle, remove the intermediate bloodline fragments The data is extracted and written to a dedicated intermediate lineage buffer asynchronously. This buffer can then be used as a reference for... This is a supplement for in-depth tracing in specific scenarios.

[0107] In this embodiment, the runtime reverse impact analysis module utilizes the stored taint label data to achieve real-time and accurate reverse tracing capability of target data. This process can be implemented through the following sub-steps:

[0108] Receives reverse trace query requests. The runtime reverse impact analysis module provides a user-facing query interface. Receive data units for a specific target The reverse trace query request. Target data unit By unique target identifier This identifier corresponds to a field in the final report, a database record, or a distributed file block.

[0109] Obtain the final taint label for the target data. The runtime back-impact analysis module is based on the target identifier. Index and read the final taint label of the target data unit from the graph database or label caching service. .

[0110] Source data location is performed based on a set of taint tags. Runtime reverse impact analysis module performs the analysis. The set of source identifiers included .gather The definition is as follows:

[0111] ;

[0112] In the formula, This is the source identifier.

[0113] For each identifier in the set The system executes a hierarchical positioning strategy: if Fine-grained source identifier The system queries the source mapping table established in the source data interception stage of step S1 to obtain... The corresponding physical storage path and offset information are used to locate the specific original record; if The coarse-grained identifier generated for the downgrade process in step S3 The system queries the corresponding aggregated metadata to locate the dataset or data batch range represented by the identifier. By directly parsing the tag set, the system can pinpoint the physical location of all relevant source data without traversing the lineage map.

[0114] Accurate impact paths are generated by combining the global graph. The runtime reverse impact analysis module uses... Each identifier in As the starting point of the path, As the endpoint of the path, in the global lineage graph The path search algorithm is executed. The path search algorithm calculates the connected path from the starting point to the ending point. :

[0115] ;

[0116] Due to the starting set As explicitly defined by the taint label, the path search algorithm only applies to... The process is performed within the subgraph of the root node; the system will calculate the set of paths. The output serves as the result of the reverse impact analysis.

[0117] Please see the appendix Figure 2 A data governance system based on data lineage tracing, comprising:

[0118] The source data interception module is used to intercept input data at the data reading operator, establish a source identifier, and initialize source data taint labels;

[0119] The taint propagation module is used to intercept data manipulation instructions, perform taint label propagation for explicit data dependencies, and maintain a context stack to handle implicit stream dependencies.

[0120] The label degradation module is used to identify high-aggregation operations or desensitization operations, and when the conditions are met, it calls the label degradation function to generate a downgraded taint label containing a coarse-grained identifier or a desensitization identifier.

[0121] The graph construction module is used to intercept output data at the data writing operator, convert the final taint label into a lineage tuple and construct a global lineage graph.

[0122] The reverse analysis module receives reverse tracing queries, uses the final taint label to locate the source data, and combines it with the global lineage map to generate the impact path.

[0123] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data governance method based on data lineage tracing, characterized in that, Includes the following steps: S1. Intercept input data at the data reading operator, determine the smallest traceable unit and generate a source identifier, and initialize a source data taint label containing the source identifier for the input data; S2. During the execution of instructions by the computing engine, intercept data operation instructions and propagate taint labels according to preset rules. The propagation includes propagation of explicit data dependencies and implicit flow dependency processing for control flow instructions. S3. Identify the type of the current data operation instruction. When the data operation instruction is a preset high-aggregation operation or desensitization operation, call the label degradation function to degrade the taint label of the input data and generate a degraded taint label containing a coarse-grained identifier or a desensitization identifier. S4. Intercept the output data at the data writing operator, parse the final taint label of the output data, convert the final taint label into a lineage tuple and send it to the graph database to construct a global lineage graph. S5. Receive a reverse tracing query for the target data, obtain the final taint label of the target data, locate the source data based on the identifier set in the final taint label, and generate an influence path in combination with the global lineage graph.

2. The data governance method based on data lineage tracing according to claim 1, characterized in that, In step S1, the source data taint tags are stored in memory using a bit vector structure; The initialization of the source data taint label containing the source identifier includes: establishing a mapping function from the source identifier to the bit vector index, generating a binary sequence as the source data taint label, and setting the bit value of the index corresponding to the source identifier to 1; A weak reference mapping table is established between the memory address of the original data object and the memory address of the source data taint tag, and the source data taint tag is managed as shadow metadata.

3. The data governance method based on data lineage tracing according to claim 1, characterized in that, In step S2, the implicit flow dependency processing for control flow instructions includes: Maintain a thread-private context stack in memory; When a control flow instruction is intercepted, the conditional variables on which the control flow instruction depends are identified, the taint labels of the conditional variables are obtained and pushed onto the context stack. When generating new data within a code block defined by the control flow instructions, the union of all tags in the context stack is calculated as an environmental taint tag, and the environmental taint tag is injected into the taint tag of the new data; When the execution flow leaves the scope of the code block, a pop operation is performed on the context stack.

4. The data governance method based on data lineage tracing according to claim 1, characterized in that, In step S2, the propagation of explicit data dependencies includes: Retrieve taint labels for all input operands of the current data manipulation instruction; Calculate the union of taint labels for all input operands, and associate the calculated union with the output data generated by the current data manipulation instruction; When the taint tags are stored as bit vectors, the union of the taint tags for all input operands is calculated using a bitwise OR operation.

5. The data governance method based on data lineage tracing according to claim 1, characterized in that, In step S3, before calling the label degradation function to downgrade the tainted labels of the input data, it is also necessary to determine whether the degradation trigger condition is met: Determine whether the size of the input data set for the current data operation instruction exceeds a preset threshold; If so, then the operation of calling the tag degradation function is triggered; The degradation process includes collecting taint tags from all data units in the input data set to form an input tag set.

6. The data governance method based on data lineage tracing according to claim 5, characterized in that, The process of calling the label degradation function to downgrade the taint labels on the input data specifically includes: When the data operation instruction is a high-aggregation operation, an aggregation degradation strategy is executed: a coarse-grained identifier representing the current operation or data batch is generated, and the coarse-grained identifier is used to replace all source identifiers in the input label set. When the data operation instruction is a de-identification operation, a de-identification and degradation strategy is executed: a de-identification identifier is generated, and the de-identification identifier is used to replace all source identifiers in the input tag set; The generated downgraded taint label after replacement is associated with the output data of the data manipulation instruction, and the shadow metadata mapping in memory is updated.

7. A data governance method based on data lineage tracing according to claim 1, characterized in that, In step S4, converting the final taint label into a kinship tuple and sending it to the graph database to construct a global kinship graph includes: Generate a target identifier for the output data; Extract all source identifiers, coarse-grained identifiers, and desensitized identifiers contained in the final taint label, and combine them with the target identifier to generate a lineage tuple; The lineage relationship tuple is sent to the graph database processing service using an asynchronous communication mechanism, and the graph database processing service writes the lineage relationship tuple into the graph database.

8. A data governance method based on data lineage tracing according to claim 6, characterized in that, The construction of the global lineage map also includes: The granularity change information generated in step S3 is stored as metadata in the nodes or edges of the global lineage graph; The granularity change information includes the downgrade operation type, downgrade timestamp, and the granularity level before downgrade. The global lineage graph is periodically topologically sorted, and redundant lineage paths in the graph are cleaned up.

9. A data governance method based on data lineage tracing according to claim 1, characterized in that, In step S5, locating the source data based on the identifier set within the final blemish label includes: Parse the final taint label and extract the identifier set; Traverse the set of identifiers: if the identifier is a source identifier, obtain the physical storage path and offset based on the mapping relationship established in step S1; if the identifier is a coarse-grained identifier or anonymized identifier generated in step S3, locate the dataset range or data batch information based on the corresponding aggregated metadata.

10. A data governance system based on data lineage tracing, and a data governance method based on data lineage tracing according to any one of claims 1-9, characterized in that, include: The source data interception module is used to intercept input data at the data reading operator, establish a source identifier, and initialize source data taint labels; The taint propagation module is used to intercept data manipulation instructions, perform taint label propagation for explicit data dependencies, and maintain a context stack to handle implicit stream dependencies. The label degradation module is used to identify high-aggregation operations or desensitization operations, and when the conditions are met, it calls the label degradation function to generate a downgraded taint label containing a coarse-grained identifier or a desensitization identifier. The graph construction module is used to intercept output data at the data writing operator, convert the final taint label into a lineage tuple and construct a global lineage graph. The reverse analysis module is used to receive reverse tracing queries, locate the source data using the final taint label, and generate the influence path by combining it with the global lineage map.

Images