A method and system for upstream dual fingerprint parsing based on bloodline recursive closure detection
The upstream dual fingerprinting method based on lineage recursive closure detection solves the fingerprint offset and misjudgment problems caused by changes in source table field types, achieving more accurate type drift observation and verification, and improving the completeness and accuracy of lineage resolution.
Patent Information
- Application Number
- CN202611627743.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot effectively correct expired column types when the source table field type changes but the lineage snapshot is not updated synchronously, leading to fingerprint offset and misjudgment of upstream closure, affecting the consistency and accuracy of lineage parsing, type drift determination, and dual fingerprint verification of indicators.
An upstream dual fingerprinting method based on lineage recursive closure detection is adopted. The closure is constructed by path hash depth-first search. Combined with exponential decay staleness calculation and weighted risk quantization, type drift observation and judgment are performed. A three-channel normalization mechanism is used to process column types, realizing literal type normalization and closure consistency verification.
It improves the integrity and consistency of lineage analysis, reduces fingerprint offset and false alarms, enhances the accuracy and traceability of type change detection, and ensures that literal normalization remains stable and reversible when the type is uncertain.
Smart Images

Figure CN123115151A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic digital data processing technology, and in particular to an upstream dual fingerprinting method and system based on bloodline recursive closure detection. Background Technology
[0002] Currently, in metric development and data warehouse lineage management scenarios, metric nodes are typically generated from upstream fields through several filtering, aggregation, and transformation operations. Lineage information records the dependencies between nodes and the column types of each field in the lineage snapshot. To ensure that the literals in the filtering conditions participate in the calculation consistently with the upstream column types, existing processing paths generally first force-convert or format the literals in the filtering expression based on the column types recorded in the lineage snapshot, and then generate metric fingerprints or job fingerprints based on the normalized expressions for change detection and impact analysis. Some solutions also update column types by periodically synchronizing metadata, but the synchronization period is often inconsistent with the lineage snapshot generation time, which may result in a time window discrepancy between the snapshot column types and the actual metadata.
[0003] For example, when the data type of a source table field is changed from string to numeric or date, the column types recorded in the lineage snapshot are not updated in a timely manner. The existing literal normalization steps still use outdated snapshot column types to forcibly convert the filtered values. String-type filtered values may be retained according to string rules, while numeric types should have trailing zeros removed or uniform precision applied, and date types should be converted to standard date formats. Once the conversion rules do not match the current actual types, the filtered values will lose comparability or undergo incorrect normalization, leading to a shift in the subsequently generated indicator fingerprints. Meanwhile, if upstream closure construction relies solely on snapshot types for node summary calculation, type expiration can cause the same logical change to be misjudged as a semantic change, or cause actual type changes to be missed as stable, affecting downstream alerts and dependency analysis.
[0004] Therefore, there is an urgent need for a method that can still correct expired column types and stably handle unknown types even when the source table field types change but the lineage snapshot is not updated synchronously, so that the literal normalization results are not affected by type changes, thereby improving the consistency, accuracy and traceability of lineage parsing, type drift determination and indicator dual fingerprint verification. Summary of the Invention
[0005] To address the aforementioned technical shortcomings, the present invention aims to propose an upstream dual fingerprint parsing method and system based on lineage recursive closure detection. This method aims to solve the technical problems in the prior art that rely on the forced conversion of lineage snapshot column types to filter literals, especially when the source table field type changes but the snapshot is not updated synchronously. This results in the inability to correct expired column types and handle unknown types stably, leading to fingerprint offset and misjudgment of upstream closure.
[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides an upstream dual fingerprint parsing method and system based on bloodline recursive closure detection, wherein the method includes:
[0007] Step S10: Obtain the upstream lineage information of the current indicator node. Based on the upstream lineage information of the current indicator node, perform the upstream closure construction task using the path hash depth-first search mechanism, and output the lineage closure parsing result.
[0008] Step S20: Based on the bloodline closure analysis results, perform the type drift observation task using the exponential decay staleness calculation mechanism, and output the type drift observation results;
[0009] Step S30: Based on the type drift observation results, perform the type drift risk determination task using a weighted risk quantification mechanism, and output the type drift determination result;
[0010] Step S40: Based on the type drift determination result, a three-channel normalization mechanism is used to execute the column type normalization context generation task, and the column type normalization processing data is output.
[0011] Step S50: Based on the column type normalization data, a dual fingerprint verification mechanism is used to perform literal type normalization and closure consistency verification tasks, and output the dual fingerprint parsing results.
[0012] Preferably, step S10, which involves obtaining the upstream lineage information of the current indicator node, performing an upstream closure construction task based on the upstream lineage information of the current indicator node using a path hash depth-first search mechanism, and outputting the lineage closure parsing result, specifically includes:
[0013] Step S101: Obtain the current indicator node, upstream bloodline edge, bloodline snapshot and original filter expression abstract syntax tree of each node from the upstream bloodline information. Using the current indicator node as the initial node, perform depth-first expansion according to the upstream bloodline edge, add the direct upstream nodes of the included nodes to the closure, and construct the closure node set until the closure node set no longer changes.
[0014] Step S102: During the expansion process, the path hash is carried. The path hash of the current node, the fixed arrow separator, and the stable identifier of the next node are concatenated in sequence to calculate the SHA-256 digest and update the path hash of the next node. When the same node arrives through different upstream paths, the path information is retained. When the same node appears repeatedly on the same path, it is marked as a node and the expansion is stopped. The set of closure nodes with retained path information is obtained.
[0015] Step S103: For each node in the closure node set, the sorted and concatenated values of the operator, field name, stable representation of the original literal, snapshot column type, and child node hash are sequentially concatenated using a fixed vertical bar separator and a SHA-256 digest is calculated to obtain the node hash; the node hashes are sorted, stably concatenated, and a SHA-256 digest is calculated to obtain the lineage closure digest; a column reference state matrix is constructed that records the node identifier, source table identifier, column name, snapshot column type, and snapshot version number; the closure node set, the path information, the lineage snapshot, the original filter expression abstract syntax tree, the lineage closure digest, and the column reference state matrix are combined to form the lineage closure parsing result.
[0016] Preferably, step S20, which involves performing a type drift observation task based on the bloodline closure analysis result using an exponential decay staleness calculation mechanism and outputting the type drift observation result, specifically includes:
[0017] Step S201: Obtain the column reference state matrix from the bloodline closure parsing result. For each column reference in the column reference state matrix, collect the current metadata column type, snapshot generation time and current time from the metadata service, collect a set of literal samples from the query log or filter conditions, and construct a type drift observation vector in combination with the snapshot column type.
[0018] Step S202: Based on the type drift observation vector, obtain the normalized write frequency of the table to which each column belongs, and use exponential decay processing with the attenuation coefficient adjusted with the normalized write frequency to calculate the snapshot staleness of each column.
[0019] Step S203: Combine the bloodline closure parsing result, the type drift observation vector, and the snapshot staleness to form the type drift observation result.
[0020] Preferably, in step S202, the step of obtaining the normalized write frequency of the table to which each column belongs based on the type drift observation vector, and calculating the snapshot staleness of each column by using exponential decay processing with an attenuation coefficient adjusted according to the normalized write frequency, specifically includes: normalizing the write frequency according to the number of writes in the data definition language or data manipulation language of the table to which each column belongs in the last 30 days to obtain the normalized write frequency with a value of 0 to 1, and calculating the snapshot staleness. The calculation relationship of the snapshot staleness is as follows:
[0021]
[0022] in, Reference to the current column, For the snapshot age of this column, The preset baseline attenuation coefficient is set to 0.05 per day. The preset write frequency amplification factor is set to 0.3. The normalized write frequency for the table to which this column belongs. The difference between the current time and the snapshot generation time in this column, in days. The natural exponential function is used to form the type of drift observation results.
[0023] Preferably, step S30, which involves performing a type drift risk determination task based on the type drift observation results using a weighted risk quantification mechanism and outputting the type drift determination result, specifically includes:
[0024] Step S301: Obtain the snapshot column type, current metadata column type, literal sample set and snapshot staleness of each column referenced from the type drift observation results. According to the preset type category semantic distance and similar precision loss rules, add the type category semantic distance and similar precision loss to obtain the type semantic distance.
[0025] Step S302: For each literal in the literal sample set, use the parser of the current metadata column type to parse it, and obtain the parsing confidence with a value of 0 to 1. If it cannot be parsed, take zero. Calculate the average parsing confidence to obtain the average interpretability.
[0026] Step S303: The type semantic distance, the snapshot staleness, and the uninterpretability (obtained by subtracting the average interpretability from one) are weighted and summed using a preset weighting coefficient to obtain a type drift risk score. The type drift risk value is obtained by subtracting the negative value of the type drift risk score from the natural exponent value. A preset first judgment threshold and a preset second judgment threshold greater than the first judgment threshold are set. When the snapshot column type or the current metadata column type is an unknown type, an unknown drift label is directly generated. Otherwise, when the type drift risk value is less than the first judgment threshold, a stable label is generated; when it is not less than the first judgment threshold and less than the second judgment threshold, an unknown drift label is generated; when it is not less than the second judgment threshold, a true drift label is generated. The judgment label, the type drift risk value, and the type drift observation result are combined to form the type drift judgment result.
[0027] Preferably, step S40, based on the type drift determination result, employs a three-channel normalization mechanism to perform a column type normalization context generation task and outputs column type normalization processed data, specifically includes:
[0028] Step S401: Obtain the judgment label, snapshot column type, current metadata column type, and lineage snapshot of each column reference from the type drift judgment result; read the type correction version number of each column reference; establish a valid type mapping and column type correction record set; the judgment label is a stable label, a true drift label, or an unknown drift label; for column references with the stable label, determine the valid type as the snapshot column type, retain the type correction version number, and set the correction flag to uncorrected;
[0029] Step S402: In the effective type mapping, for column references with the real drift label, the effective type is determined to be the current metadata column type, a column type correction record is generated and added to the column type correction record set, the old version of the lineage snapshot is retained and a new version using the current metadata column type is generated, the type correction version number is incremented, and the correction flag is set to corrected.
[0030] Step S403: In the valid type mapping, for column references with the unknown drift label, the valid type is determined as an unknown security type, the type correction version number is retained, and the correction flag is set to uncorrected; for the literal corresponding to the unknown security type, the specific type parser is not called, the literal is converted into a UTF-8 byte sequence, and the fixed prefix "U:", the byte length padded with leading zeros to 5 bits, the colon separator, the URL-secure Base64 encoding of the byte sequence, the colon separator, and the first 64 hexadecimal characters of the SHA-256 digest of the byte sequence are concatenated in sequence to obtain the unknown security reversible encoding result; the valid type mapping formed by the valid type, type correction version number, and correction flag of each column reference is determined as the column type normalization context, and the column type normalization context, the column type correction record set, and the type drift determination result are combined to form the column type normalization processing data.
[0031] Preferably, step S50, which involves performing literal type normalization and closure consistency verification tasks using a dual fingerprint verification mechanism based on the column type normalization data, and outputting the dual fingerprint parsing results, specifically includes:
[0032] Step S501: Obtain the closure node set, lineage closure summary, original filter expression abstract syntax tree of each node, column type normalization context, and decision labels of each column reference from the column type normalization processing data; the column type normalization context records the valid type, type correction version number, and correction flag of each column reference, and the decision label is a stable label, a true drift label, or an unknown drift label; normalize the leaf node literals of the original filter expression abstract syntax tree according to the valid types to obtain a normalized abstract syntax tree, wherein string types are subjected to removal of leading and trailing whitespace and Unicode. For NFC standardization, numeric types are parsed into decimal numbers, trailing zeros are removed, and a normal decimal string is output. Date types are parsed and an eight-digit year-month-day representation is output. Boolean types are converted to lowercase strings. When numeric or date parsing fails, or when the valid type is an unknown security type, a specific type parser is not called for forced conversion. Instead, the literal is converted into a UTF-8 byte sequence, and then a fixed prefix "U:", a byte length padded with leading zeros to 5 digits, a colon separator, the URL-secure Base64 encoding of the byte sequence, a colon separator, and the first 64 hexadecimal characters of the SHA-256 digest of the byte sequence are concatenated sequentially to obtain an unknown security reversible encoding result.
[0033] Step S502: For each node in the closure node set, determine the largest type correction version number within the current closure from the column type normalization context. Calculate the job fingerprint using the original filter expression abstract syntax tree and the lineage closure digest. Calculate the index fingerprint using the normalized abstract syntax tree, the column type normalization context, and the largest type correction version number. The calculation relationships between the job fingerprint and the index fingerprint are as follows:
[0034]
[0035]
[0036] in, and These are the job fingerprint and the indicator fingerprint, respectively. For SHA-256 digest function, For stable serialization functions, and These are the original filter expression abstract syntax tree and the normalized abstract syntax tree, respectively. This is the summary of the bloodline closure. For the normalization context of the column type, A stable string representation of the largest type's version number. This indicates stable string concatenation. For fixed version prefixes, Fixed separator;
[0037] Step S503: For each node in the closure node set, read the historical dual fingerprint and compare it with the new computed dual fingerprint composed of the job fingerprint and the indicator fingerprint; when both fingerprints remain unchanged, it is determined that there is no change; when only the job fingerprint changes, it is determined that only the expression form has changed and no new semantic version is created; when the job fingerprint remains unchanged, the indicator fingerprint changes, and the node has a column reference with the real drift label, it is determined to be a type drift correction, the indicator fingerprint index is updated, and the correction mark is propagated to downstream nodes without triggering a semantic change alarm; when the job fingerprint remains unchanged, the indicator fingerprint changes, and there is no column reference with the real drift label, the nodes that have not received the correction mark are listed as unknown type normalization change pending manual confirmation, and the nodes that have received the correction mark are determined to be normalization changes affected by upstream type drift, only the indicator fingerprint index is updated, and no new semantic version is created; when both fingerprints change, it is determined to be a real semantic change, and a new indicator version is created; the new computed dual fingerprint and comparison processing results are collected, and the dual fingerprint parsing result is output.
[0038] This invention also provides an upstream dual fingerprint parsing system based on bloodline recursive closure detection, comprising:
[0039] The bloodline recursive closure detection module is used to obtain the upstream bloodline information of the current indicator node, and based on the upstream bloodline information of the current indicator node, it uses a path hash depth-first search mechanism to execute the upstream closure construction task and outputs the bloodline closure parsing result.
[0040] The real-time metadata acquisition and antiquity calculation module is used to perform type drift observation tasks based on the bloodline closure parsing results, using an exponential decay antiquity calculation mechanism, and output type drift observation results.
[0041] The type drift determination module is used to perform a type drift risk determination task based on the type drift observation results, using a weighted risk quantification mechanism, and output the type drift determination result.
[0042] The column type normalization context generation module is used to perform the column type normalization context generation task based on the type drift determination result and a three-channel normalization mechanism, and output the column type normalization processed data.
[0043] The dual fingerprint generation and closure consistency verification module is used to process data based on the column type normalization, perform literal type normalization and closure consistency verification tasks using a dual fingerprint verification mechanism, and output dual fingerprint parsing results.
[0044] The present invention also provides an upstream dual fingerprint parsing device based on bloodline recursive closure detection. The upstream dual fingerprint parsing device based on bloodline recursive closure detection includes: a memory, a processor, and an upstream dual fingerprint parsing program based on bloodline recursive closure detection stored in the memory and executable on the processor. When the upstream dual fingerprint parsing program based on bloodline recursive closure detection is executed by the processor, it implements the above-described method.
[0045] The present invention also provides a computer program product, the computer program product including an upstream dual fingerprint parsing program based on bloodline recursive closure detection, wherein the upstream dual fingerprint parsing program based on bloodline recursive closure detection implements the above method when executed by a processor.
[0046] The beneficial effects of this invention are as follows:
[0047] By employing a path hash depth-first search mechanism, path hashes are carried along during the expansion process to identify node nodes. This preserves path information when the same node is reached via different upstream paths, enabling the lineage closure parsing results to fully reflect multi-path dependencies and avoiding closure bloat caused by repeated expansion. Simultaneously, a stable closure fingerprint is formed based on node hashes and lineage closure summaries, providing a traceable closure foundation for subsequent type drift observation and dual fingerprint verification, thereby improving the completeness and consistency of upstream dependency analysis.
[0048] An exponential decay aging calculation mechanism is adopted in conjunction with normalized write frequency to adjust the decay coefficient, so that the snapshot aging can reflect the impact of table write activity on the timeliness of type information. Then, a weighted risk quantification mechanism is used to integrate type semantic distance, average interpretability and snapshot aging to directly generate unknown drift labels for unknown types, avoiding misjudgment when type information is missing. This allows for more accurate identification of true type drift when lineage snapshots are not synchronized in a timely manner, reducing indicator fingerprint offset and false alarms caused by expired column types.
[0049] By employing a three-channel normalization mechanism, the stable label, the true drift label, and the unknown drift label are respectively treated as effective types using snapshot column type, current metadata column type, and unknown security type. The unknown security type is reversibly encoded instead of forcibly parsed, which ensures that literal normalization remains stable and reversible when the type is uncertain. At the same time, the dual fingerprint verification mechanism distinguishes between changes in expression form, type drift correction, and true semantic changes. Only the true semantic changes are used to create a new indicator version, thereby reducing the false positive rate and improving the accuracy of change detection in type change scenarios. Attached Figure Description
[0050] Figure 1 This is a flowchart illustrating the first embodiment of the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection of the present invention. Detailed Implementation
[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0052] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection according to the present invention. The present invention proposes the first embodiment of the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection.
[0054] In the first embodiment, the upstream dual fingerprint parsing method based on bloodline recursive closure detection includes:
[0055] Step S10: Obtain the upstream lineage information of the current indicator node. Based on the upstream lineage information of the current indicator node, perform the upstream closure construction task using the path hash depth-first search mechanism, and output the lineage closure parsing result.
[0056] The upstream lineage information covers all upstream nodes reachable from the current indicator node along the lineage relationship and their connections. This includes node operators, field names, raw literals or lists of literals, the column type of a field in the lineage snapshot, and upstream and downstream dependent edges. The path hash depth-first search mechanism maintains a path hash for each node during traversal. The path hash of the current node, the fixed arrow separator, and the stable identifier of the next node are concatenated sequentially and then calculated using the SHA-256 digest function to obtain the path hash of the next node. When the same node arrives from different upstream paths, the path hash will differ depending on the source, thus preserving multi-path information. If the same node appears repeatedly on the same path, it indicates a cycle, and the node is marked as a cycle point and further expansion stops.
[0057] Closure construction involves collecting directly upstream nodes round by round until the closure node set no longer changes. For each node in the closure node set, the node operator, field name, stable representation of the original literal, snapshot column type, and sorted concatenation value of the child node hash are sequentially concatenated using a fixed vertical bar separator, and a SHA-256 digest is calculated to obtain the node hash. After sorting all node hashes, they are stably concatenated and a SHA-256 digest is calculated to obtain the lineage closure digest.
[0058] The column reference state matrix is recorded row by row according to node identifier, source table identifier, column name, snapshot column type, snapshot version number, partition column flag, and column usage. The closure node set, path information, lineage snapshot, original filter expression abstract syntax tree, lineage closure summary, and column reference state matrix together constitute the lineage closure parsing result. The column reference state matrix is directly used as the input object for type drift observation in step S20, enabling subsequent antiquity calculation to be performed column by column.
[0059] After this step, identical nodes from different upstream paths are no longer single node records, but path instances with different path hashes, thus preserving the path source information in the lineage closure. Once a node is identified on the path, it no longer expands further downwards, preventing the closure from getting stuck in infinite recursion when building a closure in a lineage graph with loops. Node hashes incorporate node operators, field names, stable literal representations, snapshot column types, and child node hashes. Changes in the operational semantics, field source, literal content, or substructure of any upstream node are reflected in the node hash. The lineage closure digest eliminates the uncertainty of the traversal order of the closure node set through sorted concatenation, ensuring that the same lineage closure receives the same digest value regardless of the traversal order.
[0060] The column reference state matrix expands the nodes in the closure into column-level records. Subsequent steps can collect current type, metadata change time, and literal samples column by column on this matrix without retracing the lineage graph. This reduces the preprocessing overhead for type drift observation and allows for operations at the column reference level. For the snapshot column type change scenario addressed in this application, the snapshot column type and snapshot version number in the column reference state matrix are preserved, becoming the basic information for subsequent detection of type inconsistencies.
[0061] Traditional lineage resolution typically only deduplicates nodes or truncates them to a fixed number of layers when constructing upstream closures, rarely preserving path information for different paths reaching the same node. When encountering multi-source convergence, it easily loses the context of a particular source, making it difficult to distinguish between multiple sources of the same node when calculating fingerprints or judging changes. For loops, traditional approaches often rely solely on the visited set to end the traversal, or stop directly when the recursion depth reaches the upper limit. The former may treat all visited nodes as non-repeatable in multi-path scenarios, inadvertently affecting other uncovered paths; the latter, if the depth threshold is not chosen appropriately, may prematurely truncate before reaching the actual leaf node.
[0062] This step employs path hashing and node-stopping expansion to incorporate path information into the hash without introducing additional data structures. This preserves multiple path sources while also marking the boundaries of loops. The sorting and concatenation of node hashes and lineage closure summaries further suppresses summary discrepancies caused by the traversal order of closure nodes. The column reference state matrix transforms the coarse-grained lineage node parsing results into a structure that can be consumed column-by-column by subsequent type observations, thus improving the completeness of the input provided by the lineage closure parsing results for type drift observations. Even when snapshot column types have expired, the snapshot column types and snapshot type items in the node hashes are retained. Instead of directly discarding the old types, this provides a comparable original state for subsequent steps, allowing type drift to be located to a specific column.
[0063] A certain indicator node has an upstream lineage with a path from table A's field col1 through node N1 to table B's field col2, and another path from table A's field col1 through node N2 to table B's field col2. Table B's field col2 also has a backlink to itself via a self-referencing filter node. Depth-first search reaches table B's field col2 on both paths. Since the current path hash values come from N1 and N2 respectively, table B's field col2 will retain two different path hashes. The self-referencing filter node is marked as a link when it appears a second time on the same path, and will not be expanded further.
[0064] The final closure node set includes table A field col1, node N1, node N2, table B field col2, and self-referencing filter nodes. Node hashes and lineage closure summaries reproducibly generate summary values for this closure. In the lineage snapshot, the column type of table A field col1 is still recorded as STRING, while the actual metadata type has changed to DECIMAL. This column reference retains the STRING snapshot column type and corresponding snapshot version number in the column reference state matrix. Step S20 can directly read these column references for antiquity calculation and current type acquisition, eliminating the need to backtrack the entire lineage graph for subsequent type drift observations.
[0065] Step S20: Based on the bloodline closure analysis results, perform the type drift observation task using the exponential decay staleness calculation mechanism, and output the type drift observation results;
[0066] The type drift observation result is not a single timestamp or type marker, but rather structured data that organizes the lineage closure resolution result, type drift observation vector, and snapshot staleness item by item according to column references. The lineage closure resolution result comes from step S10, where the column reference state matrix records the snapshot column type, snapshot version number, and path hash information corresponding to each column reference. The type drift observation vector is constructed separately for each column reference in the column reference state matrix. During construction, the current metadata column type, snapshot generation time, and current time are collected from the metadata service, and a set of literal samples is collected from the query log or filter condition history. This information is then merged with the snapshot column type. The snapshot staleness is calculated based on the normalized write frequency of the table to which each column belongs, using an exponential decay process that adjusts the decay coefficient according to the frequency. After combining the three types of information, each column reference carries observation data in the time dimension, type dimension, and literal dimension simultaneously. Step S30 can directly read the required content from it for weighted risk quantification.
[0067] Snapshot staleness is calculated using an exponential decay method. It increases rapidly in the initial period after snapshot generation, gradually approaching its upper limit over time. Therefore, lineage snapshots have lower staleness when first generated, but approach their upper limit after a long period without updates. The decay coefficient is not fixed but dynamically adjusted based on the normalized write frequency of the table to which the column belongs: the more frequently a table is written, the larger the decay coefficient. For the same time interval, columns with high-frequency writes will have higher staleness than those with low-frequency writes, thus exposing potential type drift risks earlier.
[0068] The type drift observation results simultaneously retain the snapshot column type, the current metadata column type, the literal sample set, and the snapshot staleness. Step S30 can directly calculate the type semantic distance, literal interpretability, and risk value without re-collecting metadata, reducing data coupling and redundant collection between steps. When the snapshot column type is still a string while the current type has become a decimal number, the observation results will simultaneously retain both types and the corresponding literal samples, providing direct comparison material for subsequent judgment.
[0069] Traditionally, when determining the usability of snapshot column types, the approach often only compares whether the snapshot generation time exceeds a fixed time window, or directly trusts the column types in lineage snapshots, rarely combining write frequency and Data Definition Language events to measure obsolescence. For tables with significant differences in write frequency, a fixed time window can easily cause low-frequency tables to be marked as expired prematurely, or high-frequency tables to be considered fresh even after multiple type changes within the time window.
[0070] This step amplifies the decay coefficient by normalizing the write frequency, allowing the snapshot aging to adaptively adjust with the table's usage intensity, giving a higher aging rating to tables with high-frequency writes at the same time difference. Simultaneously, the type drift observation results not only include the time difference but also incorporate the current type, literal samples, and data definition language event sequences. This enables subsequent steps to distinguish between scenarios where the type remains unchanged despite the passage of time and scenarios where the type has changed recently, improving the granularity of observation regarding snapshot type expiration risk. For cases where source table fields are changed from strings to decimal numbers but lineage snapshots are not updated synchronously, simply looking at the time difference may not be sufficient to detect the problem; data definition language events and literal samples can provide direct evidence of type changes.
[0071] Following the scenario in step S10, the column type of field col1 in table A is still recorded as a string in the lineage snapshot, while the actual metadata type has changed to a decimal number. This column reference retains the string snapshot column type and the corresponding snapshot version number in the column reference status matrix. Step S20 collects the current metadata column type of this column from the metadata service (decimal number), the snapshot generation time (eight days ago), and the current time (the current moment). It also obtains a set of literal sample values from the query logs or filter history within the most recent window. The literals in this set have gradually changed from quoted strings to unquoted decimal numbers.
[0072] Simultaneously, the normalized write frequency of the table to which the column belongs is obtained from the metadata service. This table has been frequently written to in the last thirty days, resulting in a high normalized write frequency. Based on this normalized write frequency, the decay coefficient is adjusted, and exponential decay processing is used to calculate the snapshot staleness of the column. This staleness increases with the difference between the current time and the snapshot generation time, gradually approaching its upper limit. Ultimately, the type drift observation results simultaneously include the current decimal number type, the string snapshot column type, the literal sample set, and the snapshot staleness, providing a three-dimensional basis (time, type, and literal) for step S30 to determine whether type drift has occurred in the column, and can be directly used for weighted risk quantification.
[0073] Step S30: Based on the type drift observation results, perform the type drift risk determination task using a weighted risk quantification mechanism, and output the type drift determination result;
[0074] The type drift determination result is not an isolated label, but a combination of risk labels, risk values, and observation information from step S20. The determination labels are divided into three categories: stable, unknown drift, and true drift, corresponding to the degree of consistency between the snapshot column type, the current metadata column type, and the literal sample, respectively. The risk value comes from a weighted risk quantification mechanism, which first synthesizes the type semantic distance, snapshot staleness, and literal uninterpretability according to preset weights, and then compresses them to between zero and one through exponential mapping; the larger the value, the more severe the drift.
[0075] Type semantic distance is obtained by adding type category semantic distance and same-category precision loss. The former measures cross-category differences between different categories such as strings, numbers, dates, and booleans, while the latter measures semantic loss caused by changes in precision or scaling within the same category, such as changes in numerical precision or date granularity. Literal average interpretability is the result of averaging the parsing confidence of each literal using the parser of the current metadata column type over a sample size, reflecting the current type's interpretability of the actual literal set. Snapshot staleness directly uses the exponential decay calculation result from step S20, reflecting the decaying impact of the lineage snapshot time and the current time interval on type credibility. The weighted risk quantification mechanism sums the above three types of information according to preset weight coefficients, which are pre-set based on the importance of type semantic distance, snapshot staleness, and literal uninterpretability in determining type drift.
[0076] When the snapshot column type or the current metadata column type is unknown, a risk score is not calculated, and an unknown drift label is directly generated to avoid erroneous distances in the absence of a type comparison basis. Otherwise, the type drift risk value is compared with a preset first judgment threshold and a preset second judgment threshold that is greater than the first judgment threshold. If it is less than the first judgment threshold, a stable label is generated; if it is not less than the first judgment threshold and less than the second judgment threshold, an unknown drift label is generated; if it is not less than the second judgment threshold, a true drift label is generated. The judgment label, the type drift risk value, and the type drift observation result are combined to form the type drift judgment result, which serves as the sole basis for selecting the normalization channel in step S40.
[0077] Step S30 integrates three types of information—type semantic distance, snapshot staleness, and literal uninterpretability—into a single risk score, which is then used for exponential mapping to obtain the type drift risk value. Exponential mapping allows the risk value to increase rapidly when the risk score is low, and gradually approaches its upper limit when the risk score is high, which is beneficial for distinguishing fine-grained differences between low and medium risk ranges. Two preset judgment thresholds are set to form a three-category classification: stable, unknown drift, and true drift. Uncertainty is retained in the middle range to avoid simply classifying columns near the boundary into either stable or drift categories, leaving room for safe processing in subsequent three-channel normalization. When the snapshot column type or the current metadata column type is unknown, an unknown drift label is directly generated to prevent incorrect type distance calculations on unknown types.
[0078] The judgment label in the type drift determination result directly determines whether step S40 uses a stable channel, an unknown safe type encoding channel, or a drift correction channel, eliminating the need for secondary interpretation of the risk score. The type drift risk value can also serve as a reference for the processing intensity within subsequent channels; for example, in the unknown safe type encoding channel, the degree of coding conservatism can be adjusted based on the risk value. The intermediate state formed in this step means that type drift determination no longer relies solely on whether the snapshot type is consistent with the current type, nor solely on time staleness, but rather comprehensively considers type category differences, similar precision loss, time decay, and the actual interpretability of the literal, thereby constraining misjudgment paths caused by sudden changes in a single factor or boundary ambiguity.
[0079] Traditional type drift detection often uses string equality as the sole criterion, or only compares snapshot time and metadata time. If the snapshot column type is string and the current column type is variable-length string, comparing by string will indicate a type change, but the actual semantic category and literal interpretation capability have changed very little, which can easily lead to false positives. If the snapshot column type is integer and the current column type is large integer, although the categories are the same, the precision range has expanded. Directly judging it as stable may miss potential data overflow or conversion issues.
[0080] This step measures category difference and similarity accuracy loss using semantic distance and similarity accuracy loss, respectively. It incorporates snapshot staleness and literal interpretability into the risk score. When types are unequal but the differences are small, or when the risk score is in the middle range, an unknown drift label is output, preserving uncertainty rather than immediate correction or immediate release. This intermediate-state risk assessment method improves the robustness of type drift determination in critical and aliasing scenarios without increasing additional metadata collection.
[0081] In scenarios where source table fields change from strings to numbers, even if the current type is clear, a small number of unresolvable values mixed in with the literal samples will increase the risk score, thus entering a more suitable safe processing channel. Traditional methods, if only comparing type strings, will directly determine it as drift and may trigger unnecessary column type corrections; if only comparing time staleness, the actual type change may be ignored due to small time differences. This step, through weighted fusion and dual-threshold tri-classification, makes the judgment result more consistent with the actual semantic change, providing a more reasonable channel selection for subsequent normalization.
[0082] Following the scenario in step S20, the column type of field col1 in table A is still recorded as a string in the lineage snapshot, while the actual metadata type has changed to a decimal number. The column reference retains the string snapshot column type and the corresponding snapshot version number in the column reference status matrix. Step S20 collects the current metadata column type of this column from the metadata service (decimal number), the snapshot generation time (eight days ago), and the current time (the current moment). It also obtains a set of literal samples within the recent window from the query log or filter history. The literals in this set have gradually changed from quoted strings to unquoted decimal numbers. Simultaneously, it obtains the normalized write frequency of the table to which this column belongs from the metadata service. This table has been frequently written to in the last thirty days, resulting in a high normalized write frequency. Based on this normalized write frequency, the decay coefficient is adjusted, and exponential decay processing is used to calculate the snapshot staleness of this column. This staleness increases with the difference between the current time and the snapshot generation time, gradually approaching its upper limit. Ultimately, the type drift observation results simultaneously include the current type of the decimal number, the type of the string snapshot column, the literal sample set, and the snapshot staleness.
[0083] Step S30 obtains the snapshot column type, current metadata column type, literal sample set, and snapshot staleness for each column referenced from the type drift observation results. According to the preset type category semantic distance and similar precision loss rules, the type category semantic distance is added to the similar precision loss to obtain the type semantic distance. For each literal in the literal sample set, the parser of the current metadata column type is used for parsing, obtaining a parsing confidence score ranging from zero to one (zero if parsing is not possible). The parsing confidence scores are averaged to obtain the average interpretability. The type semantic distance, snapshot staleness, and the uninterpretability obtained by subtracting the average interpretability from one are weighted and summed using preset weighting coefficients to obtain a type drift risk score. The type drift risk value is obtained by subtracting the negative natural exponent value of the type drift risk score from one.
[0084] A preset first judgment threshold and a preset second judgment threshold greater than the first judgment threshold are set. When the snapshot column type or the current metadata column type is unknown, an unknown drift label is directly generated; otherwise, a stable label is generated when the type drift risk value is less than the first judgment threshold, an unknown drift label is generated when it is not less than the first judgment threshold and less than the second judgment threshold, and a true drift label is generated when it is not less than the second judgment threshold. The judgment label, the type drift risk value, and the type drift observation results are combined to form the type drift judgment result.
[0085] In this scenario, the combined semantic distance and precision loss terms for this column type are relatively high. The average interpretability is slightly lower than fully interpretable due to a small number of unresolvable literals. The snapshot staleness is at a moderate level. The weighted risk score falls into the middle range, and the type drift risk value is between the first and second judgment thresholds. Therefore, an unknown drift label is generated. This type drift determination result will serve as the basis for selecting the unknown security type encoding channel in step S40, rather than directly using the current metadata type or snapshot column type for normalization.
[0086] Step S40: Based on the type drift determination result, a three-channel normalization mechanism is used to execute the column type normalization context generation task, and the column type normalization processing data is output.
[0087] The column type normalization processing data is a set of column-level type criteria formed in step S40 based on the type drift determination result, which is directly consumed by step S50. This data includes the column type normalization context, the column type correction record set, and the retained type drift determination result. The column type normalization context uses each column reference as an index to record the valid type, type correction version number, and correction flag for each column reference; the column type correction record set only collects correction actions generated by the actual drift channel, and each record includes the column reference identifier, the snapshot column type before correction, the current metadata column type after correction, and the type correction version number when the correction occurred.
[0088] The three-channel normalization mechanism sends column references to the stable channel, the true drift channel, and the unknown drift channel respectively, based on the judgment label output in step S30. The stable channel, corresponding to the stable label, determines the valid type as the snapshot column type, retains the type correction version number, and sets the correction flag to uncorrected, indicating that the column reference continues to use the column type in the lineage snapshot as the basis for literal normalization. The true drift channel, corresponding to the true drift label, determines the valid type as the current metadata column type, generates a column type correction record and adds it to the column type correction record set, while retaining the old version of the lineage snapshot and generating a new version using the current metadata column type. The type correction version number is incremented, and the correction flag is set to corrected, enabling subsequent steps to recognize that the column reference has undergone type correction.
[0089] The unknown drift channel corresponds to an unknown drift label. The valid type is determined as an unknown security type, the type correction version number is retained, and the correction flag is set to uncorrected. For literals corresponding to unknown security types, instead of calling a specific type parser, the literal is converted to a UTF-8 byte sequence. This sequence is then stably concatenated with a fixed prefix "U:", padded with leading zeros to a fixed byte length, a colon separator, the URL-secure Base64 encoding of the byte sequence, another colon separator, and the first few hexadecimal characters of the SHA-256 digest of the byte sequence, resulting in an unknown security reversible encoding result. This encoding result simultaneously preserves the recoverable information and integrity verification information of the original literal, ensuring that unknown type literals can still be identified and verified in subsequent comparisons.
[0090] The column type normalization context, the column type correction record set, and the type drift determination result together constitute the column type normalization processing data. The type drift determination result continues to carry risk labels and risk values, providing a type drift background for the dual fingerprint verification in step S50.
[0091] Step S40 uses a three-channel normalization mechanism to transform the type drift determination label output in step S30 into a valid type that can be directly used by each column reference, forming a column type correction record set and an unknown safe reversible encoding result. This eliminates the need for step S50 to re-determine the type source or handle type uncertainty when normalizing literals. For stable label column references, the valid type remains the snapshot column type, and step S50 can continue to parse literals according to the type rules in the lineage snapshot, avoiding introducing additional conversions for columns that have not drifted.
[0092] For references to real drift-label columns, the valid type is switched to the current metadata column type. Simultaneously, the column type correction record set retains the type pair before and after correction, as well as the incremented type correction version number. Step S50 incorporates type correction epoch information into the comparison when calculating the index fingerprint, explicitly distinguishing the fingerprint differences of the same column reference before and after correction, rather than misjudging them as different columns. For references to unknown drift-label columns, the valid type is an unknown safe type. The literal is encoded in a stable form containing a length prefix, reversible encoding, and a digest field. Step S50 can still perform consistency checks and fingerprint calculations on the literal even when the specific type cannot be determined, avoiding the loss of original content or incorrect normalization results due to forced type conversion.
[0093] The column type normalization context compresses column type divergences into a single valid type, allowing the filtering condition abstract syntax tree normalization process in step S50 to simply read the valid type and process the literals according to the corresponding rules. This reduces the risk of type uncertainty being passed on to subsequent dual fingerprint verification stages. The type correction version number increments in the actual drift channel, providing an explicit marker for the order of type corrections in step S50. This enables dual fingerprint comparison to distinguish column type states under different correction epochs, thereby constraining fingerprint inconsistencies caused by version confusion during type drift.
[0094] Traditional processing, after determining type drift, typically outputs only a binary result indicating whether the type has changed or not. Subsequent literal normalization then forces a choice between the snapshot type and the current type. When the determination result is uncertain or the samples are mixed, continuing to use the snapshot type may apply outdated conversion rules to column references that have already undergone type changes, leading to incorrect parsing or discarding of the literal. Switching directly to the current type may force unparsable literals to be converted, corrupting the original content. This step introduces an unknown safe type channel to reversibly encode uncertain literals. The encoding result includes a length prefix and a summary field, allowing the literal to be recovered and verified even when the current type is not confirmed, avoiding forced type conversion and loss of original content.
[0095] For confirmed drifting column references, traditional methods typically modify the snapshot column type directly, but do not record the correction action and order. Subsequent fingerprint comparisons cannot distinguish the type state before and after the correction, easily misjudging the type changes of the same column reference at different points in time as different columns. This step records the order of each actual drift correction through a column type correction record set and a type correction version number, making the type correction action explicit information carried in the context and fingerprint. Step S50 can identify the type correction epoch when comparing dual fingerprints, avoiding version confusion.
[0096] Stable channels retain the original snapshot column types, reducing unnecessary context changes caused by uniform correction of drift-free columns. In contrast, traditional binary selection often uses the same type source for all column references, forcing even drift-free columns to switch types, increasing subsequent normalization errors. Overall, the three-channel normalization mechanism provides finer type processing boundaries compared to binary selection, making it particularly suitable for transitional phases such as the initial expiration of snapshot types, when the current type is still unstable, or when literal samples are insufficient.
[0097] Following the scenario in step S30, the column type of field col1 in table A is still recorded as a string in the lineage snapshot, but the actual metadata type has changed to a decimal number. This column reference was determined to have an unknown drift label in step S30, and the type drift risk value falls into the middle range. Step S40 obtains the judgment label, snapshot column type, current metadata column type, and lineage snapshot of this column reference from the type drift judgment result, reads the type correction version number of this column reference, and establishes a valid type mapping and a set of column type correction records. Because this column reference has an unknown drift label, it enters the unknown drift channel, the valid type is determined to be an unknown safe type, the type correction version number is retained, and the correction flag is set to uncorrected.
[0098] The column references a literal sample containing a value with an unresolvable character. This literal doesn't invoke a specific type parser; instead, it's converted to a UTF-8 byte sequence. This sequence is then concatenated with a fixed prefix "U:", padded with leading zeros to a fixed byte length, a colon separator, the URL-secure Base64 encoding of the byte sequence, another colon separator, and the first few hexadecimal characters of the SHA-256 digest of the byte sequence. This results in an unknown, secure, reversible encoded result. The length prefix in the encoded result records the UTF-8 byte length, the Base64Url portion can be decoded to recover the original literal, and the digest portion can be used for verification.
[0099] Another column reference is determined to be a true drift label in step S30. The snapshot column type is corrected from string to the current metadata type timestamp. The type correction version number is incremented, a column type correction record is generated and added to the column type correction record set, the old version of the lineage snapshot is retained, and a new version using the current metadata column type is generated. The correction flag is set to corrected. Another column reference is determined to be a stable label. It continues to use the snapshot column type variable-length character as the valid type, the type correction version number remains unchanged, and the correction flag is set to uncorrected.
[0100] The column type normalization context records these three different processing results. The column type correction record set contains correction records generated by the actual drift channel, and the type drift judgment result is retained as is. In step S50, when standardizing the abstract syntax tree of the filtering conditions, the valid type corresponding to each column can be directly read, the literal is processed according to the stable rules, and dual fingerprint verification is performed using the type correction version number and the unknown secure reversible encoding result.
[0101] Step S50: Based on the column type normalization data, a dual fingerprint verification mechanism is used to perform literal type normalization and closure consistency verification tasks, and output the dual fingerprint parsing results.
[0102] The dual fingerprint analysis result is a closure-level output formed based on the column type normalization data. It carries both the job fingerprint and the indicator fingerprint, as well as the comparison conclusion between the two. This result covers every node in the closure node set and retains the original filter expression abstract syntax tree, normalized abstract syntax tree, lineage closure summary, column type normalization context, type correction version number, newly calculated dual fingerprint, historical dual fingerprint comparison status, and final decision action for each node. The column type normalization data is provided by step S40, where the closure node set records all nodes within the upstream lineage closure. The lineage closure summary remains unchanged after being generated in step S10, and the original filter expression abstract syntax tree of each node preserves the filter condition structure before type normalization. The column type normalization context records the valid type, type correction version number, and correction flag column by column. The decision labels are divided into stable labels, true drift labels, and unknown drift labels.
[0103] Step S50 first normalizes the leaf node literals of the original filter expression abstract syntax tree item by item according to the valid types in the column type normalization context: string types remove leading and trailing whitespace and perform Unicode standard normalization; numeric types are parsed into decimal numbers, trailing zeros are removed, and ordinary decimal strings are output; date types are parsed and output as an eight-digit year-month-day representation; boolean types are converted to lowercase strings. When numeric or date parsing fails, or when the valid type is an unknown security type, the specific type parser is not called for forced conversion. Instead, the literal is converted into an eight-bit byte sequence in Unicode conversion format. Then, a fixed prefix is stably concatenated, padded with leading zeros to a fixed byte length, a colon separator is added, followed by the Unicode security code point of the byte sequence, the colon separator, and the first few hexadecimal characters of the security hash digest of the byte sequence, to obtain an unknown security reversible encoding result. The normalized abstract syntax tree is structurally consistent with the original filter expression abstract syntax tree, with only the leaf node literal representation changing.
[0104] The job fingerprint is generated by stable serialization and secure hashing of the original filter expression abstract syntax tree and the lineage closure digest. The indicator fingerprint is generated by stable serialization and secure hashing of the normalized abstract syntax tree, column type normalization context, and the largest type correction version number within the current closure. During dual fingerprint verification, the historical dual fingerprint is read for each node in the closure node set and compared with the newly calculated dual fingerprint. Based on whether the job fingerprint and indicator fingerprint have changed and whether the node has a real drift label column reference, it is determined whether there is no change, only a change in expression form, a type drift correction, an unknown type normalization change, a normalization change affected by upstream type drift, or a real semantic change. Based on this, it is decided whether to update the indicator fingerprint index, propagate the correction mark downstream, create a new indicator version, or put it into a state awaiting manual confirmation.
[0105] Step S50 decomposes literal type normalization and closure consistency verification into two independent computation paths: job fingerprint and indicator fingerprint. This allows changes in filtering conditions within the same lineage closure to be distinguished from three different sources: changes in expression form, changes in type normalization, and changes in true semantics. The job fingerprint relies only on the abstract syntax tree of the original filtering expression and the lineage closure summary. When the original filtering conditions have not undergone business modifications, the job fingerprint remains unchanged regardless of how the column type is modified or how the literal is re-encoded. The indicator fingerprint relies on the normalization abstract syntax tree, the column type normalization context, and the type modification version number, and can independently reflect the impact of adjustments to the type normalization context or version number.
[0106] When the job fingerprint remains unchanged while the indicator fingerprint changes, and the node has a reference to a real drift label column, the system determines it as a type drift correction, only updates the indicator fingerprint index and propagates the correction flag to downstream nodes, without triggering a semantic change alarm. When the job fingerprint remains unchanged while the indicator fingerprint changes but there is no reference to a real drift label column, the system further distinguishes between nodes that have not received a correction flag as having an unknown type normalization change pending manual confirmation, and nodes that have received a correction flag as having a normalization change affected by upstream type drift, only updating the indicator fingerprint index without creating a new semantic version. When both the job fingerprint and indicator fingerprint change, the system determines it as a real semantic change and creates a new indicator version. This dual-fingerprint parsing result not only provides the final state of each node but also preserves the source of change and propagation path, enabling downstream dependent nodes to determine whether their own indicator fingerprint change is caused by upstream type drift based on the correction flag, avoiding misjudging the representation adjustment during type synchronization as a business semantic change.
[0107] Unknown secure reversible encoding results can still maintain the reversibility and stability of literals when the specific type cannot be determined. The combination of length prefix, Uniform Resource Locator (URL) secure encoding and secure hash digest enables the encoding results to both restore the original value and verify integrity, preventing unresolvable characters from generating random differences in subsequent fingerprint calculations due to different encoding methods.
[0108] Traditional dual-fingerprint or single-fingerprint schemes typically only compare the expression digests before and after normalization, or mix lineage digests and expression digests in a single fingerprint. They cannot distinguish whether changes in the indicator fingerprint stem from adjustments to the type normalization context or modifications to the original filter conditions. When the snapshot column type is modified or an unknown type literal is re-encoded, the indicator fingerprint may change even if the business semantics of the filter conditions remain unchanged. If the system relies solely on single fingerprint changes to trigger alarms, false alarms due to type synchronization will occur. Conversely, completely ignoring indicator fingerprint comparisons may mask the true semantic changes in the filter conditions.
[0109] This step uses a combination of job fingerprints and indicator fingerprints for judgment. If the job fingerprint remains unchanged, it indicates that the original filter expression's abstract syntax tree and lineage closure structure have not undergone any business changes. Changes in the indicator fingerprint only point to adjustments in the type normalization context or version number. Combined with the reference to the true drift label column, this indicates that the change originates from type drift correction. Therefore, only the indicator fingerprint index is updated without triggering semantic change alarms or propagating correction markers downstream. When both change, it indicates that the original expression has indeed changed, and the system can determine the true semantic change accordingly. Compared to traditional methods, dual-fingerprint verification improves the distinction between type correction and semantic change without adding additional expression rewriting rules, and avoids alarm noise during type synchronization.
[0110] Traditional solutions often treat unknown literals as strings or discard them, causing fingerprint jitter due to encoding differences in different snapshots for the same unresolvable value. This step uses unknown secure reversible encoding to convert literals into stable representations with length prefixes, Uniform Resource Locator (URL) secure encoding, and secure hash digests. Even if the specific type cannot be resolved, the fingerprint remains stable, while retaining reversible recovery capabilities, enabling the original literal to be restored during subsequent manual verification or auditing.
[0111] Following the scenario in step S40, the column type of field col1 in table A is still recorded as a string in the lineage snapshot, but the actual metadata type has changed to a decimal number. This column reference was determined to have an unknown drift label in step S30, and the type drift risk value falls into the middle range. Step S40 obtains the judgment label, snapshot column type, current metadata column type, and lineage snapshot of the column reference from the type drift judgment result, reads the type correction version number of the column reference, and establishes a valid type mapping and a set of column type correction records. Because the column reference has an unknown drift label, it enters the unknown drift channel, the valid type is determined to be an unknown safe type, the type correction version number is retained, and the correction flag is set to uncorrected.
[0112] The column references a literal sample containing a value with an unresolvable character. Instead of invoking a specific type parser, this literal is converted to a Unicode Transformation Format (Unicode) 8-bit byte sequence. This sequence is then concatenated with a fixed prefix, padded with leading zeros to a fixed byte length, a colon separator, the Unicode Conversion Security (UCS) code for the byte sequence, another colon separator, and the first few hexadecimal characters of the secure hash digest of the byte sequence. This results in an unknown secure reversible encoded result. The length prefix in the encoded result records the Unicode Transformation Format 8-bit byte length, the Unicode Conversion Security (UCS) code can be decoded to recover the original literal, and the hash can be used for verification.
[0113] Another column reference is determined to be a true drift label in step S30. The snapshot column type is corrected from string to the current metadata type timestamp. The type correction version number is incremented, a column type correction record is generated and added to the column type correction record set, the old version of the lineage snapshot is retained, and a new version using the current metadata column type is generated. The correction flag is set to corrected. Another column reference is determined to be a stable label. It continues to use the snapshot column type variable-length character as the valid type, the type correction version number remains unchanged, and the correction flag is set to uncorrected. The column type normalization context records these three different processing results. The column type correction record set contains the correction records generated by the true drift channel, and the type drift determination result is retained as is.
[0114] Step S50 directly reads the valid type corresponding to each column when standardizing the abstract syntax tree of the filtering conditions, processes the literals according to the stability rules, and performs dual fingerprint verification using the type correction version number and the unknown secure reversible encoding result. For references to real drift label columns, the literals in the original filtering conditions may be date strings with time components. Step S50 parses them according to the valid type timestamp type and outputs an eight-digit year-month-day representation. Compared with the original abstract syntax tree, the generated normalized abstract syntax tree only changes the literal representation of the leaf nodes. The job fingerprint remains unchanged, while the indicator fingerprint changes due to the increment of the type correction version number and the change of the normalized abstract syntax tree. After comparing the node's historical dual fingerprints, it is determined to be a type drift correction. The indicator fingerprint index is updated but no semantic change alarm is triggered, and the correction mark is propagated to downstream dependent nodes.
[0115] For unknown drift label column references, their literals remain stable after unknown secure reversible encoding, and both the job fingerprint and indicator fingerprint remain unchanged, thus being judged as unchanged. If an equivalence judgment condition for another column reference is added to the original filtering conditions of the same node, both the job fingerprint and indicator fingerprint change, then it is judged as a true semantic change, and a new indicator version is created. Finally, the newly calculated dual fingerprints and comparison processing results of each node in the closure node set are collected, and the dual fingerprint parsing results are output, providing a complete basis for downstream indicator version management and lineage consistency auditing.
[0116] Example 2: Furthermore, the present invention provides an upstream dual fingerprint parsing system based on bloodline recursive closure detection, employing the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection in the above embodiments, which can solve the technical problem of upstream dual fingerprint parsing based on bloodline recursive closure detection. The beneficial effects of the upstream dual fingerprint parsing system based on bloodline recursive closure detection provided by the present invention are the same as those of the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection provided in the above embodiments, and other technical features of the upstream dual fingerprint parsing system based on bloodline recursive closure detection are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0117] Example 3: This invention provides an upstream dual fingerprint parsing device based on kinship recursive closure detection. The device includes at least one processor and a memory communicatively connected to the processor. The memory stores instructions executable by the processor, which are then executed to enable the processor to perform the upstream dual fingerprint parsing method and system based on kinship recursive closure detection described in Example 1. The upstream dual fingerprint parsing device based on kinship recursive closure detection in this invention can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This upstream dual fingerprint parsing device based on kinship recursive closure detection is merely an example and should not limit the functionality or scope of the invention. An upstream dual-fingerprint analysis device based on lineage recursive closure detection may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory or a program loaded from a storage device into random access memory. The random access memory also stores various programs and data required for the operation of the upstream dual-fingerprint analysis device based on lineage recursive closure detection. The processing unit, read-only memory, and random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the upstream dual-fingerprint analysis device based on lineage recursive closure detection to communicate wirelessly or wiredly with other devices to exchange data. While an upstream dual fingerprinting device based on lineage recursive closure detection with various systems has been described, it should be understood that implementation of all the described systems is not required. Alternatively, more or fewer systems may be implemented.
[0118] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection as described above. The computer program product provided by this invention can solve the technical problem of upstream dual fingerprint parsing based on bloodline recursive closure detection. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the upstream dual fingerprint parsing method and system based on bloodline recursive closure detection provided in the above embodiments, and will not be repeated here.
[0119] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.
[0120] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.
[0121] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.
Claims
1. An upstream dual fingerprint parsing method based on bloodline recursive closure detection, characterized in that, include: Step S10: Obtain the upstream lineage information of the current indicator node. Based on the upstream lineage information of the current indicator node, perform the upstream closure construction task using the path hash depth-first search mechanism, and output the lineage closure parsing result. Step S20: Based on the bloodline closure analysis results, perform the type drift observation task using the exponential decay staleness calculation mechanism, and output the type drift observation results; Step S30: Based on the type drift observation results, perform the type drift risk determination task using a weighted risk quantification mechanism, and output the type drift determination result; Step S40: Based on the type drift determination result, a three-channel normalization mechanism is used to execute the column type normalization context generation task, and the column type normalization processing data is output. Step S50: Based on the column type normalization data, a dual fingerprint verification mechanism is used to perform literal type normalization and closure consistency verification tasks, and output the dual fingerprint parsing results.
2. The upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in claim 1, characterized in that, Step S10, which involves obtaining the upstream lineage information of the current indicator node, performing an upstream closure construction task based on the upstream lineage information of the current indicator node using a path hash depth-first search mechanism, and outputting the lineage closure parsing result, specifically includes: Step S101: Obtain the current indicator node, upstream bloodline edge, bloodline snapshot and original filter expression abstract syntax tree of each node from the upstream bloodline information. Using the current indicator node as the initial node, perform depth-first expansion according to the upstream bloodline edge, add the direct upstream nodes of the included nodes to the closure, and construct the closure node set until the closure node set no longer changes. Step S102: During the expansion process, the path hash is carried. The path hash of the current node, the fixed arrow separator, and the stable identifier of the next node are concatenated in sequence to calculate the SHA-256 digest and update the path hash of the next node. When the same node arrives through different upstream paths, the path information is retained. When the same node appears repeatedly on the same path, it is marked as a node and the expansion is stopped. The set of closure nodes with retained path information is obtained. Step S103: For each node in the closure node set, the sorted and concatenated values of the operator, field name, stable representation of the original literal, snapshot column type, and child node hash are sequentially concatenated using a fixed vertical bar separator and a SHA-256 digest is calculated to obtain the node hash; the node hashes are sorted, stably concatenated, and a SHA-256 digest is calculated to obtain the lineage closure digest; a column reference state matrix is constructed that records the node identifier, source table identifier, column name, snapshot column type, and snapshot version number; the closure node set, the path information, the lineage snapshot, the original filter expression abstract syntax tree, the lineage closure digest, and the column reference state matrix are combined to form the lineage closure parsing result.
3. The upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in claim 1, characterized in that, Step S20, which involves performing a type drift observation task based on the bloodline closure analysis result using an exponential decay staleness calculation mechanism and outputting the type drift observation result, specifically includes: Step S201: Obtain the column reference state matrix from the bloodline closure parsing result. For each column reference in the column reference state matrix, collect the current metadata column type, snapshot generation time and current time from the metadata service, collect a set of literal samples from the query log or filter conditions, and construct a type drift observation vector in combination with the snapshot column type. Step S202: Based on the type drift observation vector, obtain the normalized write frequency of the table to which each column belongs, and use exponential decay processing with the attenuation coefficient adjusted with the normalized write frequency to calculate the snapshot staleness of each column. Step S203: Combine the bloodline closure parsing result, the type drift observation vector, and the snapshot staleness to form the type drift observation result.
4. The upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in claim 3, characterized in that, In step S202, based on the type drift observation vector, the normalized write frequency of the table to which each column belongs is obtained, and exponential decay processing with an attenuation coefficient adjusted according to the normalized write frequency is used to calculate the snapshot aging of each column. Specifically, this includes: normalizing the write frequency according to the number of writes in the data definition language or data manipulation language of the table to which each column belongs within the last 30 days to obtain the normalized write frequency with a value between 0 and 1, and calculating the snapshot aging. The calculation relationship for the snapshot aging is as follows: in, Reference to the current column, For the snapshot age of this column, The preset baseline attenuation coefficient is set to 0.05 per day. The preset write frequency amplification factor is set to 0.
3. The normalized write frequency for the table to which this column belongs. The difference between the current time and the snapshot generation time in this column, in days. The natural exponential function is used to form the type of drift observation results.
5. The upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in claim 1, characterized in that, Step S30, which involves performing a type drift risk determination task based on the type drift observation results using a weighted risk quantification mechanism and outputting the type drift determination result, specifically includes: Step S301: Obtain the snapshot column type, current metadata column type, literal sample set and snapshot staleness of each column referenced from the type drift observation results. According to the preset type category semantic distance and similar precision loss rules, add the type category semantic distance and similar precision loss to obtain the type semantic distance. Step S302: For each literal in the literal sample set, use the parser of the current metadata column type to parse it, and obtain the parsing confidence with a value of 0 to 1. If it cannot be parsed, take zero. Calculate the average parsing confidence to obtain the average interpretability. Step S303: The type semantic distance, the snapshot staleness, and the uninterpretability (obtained by subtracting the average interpretability from one) are weighted and summed using a preset weighting coefficient to obtain a type drift risk score. The type drift risk value is obtained by subtracting the negative value of the type drift risk score from the natural exponent value. A preset first judgment threshold and a preset second judgment threshold greater than the first judgment threshold are set. When the snapshot column type or the current metadata column type is an unknown type, an unknown drift label is directly generated. Otherwise, when the type drift risk value is less than the first judgment threshold, a stable label is generated; when it is not less than the first judgment threshold and less than the second judgment threshold, an unknown drift label is generated; when it is not less than the second judgment threshold, a true drift label is generated. The judgment label, the type drift risk value, and the type drift observation result are combined to form the type drift judgment result.
6. The upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in claim 1, characterized in that, In step S40, based on the type drift determination result, a three-channel normalization mechanism is used to execute the column type normalization context generation task, and the column type normalization processed data is output. This step specifically includes: Step S401: Obtain the judgment label, snapshot column type, current metadata column type, and lineage snapshot of each column reference from the type drift judgment result; read the type correction version number of each column reference; establish a valid type mapping and column type correction record set; the judgment label is a stable label, a true drift label, or an unknown drift label; for column references with the stable label, determine the valid type as the snapshot column type, retain the type correction version number, and set the correction flag to uncorrected; Step S402: In the effective type mapping, for column references with the real drift label, the effective type is determined to be the current metadata column type, a column type correction record is generated and added to the column type correction record set, the old version of the lineage snapshot is retained and a new version using the current metadata column type is generated, the type correction version number is incremented, and the correction flag is set to corrected. Step S403: In the valid type mapping, for column references with the unknown drift label, the valid type is determined as an unknown security type, the type correction version number is retained, and the correction flag is set to uncorrected; for the literal corresponding to the unknown security type, the specific type parser is not called, the literal is converted into a UTF-8 byte sequence, and the fixed prefix "U:", the byte length padded with leading zeros to 5 bits, the colon separator, the URL-secure Base64 encoding of the byte sequence, the colon separator, and the first 64 hexadecimal characters of the SHA-256 digest of the byte sequence are concatenated in sequence to obtain the unknown security reversible encoding result; the valid type mapping formed by the valid type, type correction version number, and correction flag of each column reference is determined as the column type normalization context, and the column type normalization context, the column type correction record set, and the type drift determination result are combined to form the column type normalization processing data.
7. The upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in claim 1, characterized in that, Step S50, based on the column type normalization data, employs a dual fingerprint verification mechanism to perform literal type normalization and closure consistency verification tasks, and outputs the dual fingerprint parsing results. Specifically, this includes: Step S501: Obtain the closure node set, lineage closure summary, original filter expression abstract syntax tree of each node, column type normalization context, and decision labels of each column reference from the column type normalization processing data; the column type normalization context records the valid type, type correction version number, and correction flag of each column reference, and the decision label is a stable label, a true drift label, or an unknown drift label; normalize the leaf node literals of the original filter expression abstract syntax tree according to the valid types to obtain a normalized abstract syntax tree, wherein string types are subjected to removal of leading and trailing whitespace and Unicode. For NFC standardization, numeric types are parsed into decimal numbers, trailing zeros are removed, and a normal decimal string is output. Date types are parsed and an eight-digit year-month-day representation is output. Boolean types are converted to lowercase strings. When numeric or date parsing fails, or when the valid type is an unknown security type, a specific type parser is not called for forced conversion. Instead, the literal is converted into a UTF-8 byte sequence, and then a fixed prefix "U:" is sequentially and stably concatenated, followed by leading zeros to a 5-digit byte length, a colon separator, the URL-secure Base64 encoding of the byte sequence, the colon separator, and the first 64 hexadecimal characters of the SHA-256 digest of the byte sequence to obtain an unknown security reversible encoding result. Step S502: For each node in the closure node set, determine the largest type correction version number within the current closure from the column type normalization context. Calculate the job fingerprint using the original filter expression abstract syntax tree and the lineage closure digest. Calculate the index fingerprint using the normalized abstract syntax tree, the column type normalization context, and the largest type correction version number. The calculation relationships between the job fingerprint and the index fingerprint are as follows: in, and These are the job fingerprint and the indicator fingerprint, respectively. For SHA-256 digest function, For stable serialization functions, and These are the original filter expression abstract syntax tree and the normalized abstract syntax tree, respectively. This is the summary of the bloodline closure. For the normalization context of the column type, A stable string representation of the largest type's version number. This indicates stable string concatenation. For fixed version prefixes, Fixed separator; Step S503: For each node in the closure node set, read the historical dual fingerprint and compare it with the new computed dual fingerprint composed of the job fingerprint and the indicator fingerprint; when both fingerprints remain unchanged, it is determined that there is no change; when only the job fingerprint changes, it is determined that only the expression form has changed and no new semantic version is created; when the job fingerprint remains unchanged, the indicator fingerprint changes, and the node has a column reference with the real drift label, it is determined to be a type drift correction, the indicator fingerprint index is updated, and the correction mark is propagated to downstream nodes without triggering a semantic change alarm; when the job fingerprint remains unchanged, the indicator fingerprint changes, and there is no column reference with the real drift label, the nodes that have not received the correction mark are listed as unknown type normalization change pending manual confirmation, and the nodes that have received the correction mark are determined to be normalization changes affected by upstream type drift, only the indicator fingerprint index is updated, and no new semantic version is created; when both fingerprints change, it is determined to be a real semantic change, and a new indicator version is created; the new computed dual fingerprint and comparison processing results are collected, and the dual fingerprint parsing result is output.
8. An upstream dual fingerprint parsing system based on bloodline recursive closure detection, applied to the upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in any one of claims 1 to 7, characterized in that, include: The bloodline recursive closure detection module is used to obtain the upstream bloodline information of the current indicator node, and based on the upstream bloodline information of the current indicator node, it uses a path hash depth-first search mechanism to execute the upstream closure construction task and outputs the bloodline closure parsing result. The real-time metadata acquisition and antiquity calculation module is used to perform type drift observation tasks based on the bloodline closure parsing results, using an exponential decay antiquity calculation mechanism, and output type drift observation results. The type drift determination module is used to perform a type drift risk determination task based on the type drift observation results, using a weighted risk quantification mechanism, and output the type drift determination result. The column type normalization context generation module is used to perform the column type normalization context generation task based on the type drift determination result and a three-channel normalization mechanism, and output the column type normalization processed data. The dual fingerprint generation and closure consistency verification module is used to process data based on the column type normalization, perform literal type normalization and closure consistency verification tasks using a dual fingerprint verification mechanism, and output dual fingerprint parsing results.
9. An upstream dual fingerprint analysis device based on bloodline recursive closure detection, characterized in that, The upstream dual fingerprint parsing device based on bloodline recursive closure detection includes: a memory, a processor, and an upstream dual fingerprint parsing program based on bloodline recursive closure detection stored in the memory and executable on the processor. When the upstream dual fingerprint parsing program based on bloodline recursive closure detection is executed by the processor, it implements the upstream dual fingerprint parsing method based on bloodline recursive closure detection according to any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes an upstream dual fingerprint parsing program based on bloodline recursive closure detection, which, when executed by a processor, implements an upstream dual fingerprint parsing method based on bloodline recursive closure detection as described in any one of claims 1 to 7.
