A data tracing method based on big data mining

Abstract

The present application relates to big data governance technical field, specifically to a kind of data traceability method based on big data mining, comprising: collecting unstructured operation log and parsing into structured event record, associated generation heterogeneous operation session set;Access mode association analysis is carried out to session set, and read-write path mode is extracted, and dependence weight is determined in combination with association strength and time decay factor and constructs blood relationship atlas;Loop is eliminated to atlas, and directed acyclic graph is generated, and generating tree and node index coding interval are constructed as retrieval backbone, and cross branch association edge is mapped to bitmap index to generate probability skeleton tree index;Interval positioning main path is used, and bitmap is called to carry out multi-path verification, and traceability path and confidence are output. By mining log, probability blood relationship atlas and probability skeleton tree index are constructed, data blood relationship reconstruction and large-scale node low-delay retrieval in heterogeneous black box environment are realized.

Description

Technical Field

[0001] This invention relates to the field of big data governance technology, specifically a data tracing method based on big data mining. Background Technology

[0002] In key infrastructure such as telecom operators' business support systems and smart city data lakes, data flow exhibits extremely high heterogeneity and complexity, with processing typically spanning legacy billing engines running on mainframes and modern streaming computing platforms.

[0003] Existing data tracing technologies mainly rely on static code analysis or intrusive metadata injection, which involves explicitly capturing data dependencies by parsing SQL statements or embedding probes in the source code; or using general traversal algorithms based on graph databases to perform full graph retrieval by treating data objects as nodes and dependencies as edges.

[0004] However, in the aforementioned heterogeneous telecommunications scenarios, existing technologies face severe challenges: On the one hand, there are a large number of tightly encapsulated binary legacy components in the telecommunications environment. Due to the inability to obtain source code permissions for instrumentation, the logical connection between file input and output is broken at the metadata level, forming a data lineage blind spot across media; on the other hand, as the scale of data assets reaches hundreds of millions of nodes, and the mined dependencies naturally have uncertain probabilistic attributes, traditional query algorithms based on full graph traversal face exponential time complexity when processing the reachability calculation of large-scale probabilistic graphs, resulting in excessively high latency in traceability query response, which cannot meet the needs of second-level real-time operation and maintenance and accurate integrity verification throughout the entire data lifecycle.

[0005] Therefore, how to reconstruct data lineage in a black-box environment lacking source code and achieve millisecond-level retrieval for massive uncertain graphs has become an urgent problem to be solved.

[0006] Therefore, this invention proposes a data tracing method based on big data mining. Summary of the Invention

[0007] The purpose of this invention is to provide a data tracing method based on big data mining, which constructs a probabilistic lineage map and a probabilistic skeleton tree index by mining logs, thereby realizing data lineage reconstruction and low-latency retrieval of large-scale nodes in a heterogeneous black-box environment.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A data tracing method based on big data mining includes:

[0010] Collect unstructured operation logs, use log parsing templates to parse the unstructured operation logs into structured event records, and associate the structured event records into a heterogeneous operation session set;

[0011] Access pattern association analysis is performed on a heterogeneous set of operation sessions to extract read and write access path patterns; the dependency weights in the read and write access path patterns are determined by combining the association strength evaluation model and the time decay factor, and a lineage dependency graph is constructed based on the dependency weights.

[0012] Perform loop elimination on the bloodline dependency graph to generate a directed acyclic topology; construct a retrieval trunk tree structure based on priority sorting rules, traverse the retrieval trunk tree structure to generate node index encoding intervals; map cross-branch association edges in the directed acyclic topology that are not included in the retrieval trunk tree structure to the association jump bitmap index to generate a probabilistic skeleton tree index structure.

[0013] Receive data tracing retrieval requests, use the node index encoding range in the probabilistic skeleton tree index structure to locate the basic retrieval path on the trunk tree structure; when the dependency weight of the basic retrieval path does not meet the preset threshold, call the associated jump bitmap index to perform multi-path retrieval verification, and output the tracing path and comprehensive confidence score.

[0014] Preferably, the step of parsing unstructured operation logs into structured event records using log parsing templates and associating the structured event records with a heterogeneous operation session set includes: parsing the unstructured operation logs using a tree-based text clustering algorithm, extracting fixed patterns from the log text as event templates, and mapping variable parts of the log text to parameter fields; constructing standardized structured event records, which include timestamp fields, operation subject fields, operation action fields, and data object path fields; and aggregating the structured event records within a preset time window using access subject identifiers to generate a heterogeneous operation session set as a logical storage unit, wherein the access subject identifiers include host address, user identifier, process identifier, thread identifier, and session start time, and the heterogeneous operation session set is formed by aggregating structured event records belonging to the same execution subject.

[0015] Preferably, the step of performing access pattern association analysis on the heterogeneous operation session set and extracting read / write access path patterns includes: storing and scanning atomic operation groups within the heterogeneous operation session set using a frequency pattern growing tree data structure; traversing frequent sub-paths that satisfy the minimum support threshold on the frequency pattern growing tree, filtering out paths that conform to the topology of reading data objects to writing data objects, and defining them as read / write access path patterns; the read / write access path pattern consists of a sequence of access operations from the input data object to the output data object within the heterogeneous operation session set.

[0016] Preferably, the step of determining the dependency weights in the read / write access path pattern by combining the association strength evaluation model and the time decay factor, and constructing a lineage dependency graph based on the dependency weights, includes: constructing an association strength evaluation model based on conditional probability statistics. The construction process of the association strength evaluation model includes: traversing the frequency pattern growth tree to establish a full object index table, and defining a conditional probability calculation logic based on support counting. The conditional probability calculation logic specifies that the global count value of the input data object node in the frequency pattern growth tree is used as the denominator, and the path co-occurrence count value of the input data object node and its descendant output data object node is used as the numerator; calculating the association strength value using the association strength evaluation model, and performing a weighted summation based on the time decay value generated by the operation time difference to determine the dependency weights; performing a graph instantiation operation to map the data object path to a graph node, establishing directed connection edges storing the dependency weights between the graph nodes, and generating a lineage dependency graph.

[0017] Preferably, the step of performing loop elimination on the lineage dependency graph to generate a directed acyclic topology and constructing a retrieval trunk tree structure based on priority sorting rules includes: processing the lineage dependency graph using a depth-first search-based strongly connected component identification algorithm, wherein the strongly connected component identification algorithm maintains a node traversal stack and records the access order identifier and minimum backtracking identifier of each graph node during traversal; when the access order identifier of the current node is detected to be equal to the minimum backtracking identifier, all nodes above the current node in the node traversal stack are popped and identified as cyclic dependency subgraphs; each identified cyclic dependency subgraph is folded into a single node, and the external connection relationships between nodes are preserved to generate a directed acyclic topology; all connecting edges in the directed acyclic topology are sorted in descending order according to the numerical value of the dependency weight; the connecting edge with the largest dependency weight is preferentially selected and added to the tree index structure until all reachable nodes in the directed acyclic topology are covered, forming a retrieval trunk tree structure.

[0018] Preferably, the step of traversing the trunk tree structure to generate the node index encoding interval includes: performing a depth-first traversal operation on the trunk tree structure and maintaining a global sequence counter; when a node in the tree structure is visited, writing the current value of the global sequence counter into the starting position of the interval of the node and incrementing the global sequence counter; when all child nodes of the node have been visited and backtracking has been performed, writing the current value of the global sequence counter into the ending position of the interval of the node and incrementing the global sequence counter; the generated node index encoding interval consists of the starting position and the ending position of the interval, and if the node index encoding interval of the first node contains the node index encoding interval of the second node, then the first node is determined to be the ancestor node of the second node in the trunk tree structure.

[0019] Preferably, the step of mapping cross-branch association edges in the directed acyclic topology that are not included in the retrieval trunk tree structure to the association jump bitmap index to generate a probabilistic skeleton tree index structure includes: identifying directed edges in the directed acyclic topology that exist but are not included in the retrieval trunk tree structure, and defining them as cross-branch association edges; constructing a bitmap index table based on bucket compression technology as the association jump bitmap index; the association jump bitmap index adopts a hierarchical storage structure combining high-bit key-value pairs and low-bit containers, wherein the high-bit part of the source node identifier of the cross-branch association edge is used as the index key value, and the low-bit part of the source node identifier is used as the offset to map to the corresponding bitmap container; wherein the position index corresponding to the set bit in the bitmap container is used to map the target node identifier that can be jumped to from the source node; performing an index encapsulation operation to combine the node index encoding range with the association jump bitmap index to generate a probabilistic skeleton tree index structure.

[0020] Preferably, the step of receiving a data tracing retrieval request and using the node index coding interval in the probabilistic skeleton tree index structure to locate the basic retrieval path on the retrieval trunk tree structure includes: extracting the first node index coding interval of the source data node identifier and the second node index coding interval of the target data node identifier; if it is determined that the starting position value of the first node index coding interval is less than or equal to the starting position value of the second node index coding interval and the ending position value is greater than or equal to the ending position value of the second node index coding interval, then it is confirmed as a tree-like inclusion relationship, and the path connecting the two node identifiers on the retrieval trunk tree structure is taken as the basic retrieval path; otherwise, the associated jump bitmap index is called to search for whether there is a cross-branch association edge connecting the subtree where the source data node identifier and the target data node identifier are located, and the path generated based on the cross-branch association edge is taken as the basic retrieval path.

[0021] Preferably, the process of calling the associated jump bitmap index to perform multi-path retrieval verification and outputting the source path and comprehensive confidence score when the dependency weight of the basic retrieval path does not meet the preset threshold includes: calculating the product of the dependency weights of all connecting edges on the basic retrieval path as the basic probability value; triggering a query operation for the associated jump bitmap index when the basic probability value is lower than the preset threshold; searching in the bitmap container for cross-branch association edges connecting the subtree where the source data node identifier is located and the subtree where the target data node identifier is located through bit operations; if cross-branch association edges exist, performing probability verification calculation, which includes calculating the first complement value of the basic probability value and the second complement value of the jump path weight through the cross-branch association edge, calculating the product of the first complement value and the second complement value, and calculating the complement value of the product as the comprehensive confidence score; the step of outputting the source path and comprehensive confidence score includes returning the set of paths whose comprehensive confidence score meets the output conditions to the retrieval interface.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. This invention transforms unstructured logs into structured records using log parsing templates, constructs a heterogeneous operation session set based on time windows, and combines frequency pattern growth tree mining and probability model calculation of dependency weights. This enables the implicit mining and quantification of causal dependencies between inputs and outputs from massive discrete logs in a telecom-grade heterogeneous computing environment lacking source code access, without the need for code instrumentation or decompiling of binary legacy components. This effectively reconstructs cross-media data lineages across mainframes and streaming computing platforms, filling metadata blind spots.

[0024] 2. This invention eliminates loops and constructs a backbone spanning tree on the lineage graph, uses node index encoding intervals to represent the backbone topology, and maps cross-branch association edges to association jump bitmap indexes to generate a probabilistic skeleton tree index structure. This enables the transformation of traditional source tracing queries based on full graph traversal into extremely low-overhead numerical interval determination and bitmap bit operations in massive data scenarios with hundreds of millions of nodes. This achieves millisecond-level source tracing response while ensuring path integrity, meeting the timeliness requirements of real-time operation and maintenance and accurate integrity verification throughout the data lifecycle.

[0025] 3. This invention establishes a tree-based trunk retrieval mechanism based on intervals and a multi-path retrieval verification mechanism based on bitmaps. By combining association strength and time decay factor to calculate the comprehensive confidence level, it achieves dynamic verification and probability compensation of the mined nondeterministic dependencies, effectively eliminating noisy paths with low confidence and ensuring that the source tracing path output under complex business logic has high credibility and reference value. Attached Figure Description

[0026] Figure 1 This is a flowchart of a data tracing method based on big data mining according to the present invention;

[0027] Figure 2 This is a flowchart illustrating the dependency weight determination process according to an embodiment of the present invention.

[0028] Figure 3 This is a flowchart illustrating the loop elimination and trunk tree construction process according to an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Other embodiments obtained by those skilled in the art based on the ideas in this specification without creative effort all fall within the protection scope of this invention.

[0030] Reference Figures 1 to 3 This invention provides a data tracing method based on big data mining, the specific technical solution of which is as follows:

[0031] Example 1:

[0032] A data tracing method based on big data mining, referring to Figure 1 ,include:

[0033] Collect unstructured operation logs, use log parsing templates to parse the unstructured operation logs into structured event records, and associate the structured event records into a heterogeneous operation session set;

[0034] Access pattern association analysis is performed on a heterogeneous set of operation sessions to extract read and write access path patterns; the dependency weights in the read and write access path patterns are determined by combining the association strength evaluation model and the time decay factor, and a lineage dependency graph is constructed based on the dependency weights.

[0035] Perform loop elimination on the bloodline dependency graph to generate a directed acyclic topology; construct a retrieval trunk tree structure based on priority sorting rules, traverse the retrieval trunk tree structure to generate node index encoding intervals; map cross-branch association edges in the directed acyclic topology that are not included in the retrieval trunk tree structure to the association jump bitmap index to generate a probabilistic skeleton tree index structure.

[0036] Receive data tracing retrieval requests, use the node index encoding range in the probabilistic skeleton tree index structure to locate the basic retrieval path on the trunk tree structure; when the dependency weight of the basic retrieval path does not meet the preset threshold, call the associated jump bitmap index to perform multi-path retrieval verification, and output the tracing path and comprehensive confidence score.

[0037] Furthermore, the step of parsing unstructured operation logs into structured event records using log parsing templates and associating these structured event records with a heterogeneous operation session set includes: parsing the unstructured operation logs using a tree-based text clustering algorithm, extracting fixed patterns from the log text as event templates, and mapping variable parts of the log text to parameter fields; constructing standardized structured event records, which include timestamp fields, operation subject fields, operation action fields, and data object path fields; and aggregating the structured event records within a preset time window using access subject identifiers to generate a heterogeneous operation session set as a logical storage unit, wherein the access subject identifiers include host address, user identifier, process identifier, thread identifier, and session start time, and the heterogeneous operation session set is formed by aggregating structured event records belonging to the same execution subject.

[0038] Specifically, a tree-based text clustering algorithm is used to parse unstructured operation logs, extracting fixed patterns from the log text as event templates and mapping variable parts of the log text to parameter fields. First, unstructured operation logs are collected. These unstructured operation logs refer to raw text data streams generated in real-time in a distributed computing environment that lack a predefined data model. This includes, but is not limited to, access logs from object storage systems, audit logs from database systems, and kernel call logs at the underlying file system level. Such logs are typically represented as string sequences composed of a mixture of natural language text and technical parameters, such as "2025-06-16 10:00:01 User_Admin_01 uploaded / data / finance / finance_report_alpha.dat from 10.0.0.5". Here, "2025-06-16 10:00:01" is the operation time, "User_Admin_01" is the operation subject, "uploaded" is the action type, " / data / finance / finance_report_alpha.dat" is the data object path, and "10.0.0.5" is the source address. However, this information has not yet been parsed into key-value pair format during the collection phase. Subsequently, a tree-based text clustering algorithm is executed: this tree-based text clustering algorithm refers to an algorithm that uses a parse tree structure to perform layer-by-layer matching and similarity clustering of log token sequences. During the algorithm's execution, this method splits the original text log (e.g., "User_A modifiedfinance_report_alpha.dat at 10:00:01") into a token sequence according to a preset delimiter. The parse tree depth is set to 4 levels, with each level's node corresponding to a word position in the sequentially segmented log text. During algorithm execution, the input log is first segmented into a word sequence, and then the word at the current position is compared character-wise with the content of the corresponding node at each level in the tree structure; the edit distance ratio between the two word strings is calculated as the similarity score. When the calculated similarity value is greater than the preset evolution threshold (set to 0.8 in this embodiment) but less than the perfect match value, it is determined that the current log belongs to the event type represented by this branch, and the words with inconsistent content at the current position are identified as dynamic variables. At this time, parameter placeholders are established on the corresponding node of the parse tree, and the text at this position in the original log (such as the file name "finance_report_alpha.dat" or the IP address "10.0.0.5") is marked as the content of the parameter field, while the consistent word sequence on the tree path is retained as a fixed event template. If the similarity value of the current layer does not reach the threshold, a new child node is created under the current node to store the word, thereby forming a new event template branch.Through this difference bit recognition mechanism, the algorithm can automatically map changed content to parameter fields while preserving static string sequences (such as "User<>modified<>at<*>"). In this process, the virtual file "finance_report_alpha.dat" is accurately identified and mapped to parameter fields representing data objects.

[0039] A standardized structured event log is constructed, containing a timestamp field, an operation subject field, an operation action field, and a data object path field. Instantiation is performed to generate a standardized JSON-formatted mapping object for each parsed log instance. The specific assignment process includes: reading the raw log time, converting it to a Unix timestamp accurate to microseconds, and writing it to the timestamp field; extracting the process ID (e.g., PID_4521) or user account (e.g., Finance_Admin) that performed the action from the log text parameters and writing it to the operation subject field; performing action normalization processing, traversing a pre-defined action mapping table, matching verbs in the raw log (e.g., variations of "updated," "wrote," "appended," etc.), converting them to standard atomic actions (e.g., "WRITE"), and writing them to the operation action field; finally, parsing the path information of the operated object, converting it to a complete absolute path, and writing it to the data object path field.

[0040] Structured event records are aggregated within a preset time window using the access subject identifier to generate a set of heterogeneous operation sessions as logical storage units. A session mapping table is maintained in memory to store the set of active heterogeneous operation sessions. The specific aggregation steps are as follows: First, extract the host address, user identifier, process identifier, thread identifier, and session start time of the current structured event record, and combine them to generate a unique five-tuple of access subject identifiers; second, use this access subject identifier as a key to perform a lookup operation in the memory session mapping table; then, execute the time window verification logic: calculate the difference between the timestamp of the current event record and the session start time, and determine whether this difference falls within the preset time window (a sliding time range used to define the logical correlation of operations, as described in this embodiment). The timeframe is set to 300 seconds, a value derived from the average time consumption of a single complex business flow (from read to write) within the cloud platform. Finally, if the access subject identifiers are consistent and the time window verification condition is met, the structured event record is appended to the corresponding session set in chronological order, thus dynamically restoring the scattered logs into an operation flow with business continuity. If the condition is not met or no corresponding session is found, a new session set is initialized. The heterogeneous operation session set refers to an ordered set that logically carries all operation sequences belonging to the same access subject identifier and occurring within the 300-second time envelope. This method maintains a session mapping table in memory. All event records with consistent access subject identifiers are pushed into the corresponding session set in chronological order, thus restoring the scattered logs into an operation flow with business continuity.

[0041] As a preferred implementation, the step of aggregating structured event records within a preset time window using the access subject identifier further includes: real-time statistical analysis of the distribution density of operation events under the current access subject identifier, and calculation of the operation behavior entropy value using the Shannon entropy formula; when the operation behavior entropy value exceeds a preset stability threshold, linearly compressing the preset time window length according to the rate of change of the entropy value to achieve fine segmentation of high-frequency concurrent operations; when the operation behavior entropy value is lower than the preset stability threshold, dynamically expanding the preset time window length according to the variance of the time interval of the operation events to achieve complete coverage of long transaction operation logic.

[0042] Specifically, the operational behavior entropy value is a mathematical quantity used to characterize the dispersion of the frequency of operations performed by an accessing subject within a unit of time. Its calculation process is as follows: First, within a preset time window, the proportion of each type of atomic operation (read, write, delete, etc.) occurring relative to the total number of operations is statistically analyzed, serving as the probability of occurrence for that operation type. Then, the product of the probability of occurrence for each operation type and the logarithm of that probability is calculated. Finally, the products of all operation types are summed and their negative values ​​are taken to obtain the operational behavior entropy value. Taking the operational subject identifier for the financial document "finance_report_alpha.dat" as an example, the preset time window of 300 seconds is divided into several 1-second observation micro-elements. If the event distribution within each observation micro-element is extremely dense and uniform, the calculated Shannon entropy value will significantly increase. The preset stability threshold is set to 2.5 bits, a value derived from the 95th percentile based on probabilistic modeling of historical stable business flow samples.

[0043] When the entropy value of an operation behavior rises to 4.8 bits and exceeds the preset stability threshold, it is determined that the current stage is a high-concurrency processing stage. The rate of change of the entropy value of the operation behavior is obtained, and a compression operation is performed on the preset time window length according to the preset linear scaling factor (e.g., the window shrinks by 50 seconds for every 1 bit increase in entropy value). For example, the window is adjusted from 300 seconds to 15 seconds in real time. This measure ensures that the heterogeneous operation session set generated under high load can accurately lock a single logical task by increasing the sampling frequency, and prevents different business instructions from being confused in the same session.

[0044] When the entropy value of the operation behavior drops to 0.5 bits and is lower than the preset stability threshold, it is determined that the current period is a long transaction execution interval. The variance of the time interval between adjacent events under this subject is extracted. If the variance value exceeds the preset fluctuation threshold (e.g., the variance value is greater than 5000), a window expansion operation is performed to extend the preset time window length from 300 seconds to 1200 seconds. During this process, even if the cross-database processing of "finance_report_alpha.dat" has a long waiting time for response, this method can still classify it into the same logical storage unit, avoiding the break in causal lineage caused by physical time slicing.

[0045] By introducing a dynamic window mechanism based on behavioral entropy, the aggregation granularity and the actual operation intensity are adaptively aligned, effectively solving the problem of discontinuous data lineage under complex loads and improving the integrity of the tracing logic.

[0046] By employing sophisticated tree-like clustering and multi-dimensional session aggregation techniques, high-entropy, disordered unstructured logs are transformed into structured event sequences with strong logical connections. This effectively filters log noise and provides standardized data input for the subsequent construction of a high-precision lineage dependency graph.

[0047] Furthermore, referring to Figure 2 The steps of performing access pattern association analysis on the heterogeneous operation session set and extracting read / write access path patterns include: storing and scanning atomic operation groups within the heterogeneous operation session set using a frequency pattern growing tree data structure; traversing frequent sub-paths that satisfy the minimum support threshold on the frequency pattern growing tree, filtering out paths that conform to the topology of reading data objects to writing data objects, and defining them as read / write access path patterns; the read / write access path pattern consists of the sequence of access operations from the input data object to the output data object within the heterogeneous operation session set.

[0048] Specifically, a frequency-pattern growing tree data structure is used to store and scan atomic operation groups within a heterogeneous operation session set. The frequency-pattern growing tree data structure refers to a prefix tree structure that compresses and stores frequent itemsets in a tree-like topology through two scans of the dataset, containing an item header table and a multi-level association tree. During storage, each heterogeneous operation session set is treated as a logical transaction, and the atomic operations within the session (e.g., READfinance_report_alpha.dat) are treated as items in the transaction. The compressed storage construction process based on two scans is executed as follows: First, the first scan is performed, traversing all session sets, counting the global frequency of each atomic operation, and establishing an item header table, sorting all operation items in descending order of frequency; second, the second scan is performed, traversing the heterogeneous operation session set again, and rearranging the atomic operations in each session according to the order of the item header table; finally, a tree structure insertion operation is performed, mapping the rearranged operation sequence to a multi-level association tree; if the current path node already exists, its cumulative count is increased; if it does not exist, a new node is created and a parent-child connection is established. This process compresses and stores large-scale log data into a prefix tree structure. Subsequent scanning processes directly perform path counting statistics on this compressed structure, eliminating the need to repeatedly read the original logs.

[0049] The frequent sub-paths that satisfy the minimum support threshold are traversed on the frequency pattern growth tree. Paths that conform to the topology of reading data objects to writing data objects are selected and defined as read / write access path patterns. The minimum support threshold refers to a numerical proportion used to filter low-frequency random noise and retain statistically significant paths. In this embodiment, it is set to 0.02 (i.e., the probability of the operation sequence appearing in the total session sample must reach 2%). This value is set based on an empirical value obtained from the distribution density test of random operation trajectories generated by concurrent conflicts in a distributed system. First, a bottom-up recursive mining operation is performed: starting from the bottom item of the item header table, low-frequency random noise is filtered using the preset minimum support threshold, conditional patterns are extracted, and a conditional frequency pattern growth tree is constructed to mine all frequent item sequences that satisfy the statistical significance. Then, a topological logic filtering operation is performed: the extracted frequent item sequences are traversed, and the logical integrity of the sequences is checked; the start operation type and end operation type of the sequence are identified, and only path segments with the start operation being "read" and the end operation being "write" or "export" and whose intermediate logic conforms to the causal law of data flow are retained, while incomplete fragmented paths are eliminated.

[0050] The read / write access path pattern consists of a sequence of access operations from the input data object to the output data object within a heterogeneous set of operation sessions. This read / write access path pattern refers to a directed logical chain describing the flow of data from the source endpoint to the destination endpoint. Structurally, it is represented as an ordered operation vector, with the vector dimension defined as [Batch, Sequence_Length], where Sequence_Length depends on the number of data processing steps in the specific business logic. Taking a virtual finance scenario as an example, a typical read / write access path pattern is: [Read original voucher A, process P processes, writes to temporary summary table B, exports finance_report_alpha.dat]. This pattern clearly defines the direct or indirect derivation relationship between the input data object (original voucher A) and the output data object (finance_report_alpha.dat), providing atomic-level logical connections for subsequent construction of a dependency graph.

[0051] By introducing frequency pattern growing trees and support filtering mechanisms, statistically significant data flow models are automatically extracted from high-concurrency, fragmented user sessions, eliminating redundant interference terms and significantly improving the accuracy and mining efficiency of lineage path identification.

[0052] Furthermore, the step of combining the association strength evaluation model and the time decay factor to determine the dependency weights in the read / write access path pattern, and constructing a lineage dependency graph based on the dependency weights, includes: constructing an association strength evaluation model based on conditional probability statistics. The construction process of the association strength evaluation model includes: traversing the frequency pattern growth tree to establish a full object index table, and defining a conditional probability calculation logic based on support counting. The conditional probability calculation logic specifies that the global count value of the input data object node in the frequency pattern growth tree is used as the denominator, and the path co-occurrence count value of the input data object node and its descendant output data object node is used as the numerator; calculating the association strength value using the association strength evaluation model, and performing a weighted summation based on the time decay value generated based on the operation time difference to determine the dependency weights; performing a graph instantiation operation to map the data object path to a graph node, establishing directed connection edges storing the dependency weights between the graph nodes, and generating a lineage dependency graph.

[0053] Specifically, a correlation strength assessment model based on conditional probability statistics is constructed. This process does not simply involve introducing a black-box algorithm, but rather establishing a defined computational logic framework. The specific construction steps include: First, establishing a full object index table, traversing the previously constructed frequency pattern growth tree, extracting the unique data object identifiers (i.e., non-repeating file path strings) of all data objects involved in read and write operations in the tree, using these identifiers as keys, and storing the global occurrence frequency of the object recorded in the tree structure as the value in the key-value pair index table, thus providing a fast lookup basis for subsequent denominator calculations; Second, defining the conditional probability calculation logic within the correlation strength assessment model. This logic explicitly specifies that the global occurrence frequency of the input data object node recorded in the full object index table is used as the divisor (i.e., the denominator), and the path co-occurrence count value of the input data object node and its descendant output data object node in the same path sequence is used as the dividend (i.e., the numerator). By establishing this operational rule, the model can quantify the statistical probability that "given that the input object has occurred, the output object will subsequently occur."

[0054] The association strength assessment model extracts the global occurrence frequency of input data objects and the joint frequency of co-occurrence of input and output data objects from the frequency pattern growth tree. The ratio of the joint frequency to the global occurrence frequency is calculated as the association strength value. The aforementioned calculation logic is executed, extracting data from the frequency pattern growth tree and the full object index table and performing calculations. Specifically, when processing a specific read / write access path pattern, the full object index table is queried using the input data object identifier as the index key to obtain its global occurrence frequency (assumed to be 1000 times). Simultaneously, in the frequency pattern growth tree, the joint frequency of the input data object and the target output data object appearing sequentially in the same session sequence is calculated (assumed to be 850 times). A division operation is performed, dividing the joint frequency by the global occurrence frequency; the quotient (e.g., 0.85) is the association strength value. This value is within the range of 0 to 1, reflecting the statistical certainty of data flow.

[0055] The time difference between read and write operations is calculated, and a time decay value is generated based on a time decay factor. First, the precise times of read and write operations in the read / write access path pattern are extracted, and the absolute time difference between them is calculated (accurate to milliseconds). Second, a mapping calculation is performed using a time decay factor. This time decay factor employs exponential decay logic based on the natural constant (i.e., the mathematical constant e) to simulate the weakening of causal relationships over time. The proportionality coefficient involved in the decay calculation is preset to 0.005 (this value is set based on the average lifespan of data processing processes within the cloud platform (approximately 200 seconds) to ensure that operational relationships with large spans are assigned lower weights). The specific calculation steps are as follows: First, the product of the aforementioned time difference and the proportionality coefficient 0.005 is calculated. Then, using the natural constant as the base and the opposite of the product (i.e., the negative value) as the exponent, an exponential operation is performed. The final result is defined as the time decay value.

[0056] The first component is calculated by multiplying the correlation strength value by a first preset adjustment coefficient, and the second component is calculated by multiplying the time decay value by a second preset adjustment coefficient. The first and second components are then added together to obtain the dependency weight. The weighted calculation process is as follows: the first preset adjustment coefficient is set to 0.7, and the second preset adjustment coefficient is set to 0.3. This coefficient ratio is based on the optimal ratio determined after a grid search on the historical source tracing accuracy validation set, aiming to assign a higher weight to statistical correlation. The specific calculation steps are as follows: the correlation strength value calculated above is multiplied by the first preset adjustment coefficient, and the resulting value is used as the first component; simultaneously, the time decay value is multiplied by the second preset adjustment coefficient, and the resulting value is used as the second component; finally, the first and second components are added together, and the sum is defined as the dependency weight. This weight comprehensively represents the statistical significance and temporal tightness of data flow.

[0057] The graph instantiation operation maps the unique data object paths involved in the read / write access path patterns to graph nodes. For each read / write access path pattern, a directed connection edge is established between the corresponding input graph node and output graph node, and the dependency weight is written into the attribute field of the directed connection edge, generating a weighted directed graph with probability attributes as a lineage dependency graph. The lineage dependency graph refers to a directed weighted graph with data objects as vertices and flow relationships as edges. During instantiation, each unique file path is traversed, and a corresponding graph node object is created in memory. For each read / write access path pattern, a directed connection edge is established between the corresponding input graph node and output graph node, pointing from the input to the output. Subsequently, an attribute writing operation is performed: the dependency weights calculated in the previous steps are used as key attribute parameters and written into the metadata field of the directed connection edge. Through the above steps, a weighted directed graph with probability attributes is generated as a lineage dependency graph, thereby constructing a topology view that can quantitatively describe the probability of data flow throughout the entire lifecycle.

[0058] By integrating conditional probability statistics with exponential time decay logic, the logical derivation strength between data objects was scientifically quantified, realizing the transformation of kinship from fuzzy association to precise probability distribution, and providing a highly reliable data foundation for subsequent confidence assessment of complex paths.

[0059] Furthermore, referring to Figure 3The steps of performing loop elimination on the lineage dependency graph to generate a directed acyclic topology and constructing a retrieval trunk tree structure based on priority sorting rules include: processing the lineage dependency graph using a depth-first search-based strongly connected component identification algorithm, wherein the strongly connected component identification algorithm maintains a node traversal stack and records the access order identifier and minimum backtracking identifier of each graph node during traversal; when the access order identifier of the current node is detected to be equal to the minimum backtracking identifier, all nodes above the current node in the node traversal stack are popped and identified as cyclic dependency subgraphs; each identified cyclic dependency subgraph is folded into a single node, while retaining the external connection relationships between nodes, generating a directed acyclic topology; all connecting edges in the directed acyclic topology are sorted in descending order according to the numerical value of the dependency weight; the connecting edge with the largest dependency weight is preferentially selected and added to the tree index structure until all reachable nodes in the directed acyclic topology are covered, forming a retrieval trunk tree structure.

[0060] Specifically, a depth-first search-based strongly connected component identification (FTCI) algorithm is used to process the lineage dependency graph. This algorithm maintains a node traversal stack and records the access order identifier and minimum backtracking identifier for each graph node during traversal. The access order identifier is an incrementing integer value (e.g., 1, 2, 3) assigned according to the order in which nodes are first visited; the minimum backtracking identifier is the node with the smallest access order identifier that can be reached via a directed edge. When processing a lineage graph such as the virtual financial file "finance_report_alpha.dat", the algorithm starts from the initial node, pushes newly discovered nodes (e.g., the node representing the "original statement") onto the node traversal stack, and initializes their access order identifier and minimum backtracking identifier. During the recursive search, if the current node points to an ancestor node already in the stack, the minimum backtracking identifier of the current node is updated to record the existence of a cycle.

[0061] When the access order identifier of the current node is detected to be equal to the minimum backtracking identifier, all nodes above the current node in the node traversal stack are popped and identified as a circular dependency subgraph. A circular dependency subgraph refers to a directed subgraph that forms a closed loop during data flow due to logical feedback or abnormal calls. When the depth-first traversal backtracks to a node (let's call it node K), and the minimum backtracking identifier of node K is still its own access order identifier, node K is determined to be the root of a strongly connected component. At this point, the top nodes of the stack are continuously popped until node K is popped. This set of popped nodes is then identified as a circular dependency subgraph, representing a closed loop path with mutual references within the generation logic of "finance_report_alpha.dat".

[0062] Each identified cyclically dependent subgraph is folded into a single node while preserving the external connections between nodes, generating a directed acyclic topology. The folding operation involves merging all member nodes within the cyclically dependent subgraph into a single logical supernode and hiding or removing directed edges within the subgraph. For incoming edges from outside the subgraph to inside nodes, their endpoints are uniformly redirected to the supernode; for outgoing edges from inside the subgraph to outside nodes, their starting points are uniformly redirected to the supernode. This process transforms the original complex directed graph into a unidirectional, directed acyclic topology, eliminating the risk of infinite loops that may arise during data tracing.

[0063] In a preferred embodiment, the step of adding the first component and the second component to obtain the dependency weight further includes: obtaining the privilege level identifier of the access subject according to the operation subject field, and obtaining the sensitivity label of the target data object according to the data object path field; calculating the product of the privilege level identifier and the sensitivity label as the privilege mapping coefficient; and using the privilege mapping coefficient as an exponent to perform a power operation on the sum of the first component and the second component to generate the corrected dependency weight.

[0064] Specifically, the privilege level identifier refers to a scalar value used to characterize the logical weight of the access subject, defined in the range of 0.1 to 1.0. During the tracing process of "finance_report_alpha.dat", this method queries a preset account dictionary based on the operation subject field, setting the privilege level identifier of the system background backup plugin to 0.2 and the privilege level identifier of the finance manager account to 0.9. The sensitivity label refers to a weighted value used to measure the security level of a data object. For files stored in the core financial database, the sensitivity label is preset to 1.0; for ordinary system operation logs, the sensitivity label is preset to 0.1.

[0065] The product of the two is calculated as the privilege mapping coefficient. When the high-privilege backup plugin (0.2) performs high-frequency scanning of the low-sensitivity runtime log (0.1), the calculated privilege mapping coefficient is 0.02. At this time, the sum of the first and second components (assuming an initial sum of 0.8 due to extremely high statistical frequency) is used as the base, and a power operation is performed with 0.02 as the exponent. The resulting corrected dependency weight will approach 1 (under this power logic, it represents the transformation of linear probability into background probability). More importantly, when the judgment subject level and object sensitivity do not match (e.g., low-privilege backend accounts frequently write high-sensitivity business files), this power operation can significantly suppress statistical deviations caused by non-business causality.

[0066] Through this nonlinear correction process, the identification and removal of "lineage noise" are achieved. In the generated lineage dependency graph, high-frequency associated edges generated by system automatic cleaning, index reconstruction and other technical behaviors are assigned extremely low significance weights, while the statistical weights of the actual business flow (highly sensitive file operations triggered by the business subject) are effectively preserved and highlighted because the privilege mapping coefficient is close to 1. By introducing semantic privilege constraints on subjects and objects, intelligent calibration of statistical weights is achieved, eliminating the interference of system background noise on data lineage from the root and significantly improving the anti-pollution capability of the tracing path.

[0067] All connecting edges in the directed acyclic topology are sorted in descending order of their dependency weights. The process iterates through all directed edges in the topology, extracting the dependency weight values ​​(e.g., 0.95, 0.88, 0.42, etc.) calculated in previous steps for each edge. A global edge weight sequence is then built using a quicksort algorithm based on these values. This sorting logic aims to ensure that, during subsequent index building, core lineage paths with the highest statistical correlation and greatest likelihood of flow are prioritized.

[0068] The process prioritizes adding the connecting edges with the largest dependency weights to the tree index structure until all reachable nodes in the directed acyclic topology are covered, forming the retrieval trunk tree structure. The retrieval trunk tree structure refers to a forest or tree-like logical index composed of edges with maximum weights, free of loops, and connecting all reachable nodes. Connecting edges are extracted sequentially from the descending edge sequence. If an extracted edge connects two nodes that are not yet connected in the current tree structure, the edge and its weight are written into the trunk index. If the edge would cause a logical conflict or duplicate connection, it is marked as a cross-branch association edge. This process continues until all data object nodes in the topology (including collapsed supernodes) are included in the trunk tree.

[0069] This method transforms complex network kinship relationships into efficient tree-like indexes by using graph compression technology based on strongly connected components and the maximum weight spanning tree algorithm. This not only eliminates the risk of loop retrieval but also establishes the main causal path of data flow.

[0070] Furthermore, the step of traversing the trunk tree structure to generate node index encoding intervals includes: performing a depth-first traversal operation on the trunk tree structure and maintaining a global sequence counter; when a node in the tree structure is visited, writing the current value of the global sequence counter into the starting position of the node's interval and incrementing the global sequence counter; when all child nodes of the node have been visited and backtracking has been performed, writing the current value of the global sequence counter into the ending position of the node's interval and incrementing the global sequence counter; the generated node index encoding interval consists of the starting position and the ending position of the interval. If the node index encoding interval of the first node contains the node index encoding interval of the second node, then the first node is determined to be the ancestor node of the second node in the trunk tree structure.

[0071] Specifically, a depth-first traversal operation is performed on the retrieval trunk tree structure, and a global sequence counter is maintained. An integer variable named "global sequence counter" is initialized, with an initial value of 1. The depth-first traversal operation starts from the root node of the retrieval trunk tree structure (e.g., the "raw log collection bus" node in an enterprise private cloud platform) and probes downwards along the deepest branch of the tree using a recursive stack-based approach. Throughout the traversal cycle, the global sequence counter serves as a unique incrementing scale, recording the logical time points of entry and exit for each node, ensuring that all nodes have a unique positional identifier in the temporal dimension.

[0072] When a node in the tree structure is accessed, the current value of the global sequence counter is written to the starting position of the node's interval, and the global sequence counter is incremented. Taking the retrieval of an intermediate processing node in the main tree structure (e.g., the virtual node "Data_Encryption_Node" that performs encryption on "finance_report_alpha.dat") as an example, when the traversal stream enters this node, the real-time value of the current global sequence counter (let's say 15) is read, and the value 15 is stored as the starting position of the node's interval in its metadata object. Subsequently, an increment operation is performed to update the value of the global sequence counter to 16.

[0073] When all child nodes of a node have been visited and backtracked, the current value of the global sequence counter is written to the end position of the node's interval, and the global sequence counter is incremented. After completing a full traversal of all sub-paths under the "Data_Encryption_Node" node (such as child nodes for encryption key generation, ciphertext verification, etc.), the traversal stream backtracks to the encryption node. At this point, the accumulated global sequence counter value (assuming it becomes 22 after multiple increments in its subtree) is read, and the value 22 is stored as the end position of the node's interval in its metadata object. Subsequently, the increment operation is performed again to update the value of the global sequence counter to 23.

[0074] The generated node index encoding interval consists of a start position and an end position. If the node index encoding interval of the first node contains the node index encoding interval of the second node, then the first node is determined to be the ancestor node of the second node in the retrieval trunk tree structure. The node index encoding interval refers to a closed interval that represents the coverage area of ​​a node in the tree topology through double-boundary numerical values. This method generates an interval identifier of the form [start position, end position] for each data object node. For example, if the interval of the first node is [15, 22] and the interval of the second node is [17, 20], numerical logic determines that 15 is less than or equal to 17 and 22 is greater than or equal to 20, thus establishing that the first node is the ancestor node of the second node. In a physical scenario, this means that data must flow from the first node to the second node, establishing a definite causal chain.

[0075] By using a depth-first temporal interval encoding mechanism, complex tree path queries are transformed into numerical interval inclusion determination with extremely low computational overhead, which significantly improves the efficiency of determining the grandparent-grandchild relationship between nodes in large-scale data lineage trees and achieves constantization of retrieval complexity.

[0076] Further, the step of mapping cross-branch association edges in the directed acyclic topology that are not included in the retrieval trunk tree structure to the association jump bitmap index to generate a probabilistic skeleton tree index structure includes: identifying directed edges in the directed acyclic topology that exist but are not included in the retrieval trunk tree structure, and defining them as cross-branch association edges; constructing a bitmap index table based on bucket compression technology as the association jump bitmap index; the association jump bitmap index adopts a hierarchical storage structure combining high-bit key-value pairs and low-bit containers, wherein the high-bit part of the source node identifier of the cross-branch association edge is used as the index key value, and the low-bit part of the source node identifier is used as the offset to map to the corresponding bitmap container; wherein the position index corresponding to the set bit in the bitmap container is used to map the target node identifier that can be jumped to from the source node; performing an index encapsulation operation to combine the node index encoding range with the association jump bitmap index to generate a probabilistic skeleton tree index structure.

[0077] Specifically, directed edges that exist in the directed acyclic topology but are not included in the retrieval trunk tree structure are identified and defined as cross-branch related edges. These cross-branch related edges refer to dependency edges that represent non-core flow paths in the directed acyclic graph but are excluded from the trunk when constructing a maximally weighted spanning tree because they do not meet tree topological constraints (e.g., they would cause duplicate in-degrees or low weights in the spanning tree). All non-tree edges are extracted by performing a difference operation between the edge set of the directed acyclic topology and the edge set of the retrieval trunk tree structure. Taking the flow scenario of the financial document "finance_report_alpha.dat" as an example, if the trunk tree records the main data processing pipeline, and a logical edge generated by a temporary offline audit backup operation does not enter the trunk, then this backup related path is identified as a cross-branch related edge.

[0078] A bitmap index table based on bucket compression technology is constructed as the associative jump bitmap index. Bucket compression technology refers to a technique that divides a large-scale integer numerical space into several logical intervals (buckets) of fixed size, and dynamically selects a bitmap or array for compressed storage based on the sparsity of the data within each interval. The associative jump bitmap index refers to a compressed index table that stores non-tree edge jump relationships. A 32-bit integer identifier space is pre-defined for all graph nodes, and its logical representation is divided into two levels: the high 16 bits and the low 16 bits of binary representation. This allows for the construction of a retrieval container supporting fast bit operations at the algorithm logic layer without relying on specific hardware memory.

[0079] The associated jump bitmap index employs a hierarchical storage structure combining high-order key-value pairs and low-order containers. The specific data dimension splicing and mapping process includes: performing dimension alignment operations for source node location and target node mapping. The high-order part of the source node identifier in the cross-branch association edge is used as the index key value to locate the storage space, while the target node identifier, after mapping, serves as the offset of the bitmap. The specific steps are: obtaining the 32-bit integer value of the source node ID in the cross-branch association edge, performing a bitwise operation to shift it 16 bits to the right to extract the high 16 bits, which serve as the index key value of the hash table to uniquely locate a logical bucket; maintaining a dedicated bitmap container within this bucket; and simultaneously performing a bitwise AND operation on the source node ID using the low 16-bit mask to extract the low 16 bits, which are then used to further lock the row bitmap region corresponding to the source node within the logical bucket.

[0080] In a bitmap container, the set bits logically represent the identifier of the target node from which a jump can occur. When it is determined that there is a cross-branch association edge between source node A and target node B, intermediate processing steps such as hash mapping or modulo operation are first performed on the identifier of target node B to generate a target offset, ensuring the consistency between the data dimension and the bitmap space. Subsequently, in the row bitmap area locked by source node A, the bit corresponding to the target offset is changed from binary "0" to binary "1". At the same time, a weight mapping sequence consistent with the logical address of the bitmap container is constructed, and the dependency weight value of the cross-branch association edge is written into the corresponding position of the weight mapping sequence. This method of recording complex mesh jump logic with bit "setting" and weight synchronization can record the complex mesh jump logic with a very high compression ratio, ensuring that each set bit represents a possible data flow from one branch of the trunk tree to another and its corresponding confidence weight.

[0081] An index encapsulation operation is performed, combining the node index encoding range with the associated jump bitmap index to generate a probabilistic skeleton tree index structure. This probabilistic skeleton tree index structure is a comprehensive index object that organically integrates tree-like range-containing decision logic with non-tree-like bitmap jump logic. It logically binds the [start position, end position] range attribute of each node in the retrieval trunk tree structure to its bitmap container in the associated jump bitmap index. When performing full lineage tracing for "finance_report_alpha.dat", this structure supports seamless switching between deterministic tree paths and probabilistic jump edges in the retrieval logic, thereby constructing a complete data flow skeleton containing the trunk and branches.

[0082] By using hierarchical bitmap indexing technology to compress and store non-core lineage relationships, the problem of memory overflow caused by storing the full adjacency matrix in a massive node environment is solved, achieving efficient indexing of complex topological paths and ensuring the integrity of source tracing retrieval.

[0083] Furthermore, the step of receiving the data tracing retrieval request and locating the basic retrieval path on the main tree structure using the node index coding interval in the probabilistic skeleton tree index structure includes: extracting the first node index coding interval of the source data node identifier and the second node index coding interval of the target data node identifier; if it is determined that the starting position value of the first node index coding interval is less than or equal to the starting position value of the second node index coding interval and the ending position value is greater than or equal to the ending position value of the second node index coding interval, then a tree-like inclusion relationship is confirmed, and the path connecting the two node identifiers on the main tree structure is taken as the basic retrieval path; otherwise, the associated jump bitmap index is called to search for whether there is a cross-branch association edge connecting the subtree where the source data node identifier and the target data node identifier are located, and the path generated based on the cross-branch association edge is taken as the basic retrieval path.

[0084] Specifically, the first node index encoding range of the source data node identifier and the second node index encoding range of the target data node identifier are extracted. The basic retrieval path refers to an initial logical sequence connecting the source data object and the target data object in a probabilistic skeleton tree index structure. When the system receives a tracing request for the virtual file "finance_report_alpha.dat", it first identifies the source data node identifier (e.g., original account statement ID) and the target data node identifier (e.g., final report ID) in the request message. By querying the index information stored in the metadata mapping table, the first node index encoding range (e.g., [10, 100]) corresponding to the source data node and the second node index encoding range (e.g., [30, 70]) corresponding to the target data node are obtained.

[0085] If the starting position of the indexed interval of the first node is less than or equal to the starting position of the indexed interval of the second node, and the ending position is greater than or equal to the ending position of the indexed interval of the second node, then a tree-like inclusion relationship is confirmed, and the path connecting the two node identifiers on the retrieval trunk tree structure is used as the basic retrieval path. Numerical inclusion logic is applied to the two extracted intervals. Taking the first interval [10, 100] and the second interval [30, 70] as an example, since 30 is greater than or equal to 10 and 70 is less than or equal to 100, it is determined that they have a definite grandparent-grandchild topological relationship on the retrieval trunk tree structure. At this point, a reverse backtracking or forward traversal is performed along the parent-child pointers of the trunk tree to extract the unique connection sequence between these two nodes, which is defined as the basic retrieval path.

[0086] Otherwise, the associated jump bitmap index is invoked to search for the existence of a cross-branch association edge connecting the source data node identifier and the target data node identifier in their respective subtrees, and the path generated based on this cross-branch association edge is used as the basic search path. If the interval inclusion logic is not true (e.g., the first interval is [10, 50], and the second interval is [60, 90]), it indicates that the two data objects are distributed on different branches of the trunk tree. In this case, the corresponding bitmap container in the associated jump bitmap index is located according to the source data node identifier, and bitwise operations are performed to search whether the branch to which the source node belongs has a cross-branch association edge pointing to the branch where the target node is located. Once a set bit is found in the bitmap, the jump edge is extracted and concatenated with the local path on the trunk tree to generate the basic search path that crosses branches.

[0087] By transforming complex graph path finding into logical judgment based on numerical ranges and bitmap jump retrieval, high-speed positioning of data lineage paths is achieved, ensuring that source tracing retrieval requests can be responded to in real time in massive data environments, while taking into account the retrieval integrity of both main and peripheral paths.

[0088] Furthermore, the process of calling the associated jump bitmap index to perform multi-path retrieval verification and outputting the source path and comprehensive confidence score when the dependency weight of the basic retrieval path does not meet the preset threshold includes: calculating the product of the dependency weights of all connecting edges on the basic retrieval path as the basic probability value; triggering a query operation for the associated jump bitmap index when the basic probability value is lower than the preset threshold; searching in the bitmap container for cross-branch associated edges connecting the subtree where the source data node identifier is located and the subtree where the target data node identifier is located through bit operations; if cross-branch associated edges exist, performing probability verification calculation, which includes calculating the first complement value of the basic probability value and the second complement value of the jump path weight through the cross-branch associated edges, calculating the product of the first complement value and the second complement value, and calculating the complement value of the product as the comprehensive confidence score; the step of outputting the source path and comprehensive confidence score includes returning the set of paths whose comprehensive confidence score meets the output conditions to the retrieval interface.

[0089] Specifically, the base probability value is calculated as the product of the dependency weights of all connected edges on the basic retrieval path. The dependency weight value of each directed connected edge in the basic retrieval path is extracted. This base probability value refers to the joint probability that data will flow completely from the source node to the target node, assuming that each level of data flow operation is an independent random event. The calculation process involves performing consecutive multiplication operations on all weight scalars on the path. Taking the source tracing path for "finance_report_alpha.dat" as an example, if the basic retrieval path contains three edges with dependency weights of 0.9, 0.8, and 0.9 respectively, then the base probability value is the product of these three values, i.e., 0.648.

[0090] When the base probability value is lower than a preset threshold, a query operation on the associated jump bitmap index is triggered. The preset threshold is a logical critical value used to determine whether a single tracing path has sufficient credibility; in this embodiment, it is set to 0.75. This value is set based on statistical analysis of the accuracy distribution of historical manual tracing samples on the enterprise cloud platform, aiming to filter out low-confidence scenarios that may have multi-path flow interference. When the calculated base probability value (e.g., 0.648) is less than 0.75, multi-path redundancy verification logic is immediately initiated.

[0091] Bitwise operations are used to search the bitmap container for cross-branch association edges connecting the subtree containing the source data node identifier and the subtree containing the target data node identifier. The specific retrieval and extraction process is as follows: First, the corresponding bitmap container and its dedicated row bitmap are located in the association jump bitmap index using the source data node identifier. Then, for candidate nodes within the target data node identifier and its subtree, a hash mapping or modulo operation, identical to that performed in the index building phase, is executed to generate the corresponding target offset set. Finally, the bits corresponding to the target offsets in the row bitmap are checked to see if they are set. Each set bit represents an auxiliary flow path that bypasses the retrieval backbone tree structure.

[0092] If a bit at a specific location is set, it indicates a potential jump path or hash mapping collision. In this case, using the target offset as an index pointer, the weight mapping sequence consistent with the bitmap container's logical address is accessed, and the data stored at that location is read. If the read content is a valid dependent weight value (i.e., a value greater than zero), the cross-branch association edge is confirmed to exist, and the weight is accurately extracted. If the read content is empty or marked as invalid, the setting is determined to be a false hit caused by a hash collision and is ignored. This dual-verification mechanism, combining initial bitmap screening with weight array review, effectively solves the problem of misjudgment caused by target node mapping collisions while confirming the existence of the path, ensuring the accuracy of the tracing results.

[0093] As a preferred implementation, the step of calling the associated jump bitmap index for multi-path retrieval verification further includes: calculating the logical hop count of the two nodes connected by the cross-branch associated edge on the retrieval trunk tree structure, defined as the topological deviation distance; generating a path stability factor based on the reciprocal of the topological deviation distance; and, when performing probability verification calculation, using the product of the jump path weight and the path stability factor as the corrected jump weight to participate in the calculation of the comprehensive confidence level; wherein, the larger the topological deviation distance, the smaller the path stability factor, in order to penalize random associated edges that are physically and logically too far apart.

[0094] Specifically, the logical hop count of the two nodes connected by the cross-branch association edge in the retrieval trunk tree structure is calculated. The logical hop count refers to the minimum total number of edges required to connect two nodes through their nearest common ancestor in the retrieval trunk tree structure. During the validation of "finance_report_alpha.dat", the logical hop count of the two nodes connected by this cross-branch association edge in the trunk tree was identified as 15 hops, which is defined as the topological deviation distance. A larger hop count reflects that the two data objects are physically and logically located on relatively isolated branches. Taking the reciprocal of the topological deviation distance (1 / 15) yields approximately 0.067, which is used as the path stability factor. Subsequently, the original jump path weight (0.5) of this cross-branch association edge is multiplied by the path stability factor (0.067) to obtain a corrected jump weight of 0.0335.

[0095] By introducing a penalty mechanism based on logical distance, accidental associations across business domains are effectively identified and suppressed, ensuring the physical rationality of multi-path probability fusion and enhancing the credibility of the tracing conclusions.

[0096] If cross-branch related edges exist, a probability verification calculation is performed. This calculation includes calculating the first complement of the base probability value and the second complement of the weight of the jump path through the cross-branch related edge. The product of the first and second complements is then calculated, and the complement of this product is used as the overall confidence level. The overall confidence level refers to the total probability that the data lineage relationship is valid after comprehensively considering the main path and cross-branch paths. First, the first complement (i.e., subtracting the base probability value from 1, for example, 1-0.648=0.352) and the second complement (i.e., subtracting the dependency weight of the jump path from 1; assuming the jump weight is 0.5, the complement is 0.5) are calculated. Multiplying these two complements yields the probability of the entire path failing (0.352 multiplied by 0.5 equals 0.176). Finally, subtracting this product (1-0.176=0.824) from 1 gives the overall confidence level. This logic scientifically integrates the evidence strength of multiple tracing paths using the union event calculation principle in probability theory.

[0097] The steps for outputting the tracing path and overall confidence score include returning the set of paths whose overall confidence score meets the output criteria to the retrieval interface. The output criteria refer to the overall confidence score needing to reach a preset minimum publication threshold (e.g., 0.6). The finally calculated overall confidence score value (e.g., 0.824) is associated and encapsulated with the basic retrieval path and the identified cross-branch related edge paths. The generated output data object contains the complete node sequence, the weights of each segment, and the quantified overall confidence score scalar. Finally, the result is returned to the front-end retrieval interface through a standard application programming interface (API) for auditors to view the tracing results of the financial document "finance_report_alpha.dat".

[0098] By introducing a multi-path probability fusion model, the problem of insufficient confidence in single-path tracing under complex network associations is solved. Bitmap verification is used to quickly complete the lineage view, providing a comprehensive confidence assessment with statistical basis, which significantly enhances the objectivity and reliability of the tracing results.

[0099] This method addresses the technical bottlenecks of accurately reconstructing data flow from massive heterogeneous logs in large-scale distributed environments and the low efficiency of retrieval of network lineage relationships by integrating unstructured text mining, probabilistic graphical modeling, and hierarchical hybrid indexing techniques. It achieves logical regularization of fragmented logs using tree-like clustering and multi-dimensional session aggregation, and constructs a weighted lineage graph with quantified semantics by combining conditional probability and time decay factors. Through a collaborative indexing mechanism of retrieval trunk tree encoding and associated jump bitmaps, it transforms complex graph path depth search into low-overhead numerical range determination and bitwise operations. While ensuring the integrity of the tracing path, it achieves second-level retrieval response and scientific evaluation of comprehensive confidence, significantly improving the accuracy, retrieval efficiency, and robustness of data lifecycle tracing.

[0100] Example 2:

[0101] This embodiment provides a data tracing method based on big data mining. It aims to address the technical challenges of accurately reconstructing data flow paths, poor retrieval performance, and a lack of quantitative confidence assessment of lineage relationships in ultra-large-scale distributed environments through in-depth mining and topological indexing of multi-source heterogeneous logs. This embodiment is applied to a large enterprise private cloud storage platform carrying petabytes of business data. The system generates unstructured operation log streams in real time, including object storage access logs, database audit logs, and file system kernel call logs. This scenario requires tracing the lineage path of a virtual file (named "finance_report_alpha.dat") throughout its entire lifecycle, from original voucher collection, intermediate operator processing, cross-database transfer, to the final report generation. This method constructs a probabilistic skeleton tree index structure to ensure second-level full-path tracing and quantitative confidence assessment of tracing conclusions even with logs of tens of billions in size.

[0102] A tree-based text clustering algorithm is used to parse the collected unstructured operation logs. This tree-based text clustering algorithm is a heuristic algorithm that performs layer-by-layer matching and similarity classification of log token sequences by constructing a depth-limited parse tree. The original log (e.g., “User_Admin_01 uploaded finance_report_alpha.dat from 10.0.0.5”) is segmented into token sequences according to delimiters and input into the parse tree. If the similarity between the input sequence and an existing path in the tree is greater than a preset evolution threshold (set to 0.85 in this embodiment, a value derived from convergence distribution tests of historical log patterns), it is assigned to that class and the pattern is updated; otherwise, a new branch is created. This process extracts fixed string sequences from the log text as event templates, and maps dynamic parts that change with operation (such as filenames and IP addresses) to parameter fields.

[0103] Construct standardized, structured event logs. Generate a description object containing a fixed pattern for each parsed log entry. The structured event log includes a timestamp field, an operation subject field, an operation action field, and a data object path field. The timestamp field stores an integer value accurate to microseconds; the operation subject field records the entity identity performing the operation (e.g., financial approval process P_102); the operation action field maps raw verbs to preset atomic predicates (e.g., READ, WRITE, DELETE); and the data object path field stores a globally unique identifier for the operated object (e.g., a file URI).

[0104] Structured event records are aggregated within a preset time window using access subject identifiers to generate a heterogeneous operation session set as a logical storage unit. The preset time window is a sliding time range used to define the logical correlation of operations; in this embodiment, it is set to 300 seconds. This value is based on empirical values ​​derived from statistics of the average time consumption of a typical single business flow (from read to write) within the cloud platform. The access subject identifier is a quintuple consisting of host address, user identifier, process identifier, thread identifier, and session start time. Scanning the event record stream, records with completely identical quintuple characteristics and time intervals within 300 seconds are pushed into the same ordered logical container in chronological order of timestamps, forming a heterogeneous operation session set. This restores discrete log entries to atomic operation flows reflecting the behavior of specific subjects.

[0105] A frequency-pattern growing tree data structure is used to store and scan atomic operation groups within a heterogeneous operation session set. This frequency-pattern growing tree data structure is a prefix tree index that compresses and stores frequent itemsets in a tree-like topology through two scans of the dataset. Each heterogeneous operation session set is treated as a transaction, with its atomic operations treated as items. When constructing the frequency-pattern growing tree, an item header table is maintained, arranged in descending order of the frequency of atomic operations across all sessions. Each node in the tree stores an operation identifier and its cumulative count value under the current path branch, and the co-occurrence order of operations is represented by the connection relationship between parent and child nodes.

[0106] The frequent sub-paths that satisfy the minimum support threshold are traversed on the frequency pattern growth tree. Paths that conform to the topology from reading a data object to writing a data object are selected and defined as read-write access path patterns. The minimum support threshold refers to a numerical proportion used to filter random noise and retain statistically significant paths; in this embodiment, it is set to 0.02 (i.e., the pattern must appear in more than 2% of sessions). A bottom-up recursive mining strategy is used to extract all frequent item sequences that satisfy this threshold from the frequency pattern growth tree. Through topological logic filtering, only path segments with the starting operation being "read," the ending operation being "write," and intermediate nodes conforming to data processing logic are retained and defined as read-write access path patterns. The read-write access path pattern consists of the access operation sequence from the input data object to the output data object within the heterogeneous operation session set, clarifying the causal chain of data flow.

[0107] A correlation strength assessment model based on conditional probability statistics is constructed. This model is a mathematical model that quantifies the derivation probability between data objects based on Bayesian principles. The frequency pattern growth tree is traversed to extract unique identifiers of all data objects involved in read / write operations (e.g., finance_report_alpha.dat), and a full object index table is established. The conditional probability calculation logic based on support counting is defined: the sum of the global counts of the input data object nodes in the frequency pattern growth tree is used as the denominator; the sum of the co-occurrence counts of the input node and its descendant output data object nodes along the same path is used as the numerator, thereby establishing a quantitative correlation assessment criterion.

[0108] The association strength assessment model extracts the global occurrence frequency of input data objects and the joint frequency of co-occurrence of input and output data objects from the frequency pattern growing tree. The ratio of the joint frequency to the global occurrence frequency is calculated as the association strength value. The global frequency (N1) of the input object and the joint frequency (N2) of both are obtained from the full object index table. N2 / N1 is then calculated to obtain the scalar association strength value (ranging from 0 to 1). This value reflects the statistical contribution of the existence of the input object to the generation of the output object.

[0109] The time difference between read and write operations is calculated, and a time decay value is generated based on a time decay factor. The difference between the read and write times in the read / write access path pattern is extracted. The time decay factor is a decay function following an exponential distribution. In this embodiment, the time decay value is generated by calculating the power of the negative λ of the natural constant e (where λ is preset to 0.005) and the time difference. This factor is set to simulate the characteristic that the causal correlation between two operations gradually weakens over time.

[0110] The product of the association strength value and a first preset adjustment coefficient is calculated as the first component, and the product of the time decay value and a second preset adjustment coefficient is calculated as the second component. The first and second components are then added together to obtain the dependency weight. The first preset adjustment coefficient is set to 0.7, and the second preset adjustment coefficient is set to 0.3. This ratio is the optimal parameter obtained by grid search based on the historical source tracing accuracy validation set, aiming to ensure that statistical probability dominates the weighting.

[0111] Perform a graph instantiation operation to generate a lineage dependency graph. The lineage dependency graph is a weighted graph with data objects as vertices and flow relationships as directed edges. The unique data object paths in the read / write access path pattern are mapped to graph nodes. Directed edges are established between each pair of input and output nodes with flow relationships, and the calculated dependency weights are written into the attribute fields of these edges, thus forming a weighted directed graph with probabilistic attributes.

[0112] A depth-first search-based strongly connected component identification algorithm is used to process the lineage dependency graph, performing loop elimination and generating a directed acyclic topology. The algorithm maintains a node traversal stack, recording the visit order identifier and minimum backtracking identifier of each node during traversal. During the depth-first search, when the visit order identifier of the current node equals the minimum backtracking identifier, a root of a strongly connected component is found. All nodes above this root node are popped from the stack and identified as cyclic dependent subgraphs.

[0113] Each identified cyclic dependency subgraph is collapsed into a single node, while preserving the external connections between nodes. The folding operation logically merges all nodes within the cyclic dependency into a single supernode, redirecting all edges pointing to or leading out of the subgraph to this supernode. This process eliminates all closed loops in the graph, ensuring the unidirectionality of the tracing path and the convergence of the retrieval process, generating a directed acyclic topology.

[0114] All edges in the directed acyclic topology are sorted in descending order of their dependency weights, and a retrieval trunk tree structure is constructed. A greedy strategy is employed, prioritizing the addition of edges with the largest dependency weights to the tree index structure until all reachable nodes in the topology are covered. If adding an edge would create a cycle in the tree structure or violate the in-degree constraint, it is marked as a cross-branch association edge. The final result is a retrieval trunk tree structure composed of edges with maximal weights.

[0115] A depth-first traversal is performed on the retrieval trunk tree structure, and a global sequence counter is maintained to generate node index encoding intervals. The global sequence counter is initialized to 1. When the traversal flow first enters a node, the current value of the counter is written to the start position of the interval of that node and the counter is incremented; when all subtrees of that node have been traversed and backtracking is performed, the current value of the counter is written to the end position of the interval of that node and incremented again. The generated node index encoding interval consists of a start position and an end position. If the interval [start1, end1] of the first node contains the interval [start2, end2] of the second node (i.e., start1 <= start2 and end1 >= end2), then the first node is determined to be the ancestor node of the second node, realizing a lineage determination with constant complexity.

[0116] Directed edges that exist in the directed acyclic topology but are not included in the retrieval backbone tree structure are identified and defined as cross-branch association edges. A bitmap index table based on bucket compression technology is constructed as the association jump bitmap index. The bucket compression technology refers to dividing the node ID space into two layers of storage: high 16 bits and low 16 bits. The association jump bitmap index adopts a hierarchical storage structure: the high-order part of the source node ID of the cross-branch association edge is used as the index key value, and the low-order part is stored in the corresponding bitmap container. The bit offset of the bit set (set to binary "1") in the bitmap container corresponds to the identifier of the target node that the edge can jump to.

[0117] An index encapsulation operation is performed, combining the node index encoding range with the associated jump bitmap index to generate a probabilistic skeleton tree index structure. This probabilistic skeleton tree index structure refers to a comprehensive index object that integrates tree-like range determination logic with non-tree-like bitmap jump logic. In physical storage, this structure manifests as each node's metadata containing both range codes and index pointers to bitmap containers, providing a complete logical skeleton for full lineage tracing.

[0118] Upon receiving a data tracing retrieval request for "finance_report_alpha.dat", the system uses the node index encoding interval in the probabilistic skeleton tree index structure to locate the basic retrieval path on the main tree structure. It extracts the interval encodings of the source and target data nodes and determines their positions on the main tree using numerical inclusion logic. If an inclusion relationship is satisfied, it directly extracts the connection sequence along the main tree as the basic retrieval path.

[0119] If the interval inclusion logic is not satisfied, the associated jump bitmap index is called to search for the existence of a cross-branch associated edge connecting the subtree where the source data node is located and the subtree where the target data node is located. By performing a bitwise AND operation on the bitmap, the jump point that crosses the tree branch is quickly located and concatenated with the main tree path to generate a basic search path that covers non-tree edges.

[0120] When the dependency weight of the basic retrieval path does not meet the preset threshold, the associated jump bitmap index is invoked to perform multi-path retrieval verification, outputting the source path and overall confidence score. The product of the weights of all connecting edges on the basic retrieval path is calculated as the basic probability value. If this value is lower than the preset threshold (e.g., 0.75), the bitmap index is immediately used to search for other parallel jump paths.

[0121] Perform probability verification calculations. Calculate the first complement of the base probability value, as well as the complement of the weights of all retrieved redirection paths. Multiply these complements together and calculate the complement of the product as the overall confidence score (calculated as: 1 - the probability that all paths are invalid). Finally, return the set of paths whose overall confidence score meets the output threshold (e.g., 0.6) and their confidence scalars to the retrieval interface to complete the source tracing.

[0122] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of protection defined in the claims.

Claims

1. A data tracing method based on big data mining, characterized in that, include: Collect unstructured operation logs, use log parsing templates to parse the unstructured operation logs into structured event records, and associate the structured event records into a heterogeneous operation session set; Access pattern association analysis is performed on a heterogeneous set of operation sessions to extract read and write access path patterns; the dependency weights in the read and write access path patterns are determined by combining the association strength evaluation model and the time decay factor, and a lineage dependency graph is constructed based on the dependency weights. Perform loop elimination on the bloodline dependency graph to generate a directed acyclic topology; The steps for constructing the retrieval trunk tree structure based on priority sorting rules include: processing the lineage dependency graph using a depth-first search-based strongly connected component identification algorithm, which maintains a node traversal stack and records the access order identifier and minimum backtracking identifier of each graph node during traversal; when the access order identifier of the current node is equal to the minimum backtracking identifier, all nodes above the current node in the node traversal stack are popped and identified as cyclic dependency subgraphs; each identified cyclic dependency subgraph is folded into a single node, while retaining the external connections between nodes, generating a directed acyclic topology; all connecting edges in the directed acyclic topology are sorted in descending order according to the numerical value of the dependency weight; the connecting edge with the largest dependency weight is preferentially selected and added to the tree index structure until all reachable nodes in the directed acyclic topology are covered, forming the retrieval trunk tree structure; the retrieval trunk tree structure is traversed to generate node indices. The steps for generating a probabilistic skeleton tree index structure include: identifying directed edges in the directed acyclic topology that are not included in the retrieval backbone tree structure and defining them as cross-branch related edges; constructing a bitmap index table based on bucket compression technology as the related jump bitmap index; the related jump bitmap index adopts a hierarchical storage structure combining high-bit key-value pairs and low-bit containers, wherein the high-bit part of the source node identifier of the cross-branch related edge is used as the index key value, and the low-bit part of the source node identifier is used as the offset to map to the corresponding bitmap container; wherein the position index corresponding to the set bit in the bitmap container is used to map the target node identifier that can be jumped to from the source node; and performing an index encapsulation operation to combine the node index encoding range with the related jump bitmap index to generate a probabilistic skeleton tree index structure. Receive data tracing retrieval requests, use the node index encoding range in the probabilistic skeleton tree index structure to locate the basic retrieval path on the trunk tree structure; when the dependency weight of the basic retrieval path does not meet the preset threshold, call the associated jump bitmap index to perform multi-path retrieval verification, and output the tracing path and comprehensive confidence score.

2. The data tracing method based on big data mining according to claim 1, characterized in that, The steps of parsing unstructured operation logs into structured event records using log parsing templates and associating these structured event records with a heterogeneous operation session set include: parsing the unstructured operation logs using a tree-based text clustering algorithm, extracting fixed patterns from the log text as event templates, and mapping variable parts of the log text to parameter fields; constructing standardized structured event records, which include timestamp fields, operation subject fields, operation action fields, and data object path fields; and aggregating the structured event records within a preset time window using access subject identifiers to generate a heterogeneous operation session set as a logical storage unit, wherein the access subject identifiers include host address, user identifier, process identifier, thread identifier, and session start time, and the heterogeneous operation session set is formed by aggregating structured event records belonging to the same execution subject.

3. The data tracing method based on big data mining according to claim 1, characterized in that, The steps of performing access pattern association analysis on the heterogeneous operation session set and extracting read / write access path patterns include: storing and scanning atomic operation groups within the heterogeneous operation session set using a frequency pattern growing tree data structure; traversing frequent sub-paths that satisfy the minimum support threshold on the frequency pattern growing tree, filtering out paths that conform to the topology of reading data objects to writing data objects, and defining them as read / write access path patterns; the read / write access path pattern consists of the sequence of access operations from the input data object to the output data object within the heterogeneous operation session set.

4. The data tracing method based on big data mining according to claim 3, characterized in that, The steps of determining the dependency weights in the read / write access path pattern by combining the association strength evaluation model and the time decay factor, and constructing a lineage dependency graph based on the dependency weights, include: constructing an association strength evaluation model based on conditional probability statistics. The construction process of the association strength evaluation model includes: traversing the frequency pattern growth tree to establish a full object index table, and defining a conditional probability calculation logic based on support counting. The conditional probability calculation logic specifies that the global count value of the input data object node in the frequency pattern growth tree is used as the denominator, and the path co-occurrence count value of the input data object node and its descendant output data object node is used as the numerator; calculating the association strength value using the association strength evaluation model, and performing a weighted summation based on the time decay value generated based on the operation time difference to determine the dependency weights; performing a graph instantiation operation to map the data object path to a graph node, establishing directed connection edges storing the dependency weights between the graph nodes, and generating a lineage dependency graph.

5. The data tracing method based on big data mining according to claim 1, characterized in that, The steps for generating node index encoding intervals by traversing and retrieving the main tree structure include: performing a depth-first traversal operation on the main tree structure and maintaining a global sequence counter; when a node in the tree structure is visited, writing the current value of the global sequence counter to the start position of the interval of the node and incrementing the global sequence counter; when all child nodes of the node have been visited and backtracking has been performed, writing the current value of the global sequence counter to the end position of the interval of the node and incrementing the global sequence counter; the generated node index encoding interval consists of the start position and the end position of the interval. If the node index encoding interval of the first node contains the node index encoding interval of the second node, then the first node is determined to be the ancestor node of the second node in the main tree structure.

6. The data tracing method based on big data mining according to claim 1, characterized in that, The steps of receiving a data tracing retrieval request and locating the basic retrieval path on the main tree structure using the node index coding interval in the probabilistic skeleton tree index structure include: extracting the first node index coding interval of the source data node identifier and the second node index coding interval of the target data node identifier; if it is determined that the starting position value of the first node index coding interval is less than or equal to the starting position value of the second node index coding interval and the ending position value is greater than or equal to the ending position value of the second node index coding interval, then a tree-like inclusion relationship is confirmed, and the path connecting the two node identifiers on the main tree structure is taken as the basic retrieval path; otherwise, the associated jump bitmap index is called to search for whether there is a cross-branch association edge connecting the subtree where the source data node identifier and the target data node identifier are located, and the path generated based on the cross-branch association edge is taken as the basic retrieval path.

7. The data tracing method based on big data mining according to claim 1, characterized in that, The process of calling the associated jump bitmap index to perform multi-path retrieval verification and outputting the source path and comprehensive confidence score when the dependency weight of the basic retrieval path does not meet the preset threshold includes: calculating the product of the dependency weights of all connecting edges on the basic retrieval path as the basic probability value; triggering a query operation for the associated jump bitmap index when the basic probability value is lower than the preset threshold; searching in the bitmap container for cross-branch association edges connecting the subtree where the source data node identifier is located and the subtree where the target data node identifier is located through bit operations; if cross-branch association edges exist, performing probability verification calculation, which includes calculating the first complement value of the basic probability value and the second complement value of the jump path weight through the cross-branch association edge, calculating the product of the first complement value and the second complement value, and calculating the complement value of the product as the comprehensive confidence score; the step of outputting the source path and comprehensive confidence score includes returning the set of paths whose comprehensive confidence score meets the output conditions to the retrieval interface.

Images