AI data warehouse full-link consanguinity tracking method and AI data warehouse full-link consanguinity tracking device

Abstract

The invention discloses an AI data warehouse full-link blood relationship tracking method and device, and relates to the technical field of data processing. The method comprises the following steps: obtaining structured log data with business terms and entity tags; establishing a knowledge graph according to the structured log data and the business document; generating blood relationship metadata according to the knowledge graph and the data job execution log; constructing a blood relationship cognition map according to the blood relationship metadata, the data operation log and the business document; generating a governance strategy vector according to the blood relationship cognitive map, the system real-time index and the service SLO, and sending the governance strategy vector to an execution engine, so that the execution engine carries out processing according to the governance strategy vector; and obtaining a governance strategy execution result, user feedback information of the governance strategy execution result and a system log generated in a governance process. According to the method, complex semantics and causal relationships behind data operation can be captured.

Description

Technical Field

[0001] This application relates to the field of data processing technologies, and in particular, to an AI data warehouse full-link lineage tracking method and an AI data warehouse full-link lineage tracking device. Background Art

[0002] In the fields of data governance and intelligent decision-making, full-link lineage tracking is a core technology to ensure the reliability of data assets and optimize the system operation efficiency. Most of the existing lineage graphs are static topological structures, which are difficult to cope with the dynamic evolution of data operations. When the business logic is adjusted (such as adding new data sources) or the system architecture is changed (such as upgrading the computing engine), the existing methods require manual intervention to reconstruct the lineage relationship, resulting in the governance strategy lagging behind the actual needs. Summary of the Invention

[0003] The purpose of the present invention is to provide an AI data warehouse full-link lineage tracking method to at least solve one of the above technical problems.

[0004] In one aspect of the present invention, there is provided an AI data warehouse full-link lineage tracking method, and the AI data warehouse full-link lineage tracking method includes: Obtain data operation logs and business documents; Obtain structured log data with business terms and entity tags; Establish a knowledge graph including the mapping relationship between business concepts and data entities according to the structured log data and business documents; Generate lineage metadata including business semantics according to the knowledge graph and data job execution logs; Construct a lineage cognitive graph according to the lineage metadata, data operation logs and business documents; Generate a governance strategy vector according to the lineage cognitive graph, system real-time metrics and business SLO, and send it to the execution engine, so that the execution engine processes according to the governance strategy vector; Obtain the governance strategy execution result, user feedback information of the governance strategy execution result, and system logs generated during the governance process.

[0005] Optionally, the obtaining of the structured log data with business terms and entity tags includes: Generate a structured log token stream according to the data operation logs; Generate a multi-modal fusion semantic vector according to the structured log token stream; Generate a spatio-temporal graph structure according to the multi-modal fusion semantic vector; Generate an implicit relationship embedding vector according to the spatio-temporal graph structure; Generate a spatio-temporal enhanced node representation through the implicit relationship embedding vector; Strengthen the node representation according to the spatio-temporal enhancement node representation generation rule; Generate a weighted rescaled semantic vector according to the rule-strengthened node representation; Generate a robustness-enhanced semantic vector according to the weighted rescaled semantic vector; Generate a probabilistic entity annotation sequence according to the robustness-enhanced semantic vector; Generate an optimized entity annotation sequence according to the probabilistic entity annotation sequence as structured log data with business terms and entity labels.

[0006] Optionally, the building of a knowledge graph including the mapping relationship between business concepts and data entities according to the structured log data and business documents includes: Extract data from the structured log data and business documents to obtain a standardized metadata set; Perform dynamic semantic injection on the standardized metadata set to obtain a semantic-enhanced metadata vector; Construct an initial cognitive graph according to the semantic-enhanced metadata vector.

[0007] Optionally, the building of a blood relationship cognitive graph according to the blood relationship metadata, data operation logs, and business documents includes: According to the original blood relationship set in the blood relationship metadata, data operation logs, and business documents, the original blood relationship set includes operation type, timestamp, and confidence score; Generate an enhanced blood relationship set according to the original blood relationship set, and the enhanced blood relationship set includes business semantic labels, influence scope, and causal strength; Generate a blood relationship cognitive graph according to the enhanced blood relationship set.

[0008] Optionally, the generating of a governance policy vector according to the blood relationship cognitive graph, system real-time metrics, and business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector includes: Use a graph attention network to encode the blood relationship cognitive graph to obtain a graph neural network encoded vector; Perform dynamic weight adjustment according to the graph neural network encoded vector to obtain a weight-adjusted graph; Obtain a governance policy action set according to the weight-adjusted graph; Generate an executable governance policy package according to the governance policy action set, and the executable governance policy package includes action type, target, priority, and specific operations; Perform conflict resolution processing according to the executable governance policy package to obtain a conflict-resolution policy package and send it to the execution engine.

[0009] Optionally, the obtaining of the governance policy action set according to the weight-adjusted graph includes: According to the weight-adjusted graph, use breadth-first search to enumerate all causal paths with a length ≤ 3, so as to obtain each path with intensity to be calculated; Perform intensity calculation on each path with intensity to be calculated, so as to obtain the causal intensity of each path with intensity to be calculated; Input the graph into a graph convolutional network, aggregate neighbor node information through multi-layer convolutional operations, and generate a latent path representation; Based on the latent path representation, calculate the indirect influence of nodes through a message passing mechanism. Each node transmits its own influence to its neighbors, so as to obtain an indirect causal reasoning result; Perform counterfactual intervention on key nodes according to the weight-adjusted graph, simulate the change amount of the target node after intervention, and use the SHAP value to calculate the marginal contribution of each feature, so as to obtain a multi-level causal reasoning result; Perform dynamic weight adjustment and fusion according to the multi-level causal reasoning result, so as to obtain a hierarchical weight allocation result; Generate a fused causal intensity graph according to the multi-level causal reasoning result and the hierarchical weight allocation result; Generate a global state representation according to the fused causal intensity graph; Input the global state representation into a deep Q-network, map the state to action values through a multi-layer network, and use the ε-greedy strategy for action exploration, so as to obtain action value evaluation result information; Select the optimal action according to the Q function value according to the action value evaluation result information, and each of the optimal actions constitutes the governance policy action set.

[0010] Optionally, the conflict resolution process according to the executable governance policy package to obtain a conflict-resolution policy package and send it to the execution engine includes: Identify the affected stakeholders according to the executable governance policy package, so as to obtain game theory model parameters, where the game theory model parameters include game players, strategy sets, utility functions, and cost functions; Iteratively update the strategy combination of the strategy set through the replicator dynamics equation, and find the equilibrium solution, so as to obtain the Nash equilibrium strategy combination and non-equilibrium strategy pairs; Generate a compensation plan according to the non-equilibrium strategy pairs and calculate the compensation cost, where the Nash equilibrium strategy combination and the compensation plan and compensation cost corresponding to each non-equilibrium strategy constitute the conflict-resolution policy package.

[0011] Optionally, after obtaining the governance policy action set adjusted according to weights, the step of inputting the lineage awareness graph, system real-time metrics, and business SLO into the governance model to generate a governance policy vector and sending it to the execution engine, so that the execution engine processes according to the governance policy vector further includes: Obtain a preset policy template database, where the preset policy template database includes at least one predefined policy template and the action type corresponding to each predefined policy template; Obtain the corresponding predefined policy template from the preset policy template database according to the action type in the obtained governance policy action set; Obtain system real-time metrics and historical execution logs; Generate a parameterized policy template according to the predefined policy template, system real-time metrics, and historical execution logs; Dynamically adjust the policy priority using the context-aware priority dynamic adjustment formula to obtain the adjusted governance policy action set; Evaluate the adjusted governance policy action set using a multi-modal policy generation quality assessment model to obtain a score value; Judge whether the score value exceeds the scoring threshold. If so, generate an executable governance policy package according to the adjusted governance policy action set.

[0012] Optionally, in the process of iteratively updating the policy combination through the replicator dynamics equation for the policy set, finding the equilibrium solution, and obtaining the Nash equilibrium policy combination and non-equilibrium policy pairs, the AI data warehouse full-link lineage tracing method further includes: In the process of iteratively updating the policy combination through the replicator dynamics equation each time, evaluate the current policy combination using the stakeholder influence weighted satisfaction formula; When the iteration change rate of the weighted satisfaction score is <1% for 3 consecutive iterations, terminate the iteration in advance to obtain the Nash equilibrium policy combination and the weighted satisfaction score for the current iteration of termination.

[0013] This application also provides an AI data warehouse full-link lineage tracing device, where the AI data warehouse full-link lineage tracing device includes: A data acquisition module, which is used to acquire data operation logs and business documents; A structured log data acquisition module, which is used to acquire structured log data with business terms and entity tags; A knowledge graph acquisition module, which is used to establish a knowledge graph containing the mapping relationship between business concepts and data entities according to the structured log data and business documents; A blood relationship metadata acquisition module, which is used to generate blood relationship metadata including business semantics according to the knowledge graph and data operation execution logs; A blood relationship cognitive graph acquisition module, which is used to construct a blood relationship cognitive graph according to the blood relationship metadata, data operation logs, and business documents; A governance strategy vector generation module, which is used to generate a governance strategy vector according to the blood relationship cognitive graph, system real-time metrics, and business SLO and send it to the execution engine, so that the execution engine processes according to the governance strategy vector; A governance data acquisition module, which is used to obtain governance strategy execution results, user feedback information on governance strategy execution results, and system logs generated during the governance process.

[0014] Beneficial effects The AI data warehouse full-link blood relationship tracking method of this application breaks the barrier between technology and business in traditional data governance through full-link data fusion, and realizes the deep association between business terms and data entities. On this basis, through semantic-enhanced metadata construction and dynamic blood relationship cognitive graph generation technology, this application can capture the complex semantics and causal relationships behind data operations. Description of the drawings

[0015] Figure 1 is a schematic flowchart of the AI data warehouse full-link blood relationship tracking method according to an embodiment of this application. Detailed implementation manners

[0016] To make the purpose, technical solutions, and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the drawings in the embodiments of this application. In the drawings, the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The described embodiments are some, but not all, of the embodiments of this application. The embodiments described below by referring to the drawings are exemplary and are intended to explain this application and should not be construed as limiting this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application. The embodiments of this application will be described in detail below with reference to the drawings.

[0017] As Figure 1 shown, the AI data warehouse full-link blood relationship tracking method includes: Step 1: Obtain data operation logs and business documents; Step 2: Obtain structured log data with business terms and entity tags; Step 3: Establish a knowledge graph containing the mapping relationship between business concepts and data entities based on the structured log data and business documents; Step 4: Generate lineage metadata including business semantics based on the knowledge graph and data job execution logs; Step 5: Construct a lineage awareness graph based on the lineage metadata, data operation logs, and business documents; Step 6: Generate a governance policy vector based on the lineage awareness graph, system real-time metrics, and business SLO, and send it to the execution engine, so that the execution engine processes according to the governance policy vector; Step 7: Obtain the governance policy execution result, user feedback information on the governance policy execution result, and system logs generated during the governance process.

[0018] In this embodiment, the data operation logs may include the following contents: structured operation records such as SQL queries, ETL job logs, model training logs, etc.; In this embodiment, the business documents may include unstructured texts or semi-structured data such as data dictionaries, requirement specifications, system logs (operating system events, network transmission records), etc.; In this embodiment, the obtaining of the structured log data with business terms and entity tags includes: Generating a structured log token stream according to the data operation logs; specifically, generating a structured log token stream according to the data operation logs includes: performing regular expression matching on the original log entries to extract the timestamp, operation type, and involved entities (such as users, files, IPs, etc.) therein. Mapping the extracted entities to predefined entity types, such as mapping user A to user ID, mapping file X to file path, and serializing the parsed timestamp, operation type, and entity and other information into a structured log token stream.

[0019] Generating a multi-modal fusion semantic vector according to the structured log token stream; specifically, using natural language processing techniques (such as word embedding) to extract text features from the log token stream and convert the text information into a vector representation. Encoding the timestamp in the log token stream to capture the time series pattern and convert the time information into a vector representation. Constructing spatial features according to the entity type and the relationship between entities and converting the spatial information into a vector representation. Fusing the text features, time features, and spatial features to generate a multi-modal fusion semantic vector, which synthesizes various information of the log entries; Generate a spatio-temporal graph structure based on the multi-modal fusion semantic vectors; construct a dynamic spatio-temporal graph based on the multi-modal fusion semantic vectors. Nodes in the graph represent entities, edges represent the relationships between entities, and the weights of the edges reflect the strength of the relationships. The spatio-temporal graph can capture the dynamic changes of entities in time and space. In this embodiment, the node attributes include the entity type (such as user, file) and the occurrence frequency of the entity in the log. Define the non-zero elements in the spatial adjacency matrix as the edges in the graph, and the weight of the edge is the co-occurrence frequency between entities. Concatenate the graphs corresponding to each time window in chronological order to form a dynamic spatio-temporal graph structure. Each spatio-temporal graph represents the entities and their relationships within a specific time window; Generate an implicit relationship embedding vector according to the spatio-temporal graph structure; In this embodiment, generating an implicit relationship embedding vector according to the spatio-temporal graph structure includes: A predefined relationship pattern library, such as user-file access pattern, IP-user login pattern, etc.

[0020] Mine typical relationship patterns from historical log data and extract their features. For example, for the user-file access pattern, features such as the time distribution and access frequency of users accessing files can be extracted.

[0021] Represent the extracted relationship patterns in the form of feature vectors. For example, represent the features of the user-file access pattern as a vector, including dimensions such as the average access interval and the maximum access frequency. Each vector forms an implicit relationship embedding vector, and each element contains the pattern ID and the corresponding feature vector; Perform edge relationship matching and initial weight assignment. Specifically, it includes the following steps: Feature similarity calculation: For each edge in the spatio-temporal graph structure, calculate its feature similarity with each implicit relationship pattern.

[0022] Pattern matching: Select the implicit relationship pattern with the highest feature similarity for each edge.

[0023] Initial weight assignment: Map the similarity between the edge and the matching pattern to the initial weight of the edge.

[0024] Output: The initialized spatio-temporal graph edge weights, including the edge ID and the initial weight.

[0025] The generation of the implicit relationship embedding vector specifically includes the following steps: Embedding vector learning: Based on the spatio-temporal graph structure and the initial weights of the edges, use graph embedding techniques (such as Node2Vec, GraphSAGE, etc.) to learn the implicit relationship embedding vectors. These vectors can capture the complex implicit relationships between entities, and thus output the implicit relationship embedding vectors. Each vector corresponds to an edge in the spatio-temporal graph and represents the implicit relationship represented by that edge; Generate spatio-temporal enhanced node representations through implicit relationship embedding vectors; specifically, use a graph attention network (GAT) to perform weighted aggregation on neighbor nodes in the spatio-temporal graph according to the weights of the implicit relationship embedding vectors. For example, the representation of node A aggregates the representations of its neighbor nodes B and C, and the aggregation weights are determined by the strength of the implicit relationships between node A and B, C. In the graph attention mechanism, both the time decay factor and the spatial adjacency relationship are considered to ensure that node representations can capture spatio-temporal dynamics. For example, recent log events have a greater impact on node representations.

[0026] In this embodiment, in the spatio-temporal enhanced node representation, each node corresponds to a vector, which synthesizes its own attributes, neighbor information, and spatio-temporal dynamics.

[0027] Strengthen node representations according to the spatio-temporal enhanced node representation generation rules; specifically, match the rules in the business rule library (such as "the user ID must be a combination of numbers and letters") with the node representations to identify the nodes that conform to the rules. For the nodes that match the rules, inject their rule compliance information (such as whether it conforms to the user ID format) into the node representations to enhance the node's ability to express business rules, thereby obtaining rule-strengthened node representations; Generate weight-rescaled semantic vectors according to the rule-strengthened node representations; specifically, construct a meta-learner, adopt a reinforcement learning strategy (PPO algorithm) to learn the feature weights in different business scenarios, design a weight generation network, output a weight vector with a dimension of 256, and perform a Hadamard product with the input vector to achieve feature rescaling, thereby obtaining weight-rescaled semantic vectors; Generate robustness-enhanced semantic vectors according to the weight-rescaled semantic vectors; obtain an adversarial sample generator, construct a semantics-preserving adversarial perturbation in the input space based on the FGSM algorithm, and adopt an alternating training strategy to first update the discriminator (to distinguish real / adversarial samples), and then jointly optimize the generator and the main network to improve the robustness of the model; Generate a probabilistic entity annotation sequence according to the robustness-enhanced semantic vectors; specifically, construct a teacher-student architecture, where the teacher network uses BERT-large to generate soft labels, and the student network uses a lightweight Transformer for distillation learning. Design a temperature-scaled cross-entropy loss function to balance hard label supervision and soft label guidance, and generate a probabilistic entity annotation sequence; Generate an optimized entity annotation sequence from the probabilistic entity annotation sequence as structured log data with business terms and entity labels. Transform entity recognition into a sequence decision problem, defining the state space (current annotation position), action space (label selection), and reward function (F1 value increment). Deploy the DQN algorithm for online decision optimization, using the experience replay mechanism to break data correlations and optimize the entity annotation sequence.

[0028] This application extracts business terms and entity labels through NLP technology, constructs a knowledge graph using graph database technology, realizes the association and mapping between business and technology, obtains data operation logs and business documents, extracts business terms and entity labels from them using natural language processing (NLP) and machine learning technologies, and associates them with structured log data. Based on these associated data, the present invention constructs a knowledge graph containing the mapping relationship between business concepts and data entities, realizing the deep integration of technology and business.

[0029] In this embodiment, establishing a knowledge graph containing the mapping relationship between business concepts and data entities based on the structured log data and business documents includes: Extract data from the structured log data and business documents to obtain a standardized metadata set; Perform dynamic semantic injection on the standardized metadata set to obtain a semantic-enhanced metadata vector; Construct an initial cognitive graph according to the semantic-enhanced metadata vector.

[0030] In this embodiment, the following processing is performed on the structured log data, business documents and other data in the above text: File format recognition: Automatically recognize formats such as CSV, JSON, XML, TXT, etc. by analyzing the magic number of the file and using file extension mapping. For example: For a file starting with 0x89PNG, recognize it as a PNG format.

[0031] Database schema parsing: Use the ANTLR library to parse SQL DDL statements, construct an abstract syntax tree (AST), and extract information such as table names, field names, and data types from it. For example: For the SQL statement CREATE TABLE customers (id INT, name VARCHAR(100)), extract the table name customers, field names id and name, and the corresponding data types.

[0032] ETL Job Configuration Analysis: Parse the DAG definition file (in JSON format) of Apache Airflow to extract task dependencies and execution order. For example: Parse from JSON that task A is executed after task B and establish the task dependency relationship.

[0033] Define a standardized representation that includes multiple core attributes. For example, in this embodiment, it includes 12 core attributes as follows: Entity ID: A unique identifier, such as "TBL_CUSTOMER_001"; Entity Type: Table, field, job, rule, etc.; Operation Timestamp: In ISO 8601 format, such as "2025-05-26T10:00:00Z"; Business Domain: Finance, risk, marketing, etc.; Data Source: Database name, file path, etc.; Data Quality Score: A floating point number between 0 and 1, such as 0.95; Semantic Tags: Customer, transaction, etc., separated by commas; Parent Entities: A list of superior entity IDs, such as ["DB_SALES", "SCH_CUSTOMER"]; Child Entities: A list of inferior entity IDs, such as ["COL_CUSTOMER_ID"]; Data Lineage: A path description from source to target, such as "From source system A through ETL job B to target table C"; Business Rules: A list of associated business rule IDs, such as ["RULE_RISK_SCORE"]; Metadata Version: A semantic version number, such as "1.0.0".

[0034] It can be understood that this application can also include a conflict detection method as follows: Use a pre-trained BERT model (such as bert-base-uncased) to calculate the semantic similarity of field descriptions. For example: For field descriptions "Customer Risk Score" and "Customer Credit Rating", calculate their cosine similarity. When the similarity is lower than a threshold (such as 0.8), trigger an artificial arbitration interface; this interface provides a web interface to display the conflicting fields and their context, such as the table name they belong to, field type, sample data, etc., for domain experts to make a decision.

[0035] This application uses multi-modal fusion technology. Multi-modal fusion can comprehensively utilize various features of log data, and the spatio-temporal graph structure can capture the temporal and spatial relationships in log data, thereby more comprehensively understanding the meaning of log data. The structured log token stream is fused with the semantic information in business documents to generate a multi-modal fusion semantic vector. Further, through spatio-temporal graph structure modeling, the temporal and spatial relationships in log data are captured to generate an implicit relationship embedding vector. Finally, through a series of complex vector transformation and optimization processes, structured log data with business terms and entity labels is generated. This significantly improves the semantic richness and accuracy of log data, providing a more reliable data basis for subsequent lineage tracing.

[0036] In this embodiment, dynamic semantic injection is performed on the standardized metadata set to obtain a semantic-enhanced metadata vector, including: Convert the timestamps in the standardized metadata set into periodic encodings. For example: Use Fourier transform to extract the daily, weekly, and monthly periodic features of the timestamps, generating a vector with 24 dimensions representing different time periods in a day.

[0037] Spatial information encoding: Map the spatial information into hierarchical encodings. For example: For a database system, encode it as "system level (4 bits) → subsystem level (4 bits) → component level (4 bits)", such as "DBMS_0001_ORACLE_0001_SALES_0001".

[0038] Business rule injector: Business document parsing: Use the SciBERT model to parse business documents and extract condition-action rules. For example: From the document "When the customer risk score > 80, trigger manual review", extract the condition "customer risk score > 80" and the action "trigger manual review".

[0039] Log pattern matching: Identify the operation sequence patterns in logs through pattern matching. For example: In the logs, find the frequently occurring operation sequence "SELECT order → UPDATE inventory", and map it to the business rule "inventory locking rule".

[0040] Feature concatenation: Concatenate the spatio-temporal encoding (48 dimensions) and business rule embedding (128 dimensions) with the original metadata features (256 dimensions) to generate a 512-dimensional semantic-enhanced metadata vector.

[0041] In this embodiment, constructing an initial cognitive graph according to the semantic-enhanced metadata vector includes: Dynamically select a graph construction strategy according to the data density. For example, for high-density data (the number of connections between nodes is greater than the threshold), use a fully connected graph and use L1 regularization pruning to retain the K edges with the highest weights (K is dynamically adjusted according to the data density, such as K = 100). For sparse data, use a k-NN graph, and the value of k is dynamically adjusted according to the data distribution (such as between k = 5 and 15).

[0042] Multi-relational graph neural network: Definition of relationship types: Support 18 predefined relationship types, such as data flow, dependency, influence, etc., and 3 dynamic relationship types (generated by business rules). For example, define the relationship type "data flow" to indicate that data flows from one table to another table.

[0043] Graph attention propagation mechanism: Multi-head attention calculation: Aggregate neighbor information through multi-head attention (16 heads). For example, each attention head independently calculates the weights of neighbor nodes, and then concatenates the outputs of the 16 attention heads, retaining the original features (using residual connections).

[0044] Through the above solution, this application first injects spatio-temporal encoding and business rules into the knowledge graph in vector form, realizes the fusion of static metadata and dynamic context, automatically selects a graph construction strategy according to the data density, balances computational efficiency and information integrity, and constructs a complete processing chain from raw data to a dynamic cognitive graph, which is particularly suitable for complex systems that require in-depth business understanding (such as financial risk control, supply chain management). Through dynamic semantic injection and continuous learning mechanisms, it can reflect the changes in the system state in real time, providing a high-quality knowledge base for subsequent spatio-temporal causal reasoning and autonomous governance.

[0045] In this embodiment, constructing a blood relationship cognitive graph according to the blood relationship metadata, data operation logs, and business documents includes: According to the original blood relationship set in the blood relationship metadata, data operation logs, and business documents, the original blood relationship set includes operation type, timestamp, and confidence score; Generate an enhanced blood relationship set according to the original blood relationship set, and the enhanced blood relationship set includes business semantic labels, influence scope, and causal strength; Generate a blood relationship cognitive graph according to the enhanced blood relationship set.

[0046] In this embodiment, first parse the structured data, and the following parsing methods can be used (for example only): SQL blood relationship extraction: Use regular expressions to match keywords. For example, keywords such as CREATE TABLE, INSERT INTO, etc., and combine syntax tree analysis (ANTLR library) to parse the field-level blood relationship. For example: Extract source_table.col1 → target_table.col1 from INSERT INTO target_table SELECT col1, col2 FROM source_table; Extract the update operation lineage of orders.status from UPDATE orders SET status=2 WHERE customer_id=1001.

[0047] In this embodiment, semi-structured logs can be parsed in the following way (for illustration only): Parse task_id and upstream_task_ids in Apache Airflow logs to construct job-level lineage. For example: extract extract_data → load_data from the log Task(task_id='load_data')executed with upstream=[Task(task_id='extract_data')]; Identify expectation_type (such as expect_column_values_to_be_unique) in Great Expectations validation logs and associate it with the corresponding field lineage; In this embodiment, unstructured logs can be parsed in the following way (for illustration only): Use the SciBERT model for named entity recognition (NER) to extract data entities (such as "customer information table") and operation verbs (such as "synchronize to the data warehouse") to construct a draft document-level lineage. For example: Extract production_library.customer_information_table → data_warehouse.customer_information_table_daily_increment from "Synchronize the customer information table in the production library to the data warehouse every day".

[0048] The above parsed lineage relationships are the original lineage relationship set.

[0049] Based on the original lineage relationship set, this application introduces attributes such as business semantic tags, scope of influence, and causal strength to generate an enhanced lineage relationship set. Further, based on these enhanced lineage relationships, the present invention constructs a lineage cognitive graph to achieve a comprehensive description and in-depth understanding of lineage relationships.

[0050] In this embodiment, generating an enhanced lineage relationship set according to the original lineage relationship set includes: Graph structure construction: Attribute Graph Storage: Use the Neo4j graph database to store lineage relationships. Node attributes include: entity type (e.g., table / field / job); business domain (e.g., finance / medical / logistics); most recent update time; Edge Attribute Design: Edge attributes include: operation type (e.g., SELECT / INSERT / UPDATE); causal strength (0 - 1); scope of influence (e.g., direct / indirect); Quality Optimization Engine: Connectivity Analysis: Calculate the global connectivity rate of the graph and mark isolated nodes (connectivity rate < 10%). For example, if a table has no upstream or downstream lineage and is not referenced by business rules, it is marked as an isolated node.

[0051] Critical Path Identification: Use the PageRank algorithm to calculate the importance of nodes and perform double-backup storage for the top 5% of nodes. For example, the PageRank value of a core dimension table (such as the customer master data table) is usually > 0.05; Dynamic Pruning Strategy: Set quality thresholds (causal strength > 0.3 & confidence > 0.6). Edges below the thresholds will be moved to the "pending confirmation lineage library"; Incremental Update Mechanism: Change Capture: Use database CDC (Change Data Capture) technology to capture table structure changes (such as adding fields) in real time. For example, when detecting ALTER TABLE orders ADD COLUMN discount, trigger the lineage update process; Influence Propagation: Use breadth-first search (BFS) for influence propagation analysis. The update scope = directly associated nodes + 2-degree indirect nodes. For example, when modifying the structure of the "order table", the lineage relationships of all directly associated ETL jobs and downstream reports need to be updated.

[0052] Through the above solution, a complete processing chain from raw data operations to semantic lineage graphs is constructed, which is particularly suitable for complex systems that require precise traceability. Through semantic enhancement and dynamic optimization mechanisms, it can reflect data association changes in real time and provide intelligent support for scenarios such as data governance and compliance auditing.

[0053] In this embodiment, generating a governance policy vector based on the lineage cognitive graph, system real-time metrics, and business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector includes: Encode the blood relationship cognitive graph using a graph attention network to obtain the graph neural network encoding vector; specifically, use a graph attention network (GAT) to encode the blood relationship graph, and the node features fuse business semantic labels, influence scope, and causal strength. For example, the node initialization features are: [business domain code, entity type code, data quality score, most recent update timestamp]; Attention mechanism calculation: Calculate the attention coefficient for each edge; after 3 layers of GAT propagation, generate a 1024-dimensional embedding vector for each node.

[0054] Perform dynamic weight adjustment based on the graph neural network encoding vector to obtain the weight-adjusted graph; specifically, dynamically adjust the edge weights through a spatio-temporal decay attention mechanism, and combine business impact factors to enhance the edge weights of critical paths (such as the core report generation link).

[0055] Obtain the governance strategy action set based on the weight-adjusted graph; Generate an executable governance strategy package based on the governance strategy action set, and the executable governance strategy package includes action type, target, priority, and specific operations; Perform conflict resolution processing based on the executable governance strategy package to obtain the conflict-resolution strategy package and send it to the execution engine.

[0056] In this embodiment, the obtaining the governance strategy action set based on the weight-adjusted graph includes: Based on the weight-adjusted graph, use breadth-first search to enumerate all causal paths with a length ≤ 3 to obtain each path with intensity to be calculated; Calculate the intensity for each path with intensity to be calculated to obtain the causal intensity of each path with intensity to be calculated; Input the graph into a graph convolutional network, and aggregate neighbor node information through multi-layer convolutional operations to generate a latent path representation; Based on the latent path representation, calculate the indirect influence of nodes through a message passing mechanism. Each node transmits its own influence to its neighbors to obtain the indirect causal reasoning result; Perform counterfactual intervention on critical nodes based on the weight-adjusted graph, simulate the change amount of the target node after intervention, and use the SHAP value to calculate the marginal contribution of each feature to obtain the multi-level causal reasoning result; for example, assume that node B is removed and simulate the change amount of the target node (such as report C) after intervention.

[0057] Perform dynamic weight adjustment and fusion based on the multi-level causal reasoning result to obtain the hierarchical weight allocation result; Generate a fused causal strength map based on the multi-level causal reasoning results and the hierarchical weight assignment results; specifically, set a basic weight, and obtain the direct causal weight by attenuating according to the path length; set a dynamic weight, and adjust the indirect causal weight according to the graph connectivity rate; set an exploration weight, and enhance the counterfactual weight according to the intervention effect; fuse the multi-level causal strengths (the direct causal weight, indirect causal weight, and counterfactual weight above) for each edge, and use the Softmax function to normalize the fused causal strength, so as to obtain the fused causal strength map.

[0058] Generate a global state representation based on the fused causal strength map; specifically, use a graph isomorphism network (GIN) to encode the map. Aggregate node features through a multi-layer network; perform average pooling on all node features to generate a global state representation; Input the global state representation into a deep Q-network, map the state to the action value through a multi-layer network, and use the ε-greedy strategy for action exploration, so as to obtain action value evaluation result information; Select the optimal action according to the Q function value based on the action value evaluation result information, and each of the optimal actions constitutes the governance strategy action set.

[0059] Through the above solution, the present application comprehensively captures causal relationships through direct path enumeration, indirect influence propagation, and counterfactual intervention simulation. According to the path length, graph connectivity rate, and intervention effect, the priority of causal weights at each level is adjusted in real time. Through graph encoding and deep Q-learning, the graph state is mapped to specific governance actions, realizing end-to-end decision optimization.

[0060] In this embodiment, the conflict resolution process according to the executable governance policy package to obtain a conflict-resolution policy package and send it to the execution engine includes: Identify the affected stakeholders according to the executable governance policy package, so as to obtain game theory model parameters, where the game theory model parameters include game players, strategy sets, utility functions, and cost functions; Iteratively update the strategy combination of the strategy set through the replicator dynamics equation to find an equilibrium solution, so as to obtain a Nash equilibrium strategy combination and non-equilibrium strategy pairs; Generate a compensation plan based on the non-equilibrium strategy pairs and calculate the compensation cost, where the Nash equilibrium strategy combination and the compensation plan and calculated compensation cost corresponding to each non-equilibrium strategy constitute the conflict-resolution policy package.

[0061] In this embodiment, the business SLO may include response time, throughput, error rate, etc. (defined by the business department, indicating the business's expectations for system performance).

[0062] In this embodiment, the system real-time metrics may include CPU utilization, remaining memory, network latency, etc. (provided by system monitoring tools, reflecting the current operating state of the system).

[0063] After obtaining the governance policy action set adjusted according to the weights, the step of inputting the lineage cognitive graph, system real-time metrics, and business SLO into the governance model to generate a governance policy vector and sending it to the execution engine, so that the execution engine processes according to the governance policy vector further includes: Obtain a preset policy template database, where the preset policy template database includes at least one predefined policy template and the action type corresponding to each predefined policy template; Obtain the corresponding predefined policy template from the preset policy template database according to the action type in the obtained governance policy action set; Obtain system real-time metrics and historical execution logs; in this embodiment, the historical execution logs can be read from the log database, such as the policy execution records in the past 24 hours.

[0064] Generate a parameterized policy template according to the predefined policy template, system real-time metrics, and historical execution logs; in this embodiment, the predefined policy template is, for example, a verification rule template. For example, if the action in the governance policy action set is to perform data quality verification, then the verification rule template is correspondingly selected; In this embodiment, fill the obtained data into the template. For example, extract the load information from the system real-time state.

[0065] Extract the execution time (such as the most recent execution = 24 hours ago) and success rate (such as N successes / N total = 10 / 12) from the historical execution logs.

[0066] Fill the template with these contents combined with the object of action (such as the table name) and the extracted context parameters. For example, set the priority to "high" (the original priority P original = high).

[0067] After filling the data into the template, dynamically adjust the policy priority using the context-aware priority dynamic adjustment formula to obtain the adjusted governance policy action set; specifically, the context-aware priority dynamic adjustment formula is as follows:

[0068] Where, is the original priority (that is, the priority recorded in the governance policy vector, which can be represented by a number. For example, 1 represents high level, and 2 represents general); is the current system load; is the load threshold (which can be set according to needs); Indicates the last execution time of the policy; Indicates the success rate of the policy; Indicates the historical success rate; 、 and are weight factors and can be set according to needs; Is the adjusted score. When the adjusted score is closer to a certain threshold, it is used as the priority corresponding to that threshold. For example, if the calculated is 1.9 and 2 is the general level, and it is closer to the general level, then it is changed to the general level.

[0069] After various adjustments are made, it is necessary to evaluate the adjusted set of governance policy actions using a multi-modal policy generation quality evaluation model to obtain a score value; In this embodiment, the multi-modal policy generation quality evaluation model is as follows: Q = ; where represents the number of lineage nodes covered by the currently adjusted set of governance policy actions; represents the number of violated dependencies of the currently adjusted set of governance policy actions; represents the predicted quality improvement number; Q is the score value; ω1, ω2, ω3 represent weights, and these weights are used to adjust the influence degree of different dimensions on the final score and can be set and adjusted according to actual needs; is the total dependency degree, which refers to the number of all possible dependencies in the lineage graph; is the maximum benefit, which refers to the maximum data quality improvement number that can be achieved in the optimal case. It represents an ideal upper limit and can be set according to needs; Judge whether the score value exceeds the scoring threshold. If so, generate an executable governance policy package according to the adjusted set of governance policy actions. For example, only when Q>0.7, the policy package (including the adjusted priority) is published to the execution engine.

[0070] This application innovatively integrates three dimensions of system load, policy execution timeliness, and historical execution success rate into the dynamic adjustment of policy priority. Through the exponential decay function , ensure that the priority of policies that have not been executed recently gradually increases to avoid policy starvation.

[0071] Through item, make the policies with excellent historical performance obtain higher priority, forming positive feedback.

[0072] This application first proposes a three-dimensional evaluation framework of coverage-conflict rate-benefit ratio to quantify the quality of policy generation.

[0073] Through items, the encouragement strategy covers more key data entities.

[0074] Through items, punish the strategies that violate the internal data dependencies.

[0075] In this embodiment, in the process of iteratively updating the strategy combination through the replicator dynamics equation for the strategy set to find the equilibrium solution, thereby obtaining the Nash equilibrium strategy combination and the non-equilibrium strategy pair, the AI data warehouse full-link lineage tracing method further includes: In the process of iteratively updating the strategy combination through the replicator dynamics equation each time, use the stakeholder influence weighted satisfaction formula to evaluate the current strategy combination; When the change rate of the weighted satisfaction score is less than 1% for three consecutive iterations, terminate the iteration in advance, thereby obtaining the Nash equilibrium strategy combination and the weighted satisfaction score for the current iteration of the terminated iteration.

[0076] In this embodiment, a strategy combination is a vector representing the strategies selected by all players in the current state, and the strategy combination may be expressed as: S = {Producer: Reject, Consumer: Delay execution}; In the initial strategy allocation session, we randomly assign an initial strategy to each player, and the set of these strategies constitutes the initial strategy combination . For example: The producer is randomly assigned the strategy "Reject".

[0077] The consumer is randomly assigned the strategy "Delay execution".

[0078] The combination of these two strategies = {Producer: Reject, Consumer: Delay execution} is the initial strategy combination.

[0079] The initial strategy combination will be input into the strategy update iteration session as the starting point of the iteration.

[0080] In each iteration, the strategy combination is updated according to the replicator dynamics equation to generate a new strategy combination .

[0081] The iteration process will continue until the strategy combination converges to the Nash equilibrium.

[0082] In this embodiment, the replicator dynamics equation describes the rate of change of the strategy combination over time, that is , where represents the player i at time t choosing the strategy Ratio (or probability).

[0083] The equation updates the strategy combination by considering the utility and average utility of the players, making it converge to the equilibrium state.

[0084] In this embodiment, the formula for the weighted satisfaction degree of the stakeholder influence is as follows:

[0085] Wherein, is the stakeholder influence weight (such as the producer in the above text); is the current utility value (such as the producer utility); is the minimum acceptable utility; is the weighted satisfaction score; is the maximum acceptable utility.

[0086] In this embodiment, when the continuous 3 - time iteration change rate < 1%, the iteration is terminated in advance.

[0087] This application combines the influence weight with the normalized utility to achieve fair coordination of unequal stakeholders. Through terms, the utility value is mapped to the range of [0, 1], which is convenient for cross - stakeholder comparison.

[0088] This application also provides an AI data warehouse full - link lineage tracking device. The AI data warehouse full - link lineage tracking device includes a data acquisition module, a structured log data acquisition module, a knowledge graph acquisition module, a lineage metadata acquisition module, a lineage cognitive graph acquisition module, a governance strategy vector generation module, and a governance data acquisition module. Among them, The data acquisition module is used to acquire data operation logs and business documents; The structured log data acquisition module is used to acquire structured log data with business terms and entity labels; The knowledge graph acquisition module is used to establish a knowledge graph containing the mapping relationship between business concepts and data entities according to the structured log data and business documents; The lineage metadata acquisition module is used to generate lineage metadata including business semantics according to the knowledge graph and data job execution logs; The lineage cognitive graph acquisition module is used to construct a lineage cognitive graph according to the lineage metadata, data operation logs, and business documents; The governance strategy vector generation module is used to generate a governance strategy vector according to the lineage cognitive graph, system real - time metrics, and business SLO and send it to the execution engine, so that the execution engine processes according to the governance strategy vector; The governance data acquisition module is used to acquire the execution results of governance policies, the user feedback information of the execution results of governance policies, and the system logs generated during the governance process.

[0089] The AI data warehouse full-linkage lineage tracing method proposed in this application breaks the barrier between technology and business through the full-linkage data fusion technology; captures the complex semantics and causal relationships behind data operations through semantic-enhanced metadata construction and dynamic lineage cognitive graph generation technology; and realizes the intelligent generation and optimization of governance policies through core technologies such as multi-level causal reasoning and intelligent decision-making closed-loop. These technical solutions jointly solve the fundamental limitations of traditional methods in semantic loss, dynamic adaptation, intelligent decision-making, and causal reasoning, providing a new solution for data governance.

[0090] In addition, this application also realizes the optimal decision-making under multi-objective trade-off through the game theory model and conflict resolution technology; improves the decision-making efficiency and flexibility through the preset template and dynamic policy generation technology; and ensures the fairness and reasonableness of the decision-making process through the stakeholder influence weighted satisfaction technology. The integration of these technical solutions makes this invention have significant technical innovation and practical application value in the field of data governance.

[0091] Although the present invention has been described in detail with general descriptions and specific implementation examples above, based on the present invention, some modifications or improvements can be made, which are obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection required by the present invention.

Claims

1. An AI data warehouse full-link lineage tracking method, characterized in that, The AI data warehouse full-link lineage tracking method includes: Obtain data operation logs and business documents; Obtain structured log data with business terms and entity tags; Build a knowledge graph containing the mapping relationship between business concepts and data entities based on the structured log data and business documents; Generate lineage metadata including business semantics based on the knowledge graph and data job execution logs; Construct a lineage cognitive graph based on the lineage metadata, data operation logs, and business documents; Generate a governance policy vector based on the lineage cognitive graph, system real-time metrics, and business SLO and send it to the execution engine, so that the execution engine processes according to the governance policy vector; Obtain the governance policy execution result, user feedback information on the governance policy execution result, and system logs generated during the governance process.

2. The AI data warehouse full-link lineage tracing method according to claim 1, wherein The obtaining of structured log data with business terms and entity tags includes: Generate a structured log token stream based on the data operation logs; Generate a multi-modal fusion semantic vector based on the structured log token stream; Generate a spatio-temporal graph structure based on the multi-modal fusion semantic vector; Generate an implicit relationship embedding vector based on the spatio-temporal graph structure; Generate a spatio-temporal enhanced node representation through the implicit relationship embedding vector; Generate a rule-reinforced node representation based on the spatio-temporal enhanced node representation; Generate a weight-rescaled semantic vector based on the rule-reinforced node representation; Generate a robustness-enhanced semantic vector based on the weight-rescaled semantic vector; Generate a probabilistic entity annotation sequence based on the robustness-enhanced semantic vector; Generate an optimized entity annotation sequence as structured log data with business terms and entity tags based on the probabilistic entity annotation sequence.

3. The AI data warehouse full-link lineage tracing method according to claim 2, wherein The building of a knowledge graph containing the mapping relationship between business concepts and data entities based on the structured log data and business documents includes: Extract data from the structured log data and business documents to obtain a standardized metadata set; Perform dynamic semantic injection on the standardized metadata set to obtain a semantic-enhanced metadata vector; Construct an initial cognitive graph based on the semantic-enhanced metadata vector.

4. The AI data warehouse full-link lineage tracing method according to claim 3, wherein, The construction of a lineage cognitive graph based on the lineage metadata, data operation logs, and business documents includes: Based on the original lineage relationship set in the lineage metadata, data operation logs, and business documents, the original lineage relationship set includes operation type, timestamp, confidence score; Generate an enhanced lineage relationship set based on the original lineage relationship set, the enhanced lineage relationship set includes business semantic tags, scope of influence, causal strength; Generate a lineage cognitive graph based on the enhanced lineage relationship set.

5. The AI data warehouse full-link lineage tracing method according to claim 4, wherein The generation of a governance policy vector based on the lineage cognitive graph, system real-time metrics, and business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector includes: Use a graph attention network to encode the lineage cognitive graph to obtain a graph neural network encoded vector; Perform dynamic weight adjustment based on the graph neural network encoded vector to obtain a weight-adjusted graph; Obtain a governance policy action set based on the weight-adjusted graph; Generate an executable governance policy package according to the governance policy action set, where the executable governance policy package includes action type, object of action, priority, and specific operations; Perform conflict resolution processing according to the executable governance policy package, so as to obtain the conflict-resolution policy package and send it to the execution engine.

6. The AI data warehouse full-link lineage tracing method according to claim 5, wherein, The obtaining of the governance policy action set according to the graph after weight adjustment includes: According to the graph after weight adjustment, use breadth-first search to enumerate all causal paths with length ≤ 3, so as to obtain each path to be calculated for intensity; Perform intensity calculation on each path to be calculated for intensity, so as to obtain the causal intensity of each path to be calculated for intensity; Input the graph into a graph convolutional network, aggregate neighbor node information through multi-layer convolutional operations, and generate a latent path representation; Based on the latent path representation, calculate the indirect influence of nodes through a message passing mechanism; each node transmits its own influence to its neighbors, so as to obtain an indirect causal inference result; Perform counterfactual intervention on key nodes according to the graph after weight adjustment, simulate the change amount of the target node after intervention, and use the SHAP value to calculate the marginal contribution of each feature, so as to obtain a multi-level causal inference result; Perform dynamic weight adjustment and fusion according to the multi-level causal inference result, so as to obtain a hierarchical weight allocation result; Generate a fused causal intensity graph according to the multi-level causal inference result and the hierarchical weight allocation result; Generate a global state representation according to the fused causal intensity graph; Input the global state representation into a deep Q-network, map the state to action value through a multi-layer network, and use the ε-greedy strategy for action exploration, so as to obtain action value evaluation result information; Select the optimal action according to the Q function value according to the action value evaluation result information, and each of the optimal actions constitutes the governance policy action set.

7. The AI data warehouse full-link lineage tracing method according to claim 6, wherein The performing conflict resolution processing according to the executable governance policy package, so as to obtain the conflict-resolution policy package and send it to the execution engine includes: Identify the affected stakeholders according to the executable governance policy package, so as to obtain game theory model parameters, where the game theory model parameters include game players, strategy sets, utility functions, and cost functions; Iteratively update the strategy combination of the strategy set through the replicator dynamics equation, search for the equilibrium solution, so as to obtain the Nash equilibrium strategy combination and non-equilibrium strategy pairs; Generate a compensation plan according to the non-equilibrium strategy pairs and calculate the compensation cost, where the Nash equilibrium strategy combination and the compensation plan and calculated compensation cost corresponding to each non-equilibrium strategy constitute the conflict-resolution policy package.

8. The AI data warehouse full-link lineage tracing method according to claim 7, wherein, After obtaining the governance policy action set according to the graph after weight adjustment, generating a governance policy vector according to the blood relationship cognitive graph, system real-time metrics, and business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector further includes: Obtain a preset policy template database, where the preset policy template database includes at least one predefined policy template and the action type corresponding to each predefined policy template; Obtain the corresponding predefined policy template from the preset policy template database according to the action type in the obtained governance policy action set; Obtain the system real-time metrics and historical execution logs; Generate a parameterized policy template based on the predefined policy template, system real-time metrics, and historical execution logs; Dynamically adjust the policy priority using the context-aware priority dynamic adjustment formula to obtain the adjusted set of governance policy actions; Evaluate the adjusted set of governance policy actions using a multi-modal policy generation quality assessment model to obtain a score value; Determine whether the score value exceeds the scoring threshold. If so, generate an executable governance policy package based on the adjusted set of governance policy actions.

9. The AI data warehouse full-link lineage tracing method according to claim 8, characterized in that, In the process of iteratively updating the policy combination through the replicator dynamics equation for the policy set to find the equilibrium solution, thereby obtaining the Nash equilibrium policy combination and non-equilibrium policy pairs, the AI data warehouse full-link lineage tracing method further includes: During each iteration of updating the policy combination through the replicator dynamics equation, evaluate the current policy combination using the stakeholder influence weighted satisfaction formula; When the change rate of the weighted satisfaction score for 3 consecutive iterations < 1%, terminate the iteration in advance to obtain the Nash equilibrium policy combination and the weighted satisfaction score for the current iteration of the terminated iteration.

10. An AI data warehouse full-link lineage tracking device, characterized in that, The AI data warehouse full-link lineage tracing device includes: A data acquisition module, which is used to acquire data operation logs and business documents; A structured log data acquisition module, which is used to acquire structured log data with business terms and entity tags; A knowledge graph acquisition module, which is used to establish a knowledge graph including the mapping relationship between business concepts and data entities based on the structured log data and business documents; A lineage metadata acquisition module, which is used to generate lineage metadata including business semantics based on the knowledge graph and data job execution logs; A lineage cognitive graph acquisition module, which is used to construct a lineage cognitive graph based on the lineage metadata, data operation logs, and business documents; A governance policy vector generation module, which is used to generate a governance policy vector based on the lineage cognitive graph, system real-time metrics, and business SLO and send it to the execution engine, so that the execution engine processes according to the governance policy vector; A governance data acquisition module, which is used to acquire governance policy execution results, user feedback information on governance policy execution results, and system logs generated during the governance process.

Images