An AI data warehouse full-link lineage tracking method and device

Abstract

The present application discloses a full-link lineage tracking method and device for an AI data warehouse, which relates to the field of data processing technology. The method includes: obtaining structured log data with business terms and entity tags; establishing a knowledge graph based on the structured log data and business documents; generating lineage metadata based on the knowledge graph and data job execution logs; building a lineage cognitive graph based on the lineage metadata, data operation logs and business documents; generating a governance policy vector based on the lineage cognitive graph, system real-time indicators and business SLOs and sending it to the execution engine, so that the execution engine processes it according to the governance policy vector; obtaining the governance policy execution results, user feedback information on the governance policy execution results and the system logs generated during the governance process. The present application can capture the complex semantics and causal relationships behind data operations.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an AI data warehouse full-link lineage tracking method and an AI data warehouse full-link lineage tracking device. Background Art

[0002] In the field of data governance and intelligent decision-making, full-link lineage tracking is a core technology for ensuring data asset reliability and optimizing system operational efficiency. Existing lineage maps are mostly static topological structures, making them difficult to adapt to the dynamic evolution of data operations. When business logic is adjusted (such as adding new data sources) or system architecture changes (such as computing engine upgrades), existing methods require manual intervention to reconstruct lineage relationships, causing governance strategies to lag behind actual needs. Summary of the Invention

[0003] The purpose of the present invention is to provide an AI data warehouse full-link lineage tracing method to solve at least one of the above-mentioned technical problems.

[0004] One aspect of the present invention provides a method for tracing lineage in an AI data warehouse across the entire chain, the method comprising:

[0005] Obtain data operation logs and business documents;

[0006] Get structured log data with business terms and entity tags;

[0007] Establishing a knowledge graph containing mapping relationships between business concepts and data entities based on the structured log data and business documents;

[0008] Generate lineage metadata including business semantics based on the knowledge graph and data job execution log;

[0009] Constructing a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents;

[0010] Generate a governance strategy vector based on the lineage cognitive map, system real-time indicators, and business SLOs and send it to the execution engine, so that the execution engine processes it according to the governance strategy vector;

[0011] Obtain governance policy execution results, user feedback on governance policy execution results, and system logs generated during the governance process.

[0012] Optionally, obtaining structured log data with business terms and entity tags includes:

[0013] Generate a structured log token stream based on the data operation log;

[0014] generating a multimodal fusion semantic vector according to the structured log token stream;

[0015] Generating a spatiotemporal graph structure according to the multimodal fusion semantic vector;

[0016] generating an implicit relation embedding vector according to the spatiotemporal graph structure;

[0017] Generate spatiotemporal enhanced node representations through implicit relation embedding vectors;

[0018] Strengthening node representation according to the spatiotemporal enhanced node representation generation rule;

[0019] Strengthen the node representation according to the rules to generate a weighted recalibrated semantic vector;

[0020] Recalibrating the semantic vector according to the weight to generate a robustness enhanced semantic vector;

[0021] Generating a probabilistic entity annotation sequence according to the robustness enhanced semantic vector;

[0022] An optimized entity annotation sequence is generated according to the probabilistic entity annotation sequence as structured log data with business terms and entity tags.

[0023] Optionally, the establishing of a knowledge graph including mapping relationships between business concepts and data entities based on the structured log data and the business documents includes:

[0024] Extracting data from the structured log data and business documents to obtain a standardized metadata set;

[0025] Dynamically inject semantics into the standardized metadata set to obtain semantically enhanced metadata vectors;

[0026] An initial cognitive graph is constructed according to the semantically enhanced metadata vector.

[0027] Optionally, constructing a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents includes:

[0028] According to the original blood relationship set in the blood relationship metadata, data operation log and business document, the original blood relationship set includes operation type, timestamp and confidence score;

[0029] generating an enhanced blood relationship set according to the original blood relationship set, wherein the enhanced blood relationship set includes a business semantic label, an impact range, and a causal strength;

[0030] A blood relationship cognitive map is generated according to the enhanced blood relationship set.

[0031] Optionally, generating a governance policy vector based on the lineage cognitive map, the system real-time indicators, and the business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector, includes:

[0032] Use graph attention network to encode the bloodline cognitive graph to obtain graph neural network encoding vector;

[0033] Dynamically adjust the weights according to the graph neural network encoding vector to obtain a weight-adjusted graph;

[0034] Obtain the governance strategy action set based on the weight-adjusted graph;

[0035] Generate an executable governance policy package according to the governance policy action set, wherein the executable governance policy package includes an action type, an object, a priority, and specific operations;

[0036] Conflict resolution is performed according to the executable governance policy package, thereby obtaining a conflict-resolved policy package and sending it to the execution engine.

[0037] Optionally, obtaining a governance policy action set according to the weight-adjusted graph includes:

[0038] According to the weight-adjusted graph, use breadth-first search to enumerate all causal paths with length ≤ 3, so as to obtain each strength path to be calculated;

[0039] Performing strength calculation on each strength path to be calculated, thereby obtaining the causal strength of each strength path to be calculated;

[0040] The graph is input into the graph convolutional network, which aggregates neighbor node information through multi-layer convolution operations to generate potential path representations;

[0041] Based on the potential path representation, the indirect influence of nodes is calculated through a message passing mechanism. Each node transmits its influence to its neighbors to obtain indirect causal reasoning results.

[0042] Perform counterfactual interventions on key nodes based on the weight-adjusted graph, simulate the changes in target nodes after the intervention, and use SHAP values ​​to calculate the marginal contribution of each feature to obtain multi-level causal reasoning results;

[0043] Dynamic weight adjustment and fusion are performed according to the multi-level causal reasoning results to obtain the hierarchical weight distribution results;

[0044] Generate a fused causal strength map based on the multi-level causal reasoning results and the level weight distribution results;

[0045] generating a global state representation according to the fused causal strength map;

[0046] The global state representation is input into the deep Q network, the state is mapped to the action value through the multi-layer network, and the ε-greedy strategy is used for action exploration to obtain the action value evaluation result information;

[0047] The optimal action is selected according to the action value evaluation result information based on the Q function value, and each of the optimal actions constitutes the governance strategy action set.

[0048] Optionally, performing conflict resolution processing according to the executable governance policy package, thereby obtaining a conflict-resolved policy package and sending it to the execution engine includes:

[0049] Identifying affected stakeholders based on the executable governance policy package, thereby obtaining game theory model parameters, wherein the game theory model parameters include game parties, policy sets, utility functions, and cost functions;

[0050] Iteratively updating the strategy combination of the strategy set through the replicator dynamics equation to find the equilibrium solution, thereby obtaining the Nash equilibrium strategy combination and the non-equilibrium strategy pair;

[0051] A compensation scheme is generated according to the non-equilibrium strategy pair and a compensation cost is calculated, wherein the Nash equilibrium strategy combination and the compensation scheme corresponding to each non-equilibrium strategy and the calculated compensation cost constitute a conflict-resolved strategy package.

[0052] Optionally, after obtaining the governance policy action set according to the weight-adjusted graph, inputting the lineage cognitive graph, system real-time indicators, and business SLO into the governance model to generate a governance policy vector and sending it to the execution engine, so that the execution engine processes according to the governance policy vector, further includes:

[0053] Acquire a preset policy template database, wherein the preset policy template database includes at least one predefined policy template and an action type corresponding to each predefined policy template;

[0054] Acquire a corresponding predefined policy template from the preset policy template database according to the action type in the acquired governance policy action set;

[0055] Obtain system real-time indicators and historical execution logs;

[0056] Generate a parameterized policy template based on the predefined policy template, system real-time indicators, and historical execution logs;

[0057] Use the context-aware priority dynamic adjustment formula to dynamically adjust the policy priority to obtain the adjusted governance policy action set;

[0058] The adjusted governance policy action set is evaluated using a multimodal strategy to generate a quality assessment model to obtain a score value;

[0059] Determine whether the score value exceeds the score threshold. If so, generate an executable governance policy package based on the adjusted governance policy action set.

[0060] Optionally, in the process of iteratively updating the strategy combination through the replica dynamics equation for the strategy set to find an equilibrium solution, thereby obtaining a Nash equilibrium strategy combination and a non-equilibrium strategy pair, the AI ​​data warehouse full-link lineage tracing method further includes:

[0061] In each process of iteratively updating the strategy combination through the replicator dynamics equation, the current strategy combination is evaluated using the stakeholder influence weighted satisfaction formula;

[0062] When the weighted satisfaction score changes by less than 1% for three consecutive iterations, the iteration is terminated early to obtain the Nash equilibrium strategy combination and weighted satisfaction score at the time of termination.

[0063] This application also provides an AI data warehouse full-link lineage tracking device, which includes:

[0064] A data acquisition module, which is used to acquire data operation logs and business documents;

[0065] A structured log data acquisition module, wherein the structured log data acquisition module is used to acquire structured log data with business terms and entity tags;

[0066] A knowledge graph acquisition module, configured to establish a knowledge graph containing mapping relationships between business concepts and data entities based on the structured log data and business documents;

[0067] A lineage metadata acquisition module, configured to generate lineage metadata including business semantics based on the knowledge graph and data job execution logs;

[0068] A blood relationship cognitive map acquisition module, which is used to construct a blood relationship cognitive map based on the blood relationship metadata, data operation logs and business documents;

[0069] A governance strategy vector generation module, which is used to generate a governance strategy vector based on the lineage cognitive map, system real-time indicators, and business SLOs and send it to the execution engine, so that the execution engine processes it according to the governance strategy vector;

[0070] The governance data acquisition module is used to obtain governance policy execution results, user feedback information on governance policy execution results, and system logs generated during the governance process.

[0071] Beneficial effects

[0072] This application's AI data warehouse full-link lineage tracking method breaks down the barriers between technology and business in traditional data governance through full-link data integration, achieving a deep connection between business terms and data entities. Furthermore, through semantically enhanced metadata construction and dynamic lineage cognitive graph generation technology, this application can capture the complex semantics and causal relationships behind data operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a flow chart of the AI ​​data warehouse full-link lineage tracing method according to one embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below in conjunction with the drawings in the embodiments of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be understood as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0075] like Figure 1 The AI ​​data warehouse full-link lineage tracing method shown includes:

[0076] Step 1: Obtain data operation logs and business documents;

[0077] Step 2: Obtain structured log data with business terms and entity labels;

[0078] Step 3: Establish a knowledge graph containing mapping relationships between business concepts and data entities based on the structured log data and business documents;

[0079] Step 4: Generate lineage metadata including business semantics based on the knowledge graph and data job execution log;

[0080] Step 5: Construct a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents;

[0081] Step 6: Generate a governance strategy vector based on the lineage cognitive map, system real-time indicators, and business SLOs and send it to the execution engine, so that the execution engine processes it according to the governance strategy vector;

[0082] Step 7: Obtain the governance policy execution results, user feedback information on the governance policy execution results, and system logs generated during the governance process.

[0083] In this embodiment, the data operation log may include the following: SQL queries, ETL job logs, model training logs and other structured operation records;

[0084] In this embodiment, the business documents may include unstructured text or semi-structured data such as data dictionaries, requirement specifications, and system logs (operating system events, network transmission records);

[0085] In this embodiment, obtaining structured log data with business terms and entity tags includes:

[0086] Generate a structured log token stream based on data operation logs. Specifically, this process involves performing regular expression matching on raw log entries to extract timestamps, operation types, and entities involved (e.g., users, files, IP addresses, etc.). The extracted entities are mapped to predefined entity types, such as mapping user A to a user ID and file X to a file path. The parsed timestamp, operation type, and entity information are then serialized into a structured log token stream.

[0087] A multimodal fused semantic vector is generated based on the structured log token stream. Specifically, natural language processing techniques (e.g., word embedding) are used to extract text features from the log token stream and convert the text information into a vector representation. Timestamps in the log token stream are encoded to capture time series patterns and convert the temporal information into a vector representation. Spatial features are constructed based on entity types and relationships between entities, and the spatial information is converted into a vector representation. The text features, temporal features, and spatial features are fused to generate a multimodal fused semantic vector that integrates various information of the log entry.

[0088] A spatiotemporal graph structure is generated based on the multimodal fusion semantic vector; and a dynamic spatiotemporal graph is constructed based on the multimodal fusion semantic vector. In the graph, nodes represent entities, edges represent relationships between entities, and the weights of edges reflect the strength of relationships. The spatiotemporal graph can capture the dynamic changes of entities in time and space. In this embodiment, the node attributes include the entity type (such as user, file) and the frequency of occurrence of the entity in the log. The non-zero elements in the spatial adjacency matrix are defined as edges in the graph, and the weights of the edges are the co-occurrence frequencies between entities. The graphs corresponding to each time window are connected in series in chronological order to form a dynamic spatiotemporal graph structure. Each spatiotemporal graph represents entities and the relationships between them within a specific time window;

[0089] generating an implicit relation embedding vector according to the spatiotemporal graph structure;

[0090] In this embodiment, generating an implicit relationship embedding vector according to the spatiotemporal graph structure includes:

[0091] Predefined relationship pattern library, such as user-file access pattern, IP-user login pattern, etc.

[0092] Mining typical relationship patterns from historical log data and extracting their features. For example, for user-file access patterns, features such as the time distribution and access frequency of users accessing files can be extracted.

[0093] Represent the extracted relational patterns as feature vectors. For example, the features of the user-file access pattern are represented as a vector containing dimensions such as average access interval and maximum access frequency. Each vector is combined to generate an implicit relation embedding vector, where each element contains the pattern ID and the corresponding feature vector.

[0094] Perform edge relationship matching and initial weight assignment, specifically, including the following steps:

[0095] Feature similarity calculation: For each edge in the spatiotemporal graph structure, calculate its feature similarity with each implicit relationship pattern.

[0096] Pattern matching: For each edge, select the implicit relationship pattern with the highest similarity to its features.

[0097] Initial weight assignment: Map the similarity between the edge and the matching pattern to the initial weight of the edge.

[0098] Output: Initialized spatiotemporal graph edge weights, including edge IDs and initial weights.

[0099] The implicit relation embedding vector generation specifically includes the following steps:

[0100] Embedding vector learning: Based on the spatiotemporal graph structure and initial edge weights, graph embedding techniques (such as Node2Vec and GraphSAGE) are used to learn implicit relationship embedding vectors. These vectors can capture the complex implicit relationships between entities, thereby outputting implicit relationship embedding vectors. Each vector corresponds to an edge in the spatiotemporal graph and represents the implicit relationship represented by that edge.

[0101] Generate spatiotemporally enhanced node representations using implicit relational embeddings. Specifically, the Graph Attention Network (GAT) performs weighted aggregation of neighboring nodes in the spatiotemporal graph based on the weights of the implicit relational embeddings. For example, the representation of node A aggregates the representations of its neighboring nodes B and C, with the aggregation weight determined by the strength of the implicit relationship between A, B, and C. The graph attention mechanism considers both temporal decay factors and spatial adjacency to ensure that node representations capture spatiotemporal dynamics. For example, recent log events may have a greater impact on node representations.

[0102] In this embodiment, in the spatiotemporal enhanced node representation, each node corresponds to a vector that integrates its own attributes, neighbor information, and spatiotemporal dynamics.

[0103] Generate a rule-enhanced node representation based on the spatiotemporal enhanced node representation. Specifically, match the rules in the business rule library (e.g., "user IDs must be a combination of numbers and letters") with the node representation to identify nodes that meet the rule. For nodes that match the rule, their rule compliance information (e.g., whether they conform to the user ID format) is injected into the node representation to enhance the node's ability to express business rules, thereby obtaining a rule-enhanced node representation.

[0104] According to the rules, node representations are strengthened to generate weighted recalibrated semantic vectors. Specifically, a meta-learner is constructed, and a reinforcement learning strategy (PPO algorithm) is used to learn feature weights in different business scenarios. A weight generation network is designed, and a weight vector with a dimension of 256 is output. The Hadamard product is performed with the input vector to achieve feature recalibration, thereby obtaining a weighted recalibrated semantic vector.

[0105] Recalibrating the semantic vector according to the weights to generate a robustness-enhanced semantic vector; obtaining an adversarial sample generator, constructing semantically-preserving adversarial perturbations in the input space based on the FGSM algorithm, and adopting an alternating training strategy to first update the discriminator (to distinguish between real and adversarial samples), and then jointly optimizing the generator and the main network to improve the robustness of the model;

[0106] Generate a probabilistic entity annotation sequence based on the robustness-enhanced semantic vector. Specifically, a teacher-student architecture is constructed, where the teacher network uses BERT-large for soft label generation and the student network uses a lightweight Transformer for distillation learning. A temperature-scaled cross-entropy loss function is designed to balance hard label supervision and soft label guidance to generate a probabilistic entity annotation sequence.

[0107] Based on the probabilistic entity annotation sequence, an optimized entity annotation sequence is generated as structured log data with business terms and entity tags. Entity recognition is transformed into a sequential decision problem, defining the state space (current annotation position), action space (label selection), and reward function (F1 value increment). The DQN algorithm is deployed for online decision optimization, using an experience replay mechanism to break data correlation and optimize the entity annotation sequence.

[0108] This application uses natural language processing (NLP) technology to extract business terms and entity labels, and utilizes graph database technology to build a knowledge graph, achieving the association and mapping of business and technology. By acquiring data operation logs and business documents, and using natural language processing (NLP) and machine learning techniques, the application extracts business terms and entity labels from these logs and associates them with structured log data. Based on this associated data, the application constructs a knowledge graph that maps business concepts to data entities, achieving a deep integration of technology and business.

[0109] In this embodiment, the step of establishing a knowledge graph including mapping relationships between business concepts and data entities based on the structured log data and business documents includes:

[0110] Extracting data from the structured log data and business documents to obtain a standardized metadata set;

[0111] Dynamically inject semantics into the standardized metadata set to obtain semantically enhanced metadata vectors;

[0112] An initial cognitive graph is constructed according to the semantically enhanced metadata vector.

[0113] In this embodiment, the structured log data and business documents mentioned above are processed as follows:

[0114] File format recognition: Automatically recognizes formats such as CSV, JSON, XML, and TXT by analyzing the file's magic number and using file extension mapping. For example, a file starting with 0x89PNG is recognized as a PNG format.

[0115] Database schema parsing: Use the ANTLR library to parse SQL DDL statements, build an abstract syntax tree (AST), and extract information such as table names, field names, and data types. For example, for the SQL statement CREATE TABLE customers (id INT, name VARCHAR(100)), extract the table name customers, field names id and name, and their corresponding data types.

[0116] ETL job configuration parsing: Parses Apache Airflow's DAG definition file (in JSON format) to extract task dependencies and execution order. For example, we can parse from JSON to determine that Task A executes after Task B, thus establishing task dependencies.

[0117] A standardized representation including multiple core attributes is defined. For example, in this embodiment, 12 core attributes are included, as follows:

[0118] Entity ID: unique identifier, such as "TBL_CUSTOMER_001";

[0119] Entity type: table, field, job, rule, etc.;

[0120] Operation timestamp: in ISO 8601 format, such as "2025-05-26T10:00:00Z";

[0121] Business domain: finance, risk, marketing, etc.;

[0122] Data source: database name, file path, etc.;

[0123] Data quality score: a floating point number between 0 and 1, such as 0.95;

[0124] Semantic tags: customer, transaction, etc., separated by commas;

[0125] Parent entity: a list of parent entity IDs, such as ["DB_SALES", "SCH_CUSTOMER"];

[0126] Sub-entity: a list of subordinate entity IDs, such as ["COL_CUSTOMER_ID"];

[0127] Data lineage: Description of the path from source to target, such as "from source system A through ETL job B to target table C";

[0128] Business rules: a list of associated business rule IDs, such as ["RULE_RISK_SCORE"];

[0129] Metadata version: semantic version number, such as "1.0.0".

[0130] It is understood that the present application may also include the following conflict detection method:

[0131] Use a pre-trained BERT model (such as bert-base-uncased) to calculate the semantic similarity of field descriptions. For example:

[0132] For the field descriptions "Customer Risk Score" and "Customer Credit Rating," we calculate their cosine similarity. When the similarity falls below a threshold (e.g., 0.8), a manual arbitration interface is triggered. This interface provides a web interface displaying the conflicting fields and their context, such as the table name, field type, and sample data, for domain experts to adjudicate.

[0133] This application adopts multimodal fusion technology. Multimodal fusion can comprehensively utilize the various characteristics of log data, and the spatiotemporal graph structure can capture the temporal and spatial relationships in the log data, so as to more comprehensively understand the meaning of the log data. The structured log token stream is fused with the semantic information in the business document to generate a multimodal fusion semantic vector. Furthermore, through spatiotemporal graph structure modeling, the temporal and spatial relationships in the log data are captured to generate implicit relationship embedding vectors. Finally, through a series of complex vector transformation and optimization processes, structured log data with business terms and entity labels is generated. It significantly improves the semantic richness and accuracy of log data, providing a more reliable data foundation for subsequent lineage tracking.

[0134] In this embodiment, dynamically injecting semantics into the standardized metadata set to obtain a semantically enhanced metadata vector includes:

[0135] Convert the timestamps in the standardized metadata collection into a periodic encoding. For example, use Fourier transform to extract the daily, weekly, and monthly periodic features of the timestamp, generating a 24-dimensional vector representing different time periods of the day.

[0136] Spatial information encoding: Map spatial information into a hierarchical encoding. For example, for a database system, the encoding is "system level (4 bits) → subsystem level (4 bits) → component level (4 bits)", such as "DBMS_0001_ORACLE_0001_SALES_0001".

[0137] Business Rule Injector:

[0138] Business document parsing: Use the SciBERT model to parse business documents and extract condition-action rules. For example, from the document "When customer risk score > 80, trigger manual review," extract the condition "customer risk score > 80" and the action "trigger manual review."

[0139] Log pattern matching: Identify operation sequence patterns in logs through pattern matching. For example, if the frequently occurring operation sequence "SELECT Orders → UPDATE Inventory" is found in the log, it can be mapped to the business rule "Inventory Locking Rule."

[0140] Feature concatenation: The spatiotemporal encoding (48 dimensions) and business rule embedding (128 dimensions) are concatenated with the original metadata features (256 dimensions) to generate a 512-dimensional semantically enhanced metadata vector.

[0141] In this embodiment, constructing an initial cognitive graph based on the semantically enhanced metadata vector includes:

[0142] Dynamically select a graph construction strategy based on data density. For example, for high-density data (the number of connections between nodes is greater than a threshold), a fully connected graph is used with L1 regularization pruning, retaining the K edges with the highest weights (K is dynamically adjusted based on data density, such as K=100). For sparse data, a k-NN graph is used, with the k value dynamically adjusted based on the data distribution (e.g., between k=5 and 15).

[0143] Multi-relational graph neural network:

[0144] Relationship type definition: supports 18 predefined relationship types, such as data flow, dependency, and influence, as well as three dynamic relationship types (generated by business rules). For example, defining the relationship type "data flow" indicates that data flows from one table to another.

[0145] Graph Attention Propagation Mechanism:

[0146] Multi-head attention calculation: Aggregate neighbor information through multi-head attention (16 heads). For example, each attention head independently calculates the weight of the neighboring nodes, and then splices the outputs of the 16 attention heads to retain the original features (using residual connections).

[0147] Through this approach, this application, for the first time, injects spatiotemporal encodings and business rules into a graph in vector form, achieving the fusion of static metadata and dynamic context. It automatically selects a composition strategy based on data density, balancing computational efficiency and information integrity. This completes the processing chain from raw data to a dynamic cognitive graph, making it particularly suitable for complex systems requiring deep business understanding (such as financial risk control and supply chain management). Through dynamic semantic injection and continuous learning mechanisms, it can reflect system state changes in real time, providing a high-quality knowledge foundation for subsequent spatiotemporal causal reasoning and autonomous governance.

[0148] In this embodiment, constructing a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents includes:

[0149] According to the original blood relationship set in the blood relationship metadata, data operation log and business document, the original blood relationship set includes operation type, timestamp and confidence score;

[0150] generating an enhanced blood relationship set according to the original blood relationship set, wherein the enhanced blood relationship set includes a business semantic label, an impact range, and a causal strength;

[0151] A blood relationship cognitive map is generated according to the enhanced blood relationship set.

[0152] In this embodiment, the structured data is first parsed, and the following parsing method can be used (for example only):

[0153] SQL lineage extraction: Use regular expressions to match keywords, such as CREATE TABLE, INSERT INTO, etc., and combine syntax tree analysis (ANTLR library) to parse field-level lineage. For example:

[0154] Extract source_table.col1 → target_table.col1 from INSERT INTO target_table SELECT col1, col2 FROM source_table;

[0155] Extract the update operation lineage of orders.status from UPDATE orders SET status=2 WHERE customer_id=1001.

[0156] In this embodiment, semi-structured logs can be parsed in the following manner (for example only):

[0157] Parse the task_id and upstream_task_ids in the Apache Airflow log to build job-level lineage. For example: extract extract_data → load_data from the log Task(task_id='load_data') executedwith upstream=[Task(task_id='extract_data')];

[0158] Identify the expectation_type (such as expectation_column_values_to_be_unique) in the Great Expectations validation log and associate it with the corresponding field lineage;

[0159] In this embodiment, unstructured logs can be parsed in the following manner (for example only):

[0160] Use the SciBERT model for named entity recognition (NER), extract data entities (such as "customer information table") and action verbs (such as "synchronize to data warehouse"), and build document-level lineage drafts. For example:

[0161] Extract the "Production Database.Customer Information Table" → "Data Warehouse.Customer Information Table_Daily Increment" from "Synchronize the Production Database Customer Information Table to the Data Warehouse Daily".

[0162] The blood relationship analyzed above is the original blood relationship set.

[0163] Based on the original set of blood relationships, this application introduces attributes such as business semantic tags, impact scope, and causal strength to generate an enhanced set of blood relationships. Furthermore, based on these enhanced blood relationships, the present invention constructs a blood relationship cognitive map, achieving a comprehensive description and in-depth understanding of blood relationships.

[0164] In this embodiment, generating an enhanced blood relationship set according to the original blood relationship set includes:

[0165] Graph structure construction:

[0166] Property graph storage: Use the Neo4j graph database to store lineage relationships. Node attributes include: entity type (e.g., table / field / job); business domain (e.g., finance / medical / logistics); and last update time.

[0167] Edge attribute design:

[0168] Edge attributes include: operation type (e.g., SELECT / INSERT / UPDATE); causal strength (0-1); impact scope (e.g., direct / indirect);

[0169] Quality Optimization Engine:

[0170] Connectivity analysis: Calculate the global connectivity of the graph and mark isolated nodes (connectivity < 10%). For example, if a table has no upstream or downstream relationships and is not referenced by any business rules, it is marked as an isolated node.

[0171] Critical path identification: Use the PageRank algorithm to calculate node importance and perform dual backup storage for the top 5% nodes. For example, the PageRank value of core dimension tables (such as the customer master data table) is usually greater than 0.05.

[0172] Dynamic pruning strategy: Set a quality threshold (causal strength > 0.3 & confidence > 0.6). Edges below the threshold will be moved to the "pending lineage library";

[0173] Incremental update mechanism:

[0174] Change Capture: Use database CDC (Change Data Capture) technology to capture table structure changes (such as newly added fields) in real time. For example, when ALTER TABLE orders ADD COLUMN discount is detected, the lineage update process is triggered.

[0175] Impact propagation: Breadth-first search (BFS) is used for impact propagation analysis. The update scope = directly related nodes + 2nd-degree indirect nodes. For example, when modifying the "Order table" structure, the lineage relationships of all directly related ETL jobs and downstream reports need to be updated.

[0176] This solution builds a complete processing chain from raw data manipulation to semantic lineage graphs, making it particularly suitable for complex systems requiring precise traceability. Through semantic enhancement and dynamic optimization mechanisms, it can reflect changes in data associations in real time, providing intelligent support for scenarios such as data governance and compliance audits.

[0177] In this embodiment, generating a governance policy vector based on the lineage cognitive map, system real-time indicators, and business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector, includes:

[0178] Use a graph attention network to encode the lineage cognitive graph to obtain a graph neural network encoding vector. Specifically, use a graph attention network (GAT) to encode the lineage graph, and integrate node features with business semantic labels, influence scope, and causal strength. For example, node initialization features include: [business domain code, entity type code, data quality score, and last update timestamp].

[0179] Attention mechanism calculation: Calculate the attention coefficient for each edge; after three layers of GAT propagation, generate a 1024-dimensional embedding vector for each node.

[0180] Dynamic weight adjustment is performed according to the graph neural network encoding vector to obtain a weight-adjusted graph. Specifically, the edge weights are dynamically adjusted through the spatiotemporal attenuation attention mechanism, and the edge weights of key paths (such as the core report generation link) are enhanced in combination with business impact factors.

[0181] Obtain the governance strategy action set based on the weight-adjusted graph;

[0182] Generate an executable governance policy package according to the governance policy action set, wherein the executable governance policy package includes an action type, an object, a priority, and specific operations;

[0183] Conflict resolution is performed according to the executable governance policy package, thereby obtaining a conflict-resolved policy package and sending it to the execution engine.

[0184] In this embodiment, obtaining a governance policy action set according to the weight-adjusted graph includes:

[0185] According to the weight-adjusted graph, use breadth-first search to enumerate all causal paths with length ≤ 3, so as to obtain each strength path to be calculated;

[0186] Performing strength calculation on each strength path to be calculated, thereby obtaining the causal strength of each strength path to be calculated;

[0187] The graph is input into the graph convolutional network, which aggregates neighbor node information through multi-layer convolution operations to generate potential path representations;

[0188] Based on the potential path representation, the indirect influence of nodes is calculated through a message passing mechanism. Each node transmits its influence to its neighbors to obtain indirect causal reasoning results.

[0189] Perform counterfactual interventions on key nodes based on the weight-adjusted graph, simulate the change in the target node after the intervention, and use SHAP values ​​to calculate the marginal contribution of each feature to obtain multi-level causal inference results; for example, assuming that node B is removed, simulate the change in the target node (such as report C) after the intervention.

[0190] Dynamic weight adjustment and fusion are performed according to the multi-level causal reasoning results to obtain the hierarchical weight distribution results;

[0191] A fused causal strength map is generated based on the multi-level causal reasoning results and the hierarchical weight distribution results. Specifically, basic weights are set to obtain direct causal weights according to path length attenuation; dynamic weights are set to adjust indirect causal weights according to the graph connectivity rate; exploration weights are set to enhance counterfactual weights according to the intervention effect; multi-level causal strengths are fused for each edge (the direct causal weights, indirect causal weights, and counterfactual weights above), and the fused causal strengths are normalized using the Softmax function to obtain a fused causal strength map.

[0192] A global state representation is generated based on the fused causal strength graph. Specifically, the graph is encoded using a graph isomorphism network (GIN). Node features are aggregated through a multi-layer network. All node features are average-pooled to generate a global state representation.

[0193] The global state representation is input into the deep Q network, the state is mapped to the action value through the multi-layer network, and the ε-greedy strategy is used for action exploration to obtain the action value evaluation result information;

[0194] The optimal action is selected according to the action value evaluation result information based on the Q function value, and each of the optimal actions constitutes the governance strategy action set.

[0195] Through the above approach, this application comprehensively captures causal relationships through direct path enumeration, indirect influence propagation, and counterfactual intervention simulation. The priority of causal weights at each level is adjusted in real time based on path length, graph connectivity, and intervention effectiveness. Through graph encoding and deep Q-learning, graph states are mapped to specific governance actions, achieving end-to-end decision optimization.

[0196] In this embodiment, performing conflict resolution according to the executable governance policy package, thereby obtaining a conflict-resolved policy package and sending it to the execution engine includes:

[0197] Identifying affected stakeholders based on the executable governance policy package, thereby obtaining game theory model parameters, wherein the game theory model parameters include game parties, policy sets, utility functions, and cost functions;

[0198] Iteratively updating the strategy combination of the strategy set through the replicator dynamics equation to find the equilibrium solution, thereby obtaining the Nash equilibrium strategy combination and the non-equilibrium strategy pair;

[0199] A compensation scheme is generated according to the non-equilibrium strategy pair and a compensation cost is calculated, wherein the Nash equilibrium strategy combination and the compensation scheme corresponding to each non-equilibrium strategy and the calculated compensation cost constitute a conflict-resolved strategy package.

[0200] In this embodiment, the business SLO may include response time, throughput, error rate, etc. (defined by the business department and representing the business's expectations for system performance).

[0201] In this embodiment, the system real-time indicators may include CPU utilization, remaining memory, network latency, etc. (provided by a system monitoring tool to reflect the current operating status of the system).

[0202] After obtaining the governance policy action set according to the weight-adjusted graph, inputting the lineage cognitive graph, system real-time indicators, and business SLO into the governance model to generate a governance policy vector and sending it to the execution engine, so that the execution engine processes according to the governance policy vector, further includes:

[0203] Acquire a preset policy template database, wherein the preset policy template database includes at least one predefined policy template and an action type corresponding to each predefined policy template;

[0204] Acquire a corresponding predefined policy template from the preset policy template database according to the action type in the acquired governance policy action set;

[0205] Obtain system real-time indicators and historical execution logs; in this embodiment, historical execution logs can be read from a log database, such as policy execution records for the past 24 hours.

[0206] A parameterized policy template is generated based on the predefined policy template, the system real-time indicators, and the historical execution log. In this embodiment, the predefined policy template is, for example, a verification rule template. For example, if the action in the governance policy action set is to perform data quality verification, then the verification rule template is selected accordingly.

[0207] In this embodiment, the acquired data is filled into the template, for example, load information is extracted from the real-time status of the system.

[0208] Extract execution time (e.g., most recent execution = 24 hours ago) and success rate (e.g., N successes / N total = 10 / 12) from historical execution logs.

[0209] Combine these contents with the target object (such as table name) and the extracted context parameters to fill in the template, for example, set the priority to "high" (original priority P original = high).

[0210] After the template is filled with data, the policy priority is dynamically adjusted using the context-aware priority dynamic adjustment formula to obtain the adjusted governance policy action set. Specifically, the context-aware priority dynamic adjustment formula is as follows:

[0211]

[0212] in, The original priority (i.e., the priority recorded in the governance policy vector, which can be represented by a number, for example, 1 for high and 2 for general); is the current system load; is the load threshold (which can be set as needed); Indicates the last execution time of the strategy; Indicates the success rate of the strategy; represents the historical success rate; 、 as well as is the weight factor, which can be set as needed; is the adjusted score. When the adjusted score is closer to a certain threshold, it is used as the priority corresponding to the threshold. For example, if the calculated is 1.9, and 2 is the general level, which is closer to the general level, then it becomes the general level.

[0213] After making various adjustments, the adjusted governance policy action set needs to be evaluated using a multimodal strategy to generate a quality assessment model to obtain a score value.

[0214] In this embodiment, the multimodal strategy generates a quality assessment model as follows:

[0215] Q = ;

[0216] in, Indicates the number of bloodline nodes covered by the currently adjusted governance strategy action set; Indicates the number of dependencies violated using the currently adjusted governance policy action set; Indicates the improvement in prediction quality; Q is the score value; ω1, ω2, and ω3 represent weights, which are used to adjust the influence of different dimensions on the final score and can be set and adjusted according to actual needs; is the total dependency, which refers to the number of all possible dependency relationships in the bloodline graph; The maximum benefit refers to the maximum data quality improvement that can be achieved under the best conditions. It represents an ideal upper limit and can be set as needed.

[0217] Determine whether the score exceeds the threshold. If so, generate an executable governance policy package based on the adjusted governance policy action set. For example, only when Q>0.7 will the policy package (including the adjusted priority) be published to the execution engine.

[0218] This application innovatively integrates the three dimensions of system load, policy execution timeliness and historical execution success rate into the dynamic adjustment of policy priority. , ensuring that the priority of policies that have not been executed recently is gradually increased to avoid policy starvation.

[0219] pass This allows strategies with good historical performance to receive higher priority, forming positive feedback.

[0220] This application proposes for the first time a three-dimensional evaluation framework of coverage-conflict rate-benefit ratio to quantify the quality of strategy generation.

[0221] pass Items are encouraged to cover more key data entities.

[0222] pass Item, punishes strategies that violate the inherent dependencies of data.

[0223] In this embodiment, in the process of iteratively updating the strategy combination through the replica dynamics equation for the strategy set to find an equilibrium solution, thereby obtaining the Nash equilibrium strategy combination and the non-equilibrium strategy pair, the AI ​​data warehouse full-link lineage tracking method further includes:

[0224] In each process of iteratively updating the strategy combination through the replicator dynamics equation, the current strategy combination is evaluated using the stakeholder influence weighted satisfaction formula;

[0225] When the weighted satisfaction score changes by less than 1% for three consecutive iterations, the iteration is terminated early to obtain the Nash equilibrium strategy combination and weighted satisfaction score at the time of termination.

[0226] In this embodiment, a strategy combination is a vector that represents the strategies selected by all game players in the current state. The strategy combination may be expressed as:

[0227] S={producer: reject, consumer: delay execution};

[0228] In the initial strategy allocation phase, we randomly assign an initial strategy to each player, and the collection of these strategies constitutes the initial strategy combination. .For example:

[0229] The producer is "rejected" by the random allocation strategy.

[0230] Consumers are randomly assigned the strategy "delayed execution".

[0231] A combination of these two strategies ={producer: reject, consumer: delay execution} is the initial strategy combination.

[0232] Initial strategy combination It will be input into the strategy update iteration phase as the starting point of the iteration.

[0233] In each iteration, the strategy combination is updated according to the replicator dynamics equation to generate a new strategy combination .

[0234] The iterative process continues until the strategy combination converges to a Nash equilibrium.

[0235] In this embodiment, the replicator dynamics equation describes the rate of change of the strategy combination over time, i.e. ,in Represents the game player i At the moment t Select a strategy The proportion (or probability) of

[0236] The equation updates the strategy combination by considering the utility and average utility of the game players so that it converges to the equilibrium state.

[0237] In this embodiment, the weighted satisfaction formula of the stakeholder influence is as follows:

[0238]

[0239] in, The influence weight of the stakeholders (such as the producers mentioned above); is the current utility value (e.g., producer utility); is the minimum acceptable utility; To provide a weighted satisfaction score; The maximum acceptable utility.

[0240] In this embodiment, when When the change rate is less than 1% for three consecutive iterations, the iteration is terminated early.

[0241] This application combines influence weights with normalized utility to achieve fair coordination among unequal interests. The utility value is mapped to the range [0,1] to facilitate comparison across stakeholders.

[0242] The present application also provides an AI data warehouse full-link lineage tracking device, which includes a data acquisition module, a structured log data acquisition module, a knowledge graph acquisition module, a lineage metadata acquisition module, a lineage cognitive graph acquisition module, a governance strategy vector generation module, and a governance data acquisition module, wherein:

[0243] The data acquisition module is used to obtain data operation logs and business documents;

[0244] The structured log data acquisition module is used to obtain structured log data with business terms and entity tags;

[0245] The knowledge graph acquisition module is used to establish a knowledge graph containing a mapping relationship between business concepts and data entities based on the structured log data and business documents;

[0246] The lineage metadata acquisition module is used to generate lineage metadata including business semantics based on the knowledge graph and data job execution log;

[0247] The blood relationship cognitive map acquisition module is used to construct a blood relationship cognitive map based on the blood relationship metadata, data operation logs and business documents;

[0248] The governance strategy vector generation module is used to generate a governance strategy vector based on the lineage cognitive map, system real-time indicators and business SLO and send it to the execution engine, so that the execution engine processes it according to the governance strategy vector;

[0249] The governance data acquisition module is used to obtain governance policy execution results, user feedback information on governance policy execution results, and system logs generated during the governance process.

[0250] The AI ​​data warehouse full-link lineage tracking method proposed in this application breaks down the barriers between technology and business through full-link data fusion technology. It captures the complex semantics and causal relationships behind data operations through semantically enhanced metadata construction and dynamic lineage cognitive map generation technology. It also achieves intelligent generation and optimization of governance strategies through core technologies such as multi-level causal reasoning and intelligent decision-making closed loops. These technical solutions collectively address the fundamental limitations of traditional methods in terms of semantic loss, dynamic adaptation, intelligent decision-making, and causal reasoning, providing a new solution for data governance.

[0251] Furthermore, this application achieves optimal decision-making under multi-objective trade-offs through game theory models and conflict resolution techniques; improves decision-making efficiency and flexibility through preset templates and dynamic strategy generation techniques; and ensures fairness and rationality in the decision-making process through stakeholder influence-weighted satisfaction techniques. The integration of these technical solutions gives this invention significant technological innovation and practical application value in the field of data governance.

[0252] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made based on the present invention. Therefore, such modifications and improvements, which do not depart from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. An AI data warehouse full-link lineage tracing method, characterized by: The AI ​​data warehouse full-link lineage tracing method includes: Obtain data operation logs and business documents; Get structured log data with business terms and entity tags; Establishing a knowledge graph containing mapping relationships between business concepts and data entities based on the structured log data and business documents; Generate lineage metadata including business semantics based on the knowledge graph and data job execution log; Constructing a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents; Generate a governance strategy vector based on the lineage cognitive map, system real-time indicators, and business SLOs and send it to the execution engine, so that the execution engine processes it according to the governance strategy vector; Obtain governance policy execution results, user feedback on governance policy execution results, and system logs generated during the governance process; The acquisition of structured log data with business terms and entity tags includes: Generate a structured log token stream based on the data operation log; generating a multimodal fusion semantic vector according to the structured log token stream; Generating a spatiotemporal graph structure according to the multimodal fusion semantic vector; generating an implicit relation embedding vector according to the spatiotemporal graph structure; Generate spatiotemporal enhanced node representations through implicit relation embedding vectors; Strengthening node representation according to the spatiotemporal enhanced node representation generation rule; Strengthen the node representation according to the rules to generate a weighted recalibrated semantic vector; Recalibrating the semantic vector according to the weight to generate a robustness enhanced semantic vector; Generating a probabilistic entity annotation sequence according to the robustness enhanced semantic vector; Generating an optimized entity annotation sequence according to the probabilistic entity annotation sequence as structured log data with business terms and entity tags; The step of establishing a knowledge graph including mapping relationships between business concepts and data entities based on the structured log data and business documents includes: Extracting data from the structured log data and business documents to obtain a standardized metadata set; Dynamically inject semantics into the standardized metadata set to obtain semantically enhanced metadata vectors; constructing an initial cognitive graph based on the semantically enhanced metadata vector; The constructing of a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents includes: According to the original blood relationship set in the blood relationship metadata, data operation log and business document, the original blood relationship set includes operation type, timestamp and confidence score; generating an enhanced blood relationship set according to the original blood relationship set, wherein the enhanced blood relationship set includes a business semantic label, an impact range, and a causal strength; A blood relationship cognitive map is generated according to the enhanced blood relationship set.

2. The AI ​​data warehouse full-link lineage tracing method according to claim 1 is characterized in that: The step of inputting the bloodline cognitive map, the system real-time indicators, and the business SLO into the governance model to generate a governance strategy vector and sending it to the execution engine, so that the execution engine processes the governance strategy vector, includes: Use graph attention network to encode the bloodline cognitive graph to obtain graph neural network encoding vector; Dynamically adjust the weights according to the graph neural network encoding vector to obtain a weight-adjusted graph; Obtain the governance strategy action set based on the weight-adjusted graph; Generate an executable governance policy package according to the governance policy action set, wherein the executable governance policy package includes an action type, an object, a priority, and specific operations; Conflict resolution is performed according to the executable governance policy package, thereby obtaining a conflict-resolved policy package and sending it to the execution engine.

3. The AI ​​data warehouse full-link lineage tracing method according to claim 2 is characterized in that: The acquisition of the governance strategy action set according to the weight-adjusted graph includes: According to the weight-adjusted graph, use breadth-first search to enumerate all causal paths with length ≤ 3, so as to obtain each strength path to be calculated; Performing strength calculation on each strength path to be calculated, thereby obtaining the causal strength of each strength path to be calculated; The graph is input into the graph convolutional network, which aggregates neighbor node information through multi-layer convolution operations to generate potential path representations; Based on the potential path representation, the indirect influence of nodes is calculated through the message passing mechanism; each node transmits its influence to its neighbors to obtain the indirect causal reasoning results; Perform counterfactual interventions on key nodes based on the weight-adjusted graph, simulate the changes in target nodes after the intervention, and use SHAP values ​​to calculate the marginal contribution of each feature to obtain multi-level causal reasoning results; Dynamic weight adjustment and fusion are performed according to the multi-level causal reasoning results to obtain the hierarchical weight distribution results; Generate a fused causal strength map based on the multi-level causal reasoning results and the level weight distribution results; generating a global state representation according to the fused causal strength map; The global state representation is input into the deep Q network, the state is mapped to the action value through the multi-layer network, and the ε-greedy strategy is used for action exploration to obtain the action value evaluation result information; The optimal action is selected according to the action value evaluation result information based on the Q function value, and each of the optimal actions constitutes the governance strategy action set.

4. The AI ​​data warehouse full-link lineage tracing method according to claim 3 is characterized in that: The performing conflict resolution processing according to the executable governance policy package, thereby obtaining a conflict-resolved policy package and sending it to the execution engine includes: Identifying affected stakeholders based on the executable governance policy package, thereby obtaining game theory model parameters, wherein the game theory model parameters include game parties, policy sets, utility functions, and cost functions; Iteratively updating the strategy combination of the strategy set through the replicator dynamics equation to find the equilibrium solution, thereby obtaining the Nash equilibrium strategy combination and the non-equilibrium strategy pair; A compensation scheme is generated according to the non-equilibrium strategy pair and a compensation cost is calculated, wherein the Nash equilibrium strategy combination and the compensation scheme corresponding to each non-equilibrium strategy and the calculated compensation cost constitute a conflict-resolved strategy package.

5. The AI ​​data warehouse full-link lineage tracing method according to claim 4 is characterized in that: After obtaining the governance policy action set according to the weight-adjusted graph, generating a governance policy vector according to the lineage cognitive graph, the system real-time indicators, and the business SLO and sending it to the execution engine, so that the execution engine processes according to the governance policy vector, further includes: Acquire a preset policy template database, wherein the preset policy template database includes at least one predefined policy template and an action type corresponding to each predefined policy template; Acquire a corresponding predefined policy template from the preset policy template database according to the action type in the acquired governance policy action set; Obtain system real-time indicators and historical execution logs; Generate a parameterized policy template based on the predefined policy template, system real-time indicators, and historical execution logs; Use the context-aware priority dynamic adjustment formula to dynamically adjust the policy priority to obtain the adjusted governance policy action set; The adjusted governance policy action set is evaluated using a multimodal strategy to generate a quality assessment model to obtain a score value; Determine whether the score value exceeds the score threshold. If so, generate an executable governance policy package based on the adjusted governance policy action set.

6. The AI ​​data warehouse full-link lineage tracing method according to claim 5, characterized in that: In the process of iteratively updating the strategy combination through the replicator dynamics equation for the strategy set to find an equilibrium solution, thereby obtaining the Nash equilibrium strategy combination and the non-equilibrium strategy pair, the AI ​​data warehouse full-link lineage tracking method further includes: In each process of iteratively updating the strategy combination through the replicator dynamics equation, the current strategy combination is evaluated using the stakeholder influence weighted satisfaction formula; When the weighted satisfaction score changes by less than 1% for three consecutive iterations, the iteration is terminated early to obtain the Nash equilibrium strategy combination and weighted satisfaction score at the time of termination.

7. An AI data warehouse full-link lineage tracking device, characterized by: The AI ​​data warehouse full-link lineage tracking device includes: A data acquisition module, which is used to acquire data operation logs and business documents; A structured log data acquisition module, wherein the structured log data acquisition module is used to acquire structured log data with business terms and entity tags; A knowledge graph acquisition module, configured to establish a knowledge graph containing mapping relationships between business concepts and data entities based on the structured log data and business documents; A lineage metadata acquisition module, configured to generate lineage metadata including business semantics based on the knowledge graph and data job execution logs; A blood relationship cognitive map acquisition module, which is used to construct a blood relationship cognitive map based on the blood relationship metadata, data operation logs and business documents; A governance strategy vector generation module, which is used to generate a governance strategy vector based on the lineage cognitive map, system real-time indicators, and business SLOs and send it to the execution engine, so that the execution engine processes it according to the governance strategy vector; A governance data acquisition module, which is used to obtain governance policy execution results, user feedback information on governance policy execution results, and system logs generated during the governance process; The acquisition of structured log data with business terms and entity tags includes: Generate a structured log token stream based on the data operation log; generating a multimodal fusion semantic vector according to the structured log token stream; Generating a spatiotemporal graph structure according to the multimodal fusion semantic vector; generating an implicit relation embedding vector according to the spatiotemporal graph structure; Generate spatiotemporal enhanced node representations through implicit relation embedding vectors; Strengthening node representation according to the spatiotemporal enhanced node representation generation rule; Strengthen the node representation according to the rules to generate a weighted recalibrated semantic vector; Recalibrating the semantic vector according to the weight to generate a robustness enhanced semantic vector; Generating a probabilistic entity annotation sequence according to the robustness enhanced semantic vector; Generating an optimized entity annotation sequence according to the probabilistic entity annotation sequence as structured log data with business terms and entity tags; The step of establishing a knowledge graph including mapping relationships between business concepts and data entities based on the structured log data and business documents includes: Extracting data from the structured log data and business documents to obtain a standardized metadata set; Dynamically inject semantics into the standardized metadata set to obtain semantically enhanced metadata vectors; constructing an initial cognitive graph based on the semantically enhanced metadata vector; The constructing of a bloodline cognitive map based on the bloodline metadata, data operation logs, and business documents includes: According to the original blood relationship set in the blood relationship metadata, data operation log and business document, the original blood relationship set includes operation type, timestamp and confidence score; generating an enhanced blood relationship set according to the original blood relationship set, wherein the enhanced blood relationship set includes a business semantic label, an impact range, and a causal strength; A blood relationship cognitive map is generated according to the enhanced blood relationship set.

Images