A data lineage full-link traceability method and system based on multi-source heterogeneous metadata and pre-trained large models

Abstract

The present invention provides a method and system for full-link data lineage traceability based on multi-source heterogeneous metadata and pre-trained large models. The method includes the following steps: S1: collecting metadata and pre-processing the data; S2: using a data lineage relationship detection model to identify the data processed by S1 and outputting a lineage probability matrix between data entities; S3: filtering the lineage probability matrix based on a dynamic threshold to generate a lineage relationship map; S4: rendering the lineage relationship map obtained in S3 to visualize the data lineage relationship. The present invention solves the problems of rigid rules, low computational efficiency, and insufficient semantic understanding of traditional technologies through end-to-end training of deep learning large models, achieving high-precision and adaptive data lineage relationship modeling.

Description

Technical Field

[0001] The present invention relates to the technical field of data lineage relationship detection, and more specifically, to a method and system for full-link data lineage tracing based on multi-source heterogeneous metadata and pre-trained large models. Background Art

[0002] Traditional data lineage detection technologies currently have the following drawbacks: 1. Strong reliance on rules. Lineage analysis based on manual rules or static labels (such as Apache Atlas) struggles to adapt to dynamically changing metadata schemas. 2. Insufficient scalability: Existing methods (such as path queries in the graph database Neo4j) experience exponentially increased computational complexity when processing extremely large amounts of data. 3. Lack of semantic understanding: Traditional algorithms (such as similarity calculations based on string matching) are unable to capture the deep semantic connections between data. Summary of the Invention

[0003] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a full-link data lineage traceability method and system based on multi-source heterogeneous metadata and pre-trained large models, which is used to solve the problems of data lineage detection in the prior art such as strong dependence on rules, insufficient scalability and lack of semantic understanding. It realizes high-precision and adaptive data lineage relationship modeling and automatic identification and dynamic tracking of data lineage relationships, and is suitable for complex data governance scenarios in the fields of finance, medical care, Internet of Things, etc.

[0004] The technical solution adopted by the present invention is a data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models, the method comprising the following steps:

[0005] S1: Collect metadata and preprocess data;

[0006] S2: Use the data blood relationship detection model to identify the data processed by S1 and output the blood relationship probability matrix between data entities;

[0007] S3: Filter the blood relationship probability matrix based on the dynamic threshold to generate a blood relationship map;

[0008] S4: Render the blood relationship map obtained in S3 to visualize the data blood relationship.

[0009] In this application, the semantic feature capabilities of blood relationships are effectively improved by performing feature vector quantization processing on the collected data, and a data blood relationship detection model is set up to solve the problems of rigid rules and low computational efficiency of traditional technical methods. Through the deep combination of deep learning large models and big data metadata, intelligent detection of data blood relationships is realized. Compared with existing technologies, it has higher accuracy, stronger generalization capabilities and better computing efficiency, providing an innovative solution for data asset management.

[0010] Preferably, in step S1, the data preprocessing includes:

[0011] S11: Perform two-dimensional convolutional neural network encoding and position encoding on the data in sequence;

[0012] S12: Perform encoding mapping on the encoded data, and then perform feature splicing on the encoded mapped data.

[0013] In this application, the data is first encoded using a two-dimensional convolutional neural network to help the model capture the features of the data. Then, since the Transformer architecture itself does not have the ability to process the sequential information of the input data, it is necessary to add position encoding to the features to help the model understand the relative or absolute position of each part of the data. The features are then further encoded and mapped to help the model capture information from multiple angles, allowing data from different modalities to effectively interact with each other and capture cross-modal information. Finally, feature splicing is performed to enable the fusion of multiple features in the same model to capture more complex feature interactions.

[0014] Preferably, step S2 also includes model construction and training: the data blood relationship detection model is a multimodal Transformer model, which takes the data after feature splicing in S1 as the input vector, and then calculates the attention weight matrix through the self-attention mechanism to capture cross-field dependencies, and finally passes through the blood relationship prediction head and the fully connected layer to output the blood relationship probability matrix.

[0015] In this application, multimodal feature fusion is performed, and features of different modalities are combined as input vectors and input into the model for training, thereby effectively improving the semantic representation ability of blood relationships. The self-attention mechanism enables the model to capture rich contextual information in the sequence, and the fully connected layer is used to enhance the expression ability of the model, and finally outputs the blood relationship probability matrix.

[0016] Preferably, the calculation formula of the attention weight matrix is:

[0017]

[0018] in, represents the attention weight matrix; represents the query vector matrix; represents the transpose of the key vector matrix; Indicates the dimension of the key vector.

[0019] The weight matrix calculated according to the above formula enables the model to establish dependencies between different parts in the input sequence and assign different weights to each part, thereby enabling the model to capture rich contextual information in the sequence.

[0020] Preferably, step S2 also includes distributed training of the model, sharding and storing the model parameters, and using asynchronous updates to train the model, while using a contrastive learning loss function to optimize the model and a graph pruning algorithm to optimize reasoning efficiency.

[0021] In this application, distributed training is used to accelerate the training process, improve computing resource utilization and system scalability. The model's representational capabilities are optimized by comparing learning loss functions, making it particularly suitable for unsupervised learning tasks and improving model accuracy and robustness. Furthermore, a graph pruning algorithm is used to optimize inference efficiency, reducing computing resource consumption and making it suitable for real-time inference tasks.

[0022] Preferably, the contrastive learning loss function formula is:

[0023]

[0024] in, represents a positive sample pair; represents a negative sample pair.

[0025] Preferably, in step S3, it also includes updating model parameters through an online learning mechanism and updating the model through incremental training to adapt to metadata pattern drift.

[0026] In this application, the model can be continuously optimized through an online learning mechanism, so that the model can dynamically adapt to metadata schema drift and adapt to changes in the data environment.

[0027] On the other hand, the present application also provides a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models, the system comprising:

[0028] Metadata collection and processing module, used to collect metadata in real time and pre-process the data to generate data sets;

[0029] The model service module is used to deploy the data kinship detection model as an API interface, perform real-time kinship query and batch prediction by calling the API interface, and generate a kinship graph; the data kinship detection model is a multimodal Transformer model;

[0030] The visualization module is used to visualize the blood relationship map, facilitating interactive exploration and traceability analysis.

[0031] This application also provides a data lineage relationship detection system, in which different modules are set up to realize real-time and batch detection of data lineage relationships. It can not only effectively store and manage the lineage relationships of data streams, but also visualize the lineage relationships, allowing users to analyze data dependencies more intuitively and efficiently, and solve the problems of traditional data lineage detection technology such as strong rule dependence, insufficient scalability, and lack of semantic understanding.

[0032] Preferably, the metadata collection and processing module includes:

[0033] Data source access unit: accesses various types of data sources through multi-adapter components, and processes and transmits data from the data source through agent design components;

[0034] Data processing pipeline unit: Use the Kafka cluster component to clean the data, and then use the storage optimization component to optimize the storage of the cleaned data;

[0035] Metadata governance unit: stores the processed data in the metadata warehouse and performs data lineage tracking and exception monitoring.

[0036] In this module, by utilizing multi-source adapters and architecture, the system can provide a unified interface according to different data source types, and perform corresponding adaptation, conversion, feature vectorization and other processing to ensure the extraction and integration of information from various data sources, and achieve efficient and reliable data transmission. In addition, a data processing pipeline unit is set up to further optimize the data, effectively improving the high availability, scalability and reliability of the data. At the same time, it can also track and monitor the data to ensure the quality of the data.

[0037] Preferably, the Agent design component includes: first, using a deployment component to encapsulate the Agent into a lightweight Docker container, then using a data extraction component to extract data using an incremental or full pull strategy to reduce unnecessary data loading, then using a format conversion and compression component to convert the format and compress the extracted data to optimize data transmission efficiency, and finally using a feature engineering component to encode the processed data and perform feature splicing.

[0038] Therefore, in this application, Agent design components are used to perform incremental extraction, full extraction, format conversion and data compression, effectively ensuring efficient and reliable data transmission.

[0039] Preferably, in the metadata governance unit, data lineage tracking and anomaly monitoring include designing a data lineage tracking mechanism and clearly marking ETL transformation logic to address the source and destination of data at each stage. Prometheus and Grafana are then used to form a monitoring component to monitor the data. This ensures the effective management and use of data assets and improves data quality and traceability.

[0040] Preferably, the model service module includes:

[0041] Model hosting unit: Use the model repository to store the multimodal Transformer model and use the TensorFlowServing cluster to deploy the model as an API interface;

[0042] API Gateway Unit: performs traffic diversion, authentication, and traffic management for API interfaces, and performs internal and external interface protocol conversion;

[0043] Monitoring and debugging unit: monitors model performance indicators and performs log analysis to generate prediction result attribution analysis reports.

[0044] In order to ensure that the data lineage relationship detection model can be called quickly and stably, a model hosting unit is set up in this system to manage the model and convert it into an API interface for external calls. At the same time, an API gateway unit is set up to manage the interface and a monitoring and control unit is set up to monitor the model performance, which effectively improves the certainty of data lineage relationship model detection.

[0045] Preferably, the API gateway unit includes:

[0046] Weighted traffic diversion component: This diverts traffic based on the weight of the HTTP request path and model version, enabling different versions of the model to process requests in parallel.

[0047] Authentication component: uses JWT Token verification + RBAC roles for permission control to increase system security;

[0048] Traffic management component: Traffic management is performed using circuit breaking, current limiting, and request caching to reduce system pressure.

[0049] Therefore, by utilizing the API network management unit to divert, authenticate, and manage the traffic of the API interface, the security of the model call is ensured and the pressure on the system can be reduced.

[0050] Preferably, the visualization module includes:

[0051] Bloodline relationship graph construction unit: Node relationship data is stored through Neo4j or JanusGraph and transmitted to the front-end through an interface. The front-end framework renders the bloodline relationship graph based on the received data and uses hooks to monitor metadata changes to automatically trigger graph updates.

[0052] Interactive Analysis Unit: This unit uses the front-end framework to draw paths, allowing users to interactively query the relationship between different data nodes. It also enables reverse querying of downstream models and reports affected by field changes. It also uses different colors or layout styles to compare graph snapshots from different periods, helping users identify differences in architectural evolution.

[0053] Integration extension unit: Displays data quality indicators in the graph, controls visualization content based on user permissions, and supports graph export.

[0054] To more intuitively visualize data lineage relationships, this system not only visualizes model detection results but also uses interactive analysis units to allow users to view data lineage relationships across different nodes, enabling more intuitive and efficient analysis of data dependencies. Furthermore, operational permissions are set within this system to control the visualization content that users can view, ensuring data isolation and security.

[0055] In addition, the present application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the program, it implements a data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models as described above, or a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models as described above.

[0056] The present application also provides a non-transitory computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models or the above-mentioned data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models.

[0057] The present application also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models as described above, or the data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models as described above.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] This invention realizes the intelligent detection of data lineage through the deep combination of deep learning large models and big data metadata. Compared with existing technologies, it has higher accuracy, stronger generalization ability and better computing efficiency, providing an innovative solution for data asset management. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Flow chart of the method of the present invention.

[0061] Figure 2 This is a model structure diagram of the present invention.

[0062] Figure 3 This is a system structure diagram of the present invention.

[0063] Figure 4 A schematic diagram of the physical structure of an electronic device. DETAILED DESCRIPTION

[0064] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present invention. To better illustrate the following embodiments, some components in the accompanying drawings may be omitted, enlarged, or reduced in size, and do not represent actual product dimensions. Those skilled in the art will appreciate that some well-known structures and their descriptions may be omitted from the accompanying drawings.

[0065] Example 1

[0066] like Figure 1 As shown, Figure 1 This is a flowchart of the method of the present invention. This embodiment provides a data lineage full-link traceability method and system based on multi-source heterogeneous metadata and pre-trained large models. The method includes the following steps:

[0067] Step S1: Collect metadata and pre-process the data;

[0068] Specifically, if Figure 2 As shown, Figure 2 It is the model structure diagram of the present invention, from Figure 2 It can be seen that the metadata collected in this embodiment include: structured data L, text description data V and log tracking data A, and then a two-dimensional convolutional neural network is used to encode these data. The encoded data is represented by C, and then a position code P is added to each data type to help the model understand the relative or absolute position of each part of the data. Then, multi-head self-attention is entered for encoding mapping. For structured data L, it includes P→L encoding mapping and C→L encoding mapping; for text description data V, it includes P→V encoding mapping and C→V encoding mapping; for log tracking data A, it includes P→A encoding mapping and C→A encoding mapping; then feature splicing is performed separately and input into the Transformer model for further processing, so that data of different modalities can be effectively combined under the same framework, thereby improving the performance of the model in multimodal tasks.

[0069] In this embodiment, in addition to the above data, metadata can also be collected from database logs, ETL tasks, API call chains, etc., including table structure, field type, operation timestamp, user tags, etc. The collected data is then vectorized, including structural features, time series features, and graph features.

[0070] For the structured features: the field type and data magnitude are mapped into vectors by performing One-Hot encoding on them, such as for the field type Map it into a structured feature vector .

[0071] For the timing characteristics, a sliding window is used to extract the operating frequency and quantize the frequency characteristics; wherein the sliding window formula is: , Indicates the number of operations in the sliding window; Indicates the length of the sliding window; Indicates the operation frequency in the sliding window.

[0072] For the graph features, an adjacency matrix is ​​constructed based on data dependencies. , and generate node vectors through graph embedding algorithms (such as Node2Vec) .

[0073] Therefore, in this embodiment, by encoding and vectorizing different modal features, the machine learning model can understand and use different types of features, thereby improving the model's prediction and reasoning capabilities. By uniformly representing various features as vectors, multiple features can be integrated in the same model to capture more complex feature interactions. In addition, the data after vectorization processing is usually lower in dimension and more structured, which helps to improve computing efficiency, especially when processing large-scale data, and can accelerate the training and reasoning process.

[0074] Step S2: Use the data blood relationship detection model to identify the data processed by S1 and output the blood relationship probability matrix between data entities;

[0075] Specifically, in this embodiment, step S2 also includes the construction and training of the model: Figure 2 As shown, the data kinship detection model is a multimodal Transformer model, which takes the data after feature splicing in S1 as the input vector , and then the attention weight matrix is ​​calculated by the self-attention mechanism to capture the cross-field dependencies. The calculation formula is ,in, represents the attention weight matrix; represents the query vector matrix; represents the transpose of the key vector matrix; Represents the dimension of the key vector. Finally, it passes through the blood relationship prediction head and the fully connected layer to output the blood relationship probability matrix ,in, Represents a data entity arrive The probability of blood relationship.

[0076] By performing multimodal feature fusion, combining structured, temporal and graph features and concatenating the vectorized features as input vectors into the model for training, the semantic representation ability of blood relationships is effectively improved. The self-attention mechanism enables the model to capture rich contextual information in the sequence, and the fully connected layer is used to enhance the model's expressive power, ultimately outputting a blood relationship probability matrix.

[0077] Further preferably, step S2 further includes distributed training of the model. In this embodiment, based on the Parameter Server architecture, the model parameters are stored in shards and updated using asynchronous SGD. The update formula is: ,in, Represents the parameters that the model needs to learn at the tth iteration; Indicates the parameters that need to be learned by the updated model at the t+1th iteration; represents the learning rate; Represents the gradient of the loss function with respect to the model parameters. Thus, by calculating the gradient of the loss function and adjusting the learning rate, the performance of the model is optimized.

[0078] In addition, a contrastive learning loss function is used to optimize the model and a graph pruning algorithm is used to optimize inference efficiency. The formula of the contrastive learning loss function is: ,in, Indicates a positive sample pair (there is a blood relationship); represents a negative sample pair (no blood relationship); and in the embodiment, the positive-negative sample ratio is set to 1:3.

[0079] In this embodiment, a distributed parameter server is used to accelerate the training process, improving computing resource utilization and system scalability. The model's representational capabilities are optimized by contrasting learning loss functions, making it particularly suitable for unsupervised learning tasks and improving model accuracy and robustness. Furthermore, a graph pruning algorithm is used to optimize inference efficiency, reducing computing resource consumption and making it suitable for real-time inference tasks.

[0080] Step S3: filtering the blood relationship probability matrix based on the dynamic threshold to generate a blood relationship map;

[0081] Preferably, in this embodiment, the dynamic threshold is set to , thereby filtering out the blood relationship matrix that does not meet the conditions and generating the final blood relationship graph.

[0082] Further preferably, in the step S3, it also includes updating the model parameters through an online learning mechanism, and updating the model through incremental training (such as Elastic Weight Consolidation) to adapt to metadata pattern drift.

[0083] In this application, the model can be continuously optimized through an online learning mechanism, so that the model can dynamically adapt to metadata schema drift and adapt to changes in the data environment.

[0084] Step S4: Render the blood relationship graph based on D3.js to visualize the data blood relationship.

[0085] In this application, the semantic feature capabilities of blood relationships are effectively improved by performing feature vector quantization processing on the collected data, and a data blood relationship detection model is set up to solve the problems of rigid rules and low computational efficiency of traditional technical methods. Through the deep combination of deep learning large models and big data metadata, intelligent detection of data blood relationships is realized. Compared with existing technologies, it has higher accuracy, stronger generalization capabilities and better computing efficiency, providing an innovative solution for data asset management.

[0086] Example 2

[0087] Based on the detection method described in Example 1, Figure 3 As shown, Figure 3 This is a schematic diagram of the system structure of the present invention. This embodiment provides a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models. The system includes:

[0088] Metadata collection and processing module, used to collect metadata in real time and pre-process the data to generate data sets;

[0089] The model service module is used to deploy the data kinship detection model as an API interface, perform real-time kinship query and batch prediction by calling the API interface, and generate a kinship graph; the data kinship detection model is a multimodal Transformer model;

[0090] The visualization module is used to visualize the blood relationship map, facilitating interactive exploration and traceability analysis.

[0091] This application also provides a data lineage relationship detection system, in which different modules are set up to realize real-time and batch detection of data lineage relationships. It can not only effectively store and manage the lineage relationships of data streams, but also visualize the lineage relationships, allowing users to analyze data dependencies more intuitively and efficiently, and solve the problems of traditional data lineage detection technology such as strong rule dependence, insufficient scalability, and lack of semantic understanding.

[0092] Preferably, the metadata collection and processing module includes:

[0093] Data source access unit:

[0094] Connect to various data sources through multiple adapter components, including structured (Hive / MySQL), semi-structured (JSON / XML), and unstructured (log / text) data sources.

[0095] The original data is then processed and transmitted through the Agent design components for feature vectorization, including:

[0096] Deployment Components: Agents are encapsulated as lightweight Docker containers, ensuring rapid deployment, upgrades, and scalability across diverse environments. Each Agent instance runs a separate container, ensuring isolation and scalability. Containerized Agents can be scheduled and managed within a Kubernetes cluster. Kubernetes helps dynamically scale the number of Agent instances, supporting high availability and load balancing, allowing Agents to automatically scale in response to changes in traffic and data sources.

[0097] Functional modules: mainly include data extraction, format conversion, feature engineering and transmission guarantee, ensuring that data can flow from the source system to the target system accurately, reliably and efficiently.

[0098] Data extraction components: Utilize incremental / full pull strategies, such as Hive partition scanning and Kafka Consumer Group offset management. For Hive, partitioning can be leveraged to implement incremental pull. For example, scanning new partitions (or newly added data) can reduce unnecessary data loading. For Kafka, offset management within the Consumer Group is used to retrieve previously consumed messages, ensuring incremental pull.

[0099] Format conversion and compression: This includes schema mapping (such as Avro / Parquet serialization) and data compression (using Snappy or Zstd compression algorithms). Different data sources may use different schema formats. When extracting data, the agent performs schema mapping to convert the data from the source system's structure to a format understandable by the target system. Furthermore, data compression is performed to reduce bandwidth consumption and improve transmission efficiency.

[0100] Feature engineering component: Use a two-dimensional convolutional neural network to encode structured data L, text description data V, and log tracking data A. The encoded data is represented by C, and then position code P is added to each data type to help the model understand the relative or absolute position of each part of the data. Then enter the multi-head self-attention for encoding mapping. For structured data L, it includes P→L encoding mapping and C→L encoding mapping; for text description data V, it includes P→V encoding mapping and C→V encoding mapping; for log tracking data A, it includes P→A encoding mapping and C→A encoding mapping; then perform feature splicing separately and input it into the Transformer model for further processing, so that data of different modalities can be effectively combined under the same framework, thereby improving the performance of the model in multimodal tasks.

[0101] Transmission assurance components: These include data resuming, ACK confirmation mechanisms, and data sharding. Resuming data transmission ensures that after a failure, transmission can resume from the point of interruption, eliminating the need to start from the beginning. This helps improve system robustness and fault tolerance and avoids duplicate data transmission. The ACK confirmation mechanism ensures that data has been successfully received and processed by the target system, ensuring data consistency and reliability. When transmitting large amounts of data, data sharding can divide large data sets into multiple smaller segments and transmit them in parallel. This not only increases transmission speeds but also avoids performance bottlenecks caused by excessive data volume in a single data source or target system, thereby balancing system load and improving overall throughput.

[0102] Data processing pipeline unit: Use the Kafka cluster component to clean the data, and then use the storage optimization component to optimize the storage of the cleaned data;

[0103] Specifically, for a Kafka cluster, first design topics: divide topics by business domain, distributing data across multiple topics, such as metadata_raw and metadata_processed. This facilitates logical isolation and topic management, making it easier for consumers to sort and process specific data. Next, implement stream processing, integrating Kafka Streams or Flink for data cleansing (deduplication and null value processing) and field desensitization (regular expression matching). For storage optimization, first design a RowKey: reverse timestamp + hash prefix. This effectively avoids hotspots during reads, ensures even data storage, and improves performance. Next, implement secondary indexes, implement SQL-based queries using Phoenix, and finally implement a TTL strategy for hot and cold data tiering. Using hot storage and S3 cold backups, this strategy stores recently used data in HBase for fast access, while migrating inaccessible data to cold storage solutions like S3 to reduce storage costs while ensuring long-term data backup and compliance.

[0104] Metadata governance unit: stores the processed data in the metadata warehouse and performs data lineage tracking and exception monitoring.

[0105] Specifically, for the metadata warehouse, choose Apache Atlas or Amundsen as the metadata repository to centrally store table structures, field lineages, and data quality rules. For data lineage tracking, design a data lineage tracking mechanism that records the following link: Hive table → Kafka topic → HBase table. This comprehensive tracking helps business analysts and data engineers understand the data flow and its transformation process. It also clearly marks the ETL transformation logic to address the data source and destination at each stage, helping data users better understand the data processing process and business rules. For anomaly monitoring, use Prometheus and Grafana to form a monitoring system to monitor agent heartbeats, Kafka lag, and HBase Region load. Metadata governance improves the manageability and transparency of data assets while ensuring data quality and compliance.

[0106] In this module, by utilizing multi-source adapters and architecture, the system can provide a unified interface according to different data source types, and perform corresponding adaptation, conversion, feature vectorization and other processing to ensure the extraction and integration of information from various data sources, and achieve efficient and reliable data transmission. In addition, a data processing pipeline unit is set up to further optimize the data, effectively improving the high availability, scalability and reliability of the data. At the same time, it can also track and monitor the data to ensure the quality of the data.

[0107] Preferably, the model service module includes:

[0108] Model hosting unit: Use the model repository to store the multimodal Transformer model and use the TensorFlowServing cluster to deploy the model as an API interface;

[0109] Specifically, the model repository described in this unit uses industry-standard packaging formats such as SavedModel, PMML, and ONNX to ensure model interoperability and ease of integration. Git-LFS (Large File Storage) is also used to manage the version history of model files, making it easier to track model changes and implement version rollbacks, thereby ensuring model traceability and security. For the TensorFlow Serving cluster, model version management is included, which supports multiple versions in parallel, and hot model updates can be implemented through the gRPC interface, allowing models to be dynamically loaded or unloaded without restarting the service, which can reduce downtime and improve system availability. In addition, resources are isolated, and the CPU and memory quotas of each model are limited based on cgroups (control groups) to ensure the rational allocation of resources and prevent mutual interference between different models, thereby improving the stability and performance of the overall service.

[0110] API Gateway Unit: performs traffic diversion, authentication, and traffic management for API interfaces, and performs internal and external interface protocol conversion;

[0111] Specifically, in this embodiment, the API gateway unit utilizes a weighted traffic diversion component to divert traffic based on the HTTP Path request path ( / api / v1 / predict) and the weight of the model version. This allows requests from different model versions to be processed in parallel, effectively managing and controlling traffic when conducting A / B testing or gradually releasing new versions. Furthermore, the authentication component uses JWT token verification and RBAC roles for permission control, effectively enhancing system security. Furthermore, the traffic management component employs circuit breaking (Hystrix), current limiting (token bucket algorithm), and request caching (Redis) to manage traffic and reduce system pressure. Finally, the API gateway can also implement conversion between the client-side RESTful API and the backend's internal gRPC protocol, enhancing system flexibility. This allows clients to use a more user-friendly RESTful interface while internally using the more efficient gRPC for service calls. The API gateway also performs necessary input data preprocessing to ensure that the data meets the model's input requirements before the request reaches the prediction model.

[0112] Monitoring and debugging unit: monitors model performance indicators and performs log analysis to generate prediction result attribution analysis reports.

[0113] Specifically, the model service module also includes monitoring and debugging during model use, recording performance indicators such as QPS, Latency (P50 / P99), and GPU utilization. Elasticsearch, Logstash, and Kibana (ELK Stack) are then used for log collection and analysis. By associating TraceIDs, full-link tracking of requests is achieved. This allows for quick location of the specific link where the problem occurred when an anomaly occurs, improving troubleshooting efficiency. To ensure the interpretability of the model, the SHAP and LIME libraries are integrated to generate attribution analysis reports for prediction results, helping users understand how the model arrives at specific predictions, thereby enhancing model transparency and trust.

[0114] In order to ensure that the data lineage relationship detection model can be called quickly and stably, a model hosting unit is set up in this system to manage the model and convert it into an API interface for external calls. At the same time, an API gateway unit is set up to manage the interface and a monitoring and control unit is set up to monitor the model performance, which effectively improves the certainty of data lineage relationship model detection.

[0115] Preferably, the visualization module includes:

[0116] Bloodline relationship graph construction unit: Node relationship data is stored through Neo4j or JanusGraph and transmitted to the front-end through an interface. In this embodiment, the front-end framework used is D3.js. D3.js renders the bloodline relationship graph based on the received data and uses Hooks to monitor metadata changes to automatically trigger graph updates.

[0117] Specifically, this unit uses Neo4j or JanusGraph as the graph storage engine, supporting node relationship operations using Gremlin or Cypher query languages. The node types described in this embodiment include: data nodes, including Hive tables, Kafka topics, and HBase tables; processing nodes, including Spark jobs, Flink operators, and ETL scripts; and business nodes, including metric definitions (such as daily active users) and reports (such as BI dashboards). A hook mechanism monitors metadata change events, such as automatically triggering graph updates when executing Hive ALTER TABLE operations, to ensure the real-time and accuracy of graph data.

[0118] Interactive Analysis Unit: Using D3.js to draw paths, users can interactively query the relationship between different data nodes and reversely query the downstream models and reports affected by field changes. Different colors or layout styles can also be used to compare graph snapshots from different periods, helping users identify differences in architectural evolution.

[0119] Specifically, in this unit, the path is the entire chain of "core order table → Kafka order stream → feature engineering → model input." Users can click or hover the mouse to view the complete data flow from the core order table to the Kafka order stream, and then to feature engineering and model input, helping them understand the data flow process. It also supports reverse query functionality, allowing users to see how changes to a certain field will affect related downstream models and reports, allowing them to make corresponding adjustments or decisions. It also allows users to compare snapshots of lineage maps at different time points to identify differences in architectural evolution and help staff understand system changes.

[0120] Integration extension unit: Displays data quality indicators in the graph, controls visualization content based on user permissions, and supports graph export.

[0121] Specifically, in this embodiment, data quality indicators (such as null value rate, expired models, etc.) are directly visualized in the graph through D3.js. For low-quality node marking: for example, a table with a null value rate exceeding 30% can be marked with a warning color (such as red) on the node rendered by D3.js to attract user attention; for expired node marking: if a model has not been called for more than 6 months, D3.js can be used to add an expired label or other conspicuous prompt to these nodes.

[0122] Secondly, D3.js can also be integrated with backend permission control systems (such as LDAP / AD), allowing data to be filtered based on user roles during front-end rendering. For example, a developer can only see the nodes and relationships belonging to their business line. When the developer views the graph, the front-end dynamically loads data nodes and relationships related to that business line based on their permissions, ensuring data isolation and security.

[0123] Furthermore, in this embodiment, the kinship graph rendered with D3.js also supports generating documents in SVG or PDF format for easy export and archiving. SVG / PDF Export: After D3.js renders the graph, you can use its built-in API to export the graph content as SVG or PDF files. These files can then be embedded in Confluence or Jira for document management and collaboration.

[0124] To more intuitively visualize data lineage relationships, this system not only visualizes model detection results but also establishes interactive analysis units that allow users to view data lineage relationships across different nodes, enabling more intuitive and efficient analysis of data dependencies. Furthermore, operational permissions are set within this system to control the visual content that users can view, ensuring data isolation and security.

[0125] Example 3

[0126] Based on the data lineage full-link traceability method and system based on multi-source heterogeneous metadata and pre-trained large models proposed in Examples 1 and 2, this embodiment proposes data lineage tracing for financial transaction data, specifically as follows:

[0127] First, we use the metadata collection and processing module to collect and process data. We collect 100,000 metadata items from the MySQL transaction database, including table structures and ETL task logs. We then encode and vectorize these data. For structured features, we encode field types (amount, date) into 128-dimensional vectors. For time series features, we update the frequency based on hourly statistics to generate time series. For graph features, we construct a table dependency adjacency matrix and use GraphSAGE to generate embeddings.

[0128] Then, in the model service module, the data lineage relationship detection model is called to perform detection by inputting the new ETL task metadata, thereby outputting a lineage probability matrix, wherein the filtering threshold is set to 0.8 to generate a data lineage relationship map.

[0129] Finally, the graph is visualized through the visualization module to locate the data source of the transaction report (such as original order table → risk control intermediate table → report view).

[0130] In addition to being used in the field of financial transactions, the data lineage relationship detection method and system described in the present invention can also be used to trace medical data and applied to the same model, method, or system as mentioned above. The specific operations will not be repeated here. Through cross-domain verification, the prediction accuracy of the lineage detection capability of this method and system in medical imaging metadata can reach 87.3%.

[0131] Example 4

[0132] Based on the above embodiment, this embodiment further provides an electronic device, such as Figure 4 As shown, Figure 4 An example of a physical structure diagram of an electronic device is provided, which may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models described in Example 1 or the data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models described in Example 2.

[0133] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0134] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models described in Example 1 or the data lineage full-link tracing system based on multi-source heterogeneous metadata and pre-trained large models described in Example 2 provided by the above methods.

[0135] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models described in Example 1 or the data lineage full-link tracing system based on multi-source heterogeneous metadata and pre-trained large models described in Example 2 provided by the above methods.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0137] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0138] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models, characterized by: The method comprises the following steps: S1: Collect metadata and preprocess data; S2: Use the data blood relationship detection model to identify the data processed by S1 and output the blood relationship probability matrix between data entities; S3: Filter the blood relationship probability matrix based on the dynamic threshold to generate a blood relationship map; S4: Render the blood relationship map obtained in S3 to visualize the blood relationship of the data; In the step S3, the model parameters are updated through an online learning mechanism, and the model is updated through incremental training to adapt to metadata pattern drift; In step S1, the data preprocessing includes: S11: Perform two-dimensional convolutional neural network encoding and position encoding on the data in sequence; S12: performing encoding mapping on the encoded data, and then performing feature splicing on the encoded mapped data; The encoding specifically includes: performing feature vectorization processing on the collected data, including vectorization processing of structural features, time series features and graph features; wherein, For the structured features: the field type and data level are mapped into vectors by One-Hot encoding. Specifically, for the field type Map it into a structured feature vector V T ; For the timing characteristics, a sliding window is used to extract the operating frequency and quantize the frequency characteristics; wherein the sliding window formula is: , Indicates the number of operations in the sliding window; Indicates the length of the sliding window; Represents the operation frequency in the sliding window; For the graph features, an adjacency matrix is ​​constructed based on data dependencies. , and generate node vectors through graph embedding algorithm.

2. According to claim 1, a data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models is characterized by: Step S2 also includes model construction and training: the data blood relationship detection model is a multimodal Transformer model, which uses the data after feature splicing in S1 as the input vector, and then calculates the attention weight matrix through the self-attention mechanism to capture cross-field dependencies. Finally, it passes through the blood relationship prediction head and the fully connected layer to output the blood relationship probability matrix.

3. The method for full-link data lineage tracing based on multi-source heterogeneous metadata and pre-trained large models according to claim 2 is characterized in that: The calculation formula of the attention weight matrix is: in, represents the attention weight matrix; represents the query vector matrix; represents the transpose of the key vector matrix; Indicates the dimension of the key vector.

4. The method for full-link data lineage tracing based on multi-source heterogeneous metadata and pre-trained large models according to claim 2 is characterized in that: Step S2 also includes distributed training of the model, sharding and storing the model parameters, and using asynchronous updates to train the model. At the same time, a contrastive learning loss function is used to optimize the model, and a graph pruning algorithm is used to optimize the inference efficiency.

5. According to claim 4, a data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models is characterized by: The contrastive learning loss function formula is: in, represents a positive sample pair; represents a negative sample pair.

6. A data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models, characterized by: A data lineage full-link traceability method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 5 is applied, wherein the system comprises: Metadata collection and processing module, used to collect metadata in real time and pre-process the data to generate data sets; The model service module is used to deploy the data blood relationship detection model as an API interface, perform real-time blood relationship query and batch prediction by calling the API interface, and generate a blood relationship map; The visualization module is used to visualize the blood relationship map, facilitating interactive exploration and traceability analysis; The metadata collection and processing module includes: Data source access unit: accesses various types of data sources through multi-adapter components, and processes and transmits data from the data source through agent design components; Data processing pipeline unit: Use the Kafka cluster component to clean the data, and then use the storage optimization component to optimize the storage of the cleaned data; Metadata governance unit: stores processed data in the metadata warehouse and performs data lineage tracking and exception monitoring; The Agent design components include a transmission guarantee component, which includes data resuming, ACK confirmation mechanism, and data sharding. The resuming ensures that after a failure, the transmission can continue from the interruption point without having to start from the beginning. The ACK confirmation mechanism ensures that the data has been successfully received and processed by the target system. When transmitting large-scale data, the data sharding divides the large data set into multiple small fragments and transmits them in parallel.

7. The data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models according to claim 6 is characterized by: The Agent design components include: first, using a deployment component to encapsulate the Agent into a lightweight Docker container; then using a data extraction component to extract data using an incremental or full pull strategy to reduce unnecessary data loading; then using a format conversion and compression component to convert the format and compress the extracted data to optimize data transmission efficiency; finally, using a feature engineering component to encode the processed data and perform feature splicing.

8. The data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models according to claim 6 is characterized by: In the metadata governance unit, the data lineage tracking and anomaly monitoring processing include: designing a data lineage tracking mechanism and clearly marking the ETL transformation logic to handle the data source and destination at each stage, and then using Prometheus and Grafana to form a monitoring component to monitor the data.

9. The data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models according to claim 6 is characterized by: The model service module includes: Model hosting unit: Use the model repository to store the multimodal Transformer model and use the TensorFlowServing cluster to deploy the model as an API interface; API Gateway Unit: performs traffic diversion, authentication, and traffic management for API interfaces, and performs internal and external interface protocol conversion; Monitoring and debugging unit: monitors model performance indicators and performs log analysis to generate prediction result attribution analysis reports.

10. A data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models according to claim 9, characterized in that: The API gateway unit includes: Weighted traffic diversion component: This component diverts traffic based on the weight of the HTTP request path and model version, enabling different versions of the model to process requests in parallel. Authentication component: uses JWT Token verification + RBAC roles for permission control to increase system security; Traffic management component: Traffic management is performed using circuit breaking, current limiting, and request caching to reduce system pressure.

11. The data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models according to claim 6 is characterized by: The visualization module includes: Bloodline relationship graph construction unit: Node relationship data is stored through Neo4j or JanusGraph and transmitted to the front-end through an interface. The front-end framework renders the bloodline relationship graph based on the received data and uses hooks to monitor metadata changes to automatically trigger graph updates. Interactive Analysis Unit: This unit uses the front-end framework to draw paths, allowing users to interactively query the relationship between different data nodes and reversely query the downstream models and reports affected by field changes. It also uses different colors or layout styles to compare graph snapshots from different periods, helping users identify differences in architectural evolution. Integration extension unit: Displays data quality indicators in the graph, controls visualization content based on user permissions, and supports graph export.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements a data lineage full-link tracing method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 5.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements a data lineage full-link tracing method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 5.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by the processor, it implements a data lineage full-link tracing method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 5.

Images