Data consanguinity full-link traceability method and system based on multi-source heterogeneous metadata and pre-training large model

Abstract

The invention provides a data consanguinity full-link traceability method and system based on multi-source heterogeneous metadata and a pre-training large model. The method comprises the following steps: S1, collecting metadata and preprocessing the data; s2, identifying the data processed in the S1 by using a data consanguinity detection model, and outputting a consanguinity probability matrix between data entities; s3, filtering the blood relationship probability matrix based on a dynamic threshold value to generate a blood relationship map; and S4, rendering the blood relationship graph obtained in the S3 to visualize the data blood relationship. By means of metadata multi-dimensional feature extraction and end-to-end training of a deep learning large model, the problems that a traditional technology is rigid in rule, low in calculation efficiency, insufficient in semantic understanding and the like are solved, and high-precision and self-adaptive data blood relationship modeling is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data lineage detection, and more specifically, to a data lineage full-link tracing method and system based on multi-source heterogeneous metadata and pre-trained large models. Background Art

[0002] In the existing technology, the traditional data lineage detection technology still has the following defects: 1. It is highly dependent on rules. Lineage analysis based on manual rules or static labels (such as Apache Atlas) is difficult to adapt to dynamically changing metadata patterns. 2. Insufficient scalability: The computational complexity of existing methods (such as path query in the graph database Neo4j) increases exponentially when processing ultra-large-scale data. 3. Lack of semantic understanding: Traditional algorithms (such as similarity calculation based on string matching) cannot capture the deep semantic associations between data. Summary of the invention

[0003] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a full-link data lineage traceability method and system based on multi-source heterogeneous metadata and pre-trained large models, which is used to solve the problems of strong rule dependence, insufficient scalability and lack of semantic understanding in the prior art of data lineage detection, and realizes high-precision and adaptive data lineage relationship modeling and automatic identification and dynamic tracking of data lineage relationships, which is suitable for complex data governance scenarios in the fields of finance, medical care, Internet of Things, etc.

[0004] The technical solution adopted by the present invention is a data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models, and the method comprises the following steps: S1: Collect metadata and preprocess data; S2: Use the data blood relationship detection model to identify the data processed by S1 and output the blood relationship probability matrix between data entities; S3: Filter the blood relationship probability matrix based on the dynamic threshold to generate a blood relationship map; S4: Render the blood relationship map obtained in S3 to visualize the data blood relationship.

[0005] In this application, the semantic feature capabilities of blood relationships are effectively improved by performing feature vector quantization processing on the collected data, and a data blood relationship detection model is set up to solve the problems of rigid rules and low computational efficiency of traditional technical methods. Through the deep combination of deep learning big models and big data metadata, intelligent detection of data blood relationships is realized. Compared with existing technologies, it has higher accuracy, stronger generalization ability and better computing efficiency, providing an innovative solution for data asset management.

[0006] Preferably, in step S1, the data preprocessing includes: S11: perform two-dimensional convolutional neural network encoding and position encoding on the data in sequence; S12: Perform encoding mapping on the encoded data, and then perform feature splicing on the encoded mapped data.

[0007] In this application, we first encode the data using a two-dimensional convolutional neural network to help the model capture the features of the data. Then, since the Transformer architecture itself does not have the ability to process the order information of the input data, it is necessary to add position encoding to the features to help the model understand the relative or absolute position of each part of the data. The features are then further encoded and mapped to help the model capture information from multiple angles, allowing data from different modalities to effectively interact with each other and capture cross-modal information. Finally, feature concatenation is performed to allow multiple features to be fused in the same model to capture more complex feature interactions. Preferably, step S2 also includes model construction and training: the data blood relationship detection model is a multimodal Transformer model, which uses the data after feature splicing in S1 as the input vector, and then calculates the attention weight matrix through the self-attention mechanism to capture cross-field dependencies, and finally passes through the blood relationship prediction head and the fully connected layer to output the blood relationship probability matrix.

[0008] In this application, multimodal feature fusion is performed to combine features of different modalities as input vectors and input them into the model for training, thereby effectively improving the semantic representation ability of blood relationships. The self-attention mechanism enables the model to capture rich contextual information in the sequence, and the fully connected layer is used to enhance the expression ability of the model, and finally outputs the blood relationship probability matrix.

[0009] Preferably, the calculation formula of the attention weight matrix is:

[0010] in, represents the attention weight matrix; represents the query vector matrix; represents the transpose of the key vector matrix; Indicates the dimension of the key vector.

[0011] The weight matrix calculated according to the above formula enables the model to establish dependencies between different parts in the input sequence and assign different weights to each part, so that the model can capture rich contextual information in the sequence.

[0012] Preferably, step S2 also includes distributed training of the model, storing the model parameters in shards, and using asynchronous updates to train the model, while using contrastive learning loss function to optimize the model and using a graph pruning algorithm to optimize reasoning efficiency.

[0013] In this application, the distributed training method is used to accelerate the training process, improve the utilization efficiency of computing resources and the scalability of the system, and optimize the representation ability of the model by comparing the learning loss function, which is particularly suitable for unsupervised learning tasks and improves the accuracy and robustness of the model. In addition, the graph pruning algorithm is used to optimize the reasoning efficiency, reduce the consumption of computing resources, and is suitable for real-time reasoning tasks.

[0014] Preferably, the contrastive learning loss function formula is:

[0015] in, represents a positive sample pair; represents a negative sample pair.

[0016] Preferably, in step S3, it also includes updating model parameters through an online learning mechanism and updating the model through incremental training to adapt to metadata model drift.

[0017] In this application, the model can be continuously optimized through an online learning mechanism, so that the model can dynamically adapt to metadata schema drift and adapt to changes in the data environment.

[0018] On the other hand, the present application also provides a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models, the system comprising: The metadata collection and processing module is used to collect metadata in real time and pre-process the data to generate a data set; The model service module is used to deploy the data blood relationship detection model as an API interface, perform real-time blood relationship query and batch prediction by calling the API interface, and generate a blood relationship map; the data blood relationship detection model is a multimodal Transformer model; The visualization module is used to visualize the blood relationship map to facilitate interactive exploration and traceability analysis.

[0019] This application also provides a data lineage detection system, in which different modules are set up to realize real-time and batch detection of data lineage. It can not only effectively store and manage the lineage of data streams, but also visualize the lineage, allowing users to analyze data dependencies more intuitively and efficiently, and solve the problems of traditional data lineage detection technology, such as strong rule dependence, insufficient scalability, and lack of semantic understanding.

[0020] Preferably, the metadata collection and processing module includes: Data source access unit: accesses various types of data sources through multi-adapter components, and performs feature processing and transmission of data source data through Agent design components; Data processing pipeline unit: Use the Kafka cluster component to clean the data, and then use the storage optimization component to perform storage optimization processing on the cleaned data; Metadata governance unit: stores the processed data in the metadata warehouse and performs data lineage tracking and exception monitoring.

[0021] In this module, by using multi-source adapters and architectures, the system can provide a unified interface according to different data source types, and perform corresponding adaptation, conversion, feature vectorization and other processing to ensure the extraction and integration of information from various data sources and achieve efficient and reliable data transmission. A data processing pipeline unit is also set up to further optimize the data, effectively improving the high availability, scalability and reliability of the data. At the same time, it can also track and monitor the data to ensure the quality of the data.

[0022] Preferably, the Agent design component includes: firstly, using a deployment component to encapsulate the Agent into a lightweight Docker container, then using a data extraction component to extract data using an incremental or full pull strategy to reduce unnecessary data loading, then using a format conversion and compression component to convert the format and compress the extracted data to optimize data transmission efficiency, and finally using a feature engineering component to encode the processed data and perform feature splicing.

[0023] Therefore, in this application, Agent design components are used to perform incremental extraction, full extraction, format conversion, data compression and other technical means, effectively ensuring efficient and reliable data transmission.

[0024] Preferably, in the metadata governance unit, the data lineage tracking and abnormal monitoring process includes: designing a data lineage tracking mechanism and clearly marking the ETL transformation logic to handle the source and destination of data at each stage, and then using Prometheus and Grafana to form a monitoring component to monitor the data. This ensures the effective management and use of data assets and improves data quality and traceability.

[0025] Preferably, the model service module includes: Model hosting unit: Use the model warehouse to store the multimodal Transformer model and use the TensorFlowServing cluster to deploy the model as an API interface; API Gateway Unit: performs traffic diversion, authentication, and traffic management on API interfaces, and performs internal and external interface protocol conversion; Monitoring and debugging unit: monitors model performance indicators and performs log analysis to generate prediction result attribution analysis reports.

[0026] In order to ensure that the data lineage detection model can be called quickly and stably, a model hosting unit is set up in this system to manage the model and convert it into an API interface for external calls. At the same time, an API gateway unit is set up to manage the interface and a monitoring and control unit is set up to monitor the model performance, which effectively improves the certainty of data lineage model detection.

[0027] Preferably, the API gateway unit includes: Weighted traffic distribution component: It distributes traffic based on the weight of HTTP request path and model version, so that different versions of models can process requests in parallel. Authentication component: Use JWT Token verification + RBAC role for permission control to increase system security; Traffic management component: Traffic management is performed using fuses, current limiting, and request caching to reduce system pressure.

[0028] Therefore, by utilizing the API network management unit to divert, authenticate, and manage traffic on the API interface, the security of model calls is ensured and the pressure on the system can be reduced.

[0029] Preferably, the visualization module includes: Bloodline relationship graph construction unit: Node relationship data is stored through Neo4j or JanusGraph and transmitted to the front end through the interface. The front end framework renders the bloodline relationship graph based on the received data and uses Hook to monitor metadata changes to automatically trigger graph updates. Interactive analysis unit: Use the front-end framework to draw paths to support users to interactively query the relationship between different data nodes, and reversely query the downstream models and reports affected by field changes. At the same time, different colors or layout styles can be used to compare snapshots of graphs in different periods to help users identify differences in architecture evolution; Integrated extension unit: Displays data quality indicators in the graph, controls visualization content based on user permissions, and supports graph export.

[0030] In order to view the data lineage relationship more intuitively, this system not only visualizes the model detection results, but also uses interactive analysis units to allow users to view the data lineage relationship of different nodes, allowing users to analyze data dependencies more intuitively and efficiently. In addition, operation permissions are set in this system to control the visualization content that users can view, ensuring data isolation and security.

[0031] In addition, the present application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the program, it implements a data lineage full-link traceability method based on multi-source heterogeneous metadata and a pre-trained large model as described above, or a data lineage full-link traceability system based on multi-source heterogeneous metadata and a pre-trained large model as described above.

[0032] The present application also provides a non-transitory computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models or the above-mentioned data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models.

[0033] The present application also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models as described above, or the data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models as described above.

[0034] Compared with the prior art, the present invention has the following beneficial effects: The present invention realizes intelligent detection of data lineage through the deep combination of deep learning big model and big data metadata. Compared with the existing technology, it has higher accuracy, stronger generalization ability and better computing efficiency, and provides an innovative solution for data asset management. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1The figure is a flow chart of the method of the present invention.

[0036] Figure 2 This is a model structure diagram of the present invention.

[0037] Figure 3 It is a system structure diagram of the present invention.

[0038] Figure 4 A schematic diagram of the physical structure of an electronic device. DETAILED DESCRIPTION

[0039] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0040] Example 1 like Figure 1 As shown, Figure 1 This is a flow chart of the method of the present invention. This embodiment provides a data lineage full-link traceability method and system based on multi-source heterogeneous metadata and pre-trained large models. The method includes the following steps: Step S1: Collect metadata and pre-process the data; Specifically, Figure 2 As shown, Figure 2 It is the model structure diagram of the present invention, from Figure 2 It can be seen that the metadata collected in this embodiment includes: structured data L, text description data V and log tracking data A, and then a two-dimensional convolutional neural network is used to encode these data. The encoded data is represented by C, and then a position code P is added to each data type to help the model understand the relative or absolute position of each part of the data, and then enter the multi-head self-attention for encoding mapping. For the structured data L, it includes P→L encoding mapping and C→L encoding mapping; for the text description data V, it includes P→V encoding mapping and C→V encoding mapping; for the log tracking data A, it includes P→A encoding mapping and C→A encoding mapping; and then feature splicing is performed separately and input into the Transformer model for further processing, so that data of different modalities can be effectively combined under the same framework, thereby improving the performance of the model in multimodal tasks.

[0041] In this embodiment, in addition to the above data, metadata can also be collected from database logs, ETL tasks, API call chains, etc., including table structure, field type, operation timestamp, user tags, etc. Then, feature vectorization processing is performed on the collected data, including vectorization processing of structural features, time series features, and graph features; For the structured features: the field type and data magnitude are mapped into vectors by One-Hot encoding, such as for the field type Map it into a structured feature vector .

[0042] For the timing characteristics, a sliding window is used to extract the operating frequency and quantize the frequency characteristics; wherein the sliding window formula is: , Indicates the number of operations in the sliding window; Indicates the length of the sliding window; Represents the operation frequency in the sliding window.

[0043] For the graph features, an adjacency matrix is ​​constructed based on data dependencies. , and generate node vectors through graph embedding algorithms such as Node2Vec .

[0044] Therefore, in this embodiment, by encoding and vectorizing different modal features, the machine learning model can understand and use different types of features, thereby improving the prediction and reasoning capabilities of the model, and uniformly representing various features as vectors, it is also possible to fuse multiple features in the same model to capture more complex feature interactions. In addition, the data after vectorization is usually lower in dimension and more structured, which helps to improve computing efficiency, especially when processing large-scale data, and can accelerate the training and reasoning process.

[0045] Step S2: Use the data blood relationship detection model to identify the data processed by S1 and output the blood relationship probability matrix between data entities; Specifically, in this embodiment, step S2 also includes the construction and training of the model: Figure 2 As shown, the data kinship detection model is a multimodal Transformer model, which uses the data after feature concatenation in S1 as the input vector , and then the attention weight matrix is ​​calculated by the self-attention mechanism to capture the cross-field dependencies. The calculation formula is: ,in, represents the attention weight matrix; represents the query vector matrix; represents the transpose of the key vector matrix; Represents the dimension of the key vector. Finally, it passes through the blood relationship prediction head and the fully connected layer to output the blood relationship probability matrix ,in, Represents a data entity arrive The probability of blood relationship.

[0046] Therefore, by performing multimodal feature fusion, combining structured, temporal and graph features and concatenating the vectorized features as input vectors and inputting them into the model for training, the semantic representation ability of blood relationships is effectively improved. The self-attention mechanism enables the model to capture rich contextual information in the sequence, and the fully connected layer is used to enhance the model's expression ability, and finally outputs the blood relationship probability matrix.

[0047] Further preferably, the step S2 also includes distributed training of the model. In this embodiment, based on the Parameter Server architecture, the model parameters are stored in shards and updated using asynchronous SGD. The update formula is: ,in, Represents the parameters that the model needs to learn at the tth iteration; Indicates the parameters that need to be learned by the updated model at the t+1th iteration; represents the learning rate; Represents the gradient of the loss function relative to the model parameters. Thus, by calculating the gradient of the loss function and adjusting the learning rate, the performance of the model is optimized.

[0048] In addition, a contrastive learning loss function is used to optimize the model and a graph pruning algorithm is used to optimize the inference efficiency. The formula of the contrastive learning loss function is: ,in, Indicates a positive sample pair (there is a blood relationship); represents a negative sample pair (no blood relationship exists); and in the embodiment, the ratio of positive and negative samples is set to 1:3.

[0049] In this embodiment, a distributed parameter server is used to accelerate the training process, improve the utilization efficiency of computing resources and the scalability of the system, and optimize the representation ability of the model by comparing the learning loss function, which is particularly suitable for unsupervised learning tasks and improves the accuracy and robustness of the model. In addition, the graph pruning algorithm is used to optimize the reasoning efficiency, reduce the consumption of computing resources, and is suitable for real-time reasoning tasks.

[0050] Step S3: filtering the blood relationship probability matrix based on the dynamic threshold to generate a blood relationship map; Preferably, in this embodiment, the dynamic threshold is set to , thereby filtering out the blood relationship matrix that does not meet the conditions and generating the final blood relationship diagram.

[0051] Further preferably, in the step S3, it also includes updating the model parameters through an online learning mechanism, and updating the model through incremental training (such as Elastic Weight Consolidation) to adapt to the metadata model drift.

[0052] In this application, the model can be continuously optimized through an online learning mechanism, so that the model can dynamically adapt to metadata schema drift and adapt to changes in the data environment.

[0053] Step S4: Render the blood relationship map based on D3.js to visualize the data blood relationship.

[0054] In this application, the semantic feature capabilities of blood relationships are effectively improved by performing feature vector quantization processing on the collected data, and a data blood relationship detection model is set up to solve the problems of rigid rules and low computational efficiency of traditional technical methods. Through the deep combination of deep learning big models and big data metadata, intelligent detection of data blood relationships is realized. Compared with existing technologies, it has higher accuracy, stronger generalization ability and better computing efficiency, providing an innovative solution for data asset management.

[0055] Example 2 Based on the detection method described in Example 1, Figure 3 As shown, Figure 3 Schematic diagram of the system structure of the present invention. This embodiment provides a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models, and the system includes: The metadata collection and processing module is used to collect metadata in real time and pre-process the data to generate a data set; The model service module is used to deploy the data blood relationship detection model as an API interface, perform real-time blood relationship query and batch prediction by calling the API interface, and generate a blood relationship map; the data blood relationship detection model is a multimodal Transformer model; The visualization module is used to visualize the blood relationship map to facilitate interactive exploration and traceability analysis.

[0056] This application also provides a data lineage detection system, in which different modules are set up to realize real-time and batch detection of data lineage. It can not only effectively store and manage the lineage of data streams, but also visualize the lineage, allowing users to analyze data dependencies more intuitively and efficiently, and solve the problems of traditional data lineage detection technology, such as strong rule dependence, insufficient scalability, and lack of semantic understanding.

[0057] Preferably, the metadata collection and processing module includes: Data source access unit: Connect to various types of data sources through multiple adapter components, supporting structured (Hive / MySQL), semi-structured (JSON / XML), and unstructured (log / text) data sources; The original data is then processed and transmitted through the Agent design components, including: Deployment components: Encapsulate the Agent into a lightweight Docker container to ensure rapid deployment, upgrade, and expansion in different environments. Each Agent instance runs an independent container to ensure isolation and scalability. The containerized Agent can be scheduled and managed through the Kubernetes cluster. Kubernetes can help dynamically expand the number of Agent instances, support high availability and load balancing, and enable the Agent to automatically scale elastically according to changes in traffic and data sources.

[0058] Functional module: mainly includes data extraction, format conversion, feature engineering and transmission guarantee, ensuring that data can flow from the source system to the target system accurately, reliably and efficiently.

[0059] Data extraction component: Use incremental / full pull strategies, such as Hive partition scanning and Kafka ConsumerGroup offset management. For Hive, you can use the partition mechanism to achieve incremental extraction. For example, by scanning new partitions (or newly added data), you can reduce unnecessary data loading. For Kafka, you can use the offset management of the Consumer Group to obtain consumed messages, thereby ensuring incremental pull.

[0060] Format conversion and compression components: including Schema mapping (such as Avro / Parquet serialization) and data compression (using Snappy or Zstd compression algorithm). Different data sources may use different Schema formats. When extracting data, the Agent needs to perform Schema mapping to convert the data from the structure of the source system to a format that the target system can understand. In addition, in order to reduce bandwidth consumption and improve transmission efficiency, the data is also compressed.

[0061] Feature engineering component: Use a two-dimensional convolutional neural network to encode structured data L, text description data V, and log tracking data A. The encoded data is represented by C, and then add position code P to each data type to help the model understand the relative or absolute position of each part of the data. Then enter the multi-head self-attention for encoding mapping. For structured data L, it includes P→L encoding mapping and C→L encoding mapping; for text description data V, it includes P→V encoding mapping and C→V encoding mapping; for log tracking data A, it includes P→A encoding mapping and C→A encoding mapping; then perform feature splicing separately and input them into the Transformer model for further processing, so that data of different modalities can be effectively combined under the same framework, thereby improving the performance of the model in multimodal tasks.

[0062] Transmission guarantee components: including data breakpoint resuming, ACK confirmation mechanism, and data sharding. Breakpoint resuming ensures that after a failure occurs, the transmission can continue from the interruption point without having to start from the beginning, which helps to improve the robustness and fault tolerance of the system and avoid duplicate data transmission; the ACK confirmation mechanism ensures that the data has been successfully received and processed by the target system, ensuring the consistency and reliability of the data; when transmitting large-scale data, data sharding can divide large data sets into multiple small fragments and transmit them in parallel, which can not only increase the transmission speed, but also avoid the performance bottleneck caused by excessive data in a single data source or target system, thereby balancing the system load and improving the overall throughput.

[0063] Data processing pipeline unit: Use the Kafka cluster component to clean the data, and then use the storage optimization component to perform storage optimization processing on the cleaned data; Specifically, for the Kafka cluster, first design the topic: divide the topic by business domain, and distribute the data to multiple topics, such as metadata_raw and metadata_processed, which helps logical isolation and topic management, and facilitates consumers to sort and process different data. Then perform stream processing, integrate Kafka Streams or Flink to implement data cleaning (deduplication / null value processing) and field desensitization (regular expression matching). For storage optimization, first design the RowKey: timestamp reversal (Reverse Timestamp) + hash prefix, which can effectively avoid hot spots during reading, ensure uniform data storage, and improve performance. Then perform secondary indexing, implement SQL query through Phoenix, and finally use the TTL strategy for hot and cold data stratification, and use Hot Storage + S3 cold backup, so that the data that has been frequently used recently is stored in HBase for fast access, and migrate the data that has not been accessed for a long time and store it in cold storage solutions such as S3 to reduce storage costs while ensuring long-term backup and compliance of data.

[0064] Metadata governance unit: stores the processed data in the metadata warehouse and performs data lineage tracking and exception monitoring.

[0065] Specifically, for the metadata warehouse, choose Apache Atlas or Amundsen as the metadata warehouse to centrally store table structure, field lineage, and data quality rules. For data lineage tracking, design a data lineage tracking mechanism to record the following links: Hive table → Kafka Topic → HBase table. This comprehensive tracking can help business analysts and data engineers understand the data flow and its transformation process, and clearly mark the ETL transformation logic to handle the source and destination of data at each stage, helping data users better understand the data processing process and business rules. For abnormal monitoring, use Prometheus and Grafana to form a monitoring system to monitor Agent heartbeat, Kafka Lag, and HBase Region load. Metadata governance is used to improve the manageability and transparency of data assets, and also ensure data quality and compliance.

[0066] In this module, by using multi-source adapters and architectures, the system can provide a unified interface according to different data source types, and perform corresponding adaptation, conversion, feature vectorization and other processing to ensure the extraction and integration of information from various data sources and achieve efficient and reliable data transmission. A data processing pipeline unit is also set up to further optimize the data, effectively improving the high availability, scalability and reliability of the data. At the same time, it can also track and monitor the data to ensure the quality of the data.

[0067] Preferably, the model service module includes: Model hosting unit: Use the model warehouse to store the multimodal Transformer model and use the TensorFlowServing cluster to deploy the model as an API interface; Specifically, the model repository described in this unit uses industry-standard packaging formats, such as SavedModel, PMML, and ONNX, to ensure interoperability and easy integration of the model, and also manages the version history of the model file through Git-LFS (Large File Storage), which is convenient for tracking model changes and implementing version rollbacks, ensuring the traceability and security of the model. For the TensorFlow Serving cluster, model version management is included, which can support multiple versions in parallel, and hot model updates can be implemented through the gRPC interface, allowing dynamic loading or unloading of models without restarting the service, which can reduce downtime and improve system availability. In addition, resources are isolated, and the CPU and memory quotas of each model are limited based on cgroups (control groups) to ensure the reasonable allocation of resources and prevent mutual interference between different models, thereby improving the stability and performance of the overall service.

[0068] API Gateway Unit: performs traffic diversion, authentication, and traffic management on API interfaces, and performs internal and external interface protocol conversion; Specifically, in this embodiment, the API gateway unit uses the weighted shunting component to shun traffic based on the HTTP Path request path ( / api / v1 / predict) and the weight of the model version, so that different versions of the model can process requests in parallel, so that when performing A / B testing or gradually releasing new versions, the traffic can be effectively managed and controlled. And the authentication component uses JWT Token verification + RBAC role for permission control, thereby effectively increasing the security of the system. In addition, the traffic management component uses fuse (Hystrix), current limiting (token bucket algorithm), and request cache (Redis) to manage traffic to reduce the pressure on the system. Finally, the API gateway can also be used to realize the conversion between the client RESTful API and the back-end internal gRPC protocol, which can enhance the flexibility of the system, so that the client can use a more friendly RESTful interface, while the internal uses a more efficient gRPC for service calls. At the same time, the API gateway also needs to perform necessary input data preprocessing to ensure that the data meets the input requirements of the model before the request reaches the model prediction.

[0069] Monitoring and debugging unit: monitors model performance indicators and performs log analysis to generate prediction result attribution analysis reports.

[0070] Specifically, the model service module also includes monitoring and debugging during the use of the model, recording performance indicators such as QPS, Latency (P50 / P99), GPU utilization, and then using Elasticsearch, Logstash and Kibana (ELK Stack) for log collection and analysis, as well as associating TraceID to achieve full-link tracking of requests, so that when an exception occurs, the specific link where the problem occurs can be quickly located to improve the efficiency of troubleshooting. At the same time, in order to ensure that the model is interpretable, the SHAP and LIME libraries are also integrated to generate attribution analysis reports of prediction results, helping users understand how the model derives specific prediction results, thereby improving the transparency and trust of the model.

[0071] In order to ensure that the data lineage detection model can be called quickly and stably, a model hosting unit is set up in this system to manage the model and convert it into an API interface for external calls. At the same time, an API gateway unit is set up to manage the interface and a monitoring and control unit is set up to monitor the model performance, which effectively improves the certainty of data lineage model detection.

[0072] Preferably, the visualization module includes: Blood relationship map construction unit: Node relationship data is stored through Neo4j or JanusGraph, and transmitted to the front end through the interface. In this embodiment, the front end framework used is D3.js. D3.js renders the blood relationship map according to the received data, and uses Hook to monitor the change of metadata to automatically trigger the map update; Specifically, in this unit, Neo4j or JanusGraph is used as the graph storage engine, which supports the use of Gremlin or Cypher query language to operate node relationships. The types of nodes described in this embodiment include: data nodes, including Hive tables, Kafka Topics, and HBase tables; processing nodes: including Spark Jobs, Flink operators, and ETL scripts; business nodes: including indicator definitions (such as DAU) and reports (BI Dashboard). And through the Hook mechanism, metadata change events are monitored, such as automatically triggering the update of the graph when executing the Hive ALTER TABLE operation, to ensure the real-time and accuracy of the graph data.

[0073] Interactive analysis unit: Use D3.js to draw paths to support users to interactively query the relationship between different data nodes, and reversely query the downstream models and reports affected by field changes. At the same time, different colors or layout styles can be used to compare snapshots of graphs at different times to help users identify differences in architecture evolution; Specifically, in this unit, the path is the full link of "core order table → Kafka order flow → feature engineering → model input". Users can click or hover the mouse to view the complete data flow from the core order table to the Kafka order flow, and then to feature engineering and model input, helping users understand the data flow process. It also supports reverse query function, allowing users to view how changes to a certain field will affect related downstream models and reports, so as to make corresponding adjustments or decisions. At the same time, it is also possible to compare snapshots of lineage maps at different time points to identify differences in architectural evolution and help staff understand changes in the system.

[0074] Integrated extension unit: Displays data quality indicators in the graph, controls visualization content based on user permissions, and supports graph export.

[0075] Specifically, in this embodiment, data quality indicators (such as null value rate, expired models, etc.) are directly visualized in the graph through D3.js, and low-quality nodes are marked: for example, a table with a null value rate exceeding 30% can be marked with a warning color (such as red) on the node rendered by D3.js to attract user attention; for expired node marking: if a model has not been called for more than 6 months, D3.js can be used to add expired labels or other conspicuous prompts to these nodes.

[0076] Secondly, D3.js can also be integrated with the backend permission control system (such as LDAP / AD) to filter and display data according to user roles when rendering on the front end. For example, developers can only see the nodes and blood relationships belonging to their business line. When developers view the graph, the front end dynamically loads the data nodes and relationships related to the business line according to their permissions to ensure data isolation and security.

[0077] In addition, in this embodiment, the blood relationship map rendered by D3.js also supports the generation of documents in SVG or PDF format, which is convenient for export and archiving. SVG / PDF export: After D3.js renders the map, the map content can be exported as SVG or PDF files through its own API, and then these files can be embedded in Confluence or Jira for document management and collaboration.

[0078] In order to view the data lineage relationship more intuitively, this system not only visualizes the model detection results, but also uses the establishment of interactive analysis units for users to view the data lineage relationship of different nodes, allowing users to analyze data dependencies more intuitively and efficiently. In addition, this system also sets operation permissions to control the visualization content that users can view, ensuring data isolation and security.

[0079] Example 3 Based on the data lineage full-link traceability method and system based on multi-source heterogeneous metadata and pre-trained large models proposed in Example 1 and Example 2, this embodiment proposes to track the data lineage of financial transaction data, as follows: First, we use the metadata collection and processing module to collect and process data. We collect 100,000 metadata from the transaction database MySQL, including table structure and ETL task logs. Then we encode and vectorize these data. For structured features, we encode the field type (amount, date) into a 128-dimensional vector. For time series features, we update the frequency according to the hourly statistics table to generate a time series. For graph features, we build a table dependency adjacency matrix and use GraphSAGE to generate embeddings.

[0080] Then, in the model service module, the data lineage detection model is called to perform detection by inputting the new ETL task metadata, thereby outputting a lineage probability matrix, wherein the filtering threshold is set to 0.8 to generate a data lineage relationship map.

[0081] Finally, the graph is visualized through the visualization module to locate the data source of the transaction report (such as original order table → risk control intermediate table → report view).

[0082] In addition to being used in the field of financial transactions, the data blood relationship detection method and system described in the present invention can also be used to trace medical data and applied to the same model, method or system as mentioned above. The specific operations will not be repeated here. Through cross-domain verification, the prediction accuracy of the blood relationship detection capability of this method and system in medical imaging metadata can reach 87.3%.

[0083] Example 4 Based on the above embodiments, this embodiment further provides an electronic device, such as Figure 4 As shown, Figure 4 An example of a physical structure diagram of an electronic device is provided, and the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models described in Example 1 or the data lineage full-link tracing system based on multi-source heterogeneous metadata and pre-trained large models described in Example 2.

[0084] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data lineage full-link tracing method based on multi-source heterogeneous metadata and a pre-trained large model described in Example 1 or the data lineage full-link tracing system based on multi-source heterogeneous metadata and a pre-trained large model described in Example 2 provided by the above methods.

[0086] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data lineage full-link tracing method based on multi-source heterogeneous metadata and a pre-trained large model described in Example 1 or the data lineage full-link tracing system based on multi-source heterogeneous metadata and a pre-trained large model described in Example 2 provided by the above methods.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A data lineage full-link traceability method based on multi-source heterogeneous metadata and pre-trained large models, characterized by: The method comprises the following steps: S1: Collect metadata and preprocess data; S2: Use the data blood relationship detection model to identify the data processed by S1 and output the blood relationship probability matrix between data entities; S3: Filter the blood relationship probability matrix based on the dynamic threshold to generate a blood relationship map; S4: Render the blood relationship map obtained in S3 to visualize the data blood relationship.

2. According to claim 1, a data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models is characterized by: In step S1, the data preprocessing includes: S11: perform two-dimensional convolutional neural network encoding and position encoding on the data in sequence; S12: Perform encoding mapping on the encoded data, and then perform feature splicing on the encoded mapped data.

3. According to claim 2, a data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models is characterized by: The step S2 also includes the construction and training of the model: the data blood relationship detection model is a multimodal Transformer model, which uses the data after feature splicing in S1 as the input vector, and then calculates the attention weight matrix through the self-attention mechanism to capture the cross-field dependency relationship, and finally passes through the blood relationship prediction head and the blood relationship probability matrix through the fully connected layer.

4. According to claim 3, a data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models is characterized in that: The calculation formula of the attention weight matrix is: in, represents the attention weight matrix; represents the query vector matrix; represents the transpose of the key vector matrix; Indicates the dimension of the key vector.

5. According to claim 3, a data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models is characterized in that: The step S2 also includes distributed training of the model, sharding and storing the model parameters, and using asynchronous updates to train the model. At the same time, a contrastive learning loss function is used to optimize the model, and a graph pruning algorithm is used to optimize the reasoning efficiency.

6. According to claim 5, a data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models is characterized in that: The contrastive learning loss function formula is: in, represents a positive sample pair; represents a negative sample pair.

7. A data lineage full-link tracing method based on multi-source heterogeneous metadata and pre-trained large models according to any one of claims 1 to 6, characterized in that: In the step S3, it also includes updating the model parameters through an online learning mechanism and updating the model through incremental training to adapt to the metadata model drift.

8. A data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models, characterized by: The system comprises: The metadata collection and processing module is used to collect metadata in real time and pre-process the data to generate a data set; The model service module is used to deploy the data blood relationship detection model as an API interface, perform real-time blood relationship query and batch prediction by calling the API interface, and generate a blood relationship map; The visualization module is used to visualize the blood relationship map to facilitate interactive exploration and traceability analysis.

9. According to claim 8, a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models is characterized by: The metadata collection and processing module includes: Data source access unit: accesses various types of data sources through multi-adapter components, and performs feature processing and transmission of data source data through Agent design components; Data processing pipeline unit: Use the Kafka cluster component to clean the data, and then use the storage optimization component to perform storage optimization processing on the cleaned data; Metadata governance unit: stores the processed data in the metadata warehouse and performs data lineage tracking and exception monitoring.

10. The data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models according to claim 9 is characterized in that: The Agent design component includes: firstly, using a deployment component to encapsulate the Agent into a lightweight Docker container, then using a data extraction component to extract data using an incremental or full pull strategy to reduce unnecessary data loading, then using a format conversion and compression component to convert the format and compress the extracted data to optimize data transmission efficiency, and finally using a feature engineering component to encode the processed data and perform feature splicing.

11. According to claim 9, a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models is characterized by: In the metadata governance unit, the data lineage tracking and exception monitoring processing include: designing a data lineage tracking mechanism and clearly marking the ETL transformation logic to handle the data source and destination at each stage, and then using Prometheus and Grafana to form a monitoring component to monitor the data.

12. According to claim 8, a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models is characterized by: The model service module includes: Model hosting unit: Use the model warehouse to store the multimodal Transformer model and use the TensorFlowServing cluster to deploy the model as an API interface; API Gateway Unit: performs traffic diversion, authentication, and traffic management on API interfaces, and performs internal and external interface protocol conversion; Monitoring and debugging unit: monitors model performance indicators and performs log analysis to generate prediction result attribution analysis reports.

13. According to claim 12, a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models is characterized in that: The API gateway unit includes: Weighted traffic distribution component: It distributes traffic based on the weight of HTTP request path and model version, so that different versions of models can process requests in parallel. Authentication component: Use JWT Token verification + RBAC role for permission control to increase system security; Traffic management component: Traffic management is performed using fuses, current limiting, and request caching to reduce system pressure.

14. According to claim 8, a data lineage full-link traceability system based on multi-source heterogeneous metadata and pre-trained large models is characterized by: The visualization module includes: Bloodline relationship graph construction unit: Node relationship data is stored through Neo4j or JanusGraph and transmitted to the front end through the interface. The front end framework renders the bloodline relationship graph based on the received data and uses Hook to monitor metadata changes to automatically trigger graph updates. Interactive analysis unit: Use the front-end framework to draw paths to support users to interactively query the relationship between different data nodes, and reversely query the downstream models and reports affected by field changes. At the same time, different colors or layout styles can be used to compare snapshots of graphs in different periods to help users identify differences in architecture evolution; Integrated extension unit: Displays data quality indicators in the graph, controls visualization content based on user permissions, and supports graph export.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements a data lineage full-link tracing method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 7 or a data lineage full-link tracing system based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 8 to 14.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements a data lineage full-link traceability method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 7 or a data lineage full-link traceability system based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 8 to 14.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by the processor, it implements a data lineage full-link traceability method based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 1 to 7 or a data lineage full-link traceability system based on multi-source heterogeneous metadata and a pre-trained large model as described in any one of claims 8 to 14.

Images