Data lineage relationship analysis method, big data platform system, equipment and storage medium

By extracting and grouping log events in the big data platform system, building write and read collections, and performing database name completion, the accuracy and stability issues of data lineage analysis are solved, and low-cost and efficient data lineage analysis is achieved.

CN119537321BActive Publication Date: 2025-09-23太保科技有限公司
2 Cites 0 Cited by

Patent Information

Application Number
CN202411215869.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2025-09-23
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

When analyzing data lineage relationships in a data computing platform, existing technologies have problems such as high development costs, low analysis accuracy, and impact on platform stability, especially when dealing with multiple programming languages ​​and high load conditions.

Method used

By extracting all events from the target log, grouping and sorting them based on session identifiers, building write and read sets, marking events with missing database names as pending completion events, and using specified database events to complete them, the data lineage relationship is finally calculated through Cartesian product, avoiding the need to develop specialized parsing programs for each programming language.

Benefits of technology

It reduces development and maintenance costs, improves the accuracy and completeness of analysis results, reduces the impact on the stability of the data computing platform, and improves analysis efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The present application discloses a method for analyzing data lineage relationships, a big data platform system, a device, and a storage medium. First, events in the target log are extracted, and events with the same identifier are grouped according to the session identifier. The data access event is determined to be the first target event, and a write set and a read set are constructed. Events with empty database names in the write or read set are marked as pending events; events without empty database names are marked as complete events. The complete event is determined by the specified database event of the completion event. Finally, the Cartesian product of the write and read sets of the complete event is calculated to obtain the data lineage relationship. The present application is based on log data extraction and processing, avoiding the development of specific parsing programs for each data programming language and reducing costs. Comprehensively grouping events, accurate lineage analysis, and an independent big data platform system reduce stability impacts and avoid load issues. Empty database events are marked as pending events, accurate and complete events, and analysis efficiency is improved.
Need to check novelty before this filing date? Find Prior Art