Autonomous data lake construction system and method based on associated data

Abstract

The invention discloses an autonomous data lake construction system and method based on associated data. The system comprises a data source input module, a heterogeneous data preprocessing module, a metadata discovery and extraction module, a metadata fusion and association module, a meta-model optimization and construction module, an instance knowledge extraction module, a knowledge packaging module, a knowledge correction and fusion module, an instance concept extraction module and a meta-model verification and evolution module. Based on the associated data, the directory index updated in real time and the instance knowledge graph capable of being quickly positioned through the directory are generated while the data lake is constructed, and the data lake with the autonomous ability is finally obtained through the internal structure and semantic association of the directory index and the instance knowledge graph, so that the data lake is easily managed and retrieved by external usersand more requirements are met.

Description

technical field [0001] The present invention relates to a technology in the field of big data information processing, in particular to a multi-source heterogeneous data-based autonomous data lake construction system and method based on associated data. Background technique [0002] In the era of big data, all data has potential value. A data lake is a centralized repository that allows all structured, semi-structured and unstructured data to be stored at any scale. Data lakes do not require pre-defined data structures and can be stored in raw form. After a long period of accumulation, a data lake without governance will become a "data swamp" that no one can clean up, and the data becomes difficult to understand and use. Therefore, how to build a data lake that can fully automatically integrate data, update the data lake model and directory in real time, and facilitate external management and use, making it an autonomous data lake becomes very important. At present, there ...

AI Technical Summary

Problems solved by technology

However, how to effectively automate the fusion of continuous heterogeneous data sources, deeply analyze the instance data in the data lake, automatically establish internal semantic associations for it, and build an autonomous data lake, so that it can be easily managed and retrieved by external users and meet more complex requirements. There is still a lack of demand

Images