A data lake file system based on object storage

Abstract

The present invention provides a data lake file system based on object storage, including a local file storage component, a file management component and a local metadata storage component, wherein the file management component includes an operation transaction management component and a file version management component; the local file storage The component is controlled by the file management component. The local file storage component is responsible for saving the business data storage object file locally, and calls the local metadata storage component to save the metadata corresponding to the business data target object; the operation transaction management component is used to control the entire local file storage The life cycle of component transactions, linking file version management components during transaction commit and rollback operations. The invention enables the application side of the component to achieve the effect of caching without being aware of the underlying file system principle, so that the user does not need to care about the details of data management, and only pays attention to the upper user interface to improve the effect and accuracy of data management, reduces the difficulty of data application, and improves the efficiency of data management. Data application flexibility.

Description

technical field [0001] The invention belongs to the field of data lake file systems, in particular to a data lake file system based on object storage. Background technique [0002] The storage of the existing data lake is generally HDFS or OSS object storage or other file storage methods (such as network storage, etc.). The existing method is mainly to solve the storage of structured, semi-structured and unstructured files. It was originally invented. This file system is for distributed computing (HDFS), easy access and use (object storage OSS, S3, CEPH, etc.), a large number of file storage (network storage GlusterFS, etc.), and lack of application scenarios that consider data lakes. [0003] In recent years, due to the storage and management of raw data, data lakes are easier and more flexible to use in subsequent big data processing and AI data applications than data warehouses and data marts, so they have developed rapidly in the market. However, there is no file system...

AI Technical Summary

Problems solved by technology

The main manifestations are: 1. There are various data sources, inconsistent processing methods, and low processing efficiency; 2. The data in the data warehouse is basically fixed, and the cost of data changes is huge; 3. Data analysis and scientific calculations brought about by data changes Subsequent data applications also need to be changed, and the cost is huge; 4. The management of data changes is missing, resulting in the lack of data governance, data blood relationship and other information, and low data utilization efficiency

Images