Data hierarchical fusion storage system, storage method and related device

By proposing a data layered fusion storage system in massive data storage, processing and analysis systems, using components such as Hudi, HDFS, HBase, ClickHouse, MPP and Flink, unified storage and flexible processing of multi-source heterogeneous data are realized, solving the problem of insufficient data processing efficiency and query performance, and improving the overall performance and scalability of the system.

CN119669229APending Publication Date: 2025-03-21CHINA ELECTRIC POWER RESEARCH INSTITUTE CO LTD
0 Cites 0 Cited by

Patent Information

Application Number
CN202411780801.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-05
Publication Date
2025-03-21

AI Technical Summary

Technical Problem

The existing massive data storage, processing and analysis systems have shortcomings in data processing efficiency and query performance, especially when handling complex queries and aggregation operations, performance may be affected, affecting the data processing efficiency of streaming data processing frameworks such as Spark.

Method used

A data hierarchical fusion storage system is proposed, including the original data layer, the data initial processing layer, the data depth processing layer and the data application layer. It realizes unified storage and flexible processing of multi-source heterogeneous data through a layered and hierarchical hybrid storage method. It uses components such as Hudi, HDFS, HBase, ClickHouse, MPP and Flink to realize real-time data access, deep processing and efficient storage.

Benefits of technology

Through the hierarchical and hierarchical hybrid storage method, unified storage and flexible processing of multi-source heterogeneous data is realized, data processing efficiency and query performance are improved, near-real-time access and distributed parallel processing of massive power grid regulation data are supported, and the problems of inconvenient storage and untimely calculation of multi-source heterogeneous data are solved.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention belongs to a storage system, and provides a data hierarchical fusion storage system, a data hierarchical fusion storage method and a related device aiming at the technical problems of tedious work and low data processing efficiency of a big data warehouse adopted for mass data storage, processing and analysis at present. Comprising an original data layer, a data initial processing layer, a data deep processing layer and a data application layer which are arranged layer by layer, data synchronization is carried out among the layers, layered and hierarchical storage is carried out on the original data layer, cooked data processing is carried out in the data initial processing layer, and data are summarized and aggregated in the data deep processing layer; and finally, connecting with an external business application interface through the data application layer. Uniform storage and flexible processing of multi-source heterogeneous data are achieved in a hierarchical and hierarchical mixed storage mode, near-real-time afflux of service data is achieved in a streaming computing mode, and batch processing of mass offline data is achieved in a batch computing mode.
Need to check novelty before this filing date? Find Prior Art