A method and device for data storage under a distributed data platform

Abstract

The invention provides a method and device for storing data under a distributed data platform. The data storage and data retrieval efficiency can be improved while data changes can be effectively recorded. The method for storing the data under the distributed data platform comprises the steps that changed data are classified by comparing intraday data and data in a data state changing table; the classified data are sorted into different catalogues and stored in corresponding partitions according to data storage rules of the catalogues; the data state changing table is updated.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for storing data on a distributed data platform. Background technique [0002] Big data—people use it to describe the current era of information explosion. It is not only reflected in the leap in data volume, but also in more and more types of data storage, from traditional relational data, Key-Value data, to formal More diverse flat files, images, audio, video, and more. To analyze such complex data, higher requirements are placed on the computing performance and storage performance of the data platform. [0003] Using a distributed Hadoop system to store and analyze big data is a common practice in the industry. Since the distributed Hadoop system uses files to store data, although the data storage capacity and throughput have been improved, it has been sacrificed. The update mechanism of the original relational database only supports the operations of i...

AI Technical Summary

Problems solved by technology

[0005] 1. The storage solution for relational databases is helpless to deal with large amounts of data; while the existing distributed file system adopts the method of snapshot accumulation, which sacrifices a large amount of storage space and is inefficient in subsequent calculations;

[0006] 2. Data retrieval often requires a full scan, which takes up a lot of system resources;

[0007] 3. Lack of flexibility for online complex and changeable data scenarios

[0008] However, in a large number of application scenarios, a piece of data often undergoes many state changes from generation to extinction. Correspondingly, the data platform generates multiple snapshots when recording data state changes, and the data storage will expand rapidly. In the process, it is often necessary to track the historical trajectory of the data, and it is necessary to scan a large amount of historical data to restore the state, which is inefficient

Images