The invention discloses a real-time storage optimization method based on block data, which relates to the technical field of
big data processing and distributed storage, introduces a block data concept, realizes block-by-block writing, in-block ordering, block-level ordering and an intelligent flush trigger mechanism, and realizes real-time writing with high
throughput,
low delay and low memory occupation. The method specifically comprises the steps that block data is introduced into a
data memory model to serve as a basic
processing unit, multiple rows of data are aggregated into a block according to an aggregation key CKValue, row data in the block are sequenced according to a
primary key PK, and write-in, namely aggregation is achieved; writing data, aggregating according to blocks, and uniformly sorting in the blocks; one block, namely multiple rows of data, generates an FRC file, and
small files are reduced; exception
processing: supporting block-level
recovery to skip a failed block; the core problems of too
high memory occupation, large CPU consumption, high writing
delay and the like in a high-
concurrency and high-
throughput real-time writing scene can be solved, and the method is widely applied to scenes with extremely high requirements on data real-time performance and
system stability.