Streaming data real-time processing method and system based on multi-level cache structure

Abstract

The invention discloses a streaming data real-time processing method and system based on a multi-level cache structure, and belongs to the technical field of big data stream processing. The method of the invention uses on-line data processing and batch processing for streaming data, so as to satisfy the requirements of low latency, high throughput and reliability of the processing. The method comprises the following steps: storing an on-line processing result in an on-line cache, and designing an on-line cache structure to automatically and efficiently delete stale data; storing data of a recent access offline persistent database in a batch cache, and adopting an LRU based strategy to efficiently eliminate the stale data; and merging a query result efficiently so as to return the query result to users quickly. The invention further realizes the streaming data real-time processing system based on the multi-level cache structure. The streaming data real-time processing method and the system based on the multi-level cache structure provided by the invention can solve the problems in the existing method that the storage and query efficiency is not high, the reliability is low and the memory occupancy rate is higher.

Description

technical field [0001] The invention belongs to the technical field of big data stream processing, and more specifically relates to a method and system for real-time processing of stream data based on a multi-level cache structure. Background technique [0002] The IDC report shows that the total amount of global data is expected to exceed 40ZB by 2020, and the global data volume is growing at a rate of 58% per year, and this speed is still accelerating. A large part of the data is generated by sensor networks, social networks, The streaming data generated by web browsing, etc., has the characteristics of real-time, volatile, bursty, random, and infinite. It needs to process the streaming data efficiently, quickly, and reliably, and output the processing results in real time or Real-time query. [0003] in terms of data processing. According to the characteristics of streaming data, it can be roughly divided into three processing frameworks: 1. Pure streaming processing fr...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the above defects or improvement needs of the prior art, the present invention provides a method and system for real-time processing of streaming data based on a multi-level cache structure. Store in the online cache, and design the online cache structure to automatically and efficiently delete expired data; store the data that recently accessed the offline persistent database in the batch cache, and use the LRU-based strategy to efficiently eliminate expired data; efficiently merge the query results to quickly Return to the user, thereby solving the problems of low storage and query efficiency, low reliability, and high memory usage in existing methods

Images