A streaming data-oriented online processing method and system

Abstract

The invention discloses an online processing method and system oriented to streamed data. The method comprises the following steps: 1, establishing an online memory cache layer, extracting the property of the streamed data according to a key value structure, and storing in the online memory cache layer; 2, establishing a hybrid index structure for the streamed data in the memory cache layer; 3, adding an access flag bit to each streamed datum for which the index structure is established, and recording the access state of the streamed data by each analysis program, wherein the flag bits are used for flagging the register condition of the streamed data by different analysis programs; 4, cleaning data: if a certain streamed datum is accessed by all specified analysis programs in the memory cache layer, cleaning the streamed datum. By adopting the method and the system, the data reading and writing stress in a streamed processing procedure is reduced greatly, the stress of a database in a large-scale streamed data processing system can be effectively relieved, and the real-time processing speed of the streamed data can be increased.

Description

technical field [0001] The invention relates to large-scale data processing, in particular to an online processing method and system for streaming data. Background technique [0002] With the progress of the times and the development of the economy, people's demand for information in daily life is increasing, especially with the increasing popularity of the Internet, a large amount of information is released and disseminated on the Internet every day. In 2011, the analysis and research organization IDC released "Extracting Value from Chaos". The report shows that the total amount of information in the world doubles every two years. In 2011, the total amount of data created and copied worldwide was 1.8ZB. For example, 1.8ZB is equivalent to the total amount of data generated by every person in the world doing 215 million high-resolution MRI examinations every day. [0003] The task of the large-scale data analysis and processing system is to process massive data and extrac...

AI Technical Summary

Problems solved by technology

The biggest disadvantage of Redis is that it does not solve the problem of data service reliability well. All data is stored in the memory space requested by the user. Once the process restarts or exits abnormally, data will be lost.

However, it cannot meet the needs of querying according to different fields of the data

[0011] To sum up, the ability to alleviate data access pressure in existing technologies is limited by various factors and cannot meet actual needs

Images