Streaming data self-adaption persistence method and system based on mixed storage

Abstract

The invention provides a streaming data self-adaption persistence method and system based on mixed storage. The method includes the steps that state feature information of a streaming data processing system is collected in real time; a multiple linear regression model based on machine learning is established, and model parameters are estimated according to the collected state feature information; the optimal persistent window size of the streaming data processing system under the current state is calculated and obtained according to the state feature information of the current streaming data processing system and the established regression model; the streaming data processing system changes the current persistent window according to the obtained persistent window size, and the middle state or the calculation result in the streaming data processing process is stored in a solid state disk; when data capacity in the solid state disk reaches a certain degree, data in the solid state disk are stored in an ordinary hard disk. By means of the method and system, the persistent window size at the moment can be calculated according to the current and historical state information, accordingly the situation that the streaming data rate is unstable is dynamically adapted, and balance between usability and consistency of the system is guaranteed.

Description

technical field [0001] The invention relates to a method and system for self-adaptive persistence of streaming data based on hybrid storage, and belongs to the technical field of software. Background technique [0002] With the rapid development of cloud computing, Internet of Things, social media and other information technologies and application models, streaming data has gradually become a key data form in information systems. In typical application fields, streaming big data has the characteristics of real-time, volatile, sudden, disordered and infinite (see literature: Sun Dawei, Zhang Guangyan, Zheng Weimin. Big data streaming computing: key technologies and System example [J]. Journal of Software, 2014, 25(4): 839-862.). In fields such as the Internet of Things, we need to quickly store data streams in non-volatile media, and in the financial field with high reliability requirements; we need to store the intermediate state and calculation results of nodes and save th...

AI Technical Summary

Problems solved by technology

[0010] The purpose of the present invention is to overcome the problem of fixed windows in the existing streaming data persistence, and propose a hybrid storage-based streaming data adaptive persistence method, which can calculate the The size of the persistent window, and then dynamically adapt to the situation of unstable streaming data rate, ensures the balance between ease of use and consistency of the system

Images