Real-time stream processing system, method and device based on real-time number warehouse and storage medium

Abstract

The invention provides a real-time stream processing system, a real-time stream processing method and a real-time stream processing device based on a real-time data warehouse and a storage medium. Inthe system, a first real-time processing unit in a real-time environment module performs regular self-inspection to obtain first detection information. A detection unit in a real-time database is usedfor obtaining second detection information about the real-time database. A second real-time processing unit in an offline environment module carries out regular self-inspection to obtain third detection information. A batch processing unit carries out regular self-inspection to obtain fourth detection information. A control module controls the switch module according to the states of the first detection information, the second detection information, the third detection information and the fourth detection information, selects one of the first real-time processing unit, the second real-time processing unit and the batch processing unit for starting to obtain output data, writs the output data into the written object, and selects one of the real-time database and the offline database as thewritten object. According to the invention, the real-time warehouse availability can be improved through automatic upgrading and downgrading. The friendliness of downstream business parties is improved, and the general control program is kept highly available.

Description

technical field [0001] The present invention relates to the field of data stream processing, in particular to a real-time stream processing system, method, device and storage medium based on a real-time data warehouse. Background technique [0002] The existing data processing of the existing data warehouse is based on the historical data extracted from the business database every night, so what can be provided to the business side is offline data, and all business development is also done through yesterday's data. However, the business scenarios that can be covered by this offline data are limited. For example, the behavior of new users can only be fed back to the business side through the data warehouse the next day after registration, so it cannot be satisfied in some business scenarios that require real-time user data. , limiting the service scope of the data warehouse. So there is a demand for real-time data warehouses. [0003] The real-time data warehouse writes the...

AI Technical Summary

Problems solved by technology

[0007] The existing problems are mainly in the link of the entire real-time data warehouse. When each component is unavailable, it cannot continue to provide data to the business side. The repair time varies depending on the component with the problem, but it will cause data corruption. Delay, which has a great impact on the business side

[0008] The disadvantage of the existing technology is the vulnerability of the overall link of the real-time data warehouse. If there is a problem in one of them, it will inevitably cause the delay of business data and affect the use of business parties.

Images