Real-time data loading method and device based on data flow technology

Abstract

The invention discloses a real-time data loading method based on a data flow technology. The method comprises the steps that a data flow operator set is set; a plurality of operators are selected from the operator set according to service demands to generate a real-time query processing expression of data flow; the real-time loading network of the data flow is set according to the real-time query processing expression; the data flow entering in the real-time loading network is driven and controlled, and buffer memory space is dynamically distributed for the data flow; result data flow is output and is loaded to a real-time data warehouse. The invention further discloses a real-time data loading device based on the data flow technology. The method and device are used for achieving real-time conversion and continuous loading of data, and the real-time loading network can be established in real time.

Description

technical field [0001] The invention relates to the technical field of data management, in particular to a real-time data loading method and device based on data stream technology in a real-time data warehouse. Background technique [0002] Real-time data warehouse is a new branch of data warehouse technology that appears to adapt to the rapid changes in the market and real-time enterprise management decisions. The real-time data warehouse expands the capabilities of the traditional data warehouse, making the development of the data warehouse enter the real-time decision-making stage after going through the report stage, analysis stage and forecast stage. In order to meet the needs of business applications for real-time data and active decision-making, real-time data warehouses need to capture changes in data sources in real time and make decisions according to preset rules, thus putting forward new requirements for timely loading of data source data. Real-time data loading...

AI Technical Summary

Problems solved by technology

However, this type of technology has the following defects: the data loading frequency is not continuous, the delay is large, and during the execution of the ETL task, the default data source does not change, and the data being loaded cannot be used for query and analysis processing

[0011] It is not difficult to find from Table 1 that although the above-mentioned various loading methods have many advantages and uses, these methods can only perform periodic loading or batch loading, and fail to realize the flexible construction of the loading network

Images