Real-time data loading method and device based on data flow technology

A real-time data and loading device technology, applied in the field of data management, can solve the problems that the data cannot be processed by query and analysis, the flexible construction of the loading network cannot be realized, and the frequency of data loading is not available, so as to ensure continuous loading and reduce loading delay. time, the effect of continuous loading frequency

Inactive Publication Date: 2014-07-02
中国移动通信集团内蒙古有限公司
View PDF8 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this type of technology has the following defects: the data loading frequency is not continuous, the delay is large, and during the execution of the ETL task, the default data source does not change, and the data being loaded cannot be used for query and analysis processing
[0011] It is not difficult to find from Table 1 that although the above-mentioned various loading methods have many advantages and uses, these methods can only perform periodic loading or batch loading, and fail to realize the flexible construction of the loading network

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time data loading method and device based on data flow technology
  • Real-time data loading method and device based on data flow technology
  • Real-time data loading method and device based on data flow technology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] The present invention utilizes the real-time processing capability of the data stream to improve the freshness of data loading in the data warehouse, and further enhance the enterprise's response capability to high real-time data. The basic idea is;

[0049] Set up the data flow operation operator set; select several operators from the operator set according to business needs to generate real-time query processing expressions for data flow; set the real-time loading network for data flow according to the real-time query processing expressions; drive control into real-time Load the data stream in the network and dynamically allocate cache space for the data stream; output the resulting data stream and load it into the real-time data warehouse.

[0050] The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

[0051] figure 1 It is a schematic diagram of the implementation flow of the real-ti...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a real-time data loading method based on a data flow technology. The method comprises the steps that a data flow operator set is set; a plurality of operators are selected from the operator set according to service demands to generate a real-time query processing expression of data flow; the real-time loading network of the data flow is set according to the real-time query processing expression; the data flow entering in the real-time loading network is driven and controlled, and buffer memory space is dynamically distributed for the data flow; result data flow is output and is loaded to a real-time data warehouse. The invention further discloses a real-time data loading device based on the data flow technology. The method and device are used for achieving real-time conversion and continuous loading of data, and the real-time loading network can be established in real time.

Description

technical field [0001] The invention relates to the technical field of data management, in particular to a real-time data loading method and device based on data stream technology in a real-time data warehouse. Background technique [0002] Real-time data warehouse is a new branch of data warehouse technology that appears to adapt to the rapid changes in the market and real-time enterprise management decisions. The real-time data warehouse expands the capabilities of the traditional data warehouse, making the development of the data warehouse enter the real-time decision-making stage after going through the report stage, analysis stage and forecast stage. In order to meet the needs of business applications for real-time data and active decision-making, real-time data warehouses need to capture changes in data sources in real time and make decisions according to preset rules, thus putting forward new requirements for timely loading of data source data. Real-time data loading...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 郭向红尤新霞庞哲翀郭翔宇孙颖飞王波张大亮
Owner 中国移动通信集团内蒙古有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap