Distributed real-time data import device

Abstract

The invention discloses a distributed real-time data import device, and particularly relates to the technical field of data import, the distributed real-time data import device comprises a manager and works, the manager is electrically connected with a data warehouse, and the manager is electrically connected with the works. According to the invention, through the redirection function, the connection of the segment can always exist, and the time consumption for repeatedly establishing and closing the connection is reduced; the data is stored in the message middleware, so that the storage cost of the data warehouse is reduced; data redistribution calculation is advanced to the device, so that the computing power of segment is released; the reprocessed data is directly stored in the segment, the process of redistributing the data is avoided, the internal network of the data warehouse is not occupied any more, and the upper limit of the scale of the data warehouse is improved; the data source supports various message middleware, and importable data source types are increased; due to a distributed deployment structure, the import rate is improved, and deployment resources can be flexibly increased and reduced according to the data volume; and a real-time data query function is realized by using an import protocol.

Description

technical field [0001] The invention relates to the technical field of data import, and more specifically, the invention relates to a distributed real-time data import device. Background technique [0002] The commercial relational database system we usually use, its main goal is to ensure the ACID characteristics of data access, and provide powerful data management and access services for various business and transaction applications. However, the real-time performance of their data services is difficult to be guaranteed. The fundamental reasons are: [0003] Traditional databases are all disk databases. The main copy of data is on the hard disk. When users need to access data, the DBMS loads the data into the main memory, that is, the management of data is "disk-based caching technology". Compared with the main memory, the disk is an extremely low-speed storage medium, and the disk access speed is also related to the physical location of the accessed data and the current ...

AI Technical Summary

Problems solved by technology

During the whole process, the master node will read and process all the data, and these data will be retransmitted in the internal network of the data warehouse, which greatly consumes the computing power of the master node and the network bandwidth between all nodes;

[0010] 3) The protocol only supports reading files

[0012] Each segment will pull data, but the data is not necessarily distributed exactly on the segment, so there will be data redistribution. The larger the amount of data, the more serious the redistribution. The computing power of the segment and the network between nodes the heavier the consumed

These consumptions restrict the further expansion of the data warehouse cluster;

[0013] Only after pulling all the data, the user can find the data. When the amount of imported data is larger, the data delay in the data warehouse is more serious, and the real-time performance is poor;

[0014] Can only pull file data

Images