A Loading Client Implementation Method Oriented to Distributed Data Warehouse

Abstract

The invention discloses oriented distribution data warehouse high efficiency load client end realization method, belongs to information processing field; detailed steps are, first step, initializing system start parameter; load device manage module each applying a load device for every route; every route each building data exposition module, each client end data expositing and transmitting to load device; each load device respective calling data examine module and examining; then, storing after examination character section data in data buffer module, managing and transmitting to data transmission module; distribution node listen module acquiring every distribution data warehouse listen status, and transmitting to data transmission module; finally, data transmission module transmitting received buffer data to healthy distribution data warehouse. The method raises overall distribution data warehouse apply efficiency and data load efficiency, conforms to current apply requirement, explores broad apply prospect.

Description

technical field [0001] The invention belongs to the field of information processing, in particular to a method for implementing a loading client facing a distributed data warehouse. Background technique [0002] With the continuous development of computer technology and the continuous improvement of informatization, people use more and more distributed storage; different from the current common centralized storage technology, distributed storage technology does not store data in a certain or Instead, use the disk space on each machine in the enterprise through the network, and form a virtual storage device with scattered storage resources, and store data scattered in every corner of the enterprise. [0003] The loading efficiency of data in a distributed environment greatly affects the efficiency of the entire cluster; in order to improve data loading efficiency, optimize the use of the entire cluster, and reduce costs, a better and faster loading method is urgently needed. ...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the problem of slow loading efficiency of existing distributed data warehouses, the present invention provides a loading client-side implementation method for distributed data warehouses, which can not only solve the simultaneous loading of multiple data tables, load balancing of multiple data warehouse nodes, but also enable Use a small amount of memory to play the highest loading rate, effectively reduce the loading time, and promote the continuous improvement of the development of the big data system

Images