Unlock instant, AI-driven research and patent intelligence for your innovation.

Distributed database system data loading method and device

A technology of system data and database, which is applied in the field of data loading in distributed database systems, can solve the problem of large computing load of loading nodes, and achieve the effect of solving the loading bottleneck

Inactive Publication Date: 2017-02-08
TIANJIN NANKAI UNIV GENERAL DATA TECH
View PDF5 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a method and device for loading data in a distributed database system, so as to solve the technical problem of loading nodes with a large amount of computation

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed database system data loading method and device
  • Distributed database system data loading method and device
  • Distributed database system data loading method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0038] figure 1 It is a flow chart of the method for loading distributed database system data provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of data loading in distributed database system, and the method can be executed by a data loading device for distributed database system. The device can be realized by software / hardware, and can be integrated in the management node of the distributed database system.

[0039] see figure 1 , the distribution method of the distributed database system data, comprising:

[0040] S110. The management node receives an instruction from a user to load data, where the instruction includes a loading parameter.

[0041] A distributed database system usually includes a cluster of management nodes and a cluster of computing nodes. Through any node of the management cluster as an entry, the user can input an instruction to load data, and the management node is responsible for receiving the user's inst...

Embodiment 2

[0048] figure 2 It is a flow chart of the method for loading distributed database system data provided by Embodiment 2 of the present invention. This embodiment is applicable to the case of loading data in a distributed database system. This method can be executed by a data loading device for a distributed database system. The device can be realized by software / hardware, and can be integrated in the computing nodes of the distributed database system.

[0049] see figure 2 , the loading method of the distributed database system data, comprising:

[0050] S210, the computing node receives the allocation instruction sent by the management node, and parses the allocation instruction.

[0051] After receiving the SQL issued by the management node, the nodes in the computing node cluster are responsible for parsing the SQL, and obtain data from the data source according to the ftp, http, and hdfs protocols from the data source set in the issued SQL.

[0052] S220. Obtain row data...

Embodiment 3

[0057] image 3 It is a schematic structural diagram of the distributed database system data distribution device provided by Embodiment 3 of the present invention, as shown in image 3 As shown, the device includes:

[0058] An instruction receiving unit 310, configured to receive an instruction from a user to load data, the instruction including loading parameters;

[0059] A determining unit 320, configured to determine the amount of data loaded according to the instruction, and determine the number of nodes for loading data;

[0060] The allocating unit 330 is configured to allocate the loaded data according to the amount of data loaded and the number of nodes for loading the data, and issue allocation instructions to computing nodes, so that the computing nodes complete the loading of data according to the allocation instructions.

[0061] The distributed database system data distribution device provided by the embodiment of the present invention determines the amount of...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a distributed database system data loading method and device. The method comprises the following steps of receiving a data loading instruction of a user by a management node, wherein the instruction comprises a loading parameter; determining the volume of loaded data by the management node according to the instruction, and determining the quantity of nodes of the loaded data; and distributing the loaded data according to the volume of the loaded data and the quantity of the nodes of the loaded data, and sending a distribution instruction to an operation node to ensure that the operation node completes the loading of the data according to the distribution operation. A loading task can be sent to a computation node to ensure that the computation node loads the data. According to the distributed database system data loading method and device provided by the invention, the problem that the file servers become loading bottlenecks is solved.

Description

technical field [0001] The invention belongs to the technical field of distributed databases, in particular to a method and device for loading data in a distributed database system. Background technique [0002] Distributed database refers to the use of high-speed computer network to connect multiple physically dispersed data storage units to form a logically unified database. The basic idea of ​​a distributed database is to disperse and store the data in the original centralized database to multiple data storage nodes connected through the network to obtain larger storage capacity and higher concurrent access. In recent years, with the rapid growth of data volume, distributed database technology has also developed rapidly. Traditional relational databases have begun to develop from centralized models to distributed architectures. Relational-based distributed databases retain traditional databases. Under the data model and basic characteristics, from centralized storage to ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 崔维力武新张绍勇
Owner TIANJIN NANKAI UNIV GENERAL DATA TECH