Cluster database data loading tool and method supporting multiple data sources

Abstract

The invention relates to a cluster database data loading tool and method capable of supporting various data sources, and aims to provide the cluster database data loading tool and method with the advantages of high speed in authentication and various authentication ways. The cluster database data loading tool comprises a loading tool use interface, a loading task management module, a loading task execution module, a data reading module, a data source access module, a data writing module and a database access module, wherein the loading tool use interface is used for starting loading tasks and transferring the parameter of each task to the loading task management module; the loading task management module is used for storing, maintaining and scheduling all received loading tasks; the loading task execution module is in charge of executing the specific loading task; the data reading module is used for providing a uniform data reading interface; the data source access module is used for reading data from a specific data source; the data writing module is used for providing a uniform data writing interface; and the database access module is used for loading the received data to a cluster database.

Description

technical field [0001] The invention relates to the field of cluster databases, in particular to a cluster database data loading tool and method supporting multiple data sources. Background technique [0002] The cluster database uses multiple ordinary servers to provide high-performance database services, and is suitable for the storage, query and analysis of massive data. Because it is suitable for big data occasions, it needs to cooperate with high-performance data loading tools to read data from various data sources and load them into the cluster database. In practical applications, due to information security considerations, data sources often have a security authentication mechanism, and there are multiple security mechanisms to choose from, and the loading tool must support the corresponding authentication mechanism in order to successfully read from the data source. data file. For example, HDFS data source supports three different authentication mechanisms: simple ...

AI Technical Summary

Problems solved by technology

[0003] When using the loading tool to read data from a data source that requires security authentication, you must know the corresponding authentication method and configure the loading tool correctly. Otherwise, you will lack sufficient permissions when reading data, resulting in loading failure.

On the other hand, in the same data source, there may be multiple authentication methods coexisting and functioning at the same time. If the loading tool cannot handle this combination of authentication methods, it will also fail to read data due to insufficient permissions.

Images