Data loading method and device

Abstract

The invention discloses a data loading method and a data loading device, which belong to the field of databases. The method comprises the steps of: creating a partition table according to partition types, wherein the partition table comprises partition fields; reading data to be loaded line by line; obtaining partition fields of the data in the line according to partition information in the partition table; determining the subordinative partition of the data in the line in the partition table according to the partition fields of the data; and loading the data in the line into the corresponding partition. The device comprises a creating module, a reading module, an obtaining module, a determining module and a loading module. The method and the device disclosed by the invention have the advantages that the partition fields of the data are obtained according to the partition information of the partition table, the subordinative partition of the data in the partition table is determined, the data in the line is automatically loaded into the corresponding partition, and the automatic data loading on a distributed file system is realized, so the data is ensured to be loaded into the correct partition, and the data loading error caused by the manual partition specifying is avoided.

Description

technical field [0001] The invention relates to the field of databases, in particular to a data loading method and device. Background technique [0002] The amount of data stored in the table in the data warehouse is often very large, so the method of partitioning is widely used to divide the data in the table into several partitions according to certain strategies such as range and discrete value. The advantage of partitioning is that you can scan and query in the corresponding partitions in the table, avoiding full table scanning and improving analysis efficiency; operations such as updating, deleting, and backing up data in the table can be performed according to partitions, improving management efficiency. Therefore, the massive data stored in the distributed data warehouse will also adopt the method of partitioning. However, the correct loading of data to the corresponding partitions is a prerequisite to ensure that the partitions are meaningful, otherwise it will lead...

AI Technical Summary

Problems solved by technology

[0005] When loading data, the user needs to specify which data to load into which partition, so the user needs to be fully familiar with the system, master the principle of partitioning and the method of loading data into the partition, the user threshold is high, and the usability is poor; The user’s negligence caused partition errors and wrong file names, etc., but the system will perform error loading without giving any checks and prompts, resulting in data loading errors; once the data is loaded into the system, because there is no partition field in the data, Unable to verify that data loaded correctly

Images