Data loading method and device

A data loading and data technology, applied in the database field, can solve problems such as wrong file names, partition errors, system error loading checks and prompts, etc., to ensure correctness and avoid data loading errors.

Inactive Publication Date: 2012-06-06
TENCENT TECH (SHENZHEN) CO LTD
View PDF6 Cites 19 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] When loading data, the user needs to specify which data to load into which partition, so the user needs to be fully familiar with the system, master the principle of partitioning and the method of loading data into the partition, the user threshold is high, and the usability is poor; The user’s negl

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data loading method and device
  • Data loading method and device
  • Data loading method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0051] The embodiment of the present invention provides a data loading method, see figure 1 , the method flow is as follows:

[0052] 101: Create a partition table according to the partition type, and the partition table includes partition fields;

[0053] 102: Read the data to be loaded line by line;

[0054] 103: Obtain the partition field of the row data according to the partition information of the partition table;

[0055] 104: Determine the partition to which the row of data belongs in the partition table according to the partition field of the row of data;

[0056] 105: Load the row of data into the corresponding partition.

[0057] In the method provided by the embodiment of the present invention, by obtaining the partition field of the data according to the partition information of the partition table, and determining the partition to which the row data belongs in the partition table, and automatically loading the row data into the corresponding partition, it is re...

Embodiment 2

[0059] The embodiment of the present invention provides a data loading method, see figure 2 , the method flow is as follows:

[0060] 201: Create a partition table according to the partition type, and the partition table includes partition fields;

[0061] Wherein, the embodiment of the present invention creates the table through the CREATE TABLE statement, and other ways may also be used to create the table, which is not specifically limited. Define the type of partition when creating a table to form a partitioned table. The type of the partition can be a partition of the range type, a partition of the list type, or a composite type of partition nested between the two. The embodiment of the present invention supports two levels of nesting, such as range-range, range- List, list-list, and list-range are four composite partition types, but there are no specific restrictions on other partition types.

[0062] Specifically, the way to create a table and define a partition typ...

Embodiment 3

[0157] see Figure 5 , the embodiment of the present invention provides a data loading device, the device includes:

[0158] The creation module 501 is used to create a partition table according to the type of the partition, and the partition table contains a partition field;

[0159] The reading module 502 is used to read the data to be loaded line by line;

[0160] An acquisition module 503, configured to acquire the partition field of the row of data read by the reading module 502 according to the partition information of the partition table;

[0161] A determining module 504, configured to determine the partition to which the row of data belongs in the partition table according to the partition field of the data obtained by the obtaining module 503;

[0162] The loading module 505 is configured to load the row of data into the corresponding partition determined by the determining module 504 .

[0163] Wherein, the obtaining module 503 is specifically conf...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data loading method and a data loading device, which belong to the field of databases. The method comprises the steps of: creating a partition table according to partition types, wherein the partition table comprises partition fields; reading data to be loaded line by line; obtaining partition fields of the data in the line according to partition information in the partition table; determining the subordinative partition of the data in the line in the partition table according to the partition fields of the data; and loading the data in the line into the corresponding partition. The device comprises a creating module, a reading module, an obtaining module, a determining module and a loading module. The method and the device disclosed by the invention have the advantages that the partition fields of the data are obtained according to the partition information of the partition table, the subordinative partition of the data in the partition table is determined, the data in the line is automatically loaded into the corresponding partition, and the automatic data loading on a distributed file system is realized, so the data is ensured to be loaded into the correct partition, and the data loading error caused by the manual partition specifying is avoided.

Description

technical field [0001] The invention relates to the field of databases, in particular to a data loading method and device. Background technique [0002] The amount of data stored in the table in the data warehouse is often very large, so the method of partitioning is widely used to divide the data in the table into several partitions according to certain strategies such as range and discrete value. The advantage of partitioning is that you can scan and query in the corresponding partitions in the table, avoiding full table scanning and improving analysis efficiency; operations such as updating, deleting, and backing up data in the table can be performed according to partitions, improving management efficiency. Therefore, the massive data stored in the distributed data warehouse will also adopt the method of partitioning. However, the correct loading of data to the corresponding partitions is a prerequisite to ensure that the partitions are meaningful, otherwise it will lead...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 赵伟郭玮李均洪坤乾张书彬徐钊
Owner TENCENT TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap