Structured line data-oriented distributed parallel data importing method

A technology of data import and columnar data, applied in structured data retrieval, database indexing, electronic digital data processing, etc., can solve the problems of lower memory database access efficiency and storage efficiency, slow import speed, low import efficiency, etc., to achieve The effect of improving technical effects and improving data import efficiency

Active Publication Date: 2015-11-18
UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF4 Cites 17 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0012] The present invention provides a distributed parallel data import method for structured columnar data, which solves the problem of slow import speed and low import efficiency in the existing data import method, and reduces the access efficiency and storage efficiency of the memory database. Technical probl...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Structured line data-oriented distributed parallel data importing method
  • Structured line data-oriented distributed parallel data importing method
  • Structured line data-oriented distributed parallel data importing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] The present invention provides a distributed parallel data import method for structured columnar data, which solves the problem of slow import speed and low import efficiency in the existing data import method, and reduces the access efficiency and storage efficiency of the memory database. Technical problems, realized the rapid import of structured data stored in the disk-type database into the distributed memory database system, and can provide personalized services according to user needs, and provide incremental data update function, improving the columnar database data The technical effect of import efficiency.

[0036] In order to better understand the above-mentioned technical solution, the above-mentioned technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0037] Please refer to Figure 1-Figure 2 , figure 1 It shows a schematic diagram of a specific embodiment of the present...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a structured line data-oriented distributed parallel data importing method, comprising: step 1: obtaining a data importing rule and generating an importing task according to the data importing rule; step 2: obtaining data scale, dividing an original data table into a plurality of sub-tables according to the data scale, and at the same time, dividing the importing task into a corresponding number of importing sub-tasks according to the number of sub-tables; step 3: reading sub-table data in parallel by the plurality of importing sub-tasks, packaging the data into a protocol message, and sending the protocol message to a data importing subsystem; and step 4: creating a data underlying index and a data distribution index by the data importing subsystem according to the original data sent in the step 3, and importing the data underlying index and the data distribution index into a distributed memory database engine, so that the technical effects of quickly importing structured data stored in a disk type database into the distributed memory database system and improving the data importing efficiency of a column-oriented database are achieved.

Description

technical field [0001] The invention relates to the field of computer software, in particular to a distributed parallel data import method for structured columnar data. Background technique [0002] The commercial relational database system we usually use, its main goal is to ensure the ACID characteristics of data access, and provide powerful data management and access services for various business and transaction applications. However, the real-time performance of their data services is difficult to be guaranteed. The fundamental reasons are: [0003] Traditional databases are all disk databases. The main copy of data is on the hard disk. When users need to access data, the DBMS loads the data into the main memory, that is, the management of data is "disk-based caching technology". Compared with the main memory, the disk is an extremely low-speed storage medium, and the disk access speed is also related to the physical location of the accessed data and the current disk he...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/219G06F16/22
Inventor 段翰聪张建闵革勇柳陆王瑾曾祥楷陈超朱越
Owner UNIV OF ELECTRONICS SCI & TECH OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap