Unlock instant, AI-driven research and patent intelligence for your innovation.

Data import method and device

A technology of data import and import line, applied in database index, database update, structured data retrieval, etc., can solve the problem of low data import efficiency

Pending Publication Date: 2021-07-30
BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present application provides a data import method and device to solve the problem of low data import efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data import method and device
  • Data import method and device
  • Data import method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0069] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

[0070] The nouns involved in the embodiments of the present application are explained as follows:

[0071] Data slice: refers to the constituent unit of the data. For example, in this embodiment, the data slice is the constituent unit of the data to be imported, and one data to be imported may include multiple data fragments.

[0072] Storage sharding: refers to the constituent units ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data import method and device, and the method comprises the steps: dividing to-be-imported data into a plurality of data fragments in response to a data import instruction, and constructing an import relationship according to each data fragment and each storage fragment of a column database management system, the import relationship being used for indicating a corresponding relationship between the data fragments and the storage fragments; and importing each data fragment into the storage fragment having the corresponding relationship with the data fragment, and importing the data fragment into the corresponding storage fragment based on the import relationship by constructing the import relationship between the data fragment and the storage fragment, so that the problems are solved that when the to-be-imported data is imported based on a random import strategy, the import rate is low due to the fact that different to-be-imported data contend for the same storage node in the prior art, the import operation is executed based on the import relation, targeted import can be achieved, and the technical effect of the import rate is improved.

Description

technical field [0001] The embodiments of the present application relate to the field of computer technology and the field of big data, and in particular to a data import method and device. Background technique [0002] The Spark processor is a fast and general-purpose computing engine designed for large-scale data processing. The Spark processor can read data in a distributed manner, and through various conversions and processing of the data, the processed data can be distributed and imported into A storage target (such as a database management system, specifically ClickHouse, which is a columnar database management system for real-time analysis of big data). [0003] In the prior art, the method for importing data includes: obtaining a data import request, the data import request includes one or more data to be imported, and in each storage slice of the database management system, one or more data to be imported is randomly selected Allocate storage slices, and import one...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/23
CPCG06F16/221G06F16/2379
Inventor 冯志恒
Owner BEIJING WODONG TIANJUN INFORMATION TECH CO LTD