Data import method and device based on Hbase database, electronic equipment and medium

A data import and database technology, applied in the field of big data, can solve problems affecting the timeliness of data import, occupancy, and the success rate of data import, and achieve the effects of improving timeliness and success rate, high concurrency, and speed

Pending Publication Date: 2020-09-11
CHINA PING AN PROPERTY INSURANCE CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, by calling the Put interface of Hbase to import data, it is easy to occupy too much Hbase database resources, which will affect the timeliness of data import; by relying on the MapReduce framework to perform data batch conversion and then import data, it will rely too much on the MapReduce framework , once the MapReduce framework fails, it will affect the success rate of data import

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data import method and device based on Hbase database, electronic equipment and medium
  • Data import method and device based on Hbase database, electronic equipment and medium
  • Data import method and device based on Hbase database, electronic equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0047] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0048] The execution subject of the Hbase database-based data import method provided in the embodiment of the present application includes but is not limited to at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the data import method based on the Hbase database can be executed by software or hardware installed on the terminal device or server device, and the software can be a block chain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.

[0049]Blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and e...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to big data, and discloses a data import method based on an Hbase database. The method comprises: receiving an original data set, and unifying the data format of the original data set by using a preset computing framework to obtain a target data set; creating an identification field of each piece of target data in the target data set, and performing hash processing on the identification field of each piece of target data to generate a target field of each piece of target data; according to the target field, creating a data pre-storage partition of an Hbase database, and storing target data corresponding to the target field into the data pre-storage partition; and loading the target data in the data pre-storage partition to a data table of the Hbase database by utilizing a preset data loading method. The invention also relates to a blockchain technology, and the original data set can be stored in a blockchain node. According to the method, the timeliness and the success rate of data import based on the Hbase database can be improved.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a method, device, electronic equipment and computer-readable storage medium for data import based on Hbase database. Background technique [0002] The Hbase database is a NoSQL database based on Apache Hadoop. It is a distributed and column-oriented dynamic schema database that can support real-time reading and writing and random access to very large-scale data sets. [0003] At present, there are two main methods for data import based on the Hbase database: method 1, directly calling the Put interface of Hbase to import data; method 2, relying on the MapReduce framework to perform data batch conversion and then import data. [0004] However, by calling the Put interface of Hbase to import data, it is easy to occupy too much Hbase database resources, which will affect the timeliness of data import; by relying on the MapReduce framework to perform data batch conversion a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/22
CPCG06F16/258G06F16/2282
Inventor 詹芮
Owner CHINA PING AN PROPERTY INSURANCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap