Method for data migration of non-relational database HBase

A non-relational, database technology, applied in the data migration field of non-relational database HBase, can solve problems such as inability to customize data by users, inflexible transfer methods, and increase user burden, to ensure data consistency, improve speed, speed effect

Inactive Publication Date: 2014-01-15
NINGBO DONGGUAN TECH
View PDF1 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since HBase is a distributed database developed on the distributed file system HDFS, most of the existing data migration solutions for HBase are based on HDFS data blocks, copying and transferring HDFS data blocks storing HBase data to new clusters , because the data block transfer of HDFS must transfer the entire

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for data migration of non-relational database HBase
  • Method for data migration of non-relational database HBase
  • Method for data migration of non-relational database HBase

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0025] The present invention will be further described below in conjunction with the drawings and specific embodiments, but the present invention is not limited to the following specific embodiments.

[0026] figure 1 The overall implementation flow chart is given, showing the main steps of the invention. Combining with the historical data migration process of a city water company, the scheme of the present invention is further explained. The specific implementation steps are as follows:

[0027] 1. Design a reasonable RowKey (that is, row key) for the data stored in the non-relational database HBase. The RowKey reflects the group information and time information of the data. The specific operations are as follows:

[0028] According to the needs of data migration, the structure of RowKey is designed as: group-year-month-day-user-defined information. The first four fields of RowKey (group-year-month-day) are called RowKey prefix. The RowKey prefix is ​​mainly used for data migration...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for data migration of a non-relational database HBase. According to the method, by reading configuration file information of users, data within a time period designated by the users are migrated from an old HBase cluster to a new HBase cluster. The method has the advantages that by designing RowKey, group information and time information of the data are embodied on the RowKey, and whether the data corresponding to the RowKey are within a migration time range designated by the users can be directly identified by scanning the RowKey during data migration; groups serve as data migration units, every group serves as a check point, the range of the non-synchronous data between the old HBase cluster and the new HBase cluster can be rigidly controlled within one group when abnormal conditions occur, and time required for restoring data synchronization is reduced; log information is recorded in the data migration process to be used for treating the abnormal conditions in the data migration process, synchronization of the data between the old cluster and the new cluster is guaranteed, and data migration between the HBase clusters can be finished accurately and efficiently.

Description

technical field [0001] The invention relates to a data migration technology of a non-relational database, in particular to a data migration method of a non-relational database HBase. Background technique [0002] Database technology is one of the fastest-growing fields in computer science and technology, and it is also one of the most widely used technologies. In recent years, with the development of the Internet, especially the rise of Internet Web2.0, traditional relational databases are dealing with The ultra-large-scale and high-concurrency SNS-type websites have been unable to do what they want, but the non-relational database has developed very rapidly due to its own characteristics in dealing with large-scale data and high-concurrency access. . [0003] HBase is a distributed, column-oriented non-relational database, suitable for applications that require real-time random read and write of very large-scale data. But as time goes by, the amount of data that needs to ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/273G06F16/214
Inventor 贝毅君唐宏波王丞淼王宽
Owner NINGBO DONGGUAN TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products