Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for migrating relational data to HBbase

A relational and data technology, applied in the field of HBase database, can solve the problems of reading and writing performance impact, loss of data mode information and data, space waste, etc., to achieve the effect of improving access performance, flexible configuration, and saving storage space

Active Publication Date: 2014-03-12
INST OF INFORMATION ENG CAS
View PDF0 Cites 75 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This kind of migration has the following problems: 1) Loss of data mode information and data uniqueness guarantee, such as data type, whether non-null is allowed
2) Too many columns will have a certain impact on read and write performance
3) Improper row key design may cause waste of space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for migrating relational data to HBbase
  • Method and system for migrating relational data to HBbase
  • Method and system for migrating relational data to HBbase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0059] In Embodiment 1, the table mapping adopts one-to-one mapping, the column mapping adopts the default mode, and the row key is a multi-key type. For example, Table 1 is the schema definition of the table named t1 in the relational database, and Table 2 and Table 3 are respectively the schema definition and the data storage table schema mapped to the corresponding table named hash(t1) in HBase.

[0060] Table 1

[0061]

[0062] Table 2

[0063]

[0064] table 3

[0065]

Embodiment 2

[0066] In Embodiment 2, the table mapping adopts many-to-one mapping, the column mapping adopts a sequential method, and the row key is a single key type. As shown in Table 4, it is a sampling table stored in the relational database. The generation time of the sampling table is November 12, 2013. Table 5 is another sampling table stored in the relational database. The generation of the sampling table The time is November 13, 2013. Since Table 4 and Table 5 express the same business and table schema, in order to control the data volume of a single table and improve local access performance, the annual log information table generated on a daily basis can be Store the date information on the timestamp of the column, thereby merging and storing tables of multiple days into one table. Therefore, Table 4 and Table 5 can be combined and stored as Table 6, where Table 6 is a data storage table in HBase.

[0067] Table 4

[0068] ID

sample value

123

12.1

...

Embodiment 3

[0073] In Embodiment 3, the table mapping adopts one-to-one mapping, the column mapping adopts a sequential manner, and the row key is a single key type. Table 7 is a data table in the relational database, and Table 8 is the corresponding data storage table mapped to HBase.

[0074] Table 7

[0075] ID

0:00

0:15

0:30

...

23:45

123

12.1

15.1

13.1

...

1.1

124

11.1

16.1

18.1

...

8.1

...

...

...

...

...

...

[0076] Table 8

[0077]

[0078] The above column mapping is carried out in a time-series manner, that is, to list all the time value attributes in the table in the relational database in the data storage table of HBase and define a column cluster separately. The column cluster contains a column, and the migrated data corresponds to the columns with different timestamps.

[0079] Batch data import is divided into two parts: data reading and data writing, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a method and system for migrating relational data to the HBbase. The system comprises a relational data base, an HBase database, an SQL analysis module, a query conversion module and a data mode management module. As a mapping relation list is designed in the HBase database, a mapping relation of mode information in an original relation database and mode information in the HBase database is stored and mode information of a data form in the original relation database the is completely reserved, a mapping manner can be flexibly configured and more options are provided for users. Due to the efficient data organizing manner of the HBase, the access performance is greatly improved and the storage space is saved. Meanwhile, the data export function is provided and data interaction with other sub-systems is achieved. In addition, the data query function supports SQL query and converts an SQL query statement into a query statement of the HBase database, so that the users can transit to the HBbase database more smoothly.

Description

technical field [0001] The invention relates to the field of data storage and management, in particular to a method and system for migrating relational data in a traditional relational database to an HBase database. Background technique [0002] With the diversification of data generation methods and the diversification and convenience of data access methods, the data in various business systems is rapidly expanding. Correspondingly, in the traditional way, the storage capacity and access volume of data managed by relational databases continue to increase in magnitude. Traditional relational libraries face severe challenges. [0003] In recent years, NoSQL databases have flourished in industry and research. The natural distributed architecture of the NoSQL database automatically completes the horizontal division of data and concurrent access, and greatly improves the data storage capacity and read and write performance compared with traditional relational databases. The i...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/214
Inventor 戴娇文杰韩冀中周薇
Owner INST OF INFORMATION ENG CAS
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com