Method for importing all data of relational database into Hive

A data import and database technology, applied in database indexing, database management system, structured data retrieval, etc., can solve the problems of differences in data import process, increase development cost and operation and maintenance cost, etc., to save operation and maintenance cost, Save development cost, accurate data effect

Pending Publication Date: 2021-04-13
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, most systems use data collection tools such as Flume, NiFi, etc. to collect data from relational databases and write them into big data storage Hive. There are still data type conversion problems in the data processing process, which leads to differences in the entire data import process. , it is impossible to form a set of processes to realize data import, which increases the development cost and operation and maintenance cost

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for importing all data of relational database into Hive
  • Method for importing all data of relational database into Hive

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it, but the examples given are not intended to limit the present invention.

[0028] Each relational database (such as MySQL, Oracle, etc.) supports multiple data types (such as character type, numeric type, date type, etc.), when the business requires data aggregation to achieve the purpose of big data analysis, due to the original relationship The table fields of a database-type database have clear business meanings, so when data is collected into big data and stored in Hive, the original table field type is also required to be retained. For example, the numerical type data in a relational database table is still required to be a value when stored in Hive. Type, which can support numerical function calculations such as addition, subtraction, multiplication...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for importing all data of a relational database into Hive, and relates to the technical field of data conversion. The method comprises the steps of: creating a Hive intermediate table by adopting the string field type, importing data of the relational database into the Hive intermediate table, mapping the table field type of the relational database into the field type of the Hive target table according to the data table field type of the relational database, creating a Hive target table, importing the data of the Hive intermediate table into the Hive target table by adopting HiveSQL, and performing data analysis based on the Hive target table.

Description

technical field [0001] The invention discloses a method and relates to the technical field of data conversion, in particular to a method for importing all data of a relational database into Hive. Background technique [0002] Most of the traditional data is stored in relational databases (such as MySQL, Oracle, etc.), these relational databases can support various data types such as characters and numbers, and different relational databases handle data types such as characters and numbers differently. [0003] At present, most systems use data collection tools such as Flume, NiFi, etc. to collect data from relational databases and write them into big data storage Hive. There are still data type conversion problems in the data processing process, which leads to differences in the entire data import process. , It is impossible to form a set of processes to realize data import, which increases the development cost and operation and maintenance cost. Contents of the invention ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/25
CPCG06F16/2282G06F16/258
Inventor 陈娟妮肖雪李国涛
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap