Supercharge Your Innovation With Domain-Expert AI Agents!

Method and system for automatic data input of Hadoop data warehouse

A technology of data warehouse and data system, which is applied in the field of automatic data import and system of Hadoop data warehouse, which can solve problems such as manual operation, and achieve the effect of saving time and making mistakes less prone to errors

Inactive Publication Date: 2017-09-08
温州市鹿城区中津先进科技研究院
View PDF5 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Aiming at the deficiencies in the prior art, the present invention provides a method and system for automatically importing data into a Hadoop data warehouse, which solves the need for manual operation when the data in the relational database is transmitted to the data warehouse of Hadoop in the prior art. inconvenience

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for automatic data input of Hadoop data warehouse
  • Method and system for automatic data input of Hadoop data warehouse
  • Method and system for automatic data input of Hadoop data warehouse

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0045] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0046] The embodiment of the automatic import data method of Hadoop data storehouse of the present invention, as figure 1 shown, including:

[0047] Step 1 100: The server C equipped with the Hadoop data warehouse pre-configures the data transmission interface for obtaining data from the server A equipped with the relational database;

[0048] Step 2 101: The server B equipped with the job scheduler pre-configures the call command used to call the data transmission interface and the execution period for executing the call command;

[0049] Step 3 102: Server B periodically executes the call command according to the execution cycle;

[0050] Step 4 103: server C obtains data from server A and generates HDFS files;

[0051] Step 5 104: Server C imports...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method for automatic data input of a Hadoop data warehouse. The method comprises the steps that (1) a server C which carries the Hadoop data warehouse configures a data transmission interface of data from a server A which carries a relation-type database in advance; (2) a server B which carries an operation dispatcher configures a calling command used to call the data transmission interface and an execution cycle used to execute the calling command in advance; (3) the server B executes the calling command periodically according to the execution cycle; (4) the server C acquires data from the server A and generates a HDFS document; and (5) the server C inputs the generated HDFS document into a Hive data warehouse. In addition, the invention also discloses a system for the automatic data input of the Hadoop data warehouse. The system comprises the server A, the server B and the server C. According to the invention, the inconvenience in the prior art that manual operations are needed when the data in the relation-type database is transmitted to the Hadoop data warehouse each time can be overcome.

Description

technical field [0001] The invention relates to a method and system for automatically importing data into a Hadoop data warehouse. Background technique [0002] With the increasing amount of data that enterprises need to store and analyze, Hadoop is getting more and more attention. Hadoop is an open source project of the Apache Software Foundation. Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. Due to its irreplaceable advantages in scalability, robustness, computing performance and cost, Hadoop has become the current mainstream big data storage and analysis platform. [0003] At present, the basic data used in big data analysis is usually stored in relational databases such as mysql, sqlsever, db2, etc. Due to the need for data analysis and processing, these basic data need to be screened and imported into Hadoop's Hive data warehouse , through the computing and processing capabilities of the Hadoop platform to achieve d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/182G06F16/258G06F16/283
Inventor 王振宇
Owner 温州市鹿城区中津先进科技研究院
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com