Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and device for integrating multiple versions of HBase by Sqoop

A sqoop-hbase, multi-version technology, applied in the field of Sqoop component improvement, can solve problems such as inability to fully support all functions of HBase2, achieve the effect of highlighting substantive features, reliable design principles, and wide application prospects

Active Publication Date: 2021-01-15
SUZHOU LANGCHAO INTELLIGENT TECH CO LTD
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] Aiming at the defects that the above-mentioned current Sqoop components of the prior art are suitable for the highest version of the production environment and cannot fully support all functions of the HBase2.x version, the present invention provides a Sqoop integrates the method and device of multi-version HBase to solve the above technical problems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for integrating multiple versions of HBase by Sqoop
  • Method and device for integrating multiple versions of HBase by Sqoop
  • Method and device for integrating multiple versions of HBase by Sqoop

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0063] Such as figure 1 As shown, the present invention provides a method for Sqoop integrated multi-version HBase, comprising the steps:

[0064] S1. Create the Sqoop-HBase parameter analysis module, the Sqoop-HBase parameter analysis module obtains the Sqoop command parameters, and parses the HBase parameter information from the Sqoop command parameters;

[0065] S2. create the HBase version selection module, the HBase version selection module selects the MapReduce parameters and strategies corresponding to the HBase version according to the HBase parameter information;

[0066] S3. update the Sqoop-MapReduce job generation module, the Sqoop-MapReduce job generation module obtains the original MapReduce job file generated according to the Sqoop command parameters, and adds the MapReduce parameters and strategies corresponding to the HBase version to the MapReduce job file;

[0067] S4. Update the MapReduce job submission module. The MapReduce job submission module packages ...

Embodiment 2

[0069] Such as figure 2 As shown, the present invention provides a method for Sqoop integrated multi-version HBase, comprising the steps:

[0070] S1. Create the Sqoop-HBase parameter analysis module, the Sqoop-HBase parameter analysis module obtains the Sqoop command parameters, and parses the HBase parameter information from the Sqoop command parameters; the specific steps are as follows:

[0071] S11. Create a Sqoop-HBase parameter parsing module;

[0072] S12. Obtain the Sqoop command script that the user uses the Sqoop component to write the imported data to HBase;

[0073] S13.Sqoop-HBase parameter parsing module saves the Sqoop command script in the shell file, and generates the shell script file;

[0074] S14. The Sqoop-HBase parameter analysis module reads the HBase parameter information related to HBase that Sqoop can identify from the shell script file, and extracts the HBase parameter information and sends it to the HBase version selection module; for example, h...

Embodiment 3

[0087] Such as image 3 As shown, the present invention provides a device for Sqoop integrating multi-version HBase, including:

[0088] Sqoop-HBase parameter analysis module creation unit 1, used to create Sqoop-HBase parameter analysis module, Sqoop-HBase parameter analysis module obtains Sqoop command parameters, and parses HBase parameter information from Sqoop command parameters; Sqoop-HBase parameter analysis module creation unit 1 includes:

[0089] Sqoop-HBase parameter analysis module creation subunit 1.1, used to create Sqoop-HBase parameter analysis module;

[0090] Sqoop command script acquisition subunit 1.2, which is used to obtain the Sqoop command script written by the user to import data to HBase using the Sqoop component;

[0091]The shell script file generation subunit 1.3 is used to set the Sqoop-HBase parameter analysis module to save the Sqoop command script into the shell file and generate the shell script file;

[0092] HBase parameter information ex...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method and device for integrating multiple versions of HBase through Sqoop, and the method comprises the following steps that an Sqoop-HBase parameter analysis module obtainsan Sqoop command parameter, and analyzes HBase parameter information; an HBase version selection module selects a MapReduce parameter and a strategy corresponding to the HBase version according to theHBase parameter information; a Sqoop-MapReduce job generation module is used for acquiring an original MapReduce job file generated according to the Sqoop command parameter, and adding the MapReduceparameter and strategy corresponding to the version of the HBase into the MapReduce job file; and a MapReduce job submitting module packages the MapReduce job file into a MapReduce cluster identification format, and submits the MapReduce cluster identification format to the MapReduce cluster for operation processing.

Description

technical field [0001] The invention belongs to the technical field of Sqoop component improvement, in particular to a method and device for Sqoop integrating multi-version HBase. Background technique [0002] Sqoop, is a tool for transferring data between Hadoop and relational databases. [0003] RDBMS is a relational database management system. [0004] Hbase, ApacheHBase is an open source, distributed, versioned, non-relational database modeled after Google's Bigtable-distributed storage system for structured data. [0005] Using Sqoop, you can import data from relational database management systems (RDBMS) such as MySQL or Oracle to Hadoop Distributed File System (HDFS), or export data from Hadoop Distributed File System (HDFS) to relational databases, such as MySQL. Sqoop uses HadoopMapReduce to extract and load data. [0006] HBase is an Apache open source distributed column-oriented database built on the Hadoop file system, which is horizontally scalable. HBase is...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/25G06F16/28G06F16/23G06F16/27G06F16/182G06F16/21
CPCG06F16/25G06F16/284G06F16/23G06F16/27G06F16/182G06F16/219
Inventor 陶钰昕秦正何金彪
Owner SUZHOU LANGCHAO INTELLIGENT TECH CO LTD