Unlock instant, AI-driven research and patent intelligence for your innovation.

Data migration method and device

A data and data source technology, applied in the computer field, can solve problems such as occupying local disk space, reducing data migration efficiency, and time-consuming, and achieve the effect of improving efficiency

Inactive Publication Date: 2016-06-29
HANGZHOU DT DREAM TECH
View PDF5 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this migration method, the user needs to operate twice to realize the final data migration. The whole process is equivalent to two complete data migrations, which reduces the efficiency of data migration; moreover, this method needs to export the data to the local, occupying Local disk space, disk I / O operations are time-consuming and reduce migration efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data migration method and device
  • Data migration method and device
  • Data migration method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] In order to overcome the problem of low migration efficiency caused by using two migration tools to perform data migration twice in the current data migration, the data migration method provided in the embodiment of this application will realize data migration in one data migration tool. To improve the efficiency of data migration across clusters.

[0022] figure 1 The principle of the data migration method of this application is illustrated, such as figure 1 As shown, assume that the first cluster 11 and the second cluster 12 are two different Hadoop clusters, for example, the first cluster 11 can be CDH5.2.0, and the second cluster 12 can be Hadoop2.6.0; or, the first cluster 11 is Hadoop1.x, the second cluster 12 is Hadoop2.x; Or, the first cluster 11 can also be HDP, the second cluster 12 is Hadoop, etc., no longer enumerate in detail, in these examples, the first cluster 11 and the second cluster The data migration in 12 is the data migration across clusters.

...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data migration method and device. The method comprises the steps as follows: a data source of a first cluster is loaded through a first class loader; the data source of a second cluster is loaded through a second class loader; the first class loader and the second class loader inherit a loader of a data migration tool; in the data migration tool, a first thread reads data of the data source of the first cluster through the first class loader and the data is put into a data queue; and in the data migration tool, a second thread writes the data in the data queue into the data source of the second cluster through the second class loader. The data migration method and device improve the data migration efficiency across the Hadoop cluster.

Description

technical field [0001] The disclosure relates to computer technology, in particular to a data migration method and device. Background technique [0002] Hadoop is a software framework capable of distributed processing of large amounts of data. It is a distributed computing platform that allows users to easily structure and use it. Users can easily develop and run applications that process massive amounts of data on Hadoop. In big data widely used in processing. With the continuous expansion of big data application requirements, Hadoop has also carried out a series of version changes to solve the technical bottleneck caused by huge demand changes. However, Hadoop versions are usually not compatible with each other, so data migration becomes an essential operation in the version upgrade process. For example, data migration between HIVE (hive is a Hadoop-based data warehouse tool) and HDFS (Hadoop Distributed File System, Hadoop Distributed File System) is a frequently encoun...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/214
Inventor 郑振峰
Owner HANGZHOU DT DREAM TECH