Inter-cluster data migration method and system, server and computer storage medium

An inter-cluster and intermediate storage technology, applied in the computer field, can solve problems such as non-support, process blocking, migration, etc., and achieve the effects of wide application range, reduced time, and improved migration speed

Active Publication Date: 2017-11-24
360 TECH GRP CO LTD
View PDF7 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The existing technical solution is to directly transmit data between two clusters when performing data migration. For each data table, each node needs to start the remote transmission process, and it takes a long time to start the service. When there is little data, the time required to transfer the data may be very short, which is less than the time required to start the service, which will cause the data migration speed to be very slow, and the existing data migration method cannot transfer empty tables between clusters. When the table is empty, the process will be blocked. In addition, the existing data migration method does not support data migration from a cluster with a large number of child nodes to a cluster with a small number of child nodes.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Inter-cluster data migration method and system, server and computer storage medium
  • Inter-cluster data migration method and system, server and computer storage medium
  • Inter-cluster data migration method and system, server and computer storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0026] figure 1 A schematic flowchart of a method for data migration between clusters according to Embodiment 1 of the present invention is shown. The method is used for data migration between the first cluster and the second cluster, and the first cluster and the second cluster both include a master node and at least one child node, such as figure 1 As shown, the method includes the following steps:

[0027] Step S100 , according to the data migration request, each child node in the first cluster writes their respective data in parallel to a designated path of the intermediate storage system.

[0028] The inter-cluster data migration method provided by the embodiment of the present invention can be used to perform data migration between the first cluster and the second cluster, for example, to migrate the data in the first cluster to the second cluster, after receiving the data migration request , each child node in the first cluster writes their respective data to the spec...

Embodiment 2

[0035] figure 2 A schematic flowchart of a method for migrating data between clusters according to Embodiment 2 of the present invention is shown. Such as figure 2 As shown, the method includes the following steps:

[0036] Step S200, according to the data migration request, start the data writing service for connecting each child node in the first cluster with the HDFS system.

[0037] Among them, the intermediate storage system includes: HDFS system. The HDFS system has the advantages of large bandwidth, large capacity, and large I / O throughput. Therefore, it can support each sub-node of the data storage system to write data into the HDFS system in parallel.

[0038] The following will take the HDFS system as an example for a detailed introduction.

[0039]After receiving the data migration request, each child node in the first cluster starts the data writing service used to connect the child node and the HDFS system according to the data migration request, wherein the ...

Embodiment 3

[0072] image 3 A schematic structural diagram of an inter-cluster data migration system according to Embodiment 3 of the present invention is shown. The system is used for data migration between the first cluster and the second cluster, such as image 3 As shown, the system includes: a first cluster 300, a second cluster 310, and an intermediate storage system 320, wherein the first cluster includes a master node 301 and at least one sub-node 302, and the second cluster includes a master node 311 and at least one A child node 312.

[0073] Each child node in the first cluster is suitable for writing respective data to a specified path of the intermediate storage system in parallel according to the data migration request;

[0074] The master node of the first cluster is suitable for the master node of the first cluster to back up the data table structure to the local master node in the form of a table file, and send the table file to the master node in the second cluster for t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an inter-cluster data migration method and system, a server and a computer storage medium. The method is used for performing data migration between a first cluster and a second cluster, and the first cluster and the second cluster each comprise a main node and at least one sub node; the method comprises the steps that according to a data migration request, data is written by the sub nodes in the first cluster concurrently into a designated path of an intermediate storage system; the main node of the first cluster backs up a data sheet structure to a main node locally in a list file mode, and sends a sheet file to the main node in the second cluster so that the main node in the second cluster can synchronize the data sheet structure to the sub nodes in the second cluster; the sub nodes in the second cluster reads and stores data from the designated path of the intermediate storage system currently according to a data redistribution strategy of the second cluster, the data migration speed is increased, and the time needed by data migration is shortened.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data migration method, system, server and computer storage medium between clusters. Background technique [0002] With the development of Internet applications, the number of users has increased sharply, and the amount of data storage has increased exponentially. Traditional single-database storage technologies cannot meet the access requirements of massive data. Distributed database applications have emerged, such as Greenplum Database. [0003] However, the actual application process inevitably involves data migration. For example, when a certain online cluster needs to be offline, or when the computer room is maintained and relocated, they will face the urgent task of massive data migration, and the data tables of the old cluster need to be migrated to In the new cluster, to continue to provide services. [0004] The existing technical solution is to directly tran...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/214G06F16/27
Inventor 张恒杨挺
Owner 360 TECH GRP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap