Eureka AIR delivers breakthrough ideas for toughest innovation challenges, trusted by R&D personnel around the world.

A distributed cluster data migration optimization method based on Ceph

A technology of distributed clustering and optimization method, applied in the field of Ceph-based distributed cluster data migration optimization, can solve the problems of high system consumption and too many migrations, and achieve the effect of improving availability, optimizing data migration, and preventing invalidity

Inactive Publication Date: 2019-02-15
电子科技大学成都学院
View PDF2 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is: the present invention provides a Ceph-based distributed cluster data migration optimization method to solve the problem of too m

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A distributed cluster data migration optimization method based on Ceph
  • A distributed cluster data migration optimization method based on Ceph
  • A distributed cluster data migration optimization method based on Ceph

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0063] Example 1

[0064] Such as Figure 1-5 As shown, the basic environment is composed of 3 nodes, each node has 3 device OSDs (50G), the number of replicas is set to 3, and the number of PGs is set to 664. When a device OSD is deleted, the migration amount generated by the traditional method is 420. The amount of migration using this method is 263, see the test results figure 2 .

[0065] The process is as follows: (Description: #Following is the execution command)

[0066] Set multiple flag bits for the device OSD in advance: flag bit norebalance-this flag bit will make the Ceph cluster not do any cluster rebalancing; flag bit nobackfill-this flag bit will make the Ceph cluster not do data backfilling; flag bit norecover-this flag The bit makes the Ceph cluster not do cluster recovery.

[0067] #Ceph osd set balance

[0068] #Ceph osd set nobackfill

[0069] #Ceph osd set norecover

[0070] Record the current PG distribution and save the migration data to the file pg1.txt:

[0071...

Example Embodiment

[0099] Example 2

[0100] Such as Figure 1-5 As shown, the basic environment used in test 2 consists of 2 nodes, each node has 4 device OSDs (50G), the number of copies is set to 2, the number of PGs is set to 664, the test results are shown in Figure 4 with Figure 5 .

[0101] a) Set the flag bit of the cluster to prevent migration:

[0102] Flag bit norebalance-this flag bit will make the Ceph cluster not do any cluster rebalance;

[0103] Flag bit nobackfill-this flag bit will make the Ceph cluster not do data backfill;

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed cluster data migration optimization method based on Ceph, belonging to the distributed cluster field of Ceph. The method comprises the following steps: 1, settinga flag bit of the PG; 2, stopping the OSD process of the device of the faulty node, and returning the weight of the device OSD to zero according to the flag bit algorithm CRUSH; 3, removing that OSDdescribed in the step 2 accord to the flag bit algorithm CRUSH; 4, adding a new OSD based on that step 3; 5, removing that flag bit of the PG; The invention solves the problem that the migration number of the PG is too much when the existing Ceph distributed storage fails, Replacing the faulty node triggers the data migration, which leads to the problem of large system consumption. It overcomes the excessive data migration load caused by Ceph storage, avoids the data loss caused by node failure, and effectively reduces the resource consumption of Ceph distributed storage.

Description

technical field [0001] The invention belongs to the field of Ceph distributed clusters, in particular to a Ceph-based distributed cluster data migration optimization method. Background technique [0002] In the Internet age, with the continuous development of cloud computing, the amount of global data has exploded, and the demand for big data storage has undergone tremendous changes; in terms of storage, Ceph is currently recognized as one of the excellent open source solutions, and its implementation The idea is that software-defined storage is SDS. Ceph provides unified, large-capacity, high-performance, and highly reliable file services by organizing the resources of multiple machines to meet the needs of large-scale applications. The architecture design can be easily extended to PB level; the logical storage unit of Ceph is the cluster PG (Placement Groups), referred to as PG. [0003] Ceph has developed the CRUSH (Controlled Replication Under Scalable Hashing) algorith...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/21G06F16/27
Inventor 黄曼绮李茂毅梁亮刘玉洁
Owner 电子科技大学成都学院
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com