Distributed file capacity balancing method and device

A distributed file and balanced technology, applied in the direction of multi-program device, resource allocation, special data processing applications, etc., can solve the problems of small number of PGs, unbalanced distribution of PGs, and the inability of the system to continue writing, etc.

Inactive Publication Date: 2017-07-28
GUANGDONG INSPUR BIG DATA RES CO LTD
View PDF7 Cites 25 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since the PG distribution calculated by the crush algorithm cannot achieve a perfect balance, some OSDs have a large number of PGs, and some OSDs have a small number of PGs.
PG is the smallest logical unit of data storage. Therefore, when data is written into the distributed file system, the intuitive phenomenon is that the data distribution on the OSD is uneven. When the overall capacity is written to 80%, some PGs are more distributed. The actual usage of the OSD has reached 95%, causing the system to fail to continue writing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed file capacity balancing method and device
  • Distributed file capacity balancing method and device
  • Distributed file capacity balancing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0037] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0038] In the traditional method, the crush algorithm is used for PG distribution. Since the crush algorithm uses a pseudo-random method for PG distribution, the problem of unbalanced PG distribution ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a distributed file system capacity balancing method and device. All OSDs included in a storage pool are subjected to initialized PG distribution according to a crush algorithm, and the target average number of PGs in the storage pool is calculated; whether PG numbers corresponding to all the OSDs exceed the target average number or not is judged according to the PG distribution; if yes, it is shown that the PG distribution is not balanced at the moment, and the weights of the OSDs are adjusted according to a pre-stored weight adjusting command; every time the weights of the OSDs are adjusted, all the OSDs in the storage pool are subjected to the PG distribution again through the crush algorithm according to the adjusted weights, the adjusted PG distribution is obtained, whether the PG numbers corresponding to all the OSDs exceed the target average number or not is judged according to the PG distribution, if not, the weights of the OSDs stop being adjusted, and the PG distribution meeting capacity balance is obtained. Thus, by constantly adjusting the weights of the OSDs, the PG distribution is balanced, capacity balance is achieved accordingly, and the capacity utilization rate of a distributed file system is effectively increased.

Description

technical field [0001] The invention relates to the technical field of distributed file systems, in particular to a method and device for balancing capacity of a distributed file system. Background technique [0002] Distributed file system (Distributed File System, DFS) can effectively solve data storage and management problems, expand a file system fixed at a certain location to any number of locations / multiple file systems, and many nodes form a File system network. Each node can be distributed in different locations, and the communication and data transmission between nodes can be carried out through the network. [0003] After using the distributed file system to create a storage pool, PG (the smallest logical unit of data storage) members will follow the pseudo-random data distribution algorithm (Controlled Replication Under Scalable Hashing, crush) on the storage resource (Object Storage Device, OSD) Make a pseudorandom distribution. Since the PG distribution calcu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/50
CPCG06F9/5083G06F16/183
Inventor 王曙光岳远方
Owner GUANGDONG INSPUR BIG DATA RES CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap