Eureka AIR delivers breakthrough ideas for toughest innovation challenges, trusted by R&D personnel around the world.

Strip-based consistent hash storage system construction method

A storage system and construction method technology, applied in the field of data placement mechanism, can solve problems such as inability to adapt to data recovery functions, and achieve the effects of ensuring the nature of MDS, improving stability, and reducing data volume

Active Publication Date: 2019-07-23
INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF11 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Because the location of a large number of data blocks has changed, additional data migration overhead is generated during the recovery process, which cannot meet the needs of the data recovery function.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Strip-based consistent hash storage system construction method
  • Strip-based consistent hash storage system construction method
  • Strip-based consistent hash storage system construction method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0056] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0057] Basic principle of the present invention is as follows:

[0058] First, build a storage system, construct a ring-shaped hash space according to the design of the consistent hash algorithm, generate a certain number of virtual nodes, and distribute these virtual nodes to the hash space evenly, and construct the storage nodes in the storage system into A number of node groups satisfying the properties of erasure code MDS, and hashing the node groups to the hash space, establishing the mapping relationship between virtual nodes and storage node groups on the hash space; each virtua...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

According to the strip-based consistent hash storage system construction method and the corresponding data placement mechanism and node change method provided by the invention, data blocks are organized by taking strips as units, storage nodes are organized by taking node groups as units, and the strips are placed on the node groups. When the nodes are organized into node groups, the number of thesame nodes in each node group is not greater than the number of the check blocks in the strip, so that the placement of the data blocks is ensured to meet the MDS property of the erasure codes, and the reliability of data storage is ensured. Meanwhile, a consistency hash algorithm is adopted; the node group with the lowest difference degree is selected to replace the node groups; through a consistency hash algorithm, only the placement positions of data on part of virtual nodes are changed, a node group with the lowest difference degree is selected as a replacement node group, only nodes at the position of a change node are different, the nodes at other corresponding positions are the same, and at the moment, the migrated data amount is the minimum.

Description

technical field [0001] The present invention relates to the fields of data storage and erasure codes, in particular to a method for constructing a stripe-based consistent hash storage system and a corresponding data placement mechanism. Background technique [0002] In the era of big data, the storage of massive data is facing the contradiction between storage reliability and space utilization. The erasure code storage method, which not only has high space utilization efficiency, but also ensures the reliability of data storage, is more and more used in storage systems, for example, in Google's GFS, Microsoft's Azure and Facebook's storage System and other commercial systems have applications. [0003] Erasure codes are similar to RAID. Data is grouped into stripes (Stripes). Each stripe contains N blocks of data. The data is encoded by an encoding matrix to generate N encoding blocks and M check blocks, which are collectively referred to as data blocks. The encoded block ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/22G06F11/10
CPCG06F11/10G06F16/2255
Inventor 魏征窦禹邢晶霍志刚马捷
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com