A model-variable data distribution method and system for distributed storage

A distributed storage and data distribution technology, applied in the direction of electrical digital data processing, input/output process of data processing, instruments, etc., can solve the problem that the consistent hash ring and CRUSH algorithm cannot be satisfied at the same time, and achieve increased data distribution balance High performance, high practicability, and the effect of reducing the pressure on the network and hard disk

Active Publication Date: 2020-06-19
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] At present, the classic distribution algorithms in the industry are the consistent hash ring and the CRUSH algorithm, which cannot satisfy the above three points at the same time.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A model-variable data distribution method and system for distributed storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0033] A data distribution method with a variable model for distributed storage, the realization process of which is as follows:

[0034] First, establish several virtual nodes in the cluster, and each virtual node has a corresponding data copy;

[0035] Build a mathematical model to generate a set of mappings from virtual nodes to actual hard disks;

[0036] Establish an evaluation function, and use the eva...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a model-variable data distribution method and system for distributed storage. The implementation method comprises the steps of firstly, establishing a plurality of virtual nodes in a cluster, wherein each virtual node has a corresponding data copy; constructing a mathematical model, and generating mapping sets from the virtual nodes to actual hard disks; establishing an evaluation function, utilizing the evaluation function to evaluate the data balance and data migration quantity conditions of the mapping sets from the virtual nodes to the actual hard disks, and outputting an evaluation value; obtaining an optimal set in all the mapping sets according to the evaluation value, and storing data to be stored into the hard disk corresponding to the virtual node in the mapping set. Compared with the prior art, the model-variable data distribution method and system for the distributed storage are applied to a distributed storage system, and the balance of data distribution is improved; the pressure of data migration networks and hard disks when a distributed storage structure changes is reduced; complex fault domain constraints are supported, and the probability of data loss is reduced.

Description

technical field [0001] The invention relates to the technical field of server clusters, in particular to a data distribution method and system with strong practicability and variable models for distributed storage. Background technique [0002] With the development of Internet technology in the past few decades, more and more Internet applications need to store massive amounts of data, such as search engines and Internet video sites. These requirements have given rise to a series of excellent large-scale distributed storage technologies. In recent years, in the context of cloud computing and big data, there are new requirements for data storage capacity, storage persistence, availability, and storage horizontal expansion capabilities. Distributed storage technology is used to meet cloud computing and big data Storage needs become an industry trend. Undoubtedly, the research and improvement of distributed storage systems are very important today. [0003] Distributed storag...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
CPCG06F3/0604G06F3/0631G06F3/0667G06F3/067
Inventor 窦伟平
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap