Distributed data storage method and system based on erasure code

A technology of distributed data and erasure codes, which is applied in the field of distributed data storage methods and systems based on erasure codes, can solve the problems of user time extension, delay dependence, disk performance waste, etc., to achieve low delay and ensure Effects of Consistency and Completeness

Active Publication Date: 2016-07-20
SHENZHEN TENCENT COMP SYST CO LTD
View PDF4 Cites 21 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This will cause a huge waste of disk performance
And because there are too many I / O times for each download, this will also cause the delay of the download operation to depend on the slowest I / O operation, and the final delay reflected to the user is too long

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data storage method and system based on erasure code
  • Distributed data storage method and system based on erasure code
  • Distributed data storage method and system based on erasure code

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0028] figure 2 It is a flow chart of the erasure code-based distributed data storage method provided in the first embodiment of the present invention, image 3 It is a sequence diagram of the erasure code-based distributed data storage method provided by the first embodiment of the present invention. Please refer to figure 2 and image 3 , the present embodiment mainly describes the flow process of data writing (uploading files), and the method in the present embodiment includes the following steps:

[0029] In step S11, the client sends a data write request to the cache server, and the data write request includes data to be stored.

[0030] Before making a data write request, the client can fragment a large file, that is, divide a large file into multiple fragments, so as to ensure the performance balance of the overall access. The size of the slice can be set according to the actual situation, and the specific implementation manner of the present invention is not limi...

no. 2 example

[0048] Figure 5 It is a flow chart of the erasure code-based distributed data storage method provided by the second embodiment of the present invention; Figure 6 It is a sequence diagram of the erasure code-based distributed data storage method provided by the second embodiment of the present invention. This embodiment is based on the first embodiment, and mainly describes the main process of data acquisition (file download), please refer to Figure 5 and Figure 6 , the method in this embodiment includes the following steps:

[0049] In step S21, the client sends a data acquisition request to the cache server, and the data acquisition request includes the key value of the data to be acquired.

[0050] In step S22, the cache server checks whether the data to be acquired is locally stored according to the key value, and if so, directly returns the data to be acquired to the client, otherwise, executes step S23.

[0051] Step S23, the cache server sends the data acquisitio...

no. 3 example

[0056] Figure 7 It is a flow chart of the erasure code-based distributed data storage method provided by the second embodiment of the present invention; Figure 8 It is a sequence diagram of the erasure code-based distributed data storage method provided by the second embodiment of the present invention. This embodiment is based on the first embodiment, and mainly describes the main process of data deletion (file deletion), please refer to Figure 7 and Figure 8 , the method in this embodiment includes the following steps:

[0057] Step S31, the client sends a data deletion request to the cache server, and the data deletion request includes the key value of the data to be deleted;

[0058] Step S32, the cache server finds and deletes the locally stored data to be deleted according to the key value;

[0059] Step S33, the cache server sends the data deletion request to the storage server;

[0060] Step S34, the storage server searches for and deletes data index informati...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a distributed data storage method and system based on an erasure code. The method comprises the steps that a client writes to-be-stored data in a cache server; the cache server caches the to-be-stored data; when a preset condition is satisfied, a relay server extracts cache data with a predetermined size from the cache server, encodes the cache data in an erasure code encoding mode, and generates calibration data; and the relay server writes the cache data in a plurality of data blocks of a storage server, and writes the calibration data in at least one calibration block of the storage server.

Description

technical field [0001] The present invention relates to computer internet technology, in particular to a distributed data storage method and system based on erasure codes. Background technique [0002] A distributed storage system stores data dispersedly on multiple independent devices. Traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, which cannot meet the needs of large-scale storage applications. The distributed network storage system adopts a scalable system structure, uses multiple storage servers to share the storage load, and uses the location server to locate and store information. It not only improves the reliability, availability and access efficiency of the system, but also is easy to expand. [0003] The current mainstream solutions for distributed storage all use a multi-copy storage method, that is, a complete copy of busi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08H04L1/00
Inventor 王炎
Owner SHENZHEN TENCENT COMP SYST CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap