Data Synchronization Method Based on High Bandwidth Storage System

Abstract

The invention provides a data synchronization method based on a high-bandwidth storage system. The method comprises the following steps: packaging a cluster into a fixed-length data segment, performing an erasure code algorithm, thus generating a plurality of coding objects, and dispersing the coding objects onto different nodes for storage; when a file reading request is received, analyzing a file identifier to inquire a cluster list of a corresponding file, then obtaining an identifier of a subordinate object and an identifier of an encoding object group, and storing the identifiers in a special structure; reading data of each cluster according to information in the structure; searching a position where the object is stored, then performing searching in a cluster index of the object according to a cluster identifier to find an offset address and a length of the cluster in the object, and finally reading data within the period; assembling the data into an original file according to a designated sequence in the structure. The data synchronization method based on the high-bandwidth storage system realizes organic combination of a high performance and low power consumption SSD (Solid State Drive) and a high-bandwidth disaster tolerant distributed storage architecture.

Description

technical field [0001] The invention relates to off-line storage, in particular to a data synchronization method based on a high-bandwidth storage system. Background technique [0002] Society has entered an era of data explosion. Ubiquitous sensors, mobile networks, social networks, Weibo, web page query and browsing records, call records, and consumption records generate a large amount of data all the time. The storage field is also facing many challenges in the era of big data. These challenges are not only about the storage of massive data, but also about providing corresponding support for upper-layer applications to efficiently process data from the aspects of architecture and system software to meet different upper-layer application requirements. The storage field is changing from the upper layer software system to the lower layer storage device. As we all know, SSD solid-state storage is compatible with traditional storage systems, and has the advantages of high p...

AI Technical Summary

Problems solved by technology

Most of the cloud computing platforms currently in use are designed based on mechanical disk devices. SSDs are completely different from disk devices in terms of internal mechanism, which makes the software system designed for mechanical disks not fully utilize the characteristics of SSDs.

Images