High-efficiency and high-reliability big data storage system and method and computer program

Abstract

The invention belongs to the technical field of information data processing, and discloses a high-efficiency and high-reliability big data storage system and method, and a computer program. The high-efficiency and high-reliability big data storage method includes the steps: a configuration center is set to obtain the data uploaded by a user; an arraying module is used for calculating a coded array; the array calculated by the array module is read, and encoding is performed according to the array; heartbeat information is sent between the master node and the slave nodes, and heartbeat delay time of the configuration center is read; failure node information is analyzed; data recovery is carried out; the position of each original data block is queried according to the ID of the data to be downloaded; the setting of the configuration center is read for distributing the coded block storage nodes and newly-generated nodes selected during data recovery; the configuration center is responsiblefor managing various parameters in the system; and the coded blocks are stored. The invention provides a big data storage solution with optimal storage efficiency, high efficiency, high reliability and easy expansion.

Description

technical field [0001] The invention belongs to the technical field of information data processing, and in particular relates to an efficient and highly reliable large data storage system, method and computer program. Background technique [0002] Currently, the closest prior art: [0003] With the continuous development of information technology in various industries and fields, the data volume shows an exponential growth trend, and the rapid growth of data volume brings about data storage reliability and data access concurrency efficiency issues. A generally effective method is to construct a distributed storage system composed of multiple data nodes (a data node can be a PC or a server, etc., which can be used as a data storage device). The continuous increase in the amount of stored data in various industries has led to an increasing scale of distributed storage systems and an increasing number of nodes. Some existing enterprises already have multiple storage systems w...

AI Technical Summary

Problems solved by technology

Although this type of array code has improved fault tolerance, it will face various problems when it is applied to a distributed storage system. It can only be realized if there is a linear relationship with the prime number, which undoubtedly brings great restrictions to the expansion of the distributed storage system, which does not match the easy scalability of the distributed storage system

Second, storage efficiency will decrease as the number of fault tolerances increases

For example, if we design a Weaver code that can tolerate 10 errors, its storage efficiency is less than 20%.

Therefore, these problems lead to the low practicality of array codes in the field of distributed storage systems

[0007] To sum up, the problems existing in the existing technology are: the space utilization rate of the fault-tolerant technology in the current distributed storage system is low, the storage efficiency is low, the reliability is poor, the scalability is poor, and the practicability is not high

[0009] One is that there are restrictions on the number of stripes or blocks, which can only be realized if the stripes or blocks are prime numbers or have a linear relationship with prime numbers. This undoubtedly brings great restrictions on the expansion of the distributed storage system. The ease of expansion of the storage system does not match

Second, storage efficiency will decrease as the number of fault tolerances increases

For example, if we design a Weaver code that can tolerate 10 errors, its storage efficiency is less than 20%.

When expanding a large-scale distributed storage system, if you still expand according to prime numbers, you need to add more nodes or virtual nodes. More nodes mean an increase in the cost of a system expansion. More virtual nodes require Waste more calculations and affect efficiency

Images