Dynamic linked storage cluster implementation method based on ceph

Abstract

The invention discloses a dynamic linked storage cluster implementation method based on ceph. The method includes a linked connected head cluster, and a hierarchical storage cluster formed by ordinaryclusters; corresponding cleaning strategies are arranged between the connected clusters; when the cleaning strategies are satisfied, data in the clusters are cleaned; the head cluster can achieve: strategy distribution including configuration and distribution of cleaning strategies between the clusters; data mapping in which all the service data IO passes through the head cluster and is mapped onto other clusters by the head cluster; and high-speed caching in which the head cluster performs high-speed caching on the data. The method can realize the high-performance hierarchical storage cluster which can be continuously and iteratively upgraded, and supports flexible dynamic expansion and contraction so as to reduce the cost, thereby achieving the goals of reasonable planning, predictable cost and giving consideration to performance.

Description

technical field [0001] The invention belongs to the technical field of data storage, and is designed and implemented to store data in a chained and hierarchical manner in complex application scenarios of massive data. Background technique [0002] As an excellent distributed file system, Ceph has high reliability. It can easily expand to several PBs while ensuring performance, fully meeting the needs of massive storage in big data and cloud computing. Now it has become an open source community. A well-known star project, especially with the development of cloud computing, Ceph has taken advantage of the spring breeze of OpenStack and has been welcomed by major manufacturers. Companies such as Intel, DreamHost, SanDisk, CISCO, Yahoo, etc. have participated more or less in. [0003] At present, with the increasing amount of data, while a large amount of data occupies storage space, some new or commonly used data needs to be frequently accessed as hot data. The common practice...

AI Technical Summary

Problems solved by technology

However, a large increase in data will inevitably bring a large amount of hot data. To ensure fast access to these hot data, the most direct way is to increase the capacity of the buffer cluster. However, this will inevitably lead to an increase in costs; In most application scenarios, the popularity of hot data is decreasing over time, such as: news, weather, and bank bills. The effect of the decreasing popularity of these data is called the diminishing value effect of data. This decline cannot be reflected in hot spot clusters, that is, hot and cold isolation. Manual intervention is time-consuming and labor-intensive and cannot be dynamically adjusted.

[0005] The industry has gradually realized this problem and proposed various hierarchical storage solutions. The general idea is to divide the combination of storage devices suitable for different performance, different storage capabilities, and different IO capabilities into multiple levels (generally three levels: high-performance , conventional, low-speed and high-capacity), but we know that the development of storage is changing with each passing day, new technologies emerge in an endless stream, performance continues to soar, and business requirements for performance are also rising; for example, in just a few years, the 4K random IOPS of SSD has increased from hundreds of thousands to It has soared to 2 million (Phison Element AIC SSD), and the capacity has expanded from several hundred G to as much as 8T (Intel recently announced a new form of SSD code-named "Ruler", which claims to be available in 1U. Astonishing capacity of 1PB (1000TB), I believe that this number will continue to be refreshed in the future when new technologies emerge, so it brings a very intuitive and realistic problem. At the beginning, the high-performance cluster that claimed to be at the highest level of hierarchical storage is in At present, it is far from being able to live up to its high-performance reputation, and it has become a drag on emerging businesses (such as: supercomputing field)

[0006] Maybe with the continuous adjustment of business needs, it is necessary to achieve the balance of cluster performance differences, or to expand the entire cluster chain to achieve higher capacity storage. At this time, we have to consider adding an ordinary cluster to the hierarchical cluster. This idea is in the current classification Storage is quite difficult, and it may be necessary to interrupt all running services, and the effect is unpredictable, and the risks caused by this are immeasurable

Images