Storage cluster multi-redundancy fault isolation method, system and device and medium

Abstract

The invention discloses a storage cluster multi-redundancy fault isolation method, system and device and a medium, and the method comprises the steps: dividing a storage cluster according to medium types, nodes or slot grouping, and forming a plurality of protection domains and / or fault domains enabling data resources to be isolated; the plurality of protection domains and / or fault domains for isolating the data resources are used for isolating different data resources in different storage pools according to different characteristics of the data resources and a hierarchical protection or access control strategy, so that the isolation of the protection domains and / or fault domains at a physical level is realized. Different types of resource pools can be created, and the data management capability, the system stability and the service continuity of the distributed storage cluster can be improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage, and in particular to a storage cluster multi-redundancy fault isolation method, system, equipment and medium. Background Art

[0002] At present, the existing multi-copy and erasure code data partitioning and slicing methods mainly include multi-copy redundancy and erasure code redundancy. Multi-copy redundancy is a simple and effective data redundancy strategy. When data is written to a storage cluster, it is copied to multiple storage devices to form multiple copies; these copies can be distributed on different fault domains, racks, and nodes to ensure data reliability and high availability; when a request is made to read data, the data can be returned from any copy. Erasure code redundancy is achieved by dividing the original data into several data blocks and generating some redundant data blocks, which are distributed to different storage devices. The erasure code algorithm can restore the lost or damaged original data through part of the redundant data blocks.

[0003] The space utilization rate of multi-copy storage is 1 / N, and the space utilization rate of N=K+M erasure code is K / N. Figure 2 The 5 nodes in the storage are 3+2 erasure codes, and the space utilization is 3 / 5. The two storage methods have different usage scenarios. The multi-copy method is mainly used in production systems with high performance and reliability requirements; the erasure code method is mainly used for cold data storage, such as backup, archiving, etc., which has high space utilization and no performance requirements.

[0004] The current redundancy scheme generally uses a single node (the vast majority) or a single cabinet as the fault domain. If there are three nodes in the cluster, erasure coding can create at most 2+1 erasure or three replicas. If a higher erasure code EC ratio or a higher number of replicas is required, it cannot be created, or the PG itself is downgraded after creation. Summary of the invention

[0005] The purpose of the present invention is to provide a storage cluster multi-redundancy fault isolation method, system, device and medium. The storage cluster of the present invention can achieve physical-level fault domain and protection domain isolation according to the business, thereby improving cluster stability and data durability.

[0006] To achieve the above purpose, the technical solution provided by the present invention is:

[0007] A first aspect of the present application provides a storage cluster multi-redundancy fault isolation method, which divides the storage cluster according to media type, node or slot grouping to form multiple protection domains and / or fault domains for isolating data resources; the multiple protection domains and / or fault domains for isolating data resources are to isolate different data resources in different storage pools according to different characteristics of data resources, hierarchical protection or access control strategies, so as to achieve isolation of protection domains and / or fault domains at the physical level.

[0008] To optimize the above technical solutions, the specific measures taken also include:

[0009] The storage cluster is divided according to the media type, node or slot grouping, and the storage cluster is divided horizontally and vertically, so that the disks of the storage cluster are divided into several disk groups; according to the storage media types in each storage pool isolated by the storage cluster, the corresponding storage protocol interface is used to support its storage service to carry the business data required by the service level agreement.

[0010] The data resources include computing resources and storage resources; logical partitions are formed by enabling different storage pools to support different redundancy strategies; and different storage pools are backed up separately.

[0011] The dividing of the storage cluster according to the media type, node or slot grouping to form multiple protection domains includes at least one of the following situations:

[0012] Divide the same storage cluster into several different fault domains based on nodes;

[0013] Divide the same storage cluster into several different protection domains based on nodes;

[0014] The same storage cluster is divided into different fault domains and protection domains based on nodes.

[0015] The isolating different data resources in different storage pools according to different characteristics, hierarchical protection or access control policies of data resources includes at least one of the following situations:

[0016] The sensitive data and the non-sensitive data are divided into different storage pools, and different security policies and permission controls are applied according to the needs;

[0017] Place hot data in a storage pool with high-performance storage devices, migrate cold data to a storage pool with low-cost storage devices, and apply different storage and network policies;

[0018] Divide large-scale files or data in systems with complex data into different storage pools, manage data in layers, and adopt different data strategies;

[0019] Divide the data with multiple units into different protection domains according to the units, and form a redundancy and backup strategy for the data of each unit;

[0020] Divide data in different regions into different storage pools for data replication and backup.

[0021] Acquire different business types and data; classify data; form a storage pool division strategy for protection domains and / or fault domains; perform performance stress testing and effective capacity assessment; create a resource pool; perform fault analysis; perform fault testing; perform business stress testing; and complete cluster creation.

[0022] A second aspect of the present application provides a storage cluster multi-redundancy fault isolation system, including:

[0023] Data acquisition module, used to obtain storage clusters of different business types and data;

[0024] A storage pool partitioning module is used to partition a storage cluster to form a storage pool of protection domains and / or fault domains; when partitioning, the storage cluster is divided according to the media type, node or slot grouping to form multiple protection domains and / or fault domains for isolating data resources; the multiple protection domains and / or fault domains for isolating data resources are to isolate different data resources in different storage pools according to different characteristics of data resources, hierarchical protection or access control policies, so as to achieve isolation of protection domains and / or fault domains at the physical level;

[0025] An evaluation module is used to evaluate the storage pool of the formed protection domain and / or fault domain.

[0026] A third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the storage cluster multi-redundancy fault isolation method as described above is implemented.

[0027] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the storage cluster multi-redundancy fault isolation method as described above.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention divides the storage cluster horizontally and vertically, splits the disk into multiple disk groups according to multiple nodes in the storage cluster, and creates one or more erasure code or replica resource pools on each disk group. Even if the number of replicas or erasure codes M+N>the number of NODEs, different types of resource pools can still be created to improve the SLA of the storage cluster.

[0030] The present invention simplifies the management and maintenance of storage clusters. Each fault domain and protection domain can be used as a separate management unit, which can be independently configured, monitored and maintained, reducing the complexity of management and providing better troubleshooting and debugging capabilities. The present invention can reduce the impact of disk and server failures on storage cluster services, achieve data balance in a more friendly way, and reduce the impact of data migration on all services of the storage cluster. The method of the present invention can improve data security and allow more disks or nodes to fail than in previous solutions.

[0031] The solution of the present invention is flexible and scalable, and can be customized according to actual needs and environment, so that the system can automatically expand according to changes in storage scale, and realize flexible management and optimization of resources while ensuring high availability. The present invention can tolerate the difference in disk media between nodes to a greater extent. For example, some nodes are equipped with SSDs, while some nodes are not. The SSDs on the nodes equipped with SSDs can be divided into a resource pool, etc.

[0032] The present invention can divide data into different types according to factors such as hardware performance and network topology to optimize the system, and improve the performance of the overall system by allocating high-performance devices and networks to key businesses or hot data and using lower-performance devices for cold data storage.

[0033] The present invention can be applied to a variety of application scenarios. As the scale of distributed storage cluster servers in public clouds or private clouds (enterprise self-built clouds) becomes larger and larger, more sophisticated and secure multi-redundant storage pool fault domain isolation strategies are needed to ensure that storage provides higher reliability, availability and data persistence; the relevant principles and methods of the present invention are also applicable to self-built distributed storage clusters, not just limited to distributed file systems (Ceph), but can be used for general distributed storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : A flow chart of the storage cluster multi-redundancy fault isolation method of the present invention.

[0035] Figure 2 : Schematic diagram of existing multi-copy and erasure code data partitioning and slicing methods.

[0036] Figure 3 : A schematic diagram of dividing a storage cluster according to nodes in a specific implementation manner of the present invention.

[0037] Figure 4 : Schematic diagram of logical partitioning supporting different business data in a specific implementation manner of the present invention.

[0038] Figure 5 : A schematic diagram of logical partitions supporting different copies in a specific implementation manner of the present invention.

[0039] Figure 6 : A specific flow chart of the storage cluster multi-redundancy fault isolation method in a specific implementation manner of the present invention.

[0040] Figure 7 : A structural diagram of the storage cluster multi-redundancy fault isolation system of the present invention. DETAILED DESCRIPTION

[0041] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0042] Some of the terms in the present invention are explained as follows:

[0043] RADOS (Reliable Autonomic Distributed Object Store): It provides distributed object storage capabilities and is used to store data and metadata.

[0044] OSD (Object Storage Device): OSD is a physical storage node in the Ceph cluster, responsible for storing and operating data.

[0045] PG (Placement Group): PG is the smallest unit of data in Ceph. Each PG contains a part of data and metadata, which are distributed on different OSDs.

[0046] CRUSH (Controlled Replication Under Scalable Hashing): CRUSH is an algorithm used by Ceph for data distribution and replica replication. It calculates which PG and OSD the data should be stored on based on a hash function.

[0047] RBD (RADOS Block Device): RBD is a block device interface provided by Ceph. It maps the Ceph storage system to a block device and can be used by virtual machines and physical servers.

[0048] SLA is the abbreviation of Service Level Agreement, which is a written contract or agreement that defines the specific service levels and performance standards that a provider must comply with when providing services to customers.

[0049] Replication: A data redundancy technique that involves storing multiple copies of the original data in different physical locations or storage nodes.

[0050] Erasure Coding: A data redundancy and recovery technique that divides the original data into multiple blocks and calculates redundant data blocks that are related to the original data blocks in some way.

[0051] In one embodiment, the present invention proposes a storage cluster multi-redundancy fault isolation method, such as Figure 1 As shown, the storage cluster is divided according to the media type, node or slot grouping to form multiple protection domains and / or fault domains for isolating data resources; the multiple protection domains and / or fault domains for isolating data resources are based on the different characteristics of data resources, such as different types of data such as K8S / VM, etc.), hierarchical protection or access control strategies, to isolate different data resources in different storage pools (storage pools are composed of multiple fault domains), thereby realizing the isolation of protection domains and / or fault domains at the physical level.

[0052] The present invention divides the storage cluster horizontally and vertically according to the nodes, so that the disk of the storage cluster containing multiple nodes is divided into several disk groups, such as Figure 3 As shown, the present invention divides the storage cluster according to different nodes in the storage cluster, and divides the storage cluster into storage pool 1, storage pool 2, ... storage pool N, so as to achieve complete isolation of computing resources and storage resources.

[0053] According to the different storage media in the storage cluster, such as SATA / SAS HDD, SATA SSD, PCIe NVMe, etc., it can be divided into full HDD, mixed flash, full flash, fast and slow pools. Through RBD / iSCSI, CephFS / NFS / CIFS, S3 / Swift, different storage services such as blocks, files, and objects are supported to carry business data with different SLA requirements.

[0054] The present invention can form logical partitions by enabling different storage pools to support different redundancy strategies, such as Figure 4 , Figure 5 As shown in the figure, different storage pools can support different redundancy strategies, such as multiple copies, erasure codes, etc. For example, business data A requires higher IOPS and low latency and flows to multi-copy hard disk pool 1, business data B requires higher effective capacity and BW and flows to hard disk pool 2, and business data C needs to be node-isolated from business data A and business data B to avoid mutual impact in the event of a failure and flows to hard disk pool 3.

[0055] The present invention achieves isolation of business data through fault domains and protection domains from both vertical and horizontal perspectives, ensuring business continuity and data reconstruction with minimal impact on business writing, reading, etc. when networks, disks or nodes of different nodes in the same storage cluster fail, thereby improving system reliability and availability, and simultaneously splitting the storage cluster horizontally and vertically to achieve isolation of different businesses of nodes and disks.

[0056] For example, the same storage cluster can be divided into several groups of different fault domains based on nodes. When the entire cabinet loses power or a cabinet-level failure occurs, the storage cluster service will not be interrupted or data will not be lost.

[0057] For example, the same storage cluster can be divided into several different protection domains based on nodes. Different protection domains can set different replication strategies. Redundant data copies are stored in the same protection domain. Data reconstruction caused by hardware failure in this protection domain is only within this protection domain and does not affect data usage in other domains.

[0058] The present invention can isolate the data of different applications by dividing different business applications or workloads into different fault domains and protection domains to prevent the problem of one application from affecting the data of other applications; at the same time, frequently accessed hot data and infrequently accessed cold data are written into different fault domains and protection domains to optimize performance and resource utilization; data with different security requirements can also be divided into separate protection domains and stricter access control and encryption policies can be applied;

[0059] In one disclosed embodiment of the present invention, Figure 6 As shown, the storage cluster multi-redundancy fault isolation method specifically includes the following steps: obtaining different business types and data; classifying data; forming a storage pool division strategy for protection domains and / or fault domains; performance stress testing, effective capacity assessment (CAP); resource pool creation; fault analysis; fault testing; business stress testing; and cluster creation is completed.

[0060] The solution of the present invention can provide high availability, data isolation and security, performance optimization, resource management and disaster recovery. These advantages help improve the data management capabilities, system stability and business continuity of distributed storage clusters, including:

[0061] Data isolation and security: Protection domains can isolate different data and resources, thereby achieving hierarchical protection and access control of data; sensitive data and non-sensitive data can be divided into different protection domains, and different security policies and permission controls can be applied according to actual needs to ensure data security;

[0062] High availability: By dividing data, nodes or disks into different protection domains, better data redundancy and backup can be achieved. Even in the event of failure or damage, data can still be restored from copies in other protection domains, ensuring high availability and continuity of the system.

[0063] Performance optimization: By dividing data into different protection domains, different storage and network policies can be applied according to the characteristics and access patterns of the data, thereby optimizing system performance. For example, hot data can be placed in high-performance storage devices and cold data can be migrated to low-cost storage devices to improve data access efficiency and response time.

[0064] Resource management and optimization: Protection domains can divide hardware resources into logically independent units, making resource allocation and management more centralized and refined, better managing system capacity, bandwidth, storage devices and other resources, improving resource utilization, and reducing resource redundancy and waste;

[0065] Disaster recovery and data backup: By dividing data into different protection domains and replicating across regions, disaster recovery and data backup capabilities can be provided. Even in the event of serious failures, disasters, or human errors, the backed-up data is still available, allowing for rapid recovery of business and data integrity.

[0066] The present invention is applicable to distributed storage cluster scenarios of various scales, so that data protection and availability can be better managed and controlled. Typical application scenarios include but are not limited to the following examples:

[0067] Multi-tenant environment: In a multi-tenant environment, different tenants may need to adopt different protection strategies for their data. By dividing the data of different tenants into different protection domains, data redundancy and backup strategies can be implemented for each tenant to ensure data security and availability.

[0068] Sensitive data isolation: Some applications may involve processing sensitive data, which requires strict isolation and protection. By dividing sensitive data into separate protection domains and applying stricter access control and encryption policies, the security of sensitive data can be ensured to prevent unauthorized access and leakage.

[0069] Complex data management: Some applications need to manage large amounts of complex data, such as large-scale file repositories or document archiving systems. By dividing this data into different protection domains, hierarchical data management and different data policies can be implemented, thereby improving the efficiency and controllability of data management.

[0070] Separation of hot and cold data: In some applications, some data is frequently accessed and updated, while other data is rarely accessed. By dividing this data into different protection domains, different storage strategies can be applied to hot and cold data, such as placing hot data on high-performance storage devices and migrating cold data to low-cost storage devices, to improve performance and resource utilization.

[0071] Regional and geographic distribution: For distributed and global applications, data in different regions or geographic locations can be divided into different protection domains to help achieve cross-regional data replication and backup to provide local data access and disaster recovery capabilities.

[0072] In another embodiment, the present invention provides a storage cluster multi-redundancy fault isolation system, comprising:

[0073] Data acquisition module, used to obtain storage clusters of different business types and data;

[0074] A storage pool partitioning module is used to partition a storage cluster to form a storage pool of protection domains and / or fault domains; when partitioning, the storage cluster is divided according to the media type, node or slot grouping to form multiple protection domains and / or fault domains for isolating data resources; the multiple protection domains and / or fault domains for isolating data resources are to isolate different data resources in different storage pools according to different characteristics of data resources, hierarchical protection or access control policies, so as to achieve isolation of protection domains and / or fault domains at the physical level;

[0075] An evaluation module is used to evaluate the storage pool of the formed protection domain and / or fault domain.

[0076] In another embodiment, the present invention proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above storage cluster multi-redundancy fault isolation method is implemented.

[0077] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above storage cluster multi-redundancy fault isolation method.

[0078] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0079] The present invention is further described in detail below with reference to specific application examples:

[0080] Example 1

[0081] The 18 servers are divided into three racks, and two storage pools are established on the application side to provide block storage customer form data (small IO structured data) and video surveillance data (large IO video large file data) to VM / K8S respectively.

[0082] Example 2

[0083] The six racks are divided into two node pools (storage pools). The node pools are physically isolated from each other. One node pool provides a storage resource pool for one K8S / VM and provides RBD block storage protocol services. The fault domain is set to the rack. Three copies of the same data are stored across different racks. For example, three copies of form data are placed on three racks respectively. At the same time, each node pool is divided into two protection domains. The different data shards of EC 2+1 of the monitoring data of business B (large IO video files) are distributed in one protection domain.

[0084] Example 3

[0085] By dividing the racks into rack one, rack two, etc., it can be achieved that a power failure of rack one as a whole will only affect one copy of the data; if the host or a single server is configured as the fault domain, multiple copies may lose power at the same time, resulting in business interruption or serious data loss; dividing the protection domain can achieve that if host one of rack one and host two of rack two fail at the same time, only one copy of the data will be affected; if no division is done, multiple copies of the data may be affected. If only the rack fault domain is divided, two copies of the same data may fall on host one of rack one and host two of rack two at the same time.

[0086] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention and based on the technical essence of the present invention, may make any simple modifications, equivalent substitutions and improvements to the above embodiments, which are still within the protection scope of the technical solution of the present invention.

Claims

1. A storage cluster multi-redundancy fault isolation method, characterized in that: The storage cluster is divided according to the media type, node or slot grouping to form multiple protection domains and / or fault domains for isolating data resources; the multiple protection domains and / or fault domains for isolating data resources are to isolate different data resources in different storage pools according to the different characteristics of the data resources, hierarchical protection or access control strategies, so as to achieve isolation of protection domains and / or fault domains at the physical level.

2. The storage cluster multi-redundancy fault isolation method according to claim 1, characterized in that: The storage cluster is divided into groups according to the media type, node or slot, and the storage cluster is divided horizontally and vertically, so that the disks of the storage cluster are divided into several disk groups; According to the types of storage media in each storage pool isolated by the storage cluster, the corresponding storage protocol interface is used to support its storage service to carry the business data required by the service level agreement.

3. The storage cluster multi-redundancy fault isolation method according to claim 1, characterized in that: The data resources include computing resources and storage resources; logical partitions are formed by enabling different storage pools to support different redundancy strategies; and different storage pools are backed up separately.

4. The storage cluster multi-redundancy fault isolation method according to claim 1, characterized in that: The said dividing the storage cluster according to the media type, node or slot grouping to form multiple protection domains includes at least one of the following situations: dividing the same storage cluster into several groups of different fault domains based on nodes; Divide the same storage cluster into several different protection domains based on nodes; The same storage cluster is divided into different fault domains and protection domains based on nodes.

5. The storage cluster multi-redundancy fault isolation method according to claim 1, characterized in that: The isolating different data resources in different storage pools according to different characteristics, hierarchical protection or access control policies of data resources includes at least one of the following situations: The sensitive data and the non-sensitive data are divided into different storage pools, and different security policies and permission controls are applied according to the needs; Place hot data in a storage pool with high-performance storage devices, migrate cold data to a storage pool with low-cost storage devices, and apply different storage and network policies; Divide large-scale files or data in systems with complex data into different storage pools, manage data in layers, and adopt different data strategies; Divide the data with multiple units into different protection domains according to the units, and form a redundancy and backup strategy for the data of each unit; Divide data in different regions into different storage pools for data replication and backup.

6. The storage cluster multi-redundancy fault isolation method according to claim 1, characterized in that: The specific steps include: Acquire different business types and data; classify data; form storage pool division strategies for protection domains and / or fault domains; perform performance stress testing and effective capacity assessment; and create resource pools; Fault analysis; fault testing; business stress testing; cluster creation completed.

7. The storage cluster multi-redundancy fault isolation system used in the method according to any one of claims 1 to 6, characterized in that: include: Data acquisition module, used to obtain storage clusters of different business types and data; A storage pool division module, used to divide the storage cluster into storage pools forming protection domains and / or fault domains; During the division, the storage cluster is divided according to the media type, node or slot grouping to form multiple protection domains and / or fault domains for isolating data resources; the multiple protection domains and / or fault domains for isolating data resources are to isolate different data resources in different storage pools according to the different characteristics of the data resources, hierarchical protection or access control strategies, so as to achieve isolation of protection domains and / or fault domains at the physical level.

8. The storage cluster multi-redundancy fault isolation system according to claim 7, characterized in that: Also includes: An evaluation module is used to evaluate the storage pool of the formed protection domain and / or fault domain.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the storage cluster multi-redundancy fault isolation method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the storage cluster multi-redundancy fault isolation method according to any one of claims 1 to 6.

Images