A data balancing method and device for write-time redirection distributed storage engine

Abstract

The present application relates to the technical field of distributed storage, and discloses a data balancing method and device for a write-time redirection distributed storage engine, which comprises the following steps: performing a preset disk selection balancing strategy based on a logical topology structure; obtaining a PG used capacity and a hard disk used capacity after performing the preset disk selection balancing strategy; performing a preset space allocation balancing strategy based on the PG used capacity and the hard disk used capacity; performing a preset use balancing strategy based on a star-type read-write mode after performing the preset space allocation balancing strategy; and performing a preset background balancing strategy in the case that the cluster is still not balanced after performing the preset use balancing strategy. The present application can effectively realize the balanced distribution of cluster data by adjusting the disk selection balancing, space allocation balancing, use balancing and background balancing, and monitoring the cluster hardware through a unified cluster management center.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and specifically to a method and apparatus for data balancing in a write-time redirection distributed storage engine. Background Technology

[0002] Distributed storage, also known in the storage industry as Software Defined Storage (SDS), is defined by the Storage Networking Industry Association (SNIA) as: a type of virtualized storage with a service management interface. SDS includes storage pooling functionality and allows for the definition of data service characteristics within the storage pool through the service management interface. The most crucial aspect of virtualized storage is the virtualization of the storage hardware. Compared to traditional storage, distributed storage no longer relies on proprietary hardware (such as storage controllers) and proprietary networks (such as FC networks), and can provide professional storage services through general-purpose hardware and a unified network.

[0003] Distributed storage technology is now widely used in traditional data centers, public cloud data centers, private cloud data centers, and hyperconverged workstations. Among them, the distributed storage engine provides high-performance, highly reliable, and efficient data access capabilities for distributed storage, and is the core component of distributed storage.

[0004] Distributed storage engines offer various implementation options, such as the widely used Ceph distributed storage system and the Google File System (GFS). Both employ a distributed storage architecture, enabling data to be distributed across multiple servers, making them suitable for scenarios like big data analytics, cloud computing, and large-scale data storage. However, they both face the problem of data imbalance, meaning that data is unevenly distributed among the nodes in the distributed system. This can cause some nodes to bear an excessive storage or processing burden, significantly impacting the system's performance, scalability, and fault tolerance. Summary of the Invention

[0005] In view of this, the present invention provides a method and apparatus for data balancing in a distributed storage engine with write-time redirection, in order to solve the problem of data imbalance in the distributed storage engine in the prior art.

[0006] In a first aspect, the present invention provides a data balancing method for a write-time redirection distributed storage engine, the method comprising:

[0007] The preset disk selection and balancing strategy is executed based on the logical topology. The logical topology is one of several different types of logical topologies determined after reorganizing the physical topology of the heterogeneous cluster, and each logical topology includes multiple logically isolated sub-topologies.

[0008] After executing the preset disk balancing strategy, obtain the used capacity of the PG and the used capacity of the hard drive;

[0009] Based on the used capacity of PG and the used capacity of hard disk, execute the preset space allocation balancing strategy;

[0010] After executing the preset space allocation balancing strategy, the preset usage balancing strategy is executed based on the star topology read / write method;

[0011] If the cluster still fails to achieve balance after executing the preset load balancing strategy, execute the preset background load balancing strategy.

[0012] This invention proposes a hierarchical data balancing method for write-time redirection distributed storage engines. It breaks down cluster data balancing into four levels: disk selection balancing, space allocation balancing, usage balancing, and background balancing adjustment. A unified cluster management center monitors the cluster hardware, and combined with the ROW storage engine and hierarchical storage architecture, achieves data balancing within the cluster. When applied to a distributed storage engine, this method distributes the capacity pressure on each storage medium and the performance pressure across each network, computing, and storage hardware within the cluster as evenly as possible. In other words, it achieves balanced data distribution within the cluster through fine-grained control of the cluster by a unified cluster management center.

[0013] In one optional implementation, a preset disk selection and balancing strategy is executed based on the logical topology, including:

[0014] Step a: Determine the weight and fault domain of each topology node in the logical topology structure;

[0015] Step b: Select the Nth layer in the logical topology, where N = 1;

[0016] Step c: Determine the first topology node in the maximum weight set of the Nth layer and the fault domain corresponding to the first topology node; the first topology node is any node in the maximum weight set.

[0017] Step d: If the fault domain corresponding to the first topology node is the set fault domain, determine whether the first topology node meets the fault requirements.

[0018] Step e1: If the fault requirements are met, select the first topology node and set N = N + 1. Return to step c until N is the last layer number in the logical topology structure, and complete the selection of the logical topology.

[0019] Step e2: If the fault requirements are not met, return to step b and take the second largest weight set in the Nth layer as the largest weight set.

[0020] Disk selection balancing organizes hardware of the same type into a logical topology. Large heterogeneous storage server clusters are managed through logical topology isolation and storage federation, elevating the load balancing of the heterogeneous cluster to a usage-balanced level. Disk selection balancing only considers the balancing of hard drives of the same type. The number of available PGs in the cluster is determined by disk selection within the storage cluster, ensuring that each hard drive appears an equal number of times in each position within the PG. Even in EC (Extended Read / Write) scenarios, it can maintain a relatively even distribution of read and write operations across the cluster.

[0021] In one optional implementation, after completing the disk selection for the logical topology, the method further includes:

[0022] The check shards and data shards of all PGs in the cluster are readjusted so that each hard drive in the logical topology is used as an EC check shard and EC data shard an equal number of times.

[0023] EC scenario disk selection ensures that all disks are evenly distributed across EC's check shards and data shards from the perspective of logical topology cluster, thereby effectively improving the overall performance balance of the cluster.

[0024] In one optional implementation, a preset space allocation balancing strategy is executed based on the used capacity of the PG and the used capacity of the hard disk, including:

[0025] Determine whether the difference in used capacity of PG exceeds the first preset capacity;

[0026] If the difference in the used capacity of PG exceeds the first preset capacity, it is determined whether the time exceeding the first preset capacity exceeds the first preset time.

[0027] If the time exceeds the first preset capacity, a first cyclic PG group is established. The first cyclic PG group is used to balance the used capacity of PGs.

[0028] Determine whether the difference in used hard drive capacity exceeds the second preset capacity;

[0029] If the difference in the used capacity of PG exceeds the second preset capacity, determine whether the time exceeding the second preset capacity exceeds the second preset time.

[0030] If the time exceeds the second preset capacity, a second cyclic PG group is established on the hard drive that exceeds the second preset capacity and the second preset time. The second cyclic PG group is used to balance the used capacity of the hard drive.

[0031] In one optional implementation, based on the star topology read / write method, a preset usage balancing strategy is executed, including:

[0032] Obtain the I / O data for each hard drive in a star topology read / write mode;

[0033] Based on IO data, the data types with different popularity levels are determined;

[0034] Data types with different popularity are assigned to the corresponding logical topology based on the type of the logical topology.

[0035] In this embodiment, a star topology is used for read / write operations, without a master node responsible for forwarding read / write operations. In scenarios with high read volumes, the master node will not become a read hotspot. When heterogeneous servers exist within a cluster, logical topology isolates servers with different structures. When using space, different topologies are used based on the characteristics of the business to achieve heterogeneous cluster load balancing, which can further effectively optimize data balance.

[0036] In one optional implementation, a preset backend load balancing strategy is executed, including:

[0037] Obtain statistical information for all PGs and all hard drives within the cluster;

[0038] Based on the statistical information of all PGs, hot PGs are identified;

[0039] Based on the statistical information of all hard drives, identify the hottest hard drives;

[0040] If a hot PG and a hot hard drive continuously exhibit a third preset time, the hot PG will be migrated from the hot hard drive to a non-hot hard drive through PG migration.

[0041] Migrate hot spot PGs to non-hot spot hard drives. Since the number of PGs to be migrated and the migration bandwidth can be set, the impact on the entire cluster is small and controllable, and it can further improve data balance.

[0042] In a second aspect, the present invention provides a data balancing device for a write-time redirection distributed storage engine, the device comprising:

[0043] The balancing strategy execution module is used to execute a preset disk selection balancing strategy based on the logical topology. The logical topology is one of several different types of logical topologies determined after reorganizing the physical topology of the heterogeneous cluster, and each logical topology includes multiple logically isolated sub-topologies.

[0044] The space allocation balancing strategy execution module is used to obtain the used capacity of the PG and the used capacity of the hard disk after executing the preset disk selection balancing strategy; and to execute the preset space allocation balancing strategy based on the used capacity of the PG and the used capacity of the hard disk.

[0045] The load balancing strategy execution module is used to execute the preset load balancing strategy based on the star topology read / write method after executing the preset space allocation load balancing strategy.

[0046] The background load balancing strategy execution module is used to execute a preset background load balancing strategy if the cluster still fails to achieve load balancing after executing the preset load balancing strategy.

[0047] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the write-time redirection distributed storage engine data balancing method described in the first aspect or any corresponding embodiment thereof.

[0048] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the write-time redirection distributed storage engine data balancing method described in the first aspect or any corresponding embodiment thereof.

[0049] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the write-time redirection distributed storage engine data balancing method described in the first aspect or any corresponding embodiment thereof.

[0050] It should be noted that the write-time redirection distributed storage engine data balancing device, computer device, and computer-readable storage medium provided in this invention correspond to the write-time redirection distributed storage engine data balancing method described above. Therefore, for the beneficial effects of the write-time redirection distributed storage engine data balancing device, computer device, and computer-readable storage medium, please refer to the description of the corresponding beneficial effects of the write-time redirection distributed storage engine data balancing method above, and will not be repeated here. Attached Figure Description

[0051] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating a data balancing method for a write-time redirection distributed storage engine according to an embodiment of the present invention.

[0053] Figure 2This is a schematic diagram of object topology according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of one of the logical topologies according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the selection panel according to an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of verification fragmentation and data fragmentation readjustment according to an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram showing the adjustment of the verification fragment and data fragment according to an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the PG cycle according to an embodiment of the present invention;

[0059] Figure 8 This is a schematic diagram of star-shaped read / write operation according to an embodiment of the present invention;

[0060] Figure 9 This is a schematic diagram illustrating the use of equalization according to an embodiment of the present invention;

[0061] Figure 10 This is a schematic diagram of PG migration according to an embodiment of the present invention;

[0062] Figure 11 This is a structural block diagram of a write-time redirection distributed storage engine data balancing device according to an embodiment of the present invention;

[0063] Figure 12 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Currently, Ceph uses relatively coarse-grained management for cluster balancing, which means that the hardware cannot fully utilize its capabilities during actual production operation. It requires maintenance by people with design and code-level understanding of Ceph to alleviate the data balancing problem to some extent. Furthermore, due to the Ceph architecture, some optimizations, such as data sharding and checksum sharding balancing in EC, cannot be achieved.

[0066] GFS offers more granular management, enabling cluster data balancing through both space allocation and background adjustment. However, it consumes significant master memory resources, and as GFS is primarily a replica architecture, it lacks design for data sharding and parity sharding balancing for EC (Extended Equivalent), thus failing to achieve optimal data balancing.

[0067] In view of this, according to an embodiment of the present invention, an embodiment of a data balancing method for a write-time redirection distributed storage engine is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0068] This embodiment provides a data balancing method for a write-time redirection distributed storage engine, which can be executed by a cluster management center in devices such as servers, terminals, and mobile terminals. Figure 1 This is a flowchart of a data balancing method for a write-time redirection distributed storage engine according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0069] Step S101: Execute a preset disk selection and balancing strategy based on the logical topology. The logical topology is one of several different types of logical topologies determined after reorganizing the physical topology of the heterogeneous cluster, and each logical topology includes multiple logically isolated sub-topologies.

[0070] The physical topology reflects the mapping relationship between physical hardware servers and hard drives. All server information and hard drive information managed by the servers within the cluster are collected at the chunkmaster (the chunkmaster, meaning the cluster management center, is responsible for data distribution, data balancing, fault recovery, cluster scaling, cluster monitoring, virtual space allocation, and virtual space to physical space conversion in the distributed storage engine of this embodiment; different storage engines use different naming methods, such as master and CM (cluster master)). Server information includes CPU, memory, number of hard drives, network bandwidth information, and hard drive information. Hard drive information includes: hard drive medium (e.g., TLC SSD, QLC+SLC SSD, HDD, etc.), hard drive protocol (SAS, SATA, NVMe), and hard drive capacity. See the schematic diagram of the object topology. Figure 2As shown, logical topology groups similar hardware into a logical topology pool based on physical topology and actual reliability requirements. Each logical topology is a logically isolated fault domain, with computing, network, and storage resources logically isolated. In other words, servers of the same type are organized into logically isolated logical topology structures, with one ultra-high-performance logical topology structure referencing... Figure 3 As shown, logical topology can also include high-performance logical topology, ordinary logical topology, and large IO capacity logical topology.

[0071] In different logical topologies, balanced disk selection is performed. Since the physical hardware characteristics are consistent within a topology, only the disk weight needs to be used as the selection criterion to complete the disk selection for the logical topology. The disk weight is calculated as capacity / number of PGs. After logical topology disk selection, because the disks are randomly selected, under EC (Erasure Code) protection mode, from the cluster's perspective, all disks may not be evenly distributed as EC check shards and EC data shards. Therefore, further balancing adjustments are needed to ensure that each disk is used as an EC check shard and EC data shard an equal number of times, thus completing the disk selection balancing strategy. (Erasure Code is an encoding technique that adds m data to n original data sets and can restore the original data from any n data sets from the n+m sets. Mathematically, encoding is represented by constructing a multivariate linear equation, and decoding is by solving the multivariate linear equation. Further abstraction leads to matrix operations. Current erasure coding techniques are mainly divided into two categories: one is erasure coding based on Galois field operations (such as RS); the other is erasure coding based on "XOR" operations (LDPC).

[0072] Step S102: After executing the preset disk balancing strategy, obtain the used capacity of the PG and the used capacity of the hard disk.

[0073] Step S103: Based on the used capacity of PG and the used capacity of hard disk, execute the preset space allocation balancing strategy.

[0074] In this embodiment, virtual addresses (chunks) are uniformly allocated by the cluster management center. A chunk (meaning a large block, in this embodiment, it describes a byte stream space allocated from a PG (Protect Group, in this embodiment, it describes a logical protection group composed of a group of independent virtual hard disks organized together using ECs or replicas to provide reliability guarantees and provide read / write space externally) is typically a maximum of 128MB, allocated from the PG and guaranteed for reliability by the PG), and is distinguished by a unique identifier (chunkID). Large-loop allocation prioritizes chunk allocation based on the principle that the number of chunks allocated on each PG is equal. For a normally functioning cluster, the number and capacity of chunks are the same. When the difference in used capacity of a PG reported by a chunkserver (meaning a single-machine storage server, in this embodiment, it describes a single-machine storage server that manages the storage space on the single-machine server and manages the single-machine byte stream chunk space) exceeds 10GB and remains so for more than one day, a "small-loop PG group" is established to catch up. When the monitoring shows that the difference in used hard drive capacity reported by a single storage server exceeds 10G and remains so for more than 2 days, a "small circular PG group" is established on these hard drives to catch up in capacity, thereby achieving a space allocation balance strategy.

[0075] Step S104: After executing the preset space allocation balancing strategy, execute the preset usage balancing strategy based on the star topology read / write method.

[0076] Reading and writing to a chunk actually involves reading and writing to a group of independent virtual hard disks. Each virtual hard disk only handles reads and writes related to itself. I / O data aggregation and distribution are handled by an external client independent of the virtual hard disks. This read / write method is called star topology in this embodiment. In this embodiment, once data is written to a chunk location by the ROW (Real-Time Writer), that data cannot be changed, and star topology is supported. Combined with evenly distributed parity disks, further balancing can be achieved. That is, when heterogeneous servers exist within a cluster, logical topology is used to logically isolate servers with different structures. When using space, different logical topologies are used based on the characteristics of the data types to achieve balancing of the heterogeneous cluster.

[0077] Step S105: If the cluster still fails to achieve balance after executing the preset load balancing strategy, execute the preset background load balancing strategy.

[0078] Specifically, the cluster management center collects statistical information on all PGs and hard drives within the cluster. Based on this information, it identifies hot PGs and hard drives that become hot due to sudden traffic spikes. If hot PGs and hard drives persist, PG migration is used to move them from hot hard drives to non-hot hard drives, thereby achieving performance and capacity balance within the cluster.

[0079] In related technologies, such as Google's publicly disclosed distributed storage engine GFS (Google File System), which provides an append-only file system, GFS uses a unified management node to manage the file-to-space mapping metadata. Data block information is queried by the management node from the specific data block server at startup. Data balancing is achieved through the management node's balanced allocation of data blocks and direct migration of data blocks. This results in a huge number of data blocks that the management node needs to manage when the GFS cluster is large. Furthermore, because it provides data as files, the memory consumption of data blocks and file information is significant. The lack of a multi-logical topology concept leads to complex handling when heterogeneous servers appear in the cluster. Another widely used technology is Ceph. Ceph is a distributed storage system that uses a distributed algorithm (CRUSH algorithm) to distribute data objects according to the weight of each storage device, making the distribution approximately uniform. Ceph calculates a list of OSDs (Object Storage Devices) for reading and writing. Since these OSD lists are calculated, it is difficult to guarantee that each OSD appears consistently at each location. Therefore, during normal EC (Extra Check) reads in the cluster, the absence of OSDs used for verification during reads causes uneven read distribution. Furthermore, because the number of Placement Groups (PGs) in Ceph is limited compared to the vast number of objects, each OSD (Optical Server Device) may act as the master PG at different times. Since Ceph handles both read and write operations through the master OSD, this can lead to cluster imbalance. While adjusting weights can address this imbalance, it causes data migration across multiple PGs, making fine-grained control as difficult as a centralized management cluster.

[0080] This invention proposes a hierarchical data balancing method for write-time redirection distributed storage engines. It breaks down cluster data balancing into four levels: disk selection balancing, space allocation balancing, usage balancing, and background balancing adjustment. A unified cluster management center monitors the cluster hardware, and combined with the ROW storage engine and hierarchical storage architecture, achieves data balancing within the cluster. When applied to a distributed storage engine, this method distributes the capacity pressure on each storage medium and the performance pressure across each network, computing, and storage hardware within the cluster as evenly as possible. In other words, it achieves balanced data distribution within the cluster through fine-grained control of the cluster by a unified cluster management center.

[0081] In some optional implementations, a preset disk selection and equalization strategy is executed based on the logical topology, including:

[0082] Step a: Determine the weight and fault domain of each topology node in the logical topology structure;

[0083] Step b: Select the Nth layer in the logical topology, where N = 1;

[0084] Step c: Determine the first topology node in the maximum weight set of the Nth layer and the fault domain corresponding to the first topology node; the first topology node is any node in the maximum weight set.

[0085] Step d: If the fault domain corresponding to the first topology node is the set fault domain, determine whether the first topology node meets the fault requirements.

[0086] Step e1: If the fault requirements are met, select the first topology node and set N = N + 1. Return to step c until N is the last layer number in the logical topology structure, and complete the selection of the logical topology.

[0087] Step e2: If the fault requirements are not met, return to step b and take the second largest weight set in the Nth layer as the largest weight set.

[0088] Reference Figure 4 As shown, this embodiment uses a greedy algorithm for selecting the disk:

[0089] Step 1: Starting from the root layer, randomly scatter the topological points layer by layer and select them sequentially according to the weight order of the logical topology points;

[0090] Step 2: When selecting down to the next level, if the logical topology point is found to be a fault domain, decide whether to select it according to the fault domain requirements;

[0091] Step 3: After cleaning up the fault domain that does not meet the requirements, return to Step 1 and select the next weighted logical topology point.

[0092] Step 4: When the fault domain requirements are met, the layer is randomly shuffled downwards, and the logical topology point with the highest weight is selected until the hard drive is selected.

[0093] Disk selection balancing organizes hardware of the same type into a logical topology. Large heterogeneous storage server clusters are managed through logical topology isolation and storage federation, elevating the load balancing of the heterogeneous cluster to a usage-balanced level. Disk selection balancing only considers the balancing of hard drives of the same type. The number of available PGs in the cluster is determined by disk selection within the storage cluster, ensuring that each hard drive appears an equal number of times in each position within the PG. Even in EC (Extended Read / Write) scenarios, it can maintain a relatively even distribution of read and write operations across the cluster.

[0094] In some optional implementations, after completing the selection of disks for the logical topology, the following steps are also included:

[0095] The check shards and data shards of all PGs in the cluster are readjusted so that each hard drive in the logical topology is used as an EC check shard and EC data shard an equal number of times.

[0096] In the storage field, it has been found that EC protection groups can provide higher read concurrency and higher disk utilization than replicas (three replicas <33%, EC 4+2 can provide >60% utilization). The limitation of writes requiring full-strip writes by EC is gradually being resolved by the industry, and the use of EC protection groups in distributed storage engines is becoming the mainstream in the distributed storage industry. EC data shards support cluster reads, while parity shards only support repair reads. When parity shards and data shards are unevenly distributed on disks, the disk read pressure distribution within the cluster is uneven when the cluster is normal. The cluster is in a normal state for more than 99.99% of the time, and in database OLTP business, it is generally 70% read, and in OLAP, it is generally more than 90% read. Read balance can significantly improve the cluster throughput. For example, the distributed storage engines used in related technologies, such as Ceph, use random object names and consistent hash calculations to assign them to a group of storage resources, and the IO flow is first concentrated on the master of this group of storage resources. Although from a mathematical probability point of view, in the case of complete randomness and a sufficiently large number of objects, the capacity and performance in the cluster are balanced, the actual business is not a perfect mathematical model. Furthermore, the EC protection mode widely used in the storage industry is divided into data sharding and parity sharding. When the cluster is normal, only data shards support reads; when the cluster is abnormal, parity shards are used to repair reads. The cluster is normal more than 99.999% of the time, which causes read imbalance. The number of PGs in Ceph is not huge, so mathematically it cannot be guaranteed that each medium will perform parity balancing, hence read imbalance cannot be achieved.

[0097] Therefore, this embodiment refers to Figure 5 as well as Figure 6 As shown, the parity shards and data shards of all PGs within the cluster are readjusted to ensure that all hard drives appear an equal number of times in each position within the PG, thereby guaranteeing that reads are evenly distributed across all network, compute, and storage resources within the cluster. In the EC scenario, disk selection ensures that all disks are evenly distributed across the EC's parity shards and data shards from the perspective of the logical topology cluster, effectively improving the overall performance balance of the cluster.

[0098] In this embodiment, the hard drives selected for erasure coding are chosen to ensure that the hard drives in the cluster appear in the same position in all erasure coding protection groups, thereby achieving erasure coding read / write balance. This method can effectively prevent uneven hard drive load and improve the performance and reliability of the overall storage system.

[0099] In some optional implementations, a preset space allocation balancing strategy is executed based on the used capacity of the PG and the used capacity of the hard disk, including:

[0100] Determine whether the difference in used capacity of PG exceeds the first preset capacity;

[0101] If the difference in the used capacity of PG exceeds the first preset capacity, it is determined whether the time exceeding the first preset capacity exceeds the first preset time.

[0102] If the time exceeds the first preset capacity, a first cyclic PG group is established. The first cyclic PG group is used to balance the used capacity of PGs.

[0103] Determine whether the difference in used hard drive capacity exceeds the second preset capacity;

[0104] If the difference in the used capacity of PG exceeds the second preset capacity, determine whether the time exceeding the second preset capacity exceeds the second preset time.

[0105] If the time exceeds the second preset capacity, a second cyclic PG group is established on the hard drive that exceeds the second preset capacity and the second preset time. The second cyclic PG group is used to balance the used capacity of the hard drive.

[0106] Specifically, a unified cluster management center allocates virtual address chunks, distinguished by a unique identifier, chunkID. The disk groups contained in PGs (PGs) with adjacent chunkIDs allocated by the cluster management center are largely non-overlapping, ensuring no disk hotspots occur during normal sequential chunk usage. Large loop priority is used to allocate chunks based on the principle of equal chunk numbers allocated to each PG; in a normally used cluster, the number and capacity of chunks are the same. When the cluster management center monitors a difference in used capacity of PGs reported by a single storage server exceeding 10GB and persisting for more than one day, it begins establishing a "small loop PG group" to catch up. When a difference in used disk capacity reported by a single storage server exceeds 10GB and persists for more than two days, a "small loop PG group" is established for the PGs on these disks to catch up in capacity. Refer to the specific example processing procedure. Figure 7 As shown, valid PGs are allocated chunks in a "round-robin" manner. When PG capacities are inconsistent, 90% of the requested chunks are allocated in a large round-robin, and 10% are allocated in a small round-robin to catch up with the PG capacities. The round-robin range is centrally controlled by the cluster management center. Furthermore, the cluster management center monitors the available disk capacity and percentage. When disk capacities are uneven, the relevant PGs are isolated from the large round-robin to allocate chunks in a small round-robin, ensuring that the chunks allocated by the cluster management center are writable as much as possible.

[0107] In this embodiment, the cluster space allocation is uniformly performed by the cluster management center based on statistical information within the cluster. Chunks are allocated through a "large loop" to ensure that the number of chunks within each PG is consistent. When recovering from a failure after a period of time or when scaling up, chunks are added through a small loop while the large loop is being used for allocation. If the number of chunks is consistent but the capacity is unbalanced for more than one day, the PG with the lower capacity is individually added to the small loop to achieve balance.

[0108] In some optional implementations, a preset usage balancing strategy is executed based on the star topology read / write method, including:

[0109] Obtain the I / O data for each hard drive in a star topology read / write mode;

[0110] Based on IO data, the data types with different popularity levels are determined;

[0111] Data types with different popularity are assigned to the corresponding logical topology based on the type of the logical topology.

[0112] Reference Figure 8 As shown, this embodiment replaces chained writes with star topology, resulting in a more balanced read / write operation within the cluster. When heterogeneous servers exist within a cluster, logical topology isolates servers with different structures. When using space, IO type analysis is performed, and different topologies are used based on business characteristics to achieve balanced heterogeneous cluster operation. For example, after long-term statistics, garbage collection (GC, an automatic memory management mechanism) is used to place hot data into a high-performance topology and cold data into a normal-performance topology, as described above. Figure 9 As shown.

[0113] By organizing similar hardware into different topologies, a heterogeneous storage cluster federation approach is used to manage ultra-large heterogeneous storage clusters. Different business types and hardware characteristics allow data to be evenly distributed across different federated clusters, thereby achieving balanced cluster performance. For example, in public clouds, there may be timed massive I / O operations where the business's I / O performance requirements are not high, but the impact on other business performance within the cluster can be mitigated by gradually redirecting traffic to a sandbox cluster without the business being aware of the impact. Through long-term hot and cold I / O analysis, the amount of hot and cold I / O can be distributed, achieving overall cluster performance balance.

[0114] In this embodiment, a star topology is used for read / write operations, without a master node responsible for forwarding read / write operations. In scenarios with high read volumes, the master node will not become a read hotspot. When heterogeneous servers exist within a cluster, logical topology isolates servers with different structures. When using space, different topologies are used based on the characteristics of the business to achieve heterogeneous cluster load balancing, which can further effectively optimize data balance.

[0115] In some optional implementations, a preset backend load balancing strategy is executed, including:

[0116] Obtain statistical information for all PGs and all hard drives within the cluster;

[0117] Based on the statistical information of all PGs, hot PGs are identified;

[0118] Based on the statistical information of all hard drives, identify the hottest hard drives;

[0119] If a hot PG and a hot hard drive continuously exhibit a third preset time, the hot PG will be migrated from the hot hard drive to a non-hot hard drive through PG migration.

[0120] If the above three layers of data balancing still fail to achieve cluster balance, the cluster management center will analyze cluster information. If the cluster experiences prolonged periods of hard drive hotspots and PG hotspots, or if hard drive and PG capacities are inconsistent, the cluster management center can perform PG migration, referring to... Figure 10 As shown, hot PGs are migrated to non-hot disks. Since the number of PGs migrated and the migration bandwidth can be configured, the impact on the entire cluster is small and controllable, and it can further effectively improve data balance. Here, authoritative shards represent reliable data.

[0121] The cluster management center monitors information about each component of the cluster. For example, if it finds that certain hard drives and certain PGs on those hard drives will have performance hotspots at fixed intervals of 1 hour, while other hard drives and PGs will not have this problem, then the hot PGs will be migrated to other hard drives without hotspots, and the non-hotspot PGs will be migrated back. This design can also solve the problem of periodically hot PGs on public clouds to some extent.

[0122] This embodiment also provides a data balancing device for a write-time redirection distributed storage engine. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0123] This embodiment provides a data balancing device for a write-time redirection distributed storage engine, such as... Figure 11 As shown, the device includes:

[0124] The balancing strategy execution module 201 is used to execute a preset disk selection balancing strategy based on the logical topology. The logical topology is one of several different types of logical topologies determined after reorganizing the physical topology of the heterogeneous cluster, and each logical topology includes multiple logically isolated sub-topologies.

[0125] The space allocation balancing strategy execution module 202 is used to obtain the used capacity of the PG and the used capacity of the hard disk after executing the preset disk selection balancing strategy; and to execute the preset space allocation balancing strategy based on the used capacity of the PG and the used capacity of the hard disk.

[0126] The balancing strategy execution module 203 is used to execute the preset space allocation balancing strategy and then execute the preset usage balancing strategy based on the star topology read / write method after executing the preset space allocation balancing strategy.

[0127] The background balancing strategy execution module 204 is used to execute a preset background balancing strategy if the cluster still fails to achieve balancing after executing the preset balancing strategy.

[0128] In this embodiment, the data balancing device for the write-time redirection distributed storage engine is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0129] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0130] This invention also provides a computer device having the above-described features. Figure 11 The diagram shows a data balancing device for a write-time redirection distributed storage engine.

[0131] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 12 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 12 Take a processor 10 as an example.

[0132] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0133] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0134] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0136] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0137] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0138] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0139] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data balancing method for a write-time redirection distributed storage engine, characterized in that, The method includes: A preset disk selection and balancing strategy is executed based on a logical topology. The logical topology is one of several different types of logical topologies determined after reorganizing the physical topology of the heterogeneous cluster, and each logical topology includes multiple logically isolated sub-topologies. After executing the preset disk balancing strategy, obtain the used capacity of the logical protection group and the used capacity of the hard disk; Based on the used capacity of the logical protection group and the used capacity of the hard disk, a preset space allocation balancing strategy is executed. After executing the preset space allocation balancing strategy, a preset usage balancing strategy is executed based on the star topology read / write method; If the cluster still fails to achieve balance after executing the preset load balancing strategy, the preset background load balancing strategy will be executed.

2. The method according to claim 1, characterized in that, The execution of the preset disk selection and balancing strategy based on the logical topology includes: Step a: Determine the weight and fault domain of each topology node in the logical topology structure; Step b: Select the Nth layer in the logical topology, where N=1; Step c: Determine the first topology node in the maximum weight set of the Nth layer and the fault domain corresponding to the first topology node; the first topology node is any node in the maximum weight set; Step d: If the fault domain corresponding to the first topology node is a set fault domain, determine whether the first topology node meets the fault requirements. Step e1: If the fault requirements are met, select the first topology node and set N=N+1, return to step c, until N is the last layer number in the logical topology structure, and complete the selection of the logical topology. Step e2: If the fault requirements are not met, return to step b and take the second largest weight set in the Nth layer as the largest weight set.

3. The method according to claim 2, characterized in that, After completing the selection of the logical topology, the following steps are also included: The check shards and data shards of all logical protection groups within the cluster are readjusted so that each hard drive in the logical topology serves as an EC check shard and an EC data shard an equal number of times.

4. The method according to claim 1, characterized in that, The step of executing a preset space allocation balancing strategy based on the used capacity of the logical protection group and the used capacity of the hard disk includes: Determine whether the difference in used capacity of the logic protection group exceeds the first preset capacity; If the difference in the used capacity of the logic protection group exceeds the first preset capacity, it is determined whether the time exceeding the first preset capacity exceeds the first preset time. If the time exceeds the first preset capacity, a first cyclic logic protection group is established, which is used to balance the capacity already used by the logic protection group. Determine whether the difference in used hard drive capacity exceeds the second preset capacity; If the difference in the used capacity of the logic protection group exceeds the second preset capacity, it is determined whether the time exceeding the second preset capacity exceeds the second preset time. If the time exceeds the second preset capacity and the second preset time, a second cyclic logical protection group is established on the hard drive that exceeds the second preset capacity and the second preset time. The second cyclic logical protection group is used to balance the used capacity of the hard drive.

5. The method according to claim 1, characterized in that, The star-based read / write method, which executes a preset usage balancing strategy, includes: Obtain the I / O data of each hard disk in the star topology read / write mode; Based on the IO data, the data types with different popularity levels are determined; The data types with different popularity are assigned to the corresponding logical topology structures according to the type of the logical topology structure.

6. The method according to claim 1, characterized in that, The execution of the preset background load balancing strategy includes: Obtain statistical information for all logical protection groups and all hard drives within the cluster; Based on the statistical information of all the aforementioned logic protection groups, hotspot logic protection groups are determined; Based on the statistical information of all the hard drives, hotspot hard drives are identified; If the hotspot logic protection group and the hotspot hard drive continue to occur for a third preset time, the hotspot logic protection group will be migrated from the hotspot hard drive to a non-hotspot hard drive through logic protection group migration.

7. A data balancing device for a write-time redirection distributed storage engine, characterized in that, The device includes: The balancing strategy execution module is used to execute a preset disk selection balancing strategy based on a logical topology structure. The logical topology structure is one of several different types of logical topologies determined after reorganizing the physical topology structure of the heterogeneous cluster, and each logical topology structure includes multiple logically isolated sub-topologies. The space allocation balancing strategy execution module is used to obtain the used capacity of the logical protection group and the used capacity of the hard disk after executing the preset disk selection balancing strategy; and to execute the preset space allocation balancing strategy based on the used capacity of the logical protection group and the used capacity of the hard disk. The balancing strategy execution module is used to execute the preset space allocation balancing strategy and then execute the preset usage balancing strategy based on the star topology read / write method after executing the preset space allocation balancing strategy. The background balancing strategy execution module is used to execute a preset background balancing strategy if the cluster still fails to achieve balancing after the preset balancing strategy has been executed.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data balancing method for a write-time redirection distributed storage engine as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the data balancing method for a write-time redirection distributed storage engine as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the write-time redirection distributed storage engine data balancing method as described in any one of claims 1-6.

Images