A method and apparatus for storage pool expansion / shrinkage recovery in a distributed storage system

Abstract

This invention relates to a method and apparatus for storage pool scaling and recovery in a distributed storage system. The method includes the following steps: obtaining a scaling request; setting the storage pool to maintenance mode; determining whether the current storage pool meets the conditions for allowed scaling; if so, performing the scaling operation; if not, sending an operation failure signal; after completing the scaling operation, all storage nodes in the storage pool, based on the updated cluster view, use a two-layer consistent static hash ring distribution algorithm to complete data rebalancing. Compared with existing technologies, this invention maintains the availability of data replicas / shards throughout the scaling process, thereby preventing storage pool degradation and minimizing the impact of data migration on front-end read / write requests.

Description

Technical Field

[0001] This invention relates to the field of computer distributed block storage technology, and in particular to a method and apparatus for expanding, shrinking and restoring storage pools in a distributed storage system. Background Technology

[0002] In recent years, distributed storage technology has been widely adopted. The core of distributed storage is a decentralized architecture, using algorithms to determine data storage locations, eliminating the need for centralized metadata management and achieving unlimited scalability. However, in distributed storage where data distribution is determined by algorithms, the offline / online status of storage cluster nodes and disk failures / replacements can significantly alter the calculated storage locations. While algorithms like consistent hashing can reduce the range of changes, this only addresses scenarios like disk failures / replacements. In scaling scenarios, such as expanding multiple fault-tolerant domain nodes simultaneously, large-scale changes to storage locations are unavoidable, resulting in a substantial, sometimes unacceptable, impact on front-end application performance. This is especially true for erasure coding distributed storage, where data shards are ordered and their sequence cannot be disrupted. Expanding a single faulty storage domain node can potentially change the distribution locations of all data shards. To avoid this issue, some storage solutions only support expansion via storage pools, prohibiting expansion within a single pool; others only support expanding one fault-tolerant domain at a time within a storage pool, with rolling expansion between different fault-tolerant domains. This approach leads to a reduction in available replicas, storage pool degradation, and a certain risk of data loss. Furthermore, it requires adding multiple fault-tolerant domain storage nodes in a single expansion upgrade; otherwise, it results in uneven storage distribution and no increase in available capacity. Expanding to multiple storage nodes at once may also exceed the customer's budget. Summary of the Invention

[0003] The purpose of this invention is to overcome the defects of the prior art by providing a method and apparatus for storage pool expansion and contraction recovery in a distributed storage system, and to solve the problems of storage performance degradation and storage pool degradation during the expansion and contraction of the fault-tolerant domain level in distributed storage.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A method for restoring storage pool scaling in a distributed storage system includes the following steps:

[0006] Get the scaling request, set the storage pool to maintenance mode, determine whether the current storage pool meets the conditions for allowing scaling, if yes, execute the scaling operation, if no, send back an operation failure signal;

[0007] After the scaling up and down operation is completed, all storage nodes in the storage pool rebalance the data based on the updated cluster view and using a two-layer consistent static hash ring distribution algorithm.

[0008] Furthermore, in the two-layer consistent static hash ring distribution algorithm, the data distribution hash ring of the storage cluster before data migration is denoted as the first data distribution hash ring, and the latest data distribution hash ring after scaling up and down is denoted as the second data distribution hash ring. The process of each storage node performing the data rebalancing includes the following steps:

[0009] Generate a list of storage objects to be recovered for the current storage node, with the current storage node as the primary storage node for all storage objects in the list;

[0010] At least one concurrent recovery worker thread is used to recover data from the storage objects in the list of storage objects to be recovered one by one, according to the data distribution positions in the first data distribution hash ring and the second data distribution hash ring.

[0011] Once the current node has completed data recovery, it will synchronize its status to all storage nodes.

[0012] After receiving signals that all other storage nodes have completed data recovery, filter out the storage objects whose second data distribution hash ring distribution position is not in the storage object list of this node, and clear the relevant storage objects.

[0013] Furthermore, the current storage node, acting as the requesting storage node, sends a request to obtain the list of migrated storage objects to all other storage nodes. The storage node that receives the request to obtain the list of migrated storage objects uses a hash algorithm based on the current cluster view to filter out the list of all storage objects in its local list of all storage objects for which the primary storage node is the requesting storage node, and returns the result to the requesting storage node. The requesting storage node then generates a list of storage objects to be restored based on the returned result.

[0014] Furthermore, the number of concurrent recovery worker threads is dynamically or statically adjusted based on the current storage cluster load status and data rebalancing flow control configuration.

[0015] Furthermore, based on the data distribution positions in the first and second data distribution hash rings, the data replicas / fragments that need to be recovered are marked, and the available data replicas / fragments are read from the storage nodes corresponding to the distribution positions in the first data distribution hash ring for recovery.

[0016] Furthermore, if a secondary failure occurs during the data rebalancing process, the data rebalancing is performed again after changing the cluster view. When rebalancing, the second data distribution hash ring is given higher priority, and the corresponding available data replicas / shards are read from the second data distribution hash ring and the first data distribution hash ring in sequence for recovery.

[0017] Furthermore, during the data rebalancing process, if a read / write request for a storage object is received, the corresponding read / write operation is performed based on the state of the storage object:

[0018] If the storage object is not recovered, the first data distribution hash ring is used to respond to the read and write requests. If the read and write request is a write request, a lock is added to protect the storage object until the write request is completed.

[0019] If the storage object is in recovery, determine whether the read / write request is a write request. If yes, respond with a retry instruction; otherwise, use the first data distribution hash ring response.

[0020] If the storage object has been recovered, then the first data distribution hash ring is used to respond to read and write requests.

[0021] Furthermore, it is determined whether a secondary fault recovery scenario exists when the read / write request is received. If so, the second data distribution hash ring is given higher priority, and the second data distribution hash ring and / or the first data distribution hash ring are used to respond to the read / write request in sequence.

[0022] Furthermore, it is determined whether the read / write request is received during the process of generating the list of storage objects to be recovered and whether the read / write request is a write request. If so, the storage objects corresponding to the write request whose primary storage node is a local storage node are directly added to the list of storage objects to be recovered.

[0023] The present invention also provides a storage pool scaling up / down recovery device in a distributed storage system, comprising one or more processors, a memory, and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the storage pool scaling up / down recovery method in a distributed storage system as described above.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The scaling process of this invention adopts a dual data distribution hash ring to serve front-end business requests, which supports the scaling process without reducing the number of available data replicas and without degrading the storage pool. At the same time, it avoids the problem of a significant drop in front-end application performance caused by a large number of changes in the data distribution location during the scaling process, and will not lead to a reduction in data replicas / shards, thus ensuring data availability during the scaling process.

[0026] 2. During the scaling up and down process of this invention, the number of available data copies / shards is never reduced, thereby ensuring that the storage pool does not degrade and minimizing the impact of data migration on front-end read and write requests.

[0027] 3. In multi-storage node scaling scenarios, instead of using a rolling scaling mode where the next storage node is scaled up or down after the previous one is completed, this invention can support scaling up or down multiple fault domain storage nodes at once, thereby reducing the overall data migration time, reducing the difficulty of operation and maintenance, and reducing the risk of data loss.

[0028] 4. Storage nodes have an offline state. During the expansion and contraction of the storage system, offline nodes are not removed from the hash distribution ring. Storage node-level offline failures will not cause secondary changes in the data distribution location. Disk-level failures only affect data replicas / shards on a single storage node within a single fault-tolerant domain. Their distribution location only changes within the storage node. The distribution locations of other data replicas / shards do not change. This effectively avoids the possibility of secondary failures during expansion and contraction causing further changes in data distribution, which could lead to the absence of usable data replicas / shards. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the capacity expansion and reduction operation process of the present invention;

[0030] Figure 2 This is a schematic diagram illustrating the data rebalancing state transition during the storage node scaling up / down process of the present invention.

[0031] Figure 3 This is a schematic diagram of the read / write processing logic during the expansion and contraction of the replica storage pool storage nodes in this invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0033] Example 1

[0034] This embodiment provides a method for storage pool scaling up / down recovery in a distributed storage system, including the following steps: obtaining a scaling up / down request (new storage node coming online or storage node scaling down event); setting the storage pool to maintenance mode; determining whether the current storage pool meets the conditions for allowing scaling up / down; if so, performing the scaling up / down operation; if not, sending back an operation failure signal; after completing the scaling up / down operation, all storage nodes in the storage pool, based on the updated cluster view, use a two-layer consistent static hash ring distribution algorithm to complete data rebalancing.

[0035] The above method employs a two-layer consistent static hash ring distribution algorithm. The hash ring of the storage cluster data distribution before data migration is denoted as the first data distribution hash ring, and the latest data distribution hash ring after scaling up or down is denoted as the second data distribution hash ring. Storage nodes in the storage pool are responsible for responding to read and write requests from front-end services for storage objects. Each storage object generates a primary storage node and a backup storage node in the current cluster view based on the hash algorithm. The primary storage node of a storage object is responsible for responding to read and write requests and for data migration and recovery of the storage object. If the storage node receiving the read / write request is not the primary storage node of that storage object, the read / write request is forwarded to the corresponding storage node for execution. If the primary storage node of a storage object fails, after the cluster view is updated, the backup storage node is promoted to primary storage node and responds to read and write requests for the storage object.

[0036] like Figure 1 As shown, the scaling operation process in this embodiment includes:

[0037] Before performing any scaling up or down operations, the storage pool must be set to maintenance mode. Storage pool recovery in maintenance mode will be paused, but data distribution will remain unchanged. Each scaling up or down operation requires a pre-check. Operations are considered disallowed if the storage pool is in a degraded state or if the scaling up or down operation would result in the number of available replicas not meeting configuration requirements. For scaling up, only one expansion storage node is added to the storage pool at a time. That is, one expansion storage node is added to the storage pool first, and after the storage pool status is updated, a second expansion storage node is added. This process is repeated until all storage nodes to be expanded are added. For scaling down, only one scaling down node is marked and updated at a time. That is, one scaling down node is marked as Offlining, and after the storage pool status is updated, the next storage node to be scaled down is marked as Offlining. This process is repeated until all storage nodes to be scaled down have completed their status updates. After the scaling up / down operation is completed and maintenance mode is exited, all storage nodes begin performing data rebalancing. At this point, no further scaling up / down operations are permitted; operations must wait until the data rebalancing is complete and the storage pool's status returns to Healthy before proceeding. Once all storage nodes have completed data rebalancing, the scaling-down node is taken offline, officially removing it from the storage pool view (cluster view), thus completing the entire scaling up / down process.

[0038] This embodiment introduces an offline state for storage nodes. During storage system expansion and contraction, offline nodes are not removed from the hash distribution ring, and storage node-level offline failures do not cause secondary changes in data distribution locations. Simultaneously, disk-level failures only affect data replicas / shards on a single storage node within a single fault-tolerant domain. Their distribution locations only change within the storage node; the distribution locations of other data replicas / shards remain unchanged. This algorithm layer avoids secondary failures during expansion and contraction that could cause further changes in data distribution, leading to unusable data replicas / shards.

[0039] After the scaling up / down operation is completed (new storage node coming online or storage node scaling down event), all storage nodes in the storage pool enter a static scaling up / down data rebalancing mode, and all storage nodes perform data migration work in parallel. The specific steps for each storage node to perform the data rebalancing include:

[0040] 101) Generate a list of storage objects to be recovered for the current storage node. All storage objects in the list of storage objects to be recovered are based on the current storage node as the primary storage node.

[0041] The current storage node acts as the requesting storage node, sending a request to obtain the list of migrated storage objects to all other storage nodes. The storage node that receives the request to obtain the list of migrated storage objects uses a hash algorithm to filter out the list of all storage objects in its local list of all storage objects for which the primary storage node is the requesting storage node, based on the current cluster view, and returns the result to the requesting storage node. The requesting storage node then generates a list of storage objects to be restored based on the returned result.

[0042] In this step, the main thread for restoring the storage node sets the storage node's recovery status to the Preparing state, sorts the list of storage objects to be restored by storage object ID, and sets the initial state of all storage objects to be restored to Undone.

[0043] 102) The main recovery thread exits the Preparing recovery state and enters the data rebalancing stage. It switches to the Rebalancing recovery state. In this state, the main recovery thread sets the storage object state to the Processing state from the list of storage objects to be recovered, starts multiple recovery worker threads, and performs data recovery on the storage objects in the list of storage objects to be recovered one by one according to the data distribution position in the first data distribution hash ring and the second data distribution hash ring. It migrates part of the replica / shard data of each storage object from the distribution position before scaling up or down to the new scaling node or the new distribution position.

[0044] The number of concurrent worker threads is adjusted dynamically or statically based on the current storage cluster load and data rebalancing flow control configuration. When dynamic configuration is used, the storage node dynamically adjusts the data scaling migration rate according to the business pressure of front-end read and write requests, reducing the performance impact of data migration data flow on front-end read and write requests. When static configuration is used, data scaling migration is completed at a fixed rate or at the fastest speed and in the shortest time.

[0045] The worker thread resumes calculating the data distribution position of the storage object on the first and second data distribution hash rings. If the distribution position of the storage object on the first and second data distribution hash rings has not changed, then the storage object does not need to be migrated; if only the distribution positions of some data replicas / shards have changed, then only the replicas / shards whose positions have changed need to be migrated.

[0046] The recovery worker thread reads available data replicas / shards from the storage nodes at the first data distribution hash ring distribution location. Storage objects with the replica strategy are directly copied and restored to the storage nodes at the second data distribution hash ring distribution location. For storage objects configured with the N+M (N data shards, M check shards) erasure coding (EC) strategy, at least N data shards are read, EC encoding is used to generate the corresponding data / check shards, and they are restored to the storage nodes at the second data distribution hash ring distribution location in sequence.

[0047] If a node-level or disk-level secondary failure occurs during the scaling up / down process, the local recovery task is abandoned, the list of storage objects to be migrated is rebuilt, and the data migration task is re-executed. The number of fault-tolerant domain failures that the scaling up / down process can tolerate remains consistent with the storage pool state before migration. In the recovery scenario of secondary failures, regardless of whether the distribution location has changed, the recovery thread first checks whether the data replica / shard of the storage object exists at its distribution location on the second data distribution hash ring. If it does not exist, the corresponding data replica / shard is marked as needing recovery; if it exists, available data replicas / shards are read from the second data distribution hash ring for recovery.

[0048] In the event of a crash recovery scenario where all storage nodes fail, if the recovery thread cannot obtain available data copies or sufficient data shards, it will execute rollback recovery logic to recover from the historical version of the storage object. If recovery still fails, it will be marked as recovery failure, and read / write requests for the corresponding storage object will be returned to EIO processing.

[0049] Once the worker thread has resumed, a notification will be sent to resume the main thread. The main thread will then mark the corresponding storage object as "Done" and continue resuming the next storage object.

[0050] 103) After the main thread completes the recovery of all storage objects, it marks the node recovery status as Pre_done and synchronizes the status to all storage nodes, waiting for other storage nodes to complete data recovery.

[0051] 104) After all storage nodes have completed the data migration, the storage node checks all storage objects on its own node and performs deletion operations on all data replicas / shards that belong to the first data distribution hash ring distribution location but not to the second data distribution hash ring distribution location.

[0052] If a secondary failure occurs during the above steps, the recovery main thread will abandon the current data rebalancing process, change the storage cluster status, and then re-execute the recovery process according to the above steps.

[0053] like Figure 2 As shown, the data rebalancing process for scaling up and down storage nodes in this embodiment specifically includes:

[0054] After the cluster view is updated, the recovery main thread sets the storage nodes to the Preparing recovery state, sends requests to all storage nodes in the cluster, builds a list of storage objects to be recovered, and then the storage node enters the Rebalancing recovery state. After all data objects have been migrated, the recovery main thread completes the data migration work for its own storage node. The local storage node enters the Pre_done recovery state. In this state, the recovery main thread notifies all storage nodes of its recovery status and waits for the recovery completion event of all storage nodes. After all storage nodes have completed the data migration, old data replicas / shards that do not exist on the second data distribution hash ring are cleared, and the storage pool completes the data migration. If a secondary failure occurs during the Preparing or Rebalancing recovery states, the local recovery process is canceled, and the recovery process restarts.

[0055] In this embodiment, the scaling up / down recovery process uses a dual data distribution hash ring to serve front-end business requests. The storage pool does not degrade, preventing a reduction in data replicas / shards, and ensuring data availability during scaling up / down. Specifically, during data rebalancing, if a read / write request for a storage object is received, the corresponding read / write operation is executed based on the storage object's status: if the storage object is not recovered, the first data distribution hash ring is used to respond to the read / write request, and if the read / write request is a write request, a lock is simultaneously added to protect the storage object until the write request is completed; if the storage object is recovering, it is determined whether the read / write request is a write request; if so, a retry instruction is responded to; otherwise, the first data distribution hash ring is used to respond; if the storage object has recovered, the first data distribution hash ring is used to respond to the read / write request. The recovery thread must wait for the storage object to unlock before executing the data migration and recovery logic for that storage object.

[0056] If a read / write request is received during recovery from a secondary failure in the scaling-up / scaling process, the second data distribution hash ring will be used to respond to the request first. If the second data distribution hash ring cannot find a usable data replica or sufficient data shards, the first data distribution hash ring will be used to retry the read / write request. When the storage node recovery status is in the stage of building the list of storage objects to be migrated, if the primary node of the storage object corresponding to the write request is a local storage node, the corresponding storage object will be directly added to the list of storage objects to be migrated. This avoids newly created storage objects being created in the second data distribution hash ring but processed in the first data distribution hash ring, which could cause IO errors.

[0057] The logic for handling read / write requests during the migration of storage objects during storage node scaling is as follows:

[0058] 201) When the storage node is in the Preparing state, write requests directly add the corresponding storage object to the list of storage objects to be restored, add lock protection to the storage object, and use the first data distribution hash ring data replica / sharding to process read and write requests. After the write request is completed, the lock protection is released, and the recovery thread waits for the storage object lock reference technology to return to zero before performing the recovery operation.

[0059] 202) When the storage node recovery status is Rebalancing, query the status of the storage object corresponding to the read / write request in the list of storage objects to be recovered. If the storage object status is Undone, the first data distribution hash ring is used to process the read / write request, and the write request locks the storage object. If the storage object status is Recovering, the read / write request is added to the storage object's waiting recovery completion list. After the recovery thread completes the recovery, the read / write request is retrieved from the waiting recovery list and retried. If the storage object status is Done or the corresponding storage object does not exist in the list of storage objects to be recovered, the second data distribution hash ring is used directly to process the read / write request, without the need to lock the storage object. If a secondary failure occurs during storage pool reconstruction, all read / write requests are processed first on the second data distribution hash ring. If a request fails and the error code is that the requested object replica / shard does not exist, the read / write request is retried on the first data distribution hash ring.

[0060] 203) When the storage node recovers to the Pre_done state, the second data distribution hash ring is used directly to process read and write requests.

[0061] 204) In storage pool crash recovery scenarios where the number of failures exceeds the tolerable fault tolerance domain, if data recovery still cannot be completed following the above data rebalancing process, a rollback recovery will be attempted. Before a storage node is rejoined to the storage cluster after a failure and going offline, all such storage nodes will be marked with the cluster version number at the time of offline. The recovery worker thread will attempt to read the historical version of each data replica / shard of the corresponding storage node for recovery based on the data replica / shard distribution location of the first data distribution hash ring.

[0062] Taking the expansion of a three-replica storage pool from four fault-tolerant domain storage nodes to five fault-tolerant domain storage nodes as an example, such as Figure 3 As shown, the read / write processing logic during the scaling up and down of the replica storage pool storage nodes in this embodiment specifically includes:

[0063] The read / write request processing thread queries the status of the corresponding storage objects in the list of storage objects to be restored. For storage objects that have been restored, the second data distribution hash ring is used to respond to read / write requests, such as... Figure 3 For data object 1, write requests are completed using replicas from storage nodes 2, 3, and 5. For storage objects that have not yet been recovered, read and write requests are responded to using the first data distribution hash ring. Write operations lock the storage object, such as... Figure 3 For data object 2, write requests are responded to using data replicas from storage nodes 2, 3, and 4. Storage objects in the recovery process will retry after recovery is complete. If a secondary failure occurs during the scaling up / down process, read / write I / O requests will preferentially use the second data distribution hash ring to respond to read / write requests. If the corresponding storage replica data is missing, causing the read / write request to fail, the request will be retried using the first data distribution hash ring. Throughout the entire data rebalancing process, all three replicas remain available, and the storage pool does not degrade.

[0064] In the above process, the storage node is responsible for migrating the storage objects during the scaling up and down process, and responds to the read and write requests of the front-end application. The storage node that handles the read and write requests of the storage objects must be the same as the storage node responsible for migrating the storage objects. If the storage node that receives the read and write request is not the main storage node responsible for migrating the storage objects, the read and write request will be forwarded to the corresponding storage node for processing.

[0065] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] Example 2

[0067] This embodiment provides a storage pool scaling up / down recovery device in a distributed storage system, including one or more processors, a memory, and one or more programs stored in the memory. The one or more programs include instructions for executing the storage pool scaling up / down recovery method in a distributed storage system as described in Embodiment 1.

[0068] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0069] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0073] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for restoring storage pool capacity expansion / contraction in a distributed storage system, characterized in that, Includes the following steps: Get the scaling request, set the storage pool to maintenance mode, determine whether the current storage pool meets the conditions for allowing scaling, if yes, execute the scaling operation, if no, send back an operation failure signal; After the scaling up and down operation is completed, all storage nodes in the storage pool rebalance the data based on the updated cluster view and using a two-layer consistent static hash ring distribution algorithm. In the two-layer consistent static hash ring distribution algorithm, the data distribution hash ring of the storage cluster before data migration is denoted as the first data distribution hash ring, and the latest data distribution hash ring after scaling up and down is denoted as the second data distribution hash ring. The process of each storage node performing the data rebalancing includes the following steps: Generate a list of storage objects to be recovered for the current storage node, with the current storage node as the primary storage node for all storage objects in the list; At least one concurrent recovery worker thread is used to recover data from the storage objects in the list of storage objects to be recovered one by one, according to the data distribution positions in the first data distribution hash ring and the second data distribution hash ring. Once the current node has completed data recovery, it will synchronize its status to all storage nodes. After receiving signals that all other storage nodes have completed data recovery, filter out the storage objects whose second data distribution hash ring distribution position is not in the storage object list of this node, and clear the relevant storage objects; The current storage node acts as the requesting storage node, sending a request to obtain the list of migrated storage objects to all other storage nodes. The storage node that receives the request to obtain the list of migrated storage objects uses a hash algorithm to filter out the list of storage objects of all primary storage nodes that are requesting storage nodes from the local list of all storage objects based on the current cluster view, and returns the result to the requesting storage node. The requesting storage node generates a list of storage objects to be restored based on the returned result. The number of concurrent recovery worker threads is dynamically or statically adjusted based on the current storage cluster load status and data rebalancing flow control configuration. Based on the data distribution positions in the first and second data distribution hash rings, the data replicas / fragments that need to be recovered are marked, and the available data replicas / fragments are read from the storage nodes corresponding to the distribution positions in the first data distribution hash ring for recovery.

2. The method for expanding, shrinking, and restoring a storage pool in a distributed storage system according to claim 1, characterized in that, If a secondary failure occurs during the data rebalancing process, the cluster view is changed and the data rebalancing is performed again. When rebalancing, the second data distribution hash ring has a higher priority, and the corresponding available data replicas / shards are read from the second data distribution hash ring and the first data distribution hash ring in sequence for recovery.

3. The method for expanding, shrinking, and restoring a storage pool in a distributed storage system according to claim 1, characterized in that, During the data rebalancing process, if a read / write request for a storage object is received, the corresponding read / write operation is performed based on the state of the storage object: If the storage object is not recovered, the first data distribution hash ring is used to respond to the read and write requests. If the read and write request is a write request, a lock is added to protect the storage object until the write request is completed. If the storage object is in recovery, determine whether the read / write request is a write request. If yes, respond with a retry instruction; otherwise, use the first data distribution hash ring response. If the storage object has been recovered, then the first data distribution hash ring is used to respond to read and write requests.

4. The method for expanding, shrinking, and restoring a storage pool in a distributed storage system according to claim 3, characterized in that, When the read / write request is received, it is determined whether there is a secondary fault recovery scenario. If so, the second data distribution hash ring is given higher priority, and the second data distribution hash ring and / or the first data distribution hash ring are used to respond to the read / write request in sequence.

5. The method for restoring storage pool expansion and contraction in a distributed storage system according to claim 3, characterized in that, If the read / write request is received, it is determined whether the process is in the process of generating the list of storage objects to be recovered and whether the read / write request is a write request. If so, the storage objects corresponding to the write request whose primary storage node is a local storage node are directly added to the list of storage objects to be recovered.

6. A storage pool expansion / contraction recovery device in a distributed storage system, characterized in that, It includes one or more processors, memory, and one or more programs stored in the memory, said one or more programs including instructions for executing the storage pool scaling up / down recovery method in any of claims 1-5.

Images