Resource management methods, computer program products, devices and storage media

Abstract

This application relates to a resource management method, computer program product, device, and storage medium. The method includes creating a heartbeat thread for storage nodes and electing a management storage node from each storage pool; when the management storage node detects that all storage nodes in its storage pool have joined, it sends a ready notification; the storage node receiving the ready notification initializes its storage pool version number to an initial value; when a first storage node detects through the heartbeat thread that the difference between the last update timestamp and the current time of a second storage node exceeds a preset threshold, it determines the target type attribute of the target storage pool to which the second storage node belongs; based on the target type attribute and the affiliation of the first storage node with the target storage pool, it determines whether to submit an exception event and the storage node that submits the exception event; based on the target type attribute and whether the storage node receiving the exception event belongs to the target storage pool, it determines the exception event handling scheme. Using this method can improve system reliability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a resource management method, computer program product, device and storage medium. Background Technology

[0002] With the development of cloud computing and big data technologies, distributed storage systems have become the core infrastructure of data centers and are widely used in cloud computing and big data fields. However, traditional distributed storage systems typically mount hard drives directly onto computing storage nodes, leading to problems such as low resource utilization, complex management, and difficulty in expansion.

[0003] Related solutions such as RAID technology require substantial hardware resources, while distributed file systems demand complex configuration and management. Furthermore, existing distributed storage systems suffer from several shortcomings in resource management, including: coarse-grained resource management that fails to finely allocate storage resources based on business needs, leading to resource contention when high-performance and high-capacity storage services share the same storage pool; a simplistic fault handling mechanism that lacks differentiated fault detection and submission strategies based on storage pool characteristics, resulting in low fault handling efficiency and a high risk of fault propagation; and an inadequate state synchronization mechanism, lacking version-based state synchronization, making it difficult to ensure state consistency across storage nodes in a multi-storage-pool architecture during fault handling. Summary of the Invention

[0004] Therefore, it is necessary to provide a resource management method, computer program product, device, and storage medium that can achieve fine-grained partitioning of storage resources, differentiated fault handling, and efficient state synchronization to address the above-mentioned technical problems.

[0005] To address the aforementioned technical problems, firstly, a resource management method is provided, the method comprising: Create multiple storage pools and assign a unique identifier to each storage pool; In response to a storage node joining a storage pool, a heartbeat thread is created for the storage node that has joined the storage pool, and a management storage node is elected from among the storage nodes corresponding to each storage pool; In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, it sends a ready notification to the storage nodes in its storage pool; the storage node that receives the ready notification initializes its storage pool version number to the initial value; In response to the first storage node detecting through its heartbeat thread that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the target type attribute of the target storage pool to which the second storage node belongs is determined. Based on the target type attribute and the affiliation between the first storage node and the target storage pool, determine whether to submit an abnormal event and the storage node that submits the abnormal event; In response to a storage node receiving an abnormal event, the handling plan for the abnormal event is determined based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool.

[0006] In one embodiment, multiple storage pools are created, and a unique identifier is assigned to each storage pool, including: In response to receiving an instruction to create a storage pool, select the corresponding host and disk resources for each storage pool; Create multiple storage pools and assign a unique identifier to each storage pool; The configuration information of each storage pool is written into the database and configuration file. The configuration information includes the unique identifier, host and disk resources of the storage pool. The storage pool includes an all-flash storage pool and a hybrid flash storage pool. The all-flash storage pool consists of solid-state drives (SSDs), while the hybrid flash storage pool consists of a mixture of SSDs and hard disk drives (HDDs).

[0007] In one embodiment, in response to a storage node joining the storage pool, creating a heartbeat thread for the storage node joining the storage pool includes: Start the storage service on the storage node and read the storage pool configuration information from the configuration file; The storage pool corresponding to the storage node is determined based on the network address of the storage node, and the configuration information of the storage pool is loaded. The storage node corresponds to at least one storage pool. A heartbeat thread is created for each storage pool corresponding to the storage node, and each heartbeat thread independently maintains the heartbeat status of the storage node in each storage pool. Each heartbeat thread periodically writes the timestamp of the storage node in each storage pool and reads the timestamp of each storage node in the corresponding storage pool. When the difference between the timestamp of any storage node and the current time exceeds a preset threshold, the storage node is identified as an abnormal storage node and removed from the corresponding storage pool.

[0008] In one embodiment, electing a management storage node from among the storage nodes corresponding to each storage pool includes: Create distributed locks for each storage pool; Each storage node is elected by acquiring the distributed lock corresponding to its storage pool, and the storage node that successfully acquires the distributed lock becomes the management storage node of the storage pool.

[0009] In one embodiment, in response to a management storage node detecting that all storage nodes in its storage pool have joined the storage pool, it sends a readiness notification to the storage nodes in its storage pool; the storage node receiving the readiness notification initializes its storage pool version number to an initial value including: The management storage node checks whether all storage nodes in its storage pool have been added to that storage pool; In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, the management storage node sends a ready notification carrying the storage pool identifier corresponding to the management storage node to each storage node in its storage pool; The storage node that receives the ready notification parses the storage pool identifier carried in the ready notification; If a storage node that receives a ready notification belongs to the storage pool corresponding to the storage pool identifier, the storage pool status of the storage node is set to normal, and the storage pool version number of the storage node is initialized to the initial value. If the storage node receiving the ready notification does not belong to the storage pool corresponding to the storage pool identifier, the ready notification is ignored.

[0010] In one embodiment, after the storage node that receives the readiness notification initializes its storage pool version number to an initial value, the process includes: Create an access endpoint for the storage pool and configure endpoint properties for the access endpoint. The endpoint properties are used to indicate the number of storage pools associated with the access endpoint.

[0011] In one embodiment, in response to the first storage node detecting through its heartbeat thread that the difference between the last update timestamp and the current time of the second storage node exceeds a preset threshold, the target type attribute of the target storage pool to which the second storage node belongs is determined to include: The first storage node detects the timestamps of each storage node through a heartbeat thread; In response to the first storage node detecting that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the second storage node is determined to be a faulty storage node. Get the target storage pool to which the second storage node belongs and the target type attribute of the target storage pool.

[0012] In one embodiment, the target type attribute includes a regular storage pool and an extended storage pool. Based on the target type attribute and the affiliation between the first storage node and the target storage pool, it is determined whether to submit an abnormal event and the storage nodes that submit the abnormal event include: If the target storage pool is a regular storage pool, determine whether the first storage node belongs to the target storage pool; if the first storage node belongs to the target storage pool, the first storage node submits an exception event; if the first storage node does not belong to the target storage pool, ignore the heartbeat expiration event. If the target storage pool is an extended storage pool, the first storage node directly submits an exception event.

[0013] In one embodiment, the handling scheme for the abnormal event is determined based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool, including: If the storage pool to which the second storage node belongs is a normal storage pool, then determine whether the storage node that received the abnormal event belongs to the target storage pool. If the storage node receiving the abnormal event belongs to the target storage pool, then the hash mapping of the target storage pool is updated to generate a new storage pool version number. The synchronization task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its synchronization task. The data recovery task is then executed to rebuild the data replica on the second storage node within the target storage pool. The recovery task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its asynchronous task. If the last storage node detects that all storage nodes in the target storage pool have completed both their synchronization and asynchronous tasks, then the abnormal event handling is considered complete. If the storage node that received the abnormal event does not belong to the target storage pool, the hash mapping of the target storage pool is updated, a new storage pool version number is generated, the synchronization task completion flag is written to the distributed coordination component, and the synchronization task of the storage node that received the abnormal event is marked as completed. If the last storage node detects that the synchronization task of all storage nodes in the target storage pool has been completed, the abnormal event handling is determined to be completed.

[0014] In one embodiment, determining the handling scheme for the abnormal event based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool further includes: If the storage pool to which the second storage node belongs is an extended storage pool, then determine whether the storage node that received the abnormal event belongs to the target storage pool. If the storage node receiving the abnormal event belongs to the target storage pool, the hash mapping of the target storage pool is updated, a new storage pool version number is generated, and the extension version numbers of the target storage pool and the target extended storage pools that have an extension relationship with the target storage pool are updated synchronously. The synchronization task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its synchronization task. The data recovery task is executed, prioritizing data recovery from non-faulty storage nodes in the target storage pool. If there is no valid data in the target storage pool, data is retrieved from the target extended storage pool for recovery. The recovery task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its asynchronous task. If the last storage node detects that the synchronization and asynchronous tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, and the storage pool version number and extension version number of all storage nodes in the target storage pool and the target extended storage pool are consistent, the abnormal event handling is determined to be complete. If the storage node receiving the abnormal event belongs to the target extended storage pool, the extension version number of the target storage pool and the target extended storage pools that have an extension relationship with the target storage pool are updated synchronously. The synchronization task completion flag is written to the distributed coordination component, marking that the synchronization task of the storage node that received the abnormal event is completed. If the last storage node detects that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed and the extension version number of all storage nodes in the target storage pool and the target extended storage pool is consistent, the abnormal event handling is determined to be completed.

[0015] In one embodiment, determining the handling scheme for the abnormal event based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool further includes: If the storage node that received the abnormal event does not belong to the target storage pool or the target extended storage pool, it writes the synchronization task completion flag to the distributed coordination component, marking that the synchronization task of the storage node that received the abnormal event is complete; if the last storage node detects that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, it determines that the abnormal event handling is complete.

[0016] In one embodiment, the storage pool version number is used to record the status information of the storage pool, and the extended version number is used to record the collaborative status of two storage pools with an extended relationship.

[0017] To address the aforementioned technical problems, a second aspect provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0018] To address the aforementioned technical problems, a third aspect provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: the processor executes the steps of the method described in the first aspect.

[0019] In order to solve the above-mentioned technical problems, in a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0020] This application provides a resource management method that achieves logical isolation and unified planning of storage resources by creating multiple storage pools and assigning a unique identifier to each storage pool, thus avoiding resource silos and improving resource utilization. It establishes a real-time monitoring mechanism for storage node status and the self-management capability of the distributed cluster by creating heartbeat threads for storage nodes and electing management storage nodes. By managing storage nodes to detect the cluster's ready status and broadcast ready notifications, it ensures that the cluster only starts providing services after it is fully assembled, and initializes the storage pool version number to an initial value, establishing a version baseline for state synchronization. Furthermore, it quickly discovers second storage nodes using the real-time timestamp of the first storage node through the heartbeat threads and obtains the storage to which the second storage node belongs. The pool type attributes provide a basis for differentiated fault handling decisions. Based on the storage pool type and whether the first storage node belongs to the faulty storage pool, a differentiated submission strategy is adopted. Ordinary storage pools are only submitted by storage nodes belonging to the faulty storage pool, while extended storage pools are directly submitted by the first storage node, achieving fault reporting isolated by storage pool and reducing system overhead. Based on the storage pool type and whether the receiving storage node belongs to the faulty storage pool, a differentiated processing strategy is adopted. Storage nodes belonging to the faulty storage pool perform state synchronization and data recovery, while storage nodes not belonging to the faulty storage pool only perform state synchronization. In extended storage pool scenarios, the extended version number is updated synchronously to ensure the state consistency of the two extended storage pools. Through these steps, fine-grained partitioning of storage resources, differentiated fault handling, and efficient state synchronization are achieved, significantly improving the resource utilization, reliability, and operational efficiency of the distributed storage system. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a resource management method in one embodiment; Figure 2 This is a schematic diagram of a storage cluster in one embodiment; Figure 3 This is a flowchart illustrating the resource management method in another embodiment; Figure 4 This is a flowchart illustrating the resource management method in yet another embodiment; Figure 5 This is a structural block diagram of a resource management device in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] With the rapid development of cloud computing and big data technologies, distributed storage systems have become a core infrastructure of data centers, playing a crucial role in scenarios such as cloud computing and big data analytics. However, traditional distributed storage systems typically mount hard drives directly to computing storage nodes, an architecture with inherent drawbacks such as low resource utilization, complex management and maintenance, and difficulty in horizontal scaling. As the scale and complexity of data centers continue to grow, distributed storage systems urgently need to possess higher reliability, scalability, and automated operation and maintenance capabilities.

[0025] To address the aforementioned issues, the industry has proposed solutions such as RAID (Redundant Array of Independent Disks) technology and distributed file systems. However, these solutions still have significant limitations: RAID technology has high hardware resource requirements and limited scalability; distributed file systems face challenges such as complex configuration and high management threshold, making it difficult to meet the needs of large-scale data centers for elastic resource scheduling and refined operation and maintenance.

[0026] This application provides a resource management method that logically combines physical disk resources, enabling distributed storage systems to natively support storage pool abstraction. Furthermore, it offers advanced capabilities such as fast and slow storage pool partitioning and cluster extension, thereby systematically improving system reliability and scalability. This method aggregates dispersed physical hard drives into a unified storage resource pool based on business needs through storage pooling technology, achieving centralized resource management and on-demand allocation, significantly improving resource utilization efficiency and system flexibility. Simultaneously, through a differentiated fault detection and handling mechanism based on storage pool type attributes and a version number-based state synchronization mechanism, it achieves fault isolation and rapid recovery by storage pool, effectively ensuring cluster state consistency and business continuity, providing a superior storage infrastructure solution for large-scale data centers.

[0027] To address the aforementioned technical problems, in one embodiment, such as Figure 1 As shown, this application provides a resource management method, which specifically includes the following steps: Step 101: Create multiple storage pools and assign a unique identifier to each storage pool.

[0028] Specifically, in response to receiving the instruction to create a storage pool, the system selects the corresponding host and disk resources for each storage pool; creates multiple storage pools and assigns a unique identifier to each storage pool; and writes the configuration information of each storage pool into the database and configuration file. The configuration information includes the unique identifier, host, and disk resources corresponding to the storage pool. The storage pools include all-flash storage pools and hybrid flash storage pools. The all-flash storage pools are composed of solid-state drives (SSDs), and the hybrid flash storage pools are composed of a mixture of SSDs and hard disk drives (HDDs).

[0029] A storage pool is a collection of storage resources logically combined from multiple physical disk resources. It is the basic unit for resource management and scheduling in a distributed storage system. Each storage pool has independent configuration information, member storage nodes, and storage capacity.

[0030] In this embodiment, multiple storage pools are first created in the distributed storage system, and a unique identifier is assigned to each storage pool to achieve logical isolation and unified planning of storage resources. Specifically, in response to a storage pool creation command issued by the administrator or the system through the management interface, the system selects the corresponding host and disk resources for the storage pool to be created based on the host list and disk type specified by the user. The user can choose to build an all-flash storage pool using high-performance solid-state drives (SSDs) or a hybrid flash storage pool using a combination of SSDs and HDDs, depending on business needs. Subsequently, a storage pool object is created and a unique identifier (such as a UUID or an auto-incrementing ID) is generated for it. This unique identifier is used to distinguish different storage pools in the system, and all subsequent resource management, fault handling, and other operations are isolated and addressed based on this identifier. Next, the configuration information of the storage pool is written into the database and configuration file. The configuration information includes the unique identifier of the storage pool, the list of selected host resources, the list of selected disk resources, and the storage pool type (all-flash storage pool or hybrid flash storage pool), etc., which is used to restore the storage pool state when the system restarts or the storage node starts.

[0031] In this way, by setting up all-flash storage pools and hybrid flash storage pools, centralized management of storage resources is achieved through storage pooling. The all-flash storage pool consists of solid-state drives (SSDs), which can provide high-IOPS, low-latency, and high-quality I / O response for high-priority services. The hybrid flash storage pool consists of a mix of SSDs and HDDs, which can provide ample storage space for large-capacity services. Users can deploy different services on corresponding storage pools according to their service priorities, thereby meeting the differentiated performance and capacity requirements of different services. This effectively avoids the problems of resource silos and resource contention in traditional architectures and significantly improves the utilization efficiency of hardware resources.

[0032] The resource management method provided in this application can be applied to hyperconverged platforms. A hyperconverged platform refers to a hardware cluster that deploys and runs hyperconverged software, consisting of multiple standardized x86 servers interconnected via a network, with each server simultaneously providing computing and storage resources. In a hyperconverged platform, the method of this application achieves deep integration of computing and storage by logically combining the local disks on each server to form a unified distributed storage resource pool.

[0033] like Figure 2 As shown, this embodiment creates multiple storage pools on the hyperconverged platform, including an all-flash storage pool and a hybrid flash storage pool. The all-flash storage pool is composed of SSD disks and is used to deploy high-priority services. The hybrid flash storage pool is composed of a mixture of SSDs and HDDs and is used to deploy large-capacity services. Each storage pool has an independent and unique identifier, and its configuration information is uniformly stored in the database and configuration files for each storage node of the system to read and load.

[0034] This application avoids resource silos by logically combining physical disk resources according to business needs to form a unified storage resource pool; it meets the differentiated performance and capacity requirements of different businesses by dividing the storage pool into all-flash storage pool and hybrid flash storage pool; and it provides a foundation for subsequent storage node startup and state recovery by centralizing storage pool configuration information through database and configuration files.

[0035] Step 102: In response to a storage node joining a storage pool, a heartbeat thread is created for the storage node joining the storage pool, and a management storage node is elected from among the storage nodes corresponding to each storage pool.

[0036] Specifically, the storage service of the storage node is started, and the configuration information of the storage pool in the configuration file is read. The storage pool corresponding to the storage node is determined according to the network address of the storage node, and the configuration information of the storage pool is loaded. The storage node corresponds to at least one storage pool. A heartbeat thread is created for each storage pool corresponding to the storage node, and each heartbeat thread independently maintains the heartbeat status of the storage node in each storage pool. Each heartbeat thread periodically writes the timestamp of the storage node in each storage pool and reads the timestamp of each storage node in the corresponding storage pool. When the difference between the timestamp of any storage node and the current time exceeds a preset threshold, the storage node is determined to be an abnormal storage node and removed from the corresponding storage pool.

[0037] The management storage node can be elected from among the storage nodes corresponding to each storage pool by creating a distributed lock for each storage pool. Specifically, each storage node elects a management storage node by acquiring the distributed lock corresponding to each storage pool, and the storage node that successfully acquires the distributed lock becomes the management storage node of the storage pool.

[0038] Here, the heartbeat thread refers to an independent thread created by each storage node for each storage pool it belongs to. This thread is used to periodically write the storage node's timestamp within that storage pool to a distributed coordination component (such as ETCD) and to read the timestamps of other storage nodes within the same pool, enabling real-time monitoring of the storage node's liveness status. When a storage node belongs to multiple storage pools simultaneously, multiple heartbeat threads are created, each independently maintaining the storage node's heartbeat status within its respective storage pool.

[0039] A timestamp is the current time value that the heartbeat thread periodically writes to the distributed coordination component. It records the last live time of a storage node in a specific storage pool. Other storage nodes read this timestamp and compare it with the current time to determine whether the storage node has failed.

[0040] The preset threshold is a pre-defined upper limit for the time difference. When the difference between the current time and the last updated timestamp of a storage node exceeds this threshold, the storage node is considered abnormal. This threshold can be dynamically configured based on factors such as system size and network latency, for example, it can be set to 30 seconds.

[0041] Once a storage node joins a storage pool, the system creates a heartbeat thread for that node to monitor its status in real time and elects a management storage node from each pool to coordinate the cluster. Specifically, regarding the creation and operation mechanism of the heartbeat thread, after a storage node starts its storage service, it first reads the storage pool configuration information from the configuration file to obtain the member information of each storage pool. Then, based on its own network address, it determines the storage pool to which it belongs (a storage node can belong to multiple storage pools simultaneously) and creates an independent heartbeat thread for each storage pool. Each heartbeat thread independently maintains the heartbeat status of the storage node in its corresponding storage pool, without interfering with each other. Each heartbeat thread periodically writes its current timestamp in the corresponding storage pool to the distributed coordination component and simultaneously reads the timestamps of all storage nodes in their respective storage pools. When the difference between the timestamp of another storage node and the current time exceeds a preset threshold, that other storage node is determined to be an abnormal storage node and removed from its corresponding storage pool.

[0042] Regarding the election mechanism for managing storage nodes, the system creates a distributed lock for each storage pool. A distributed lock is a distributed coordination mechanism used to achieve mutual exclusion access among multiple storage nodes. In this invention, a distributed lock is created for each storage pool, and each storage node competes for this lock to elect a managing storage node. The storage node that successfully acquires the lock becomes the managing storage node, and ownership of the lock is maintained through periodic renewals.

[0043] When each storage node starts up, it attempts to acquire the distributed lock corresponding to its storage pool. Since the distributed lock is mutually exclusive, the storage node that successfully acquires the lock becomes the management storage node of the storage pool. The management storage node periodically renews the lock during the period of holding the lock to maintain ownership of the lock. When the management storage node fails and cannot renew the lock, the distributed lock is automatically released. Other storage nodes detect the lock release and retry to acquire the lock, thereby electing a new management storage node. Through the aforementioned heartbeat mechanism and management storage node election mechanism, this embodiment achieves the beneficial effects of real-time fault detection, independent monitoring of multiple storage pools, high availability of management storage nodes, and state consistency assurance: the heartbeat thread's periodic writing and reading of timestamps can quickly detect storage node faults, providing timely basis for subsequent fault handling; when a storage node belongs to multiple storage pools, an independent heartbeat thread is created for each storage pool, realizing fault monitoring isolated by storage pool and avoiding mutual interference between different storage pools; the distributed lock mechanism elects management storage nodes and supports automatic re-election in case of failure, ensuring that each storage pool always has an available management storage node responsible for coordinating work; the timestamp threshold detection and distributed lock renewal mechanism ensure accurate judgment of storage node status and lock ownership, providing a foundation for cluster state consistency.

[0044] Step 103: In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, it sends a ready notification to the storage nodes in its storage pool; the storage node that receives the ready notification initializes its storage pool version number to the initial value.

[0045] Specifically, the management storage node checks whether all storage nodes in its storage pool have joined the storage pool. In response to this, the management storage node sends a ready notification carrying the storage pool identifier corresponding to the management storage node to each storage node in its storage pool. The storage node receiving the ready notification parses the storage pool identifier carried in the ready notification. If the storage node receiving the ready notification belongs to the storage pool corresponding to the storage pool identifier, its storage pool status is set to normal, and its storage pool version number is initialized to its initial value. If the storage node receiving the ready notification does not belong to the storage pool corresponding to the storage pool identifier, the ready notification is ignored.

[0046] Storage pool status refers to the operational status of a storage node within a specific storage pool, including two states: initialization and normal. The initialization state indicates that the storage node has been added to the storage pool but has not yet received a readiness notification; at this time, the storage node does not process business I / O. The normal state indicates that the storage node has received a readiness notification and can begin processing business I / O.

[0047] Once the storage pool cluster is fully assembled, the management storage node sends a readiness notification to all storage nodes within the pool. Upon receiving the notification, each storage node initializes its storage pool version number, signifying that the storage pool has entered a normal service state. Considering that the addition of storage nodes to a storage pool in a distributed storage system is a dynamic process, factors such as the startup speed of different storage nodes and network latency may lead to nodes joining in batches. If storage nodes begin processing business I / O before the cluster is fully assembled, problems such as insufficient data replicas, inconsistent perceptions of the cluster status among storage nodes, and inability to correctly determine the cluster status during fault handling may occur. Therefore, this embodiment introduces a readiness notification mechanism to ensure that all storage nodes uniformly enter a normal service state only after the cluster is fully assembled. In the specific implementation process, after the management storage node is elected, it is responsible for monitoring the joining status of all expected storage nodes in its storage pool. The management storage node continuously detects whether all storage nodes in its storage pool have successfully joined the storage pool. The storage node joining information can be obtained from the member list in the distributed coordination component (such as ETCD). When the management storage node detects that all storage nodes in its storage pool have joined, the management storage node sends a ready notification carrying the storage pool identifier of the management storage node to each storage node in its storage pool. The storage node that receives the ready notification first parses the storage pool identifier carried in the notification, and then determines whether the storage node belongs to the storage pool corresponding to the storage pool identifier. If the storage node belongs to the storage pool, the storage pool status of the storage node in the storage pool is set to normal, and the storage pool version number is initialized to the initial value (e.g., 1). At this time, the storage node officially enters the normal service state and can start processing business I / O requests. If the storage node does not belong to the storage pool, the ready notification is ignored and no state change is made. After the storage pool version number is initialized to its initial value, it increments every time a storage node joins, leaves, or undergoes a configuration change. Each storage node compares its own storage pool version number with the latest version number stored in the distributed coordination component when processing business I / O or participating in fault recovery. If they are inconsistent, the node refuses to process the request or waits for synchronization, thus ensuring cluster consistency. Regarding state transitions, a storage node is in the initialization state before receiving a readiness notification. At this time, although the storage node has joined the storage pool and created a heartbeat thread, it has not yet processed business I / O. After receiving the readiness notification and confirming its membership in the storage pool, the storage node's state transitions to normal and it begins processing business I / O. This state transition mechanism effectively avoids erroneous business operations when the cluster is not ready.

[0048] This application configures the management storage node to send a readiness notification after all storage nodes have joined, ensuring that all storage nodes enter normal service status at the same time, thus avoiding insufficient data replicas caused by storage nodes prematurely processing I / O. The storage pool version number is initialized to an initial value, establishing a unified version baseline for subsequent state changes, allowing each storage node to determine whether its state is consistent based on the version number. The receiving storage node determines whether it belongs to the storage pool corresponding to the notification and only processes the relevant notification, avoiding invalid state changes. The system automatically detects the cluster's readiness status and notifies all storage nodes without manual intervention, improving the system's operational efficiency.

[0049] In one implementation, after a storage node receives a readiness notification and initializes its storage pool version number to an initial value, it includes: creating an access endpoint for the storage pool and configuring endpoint attributes for the access endpoint, the endpoint attributes indicating the number of storage pools associated with the access endpoint.

[0050] Here, the access endpoint is the network entry point that provides storage access services to the outside world. Clients connect to and access backend storage resources through this endpoint. In this embodiment, the access endpoint is specifically an iSCSI target. Clients connect to this endpoint through an iSCSI initiator to map storage volumes in the backend storage pool as local disks. The access endpoint can also be implemented using other storage protocols, such as NVMe-oF, FC, etc.

[0051] Endpoint attributes are used to configure the association between an access endpoint and a storage pool, indicating the number of storage pools associated with that access endpoint. Endpoint attributes include two types: Type 1 indicates that the access endpoint is associated with a single storage pool, suitable for normal storage pool scenarios; Type 2 indicates that the access endpoint is associated with two storage pools, suitable for extended storage pool scenarios, where the two storage pools form an extended pair, and data is synchronously replicated to achieve high availability across data centers.

[0052] The number of storage pools refers to the number of storage pools associated with the access endpoint. When the endpoint attribute is of type 1, it is associated with 1 storage pool; when the endpoint attribute is of type 2, it is associated with 2 storage pools (i.e., extended storage pools). By configuring endpoint attributes, the system can flexibly select data distribution strategies according to business needs: ordinary business uses a single storage pool association, while critical business uses dual storage pool associations to achieve disaster recovery.

[0053] Step 104: In response to the first storage node detecting through its heartbeat thread that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the target type attribute of the target storage pool to which the second storage node belongs is determined.

[0054] Specifically, the first storage node detects the timestamps of each storage node through a heartbeat thread; in response to the first storage node detecting that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the second storage node is determined to be a faulty storage node; the target storage pool to which the second storage node belongs and the target type attribute of the target storage pool are obtained.

[0055] The first storage node is the storage node that performs fault detection operations, that is, it reads the timestamps of other storage nodes through a heartbeat thread and determines whether they have expired. Any storage node can serve as the first storage node, including management storage nodes and ordinary storage nodes.

[0056] The second storage node is the storage node that is detected as abnormal, that is, the difference between its last update timestamp and the current time exceeds a preset threshold, and is determined to be a storage node that has failed (such as crashing, network interruption, etc.).

[0057] The last update timestamp is the time value at which the second storage node last wrote to the distributed coordination component in its storage pool, used to record the last survival time of the storage node in that storage pool.

[0058] The preset threshold can be a pre-defined upper limit for the time difference, used to determine whether a storage node is faulty. For example, if it is set to 30 seconds, and the difference between the current time and the storage node's last updated timestamp exceeds 30 seconds, the storage node is considered faulty. This threshold can be dynamically configured based on factors such as system size and network latency.

[0059] The first storage node monitors the timestamps of each storage node in real time via a heartbeat thread. When a second storage node is detected, it determines the target storage pool to which the second storage node belongs and its type attribute, providing a basis for subsequent differentiated fault handling decisions. Considering that storage node failures are common in distributed storage systems, an efficient fault detection mechanism is needed to respond to faults promptly and take appropriate measures. This mechanism uses a heartbeat thread to periodically write and read timestamps to achieve real-time monitoring of the storage node status. Furthermore, since a storage node can belong to multiple storage pools simultaneously, and different storage pools may have different type attributes (ordinary storage pool or extended storage pool), the fault handling strategy needs to be differentiated based on the type of storage pool to which the second storage node belongs. Therefore, after detecting a second storage node, it is necessary to obtain the target storage pool to which the second storage node belongs and its type attribute.

[0060] In the specific implementation process, the heartbeat thread of the first storage node (i.e., the storage node that performs fault detection) periodically reads the timestamps of each storage node recorded in the distributed coordination component. Each heartbeat thread corresponds to a specific storage pool, so the timestamps read are the timestamps of all storage nodes in that storage pool. For each other storage node's timestamp read, the first storage node calculates the difference between the current time and the timestamp, and compares the difference with a preset threshold (e.g., 30 seconds). If the difference is less than or equal to the preset threshold, the storage node is determined to be normal. If the difference exceeds the preset threshold, the storage node is determined to be the second storage node. When the second storage node is detected, the first storage node determines the storage pool to which the second storage node belongs (i.e., the storage pool corresponding to the current heartbeat thread) as the target storage pool, and obtains the type attribute of the target storage pool from the configuration information or the distributed coordination component. The type attribute includes ordinary storage pool (suitable for resource isolation and performance optimization scenarios within a single data center) and extended storage pool (suitable for high availability and disaster recovery scenarios across data centers, consisting of two storage pools with an extension relationship).

[0061] This embodiment implements fault detection isolated by storage pool. Each heartbeat thread only monitors the timestamp of the storage node in its corresponding storage pool. Fault detection in different storage pools is independent of each other. When a storage node belongs to multiple storage pools at the same time, it may be detected as faulty in one storage pool but remain normal in another. The type attribute of the target storage pool is the key decision basis for subsequent fault handling. Ordinary storage pools adopt conditional submission rules (the abnormal event is submitted only when the first storage node belongs to the target storage pool), while extended storage pools adopt unconditional submission rules (the first storage node directly submits the abnormal event). The preset threshold can be dynamically configured according to factors such as system scale and network latency. Setting the threshold too small may lead to misjudgment, while setting it too large will delay the fault detection time. It can usually be set to an integer multiple of the heartbeat period (e.g., a heartbeat period of 3 seconds and a threshold set to 30 seconds).

[0062] Taking an application scenario as an example, suppose storage node A belongs to both the all-flash storage pool (ordinary storage pool) and extended storage pool A (part of the extended storage pool). If the heartbeat thread of storage node A detects that the timestamp of storage node B in the all-flash storage pool has expired, then storage node B is determined to be the second storage node, the target storage pool is the all-flash storage pool, and the target type attribute is the ordinary storage pool. If the heartbeat thread of storage node A detects that the timestamp of storage node C in the extended storage pool A has expired, then storage node C is determined to be the second storage node, the target storage pool is the extended storage pool A, and the target type attribute is the extended storage pool. Different fault handling strategies will be adopted according to different type attributes.

[0063] This application's central jump thread periodically monitors timestamps to quickly detect storage node failures and shorten failure response time; each storage pool performs independent failure detection to avoid mutual interference between different storage pools, and a failure in one storage pool does not affect the detection of other storage pools; it accurately determines the target storage pool to which the second storage node belongs and its type attributes, providing a precise basis for subsequent differentiated failure handling; the preset threshold can be dynamically adjusted according to the actual scenario, balancing the timeliness of failure detection with the false positive rate.

[0064] Step 105: Based on the target type attribute and the affiliation relationship between the first storage node and the target storage pool, determine whether to submit an abnormal event and the storage node that submits the abnormal event.

[0065] Specifically, in response to the target storage pool being a regular storage pool, it is determined whether the first storage node belongs to the target storage pool; in response to the first storage node belonging to the target storage pool, the first storage node submits an exception event; in response to the first storage node not belonging to the target storage pool, the heartbeat expiration event is ignored.

[0066] If the target storage pool is an extended storage pool, the first storage node directly submits an exception event.

[0067] Submitting an anomaly event refers to the operation whereby the first storage node, upon discovering the second storage node, writes fault information (including the second storage node's identifier and the identifier of the storage pool to which the second storage node belongs) to a distributed coordination component (such as ETCD). This event will be detected by other storage nodes in the storage pool, thereby triggering subsequent fault handling procedures.

[0068] In this embodiment, the detection node (first storage node) determines whether to submit an abnormal event and by whom to submit the abnormal event based on the type attribute of the storage pool to which the faulty node (second storage node) belongs and its own affiliation with the target storage pool, thereby realizing differentiated fault reporting isolated by storage pool.

[0069] Specifically, in the steps described above, the detection node has determined the target storage pool and its type attribute to which the faulty node belongs. However, different types of storage pools have different requirements for fault handling: the impact of a fault in a normal storage pool is limited to the pool itself, requiring only the nodes within that pool to participate in fault handling; unrelated nodes should not intervene to avoid unnecessary system overhead. In contrast, a fault in an extended storage pool involves two storage pools with an extension relationship, requiring rapid reporting and synchronous awareness by nodes in both pools to collaboratively complete fault recovery. Therefore, this embodiment introduces differentiated submission rules, determining whether to submit the abnormal event and by whom, based on the storage pool type and the relationship between the detection node and the target storage pool.

[0070] When the target storage pool is a regular storage pool, a conditional commit rule is used. The detection node determines whether it belongs to the target storage pool. If it does, it commits an exception event, which includes the faulty node identifier and the storage pool identifier to which the faulty node belongs. This event is written to the distributed coordination component, and other nodes in that storage pool will listen for the event and initiate a fault handling process. If the detection node does not belong to the target storage pool, the heartbeat expiration event is ignored, and no exception event is committed, as the fault only affects the target storage pool, and the detection node, not belonging to that pool, does not need to participate in the handling. When the target storage pool is an extended storage pool, an unconditional commit rule is used. The detection node directly commits the exception event without needing to determine whether it belongs to the target storage pool. Regardless of whether the detection node is a node in the faulty storage pool, a node in the peer extended storage pool, or any other unrelated node, it immediately commits the exception event, ensuring that the fault is broadcast quickly so that all nodes in both extended storage pools can receive the event promptly and initiate collaborative processing.

[0071] Conditional commits for ordinary storage pools are based on the principle of fault isolation. Failures in an ordinary storage pool only affect nodes within that pool, and only these nodes need to perform data recovery operations. Allowing nodes outside the storage pool to commit events would not only increase system overhead but also lead to duplicate event commits. Therefore, commits are only made when the detecting node belongs to the target storage pool, ensuring that the fault is perceived by nodes within the storage pool while avoiding interference from unrelated nodes. Unconditional commits for extended storage pools are based on the principle of rapid collaboration. Extended storage pools consist of two storage pools, and a node failure in either pool can affect the service capability of the entire extended pair. It is necessary for all nodes in both storage pools to perceive the fault as quickly as possible. Allowing any detecting node to directly commit events allows the fault information to be broadcast to all nodes in both storage pools as quickly as possible, buying time for subsequent collaborative recovery. The committed abnormal event includes at least the identifier of the faulty node and the identifier of the storage pool to which the faulty node belongs. For extended storage pool scenarios, the event may also carry the identifier of the extended storage pool pair so that the receiving node can identify the other storage pool requiring collaborative processing.

[0072] In terms of application scenarios, for a node in a normal storage pool that detects a fault in another node in the same storage pool, assuming that node A and node B both belong to normal storage pool P1, if node A detects that node B's heartbeat has expired, and the target storage pool is P1 and node A belongs to P1, then node A submits an abnormal event, and other nodes in storage pool P1 initiate fault handling after receiving the event.

[0073] In the case where a node outside a normal storage pool detects a fault in a node within the storage pool, assuming node A belongs to normal storage pool P1 and node B belongs to normal storage pool P2, if node A detects that node C (belonging to P1) has exceeded its heartbeat deadline, and the target storage pool is P1 but node A does not belong to P1, then node A ignores the event, and the fault is detected and the event is submitted by other nodes in storage pool P1 (such as node D).

[0074] In the extended storage pool scenario, assuming storage pool A and storage pool B are an extended pair, node X (belonging to storage pool A) detects that the heartbeat of node Y (belonging to storage pool A) has expired. The target storage pool is storage pool A. Node X directly submits an exception event, and all nodes in storage pool A and storage pool B will receive the event. Similarly, node Z (belonging to storage pool B) will also directly submit an event if it detects that node Y has failed, thus realizing dual storage pool collaboration.

[0075] This application sets up a standard storage pool scenario where only nodes within the storage pool are allowed to submit events, achieving fault isolation and preventing unrelated nodes from participating in the processing, thus reducing system overhead. In the extended storage pool scenario, any detection node is allowed to directly submit events, ensuring that faults can be broadcast quickly and shortening fault response time. The submission rules are automatically adapted according to the storage pool type without manual intervention, improving the system's automation level. In the standard storage pool scenario, the attribution relationship is used to determine whether multiple nodes repeatedly submit the same fault event, reducing the load on the distributed coordination component.

[0076] Step 106: In response to the storage node receiving an abnormal event, determine the handling plan for the abnormal event based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool.

[0077] If the storage pool to which the second storage node belongs is a normal storage pool, then determine whether the storage node that received the abnormal event belongs to the target storage pool. If the storage node receiving the abnormal event belongs to the target storage pool, then the hash mapping of the target storage pool is updated to generate a new storage pool version number. The synchronization task completion flag is written to the distributed coordination component to mark the synchronization task of the storage node that received the abnormal event as completed. The data recovery task is executed to rebuild the data copy on the second storage node in the target storage pool. The recovery task completion flag is written to the distributed coordination component to mark the asynchronous task of the storage node that received the abnormal event as completed. If the last storage node detects that the synchronization and asynchronous tasks of all storage nodes in the target storage pool have been completed, the abnormal event handling is determined to be completed.

[0078] If the storage node that received the abnormal event does not belong to the target storage pool, the hash mapping of the target storage pool is updated, a new storage pool version number is generated, the synchronization task completion flag is written to the distributed coordination component, and the synchronization task of the storage node that received the abnormal event is marked as completed. If the last storage node detects that the synchronization task of all storage nodes in the target storage pool has been completed, the abnormal event handling is determined to be completed.

[0079] When a storage node receives an abnormal event and the storage pool to which the faulty node belongs is a normal storage pool, the system executes a differentiated fault handling plan based on whether the receiving node belongs to the target storage pool. This plan includes synchronization tasks and data recovery tasks. The last node to complete the task checks the completion of all tasks to confirm that the fault handling is finished.

[0080] Once the detection node has written the abnormal event to the distributed coordination component according to the submission rules, the event will be listened to by each storage node in the target storage pool. Since the impact of a failure in a normal storage pool is limited to the storage pool itself, the nodes within the storage pool need to perform data recovery to ensure data redundancy, while the nodes outside the storage pool only need to synchronize status information. Therefore, this application sets up a differentiated processing strategy based on whether the receiving node belongs to the target storage pool, which ensures the integrity of the failure recovery and avoids unnecessary resource consumption.

[0081] After parsing the abnormal event and obtaining information about the faulty node and its storage pool, the receiving node performs processing based on the type of the storage pool to which the faulty node belongs and its affiliation with the target storage pool: when the receiving node belongs to the target storage pool to which the faulty node belongs, the node performs synchronization and data recovery tasks.

[0082] Synchronization tasks may include updating the hash mapping of the target storage pool to recalculate the hash distribution based on changes in the node members of the current storage pool and generate a new storage pool version number. After the synchronization task is completed, a synchronization task completion flag is written to the distributed coordination component to mark that the synchronization task of this node has been completed.

[0083] Data recovery tasks may include rebuilding data replicas on the failed node within the target storage pool, selecting other surviving nodes as data sources based on the data replica distribution strategy, and re-copying the data blocks carried on the failed node to other nodes to ensure that the number of data replicas is restored to the set value. After data recovery is completed, a recovery task completion flag is written to the distributed coordination component to mark that the asynchronous task of this node has been completed. When the last node detects that the synchronous and asynchronous tasks of all nodes in the target storage pool have been completed, the node determines that the abnormal event handling is complete, and the failed node is officially removed from the storage pool.

[0084] When the receiving node does not belong to the target storage pool to which the faulty node belongs, the node only performs the synchronization task and does not perform the data recovery task. That is, it updates the hash mapping of the target storage pool to generate a new storage pool version number and writes the synchronization task completion flag to the distributed coordination component to mark that the synchronization task of this node has been completed. When the last node detects that the synchronization tasks of all nodes in the target storage pool have been completed, it determines that the abnormal event handling is complete.

[0085] Understandably, the distinction between synchronous and asynchronous tasks means that all nodes that receive an abnormal event, regardless of whether they belong to the target storage pool, need to execute synchronous tasks to update the hash mapping and increment the storage pool version number. This ensures that all nodes are synchronized with the target storage pool's state changes and that the cluster state is consistent. Asynchronous tasks, on the other hand, are only required to be executed by nodes belonging to the target storage pool. By separating synchronous and asynchronous tasks, nodes outside the storage pool can quickly complete state synchronization without waiting for time-consuming data recovery, thus improving the overall efficiency of fault handling.

[0086] The last node detection mechanism ensures that all nodes' synchronous tasks and asynchronous tasks belonging to the target storage pool nodes are completed before marking the exception event handling as completed. After each node completes its synchronous task, it records its synchronization completion status in the distributed coordination component. After each node belonging to the target storage pool completes its asynchronous task, it records its asynchronous completion status. The last node to complete its task checks the status of all nodes. When all nodes' synchronous tasks and asynchronous tasks of nodes within the target storage pool have been completed, the node marks the exception event handling status as completed. This ensures the atomicity of fault handling and prevents the cluster from entering an inconsistent state when some nodes have not completed their tasks.

[0087] Hash mapping update is the core of the synchronization task. After a node leaves the storage pool, the mapping relationship between data blocks and nodes is recalculated to ensure that subsequent data reads and writes can be correctly routed to the live nodes. A consistent hashing algorithm is usually used to remap data blocks that were originally mapped to the failed node to other nodes.

[0088] The storage pool version number is used to record the version of the storage pool state change with each increment. Each node carries the current storage pool version number when processing business I / O. The server node verifies whether the version number is consistent with the latest version number. If they are inconsistent, the processing is rejected or a redirect is returned, ensuring that the client obtains the latest cluster state and avoiding the client using an expired state to access a failed node during the fault handling process.

[0089] In practical applications, when the receiving node belongs to the target storage pool, assuming that the ordinary storage pool P1 contains three nodes A, B, and C, after node B fails, node A receives an abnormal event. Since node A belongs to P1, it executes a synchronization task to update the hash mapping, incrementing pool_epoch (storage pool version number) from 5 to 6, and executes a data recovery task to copy the data copy on node B back to node C. Node C, which also belongs to P1, also executes a synchronization task and a data recovery task. Node A and node C record the synchronization and asynchronous completion flags, respectively. The last node to complete the task (let's say node C) detects that the synchronization and asynchronous tasks of all nodes (A and C) in P1 have been completed, confirms that the abnormal event handling is complete, and node B is officially removed from P1. For cases where the receiving node does not belong to the target storage pool, assuming that ordinary storage pool P1 contains nodes A, B, and C, and ordinary storage pool P2 contains nodes D, E, and F, after node B fails, node D (belonging to P2) receives an abnormal event. Since node D does not belong to P1, it only performs a synchronization task to update the hash mapping of P1, incrementing pool_epoch from 5 to 6 and marking the synchronization task as complete. Nodes A and C belong to P1 and perform synchronization and data recovery tasks. The last node to complete its task (assuming it is node A) detects that the synchronization tasks of all nodes (A, C, and D) in P1 have been completed, and the asynchronous tasks of nodes (A and C) within P1 have also been completed, thus determining that the abnormal event handling is complete.

[0090] In this application, synchronization and recovery or only synchronization strategies are executed depending on whether the receiving node belongs to the target storage pool. This ensures the integrity of fault recovery and avoids time-consuming data recovery by nodes outside the storage pool, thus improving processing efficiency. By updating the hash mapping and storage pool version number through synchronization tasks, it ensures that all nodes have a consistent understanding of the target storage pool's state, providing accurate routing information for subsequent I / O processing. The mechanism of the last node detecting the completion of all synchronous and asynchronous tasks ensures the atomicity of fault handling, preventing the cluster from entering an inconsistent state when some nodes have not completed their tasks. Nodes belonging to the target storage pool execute data recovery tasks to rebuild data copies on the faulty nodes, ensuring that data redundancy is restored to the set value and guaranteeing data reliability.

[0091] If the storage pool to which the second storage node belongs is an extended storage pool, then determine whether the storage node that received the abnormal event belongs to the target storage pool. If the storage node receiving the abnormal event belongs to the target storage pool, the hash mapping of the target storage pool is updated, a new storage pool version number is generated, and the extension version numbers of the target storage pool and the target extended storage pools that have an extension relationship with the target storage pool are updated synchronously. The synchronization task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its synchronization task. The data recovery task is executed, prioritizing data recovery from non-faulty storage nodes in the target storage pool. If there is no valid data in the target storage pool, data is retrieved from the target extended storage pool for recovery. The recovery task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its asynchronous task. If the last storage node detects that the synchronization and asynchronous tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, and the storage pool version number and extension version number of all storage nodes in the target storage pool and the target extended storage pool are consistent, the abnormal event handling is determined to be complete. If the storage node receiving the abnormal event belongs to the target extended storage pool, the extension version number of the target storage pool and the target extended storage pools that have an extension relationship with the target storage pool are updated synchronously. The synchronization task completion flag is written to the distributed coordination component, marking that the synchronization task of the storage node that received the abnormal event is completed. If the last storage node detects that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed and the extension version number of all storage nodes in the target storage pool and the target extended storage pool is consistent, the abnormal event handling is determined to be completed.

[0092] In this embodiment, when a storage node receives an abnormal event and the storage pool to which the faulty node belongs is an extended storage pool, the system executes a differentiated fault handling scheme based on the affiliation between the receiving node and the target storage pool. This scheme includes synchronization tasks, data recovery tasks, and a version number synchronization mechanism for dual storage pool collaboration. The last node to complete the process checks that all tasks are completed and the version numbers are consistent before confirming the end of the fault handling. In the extended storage pool scenario, two storage pools with an extension relationship (denoted as storage pool A and storage pool B) jointly provide high-availability services across data centers. When a node in one of the storage pools fails, the failure not only affects that storage pool itself but may also affect the service capabilities of the entire extended pair. Therefore, fault handling requires dual storage pool collaboration: nodes in both storage pools need to synchronize their status information; nodes in storage pool A need to perform data recovery, while nodes in storage pool B also need to synchronize their extended version numbers to maintain consistency between the two storage pools. This application adopts a differentiated processing strategy based on whether the receiving node belongs to the faulty storage pool, the peer extended storage pool, or neither storage pool, ensuring fault isolation and data security while achieving eventual consistency between the states of the two storage pools.

[0093] After receiving the abnormal event and obtaining information about the faulty node and its storage pool, the receiving node performs the following processing based on whether the faulty node belongs to an extended storage pool and its affiliation with the target storage pool.

[0094] When the receiving node belongs to the target storage pool (i.e., storage pool A) of the faulty node, it performs a synchronization task and a data recovery task. The synchronization task includes updating the hash mapping of the target storage pool to recalculate the hash distribution based on the changes in the node members of the current storage pool and generating a new storage pool version number (pool_epoch increments). At the same time, the node synchronously updates the extension version number (stc_epoch). This extension version number is synchronously updated by the target storage pool and the target extension storage pools that have an extension relationship with the target storage pool. The extension version numbers of the two storage pools are kept in a consistent increment. After the synchronization task is completed, the node writes the synchronization task completion flag to the distributed coordination component to mark the synchronization task completed by this node. The task has been completed; the data recovery task includes prioritizing data recovery from non-faulty storage nodes in the target storage pool to achieve fault isolation and avoid cross-storage pool traffic affecting the normal service of the peer storage pool. When there is no valid data in the target storage pool, data is pulled from the target extended storage pool for recovery to ensure that no data is lost. After the data recovery is completed, the node writes the recovery task completion flag to the distributed coordination component to mark that the asynchronous task of this node has been completed. When the last node detects that the synchronous and asynchronous tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, and the storage pool version number and the extended version number of all storage nodes in the two storage pools are consistent, the node determines that the abnormal event handling is complete.

[0095] When the receiving node belongs to a target extended storage pool (i.e., storage pool B) that has an extension relationship with the storage pool to which the faulty node belongs, the node only performs a synchronization task and does not perform a data recovery task: the node synchronously updates the extended version number, which is synchronously updated by both the target storage pool and the target extended storage pool. The extended version numbers of the two storage pools are consistent and incrementing. The storage pool version number of the target storage pool is only updated by nodes in storage pool A; nodes in storage pool B do not update the storage pool version number. After the synchronization task is completed, the node writes a synchronization task completion flag to the distributed coordination component to mark that the node's synchronization task has been completed. When the last node detects that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, and the extended version numbers of all storage nodes in the two storage pools are consistent, the node determines that the exception event handling is complete. When the receiving node does not belong to either the target storage pool or the target extended storage pool, the node only writes a synchronization task completion flag to the distributed coordination component to mark that the node's synchronization task has been completed, without updating any version number. When the last node detects that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, the node determines that the exception event handling is complete.

[0096] This application employs a dual-layer versioning mechanism to ensure consistency between the two storage pools. Specifically, the storage pool version number records the state information of a single storage pool, which is updated only by the nodes within that storage pool. When a node in storage pool A fails, the pool_epoch (storage pool version number) of storage pool A increments while the pool_epoch of storage pool B remains unchanged. The extended version number records the collaborative state of two storage pools with an extended relationship, which is updated synchronously by both storage pools. When a node in either storage pool fails, the stc_epoch (extended version number) of both storage pools increments synchronously, ensuring that the two storage pools have a consistent understanding of the overall state of the extended version. Through the dual-layer versioning system, it is possible to accurately distinguish between changes in the internal state of a single storage pool and changes in the collaborative state of the two storage pools, providing an accurate version basis for fault handling.

[0097] In extended storage pool scenarios, completion detection requires simultaneous completion of both synchronous and asynchronous tasks, as well as version number consistency. Specifically, all nodes in storage pool A must have completed both synchronous and asynchronous tasks, all nodes in storage pool B must have completed synchronous tasks, and all nodes in both storage pools must have the same storage pool version number, while all nodes in both storage pools must have the same extended version number. Fault handling is considered successful only when both conditions are met simultaneously. If the version number of any node is inconsistent with other nodes, the task result will be marked as a failure, triggering a retry or manual intervention. Regarding data recovery strategies, data recovery is prioritized from nodes within the same storage pool based on fault isolation principles: when a node in storage pool A fails, data recovery is first attempted from other surviving nodes within storage pool A to minimize cross-storage pool network traffic, reduce the impact on storage pool B, and lower cross-data center bandwidth consumption; when there is no valid data in storage pool A, data is retrieved from storage pool B for recovery to ensure no data loss. This strategy maximizes fault isolation while ensuring data security. The core value of extended storage pools lies in high availability across data centers. Through a dual-versioning mechanism and rigorous completion checks, this embodiment ensures that the states of the two storage pools are completely synchronized after fault handling, providing consistent routing information for subsequent read and write operations and avoiding the risks of split-brain or data inconsistency.

[0098] In practical applications, assume that storage pool A and storage pool B are extensions of each other. Storage pool A contains nodes A1, A2, and A3, and storage pool B contains nodes B1, B2, and B3. Scenario 1: Node A1 fails. Node A2, belonging to storage pool A, receives an abnormal event and executes a synchronization task to update the hash mapping of storage pool A, increasing `pool_epoch` from 5 to 6 and `stc_epoch` from 10 to 11. It then executes a data recovery task, prioritizing data recovery from node A3 within storage pool A. Node A3 also executes both synchronization and data recovery tasks. Nodes B1, B2, and B3 in storage pool B, upon receiving the event, execute synchronization tasks, updating `stc_epoch` from 10 to 11 while keeping `pool_epoch` unchanged. The last node to complete its task detects that the synchronization tasks of all nodes in storage pools A and B are complete, and all nodes have `stc_epoch` of 11, all nodes in storage pool A have `pool_epoch` of 6, and all nodes in storage pool B have `pool_epoch` of 3, thus confirming that the abnormal event handling is complete.

[0099] If node B1 receives an exception event, and node B1 belongs to storage pool B, it executes a synchronization task to update stc_epoch from 10 to 11 without updating pool_epoch. Nodes B2 and B3 also execute synchronization tasks. The last node to complete its task detects that the synchronization tasks of all nodes in storage pool A and storage pool B are complete and that the stc_epochs of all nodes are consistent, thus confirming that the exception event handling is complete. Scenario 3: Assume there is also a regular storage pool P1 containing nodes C1 and C2. If node C1 receives an exception event, and node C1 does not belong to storage pool A or storage pool B, it only writes the synchronization task completion flag to the distributed coordination component. The last node to complete its task detects that the synchronization tasks of all nodes in storage pool A and storage pool B are complete, thus confirming that the exception event handling is complete.

[0100] Thus, a dual-version number mechanism is used to synchronize the collaborative states of the two storage pools in the extended storage pool, ensuring that the states of the two storage pools are consistent after fault handling, thus guaranteeing high availability across data centers. Differentiated processing strategies are executed based on whether the receiving node belongs to the faulty storage pool, the peer storage pool, or an unrelated storage pool, ensuring the integrity of fault recovery while avoiding unnecessary resource consumption. Prioritizing data recovery from the local storage pool achieves fault isolation and reduces cross-storage pool network traffic. When there is no data in the local storage pool, data is fetched from other storage pools to ensure no data loss, balancing performance and reliability. The atomicity of fault handling is ensured by a mechanism where the last node detects the completion of synchronous and asynchronous tasks on all nodes and ensures that the version numbers are consistent, avoiding the risk of split-brain caused by inconsistent states between the two storage pools. In the extended storage pool scenario, if any data center fails, the other data center can still provide services, achieving disaster recovery guarantee with RPO=0.

[0101] It is understood that in this application, timestamps are stored at the storage pool level. Each storage node has an independent timestamp in each storage pool. Any heartbeat thread can read the timestamp of any storage node in any storage pool (because ETCD is global).

[0102] In one embodiment, the load status information of the surviving storage nodes in the target storage pool can be obtained. The load status information includes CPU utilization, disk I / O utilization, and network bandwidth utilization. The storage node with the lowest load is selected as the target node for data recovery based on the load status information. During the data recovery process, the resource allocation weight of the data recovery task is dynamically adjusted according to the business priority.

[0103] To optimize data recovery efficiency and minimize the impact on normal business operations, the system first acquires real-time load status information of all surviving storage nodes in the target storage pool before executing data recovery tasks. This includes metrics such as CPU utilization, disk I / O utilization, and network bandwidth utilization. A weighted comprehensive scoring method can be used to quantitatively evaluate the load of each node, prioritizing the node with the lowest comprehensive load score as the target node for data recovery. This avoids concentrating recovery tasks on high-load nodes, achieving cluster load balancing. During data recovery, the resource allocation weight of the recovery task can be dynamically adjusted based on the priority of the business to which the data to be recovered belongs: high-priority businesses (such as core transaction databases) are allocated higher resource weights and are not subject to rate limiting, while low-priority businesses (such as archive backups) are allocated lower resource weights and I / O rate limiting thresholds are set. Simultaneously, the system monitors the I / O latency of normal business operations in real time. When the business latency exceeds the threshold, the resource consumption of the recovery task is automatically reduced to prioritize the performance of normal business operations; when the business is idle, the recovery resource weight is increased to accelerate data recovery. In this way, while ensuring data reliability, intelligent coordination between recovery tasks and normal business operations is achieved, significantly improving the cluster's resource utilization efficiency and business service quality.

[0104] In one specific way, please refer to Figure 3 The document specifically illustrates the multi-storage pool deployment scheme of this application.

[0105] S10: Storage pool creation and initialization.

[0106] In response to the command to create a storage pool, a storage pool is created on the hyperconverged platform. Host and disk resources are selected for the storage pool, a unique identifier (pool_id) is generated for the storage pool, and the relevant information of the storage pool is written to the database and configuration file. The storage pool includes an all-flash storage pool and a hybrid flash storage pool. The all-flash storage pool consists of solid-state drives (SSDs), while the hybrid flash storage pool consists of a mixture of SSDs and hard disk drives (HDDs).

[0107] S11: Heartbeat thread creation and fault detection.

[0108] After a storage node starts its storage service, it reads the storage pool configuration information from the configuration file, detects the storage pool it belongs to based on its network address, and loads the configuration and disk information of that storage pool. Each node creates an independent heartbeat thread for each of its storage pools; if a node belongs to multiple storage pools, multiple heartbeat threads are created. Each heartbeat thread periodically writes its timestamp in the corresponding storage pool to the distributed coordination component and reads the timestamps of other nodes in that pool. When the difference between the timestamp of another node and the current time exceeds a preset threshold, the node is deemed abnormal and removed from the corresponding storage pool.

[0109] S12: Management node election.

[0110] After the storage service starts, each storage node competes for the management node role of its respective storage pool through the distributed lock mechanism in the distributed coordination component. The node that successfully acquires the distributed lock becomes the management node of that storage pool. The management node is responsible for managing all requests from nodes to join the storage pool.

[0111] S13: Cluster readiness notification and version number initialization.

[0112] Once the management node detects that all storage nodes in its storage pool have joined, it sends a ready notification carrying the storage pool identifier to each storage node in the pool through the distributed coordination component. Storage nodes receiving the ready notification parse the storage pool identifier in the notification. If the node belongs to that storage pool, it sets the storage pool status to normal and initializes the storage pool version number to its initial value; if the node does not belong to that storage pool, it ignores the notification.

[0113] S14: Access endpoint creation and type attribute configuration.

[0114] The system creates iSCSI targets as access endpoints for the storage pool and configures extension attributes for these targets. Here, the iSCSI target is the network access object that provides storage mounting services to the outside world. Clients establish connections with this target through the iSCSI initiator to read and write disk resources within the storage pool. When the extension attribute is the first value, the target is associated with a single storage pool; when the extension attribute is the second value, the target is associated with two storage pools, forming an extension pair, and an extension version number is initialized. This extension version number is used to record the collaborative state of the two associated storage pools. Through the storage pool version number and the extension version number, the system can effectively control the logical processing of faults across multiple storage pools and extended storage pools, ensuring cluster state consistency and guaranteeing correct processing of business I / O.

[0115] S15: Resource Isolation and Event Handling.

[0116] All resources are planned uniformly on a storage pool basis, with the unique identifier of the storage pool serving as the path for resource isolation. Storage pool-related information is stored in both the local and distributed coordination components, indexed by the storage pool identifier. During cluster event processing, the information of the corresponding storage pool is first retrieved based on the storage pool identifier, and the storage pool information is updated during the event processing.

[0117] In one specific way, please refer to Figure 4 The specific single-node fault handling scheme of this application is shown.

[0118] When each node storage service starts and joins the pool, it creates a keep-alive thread for the storage pool. This thread periodically updates the timestamp to the ETCD database. If its own timestamp updates normally, it means that the node's status in the storage pool is normal. At the same time, it reads the heartbeat data of other nodes in the pool to check for any abnormalities.

[0119] If a node's timestamp update expires, the system first checks the pool containing the failed node and obtains the pool's extension attributes. If it's not an extended storage pool, it checks if the node is in that pool. If not, the heartbeat expiration event is ignored. If it is in the storage pool, an event indicating that the node has left the pool is submitted to ETCD, notifying other nodes in the pool that a node has failed and left the cluster. If a node belongs to multiple pools, a corresponding number of tasks will be generated to indicate that a node has left its storage pool. If it belongs to an extended storage pool, an event indicating that the failed node has left the pool is submitted directly.

[0120] When a regular storage pool receives a task from the environment indicating that a node has left the storage pool, if the node receiving the leave event is not in the corresponding storage pool, it only performs a synchronization task. This involves updating the pool's first-level hash, generating a new pool epoch, and then updating the synchronization task key to ETCD, indicating that the node has completed its synchronization task. If the node is in the corresponding storage pool, in addition to performing the above synchronization task, it also needs to perform a data recovery task for the corresponding storage pool. After the data recovery is complete, it updates the asynchronous key to ETCD, indicating that the node has completed its asynchronous task. Only when the last node to complete the task detects that all synchronization and asynchronous processes have been processed normally is the task for the faulty node leaving the storage pool considered complete.

[0121] In an extended storage pool environment, the difference from the above process is that `stc_epoch` also needs to be updated during the synchronization phase. `stc_epoch` is updated based on the pool epoch. For example, if storage pools A and B are in an extended relationship, and a node in storage pool A fails, both storage pools will synchronize and update their `stc_epoch`. However, only storage pool A's pool epoch will be updated; storage pool B's pool epoch will remain unchanged. Finally, it will be checked that all nodes' pool epochs and `stc_epoch` are synchronized and consistent for the failed task to be considered successfully processed. If any node processes an epoch or its contents are inconsistent with other nodes, the task result will be marked as failed.

[0122] In an extended storage pool environment, data recovery within a storage pool is prioritized and completed from nodes within that pool, without affecting other extended storage pools. This fault isolation mechanism effectively improves system stability. If a storage pool lacks valid data, data will be retrieved from another extended storage pool for recovery, enhancing the data security of the storage system.

[0123] This application achieves the following significant benefits by introducing a multi-storage pool design and management method for distributed storage systems based on hyperconverged architecture: Significantly improved resource utilization: By logically pooling physical disk resources according to business needs, unified planning and scheduling of storage resources are achieved, avoiding the problems of resource silos and low utilization in traditional distributed storage systems, and greatly improving the utilization efficiency of hardware resources.

[0124] Enhanced system reliability and fault isolation capabilities: Through a multi-storage pool mechanism, the system can isolate faults by storage pool, preventing single-point failures from spreading to the entire system. Especially in high-availability extended cluster scenarios, the fault recovery process is more controllable and reliable, effectively ensuring business continuity and data security.

[0125] Flexible performance and capacity adaptation: Supports the division of fast and slow storage pools (such as all-flash storage pools and mixed flash storage pools), allowing services of different priorities to be deployed in storage pools with different performance characteristics. This satisfies the stringent requirements of high-performance services for IOPS and latency, while also taking into account the storage needs of large-capacity services, achieving the optimal balance between performance and capacity.

[0126] Automation and intelligence in operation and maintenance management: Through ETCD-based distributed state management (such as pool_epoch and stc_epoch mechanisms), heartbeat detection, and Leader (management storage node) election mechanisms, automatic discovery, state synchronization, fault detection and recovery of resource storage pools are realized, which greatly reduces the complexity of manual intervention and improves the system's operability and management efficiency.

[0127] Excellent scalability: The multi-storage pool architecture supports dynamic scaling up and down. New storage nodes can be added to specific storage pools as needed. The system can be flexibly expanded according to business growth, with good horizontal scaling capabilities, adapting to the ever-changing storage needs of large-scale data centers.

[0128] It should be understood that, although Figure 1 , Figure 3 , Figure 4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 3 , Figure 4 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0129] In one embodiment, such as Figure 5 As shown, a resource management device is provided, including: an allocation module 20, a creation module 21, a sending module 22, and a determination module 23, wherein: The allocation module 20 is used to create multiple storage pools and assign a unique identifier to each storage pool. Create module 21 to respond to storage nodes joining the storage pool, create a heartbeat thread for the storage nodes joining the storage pool, and elect a management storage node among the storage nodes corresponding to each storage pool; The sending module 22 is used to send a ready notification to the storage nodes in its storage pool in response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool; the storage node that receives the ready notification initializes its storage pool version number to an initial value; The determination module 23 is used to determine the target type attribute of the target storage pool to which the second storage node belongs when the first storage node detects through the heartbeat thread that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold; determine whether to submit an abnormal event and the storage node that submits the abnormal event based on the target type attribute and the affiliation relationship between the first storage node and the target storage pool; and determine the handling scheme for the abnormal event based on the target type attribute and whether the storage node that received the abnormal event belongs to the target storage pool when the storage node receives the abnormal event.

[0130] Specific limitations regarding the resource management device can be found in the limitations of the resource management method described above, and will not be repeated here. Each module in the aforementioned resource management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0131] In one embodiment, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the resource management methods provided by the above methods.

[0132] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a resource management method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0133] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Step 101: Create multiple storage pools and assign a unique identifier to each storage pool.

[0135] Step 102: In response to a storage node joining a storage pool, a heartbeat thread is created for the storage node joining the storage pool, and a management storage node is elected from among the storage nodes corresponding to each storage pool.

[0136] Step 103: In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, it sends a ready notification to the storage nodes in its storage pool; the storage node that receives the ready notification initializes its storage pool version number to the initial value.

[0137] Step 104: In response to the first storage node detecting through its heartbeat thread that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the target type attribute of the target storage pool to which the second storage node belongs is determined.

[0138] Step 105: Based on the target type attribute and the affiliation relationship between the first storage node and the target storage pool, determine whether to submit an abnormal event and the storage node that submits the abnormal event.

[0139] Step 106: In response to the storage node receiving an abnormal event, determine the handling plan for the abnormal event based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Step 101: Create multiple storage pools and assign a unique identifier to each storage pool.

[0141] Step 102: In response to a storage node joining a storage pool, a heartbeat thread is created for the storage node joining the storage pool, and a management storage node is elected from among the storage nodes corresponding to each storage pool.

[0142] Step 103: In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, it sends a ready notification to the storage nodes in its storage pool; the storage node that receives the ready notification initializes its storage pool version number to the initial value.

[0143] Step 104: In response to the first storage node detecting through its heartbeat thread that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the target type attribute of the target storage pool to which the second storage node belongs is determined.

[0144] Step 105: Based on the target type attribute and the affiliation relationship between the first storage node and the target storage pool, determine whether to submit an abnormal event and the storage node that submits the abnormal event.

[0145] Step 106: In response to the storage node receiving an abnormal event, determine the handling plan for the abnormal event based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A resource management method, characterized in that, include: Create multiple storage pools and assign a unique identifier to each storage pool; In response to a storage node joining a storage pool, a heartbeat thread is created for the storage node that has joined the storage pool, and a management storage node is elected from among the storage nodes corresponding to each storage pool; In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, it sends a ready notification to the storage nodes in its storage pool. Upon receiving the readiness notification, the storage node initializes its storage pool version number to an initial value; In response to the first storage node detecting through its heartbeat thread that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the target type attribute of the target storage pool to which the second storage node belongs is determined. Based on the target type attribute and the affiliation between the first storage node and the target storage pool, determine whether to submit an abnormal event and the storage node that submits the abnormal event; In response to a storage node receiving an abnormal event, the handling plan for the abnormal event is determined based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool.

2. The method according to claim 1, characterized in that, The process of creating multiple storage pools and assigning a unique identifier to each storage pool includes: In response to receiving an instruction to create a storage pool, select the corresponding host and disk resources for each storage pool; Create multiple storage pools and assign a unique identifier to each storage pool; The configuration information of each storage pool is written into the database and configuration file. The configuration information includes the unique identifier, host and disk resources corresponding to the storage pool. The storage pool includes an all-flash storage pool and a hybrid flash storage pool. The all-flash storage pool is composed of solid-state drives (SSDs), and the hybrid flash storage pool is composed of a mixture of SSDs and hard disk drives (HDDs).

3. The method according to claim 1, characterized in that, The step of creating a heartbeat thread for a storage node that has joined the storage pool in response to the storage node joining the storage pool includes: Start the storage service on the storage node and read the storage pool configuration information from the configuration file; The storage pool corresponding to the storage node is determined based on the network address of the storage node, and the configuration information of the storage pool is loaded. The storage node corresponds to at least one storage pool. A heartbeat thread is created for each storage pool corresponding to the storage node, and each heartbeat thread independently maintains the heartbeat status of the storage node in each storage pool. Each heartbeat thread periodically writes the timestamp of the storage node in each storage pool and reads the timestamp of each storage node in the corresponding storage pool. When the difference between the timestamp of any storage node and the current time exceeds a preset threshold, the storage node is identified as an abnormal storage node and removed from the corresponding storage pool.

4. The method according to claim 1, characterized in that, The process of electing a management storage node from among the storage nodes corresponding to each storage pool includes: Create distributed locks for each storage pool; Each storage node is elected by acquiring the distributed lock corresponding to each storage pool. The storage node that successfully acquires the distributed lock in each storage pool is designated as the management storage node of that storage pool.

5. The method according to claim 1, characterized in that, The response is that when the management storage node detects that all storage nodes in its storage pool have joined the storage pool, it sends a ready notification to the storage nodes in its storage pool. Upon receiving the readiness notification, the storage node initializes its storage pool version number to an initial value, including: The management storage node detects whether all storage nodes in its storage pool have been added to the storage pool; In response to the management storage node detecting that all storage nodes in its storage pool have joined the storage pool, the management storage node sends a ready notification carrying the storage pool identifier corresponding to the management storage node to each storage node in its storage pool; The storage node that receives the ready notification parses the storage pool identifier carried in the ready notification; If a storage node that receives a ready notification belongs to the storage pool corresponding to the storage pool identifier, the storage pool status of the storage node is set to normal, and the storage pool version number of the storage node is initialized to the initial value. If the storage node receiving the ready notification does not belong to the storage pool corresponding to the storage pool identifier, the ready notification is ignored.

6. The method according to claim 1, characterized in that, After the storage node that receives the ready notification initializes its storage pool version number to an initial value, the following steps are included: Create an access endpoint for the storage pool and configure endpoint properties for the access endpoint, the endpoint properties being used to indicate the number of storage pools associated with the access endpoint.

7. The method according to claim 1, characterized in that, The response of the first storage node detecting through its heartbeat thread that the difference between the last update timestamp and the current time of the second storage node exceeds a preset threshold, determines the target type attribute of the target storage pool to which the second storage node belongs, including: The first storage node detects the timestamps of each storage node through a heartbeat thread; In response to the first storage node detecting that the difference between the last update timestamp of the second storage node and the current time exceeds a preset threshold, the second storage node is determined to be a faulty storage node. Get the target storage pool to which the second storage node belongs and the target type attribute of the target storage pool.

8. The method according to claim 1, characterized in that, The target type attribute includes ordinary storage pools and extended storage pools. The step of determining whether to submit an abnormal event and the storage node that submits the abnormal event based on the target type attribute and the affiliation relationship between the first storage node and the target storage pool includes: In response to the target storage pool being a normal storage pool, determine whether the first storage node belongs to the target storage pool; If the first storage node belongs to the target storage pool, then the first storage node submits an exception event; If the first storage node does not belong to the target storage pool, the heartbeat expiration event is ignored; If the target storage pool is an extended storage pool, the first storage node directly submits an exception event.

9. The method according to claim 1, characterized in that, The step of determining the handling scheme for abnormal events based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool includes: If the storage pool to which the second storage node belongs is a normal storage pool, then determine whether the storage node that received the abnormal event belongs to the target storage pool. In response to the storage node receiving the abnormal event belonging to the target storage pool, the hash mapping of the target storage pool is updated to generate a new storage pool version number. A synchronization task completion flag is written to the distributed coordination component to mark the storage node that received the abnormal event as having completed its synchronization task. A data recovery task is then executed to rebuild the data copy on the second storage node within the target storage pool. A recovery task completion flag is written to the distributed coordination component to mark the storage node that received the abnormal event as having completed its asynchronous task. In response to the last storage node detecting that all storage nodes in the target storage pool have completed both their synchronization and asynchronous tasks, the abnormal event handling is determined to be complete. If the storage node that received the abnormal event does not belong to the target storage pool, the hash mapping of the target storage pool is updated, a new storage pool version number is generated, the synchronization task completion identifier is written to the distributed coordination component, and the synchronization task of the storage node that received the abnormal event is marked as completed. If the last storage node detects that the synchronization tasks of all storage nodes in the target storage pool have been completed, the abnormal event handling is determined to be completed.

10. The method according to claim 1, characterized in that, The step of determining the handling scheme for abnormal events based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool also includes: If the storage pool to which the second storage node belongs is an extended storage pool, then determine whether the storage node that received the abnormal event belongs to the target storage pool. In response to the storage node receiving the abnormal event belonging to the target storage pool, the hash mapping of the target storage pool is updated, a new storage pool version number is generated, and the extension version numbers of the target storage pool and the target extended storage pools with an extension relationship to the target storage pool are updated synchronously. The synchronization task completion flag is written to the distributed coordination component to mark the storage node that received the abnormal event as having completed its synchronization task. The data recovery task is executed, prioritizing data recovery from non-faulty storage nodes in the target storage pool. When there is no valid data in the target storage pool, data is retrieved from the target extended storage pool for recovery. The recovery task completion flag is written to the distributed coordination component to mark the storage node that received the abnormal event as having completed its asynchronous task. In response to the last storage node detecting that the synchronization and asynchronous tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed and that the storage pool version number and the extension version number of all storage nodes are consistent, the abnormal event handling is determined to be complete. In response to the storage node receiving the abnormal event belonging to the target extended storage pool, the extension version number of the target storage pool and the target extended storage pools with an extension relationship with the target storage pool are synchronously updated, and the synchronization task completion flag is written to the distributed coordination component to mark the storage node that received the abnormal event as having completed its synchronization task; in response to the last storage node detecting that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed and that the extension version numbers of all storage nodes are consistent, the abnormal event handling is determined to be completed.

11. The method according to claim 10, characterized in that, The step of determining the handling scheme for abnormal events based on the target type attribute and whether the storage node receiving the abnormal event belongs to the target storage pool also includes: In response to the storage node receiving the abnormal event not belonging to the target storage pool or the target extended storage pool, a synchronization task completion flag is written to the distributed coordination component, marking the storage node that received the abnormal event as having completed its synchronization task; in response to the last storage node detecting that the synchronization tasks of all storage nodes in the target storage pool and the target extended storage pool have been completed, the abnormal event handling is determined to be complete.

12. The method according to claim 10, characterized in that, The storage pool version number is used to record the status information of the storage pool, and the extended version number is used to record the collaborative status of two storage pools with an extended relationship.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images