Processing method and device after active fault of node in SDS storage domain

Abstract

The invention provides a processing method and device for a node after an active fault in an SDS storage domain, and relates to the technical field of software defined storage node fault processing technology.According to the processing method for the node after the active fault in the SDS storage domain, intelligent role dynamic adjustment after the active fault of the node can be achieved, and the processing efficiency of the node after the active fault of the node is improved. Consistency of storage domain configuration and service states is ensured, system stability is improved, and operation and maintenance cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of software-defined storage node fault handling technology, and more particularly to a method and apparatus for handling active node failures in an SDS storage domain. Background Technology

[0002] Software-defined storage (SDS), as a core technology in cloud computing and big data storage architectures, is widely used in enterprise data centers, cloud platforms, and distributed storage systems. Related technologies utilize the collaborative operation of cloud management platforms, node role allocation mechanisms, and service status monitoring modules to construct a storage domain architecture with high scalability and flexibility. Specifically, this architecture covers the entire process from node initialization and configuration to dynamic role adjustment, including key aspects such as metadata management of Master nodes, data storage of Storage nodes, and real-time perception and response to node status. With the evolution of SDS architecture, dynamic adjustment of node roles has become an important means of ensuring stable system operation, especially in scenarios involving active node failures (such as master node shutdown), where a processing mechanism that can balance system consistency, stability, and automated response is urgently needed.

[0003] However, existing node failure handling methods, which rely on manual intervention or automated strategies lacking role awareness, fail to achieve a strong consistency between service status and role information. This can lead to inconsistencies between roles and actual service status, resulting in abnormal data distribution and metadata management failures. Specifically, traditional solutions typically rely on administrators to manually assess and execute upgrade / downgrade operations after detecting a node failure, which not only has high response latency but is also prone to human error. Furthermore, some automated mechanisms do not consider the threshold relationship between the number of Master nodes and system stability, and blindly triggering role changes may cause cluster instability or configuration conflicts. In addition, existing technologies lack orderly processing logic in multi-node concurrent failure scenarios, making it difficult to achieve efficient and reliable fault recovery in large-scale storage domains, thus affecting the overall availability and operational efficiency of the system. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for handling node active failures in an SDS storage domain.

[0006] The second objective of this invention is to provide a processing device for node active failure in an SDS storage domain.

[0007] The third objective of this invention is to provide an electronic device.

[0008] The fourth objective of this invention is to provide a computer-readable storage medium.

[0009] The fifth objective of this invention is to provide a computer program product.

[0010] To achieve the above objectives, a first aspect of the present invention provides a method for handling node active failure in an SDS storage domain, comprising: S1. Detect node failure and determine the role of the failed node; S2. When the failed node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes, randomly select a node from the surviving Storage nodes and promote it to Master node; S3. When the failed node comes back online, if it was originally a Master node, trigger a downgrade operation and change its role from Master node to Storage node; S4. Through a mutex lock mechanism, block manual upgrade / downgrade operations during the execution of automatically triggered upgrade or downgrade operations to prevent configuration file conflicts.

[0011] In one embodiment of the present invention, the role of detecting node active failure and determining the faulty node further includes: S11, detecting the service status of the node in real time through the underlying service status monitoring module, including whether the etcd service and superx-manage service are closed; S12, determining whether the node is actively failing based on the service status change, and if the service status is abnormal and no hardware failure is detected, it is determined to be an active failure of the node.

[0012] In one embodiment of the present invention, the step of randomly selecting a node from the surviving Storage nodes and upgrading it to a Master node further includes: S21, performing a weighted random selection based on the load status and storage capacity of the surviving Storage nodes, and prioritizing the selection of nodes with lower load and sufficient storage capacity; S22, verifying whether the selected Storage node has the basic service components and configuration conditions required for the upgrade before the upgrade operation.

[0013] In one embodiment of the present invention, the step of triggering the downgrade operation and adjusting its role from Master node to Storage node further includes: S31, when the faulty node comes back online, detecting whether its service status has returned to normal and confirming whether it still has the service components required by the Master node; S32, if it is detected that the number of surviving Master nodes exceeds a preset threshold, then the node is downgraded to Storage node first to maintain the cluster role balance.

[0014] In one embodiment of the present invention, the step of blocking manual upgrade or downgrade operations during the execution of automatically triggered upgrade or downgrade operations via a mutex lock mechanism further includes: S41, locking the upgrade / downgrade configuration file in the cloud management platform through a distributed lock mechanism to prevent multiple operations from modifying the same configuration simultaneously; S42, when a manual operation request occurs, if an automatic operation is detected to be in progress, the manual operation request is added to a waiting queue and executed only after the automatic operation is completed.

[0015] In one embodiment of the present invention, the method further includes: S5, when multiple nodes are detected to have actively failed at the same time, generating a processing queue based on the timestamp of the failure event, and performing role compensation operations in sequence according to the queue order.

[0016] To achieve the above objectives, a second aspect of the present invention provides a processing apparatus for a node in an SDS storage domain that handles active failures, comprising: The fault detection and role determination module is used to detect active node failures and determine the role of the failed node; the Master node upgrade module is used to randomly select a node from the surviving Storage nodes and upgrade it to a Master node when the failed node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes; the Master node demotion module is used to trigger a demotion operation when a failed node comes back online, if it was originally a Master node, and change its role from Master node to Storage node; the mutex lock control module is used to block manual upgrade or demotion operations during the execution of automatically triggered upgrade or demotion operations through a mutex lock mechanism to prevent configuration file conflicts.

[0017] In one embodiment of the present invention, the fault detection and role determination module is further configured to: detect the service status of the node in real time through the underlying service status monitoring module, including whether the etcd service and superx-manage service are shut down; determine whether the node is actively faulty based on the changes in service status, and if the service status is abnormal and no hardware failure is detected, it is determined to be an actively faulty node.

[0018] In one embodiment of the present invention, the Master node upgrade module is further configured to: perform weighted random selection based on the load status and storage capacity of the surviving Storage nodes, and prioritize the selection of nodes with lower load and sufficient storage capacity; before the upgrade operation, verify whether the selected Storage node has the basic service components and configuration conditions required for the upgrade.

[0019] In one embodiment of the present invention, the Master node degradation module is further configured to: detect whether the service status of a faulty node has returned to normal after the faulty node comes back online, and confirm whether it still has the service components required by the Master node; if the number of surviving Master nodes exceeds a preset threshold, the node is preferentially downgraded to a Storage node to maintain the cluster role balance.

[0020] In one embodiment of the present invention, the mutex lock control module is further configured to: lock the upgrade / downgrade configuration file in the cloud management platform through a distributed lock mechanism to prevent multiple operations from modifying the same configuration at the same time; when a manual operation request occurs, if an automatic operation is detected to be being executed, the manual operation request is added to the waiting queue and executed after the automatic operation is completed.

[0021] In one embodiment of the present invention, it further includes: a multi-node fault handling module, which is used to generate a processing queue based on the timestamp of the fault event when multiple nodes are detected to be actively faulty at the same time, and to perform role compensation operations in sequence according to the queue order.

[0022] To achieve the above objectives, a third aspect of the present invention provides an electronic device comprising: processor; A memory that stores executable instructions; when the processor executes the instructions, it implements the method as claimed in any one of claims 1-6.

[0023] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method as claimed in any one of claims 1-6.

[0024] The methods, apparatus, electronic devices, and computer-readable storage media of the present invention can realize intelligent dynamic adjustment of roles after node active failure, ensure system stability and data consistency, reduce operation and maintenance costs, and avoid configuration conflicts.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a method for handling node active failure in an SDS storage domain, as provided in an embodiment of the present invention. Figure 2This is a schematic diagram of a device for handling node active failure in an SDS storage domain, provided in an embodiment of the present invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] Figure 1 This is a flowchart illustrating a method for handling active node failures in an SDS storage domain, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: S1 detects node active failures and determines the role of the failed node.

[0029] Specifically, in the SDS (Software-Defined Storage) domain, detecting node-initiated failures and determining their roles is a crucial step in implementing a dynamic role compensation mechanism. This step involves real-time monitoring of node status through a cloud management platform, combined with the reporting mechanism of underlying service status (such as etcd, superx-manage, and other critical services), to achieve accurate identification and role determination of node-initiated failure behaviors.

[0030] At the technical implementation level, the cloud management platform communicates with nodes through a heartbeat monitoring mechanism to detect whether a node is in a state of active failure. Active failure typically refers to offline behavior caused by non-abnormal reasons such as master shutdown or service voluntary cessation, which is different from passive failures such as network interruption or hardware failure. When a node fails to respond for a preset number of consecutive times (e.g., 3 heartbeat timeouts with an interval of 5 seconds), the system determines it to be in a "active failure". Subsequently, the platform queries the node registration information database to obtain the node's current role (Master or Storage) and further confirms the consistency of the role status by combining it with its service status (e.g., whether the etcd service is shut down).

[0031] At the parameter level, this step involves several key parameters: the heartbeat interval is typically set to 5 seconds, and the timeout count is 3 times; the total number of surviving nodes is used to calculate the Master node threshold, ensuring that the system has sufficient redundancy before automatic upgrades. In addition, the service status monitoring module needs to support millisecond-level response (e.g., completing status reporting within 100ms) to ensure the real-time nature of role correction.

[0032] In application scenarios, this step is widely applicable to large-scale distributed storage systems, especially in cloud data centers and virtualized storage platforms. When the Master node actively fails, the system needs to quickly identify and initiate a role compensation process to avoid metadata service interruption and impact on the availability of the entire storage domain. Through this step, the cloud management platform can respond immediately after a failure occurs, ensuring that the system configuration remains consistent with the actual service status.

[0033] The technical advantage of this step lies in its ability to automatically identify node roles and fault types, effectively avoiding system instability issues caused by delays or errors in manual judgment, as seen in traditional solutions. Simultaneously, this mechanism provides accurate decision-making basis for subsequent automatic upgrade or downgrade operations, making it a core component for achieving high availability and elastic scaling of SDS storage domains.

[0034] Furthermore, S1 includes: S11 uses the underlying service status monitoring module to detect the service status of nodes in real time, including whether the etcd service and superx-manage service are shut down.

[0035] Specifically, this step involves real-time monitoring of the service status of nodes in the SDS storage domain through the underlying service status monitoring module, including determining whether the etcd and superx-manage services are shut down. This step is a crucial preliminary step in the entire proactive node failure handling mechanism. Its technical implementation is based on a proactive service status detection and reporting mechanism, ensuring that the system can quickly and accurately identify proactive failure behaviors of nodes, thereby triggering subsequent automatic upgrade / downgrade processes.

[0036] At the technical implementation level, the underlying service status monitoring module is typically deployed at the operating system level of each node. It periodically collects the running status of etcd and superx-manage services through methods such as heartbeat detection, service port listening, and process status polling. etcd is a distributed key-value store system used to store cluster metadata and configuration information; its service status directly affects whether a node is qualified to play the Master role. superx-manage is a node management service responsible for node status reporting, configuration synchronization, and receiving control commands; its shutdown usually indicates that the node has actively gone offline or entered maintenance mode. The monitoring module determines the status based on a preset detection period (e.g., every 30 seconds) and a timeout threshold (e.g., three consecutive no-response attempts are considered a service shutdown).

[0037] In terms of parameters, the service status monitoring module needs to be configured with key parameters such as the detection cycle, service response timeout, and consecutive failure threshold to ensure that the system can accurately identify service anomalies within 90 seconds after a node is actively shut down. In addition, the system needs to maintain a global node status database to record the current service status and role information of each node, ensuring the real-time nature and consistency of the status information.

[0038] In application scenarios, this step applies to situations where the Master node or Storage node in an SDS storage domain experiences service interruptions due to proactive actions such as master shutdown or maintenance restarts. By monitoring the status of etcd and superx-manage services in real time, the system can distinguish between proactive and reactive failures (such as network interruptions), thereby avoiding unnecessarily triggering role changes and improving the accuracy of fault handling and system stability.

[0039] The technical benefit of this step is that it provides a reliable state basis for the subsequent automatic upgrade and downgrade mechanism, ensuring that the system can promptly trigger the upgrade operation of Storage nodes when the number of Master nodes falls below a threshold, thereby maintaining the high availability of the cluster. Simultaneously, through real-time role correction driven by service status, strong consistency is achieved among "service status - database roles - configuration information," effectively avoiding configuration conflicts and metadata inconsistencies caused by state lag, and improving the self-healing capability and operational efficiency of the SDS storage domain in complex fault scenarios.

[0040] S12, determine whether the node is actively failing based on changes in service status. If the service status is abnormal and no hardware failure is detected, it is determined to be an actively failing node.

[0041] Specifically, in this invention, the step of "determining whether a node has committed an active failure based on changes in service status, and determining it as an active failure if the service status is abnormal and no hardware failure is detected" is a key prerequisite for implementing a dynamic compensation mechanism for node roles in an SDS (Software-Defined Storage) storage domain. The core of this step lies in achieving accurate differentiation between "active failures" and "passive failures" through the collaborative judgment of service status monitoring and hardware status detection, thereby providing accurate decision-making basis for subsequent automatic upgrade / downgrade operations.

[0042] At the technical implementation level, this step relies on the service status monitoring module built into the cloud management platform. This module continuously polls or subscribes to the status of key services on the nodes, such as etcd and superx-manage. When the service status detection module of a node reports a service anomaly (such as service stoppage, response timeout, heartbeat loss, etc.), the system will first call the hardware status detection interface (such as IPMI, iLO, BMC, etc.) to confirm whether the node has experienced hardware failure (such as power failure, motherboard malfunction, network disconnection, etc.). If the hardware status is normal, but the service status remains abnormal (such as three consecutive failed heartbeat detections with an interval of 5 seconds), then the node is determined to have a deliberate failure, i.e., a human-induced or planned shutdown, rather than a sudden hardware failure.

[0043] At the parameter level, the determination of service status anomalies is typically based on preset detection cycles and failure thresholds. For example, the service status detection cycle can be set to 5 seconds, the consecutive failure threshold to 3 times, and the total timeout to 15 seconds. Hardware failure detection relies on the response status codes of the hardware management interface. For instance, "0x00" returned by the IPMI interface indicates that the node is normal, while "0x01" or "0x02" indicates a hardware failure. In addition, the system needs to maintain a node status database to record the current service status and hardware status of each node, ensuring the real-time nature and consistency of status information.

[0044] At the application level, this step is widely applicable to proactive shutdown scenarios for Master or Storage nodes in SDS storage domains, such as planned maintenance, node migration, and load balancing. By distinguishing between proactive and reactive failures, the system can avoid unnecessary role changes for unplanned failure nodes, thereby preventing cluster instability caused by misjudgment.

[0045] The technical benefit of this step lies in its intelligent identification of node failure types, providing precise triggering conditions for subsequent automatic upgrade / downgrade mechanisms. Through dual verification of service status and hardware status, the system effectively avoids accidental role changes triggered by non-faulty events such as network jitter or temporary service restarts, thereby improving the stability of the entire SDS storage domain and the accuracy of automated processing. Furthermore, this step lays the foundation for achieving "service status-driven real-time role correction," ensuring strong consistency among "service status, database roles, and configuration information."

[0046] S2, when the failed node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes, randomly select one node from the surviving Storage nodes and promote it to Master node. Specifically, when the failed node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes, the system will randomly select one node from the surviving Storage nodes and promote it to Master node. This step is one of the core implementations of the "dynamic role compensation mechanism" in this invention, which aims to ensure that the SDS (Software Defined Storage) storage domain can still maintain high availability and system stability after a node fails actively (such as a master shutdown).

[0047] At the technical implementation level, this step relies on the cloud management platform's real-time monitoring capabilities of node status. The cloud management platform continuously collects node operational status information through underlying service status monitoring modules (such as the status detection of critical services like etcd and superx-manage). When a Master node is detected to have actively shut down (i.e., a planned or unexpected shutdown rather than a crash-related failure), the system first calculates the total number of currently surviving nodes (…). ) and the number of surviving Master nodes ( ), and according to the preset threshold formula The system determines whether the upgrade conditions are met. If not, it indicates that the current number of Master nodes is insufficient to maintain the quorum of the cluster, posing a risk of metadata management failure. In this case, the system will trigger the role compensation process.

[0048] The specific operation is as follows: The system selects a node as a candidate for upgrade from all nodes currently in the "Storage" role and with normal service status using a random selection algorithm (such as weighted random or uniform random). This algorithm can be further optimized to a weighted selection based on indicators such as node load, storage capacity, and network latency to improve overall system performance. After selecting a node, the cloud management platform will send a role upgrade command to that node through a configuration management interface (such as REST API or gRPC), upgrading it from a Storage node to a Master node, and synchronously updating the cluster configuration database (such as etcd cluster configuration) and node status table to ensure that all nodes agree on the role change.

[0049] At the parameter level, the key parameters involved in this step include the total number of surviving nodes. Number of surviving Master nodes Threshold and the weighting factors of the random selection algorithm (such as load balancing weights) Storage capacity weight In addition, the system must ensure that the availability and consistency requirements of the CAP theorem are met during the upgrade process. Raft consensus algorithm or Paxos protocol are usually used to ensure the atomicity and consistency of configuration changes.

[0050] In application scenarios, this step is suitable for situations where the Master node in an SDS storage domain experiences a proactive failure, such as master node shutdown for maintenance, node migration, or temporary offline operation. In large-scale distributed storage systems, this mechanism can effectively avoid cluster unavailability issues caused by insufficient Master nodes, while reducing manual intervention and improving the system's self-healing capabilities.

[0051] The technical advantage of this step is that by dynamically compensating for the number of Master nodes, the cluster always has enough management nodes to maintain the normal operation of metadata services, thereby improving the system's fault tolerance and service continuity. At the same time, the random selection strategy avoids the uneven load distribution that may result from a fixed strategy, enhancing the system's robustness and scalability.

[0052] Furthermore, S2 includes: S21, perform weighted random selection based on the load status and storage capacity of surviving Storage nodes, prioritizing nodes with lower load and sufficient storage capacity. Specifically, in some implementations, a weighted random selection based on the load status and storage capacity of surviving Storage nodes, prioritizing nodes with lower load and sufficient storage capacity, is one of the core steps in implementing the dynamic role compensation mechanism in this invention. This step aims to quickly and reasonably select a node from the Storage nodes for role upgrade after the Master node actively fails, in order to maintain the high availability and data consistency of the cluster.

[0053] From a technical implementation perspective, this step involves using the cloud management platform's monitoring module to collect real-time data on the operational status of each Storage node, including load metrics such as CPU utilization, memory usage, disk I / O throughput, and network latency, as well as storage capacity parameters such as remaining disk space and available storage pool capacity. The system normalizes these metrics and calculates a comprehensive score for each node based on preset weighting coefficients (e.g., load weight 0.6, storage capacity weight 0.4). A lower score indicates a lighter node load, more abundant storage resources, and greater suitability as an upgrade candidate node.

[0054] At the parameter level, the load status is typically collected every 30 seconds, and the storage capacity is monitored every 5 minutes to ensure data real-time performance and efficient use of system resources. Weighting coefficients can be dynamically adjusted based on the actual deployment environment. For example, in high-concurrency write scenarios, the weight of storage capacity can be appropriately increased to prevent insufficient disk space from causing the upgraded Master node to be unable to support metadata services.

[0055] In terms of application scenarios, this step is applicable to situations where the system needs to quickly restore control plane functionality after a Master node in an SDS storage domain experiences a proactive failure (such as a master shutdown). By using a weighted random selection strategy, the system can avoid performance bottlenecks or data migration failures caused by selecting nodes with high load or limited storage resources, while ensuring load balancing.

[0056] From a technical perspective, this step effectively improves the system's recovery efficiency and stability after node failures, avoiding resource contention issues that may arise from traditional random selection strategies. By introducing a multi-dimensional evaluation model, the system achieves optimal resource allocation during role upgrades, thereby enhancing the resilience and fault tolerance of the SDS storage domain.

[0057] S22. Before the upgrade operation, verify whether the selected Storage node has the basic service components and configuration conditions required for the upgrade.

[0058] Specifically, verifying whether the selected Storage node possesses the necessary basic service components and configuration conditions for the upgrade before the upgrade operation is a key preliminary step in implementing the dynamic role compensation mechanism in this invention. The core technical principle of this step lies in ensuring that the candidate node meets the operating requirements of the Master node before the role switch through service status detection and configuration verification mechanisms, thereby avoiding system instability or service interruption caused by erroneous role upgrades.

[0059] At the technical implementation level, the cloud management platform collects real-time service status information of nodes through underlying service monitoring modules (such as etcd, superx-manage, and other key services). When the system determines that a Storage node needs to be upgraded to a Master node, it first executes a pre-upgrade check process. This process includes, but is not limited to, the following verification items: 1) Whether the node's operating system version meets the Master node compatibility requirements (e.g., CentOS 7.6 and above); 2) Whether the node has installed and configured the necessary storage management services (e.g., superx-manage, etcd, ceph-mon, etc.); 3) Whether the node's network connectivity meets the cluster communication standards (e.g., ping latency is less than 50ms, and whether ports 3370, 2379, etc. are open); 4) Whether the node's hardware resources meet the minimum configuration requirements for a Master node (e.g., CPU ≥ 4 cores, memory ≥ 16GB, disk space ≥ 50GB).

[0060] At the parameter level, the system sets a set of pre-upgrade verification thresholds. For example, the service process running status code must be "active", network connectivity must complete the response within 10 seconds, and hardware resource utilization must not exceed 80%. If any verification item is not met, the node is marked as "not upgradable", and the system will automatically select the next eligible Storage node as a candidate.

[0061] In application scenarios, this step is widely applicable to automatic recovery scenarios after node active failure in SDS storage domains. Especially in large-scale distributed storage systems, it can effectively prevent upgrade failures caused by insufficient resources or service loss, thereby ensuring high availability and data consistency of the cluster.

[0062] The technical effect of this step is that, through a strict pre-upgrade verification mechanism, it ensures that only nodes with complete service capabilities and resource configurations can be upgraded to Master nodes, thereby improving the success rate and stability of system upgrades and avoiding cluster splits or metadata inconsistencies caused by inconsistent configurations or service anomalies. It is a key guarantee for achieving strong consistency between "service status, database roles, and configuration information".

[0063] S3: When a failed node comes back online, if it was originally a Master node, a downgrade operation is triggered, changing its role from Master node to Storage node.

[0064] Specifically, when a failed node comes back online, if it was originally a Master node, the system will automatically trigger a degradation operation, changing its role from Master node to Storage node. This step is a key component of the "dynamic role compensation mechanism" in this invention, aiming to maintain a dynamic balance in the number of Master nodes in the SDS storage domain, thereby ensuring high availability and data consistency of the cluster.

[0065] At the technical implementation level, this degradation operation relies on the cloud management platform's real-time monitoring mechanism for node status. When a node actively fails (e.g., the master node shuts down), the system identifies the node's offline status through heartbeat detection and service status reporting (e.g., the status of critical services such as etcd and superx-manage). Once the node comes back online and restores its service status, the cloud management platform will obtain its historical role information through the status synchronization interface. If the node was previously a master node, the system will compare the current number of surviving master nodes in the cluster with a preset threshold. If the current number of master nodes has recovered to above the threshold, the system will automatically trigger the degradation process, switching the node's role from master to storage, and synchronously updating the role information and cluster metadata in the configuration database.

[0066] At the parameter level, this step involves several key parameters, including but not limited to: the total number of surviving nodes and the number of Master nodes. Threshold The frequency of service status checks (e.g., once every 30 seconds) and the role switch timeout (e.g., a maximum wait of 120 seconds to confirm that the node has fully recovered) are important parameters. These settings must comply with the high availability standards of the SDS architecture (e.g., the Raft consensus protocol's requirements for the number of Master nodes) and ensure that the normal operation of the cluster is not affected during the degradation process.

[0067] In application scenarios, this step applies to situations where nodes in an SDS storage domain recover after going offline due to proactive failures (such as planned maintenance or power outages). For example, in a cluster of 5 nodes, if there were originally two Master nodes, and one went offline due to a proactive failure, the system automatically promoted a Storage node to Master. When the original Master node recovers and comes back online, the system will automatically demote it to a Storage node to maintain the number of Master nodes below a threshold and prevent election conflicts or configuration inconsistencies caused by Master node redundancy.

[0068] The technical benefit of this step is that, through an automated degradation mechanism, the system can quickly adjust the role of nodes after recovery, ensuring that the cluster always remains within the optimal range of Master node numbers, thereby improving system stability and fault tolerance. At the same time, this mechanism avoids configuration errors caused by untimely manual intervention or misoperation, reduces operational complexity, and enhances the adaptability of the SDS storage domain in dynamic environments.

[0069] Furthermore, S3 includes: S31. After the failed node comes back online, check whether its service status has returned to normal and confirm whether it still has the service components required by the Master node. Specifically, after a failed node comes back online, checking whether its service status has returned to normal and confirming whether it still has the service components required by the Master node are key steps in this invention to achieve system self-healing and role consistency assurance. The core of this step lies in ensuring that nodes can be correctly reclassified to their proper roles after recovery through service status monitoring mechanisms and role verification logic, thereby maintaining the high availability and data consistency of the SDS storage domain.

[0070] At the technical implementation level, the cloud management platform obtains the real-time running status of nodes through heartbeat detection and service status reporting mechanisms. Specifically, the system detects the running status of key service components on the node (such as etcd, superx-manage, etc.) to determine whether it has recovered to a level where it can normally provide Master services. If it detects that the node has come back online and all service components are running (e.g., etcd service port 12379 is listening normally, superx-manage service process is alive), it further verifies whether it still has the configuration information of the Master node, such as whether it contains cluster metadata and whether it holds an election token (lease token). If the verification passes, the node is considered to have met the conditions for recovery as the Master.

[0071] At the parameter level, the system uses a preset health check timeout (e.g., if no heartbeat is received within 30 seconds, it is considered a fault) and service component status detection standards (e.g., process liveness, port listening, service response time less than 500ms) for judgment. Simultaneously, the system maintains a Master node count threshold to control whether a node needs to be demoted, preventing the number of Master nodes from exceeding the legal threshold, which could lead to cluster decision-making mechanism failure or configuration conflicts.

[0072] In application scenarios, this step is suitable for recovery after a node's active failure (such as planned shutdown, network interruption, etc.) in an SDS storage domain. For example, in a large-scale distributed storage system, when a Master node comes back online for maintenance, the system needs to quickly determine whether it should retain its Master role or be demoted to a Storage node to maintain cluster role balance. This mechanism can effectively avoid metadata inconsistencies or election anomalies caused by residual node roles.

[0073] The technical advantage of this step lies in ensuring that the system can automatically correct role information after a node recovers through dual verification of service status and role configuration, achieving strong consistency between "service status, database roles, and configuration information." Simultaneously, by combining mutual exclusion mechanisms and threshold control, it effectively prevents system instability and configuration conflicts caused by role recovery, thereby improving the stability and automated operation and maintenance capabilities of the entire SDS storage domain.

[0074] S32, if the number of surviving Master nodes exceeds a preset threshold, the node will be downgraded to a Storage node to maintain the balance of cluster roles.

[0075] Specifically, in some implementations, when the number of surviving Master nodes exceeds a preset threshold, the system prioritizes downgrading that node to a Storage node to maintain cluster role balance. This step is technically based on a dynamic node role adjustment mechanism within the SDS (Software-Defined Storage) domain. Its core principle is to intelligently control node roles by real-time monitoring of node status and service processes, combined with a cluster health assessment model, thereby avoiding system instability risks caused by Master node redundancy.

[0076] The specific operation is as follows: The cloud management platform obtains node status information in real time through a heartbeat detection mechanism (such as health checks based on etcd or superx-manage services). When a Master node actively fails (e.g., the master node shuts down or the service stops), the system first determines its role and, if the upgrade conditions are met, randomly selects one from the surviving Storage nodes for upgrade. If the failed Master node comes back online and resumes service, the system will check the number of currently surviving Master nodes again. If the number of surviving nodes is insufficient at this time, a degradation mechanism is triggered, downgrading the node from the Master role to the Storage role to ensure that the number of Master nodes does not exceed the set threshold and maintain the effectiveness of the cluster's Quorum mechanism.

[0077] Among them, threshold Typically, the number of currently surviving nodes is set to half of the total number of nodes to ensure sufficient fault tolerance in the cluster. In implementation, the system employs a mutex lock mechanism to ensure that automatic degradation and manual upgrade / degrade operations do not execute concurrently, thus avoiding configuration file conflicts or metadata inconsistencies. Furthermore, in scenarios where multiple nodes fail simultaneously, this step is serialized using an event queue mechanism to ensure the orderliness and consistency of role adjustments.

[0078] This step is widely applicable to high availability management of large-scale SDS clusters, especially in scenarios such as cloud data centers and distributed storage systems. It can effectively prevent cluster splits or metadata service unavailability caused by an imbalance in the number of master nodes. Through this mechanism, the system can maintain service continuity even after a node fails, improving the stability and fault tolerance of the cluster, while reducing the frequency of intervention by operations and maintenance personnel, and achieving automated and intelligent cluster management.

[0079] S4 uses a mutex lock mechanism to block manual upgrade or downgrade operations during automatically triggered upgrade or downgrade operations to prevent configuration file conflicts.

[0080] Specifically, this step uses a mutex lock mechanism to block manual upgrade / downgrade operations during automatically triggered upgrade or downgrade operations to prevent configuration file conflicts. Its technical implementation is based on concurrency control theory, employing a mutex lock mechanism to ensure that only one operation (automatic or manual) can modify node role configuration information at a time, thereby guaranteeing the consistency and integrity of the system configuration.

[0081] In its implementation, after detecting a node's active failure (such as a Master node shutting down), the cloud management platform first determines whether the automatic upgrade conditions are met (i.e., the number of surviving Master nodes is less than a threshold). If the conditions are met, the system initiates the automatic upgrade process, acquiring a global mutex lock (such as a Redis-based distributed lock or a local file lock) at the start of the process. This lock controls access permissions to the configuration file. During this period, any manual upgrade / downgrade requests (such as operations initiated by administrators via the console or API) will be blocked after detecting the mutex lock's holding status, returning a "operation unavailable" or "please wait for the automatic process to complete" message. Manual operations can only continue after the automatic process has finished executing and released the mutex lock.

[0082] The implementation of mutex locks must meet the following technical specifications: lock acquisition and release must be atomic to prevent deadlocks; lock holding time should be controlled within a reasonable range (e.g., no more than 30 seconds) to avoid prolonged blocking affecting system responsiveness; the granularity of the lock should be at the cluster level for role configuration operations, rather than for individual nodes or services, to ensure global consistency of operations. Furthermore, the system should support an automatic unlocking mechanism based on timeouts to prevent locks from being unable to be released due to abnormal exits.

[0083] This step is applicable in practical applications to scenarios where the number of Master nodes in an SDS storage domain is dynamically adjusted, especially in large-scale distributed clusters where nodes frequently go online and offline or where administrators need to intervene temporarily. The mutex lock mechanism can effectively prevent configuration conflicts and ensure the integrity of cluster metadata. Through this mechanism, the system establishes a clear priority and isolation strategy between automatic and manual operations, improving the reliability of fault recovery and the controllability of operation and maintenance.

[0084] Furthermore, S4 includes: S41 uses a distributed locking mechanism to lock the upgrade / downgrade configuration file in the cloud management platform, preventing multiple operations from modifying the same configuration simultaneously. Specifically, in this invention, a distributed lock mechanism is used to lock the upgrade / downgrade configuration file in the cloud management platform. The core purpose is to ensure read / write consistency of the configuration file in multi-node concurrent operation scenarios, preventing metadata conflicts or inconsistent system states caused by multiple operations simultaneously modifying the same configuration. This step plays a crucial coordinating and controlling role in the proactive fault handling process of nodes in the SDS (Software-Defined Storage) storage domain.

[0085] At the technical implementation level, distributed lock mechanisms are typically based on distributed coordination services (such as Apache ZooKeeper, etcd, or Redis's RedLock algorithm). When the cloud management platform detects a node's active failure (such as the Master node shutting down) and prepares to execute an automatic upgrade / degrade operation, it first initiates a lock request to the distributed lock service to obtain exclusive access to the configuration file. The granularity of the lock can be set to target a specific configuration file (such as node_role_config.json) or a specific field in the configuration file (such as master_list) to achieve finer-grained concurrency control. The lock's leasetime is typically set to 30 seconds to 5 minutes, depending on the system response latency and the expected time window for fault recovery.

[0086] Regarding parameters, the lock acquire timeout should ideally be set to no more than 3 seconds to avoid system response delays due to prolonged waiting. The lock retry count can be set to 3-5 times, with intervals of 500ms-1s, to handle brief network fluctuations or service unavailability. Furthermore, the system must support an automatic lock renewal mechanism to ensure that operations are not interrupted by lock expiration during complex upgrade / downgrade processes.

[0087] In application scenarios, this step is suitable for situations where the number of Master nodes in an SDS storage domain is dynamically adjusted, especially in large-scale distributed clusters where multiple nodes may experience active failures or recoveries simultaneously, leading to concurrent access to the configuration file by multiple threads / processes. Through a distributed lock mechanism, the cloud management platform can ensure that at any given time, only one operation (automatic or manual) modifies the configuration file, thereby guaranteeing the integrity and consistency of cluster metadata.

[0088] The technical advantage of this step lies in effectively avoiding configuration conflicts caused by concurrent operations, thereby improving the stability and reliability of the system in high-concurrency fault handling scenarios. Simultaneously, by introducing a locking mechanism, the system possesses excellent scalability and fault tolerance, providing a solid foundation for subsequent automated operation and maintenance and intelligent decision-making.

[0089] S42, when a manual operation request occurs, if an automatic operation is detected to be in progress, the manual operation request is added to the waiting queue and executed after the automatic operation is completed.

[0090] Specifically, the step in this invention, "when a manual operation request occurs, if an automatic operation is detected to be in progress, the manual operation request is added to a waiting queue and executed only after the automatic operation is completed," is a key control mechanism for achieving system configuration consistency and avoiding operation conflicts. Its technical implementation principle is based on the coordinated control of a mutex lock and a task scheduling queue, ensuring that manual operations do not interfere with the ongoing automatic process during the automatic upgrade / downgrade process, thereby avoiding metadata inconsistencies or configuration conflicts caused by concurrent operations.

[0091] In its implementation, when the cloud management platform receives a manual upgrade / downgrade request, it first checks the status detection module to see if any automatic upgrade / downgrade tasks are already running in the system. This detection is typically based on task status flags (such as the `auto_upgrade_in_progress` boolean variable) or task queue locking mechanisms (such as Redis distributed locks or ZooKeeper node locks). If automatic operation is detected, the system encapsulates the manual request as a task object and adds it to a priority queue or a FIFO (First In First Out) queue, waiting for the automatic operation to complete before processing. A maximum waiting time (e.g., 300 seconds) and timeout handling policies can be set in the queue to prevent requests from waiting indefinitely.

[0092] Regarding parameter settings, the system needs to define the timeout period for the mutex lock (e.g., lock_timeout=60s) to avoid deadlocks; at the same time, the manual request queue should have a capacity limit (e.g., max_queue_size=100) to prevent system resource exhaustion. In addition, the system also needs to configure a callback mechanism after the automatic operation is completed to trigger the execution of manual requests in the queue.

[0093] This step is particularly useful in large-scale SDS clusters where administrators may initiate manual configuration changes during the automatic recovery process. For example, if an administrator attempts to manually downgrade a node while the system is automatically upgrading after a Master node failure, the request will be blocked and queued until the automatic process is complete, thus ensuring the atomicity and consistency of the cluster configuration.

[0094] In terms of technical effectiveness, this step effectively avoids configuration conflicts caused by concurrent execution of automatic and manual operations, improving the stability and reliability of the system in high-concurrency scenarios. By introducing a queuing mechanism, the system not only ensures the order of operations but also improves the controllability and predictability of operation responses, providing a solid guarantee for the automated operation and maintenance of SDS storage domains.

[0095] In summary, this step, by introducing a mutex lock mechanism, achieves mutual exclusion control between automatic and manual upgrade / downgrade operations, effectively avoiding conflicts caused by concurrent modification of configuration files. This is a key technical means for ensuring stable system operation in high-concurrency and high-availability scenarios.

[0096] S5: When multiple nodes are detected to be actively failing at the same time, a processing queue is generated based on the timestamp of the failure event, and role compensation operations are executed sequentially according to the queue order.

[0097] Specifically, when multiple nodes are detected to be actively failing simultaneously, the system generates a processing queue based on the timestamps of the failure events and executes role compensation operations sequentially according to the queue order. This step is the core implementation method of the innovative point of "queuing-based processing of multi-node failures" in this invention, aiming to solve the problems of processing chaos, configuration conflicts and system instability that may occur when high-concurrency nodes actively fail in large-scale SDS (Software-Defined Storage) storage domains.

[0098] At the technical implementation level, the system collects real-time node runtime status information through underlying service status monitoring modules (such as etcd and superx-manage services), and records node-initiated failure events (such as master shutdown) as timestamped event logs. When multiple nodes experience simultaneous initiated failures, the system sorts the failed nodes according to the timestamps of the events, constructing a priority queue to ensure the orderly and traceable handling of failures. After the queue is constructed, the system executes role compensation operations sequentially according to the queue order, such as promoting a Storage node to a Master node, or demoting a failed node to a Storage node after recovery, to maintain a balance in the number of Master nodes.

[0099] At the parameter level, the system sets the timestamp precision to milliseconds (ms) to ensure the accuracy of event sorting. Meanwhile, the maximum concurrency of queue processing can be configured from 1 to N, typically set to 1 in large-scale clusters to avoid configuration conflicts caused by multi-threaded operations. The execution interval for role compensation operations can be set from 5 to 30 seconds to ensure the system completes state synchronization and resource assessment before processing.

[0100] At the application scenario level, this step is applicable to the high availability management of SDS storage domains in cloud management platforms. Especially when nodes fail in batches (such as planned maintenance, hardware replacement, etc.), it can ensure that the system maintains configuration consistency during the fault recovery process and avoid metadata inconsistency or cluster splitting caused by parallel processing.

[0101] The technical advantage of this step lies in achieving serialized processing in multi-node proactive failure scenarios through a timestamp-driven queue mechanism, thereby ensuring the orderly recovery and stability of the system under high-concurrency failure conditions. Simultaneously, this mechanism, combined with "operation mutual exclusion and conflict avoidance design," further enhances the reliability and consistency of configuration changes, serving as a key technical support for the efficient, intelligent, and stable node management of this invention.

[0102] The method for handling node active failure in the SDS storage domain of this invention realizes automated role adjustment after node active failure, ensuring that the storage domain configuration is consistent with the actual service status, and improving system stability and operation and maintenance efficiency.

[0103] To implement the above embodiments, the present invention also proposes a processing device for node active failure in an SDS storage domain. Figure 2 This is a schematic diagram of a processing device for active node failure in an SDS storage domain, provided as an embodiment of the present invention. Figure 2 As shown, the device includes: The fault detection and role determination module is used to detect active faults in nodes and determine the role of the faulty nodes. The Master node upgrade module is used to randomly select a node from the surviving Storage nodes and upgrade it to a Master node when the failed node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes. The Master node degradation module is used to trigger a degradation operation when a failed node comes back online. If it was originally a Master node, the module will change its role from Master node to Storage node. The mutex control module is used to block manual upgrade or downgrade operations during automatically triggered upgrade or downgrade operations via a mutex mechanism to prevent configuration file conflicts.

[0104] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0105] To implement the above embodiments, the present invention also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0106] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0107] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0108] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0110] This invention is intended to provide implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0111] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0117] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0119] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0120] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for handling node active failure in an SDS storage domain, characterized in that, include: S1, detect node active failure and determine the role of the faulty node; S2, when the faulty node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes, randomly select a node from the surviving Storage nodes and promote it to a Master node. S3, when a faulty node comes back online, if it was originally a Master node, a downgrade operation is triggered, and its role is changed from Master node to Storage node; S4 uses a mutex lock mechanism to block manual upgrade or downgrade operations during automatically triggered upgrade or downgrade operations to prevent configuration file conflicts.

2. The method as described in claim 1, characterized in that, The role of detecting and identifying faulty nodes also includes: S11 uses the underlying service status monitoring module to detect the service status of nodes in real time, including whether the etcd service and superx-manage service are shut down; S12, determine whether the node is actively failing based on changes in service status. If the service status is abnormal and no hardware failure is detected, it is determined to be an actively failing node.

3. The method as described in claim 1, characterized in that, The step of randomly selecting a node from the surviving Storage nodes and promoting it to a Master node also includes: S21, a weighted random selection is performed based on the load status and storage capacity of the surviving Storage nodes, prioritizing nodes with lower load and sufficient storage capacity; S22. Before the upgrade operation, verify whether the selected Storage node has the basic service components and configuration conditions required for the upgrade.

4. The method as described in claim 1, characterized in that, The triggering of the downgrade operation, which changes its role from Master node to Storage node, also includes: S31. When the faulty node comes back online, check whether its service status has returned to normal and confirm whether it still has the service components required by the Master node. S32, if the number of surviving Master nodes exceeds a preset threshold, the node will be downgraded to a Storage node to maintain the balance of cluster roles.

5. The method as described in claim 1, characterized in that, The method of blocking manual upgrade or downgrade operations during automatically triggered upgrade or downgrade operations via a mutex lock mechanism also includes: S41 uses a distributed lock mechanism to lock the upgrade / downgrade configuration file in the cloud management platform to prevent multiple operations from modifying the same configuration at the same time. S42, when a manual operation request occurs, if an automatic operation is detected to be in progress, the manual operation request is added to the waiting queue and executed after the automatic operation is completed.

6. The method as described in claim 1, characterized in that, Also includes: S5: When multiple nodes are detected to be actively failing at the same time, a processing queue is generated based on the timestamp of the failure event, and role compensation operations are executed sequentially according to the queue order.

7. A processing device for node active failure in an SDS storage domain, characterized in that, include: The fault detection and role determination module is used to detect active faults in nodes and determine the role of the faulty nodes. The Master node upgrade module is used to randomly select a node from the surviving Storage nodes and upgrade it to a Master node when the failed node is a Master node and the number of surviving Master nodes is less than half of the total number of surviving nodes. The Master node degradation module is used to trigger a degradation operation when a failed node comes back online. If it was originally a Master node, the module will change its role from Master node to Storage node. The mutex control module is used to block manual upgrade or downgrade operations during automatically triggered upgrade or downgrade operations via a mutex mechanism to prevent configuration file conflicts.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images