Fault-tolerant and recovery method, device and electronic equipment of distributed cluster

Abstract

The application relates to a fault-tolerant and recovery method, device and electronic equipment of a distributed cluster. The method comprises the following steps: a central controller constructs a protection graph and obtains recovery priority information of each backup node in different fault scenarios in the protection graph based on cluster information, wherein the protection graph represents the protection relationship between the backup nodes and functions in a topological form, and the cluster information comprises the protection capability of the backup nodes and the resource status of the primary node corresponding to the functions; the central controller sends the recovery priority information of each backup node and adjacent backup node information to each backup node as preprocessing information, wherein the adjacent backup node is used for indicating the backup nodes protecting at least one same function; and after the backup node monitors function failure information, the function recovery is performed according to the preprocessing information. The fault-tolerant capability and the fault recovery efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of disaster recovery technology, and in particular to a method, apparatus and electronic device for fault tolerance and recovery of a distributed cluster. Background Technology

[0002] In existing centralized control fault recovery strategies, the system heavily relies on a central controller to coordinate and allocate resources, leading to several significant technical drawbacks. First, the presence of the central controller exposes the system to a single point of failure; if the central controller fails, the entire system recovery process will be paralyzed, severely impacting system stability. Second, in large-scale distributed systems, the central controller needs to handle status information and recovery requests from numerous backup nodes, which can easily lead to performance bottlenecks, especially under high load conditions, where the response speed of the recovery process will significantly decrease. Furthermore, as the system scales, the burden on the central controller becomes increasingly heavy, limiting system scalability and making it difficult to effectively cope with ultra-large-scale distributed environments. Summary of the Invention

[0003] This application provides a method, apparatus, and electronic device for fault tolerance and recovery of a distributed cluster.

[0004] In a first aspect, embodiments of this application provide a fault tolerance and recovery method for a distributed cluster, the distributed cluster including a central controller and multiple shared backup nodes, the method comprising: The central controller constructs a protection graph based on cluster information and obtains the recovery priority information of each backup node in the protection graph under different fault scenarios. The protection graph represents the protection relationship between the backup nodes and functions in a topological form. The cluster information includes the protection capabilities of each backup node and the resource status of the primary node corresponding to the function. The central controller sends the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessed information to each backup node, wherein the adjacent backup nodes are used to indicate the protection of at least one backup node with the same function. After the backup node detects a functional failure, it restores the function based on the preprocessed information.

[0005] Secondly, embodiments of this application provide a fault tolerance and recovery device for a distributed cluster, the distributed cluster including a central controller and multiple backup nodes, the device comprising: The information generation module is used by the central controller to construct a protection graph based on cluster information and obtain the recovery priority information of each backup node in the protection graph under different fault scenarios. The protection graph represents the protection relationship between the backup nodes and functions in a topological form. The cluster information includes the protection capabilities of each backup node and the resource status of the primary node corresponding to the function. The information sending module is used by the central controller to send the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessed information to each backup node. The adjacent backup nodes are used to indicate the protection of at least one backup node with the same function. The fault recovery module is used to restore functionality based on the preprocessed information after the backup node detects a functional fault.

[0006] Thirdly, embodiments of this application provide an electronic device, including: One or more processors; The processor is used to invoke instructions to cause the electronic device to perform the method described in the first aspect above.

[0007] Fourthly, embodiments of this application provide a storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect above.

[0008] Fifthly, embodiments of this application provide a program product, including at least one of a program and instructions, wherein when the program and instructions are executed by an electronic device, they implement the steps of the method described in the first aspect.

[0009] According to the technical solution of this application, in the preprocessing stage, the central controller generates a backup resource allocation strategy and a recovery priority for each backup node. Based on the recovery priority information, the backup node can independently recover from the fault or work with adjacent backup nodes to recover from the fault. This realizes a local recovery mechanism based on preprocessing information, improves the fault tolerance of the cluster under limited resources, avoids the limitations of centralized control, enables rapid initiation of recovery operations when a fault occurs, effectively reduces response time, achieves rapid response after a fault, and improves resource utilization efficiency.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] Figure 1 This is a flowchart illustrating a fault tolerance and recovery method for a distributed cluster according to an exemplary embodiment.

[0013] Figure 2 This is a flowchart illustrating a fault tolerance and recovery method for a distributed cluster according to another exemplary embodiment.

[0014] Figure 3 This is a block diagram illustrating a fault-tolerant and recovery apparatus for a distributed cluster according to an exemplary embodiment.

[0015] Figure 4 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0016] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0017] The embodiments described in this application are not exhaustive, but merely illustrative of some embodiments, and are not intended to limit the scope of protection of this application. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined with each other. For example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0018] In each embodiment of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0019] It should be noted that the acquisition, transmission, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0020] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0021] It is worth noting that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0022] In telecommunications network service systems, distributed clusters typically consist of a large number of computing nodes, ranging in size from thousands to tens of thousands. Each node is responsible for processing data, regulating traffic, and ensuring security. When a node fails and cannot be recovered quickly, it can negatively impact the network experience of millions of users. Therefore, how to quickly restore failed functions and ensure uninterrupted service is a critical issue that needs to be addressed. Efficient resource utilization and ensuring high system availability are always the core objectives of distributed cluster design.

[0023] Commonly known technologies encompass the core methods of fault-tolerant design and resource scheduling in distributed computing systems. These technologies typically enhance system reliability and high availability through fault-tolerant recovery mechanisms. In distributed cluster scenarios, existing methods largely rely on strategies such as redundant backups, containerized deployment, and cross-platform data recovery to improve resource utilization efficiency and recovery speed in the event of a failure. For example, containerization technology enables dynamic migration and rapid recovery of tasks, while redundant backup strategies maintain service continuity by replicating data and tasks across different nodes.

[0024] In related technologies, centralized control fault-tolerant recovery schemes rely on a central controller to monitor and coordinate resources to complete recovery operations. This method is suitable for making optimal recovery decisions from a global perspective, but in large-scale distributed systems, the central controller's efficiency is reduced due to performance bottlenecks. Firstly, centralized recovery relies on a central controller, posing a single point of failure risk. Once the central controller fails, the recovery capability of the entire system will be severely affected. Secondly, centralized schemes require uploading fault information to the central controller for global coordination, typically leading to high latency. Thirdly, centralized control requires frequent uploading of backup node status and distributing recovery commands, which burdens network traffic. Traditional centralized recovery, relying on a central controller, exhibits significant limitations in addressing high communication latency and single point of failure issues, especially in large-scale clusters where performance breakthroughs are difficult to achieve through simple optimization. Fault-tolerant methods based on redundant backups provide backups for critical resources, but their resource utilization is low, lacking flexibility, and the system needs to maintain multiple backup nodes in a standby state, increasing costs. Distributed task rescheduling strategies are mainly used for task reallocation after a failure, but their recovery time is long, and resource allocation often exhibits unevenness.

[0025] Based on this, embodiments of this application provide a method, apparatus, and electronic device for fault tolerance and recovery of distributed clusters, so as to improve the speed of fault recovery and resource utilization efficiency.

[0026] The following description, with reference to the accompanying drawings, describes a method, apparatus, and electronic device for fault tolerance and recovery of a distributed cluster according to embodiments of this application.

[0027] It should be noted that the execution subject of the fault tolerance and recovery method of the distributed cluster in the embodiments of this application can be a fault tolerance and recovery device of the distributed cluster. The device can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0028] Figure 1 This is a flowchart illustrating a fault tolerance and recovery method for a distributed cluster according to an exemplary embodiment. For example... Figure 1 As shown, the fault tolerance and recovery method of this distributed cluster may include, but is not limited to, the following steps 101-103.

[0029] In step 101, the central controller constructs a protection graph based on the cluster information and obtains the recovery priority information of each backup node in the protection graph under different fault scenarios.

[0030] The protection graph represents the protection relationship between backup nodes and functions in a topological form. The cluster information includes the protection capabilities of backup nodes and the resource status of the primary node corresponding to the function.

[0031] In this embodiment, the distributed cluster allows multiple functions to share backup resources, improving resource utilization and reducing system operating costs. By centrally managing and dynamically allocating shared backup resources, a balance between system efficiency and flexibility is achieved. By adopting a shared protection strategy, resource waste is significantly reduced while ensuring system reliability.

[0032] In some embodiments, the protection graph represents the protection relationship between backup nodes and functions in a bidirectional topology form, with backup nodes and functions (such as virtual network functions) as vertices and protection relationships as connecting edges between vertices.

[0033] This can be understood as follows: a distributed cluster includes a central controller and multiple shared backup nodes. The protection graph is used to indicate backup resource allocation information. Backup resource allocation can be performed according to the protection graph. The edges in the protection graph represent the allocation relationship of backup resources. The backup nodes in the protection graph are dedicated backup servers that can monitor the running status of the primary server for their protection functions. For example, if backup node 1 protects 3 functions, it can detect the survival status of the corresponding functions on the primary node. If backup node 1 detects that function 1 it protects needs to be restored, it will start the function on this backup node and complete the function restoration.

[0034] In some embodiments, the weights of the edges are used to further represent resource occupancy or communication latency between backup nodes, thereby providing basic support for the dynamic optimization of the protection graph structure.

[0035] In some embodiments, cluster information may include, but is not limited to: the functions running on each primary node (such as firewalls, load balancers, cache servers, etc.), the resource configuration parameters of the primary node (such as the number of CPU cores, memory capacity, network bandwidth and storage space), and the protection capabilities of the shared backup nodes.

[0036] In some embodiments, a protection graph is created by analyzing backup requirements and functional correspondences based on cluster information.

[0037] For example, during the system initialization phase, the central controller reads cluster information from configuration files or databases to obtain information such as the resource distribution of backup nodes and the functional deployment of primary nodes, and generates a protection diagram.

[0038] In some embodiments, after the protection map is generated, i.e. after the backup resource allocation policy is obtained, the central controller allocates backup resources according to the obtained backup resource allocation policy.

[0039] In some embodiments, backup resources are divided into two modes—hot backup and cold backup—based on the importance of the functions. Hot backup is used for critical functions, such as core routers and firewalls, whose failure would significantly impact the overall system operation. To ensure service continuity, hot backup operates synchronously with the primary function, and the central controller ensures that backup resources are always ready to take over through periodic status synchronization. Cold backup, on the other hand, is for non-critical functions, such as cache servers or general application services. It typically remains inactive and is only activated when a failure of the primary function is detected, thus significantly reducing the idle rate of backup resources.

[0040] It should be noted that the recovery priority information of each backup node under different failure scenarios refers to the recovery priority of the corresponding function of each backup node.

[0041] In some embodiments, during the system initialization phase, the central controller comprehensively analyzes information such as the protection capabilities (i.e., capacity), resource status (including backup resource allocation information), and network topology latency of each backup node to generate the recovery priority of each backup node in the protection diagram under different fault scenarios, ensuring that it can reflect the actual recovery needs and constraints; the recovery priority here is used to indicate the recovery order of each function.

[0042] In some embodiments, the central controller receives fault-related information and calculates the optimal recovery priority for each backup node. ,in, f It's a function. n It is a backup node. k This represents the number of possible failures. During the calculation, the central controller considers the neighbor relationships between backup nodes and the priority of protection functions to ensure that repeated or omitted recovery steps can be effectively avoided in the event of a multi-functional failure. The specific steps are as follows: For each backup node... and each function The central controller calculates the optimal recovery priority based on the number of faults k.

[0043] In some embodiments, when k When the value is 1, recovery priority information is directly generated based on the relevant information of each backup node; when... k When the value is greater than 1, relevant information of adjacent backup nodes is obtained to generate recovery priority information.

[0044] In some embodiments, the above-mentioned construction of a protection graph based on cluster information and acquisition of recovery priority information of each backup node in the protection graph under different fault scenarios includes: obtaining backup resource allocation information and recovery priority information based on cluster information and combined with a bipartite graph-based fault tolerance optimization model; wherein, the bipartite graph-based fault tolerance optimization model aims to maximize the fault tolerance capability for potential future faults, and uses the protection capability and function of each backup node and the resource status of the primary node corresponding to it as constraints.

[0045] For example, to achieve efficient fault-tolerant recovery, a bipartite graph-based fault-tolerant optimization model is used to model and solve the set of backup nodes and the set of functional protections to ensure rapid recovery of critical functions in the event of a fault. Assume... It is a collection of backup nodes that can be used for backup functions. Indicates by The set of protection functions, and the allocation of backup resources (i.e., backup paths) are determined by a given binary value. It means that, among them ;if =1, then the function f By backup node n Protect; otherwise, set to 0; let This represents a set of features that are unavailable. Represents the set of available features. Backup node The current available capacity for recovery is This indicates that this is a non-negative integer representing the number of functionalities that the backup node can restore; the available capacity of the backup node. The number of backup nodes may be less than the number of protected functions, meaning that in cases of insufficient capacity, not all protected functions can be restored. Failures during the recovery process should be strictly identified and avoided; that is, multiple backup nodes should not restore the same function, and each failed function must be restored by a specific backup node. Binary decision variables should be used. Indicates restoration, among which, If the backup node n Selected for recovery function f ,but =1; otherwise 0; the goal is to maximize the fault tolerance for potential future failures, i.e., the number of functions protected by remaining capacity after recovery; a binary auxiliary variable is introduced. , used to calculate the target value; if =1 indicates a backup node n Function can be restored f Otherwise, the value is 0. The objective function and constraints of this optimization problem are as follows: Objective function:

[0046] Constraints:

[0047] This embodiment solves the problem of modeling the protection relationship between functions and backup nodes by introducing the above-mentioned bipartite graph-based fault-tolerant optimization model. It can provide the best recovery strategy under resource-constrained conditions, while dynamically adjusting the recovery path to optimize the overall fault-tolerant performance and improve the fault tolerance capability of the cluster. By using mathematical modeling to accurately obtain the mapping relationship between backup nodes and functions, it not only enhances the system's recovery capability in scenarios with multiple backup node failures, but also provides strong theoretical support for global optimization.

[0048] By modeling and solving the optimization problem described above, the system can provide the best recovery strategy under resource-constrained conditions. Mathematical modeling is used to accurately describe the mapping relationship between backup nodes and functions, which not only enhances the system's recovery capability in scenarios with multiple backup node failures, but also provides strong theoretical support for global optimization. It can achieve efficient local fault recovery in a shared protection environment, ensuring that the system can respond quickly and restore functions when a fault occurs, minimizing service interruption time and improving the system's high availability and reliability.

[0049] This can be understood as follows: when multiple functions fail simultaneously and shared protection resources are insufficient to support all recovery needs, the system may face resource contention, recovery delays, or even the inability to recover some functions. To address this challenge, this solution proposes a priority-based recovery strategy; during the initialization phase, the central controller combines the structural information of the protection graph to assign priorities to each function, and generates a backup resource allocation strategy during the preprocessing phase.

[0050] In some embodiments, the calculation of recovery priority may also take into account factors such as the importance of the function, the resource load of the backup node, and the weight of the backup path, so as to ensure that in the case of resource shortage, the key functions that have the greatest impact on the system are restored first. This strategy not only improves the efficiency of resource scheduling, but also significantly reduces service interruption time in complex scenarios.

[0051] In step 102, the central controller sends the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessing information to each backup node.

[0052] The adjacent backup node is used to indicate the protection of at least one backup node with the same function.

[0053] In some embodiments, the preprocessing information may include, but is not limited to, the neighbor structure of the backup node and the priority order of each function. The preprocessing information is used to guide the backup node to make the optimal recovery decision quickly when a failure occurs, thereby reducing recovery time and communication overhead.

[0054] In some embodiments, the generated recovery priority information is stored together with the neighbor backup node association information in the corresponding backup node and distributed to all relevant backup nodes through the central controller.

[0055] In one example, the preprocessing information includes the neighboring backup nodes of each backup node, and related functions. f Relevant recovery priorities The backup resource allocation information is backed up, and pre-processed information is sent to each backup node for use in the local recovery phase. The pre-processed information ensures that when a failure occurs, the backup node can respond quickly and perform recovery based on the pre-calculated recovery priority.

[0056] In some embodiments, the method of this embodiment further includes: the central controller acquiring the operating data of each backup node in real time; updating the cluster information based on the operating data of the backup nodes; and obtaining updated backup resource allocation information and recovery priority information based on the updated cluster information.

[0057] In some embodiments, updated backup resource allocation information and recovery priority information are obtained based on the updated cluster information and combined with a bipartite graph-based fault tolerance optimization model.

[0058] This can be understood as follows: to adapt to dynamic changes in a distributed cluster system, a dynamic resource scheduling algorithm is introduced to achieve dynamic scheduling. This can be achieved by continuously adjusting the allocation strategy of backup resources by collecting real-time operating data of the primary node, including current load, remaining resources, link status, and network traffic. The central controller combines the structural information of the protection graph with real-time monitoring data to dynamically update the protection relationships between backup nodes. For example, when the load of a node suddenly increases, the dynamic scheduling algorithm will prioritize adjusting its backup resource allocation to avoid system bottlenecks. The dynamic resource scheduling algorithm can quickly recalculate backup paths to ensure efficient transmission of backup resources and rapid recovery of functions. This dynamic adjustment capability is crucial for large-scale cluster systems and can significantly improve the robustness and adaptability of the system.

[0059] This embodiment, by collecting system data in real time and combining it with a dynamic resource scheduling algorithm, enables the central controller to continuously adjust recovery priority information and resource allocation strategies during operation. For example, when the load on a node suddenly increases, the system can quickly reassess the recovery priority information and prioritize the allocation of protection resources to the functions on that node to ensure the stable operation of the system. This significantly enhances the system's dynamic adaptability, allowing it to flexibly respond to changes in the status of the primary node and network topology, achieving efficient recovery in complex multi-node functional failure scenarios and providing reliable support for the high availability and stability of distributed clusters.

[0060] In step 103, after the backup node detects the fault information, it performs function recovery based on the preprocessed information.

[0061] In some embodiments, during the operational phase, each backup node monitors the operational status of the function in real time, quickly detects anomalies based on this status, obtains fault information, and triggers the fault recovery process. In other words, when a backup node detects a fault, the fault recovery process is immediately initiated.

[0062] In some embodiments, a fault detection module is deployed on each backup node to obtain fault information.

[0063] In some embodiments, the fault detection module combines multiple monitoring methods, such as link status analysis, functional heartbeat detection, and system log monitoring, to immediately generate a fault report and send a signal to the central controller when a functional failure is detected. The central controller issues recovery instructions based on the current protection map and recovery priority information, and simultaneously distributes the task to relevant backup nodes to quickly complete fault recovery through collaborative operation.

[0064] In one example, during the runtime phase, fault detection modules deployed on each backup node monitor the functionality and resource status of the primary node in real time. Monitoring parameters include CPU utilization, memory usage, network transmission latency, link status, and packet loss rate. Data is collected using tools (such as Prometheus or Nagios) and aggregated through a central monitoring server, enabling global control over the health status of the primary node. To ensure rapid detection of functional failures, a heartbeat mechanism is used to periodically check the liveness of functions on the primary node. If no heartbeat signal is received within a specified time, the central controller determines that the node has failed and immediately triggers the recovery process. Simultaneously, by analyzing the node's runtime logs using log analysis tools (such as ELK Stack), the system can quickly locate abnormal patterns or error messages to further optimize fault prevention and recovery strategies.

[0065] In some embodiments, when a fault is detected based on the functional operation data, the backup node determines, according to preprocessed information, whether it needs to wait for unified scheduling by the central controller, or whether it needs to communicate with neighboring backup nodes to exchange fault information and recovery priority information, or execute a local recovery strategy. The backup node performs corresponding operations according to the obtained recovery strategy, prioritizing the recovery of high-priority functions to ensure the continuity of services corresponding to critical functions. After recovery, it feeds back to the central controller to update the backup resource allocation strategy and priority information.

[0066] In some embodiments, when it is determined that there is no need to wait for unified scheduling by the central controller and there is no need to communicate with neighboring backup nodes, i.e. in a single-function failure scenario, the backup node that detects the failure information can directly restart the failed function from the backup resources or switch to the backup path, avoiding reliance on further instructions from the central controller, thereby reducing the delay caused by relying on global scheduling.

[0067] In some embodiments, in multi-functional failure scenarios, i.e., when communication with neighboring backup nodes is required, backup nodes exchange failure information and recovery requirements through communication, and collaboratively adjust recovery priorities. During communication, each backup node dynamically assesses its current load, resource usage, and neighbor status to ensure that critical functions are restored first and to avoid resource contention or duplicate recovery issues. In complex failure scenarios involving multiple backup nodes, backup nodes can further optimize backup resource allocation strategies through communication, ensuring efficient utilization and rapid response of backup resources.

[0068] In some embodiments, during the recovery operation, the backup node can restart the affected functions or migrate the function tasks to the backup node according to the recovery priority.

[0069] To ensure the correctness of the recovery operation, in some embodiments, after fault recovery based on preprocessed information, the process includes: the backup node verifying the function corresponding to the fault information; if the verification is successful, sending recovery result information to the central processing unit; the central processing unit updating the cluster information based on the recovery result information; and based on the updated cluster information, combined with a bipartite graph-based fault tolerance optimization model, obtaining updated backup resource allocation information and recovery priority information.

[0070] In some embodiments, after recovery is completed, the system analyzes the cause of the fault, updates the protection graph and recovery priority information, thereby further optimizing subsequent fault prevention and recovery strategies and continuously improving system reliability and resource utilization efficiency.

[0071] For example, after performing the recovery operation, the backup node verifies the recovery result, including testing the performance indicators after the fault recovery, the normal operation of the functions, and the compatibility with other nodes. After the verification is completed, the fault recovery result is recorded and synchronized to the central controller to update the protection graph and recovery priority information, providing optimization support for subsequent possible faults.

[0072] Through the above embodiments, in the preprocessing stage, the central controller generates a resource backup strategy for shared backup resources and recovery priority information for each backup node. Based on the recovery priority information, backup nodes can independently recover from faults or collaboratively recover from faults with adjacent backup nodes, realizing a local recovery mechanism based on preprocessed information. This avoids the limitations of centralized control, enables rapid initiation of recovery operations when a fault occurs, effectively reduces response time, and achieves rapid response after a fault. In a distributed cluster, this significantly reduces service interruption time and improves fault recovery efficiency. This solution introduces a fault-tolerant optimization model based on a bipartite graph to solve the problem of modeling the protection relationship between functions and backup nodes. It can provide the best recovery strategy under resource-constrained conditions and improve the fault tolerance capability of the cluster under resource-constrained conditions. By using mathematical modeling to accurately obtain the mapping relationship between backup nodes and functions, it not only enhances the system's recovery capability under multi-functional fault scenarios but also provides strong theoretical support for global optimization. By completing most of the computational tasks in advance during the preprocessing stage, each backup node can quickly perform recovery when a fault occurs, improving the availability and reliability of the system. Through collaborative recovery with multiple backup nodes, backup nodes can optimize recovery paths via communication in complex fault scenarios. When resources are limited or recovery priorities conflict, backup nodes can exchange information and collaboratively select the optimal recovery strategy. This achieves a globally optimal recovery strategy in a shared protection environment, effectively avoiding resource contention and redundant operations, reducing resource overhead, and enabling more efficient resource utilization, thus improving system stability and recovery efficiency. This solution enables efficient local fault recovery in a shared protection environment, ensuring the system can respond quickly and restore functionality when a fault occurs, minimizing service interruption time and improving system high availability and reliability. It achieves efficient utilization of shared protection resources and optimizes resource allocation through dynamic resource scheduling and real-time adjustment, improving recovery efficiency and balance, enhancing the system's recovery capabilities in complex dynamic environments, providing strong support for the stable operation of highly available distributed clusters, improving system recovery capabilities and resource utilization efficiency in complex fault scenarios, and reducing operating costs.

[0073] To further clarify how step 104 above performs fault recovery based on preprocessed information, based on any one or more combined embodiments of the above examples, such as... Figure 2 As shown, the fault tolerance and recovery method of this distributed cluster may include, but is not limited to, the following steps 201-206.

[0074] In step 201, the central controller constructs a protection graph based on the cluster information and obtains the recovery priority information of each backup node in the protection graph under different fault scenarios. The protection graph represents the protection relationship between backup nodes and functions in a topological form, and the cluster information includes the protection capabilities of backup nodes and the resource status of the primary node corresponding to the function.

[0075] It should be noted that the specific implementation method of step 201 can be found in step 101 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0076] In step 202, the central controller sends the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessed information to each backup node, wherein the adjacent backup nodes are used to indicate the protection of at least one backup node with the same function.

[0077] It should be noted that the specific implementation method of step 202 can be found in step 102 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0078] In step 203, based on the preprocessed information, it is determined whether it is necessary to wait for unified scheduling by the central controller.

[0079] In some embodiments, whether a backup node needs to wait for unified scheduling by the central controller depends on the scope of the fault and global resource management requirements.

[0080] In some embodiments, based on preprocessed information, it is determined whether the fault affects critical functions that rely on global coordination. If so, it is necessary to wait for unified scheduling by the central controller. Alternatively, based on preprocessed information, the scope of the fault's impact is determined. If the scope of the fault's impact is wide and involves multiple backup nodes, it is also necessary to wait for unified scheduling by the central controller. The central controller has a global view of the entire cluster and can comprehensively consider the current resource distribution, the scope of the fault's impact, and priority information. Through unified scheduling, it can achieve optimized resource allocation across backup nodes. For scenarios that require coordination of multiple backup nodes, the central controller can adjust the recovery strategy based on global status information to ensure the stability and efficiency of the overall system recovery. Furthermore, when communication between backup nodes is insufficient to resolve complex faults, or when bottlenecks occur due to limited communication links, the global scheduling of the central controller can compensate for the lack of local information, avoiding suboptimal recovery decisions made by backup nodes based solely on local views, thereby ensuring the overall performance and reliability of the system.

[0081] In step 204, if there is no need to wait for unified scheduling by the central controller, based on the preprocessed information and the number of faults, it is determined whether this backup node needs to communicate with adjacent backup nodes.

[0082] In some embodiments, the method for determining whether a backup node needs to communicate with neighboring backup nodes based on preprocessed information and the number of faults includes: determining whether the number of faults exceeds a fault threshold; if the number of faults exceeds the fault threshold, determining that the backup node needs to communicate with neighboring backup nodes; if the number of faults does not exceed the fault threshold, determining whether the recovery priority of the neighboring backup nodes is consistent with the recovery priority of the backup node based on preprocessed information; if not, determining that the backup node needs to communicate with neighboring backup nodes.

[0083] It should be noted that adjacent backup nodes refer to backup nodes that protect the same function. For example, backup node 1 protects functions 1 and 2, backup node 2 protects functions 2 and 3, and backup node 3 protects functions 1 and 3. Therefore, the adjacent backup nodes of backup node 1 include backup node 2 and backup node 3. If adjacent backup nodes can simultaneously detect a failure in the same function, they will communicate to determine which backup node will perform the recovery. If they agree, it means only one backup node will perform the recovery; that is, only one backup node among the adjacent backup nodes will perform the recovery. The value is 1, and all others are 0; if there is a discrepancy, the existing distributed algorithm will determine which backup node will be restored.

[0084] In some embodiments, if the number of faults detected by the backup node exceeds K or the recovery priority of the adjacent backup node is inconsistent with its own, it is determined that the backup node needs to communicate with the adjacent backup node.

[0085] In some embodiments, determining whether a backup node needs to communicate with other backup nodes depends on the complexity of the fault and the availability of resources. When a fault occurs, the backup node first attempts to autonomously complete the recovery operation based on the recovery priority information generated during the preprocessing phase and its local resource status. When the detected fault is relatively simple and the backup node has sufficient local resources, recovery can be completed without interaction with other backup nodes, thus avoiding additional communication overhead. However, if the fault involves multiple related functions, or if the backup node's local resources are insufficient to support the recovery operation, communication with neighboring backup nodes is required. This communication aims to exchange fault status and recovery priority strategies to ensure the rationality of resource allocation.

[0086] In step 205, if it is determined that this backup node needs to communicate with adjacent backup nodes to obtain the recovery priority information and fault information of the adjacent backup nodes, the recovery priority information of this backup node is adjusted based on the recovery priority information and fault information of the adjacent backup nodes, and the function is restored according to the adjusted recovery priority information.

[0087] In some embodiments, the backup node will obtain fault information from neighboring backup nodes and update the recovery priority, select the optimal recovery strategy, and perform the recovery operation locally.

[0088] This can be understood as follows: in complex scenarios, collaboration among backup nodes coordinates the recovery process through information sharing. Especially when the recovery priorities of multiple backup nodes overlap, the communication mechanism can dynamically adjust resource allocation, prioritizing high-priority functions and avoiding recovery delays due to resource contention. Furthermore, when multiple backup nodes detect failures simultaneously, the communication mechanism can quickly coordinate priority conflicts among them, thereby improving recovery efficiency. Through this collaborative mode, the system can more efficiently integrate distributed resources, ensuring rapid recovery of high-priority functions in a shared protection environment while reducing the possibility of resource conflicts, thus guaranteeing the overall recovery efficiency of the system.

[0089] In step 206, if it is determined that this backup node does not need to communicate with adjacent backup nodes, functional recovery is performed based on the recovery priority information of this backup node in the preprocessing information.

[0090] In some embodiments, if the number of faults detected by the backup node does not exceed K, and the recovery priority of adjacent backup nodes is consistent with its own, the backup node can directly restore the faulty function. In this case, the backup node... Select the function to be restored based on the restoration priority and perform the corresponding restoration operation.

[0091] In some embodiments, for recovery from a single-function failure, the backup node directly adopts the optimal priority obtained during the preprocessing phase. Perform the recovery operation.

[0092] In some embodiments, when a failure involves multiple backup nodes but communication is restricted, the backup nodes will independently perform recovery operations based on the optimal recovery priority information obtained in the preprocessing stage, in the absence of communication, to avoid resource contention and duplicate recovery.

[0093] Through the above embodiments, by leveraging the autonomous decision-making capabilities of backup nodes in the event of a failure, immediate responses can be made, reducing latency and enabling independent or collaborative recovery operations to achieve local recovery. This improves the overall efficiency of the system. Local recovery is more suitable for large-scale distributed systems. By decentralizing processing, local recovery avoids the performance bottlenecks of centralized control and can efficiently handle larger-scale system scenarios. Local recovery primarily completes processing within the backup nodes, greatly reducing global communication requirements and significantly lowering communication overhead. Combining the efficiency and flexibility of local recovery, it addresses the shortcomings of existing technologies, providing a more intelligent, efficient, and reliable solution for distributed systems. This solution is more suitable for large-scale distributed scenarios and can solve the problems of high resource consumption and insufficient fault recovery capabilities of multiple backup nodes in existing solutions.

[0094] Figure 3 This is a block diagram illustrating a fault-tolerance and recovery apparatus for a distributed cluster according to an exemplary embodiment. Figure 3 As shown, the distributed cluster includes a central controller and multiple shared backup nodes. The fault tolerance and recovery device of the distributed cluster may include: an information generation module 310, an information sending module 320, and a fault recovery module 330.

[0095] in, The information generation module 310 is used by the central controller to construct a protection graph based on cluster information and obtain the recovery priority information of each backup node in the protection graph under different fault scenarios. The protection graph represents the protection relationship between backup nodes and functions in a topological form. The cluster information includes the protection capabilities of backup nodes and the resource status of the primary node corresponding to the function. The information sending module 320 is used by the central controller to send the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessed information to each backup node, wherein the adjacent backup nodes are used to indicate the protection of at least one backup node with the same function. The fault recovery module 330 is used to restore functionality based on preprocessed information after the backup node detects functional fault information.

[0096] In some implementations, the information generation module 310 is specifically used for: Based on cluster information and combined with a bipartite graph-based fault tolerance optimization model, backup resource allocation information and recovery priority information are obtained. The bipartite graph-based fault tolerance optimization model aims to maximize the fault tolerance capability for potential future failures, and uses the protection capability and function of each backup node and the resource status of the primary node corresponding to it as constraints.

[0097] In some implementations, the fault recovery module 330 is specifically used for: Based on the preprocessed information, determine whether it is necessary to wait for unified scheduling by the central controller; If there is no need to wait for unified scheduling by the central controller, based on preprocessed information and the number of faults, determine whether this backup node needs to communicate with adjacent backup nodes; If it is determined that this backup node needs to communicate with neighboring backup nodes to obtain the recovery priority information and fault information of the neighboring backup nodes; Based on the recovery priority information and fault information of adjacent backup nodes, adjust the recovery priority information of this backup node; Functionality is restored based on the adjusted recovery priority information.

[0098] In some implementations, when the fault recovery module 330 determines whether the current backup node needs to communicate with neighboring backup nodes based on preprocessed information and the number of faults, it is specifically used for: Determine if the number of faults exceeds the fault threshold; If the number of faults exceeds the fault threshold, it is determined that this backup node needs to communicate with neighboring backup nodes; If the number of faults does not exceed the fault threshold, based on the preprocessing information, determine whether the recovery priority of the adjacent backup node is consistent with the recovery priority of this backup node; If not, it indicates that this backup node needs to communicate with neighboring backup nodes.

[0099] In some implementations, the fault recovery module 330 is also used for: If it is determined that this backup node does not need to communicate with adjacent backup nodes, functional recovery is performed based on the recovery priority information of this backup node in the preprocessing information.

[0100] In some implementations, the information generation module 310 is also used for: Real-time acquisition of operational data from each backup node; Update cluster information based on the operational data of backup nodes; Based on the updated cluster information, combined with the bipartite graph-based fault tolerance optimization model, the updated backup resource allocation information and recovery priority information are obtained.

[0101] In some implementations, the fault recovery module 340 is also used for: The backup node verifies the function corresponding to the fault information. If the verification is successful, it sends the recovery result information to the central processing unit. The information generation module 310 is also used for: Update cluster information based on recovery results; Based on the updated cluster information, combined with the bipartite graph-based fault tolerance optimization model, the updated backup resource allocation information and recovery priority information are obtained.

[0102] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0103] To implement the above embodiments, this application also proposes an electronic device. Please see [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 includes: a processor 401, and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer execution instructions; the processor 401 executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0104] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0105] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0106] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0108] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0109] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0111] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0113] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0114] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0116] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0117] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A fault tolerance and recovery method for a distributed cluster, characterized in that, The distributed cluster includes a central controller and multiple shared backup nodes, and the method includes: The central controller constructs a protection graph based on cluster information and obtains the recovery priority information of each backup node in the protection graph under different fault scenarios. The protection graph represents the protection relationship between the backup nodes and functions in a topological form. The cluster information includes the protection capabilities of each backup node and the resource status of the primary node corresponding to the function. The central controller sends the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessed information to each backup node, wherein the adjacent backup nodes are used to indicate the protection of at least one backup node with the same function. After the backup node detects a functional failure, it restores the function based on the preprocessed information. The process of constructing a protection graph based on cluster information and obtaining the recovery priority information of each backup node in the protection graph under different failure scenarios includes: Based on cluster information and combined with a bipartite graph-based fault-tolerant optimization model, backup resource allocation information and recovery priority information are obtained. The bipartite graph-based fault-tolerant optimization model aims to maximize the fault tolerance capability for potential future failures, using the protection capability of each backup node and the resource status of the primary node corresponding to its function as constraints. The bipartite graph-based fault-tolerant optimization model is an optimization model that models and solves for the set of backup nodes and the set of functions. The set of backup nodes and the set of functions are two vertex sets in a bipartite graph, and the protection relationship is the connecting edge between the vertices. Maximizing the fault tolerance capability for potential future failures is the number of functions protected by the remaining capacity after recovery.

2. The method according to claim 1, characterized in that, The function recovery based on the preprocessed information includes: Based on the preprocessed information, it is determined whether it is necessary to wait for the central controller to schedule the data. If there is no need to wait for the central controller to schedule, based on the preprocessed information and the number of faults, determine whether this backup node needs to communicate with neighboring backup nodes; If it is determined that this backup node needs to communicate with adjacent backup nodes, it can obtain the recovery priority information and functional fault information of the adjacent backup nodes; Based on the recovery priority information and functional failure information of the adjacent backup nodes, adjust the recovery priority information of this backup node; Functionality is restored based on the adjusted recovery priority information.

3. The method according to claim 2, characterized in that, The step of determining whether this backup node needs to communicate with neighboring backup nodes based on the preprocessed information and the number of faults includes: Determine whether the number of faults exceeds the fault threshold; If the number of faults exceeds the fault threshold, it is determined that this backup node needs to communicate with adjacent backup nodes; If the number of faults does not exceed the fault threshold, based on the preprocessing information, it is determined whether the recovery priority of the adjacent backup node is consistent with the recovery priority of this backup node; wherein, consistent recovery priority means that only one backup node performs functional recovery. If not, it indicates that this backup node needs to communicate with neighboring backup nodes.

4. The method according to claim 2, characterized in that, The method further includes: If it is determined that this backup node does not need to communicate with adjacent backup nodes, the function is restored based on the recovery priority information of this backup node in the preprocessed information.

5. The method according to claim 1, characterized in that, The method further includes: Real-time acquisition of operational data from each of the backup nodes; Update the cluster information based on the operational data of the backup nodes; Based on the updated cluster information, combined with the bipartite graph-based fault tolerance optimization model, the updated backup resource allocation information and recovery priority information are obtained.

6. The method according to claim 1, characterized in that, After restoring the function based on the preprocessed information, the following steps are included: The backup node verifies the function corresponding to the functional failure information. If the verification is successful, it sends the recovery result information to the central controller. The central controller updates the cluster information based on the recovery result information; Based on the updated cluster information, combined with the bipartite graph-based fault tolerance optimization model, the updated backup resource allocation information and recovery priority information are obtained.

7. A fault-tolerant and recovery device for a distributed cluster, characterized in that, The distributed cluster includes a central controller and multiple backup nodes, and the device includes: The information generation module is used by the central controller to construct a protection graph based on cluster information and obtain the recovery priority information of each backup node in the protection graph under different fault scenarios. The protection graph represents the protection relationship between the backup nodes and functions in a topological form. The cluster information includes the protection capabilities of each backup node and the resource status of the primary node corresponding to the function. The information sending module is used by the central controller to send the recovery priority information of each backup node and the information of adjacent backup nodes as preprocessed information to each backup node. The adjacent backup nodes are used to indicate the protection of at least one backup node with the same function. The fault recovery module is used to restore the function based on the preprocessed information after the backup node detects the functional fault information. The information generation module is specifically used for: Based on cluster information and combined with a bipartite graph-based fault-tolerant optimization model, backup resource allocation information and recovery priority information are obtained. The bipartite graph-based fault-tolerant optimization model aims to maximize the fault tolerance capability for potential future failures, using the protection capability of each backup node and the resource status of the primary node corresponding to its function as constraints. The bipartite graph-based fault-tolerant optimization model is an optimization model that models and solves for the set of backup nodes and the set of functions. The set of backup nodes and the set of functions are two vertex sets in a bipartite graph, and the protection relationship is the connecting edge between the vertices. Maximizing the fault tolerance capability for potential future failures is the number of functions protected by the remaining capacity after recovery.

8. An electronic device, characterized in that, include: One or more processors; The processor is used to invoke instructions to cause the electronic device to perform the method of any one of claims 1 to 6.

9. A storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method of any one of claims 1 to 6.

Images