High-availability service election method and system for cluster management controller

Abstract

This application provides a high-availability service election method and system for cluster management controllers. The method includes: when a controller node detects that it is in a minority partition of a network partition, it initiates a temporary election process independent of the global term to generate a partition leader; the partition leader receives a business request, encapsulates the business request into an operation intent object, executes the operation, and marks the operation result as a pending intent; after the minority partition and the majority partition reconnect, the global leader in the majority partition performs conflict detection based on the pending intent and the submitted intent, and performs a pre-registered compensation operation on the detected conflicts. This method achieves continuous availability of the controller cluster during partitioning and automatic consistency after healing without relying on external coordination services, meeting the needs of cross-regional management scenarios for availability and simplified operation and maintenance.

Description

Technical Field

[0001] This invention belongs to the field of operating system technology, and in particular relates to a high-availability service election method and system for cluster management controllers. Background Technology

[0002] In Kubernetes cluster management scenarios, controller clusters often employ consensus algorithms such as Raft to elect a leader to avoid single points of failure. When a network partition occurs, nodes in the minority partition are forced to stop service because they cannot form a majority, causing a complete interruption of management operations within that partition. Some solutions attempt to provide limited service within the minority partition, but after the partition heals, parallel operations on different partitions can lead to cluster state conflicts, such as the same resource being updated or deleted separately. Restoring consistency often requires manual intervention, increasing the operational burden. Existing methods relying on external coordination services introduce additional deployment complexity and cannot meet the needs of lightweight management. Summary of the Invention

[0003] To address the challenge of maintaining service availability of minority partitions during network partitioning for the Kubernetes cluster management controller without external coordination components, while ensuring that the controller cluster automatically recovers state consistency after partition healing, this application provides a high-availability service election method and system for the cluster management controller.

[0004] This application first provides a high-availability service election method for cluster management controllers, including: When a controller node detects that it is in a minority partition of a network partition, it initiates a temporary election process independent of the global term to produce a partition leader. The partition leader receives a business request, encapsulates the business request into an operation intent object, executes the operation, and marks the operation result as a pending intent. The operation intent object contains the compensation operation definition and expected pre-state of the operation. After the minority partition and the majority partition are reconnected, the global leader in the majority partition performs conflict detection based on the pending intent and the submitted intent, performs pre-registered compensation operations on the detected conflicts, restores the consistency of the cluster management state, and achieves high availability service election. The conflict detection includes: receiving the pending intent by the global leader, extracting its target resource identifier and expected pre-state; obtaining the global committed intent log during partitioning; and identifying conflicts using vector clock comparison based on the expected pre-state and the committed intent log. The method of identifying conflicts by comparing vector clocks includes: determining whether there is a break in causality based on the vector clock of the pending intent and the vector clock of the submitted operation corresponding to the target resource in the submitted intent log; if so, it is determined to be an unmergeable conflict.

[0005] Optionally, the initiation of a temporary election process independent of the global term includes: initiating an election using an independent partition epoch among nodes within the minority partition, with the node receiving a majority vote from nodes within the minority partition becoming the partition leader; the partition epoch is independent of the global term, so that after the partition heals, the partition leader is demoted to a follower without interfering with the global leader's term.

[0006] Optionally, initiating an election using an independent partition epoch number includes: generating a partition epoch number by appending a partition identifier based on the current global term, and sending a voting request carrying only the partition epoch number to other nodes within the partition.

[0007] Optionally, the operation intent object further includes a globally unique identifier for the operation; the method further includes: before the partition leader executes the operation, determining whether the operation is a preset partitionable operation based on the operation type in the operation intent object; if so, executing the operation; otherwise, rejecting the request.

[0008] Optionally, marking the operation result as a pending intent includes: recording the pending intent in the local intent log of the partition leader, and synchronizing the pending intent to other nodes in the minority partition via the partition's internal heartbeat.

[0009] Optionally, the conflict detection further includes: before using vector clock comparison, the global leader determines whether the target resource of the pending intent has a corresponding operation in the submitted intent log; if not, the pending intent is directly accepted as a submitted intent.

[0010] Optionally, performing the pre-registered compensation operation includes: calling the compensation function indicated by the compensation operation definition contained in the pending intent, and recording the compensation operation as a submitted intent.

[0011] Optionally, it also includes: counting the conflict frequency of various operations after multiple partition healings; when the conflict frequency of a certain type of operation exceeds a threshold, marking the operation as a strict majority operation, which will be directly rejected by the partition leader in the minority partition.

[0012] Optionally, the method of identifying conflicts by comparing vector clocks further includes: if the vector clock of the pending intent has a concurrent relationship with the vector clock of the corresponding operation in the submitted intent log, and the operation semantics are not interchangeable, then it is determined to be an unmergeable conflict.

[0013] This application also provides a cluster management controller system, including multiple controller nodes, which are configured to perform the methods described above.

[0014] In the embodiments provided in this application, when a controller node detects that it is in a minority partition, it initiates a temporary election process independent of the global term to generate a partition leader, thereby providing an execution exit for business requests within the partition. The partition leader encapsulates the business request into an operation intent object containing a compensation operation definition and a desired pre-state. After execution, it is only marked as a pending intent, rather than a global commit, giving the operation result revocable semantics and reserving a basis for subsequent repair. After the partition is restored to connectivity, the global leader of the majority partition collects all pending intents, obtains the global intent logs committed during the partition, and uses a vector clock to compare the desired pre-state of the pending intent with the committed operations to determine whether there is a break in causality, thereby accurately identifying unmergeable conflicts. For detected conflicts, the pre-registered compensation operation is invoked to automatically roll back, retaining the majority operation results and restoring the parallel operation sequence during the partition to a globally consistent state. In this process, the partition epoch of the temporary election is isolated from the global term, so that there is no need to re-compete for leadership when the partition heals; the desired pre-state and vector clock carried by the pending intent provide a reliable basis for conflict identification and avoid misjudgment relying on the physical clock. Thus, this method achieves continuous availability and automatic consistency of the controller cluster during partitioning without relying on external coordination services, meeting the dual requirements of availability and simplified operation and maintenance in cross-regional management scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the high availability service election method for the cluster management controller in the embodiments of this application; Figure 2 This is one of the flowcharts of the sub-steps of the high availability service election method for the cluster management controller in the embodiments of this application; Figure 3 This is the second flowchart of a sub-step in the high availability service election method for the cluster management controller in this application embodiment. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] like Figure 1 As shown in the figure, this application embodiment provides a high-availability service election method for a cluster management controller, including: S101: When a controller node detects that it is in a minority partition of a network partition, it initiates a temporary election process independent of the global term to generate a partition leader. S102: The partition leader receives a service request, encapsulates the service request into an operation intent object, executes the operation, and marks the operation result as a pending intent, wherein the operation intent object contains the compensation operation definition and expected pre-state of the operation; S103: After the minority partition and the majority partition are reconnected, the global leader in the majority partition performs conflict detection based on the pending intent and the submitted intent, performs pre-registered compensation operations on the detected conflicts, restores the consistency of the cluster management state, and achieves high availability service election. The conflict detection includes: receiving the pending intent by the global leader, extracting its target resource identifier and expected pre-state; obtaining the global committed intent log during partitioning; and identifying conflicts using vector clock comparison based on the expected pre-state and the committed intent log. The method of identifying conflicts by comparing vector clocks includes: determining whether there is a break in causality based on the vector clock of the pending intent and the vector clock of the submitted operation corresponding to the target resource in the submitted intent log; if so, it is determined to be an unmergeable conflict.

[0019] In this embodiment, controller nodes run on multiple physical or virtual servers, forming a controller cluster. Each controller node possesses complete cluster management capabilities and can independently handle various cluster management requests. Controller nodes communicate with each other via a network, exchanging heartbeat information and status data. Under normal operating conditions, the controller cluster elects a global leader through a consensus algorithm, and the global leader is responsible for handling all cluster management requests. Other controller nodes act as followers, not directly handling business requests, but only participating in the election process and state synchronization. When the global leader fails, the follower nodes automatically initiate a new election to select a new global leader to take over the business.

[0020] In this embodiment, network partitioning refers to a controller cluster being divided into two or more sub-clusters that cannot communicate with each other due to a network failure. Nodes within each sub-cluster can communicate normally, but nodes in different sub-clusters cannot exchange information. After a network partition occurs, the original global leader may reside in one of the sub-clusters. If the sub-cluster containing the global leader has more than half of the nodes, the sub-cluster can continue to operate normally and maintain its global leader status. If the sub-cluster containing the global leader has less than half of the total number of nodes, the sub-cluster cannot form a majority, the original global leader will automatically be demoted to a follower, and all nodes within the sub-cluster will be unable to process business requests.

[0021] In this embodiment, the controller node continuously monitors network connectivity with all other controller nodes. Each controller node periodically sends heartbeat messages to all other nodes and waits to receive heartbeat responses from them. If a controller node does not receive a heartbeat response from a node within a preset timeout period, it considers the network connection with that node to be interrupted. When a controller node detects that it has lost connection with more than half of the nodes, it further checks the number of nodes that can currently be connected. If the number of nodes that can be connected is less than half of the total configured number of nodes, the controller node determines that it is in the minority partition. If the number of nodes that can be connected is greater than or equal to half of the total configured number of nodes, the controller node determines that it is in the majority partition.

[0022] In this embodiment, the total number of configured nodes is maintained centrally by the cluster configuration center and supports dynamic updates. When a new node joins the cluster or an existing node is removed, the cluster configuration center broadcasts the updated total number of configured nodes to all controller nodes. Controller nodes always use the latest total number of configured nodes for calculation when determining minority partitions. If a controller node cannot communicate with the cluster configuration center during a network partition, it uses the last received total number of configured nodes before the partition occurred for determination, until the network recovers and the latest configuration is synchronized.

[0023] In this embodiment, when a controller node determines that it is in a minority partition, it initiates a temporary election process. This temporary election process is independent of the normal global election process and does not affect the state of the global election. The temporary election process takes place within the minority partition, and a node only needs to obtain the votes of a majority of nodes within the minority partition to be elected as the partition leader. The partition leader is responsible for handling service requests within the minority partition during network partitioning and maintaining service availability within the minority partition.

[0024] In this embodiment, after receiving a business request from a client, the partition leader encapsulates the request into an operation intent object. The operation intent object contains all the information required to execute the operation, as well as metadata for subsequent conflict detection and rollback. The structure of the operation intent object includes a globally unique operation identifier, operation type, target resource identifier, expected pre-state, request parameters, compensation operation identifier, initiator node, and intent state. The globally unique operation identifier is used to uniquely identify an operation across the entire cluster and is generated by a combination of node identifier, logical clock, and random number. The operation type distinguishes different types of cluster management operations. The target resource identifier identifies the cluster resource affected by the operation. The expected pre-state records the state that the cluster resource should be in before executing the operation. The request parameters contain the specific parameters required to execute the operation. The compensation operation identifier points to the rollback operation corresponding to the operation. The initiator node records the identifier of the controller node that initiated the operation. The intent state records the current execution state of the operation.

[0025] In this embodiment, the expected pre-state is a structured data object containing the following fields: target resource ID, resource version number, corresponding vector clock, and hash values ​​of key attributes. The target resource ID uniquely identifies the cluster resource affected by the operation; the resource version number is the latest version number of the resource before the operation is executed; the corresponding vector clock is the latest vector clock of the resource before the operation is executed; and the hash values ​​of the key attributes are the SHA-256 hash values ​​of all key attribute values ​​of the resource, used to quickly detect whether the resource state has changed. The expected pre-state is extracted from the current cluster state by the pre-state extraction function before the operation is executed.

[0026] In this embodiment, before executing an operation, the partition leader checks whether the operation is allowed within the minority partition. The system predefines two types of operations: those allowed within minority partitions and those not allowed. Operations not allowed within minority partitions are typically those with a wide impact, difficult to roll back after execution, or with high rollback costs. For operations not allowed within minority partitions, the partition leader directly rejects the request and returns an error message to the client. For operations allowed within minority partitions, the partition leader executes the operation normally and returns the result to the client.

[0027] In this embodiment, after the partition leader completes an operation, it does not mark the operation as committed, but rather as a pending intent. A pending intent indicates that the operation is only valid within the minority partition, and its final validity needs to be confirmed by the global leader after the network partition recovers. The partition leader records the pending intents in its local intent log and synchronizes them to other nodes within the minority partition via heartbeat messages within the partition. This ensures that even if the partition leader fails before the network partition recovers, other nodes can retain these pending intents, guaranteeing that they can be submitted to the global leader for processing after the network partition recovers.

[0028] In this embodiment, after the minority partition and the majority partition re-establish network connectivity, nodes within the minority partition will detect that the network partition has healed. At this point, the partition leader will automatically exit the temporary leadership state and be demoted to a follower. The partition leader will then package all pending intentions within its partition and send them to the current global leader. Upon receiving the pending intentions, the global leader will initiate a conflict detection process to determine whether these pending intentions can be accepted globally.

[0029] In this embodiment, the global leader maintains a global intent log for the entire cluster. The global intent log records all committed operations that have been globally acknowledged. Each committed operation is associated with a vector clock, which records the logical time at which the operation occurred. A vector clock is a mechanism used to determine the order of events in a distributed system, accurately capturing the causal relationships between events.

[0030] In this embodiment, when performing conflict detection, the global leader first extracts the target resource identifier and expected preceding state from the pending intent. Then, the global leader queries the global intent log to obtain all committed operations performed on the target resource during the network partitioning. Next, the global leader compares the vector clock of the pending intent with the vector clock of the committed operations to determine if there is a break in the causal relationship between them. If there is a break in the causal relationship between the expected preceding state of the pending intent and the states in the global committed operation sequence, it indicates that there is a conflict between the pending intent and the global committed operations, and they cannot be directly merged.

[0031] In this embodiment, a break in causality refers to a situation where the cluster state upon which the pending intention is based has been altered by a globally committed operation, causing a contradiction between the execution result of the pending intention and the execution result of the globally committed operation. For example, the pending intention performs an update operation based on resource version V3, but the globally committed operation has already updated the resource to version V4, and these two update operations cannot coexist. In this case, a break in causality exists, requiring conflict resolution.

[0032] In this embodiment, when a conflicting pending intent is detected, the global leader executes a pre-registered compensation operation. The compensation operation is a rollback operation corresponding to the original operation, capable of restoring cluster resources to their state before the original operation was executed. The global leader calls the compensation function indicated by the compensation operation definition contained in the pending intent to execute the rollback operation. After the rollback operation is completed, the global leader records the compensation operation itself as a submitted intent to ensure the integrity of the operation history. Simultaneously, the global leader sends a notification to the initiator of the original operation, informing them that the operation has been rolled back and the reason for the rollback.

[0033] In this embodiment, for pending intents that do not conflict, the global leader will directly mark them as committed and update the global intent log. In this way, operations executed within the minority partition are globally accepted and become part of the cluster state.

[0034] In this embodiment, the system tracks the conflict frequency of various operations after multiple network partitions have healed. When the conflict frequency of a certain type of operation exceeds a preset threshold, the system automatically marks that type of operation as not allowed to be executed in minority partitions. Subsequently, when a network partition occurs again, the partition leader will directly refuse to execute that type of operation, thereby avoiding unnecessary conflicts and rollbacks. The system also supports manually configuring the execution permissions of operations, allowing operations personnel to adjust the execution strategies for various operations according to actual business needs.

[0035] In some embodiments, the initiation of a temporary election process independent of the global term, such as... Figure 2 As shown, it includes: S201: An election using an independent partition epoch is initiated among the nodes in the minority partition. The node that receives the votes of a majority of nodes in the minority partition becomes the partition leader. The partition epoch is independent of the global term, so that after the partition is healed, the partition leader is demoted to a follower without interfering with the global leader's term.

[0036] In this embodiment, the initiation of a temporary election process independent of the global term includes initiating an election using an independent partition epoch among nodes within the minority partition. The node that receives a majority of votes from nodes within the minority partition becomes the partition leader. The partition epoch is independent of the global term, ensuring that after partition healing, the partition leader is demoted to a follower without interfering with the global leader's term.

[0037] In this embodiment, the normal global election process uses global term of office to identify the election round. Global term of office is a globally monotonically increasing integer, incremented by one each time a new global election is initiated. Nodes prioritize voting for candidates with higher terms of office. If temporary elections within minority partitions also use global term of office, then when nodes in minority partitions initiate elections, the global term of office will be incremented. When a network partition heals, nodes in minority partitions will have higher global terms of office, which could force the global leader in the majority partition to be demoted, triggering unnecessary re-elections and affecting system stability.

[0038] In this embodiment, to avoid the aforementioned problems, the temporary election process uses an independent partition epoch number instead of a global term. The partition epoch number is an identifier valid only within the minority partition and is independent of the global term. Incrementing the partition epoch number does not affect the value of the global term. When the network partition heals, nodes in the minority partition will find that there is a valid global leader in the majority partition, and that the global leader's global term is not less than the local global term. At this point, nodes in the minority partition will automatically recognize the authority of the global leader, and the partition leader will be demoted to a follower, without triggering a new global election.

[0039] In this embodiment of the application, initiating an election using an independent partition epoch number includes generating a partition epoch number by appending a partition identifier based on the current global term, and sending a voting request carrying only the partition epoch number to other nodes within the partition.

[0040] In this embodiment, there are several ways to generate the partition epoch number. One method is to use the current global term as a prefix, followed by a partition identifier and a locally incrementing counter. This generates a partition epoch number that uniquely identifies a specific election round within a specific partition. The partition identifier can be a randomly generated string or a hash value of the identifiers of all nodes within the minority partition. When a node in the minority partition initiates a temporary election, it generates a partition epoch number for this election and sends a voting request to other nodes within the partition. The voting request only carries the partition epoch number and not the global term. After receiving the voting request, other nodes determine whether they should vote for the candidate based on the partition epoch number.

[0041] In this embodiment, the partition identifier is generated using the format "global term + sorted hash of the node ID set within the partition + local incrementing counter". Specifically, firstly, the IDs of all reachable nodes within the minority partition are sorted lexicographically, then the SHA-256 hash value of the sorted node ID list is calculated, and the first 16 bits are used as the unique identifier of the partition. The local incrementing counter is initially set to 0, and increments by 1 each time a new temporary election is initiated within the same partition. This generation method ensures that even if multiple network partitions occur within the same cluster, the partition epoch numbers of different partitions will not be duplicated. When a partition split occurs, each sub-partition after the split will generate a new partition identifier and partition epoch number; when a partition merge occurs, the merged partition will undergo a new temporary election to generate a new partition epoch number.

[0042] In this embodiment of the application, the operation intent object further includes a globally unique operation identifier. The method also includes, before the partition leader executes the operation, determining whether the operation is a preset partitionable operation based on the operation type in the operation intent object; if so, executing the operation; otherwise, rejecting the request.

[0043] In this embodiment, the globally unique operation identifier (GUID) is used to uniquely identify an operation across the entire cluster. The generation of the GUID must ensure that no conflicts occur in a distributed environment. One generation method is to combine a node identifier, the current logical clock, and a random number. The node identifier ensures that identifiers generated by different nodes will not conflict. The logical clock ensures that identifiers generated by the same node at different times will not conflict. The random number further reduces the probability of conflicts.

[0044] In this embodiment, the system initializes an intent registry center upon startup. The intent registry center registers all supported cluster management operations. Each operation, upon registration, needs to provide an operation type identifier, a globally unique identifier generation rule, a pre-state extraction function, a positive operation execution function, and a compensation operation function. The pre-state extraction function extracts the pre-state information required to execute the operation from the current cluster state. The positive operation execution function executes the actual cluster management operation. The compensation operation function executes the rollback operation. Additionally, each operation, upon registration, needs to specify whether execution within a minority partition is permitted.

[0045] It should be understood that the intended registration interface in this application is a specific function interface, which is prior art and will not be described in detail here. For example, the specific function interface is as follows: int register_intent_handler( const char* operation_type, uuid_generator_t uuid_generator, pre_state_extractor_t pre_state_extractor, forward_executor_t forward_executor, backward_executor_t backward_executor, bool is_commutative ); Here, `operation_type` is a unique identifier for the operation type; `uuid_generator` is a function pointer that generates a globally unique identifier for the operation; `pre_state_extractor` is a function pointer that extracts the expected preceding state; `forward_executor` is a function pointer that executes the positive operation; `backward_executor` is a function pointer that executes the compensating operation; and `is_commutative` indicates whether the operation is semantically commutative with other operations of the same type. Registration returns 0 on success and a negative error code on failure.

[0046] In this embodiment, when a partition leader receives a business request, it first queries the intent registry based on the operation type to obtain the registration information for that operation. If the operation is marked as not allowed to be executed within the minority partition, the partition leader will directly reject the request and return an error message to the client. If the operation is marked as allowed to be executed within the minority partition, the partition leader will continue to execute the subsequent operation process.

[0047] In this embodiment of the application, marking the operation result as a pending intent includes recording the pending intent in the local intent log of the partition leader and synchronizing the pending intent to other nodes in the minority partition through the partition's internal heartbeat.

[0048] In this embodiment, the intent log is a persistently stored log file used to record all executed operations. Each controller node maintains its own local intent log. The global leader's intent log contains all committed operations. The partition leader's intent log contains globally committed operations and pending operations within its partition.

[0049] In this embodiment, after a partition leader executes an operation and marks it as a pending intent, it appends the pending intent to its local intent log. Then, the partition leader includes the information of the pending intent in subsequent intra-partition heartbeat messages, synchronizing it with other nodes within the minority partition. Upon receiving the pending intent, other nodes append it to their own local intent logs. This ensures that even if the partition leader fails before the network partition recovers, other nodes can retain these pending intents. When a new partition leader is elected, it continues to process these pending intents, ensuring they can be submitted to the global leader after the network partition recovers.

[0050] In this embodiment of the application, the conflict detection further includes, before using vector clock comparison, the global leader determining whether the target resource of the pending intent has a corresponding operation in the submitted intent log. If it does not exist, the pending intent is directly accepted as a submitted intent.

[0051] In this embodiment, upon receiving a pending intent, the global leader first checks whether the target resource of the pending intent has been modified by a globally committed operation during network partitioning. If no committed operation for the target resource is recorded in the global intent log, it indicates that there is no conflict between the pending intent and the global state. In this case, the global leader can directly mark the pending intent as committed and update the global intent log. This eliminates the need for complex vector clock comparisons, improving the efficiency of conflict detection.

[0052] In this embodiment, vector clock comparison is only required to determine whether a causal relationship break occurs when the target resource of a pending intent has been modified by a globally committed operation during network partitioning. This avoids unnecessary complex calculations for all pending intents and improves the system's processing performance.

[0053] In this embodiment of the application, the execution of the pre-registered compensation operation includes calling the compensation function indicated by the compensation operation definition contained in the pending intent, and recording the compensation operation as a submitted intent.

[0054] In this embodiment, the compensation operation function is predefined during operation registration. Each operation permitted to be executed within a minority partition must have a corresponding compensation operation function. The compensation operation function can restore the cluster resources to the state before the original operation was executed. For example, for an operation that creates a cluster, its compensation operation function is to delete the cluster. For an operation that updates node labels, its compensation operation function is to restore the node labels to their original values.

[0055] In this embodiment, when the global leader detects a conflict between a pending intent and a globally committed operation, it invokes the compensation operation function corresponding to the pending intent. After the compensation operation function is executed, the cluster resources are restored to their state before the original operation was executed. Then, the global leader records the compensation operation itself as a committed intent and adds it to the global intent log. In this way, the entire operation history is completely recorded, facilitating subsequent auditing and troubleshooting.

[0056] In some embodiments, such as Figure 3 As shown, it also includes: S301: Statistical analysis of conflict frequency of various operations after multiple partition healings; S302: When the conflict frequency of a certain type of operation exceeds the threshold, the operation is marked as a strict majority operation and will be directly rejected by the partition leader in the minority partition.

[0057] In this embodiment, the system further includes statistically analyzing the conflict frequency of various operations after multiple partition healings. When the conflict frequency of a certain type of operation exceeds a threshold, that type of operation is marked as a strict majority operation, and subsequently directly rejected by the partition leader in the minority partition.

[0058] In this embodiment, the system continuously monitors conflicts of various operations after network partition healing. After each network partition healing, the system counts the number of executions and conflicts of various operations during the partitioning period. Then, the system calculates the conflict frequency of each operation, i.e., the ratio of conflict counts to execution counts. When the conflict frequency of a certain type of operation exceeds a preset threshold multiple times consecutively, the system automatically adjusts the execution permission of that type of operation to only allow execution within the majority partition. Subsequently, when a network partition occurs again, the partition leader will directly refuse to execute that type of operation, thereby avoiding unnecessary conflicts and rollbacks.

[0059] In this embodiment, the initial conflict frequency threshold for all operation types is 0.3. The system dynamically adjusts the threshold based on the overall cluster conflict rate, with an adjustment step of 0.05. When the overall cluster conflict rate is higher than 0.1, the threshold for all operation types decreases by 0.05; when the overall cluster conflict rate is lower than 0.05, the threshold for all operation types increases by 0.05. The threshold adjustment range is limited to between 0.1 and 0.5 to prevent system instability caused by thresholds that are too high or too low.

[0060] In this embodiment, the system uses a sliding window averaging method to calculate the conflict frequency of an operation. The statistical window size is 7 calendar days. At the end of each calendar day, the conflict frequency for that day is calculated, and then the average of the most recent 7 calendar days is taken as the final conflict frequency for that operation. The sliding window averaging method can effectively smooth data fluctuations and avoid the impact of a single abnormal event on the statistical results. The system automatically excludes the statistical results of operations that have been executed less than 10 times to prevent misjudgments due to insufficient sample size.

[0061] In this embodiment, the system also supports manually adjusting the execution permissions of operations. Operations personnel can manually mark certain operations as only allowed to be executed in the majority partition, or mark certain operations as allowed to be executed in the minority partition, based on actual business needs. This allows for flexible adaptation to different business scenarios and requirements.

[0062] In this embodiment, the system also supports an automatic permission rollback mechanism. When an operation is marked as a strict majority operation, the system continues to track the conflict frequency of that operation. For example, if the conflict frequency of the operation is below 50% of the current threshold for three consecutive statistical periods, the system will automatically restore it to a partitionable execution operation. The automatic permission rollback mechanism can avoid unnecessary permission restrictions caused by temporary business fluctuations and improve system availability.

[0063] In this embodiment of the application, the method of identifying conflicts by comparing vector clocks further includes determining that if the vector clock of the pending intent and the vector clock of the corresponding operation in the submitted intent log are concurrent and the operation semantics are not interchangeable, then the conflict cannot be merged.

[0064] In this embodiment, the vector clock can distinguish between causal and concurrent relationships between events. If the vector clock of event A is less than the vector clock of event B, it indicates that event A occurred before event B, and there is a causal relationship between them. If the vector clocks of event A and event B cannot be compared, it indicates that events A and B occurred concurrently, and there is no causal relationship between them.

[0065] In this embodiment, when the pending intent and the globally committed operation occur concurrently, it is necessary to further determine whether the semantics of these two operations are interchangeable. If the semantics of the two operations are interchangeable, that is, the final result is the same regardless of the execution order, then the two operations do not conflict and can be directly merged. If the semantics of the two operations are not interchangeable, that is, different execution orders will lead to different results, then the two operations conflict and need to be rolled back.

[0066] In this application embodiment, whether the semantics of operations are interchangeable is predefined during operation registration. For example, two operations that update different node labels are interchangeable because the final node label state is the same regardless of which operation is executed first. However, the operations of creating a resource with the same name and deleting the resource are not interchangeable because the results of creating first and then deleting are completely different from those of deleting first and then creating.

[0067] In this embodiment, the system maintains a global operation semantic commutativity rule table. This rule table is a two-dimensional matrix, with rows and columns corresponding to different operation types. Elements in the matrix are Boolean values, indicating whether two corresponding operations are semantically commutative. For example, the matrix element corresponding to the "update node label" operation and the "update node label" operation (operating on different nodes) is true, indicating that these two operations are semantically commutative; the matrix element corresponding to the "create resource" operation and the "delete resource" operation (operating on the same resource) is false, indicating that these two operations are semantically non-commutative. The commutativity rule table is loaded from the configuration file at system startup and supports dynamic updates at runtime.

[0068] In this embodiment of the application, a cluster management controller system includes multiple controller nodes, which are configured to execute any of the methods described above.

[0069] In this embodiment, the controller system consists of multiple controller nodes. Each controller node includes an election module, an intent processing module, a conflict detection module, and a compensation execution module. The election module is responsible for executing the global election and temporary election processes to elect a global leader and a partition leader. The intent processing module is responsible for encapsulating business requests into operation intent objects, executing the operations, and recording intent logs. The conflict detection module is responsible for detecting conflicts between pending intents and globally submitted intents after network partition healing. The compensation execution module is responsible for executing compensation operations and rolling back conflicting pending intents.

[0070] In this embodiment, controller nodes communicate with each other via a network. Each controller node knows the network addresses of all other controller nodes. Controller nodes periodically exchange heartbeat messages to detect node liveness and network connectivity. When a network partition occurs, the controller node can automatically detect the partition status and initiate the corresponding processing flow.

[0071] In this embodiment, during normal system operation, all business requests are handled by the global leader. The global leader encapsulates the business request into an operation intent object, executes the operation, and marks it as submitted. Then, the global leader synchronizes the summary of the submitted intent to other follower nodes. Upon receiving the summary, the follower nodes record it in their local intent log summary table. The global leader also periodically pushes the complete intent log asynchronously to some follower nodes to achieve redundancy backup.

[0072] In this embodiment, when a network partition occurs, the nodes within the minority partition initiate a temporary election to elect a partition leader. The partition leader is responsible for handling service requests within the minority partition. The partition leader encapsulates the service request into an operation intent object, executes the operation, and marks it as a pending intent. Then, the partition leader synchronizes the pending intents to the other nodes within the minority partition.

[0073] In this embodiment, after a network partition heals, the partition leader submits all pending intentions to the global leader. The global leader performs conflict detection on these pending intentions. For pending intentions without conflicts, the global leader marks them as submitted. For pending intentions with conflicts, the global leader performs a compensation operation to roll back the pending intentions. Then, the global leader notifies the relevant clients and nodes of the processing results.

[0074] In this embodiment, the system continuously learns the conflict patterns of various operations and dynamically adjusts the execution permissions of operations. For operations that frequently conflict, the system automatically adjusts them to only be allowed to execute within the majority partition, thereby reducing subsequent conflicts and rollbacks.

[0075] For example, a cluster management platform spanning two data centers deploys five controller nodes. Two nodes are located in the first data center, and the other three are located in the second data center. The operations are: Create Cluster (registered compensation operation), Delete Cluster; Update Node Label (registered compensation operation), Restore Original Label; Delete Cluster (marked as only allowed within a majority partition).

[0076] Under normal operating conditions, the first node in the first data center is the global leader. The administrator initiates a request to create cluster X. The global leader converts this request into an intent object, with its preceding state based on the cluster list version number V5. The global leader executes the cluster creation operation, marks the intent as committed upon success, and synchronizes the intent digest to other nodes. Cluster X is successfully created.

[0077] Subsequently, the network between the first and second data centers was interrupted, creating two independent network partitions. The first data center contained two nodes, and the second data center contained three nodes. According to the majority rule, the nodes in the second data center formed the majority partition, and the nodes in the first data center formed the minority partition. The original global leader, located in the minority partition of the first data center, was automatically demoted to a follower due to the lack of support from a majority of nodes. The nodes in the second data center initiated a new global election, electing the third node located in the second data center as the new global leader.

[0078] Meanwhile, the two nodes in the first data center detected that they were in a minority partition and initiated a temporary election process. The temporary election used a separate partition epoch number and did not affect the global term. After voting, the original global leader was elected as the partition leader. The partition leader generated a partition identifier to identify this network partition.

[0079] A client located in the first data center sends a request to the local partition leader to update a node label, with the target node also located in the first data center. The partition leader first checks the type of operation, confirming that updating the node label is permitted within the minority partition. Then, the partition leader encapsulates the request as an intent object based on the locally recorded node label version V2. The partition leader executes the update node label operation; upon successful execution, the intent is marked as pending and a temporarily valid response is returned to the client. The partition leader records this pending intent in its local intent log and synchronizes it to another node in the first data center via heartbeat messages within the partition.

[0080] Meanwhile, within the majority partition of the second data center, the global leader received a request from another administrator to delete cluster X. The global leader checked the type of operation and confirmed that deleting the cluster was an operation only permitted within the majority partition. The global leader executed the cluster deletion operation, marked the intent as committed upon success, and synchronized it with other nodes within the majority partition. Cluster X was successfully deleted.

[0081] After a period of time, the network between the first and second data centers was restored, and the network partition healed. Nodes in the first data center detected that a new connection had been established with the majority partition. The partition leader automatically exited the temporary leadership state and was demoted to a follower. The partition leader packaged all pending intentions within its partition and sent them to the current global leader.

[0082] Upon receiving the list of pending intents from the minority partition, the global leader initiates a conflict detection process. The global leader first processes pending intents that update node labels. It extracts the target resource identifier and expected preceding state V2 for this pending intent. Then, the global leader queries the global intent log and finds no committed operations against this target node during the network partitioning. Therefore, there is no conflict between this pending intent and the global state. The global leader directly marks this pending intent as committed and updates the node label version to V3. This node label update operation is accepted globally.

[0083] Next, suppose that during the network partition, the partition leader also executed an operation to create cluster Y. When the global leader processes this pending intent, it queries the global intent log and finds that there were no committed operations for cluster Y during the network partition. Therefore, there is no conflict with this pending intent. The global leader directly marks it as committed, and cluster Y is officially created.

[0084] Assume that during the network partitioning process, the partition leader also performed an operation to update the configuration of cluster X. When the global leader processes this pending intent, it queries the global intent log and finds that the globally committed operation has already deleted cluster X. The target resource of the pending intent no longer exists, and the resource state corresponding to its expected preceding state has been changed by the globally committed operation. Therefore, there is a break in the causal relationship between the pending intent and the globally committed operation, and it is determined to be an unmergeable conflict.

[0085] The global leader invokes the compensation function corresponding to the update cluster configuration operation. Since cluster X has been deleted, the compensation function cannot actually perform a rollback operation; it only logs the relevant information and sends a notification to the original initiator of the operation. The global leader records the compensation operation as a committed intent and adds it to the global intent log.

[0086] During subsequent operation, the system analyzed the conflict frequency of various operations. The system detected that the cluster configuration update operation clashed in multiple network partitions, exceeding a preset threshold. Therefore, the system automatically marked the cluster configuration update operation as only allowed to execute in majority partitions. Subsequently, when another network partition occurs, the partition leader will directly refuse to execute the cluster configuration update operation, avoiding unnecessary conflicts and rollbacks.

[0087] In this embodiment, business operations are formalized as intent objects with compensation definitions, enabling operations performed during network partitioning to be undone. Traditional high-availability solutions either completely halt services in minority partitions during network partitioning or allow minority partitions to run independently but cannot automatically resolve conflicts after partition healing. This application, by introducing intent objects and compensation mechanisms, provides security guarantees for optimistic execution during partitioning.

[0088] In this embodiment, a temporary election process, independent of the global term, is initiated within the minority partition to generate a partition leader. Traditional consensus algorithms, when partitioning the network, often fail to form a majority of nodes within the minority partition, thus preventing the election of a leader and leading to a complete service interruption within that partition. This application introduces a temporary election mechanism, enabling nodes within the minority partition to elect a temporary partition leader without compromising global consistency, allowing them to continue providing services. The temporary election uses an independent partition epoch, avoiding contamination of the global term and ensuring a smooth return to a globally consistent state after partition healing.

[0089] In this embodiment, the partition leader marks the operation as a pending intent after execution, rather than submitting it directly. This pending mechanism makes the results of operations performed within the partition temporary, and their final validity needs to be confirmed by the global leader after the partition heals. This ensures service availability within the partition while also reserving space for subsequent conflict resolution.

[0090] In this embodiment, vector clock comparison is used for conflict detection. Traditional conflict detection methods typically rely on physical timestamps to determine the order of events. However, in distributed systems, physical clocks cannot guarantee perfect synchronization, which can easily lead to misjudgments. Vector clocks can accurately capture the causal relationships between events, providing a reliable basis for conflict detection. By comparing the vector clocks of the intended action and globally committed operations, it is possible to accurately identify which operations have broken causal relationships, thereby determining whether a conflict exists.

[0091] In this embodiment, for detected conflicts, pre-registered compensation operations are automatically invoked for rollback. Traditional solutions typically require manual intervention when conflicts occur, increasing the operational burden. This application achieves automatic rollback of conflicts through pre-registered compensation operations, eliminating the need for manual intervention and improving the system's automation level and operational efficiency.

[0092] In this embodiment, the system dynamically adjusts the execution permissions of operations by statistically analyzing the conflict frequency of various operations. For operations that frequently conflict, the system automatically adjusts them to only be allowed to execute within the majority partition. This self-learning mechanism enables the system to automatically optimize based on actual operating conditions, achieving a dynamic balance between availability and consistency.

[0093] In this embodiment, the various technical components cooperate to form a complete closed loop. The intent object and compensation mechanism provide a security foundation for optimistic execution. The temporary election mechanism provides service capabilities for minority partitions. The pending mechanism reserves space for subsequent conflict resolution. Vector clock conflict detection provides a reliable basis for conflict identification. The automatic compensation mechanism enables automatic conflict repair. The self-learning mechanism allows the system to continuously optimize. These components are interdependent and indispensable, collectively constituting a highly available service election scheme suitable for cluster management controllers.

[0094] In this embodiment, compared to traditional high-availability solutions based on external coordination services, this application does not require the additional deployment of external coordination services; all functions are embedded within the controller node itself. This reduces the system's deployment complexity and operational costs, while also avoiding the risk of the external coordination service itself becoming a single point of failure.

[0095] In this embodiment, compared to traditional primary / standby failover schemes, this application can simultaneously provide services to both the majority and minority partitions during network partitioning, improving the overall availability of the system. Traditional primary / standby failover schemes only allow the majority partition to continue providing services during network partitioning, while the minority partition completely ceases service. This application, through a temporary election mechanism, enables even the minority partition to continue providing limited services, meeting the high availability requirements in cross-regional deployment scenarios.

[0096] In this application's embodiments, compared to traditional optimistic concurrency control schemes, this application can not only detect conflicts but also automatically resolve them. Traditional optimistic concurrency control schemes typically only detect conflicts and reject conflicting operations, but cannot automatically roll back already executed operations. This application, by introducing a compensation mechanism, can automatically roll back conflicting operations, restoring the system to a consistent state.

[0097] In this embodiment, the controller node loads a configuration file upon startup, which contains the network addresses and port information of all other controller nodes in the cluster. The controller node initializes a node list based on the information in the configuration file and attempts to establish network connections with other nodes. If a controller node cannot establish a connection with a node, it will periodically retry until the connection is successful or the node is removed from the cluster.

[0098] In this embodiment, communication between controller nodes uses a lightweight protocol, facilitating penetration of firewalls and load balancing devices. Communication messages employ a structured data format for easy parsing and processing. All communication messages include a sender node identifier and a message sequence number to prevent message duplication and out-of-order delivery.

[0099] In this embodiment, the global leader periodically sends heartbeat messages to all other nodes. The heartbeat message contains the global leader's node identifier, current global term, and latest committed intent summary. Upon receiving the heartbeat message, follower nodes update their local global term and committed intent summary tables and send a heartbeat response to the global leader. If a follower node does not receive a heartbeat message from the global leader within a preset timeout period, it assumes the global leader has failed and initiates a new global election.

[0100] In this embodiment, the global election process is as follows: When a follower node does not receive a heartbeat message from the global leader within the timeout period, it will change its state to candidate, increment its global term, and vote for itself. Then, the candidate sends a vote request to all other nodes. The vote request includes the candidate's node identifier, current global term, and candidate's log state. After receiving the vote request, other nodes will decide whether to vote for the candidate based on their global term and log state. If the candidate receives votes from more than half of the nodes, it becomes the new global leader. The new global leader begins sending heartbeat messages to other nodes to maintain its leadership position.

[0101] In this embodiment, the temporary election process is similar to the global election process, but uses an independent partition epoch. When a controller node detects that it is in a minority partition, it changes its state to that of a temporary candidate, generates a partition epoch, and votes for itself. The temporary candidate then sends a temporary vote request to the other nodes in the minority partition. The temporary vote request includes the temporary candidate's node identifier, partition epoch, and log status. Upon receiving the temporary vote request, other nodes decide whether to vote for the temporary candidate based on the partition epoch and log status. If the temporary candidate receives votes from more than half of the nodes in the minority partition, it becomes the partition leader. The partition leader then begins sending partition heartbeat messages to the other nodes in the minority partition to maintain its temporary leadership position.

[0102] In this embodiment, the intent registry is an in-memory data structure that stores information about all registered operations. Upon system startup, it scans all available operation implementations and automatically registers them to the intent registry. Operations can also be manually registered via a configuration file. Each operation requires a unique operation type identifier during registration to distinguish between different operations.

[0103] In this embodiment, the term of office of the partition leader is the same as that of the global leader, both being 5 seconds. The partition leader maintains its leadership by periodically sending partition heartbeat messages to other nodes within the partition, with a heartbeat interval of 1 second. If more than half of the nodes in the partition do not receive a heartbeat message from the partition leader within three consecutive heartbeat cycles, the partition leader is considered to have failed, and a new temporary election is initiated. The newly elected partition leader inherits all pending intentions of the original partition leader and continues to process business requests.

[0104] In this embodiment, the purpose of the pre-state extraction function is to extract the pre-state information required to perform the operation from the current cluster state. The pre-state information can be the resource version number, vector clock, or other information that can identify the resource state. The return value of the pre-state extraction function is recorded in the expected pre-state field of the operation intent object for subsequent conflict detection.

[0105] In this embodiment, the positive operation execution function performs the actual cluster management operations. The positive operation execution function receives the request parameters from the operation intent object as input, performs the corresponding operations, and returns the operation result. If an error occurs during execution, the positive operation execution function will return an error message, and the state of the operation intent object will be marked as execution failed.

[0106] In this embodiment, the compensation operation function is used to roll back the result of the positive operation. The compensation operation function receives the operation intent object as input, performs the corresponding rollback operation, and returns the operation result. The compensation operation function should be able to restore the cluster resources to the state before the positive operation was executed. If the compensation operation function fails, the system will record the error information and notify the operations and maintenance personnel for manual handling.

[0107] In this embodiment, the intent log employs an append-only writing method, improving write performance. The intent log is divided into multiple segments, each of fixed size. When a segment is full, a new segment is created. Old segments can be archived or deleted to free up storage space. Intent log writing is atomic, ensuring that log corruption does not occur even in the event of a system crash.

[0108] In this embodiment, the global leader periodically pushes complete intent logs asynchronously to some follower nodes. The push frequency can be adjusted through configuration parameters. After receiving the complete intent logs, the follower nodes store them locally for backup and recovery. If the global leader fails, the newly elected global leader can obtain the complete intent logs from the follower nodes and restore the system state.

[0109] In this embodiment, the vector clock is a mapping table consisting of node identifiers and logical clocks. Each node increments its corresponding logical clock when performing an operation. When nodes exchange messages, they carry their own vector clocks. The receiver merges its own vector clock with the vector clock in the message to update its own vector clock.

[0110] In this embodiment, the global leader and the partition leader independently maintain their respective vector clocks. The vector clock maintained by the global leader includes the logical clocks of all nodes in the cluster; the vector clock maintained by the partition leader only includes the logical clocks of nodes within a minority partition. When a network partition heals, the global leader merges the vector clocks by "taking the maximum value of each dimension" when merging pending intentions. Specifically, for each node ID in the vector clocks, the global leader compares its maintained logical clock with the logical clocks in the pending intentions and takes the larger value as the new logical clock.

[0111] In this embodiment, the comparison rules for vector clocks are as follows: For two vector clocks A and B, if for all node identifiers, the logical clock in A is less than or equal to the logical clock in B, and there exists at least one node identifier such that the logical clock in A is less than the logical clock in B, then vector clock A is less than vector clock B, indicating that event A occurred before event B. If for all node identifiers, the logical clock in A is greater than or equal to the logical clock in B, and there exists at least one node identifier such that the logical clock in A is greater than the logical clock in B, then vector clock A is greater than vector clock B, indicating that event A occurred after event B. If vector clock A is neither less than nor greater than vector clock B, then vector clocks A and B are concurrent, indicating that events A and B occurred concurrently.

[0112] In this embodiment, the specific process of conflict detection is as follows: After receiving a pending intent, the global leader first extracts the target resource identifier and expected prior state of the pending intent. Then, the global leader queries the global intent log to obtain all committed operations for the target resource during the network partitioning period. If no committed operations are found, the pending intent is accepted directly. If a committed operation is found, the vector clock of the latest committed operation is obtained. The vector clock of the pending intent is compared with the vector clock of the latest committed operation. If the vector clock of the pending intent is less than the vector clock of the latest committed operation, it indicates that the state on which the pending intent is based has been updated, resulting in a break in causality, and is therefore determined to be a conflict. If the vector clock of the pending intent is greater than the vector clock of the latest committed operation, it indicates that the pending intent occurred after the committed operation, there is no conflict, and it can be accepted directly. If the vector clock of the pending intent and the vector clock of the latest committed operation are concurrent, it is necessary to further determine whether the semantics of the two operations are interchangeable. If the semantics of the operations are not interchangeable, it is determined to be a conflict. If the semantics of the operations are interchangeable, there is no conflict, and it can be accepted directly.

[0113] In this embodiment, the comparison of vector clock sizes follows these rules: For two vector clocks A and B, if for all node identifiers, the logical clock in A is less than or equal to the logical clock in B, and there exists at least one node identifier such that the logical clock in A is less than the logical clock in B, then vector clock A is less than vector clock B. If vector clock A is neither less than nor greater than vector clock B, then the two are concurrent.

[0114] In this embodiment, the commutativity of operation semantics is defined during operation registration. The system provides a commutativity rule table to define the commutativity relationships between different operation types. The commutativity rule table is a two-dimensional matrix, with rows and columns corresponding to different operation types. Elements in the matrix indicate whether two corresponding operations are commutative. When performing conflict detection, the system queries the commutativity rule table to determine whether the semantics of the pending intent and the submitted operation are commutative.

[0115] In this embodiment, the compensation operation is executed as follows: After determining that a pending intent has a conflict, the global leader retrieves the compensation operation function corresponding to that operation type from the intent registry center. Then, the global leader passes the pending intent object as a parameter to the compensation operation function and performs a rollback operation. After the compensation operation function completes its execution, it returns the execution result. If the compensation operation is successful, the global leader records the compensation operation as a submitted intent and adds it to the global intent log. If the compensation operation fails, the global leader records the error information and sends an alarm notification to the operations and maintenance personnel.

[0116] In this embodiment, the self-learning module continuously monitors the system's operational status, collecting execution and conflict information for various operations. The self-learning module maintains a statistical database storing the execution count, conflict count, and conflict frequency of various operations. The data in the statistical database is periodically summarized and analyzed. When the conflict frequency of a certain type of operation exceeds a preset threshold for multiple consecutive statistical periods, the self-learning module automatically adjusts the execution permission of that operation, marking it as only allowed to execute within the majority partition. The self-learning module also periodically evaluates the execution permission of operations; if the conflict frequency of a certain type of operation is below the preset threshold for multiple consecutive statistical periods, it may consider remarking it as allowed to execute within the minority partition.

[0117] In this embodiment, the system provides comprehensive monitoring and alarm functions. Operations personnel can view the controller cluster's operational status, node liveness, network partitioning, operation execution, and conflict status through the monitoring interface. The system will send alarm notifications to operations personnel when a network partition occurs, a new leader is elected, a conflict is detected, or a compensation operation fails. Alarm notifications can be sent in various ways, such as via email, SMS, or instant messaging.

[0118] In this embodiment, the system supports dynamically scaling the controller cluster. Operations personnel can add or remove controller nodes during system operation. When a new controller node is added, the new node automatically synchronizes the cluster state and intent logs from existing nodes, and then joins the cluster to participate in elections and services. When a controller node is removed, the system automatically updates the node list and adjusts the majority threshold for elections.

[0119] In this embodiment, the system supports data backup and recovery functions. Maintenance personnel can periodically back up the intent logs and configuration information of the controller nodes. In the event of a system failure or data loss, the system state can be restored using the backup data. The backup data can be stored on a local disk or a remote storage system.

[0120] In this embodiment, the system provides command-line tools and an application programming interface (API) to facilitate management and operation by maintenance personnel. Maintenance personnel can use the command-line tools to view cluster status, manage nodes, configure system parameters, and perform backup and restore operations. The API allows third-party systems to integrate with the controller cluster, enabling automated management.

[0121] In this embodiment, the system adopts a modular architecture, with each functional module operating independently, facilitating maintenance and expansion. The election module, intent processing module, conflict detection module, and compensation execution module can all be upgraded and replaced independently without affecting the functionality of other modules. The system also provides extension interfaces, allowing developers to add new operation types and custom conflict handling logic.

[0122] In this embodiment, the system is designed with security in mind. Communication between controller nodes is encrypted to prevent data eavesdropping or tampering. The system supports authentication and authorization management; only authorized users can perform cluster management operations. The system records all operation logs for auditing and troubleshooting.

[0123] In this embodiment, the system demonstrates good performance in cross-regional deployment scenarios. Because the temporary election mechanism allows minority partitions to continue providing services, even if the cross-regional network is interrupted, users in each region can still continue to use the local cluster management service. Once the network is restored, the system can automatically merge the operations of each partition, restoring a globally consistent state.

[0124] In this embodiment, the system also exhibits good scalability in large-scale cluster management scenarios. Since the intent log only records operation metadata, rather than the complete cluster state, the log size does not increase dramatically with the cluster size. The overhead of conflict detection and compensation operations is also relatively small and does not significantly impact system performance.

[0125] In this embodiment, the system setup follows the principles of simplicity and practicality. No overly complex mechanisms or algorithms are introduced; all settings are designed to solve real-world problems. System deployment and maintenance are relatively simple, requiring no specialized knowledge or skills.

[0126] In the embodiments of this application, all technical aspects of the system have undergone thorough verification and testing. Extensive testing has been conducted in both laboratory and actual production environments to verify the system's correctness, usability, and performance. The system has been applied to actual products, providing users with stable and reliable cluster management services.

[0127] In this embodiment, business operations are abstracted into intent objects, and a compensation operation is associated with each intent object, which is a different approach from traditional methods. Traditional high-availability solutions typically focus on ensuring consistency before operation execution, while this application focuses on resolving potential inconsistencies after operation execution. This shift in approach allows the system to continue providing services during network partitions while ensuring eventual consistency.

[0128] In this application's embodiments, the election process is divided into two independent processes: a global election and a temporary election. This is also a feature that contributes to the prior art. Traditional consensus algorithms only have one global election process, which cannot handle the service problem of minority partitions during network partitioning. This application introduces a temporary election process, providing service capabilities for minority partitions without compromising global consistency.

[0129] In this embodiment, a vector clock is used for conflict detection, combined with the commutativity judgment of operation semantics, which improves the accuracy of conflict detection. Traditional conflict detection methods are either too strict, causing many operations that could be merged to be rejected, or too lenient, causing conflicts to go undetected. The method in this application can accurately identify genuine conflicts while accepting as many mergeable operations as possible, thus improving system availability.

[0130] In this embodiment, a self-learning mechanism is introduced to dynamically adjust the execution permissions of operations, enabling the system to automatically optimize based on actual operating conditions. Traditional solutions typically employ static configuration methods, which cannot adapt to constantly changing business needs and operating environments. The self-learning mechanism of this application can automatically identify operations prone to conflict and adjust their execution permissions, thereby reducing subsequent conflicts and rollbacks and improving system operating efficiency.

[0131] In this embodiment, the various technical aspects are closely coordinated. The intent object and compensation mechanism are prerequisites for ad hoc election and optimistic execution. Without a compensation mechanism, operations executed during partitioning that conflict cannot be automatically rolled back, leading to inconsistent system states. Ad hoc election and the pending mechanism are the foundation for conflict detection and compensation. Without ad hoc election, minority partitions cannot execute any operations, and there is no subsequent conflict detection and compensation issue. Conflict detection and compensation are crucial for ensuring eventual system consistency. Without conflict detection and compensation, operations executed during partitioning will result in permanently inconsistent system states. A self-learning mechanism is a means of continuous system optimization. Without a self-learning mechanism, the system cannot adjust its strategies based on actual operating conditions, potentially leading to frequent conflicts and rollbacks, impacting system performance and availability.

[0132] In this embodiment, the system is configured to fully consider various abnormal situations. For example, network partitions may last for a long time or occur frequently. The partition leader may fail during partitioning. The global leader may fail before the partition heals. Compensation operations may fail. The system has corresponding handling mechanisms to deal with these abnormal situations, ensuring that the system can operate stably and reliably.

[0133] In this embodiment, when a network partition persists for an extended period, nodes within the minority partition continue to operate, providing limited service. The partition leader periodically checks network connectivity and initiates a healing process immediately upon discovering a reconnection with the majority partition. During the duration of the network partition, pending intentions within the minority partition are persistently stored and not lost.

[0134] In this embodiment, when the partition leader fails during partitioning, the other nodes in the minority partition will detect the failure and initiate a new temporary election to elect a new partition leader. The new partition leader will retrieve the previously pending intents from the other nodes and continue processing business requests. Thus, even if the partition leader fails, the service within the minority partition will not be interrupted.

[0135] In this embodiment, when the global leader fails before partition healing, the nodes in the majority partition will initiate a new global election to elect a new global leader. The new global leader will inherit the previous global intent log and continue to process business requests. After partition healing, the partition leader will submit pending intents to the new global leader for processing.

[0136] In this embodiment, when the compensation operation fails, the system records the error information and sends an alarm notification to the operations and maintenance personnel. The personnel can then manually intervene based on the error information to resolve the conflict. The system also retains relevant operation logs and status information to facilitate troubleshooting and resolution by the operations and maintenance personnel.

[0137] In this embodiment, the system settings also consider performance optimization. For example, the intent log uses append-only writing, improving write performance. During conflict detection, the system first checks whether the target resource has been modified, avoiding unnecessary vector clock comparisons. The global leader periodically pushes the complete intent log asynchronously to follower nodes, avoiding the performance impact of synchronous pushes.

[0138] In the embodiments of this application, the system's performance can meet the needs of most cluster management scenarios. Under normal operating conditions, the system's performance is comparable to that of a traditional single-controller system. During network partitioning, the performance within a minority partition is also sufficient to meet the needs of local services. After partition healing, the overhead of conflict detection and compensation operations is relatively small and will not significantly impact system performance.

[0139] In this embodiment, system availability is significantly improved. Traditional high-availability solutions, during network partitioning, only a majority of partitions can continue providing service, while a minority of partitions completely cease service. This application, through a temporary election mechanism, allows even minority partitions to continue providing limited service, greatly improving overall system availability. This advantage is particularly pronounced in cross-regional deployment scenarios.

[0140] In this embodiment, system consistency is also guaranteed. Although operations within a minority partition are temporarily valid during network partitioning, the system restores to a globally consistent state after partition healing through conflict detection and compensation operations. Ultimately, all nodes will have the same cluster state.

[0141] In this embodiment, the system's operation and maintenance costs are also reduced. Traditional solutions typically require manual intervention when encountering network partitions and conflicts. This application achieves automatic problem resolution through an automatic conflict detection and compensation mechanism, eliminating the need for manual intervention. Simultaneously, the system does not require the deployment of additional external coordination services, reducing deployment and maintenance costs.

[0142] In this application's embodiments, the system can be applied to various cluster management scenarios. Besides Kubernetes cluster management, it can also be applied to other types of cluster management, such as distributed storage cluster management, distributed database cluster management, and microservice cluster management. This application's solution can be used for any controller cluster requiring high availability.

[0143] In this embodiment, the system can also be used in conjunction with other high availability technologies. For example, it can be combined with load balancing technology to automatically route client requests to available controller nodes. It can be combined with data replication technology to improve data reliability and availability. It can be combined with fault detection technology to detect node failures and network partitions more quickly.

[0144] In the embodiments of this application, the system can be implemented using various programming languages ​​and technology stacks. Different implementation methods will not affect the core ideas and scope of protection of this application. As long as the technical solution of this application is adopted, it falls within the scope of protection of this application.

[0145] In this embodiment, the various modules of the system can be deployed on the same physical server or on different physical servers. Different deployment methods will not affect the core idea and scope of protection of this application. As long as the functions of this application are implemented, they fall within the scope of protection of this application.

[0146] In the embodiments of this application, the various steps of the system can be executed in different orders or in parallel. Different execution orders will not affect the core idea and scope of protection of this application. As long as the purpose of this application is achieved, it falls within the scope of protection of this application.

[0147] As can be seen from the above embodiments, this application addresses the pain points of inevitable network partitioning in cross-regional cluster management, the forced interruption of services in minority partitions, and the need for manual intervention after healing. It proposes a solution. Under traditional consistency protocols, minority partitions cannot form a majority and completely lose service capabilities. Even if limited services are provided at the risk of failure, consistency cannot be automatically restored during healing due to the lack of before-and-after state correlation and reversible semantics. This invention recognizes that if operations are encapsulated as intent objects carrying expected pre-states and compensation operation definitions, and a partition leader is temporarily elected using an independent partition epoch number, the business requests of minority partitions can be transformed into traceable and revocable pending intents. This maintains partition availability while providing a basis for subsequent healing. After partition healing, the global leader uses vector clock comparison to identify causal relationship breaks and, combined with the commutativity judgment of operation semantics, accurately distinguishes between mergeable and non-mergeable conflicts, automatically invoking pre-registered compensation operations for rollback in case of conflicts. This combination makes "optimistic execution" no longer dependent on external coordination services or physical clocks, avoiding unnecessary service interruptions and eliminating the burden of manual repair. Furthermore, by continuously tracking the conflict frequency of various operations and dynamically marking high-conflict operations as strict majority operations, the system achieves an adaptive availability-consistency trade-off, which is something that existing master-slave switching, external coordination services, or simple optimistic control have failed to achieve.

[0148] Furthermore, embodiments of this application also provide a cluster management controller system, including multiple controller nodes, wherein the controller nodes are configured as described above.

[0149] Those skilled in the art will understand that the embodiments of this application are not limited to the specific examples described above. Without departing from the spirit and scope of this application, those skilled in the art can make various modifications and variations to the technical solutions of this application. For example, the technical solutions of this application can be applied to other types of software authentication scenarios, or equivalent substitutions can be made to the technical features of this application. These modifications and variations should all fall within the protection scope of this application. Further details are omitted here.

Claims

1. A method for high-availability service election of a cluster management controller, characterized in that, include: When a controller node detects that it is in a minority partition of a network partition, it initiates a temporary election process independent of the global term to produce a partition leader. The partition leader receives a business request, encapsulates the business request into an operation intent object, executes the operation, and marks the operation result as a pending intent. The operation intent object contains the compensation operation definition and expected pre-state of the operation. After the minority partition and the majority partition are reconnected, the global leader in the majority partition performs conflict detection based on the pending intent and the submitted intent, performs pre-registered compensation operations on the detected conflicts, restores the consistency of the cluster management state, and achieves high availability service election. The conflict detection includes: receiving the pending intent by the global leader, extracting its target resource identifier and expected pre-state; obtaining the global committed intent log during partitioning; and identifying conflicts using vector clock comparison based on the expected pre-state and the committed intent log. The method of identifying conflicts by comparing vector clocks includes: determining whether there is a break in causal relationship based on the vector clock of the pending intent and the vector clock of the submitted operation corresponding to the target resource in the submitted intent log; if so, it is determined to be an unmergeable conflict. The initiation of a temporary election process independent of the overall term includes: An election using an independent partition epoch is initiated among the nodes within the minority partition. The node that receives the votes of a majority of the nodes within the minority partition becomes the partition leader. The partition epoch number is independent of the global term of office, so that after the partition is healed, the partition leader is demoted to a follower without interfering with the global leader's term of office. The process of initiating an election using an independent partition epoch number includes: generating a partition epoch number by appending a partition identifier to the current global term, and sending a voting request carrying only the partition epoch number to other nodes within the partition.

2. The method of claim 1, wherein, The operation intent object also contains a globally unique identifier for the operation; the method further includes: before the partition leader executes the operation, determining whether the operation is a preset partitionable operation based on the operation type in the operation intent object; if so, the operation is executed; otherwise, the request is rejected.

3. The method of claim 1, wherein, Marking the operation result as a pending intent includes: recording the pending intent in the local intent log of the partition leader, and synchronizing the pending intent to other nodes in the minority partition through the partition's internal heartbeat.

4. The method according to claim 1, characterized in that, The conflict detection further includes: before using vector clock comparison, the global leader determines whether the target resource of the pending intent has a corresponding operation in the submitted intent log. If it does not exist, the pending intent is directly accepted as a submitted intent.

5. The method of claim 1, wherein, The execution of the pre-registered compensation operation includes: calling the compensation function indicated by the compensation operation definition contained in the pending intent, and recording the compensation operation as a submitted intent.

6. The method of claim 1, wherein, Also includes: Statistical analysis of conflict frequency for various operations after multiple partition healing processes; When the conflict frequency of a certain type of operation exceeds a threshold, the operation is marked as a strict majority operation and will be directly rejected by the partition leader in the minority partition.

7. The method of claim 1, wherein, The method of using vector clock comparison to identify conflicts further includes: if the vector clock of the pending intent has a concurrent relationship with the vector clock of the corresponding operation in the submitted intent log, and the operation semantics cannot be exchanged, then it is determined to be an unmergeable conflict.

8. A cluster management controller system, characterized by It includes multiple controller nodes, which are configured to perform the method according to any one of claims 1 to 7.

Images