A method and system for fault migration of industrial cluster nodes

By deploying multiple failover controllers and local fault handling services in an industrial cluster, node failures are determined using health probes, network probes, and majority arbitration. Node action locks are used to record migration status and local network isolation, thus solving the migration uncertainty problem caused by a single management process and achieving highly available failover and business continuity.

CN123123130APending Publication Date: 2026-11-17SUPCON TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611535469.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-30
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, industrial cluster node failure migration relies on a single management process, which leads to the loss of migration capability when the management process is abnormal or the master node crashes. It cannot take over unfinished migration actions, which can easily result in repeated migrations or incomplete states. Furthermore, the lack of cross-verification of multiple observation signals may lead to erroneous migrations or migration delays. After the faulty node recovers, there is a risk of address conflicts between old and new business instances.

Method used

In an industrial cluster, multiple failover controllers and local fault handling services are deployed. Node failures are determined through dual confirmation of health probes and network probes, as well as majority arbitration. Node action locks are used to record the migration status to ensure the integrity of the migration process. Network isolation is performed locally on the faulty node, and the isolation status is recorded to avoid address conflicts.

Benefits of technology

It effectively filters short-term network jitter and one-sided observation bias, avoids repeated migration and concurrent double writes, reduces the risk of address conflicts between old and new service instances, and ensures determinism and business continuity in industrial communications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123130A_ABST
    Figure CN123123130A_ABST
Patent Text Reader

Abstract

This application relates to the field of high availability technology for industrial clusters, and provides a method and system for fault migration of industrial cluster nodes. The method includes: the main controller performs health probes and network probes on the target node; when both probes confirm unreachability and the failure confirmation window is reached, the main controller requests observation from the auxiliary controller; when more than half of the observation results confirm anomalies, the target node is determined to be faulty; node action locks are acquired at the target node level, and the fault migration stage, completed migration scope, and takeover status are recorded; based on the node action locks, migrateable service instances to healthy nodes, so that after the main controller switchover, the new main controller takes over the incomplete migration actions; after the local fault handling service detects the node failure, it performs local network isolation on the old service instances and records the isolation status; after the target node recovers and becomes ready, the main controller re-enables the target node to participate in scheduling based on the isolation status. This application improves the availability of the fault migration control plane and reduces the risk of erroneous migration and address conflicts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of high availability technology for industrial clusters, and in particular to a method and system for fault migration of industrial cluster nodes. Background Technology

[0002] As industrial control systems gradually adopt cloud-native architectures, control, data acquisition, and general platform services are beginning to run on container orchestration clusters. Industrial environments have high requirements for business continuity, network address stability, and deterministic fault recovery. When a single compute node experiences a network failure, power outage, or hardware failure, it is necessary to migrate migrateable services from the failed node to a healthy node within a short confirmation window, and to avoid old and new service instances simultaneously occupying industrial communication addresses after the failed node recovers.

[0003] In existing technologies, cluster node failure migration typically relies on a single management process for fault diagnosis and migration scheduling. However, this approach loses its migration capability when the management process itself malfunctions or the master node crashes, leaving unfinished migration actions unmanaged and prone to duplicate migrations or incomplete migrations. Furthermore, existing solutions lack a cross-validation mechanism using multiple observation signals in the fault confirmation phase, making it difficult to distinguish between short-term network jitter and genuine node failures, potentially leading to erroneous migrations or migration delays. Moreover, after a failed node recovers, the communication addresses of old service instances are not effectively isolated, posing a risk of address conflicts with new service instances and impacting the determinism of industrial communication.

[0004] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for fault migration of industrial cluster nodes.

[0006] In a first aspect, the present invention provides a method for fault migration of nodes in an industrial cluster. Multiple fault migration controllers are deployed on multiple master control nodes of the industrial cluster, and local fault handling services are deployed on each computing node. The multiple fault migration controllers include a master controller and auxiliary controllers. The technical solution of this method is as follows: The main controller performs health detection and network detection on the target node. When both the health detection and the network detection confirm that the target node is unreachable and continues to reach a preset fault confirmation window, it requests the auxiliary controller to perform auxiliary observation on the target node. When the combined observation results of the main controller and the auxiliary controller exceed half of the confirmed anomalies, the target node is determined to be faulty. The main controller acquires node action locks at the granularity of the target node. The node action locks record the stage of the fault migration, the scope of the completed migration, and the takeover status. The main controller migrates the migrateable service instances on the target node to healthy nodes according to the node action locks, so that after the main controller is switched over, the new main controller takes over the incomplete migration actions according to the node action locks. After detecting a fault in the local node, the local fault handling service performs local network isolation on the old service instances on the local node and records the isolation status; after the target node recovers and becomes ready, the main controller enables the target node to re-participate in scheduling based on the isolation status.

[0007] The above solution employs a multi-controller collaborative fault confirmation mechanism, utilizing dual confirmation through health probes and network probes, along with majority arbitration (more than half of the votes), to filter out short-term network jitter and one-sided observation biases in fault determination. By recording the migration stage and completed migration range using node action locks at the faulty node level, the new master controller can take over unfinished migration actions after a master controller switchover, avoiding duplicate migrations and concurrent double writes. Furthermore, by performing network isolation locally on the faulty node and recording the isolation status, the risk of address conflicts between old and new service instances is reduced after the node recovers.

[0008] Secondly, the present invention provides an industrial cluster node fault migration system, wherein multiple fault migration controllers are deployed on multiple master control nodes of the system, and local fault handling services are deployed on each computing node. The multiple fault migration controllers include a master controller and an auxiliary controller. The technical solution of the system is as follows: The main controller is used to perform health detection and network detection on the target node. When both the health detection and the network detection confirm that the target node is unreachable and continues to reach a preset fault confirmation window, it requests the auxiliary controller to perform auxiliary observation on the target node. When the observation results of the main controller and the auxiliary controller together confirm an anomaly by more than half, the target node is determined to be faulty. Node action locks are obtained at the granularity of the target node. The node action locks record the stage of fault migration, the range of migration completed, and the takeover status. The main controller is also used to migrate the migrateable service instances on the target node to a healthy node according to the node action lock, so that after the main controller is switched, the new main controller takes over the unfinished migration action according to the node action lock. After detecting a fault in the local node, the local fault handling service performs local network isolation on the old service instances on the local node and records the isolation status; after the target node recovers and becomes ready, the main controller enables the target node to re-participate in scheduling based on the isolation status.

[0009] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0010] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of the industrial cluster node fault migration method according to an embodiment of this application; Figure 2 This is a schematic diagram of the industrial cluster node fault migration system architecture according to an embodiment of this application; Figure 3 This is a flowchart of the node fault confirmation and arbitration process according to an embodiment of this application; Figure 4 This is a timing diagram of the service migration of the fault node in an embodiment of this application; Figure 5 This is a timing diagram of node action lock takeover after a main controller failure according to an embodiment of this application; Figure 6 This is a flowchart illustrating the local network isolation process performed by the local fault handling service in this application embodiment; Figure 7 This is a flowchart of the fault node recovery process in an embodiment of this application. Detailed Implementation

[0011] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0012] Example 1: like Figure 1 As shown, this embodiment provides a method for fault migration of industrial cluster nodes. Figure 2 As shown, multiple failover controllers are deployed on multiple master control nodes of the industrial cluster, and local fault handling services are deployed on each computing node. The multiple failover controllers include master controllers and auxiliary controllers.

[0013] Specifically, an industrial cluster refers to a container orchestration cluster designed for industrial control, data acquisition, and general business operations, typically comprising multiple master control nodes and multiple compute nodes. A fault migration controller is deployed as an independent process on the master control node, responsible for fault confirmation, business migration, and recovery processing at the cluster level. Local fault handling services are deployed as system services on each compute node, responsible for fault detection and local network isolation for that node. In this embodiment, the fault migration controller is denoted as nyx-fault-controller in the accompanying drawings, where the instance running as the master controller corresponds to the Leader in the drawings, and the instance running as an auxiliary controller corresponds to the Follower in the drawings; the local fault handling service is denoted as processor in the accompanying drawings. Furthermore, a status acquisition service, denoted as node-supervisor in the accompanying drawings, can also be deployed on the compute nodes to collect the node's operating status for querying by the master controller. Multiple fault migration controllers elect a master through a distributed lease mechanism; at any given time, only one fault migration controller acts as the master controller to perform cluster write operations, while the remaining fault migration controllers run as auxiliary controllers. For example, an industrial cluster may include three master nodes, each with a failover controller deployed on it. The three failover controllers use leased resources to elect a master and perform auxiliary observation. It should be understood that although this embodiment uses three master nodes as an example, in other embodiments, the number of master nodes may be five or more, as long as the majority arbitration requirement is met.

[0014] like Figure 1 As shown, the method includes the following steps.

[0015] In step S100, the main controller performs health detection and network detection on the target node. When both health detection and network detection confirm that the target node is unreachable and the preset fault confirmation window is reached, the main controller requests the auxiliary controller to perform auxiliary observation on the target node. When the combined observation results of the main controller and the auxiliary controller exceed half of the confirmed abnormalities, the target node is determined to be faulty.

[0016] Specifically, the target node refers to the computing node in the industrial cluster that needs to be monitored for failures; it can also be called a faulty node. Health probing involves accessing the node agent of the target node to confirm whether the node agent is responding normally. The node agent, for example, is the kubelet process running on the node. Network probing involves sending network probe packets to the target node to confirm its network reachability. Network probe packets, for example, are ICMP probe packets. The fault confirmation window is a pre-set continuous observation time used to filter short-term network jitter. This fault confirmation window can be configured according to the network latency and service recovery requirements of the industrial site, preferably 10 seconds.

[0017] When the main controller initiates the fault confirmation process, it first obtains the node status and lease status of the target node. Only when the node status is ready and the lease status is normal will it perform health probes and network probes. If both health probes and network probes confirm that the target node is unreachable, meaning the node agent is inaccessible and the network probe packets are unreachable, the target node is determined to be unreachable only if both fail simultaneously. This design is to prevent misjudgments caused by biases in one-sided observations, such as abnormal node agent processes despite network connectivity or temporary network congestion despite normal node agent operation. When the unreachable state of the target node persists to the fault confirmation window, the main controller requests the auxiliary controller to perform auxiliary observations on the same target node. The auxiliary observations performed by the auxiliary controller are the same as those for health probes and network probes. The main controller aggregates its own probe results and the observation results of the auxiliary controllers, counting the total number of probe and observation results confirming target node abnormality across multiple fault migration controllers. When this number exceeds half of the total, the target node is determined to be faulty. For example, in the case of three fault migration controllers, if the primary controller and at least one auxiliary controller both confirm an anomaly in the target node, then the number of confirmed anomalies is two, which exceeds half of the total of three, and the target node is determined to be faulty. Through majority arbitration, fault determination does not depend on the observation results of a single controller, reducing the impact of one-sided observation bias on fault confirmation.

[0018] In step S200, the main controller acquires node action locks at the granularity of the target node. The node action locks record the stage of the fault migration, the scope of the completed migration, and the takeover status. The main controller migrates the migrateable service instances on the target node to healthy nodes according to the node action locks, so that after the main controller is switched over, the new main controller takes over the unfinished migration actions according to the node action locks.

[0019] Specifically, a node action lock is a data object at the target node level, used to indicate whether a fault migration controller is already processing the target node and to record the fault migration controller holding the lock. The node action lock records the stage of the fault migration, the scope of completed migrations, and the takeover status. The stage of the fault migration refers to the current stage of the migration action, such as migration start, service eviction, service rebuilding, or migration completion. The scope of completed migrations refers to the set of service instances that have completed migration, such as a list of service identifiers. The takeover status indicates whether the node action lock can be taken over by a new master controller, such as lock holding, takeover possible, or completed. The node action lock is carried by resources with consistent update capabilities in the industrial cluster, such as Kubernetes native objects or custom resources. Only the fault migration controller holding the node action lock can perform fault migration actions on the target node. By setting node action locks at the target node level and limiting the execution of fault migration actions to only the lock holder, migration actions on the same fault node (i.e., the target node) remain mutually exclusive, avoiding migration conflicts caused by multiple controller instances concurrently operating on the same node.

[0020] After acquiring the node action lock, the master controller migrates the migrateable service instances on the target node to healthy nodes based on the node action lock. A migrateable service instance refers to a service instance on the target node that needs to be migrated to a healthy node for re-running. A healthy node is a computing node in the industrial cluster that is in normal operating condition and capable of supporting the migrated services. During the migration process, the master controller continuously records the migrated service instances in the completed migration range of the node action lock. When the master controller switches, for example, if the master node where the master controller resides crashes or the master controller process exits abnormally, multiple fault migration controllers re-elect a master through a distributed lease, with auxiliary controllers competing for the distributed lease to become the new master controller. The new master controller does not inherit the memory state of the original master controller; instead, it reconstructs the node state, service state, scheduling constraint state, and node action lock state of the target node from the industrial cluster's interface services. Then, based on the fault migration stage and completed migration range recorded by the node action lock, it skips the migrated service instances and continues executing the unfinished migration actions. The rationale behind this setup is that the original master controller may have crashed, and its memory state is unreliable. The state is rebuilt from the interface service and the takeover is based on the node action lock, so that the migration action can continue idempotently after the master controller is switched, reducing the risk of repeated evictions and incomplete states.

[0021] In step S300, after the local fault handling service detects a fault in the local node, it performs local network isolation on the old service instances on the local node and records the isolation status; after the target node recovers and is ready, the main controller enables the target node to re-participate in scheduling based on the isolation status.

[0022] Specifically, the local fault handling service runs locally on the compute node and can detect faults on the node even when it experiences network failures, power outages, or hardware failures while the operating system remains operational. The local fault handling service detects faults based on the network status of the node's network interface card (NIC), disk status, memory status, and heartbeat connection with the main controller. When preset local isolation trigger conditions are met, the local fault handling service obtains the containers and service instances on the node through the container runtime interface, identifying the old service instance. The old service instance refers to the service instance that was already running on the node before the fault occurred. The local fault handling service enters the network namespace of the old service instance through the container runtime interface and sets the container NIC of the old service instance to an unavailable state, for example, setting the container NIC to a down state. This setting is based on the consideration that dedicated networks carrying control, acquisition, or I / O service communications in industrial scenarios have high requirements for the stability of industrial communication addresses (also referred to as communication addresses). If the old service instance still occupies the original communication address after the faulty node recovers, it may conflict with the address of the new service instance migrated to the healthy node. Network isolation is performed locally rather than remotely by the main controller because the local fault handling service can still operate when a node fails, while the main controller may be unable to access the node due to network failure. Setting the container's network interface card to an unavailable state instead of deleting the container is to preserve the container for subsequent recovery and status confirmation. The local fault handling service records the isolation status to the local storage of this node. The isolation status includes the isolation round, the service instance identifier, the isolation result, and the reason for the exception. The isolation round indicates the number of times the isolation action was executed, the service instance identifier identifies the isolated service instance, the isolation result indicates whether the isolation was successful, and the reason for the exception records the reason for isolation failure. By recording the isolation status to local storage, the isolation action can be idempotently recovered after the local fault handling service restarts.

[0023] After the target node recovers and becomes ready, the main controller retains the unschedulable flag of the target node. While the target node remains ready for a preset recovery stabilization window, the main controller queries the local fault handling service of the target node via a private protocol over the network connection to check its isolation status. If there are unresolved fault records in the isolation status, the unschedulable flag is retained and confirmation is awaited. If there are no unresolved fault records in the isolation status, the old service remnants on the target node are cleaned up and the unschedulable flag is removed, allowing the target node to rejoin scheduling. By setting a recovery stabilization window, the node is prevented from immediately resuming service operations before its status has stabilized. By querying the isolation status of the local fault handling service, the node recovery decision is based on the actual local isolation situation rather than solely on the node's ready status, reducing the risk of address conflicts caused by re-including the node in scheduling while the isolation is not yet resolved.

[0024] This embodiment establishes a complete closed loop from fault confirmation, majority arbitration, node action lock acquisition, service migration, local network isolation to node recovery through the above scheme. The fault confirmation stage uses dual confirmation through health probes and network probes, along with majority arbitration, to filter out short-term network jitter and one-sided observation biases. The migration stage records the migration phase and completed migration range using node action locks at the target node level, enabling the new master controller to take over unfinished migration actions after a master controller switchover, avoiding duplicate migrations and concurrent double writes. The recovery stage reduces the risk of address conflicts between old and new service instances after node recovery by performing network isolation locally on the faulty node and recording the isolation status.

[0025] Example 2: like Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed explanation of the fault confirmation and majority arbitration implementation. The difference between this embodiment and Embodiment 1 lies in the further definition of the preconditions, detection methods, unreachability determination rules, and statistical logic for majority arbitration for health detection and network detection. The above steps are further explanations of step S100 in Embodiment 1.

[0026] In step S110, the main controller obtains the node status and lease status of the target node. When the node status is ready and the lease status is normal, it performs health detection and network detection.

[0027] Specifically, node status refers to the operational status reported by the target node in the industrial cluster, such as the Ready state of a Node object in a Kubernetes cluster. Lease status refers to the liveness status maintained between the target node and the industrial cluster through a lease mechanism, such as the renewal status of a Node Lease object in a Kubernetes cluster. When the main controller initiates the fault confirmation process, it first reads the node status and lease status of the target node from the interface service of the industrial cluster. Only when the node status is Ready and the lease status is normal will subsequent health probes and network probes be executed. The consideration behind this setting is that if the node status itself is in a non-Ready state or the lease status is abnormal, it indicates that the target node may already be in a known abnormal state. At this time, the significance of performing probes is limited, and misjudgments may be introduced due to the abnormality of the node status itself. By initiating probes only when the node status is Ready and the lease status is normal, the probe process is based on the baseline of the node's original normal operation, and changes in subsequent probe results are more likely to reflect the actual occurrence of faults.

[0028] Among them, health detection involves accessing the node proxy of the target node, and network detection involves sending network detection messages to the target node.

[0029] Specifically, a node proxy refers to a proxy process running on a target node that manages the lifecycle of business instances on that node, such as the kubelet process in a Kubernetes cluster. Health probes confirm whether the node proxy is responding normally by accessing its status interface. If the node proxy is inaccessible, it indicates that the node proxy process on the target node may be malfunctioning or the host is unavailable. Network probes send network probe packets to the target node to confirm its network reachability. Network probe packets, such as ICMP probe packets, are used. If the network probe packet is unreachable, it indicates that the network link connecting the target node may be interrupted. Health probes and network probes observe the reachability of the target node from the node proxy level and the network link level, respectively. Because they observe different dimensions, they complement each other.

[0030] Step S120: When the node agent is inaccessible and the network probe message is unreachable and continues to reach the fault confirmation window, the target node is confirmed to be unreachable; when the target node becomes reachable within the fault confirmation window, the target node is determined to be experiencing network jitter, and the node status and lease status are reacquired.

[0031] Specifically, the fault confirmation window refers to a pre-set continuous observation period used to filter short-term network jitter. This window can be configured according to the network latency and service recovery requirements of the industrial site, preferably 10 seconds. The main controller only confirms that the target node is unreachable when both the node agent and network probe packets are unreachable, and these two probe results consistently reach the fault confirmation window. This design is based on the consideration that if unreachability is determined solely by a single observation result, misjudgments may occur due to biases such as an abnormal node agent process while the network remains reachable, or temporary network congestion while the node agent remains normal. For example, if the node agent process is temporarily unresponsive due to an internal anomaly but the network link remains open, determining unreachability solely based on a failed health probe might misjudge a node that can still support services as faulty; conversely, if the network link is temporarily congested but the node agent is still operating normally, determining unreachability solely based on a failed network probe could also lead to misjudgment. Therefore, using a method where both probes fail simultaneously before determining unreachability effectively avoids the impact of single-sided observation bias on fault confirmation.

[0032] When the target node becomes reachable again within the fault confirmation window (i.e., at least one of the health probe and network probe returns to normal within the fault confirmation window), the main controller determines that the target node experienced network jitter, does not trigger the migration process, and reacquires the node status and lease status, returning to step S110 to restart the fault confirmation process. This design takes into account that short-term network jitter may occur in industrial environments. If the target node has become reachable again within the fault confirmation window, it indicates that the unreachability was not a genuine node failure, and no migration action is required, thereby reducing false migrations caused by short-term network jitter.

[0033] Step S130: The auxiliary controller performs the same auxiliary observation, health probe, and network probe on the target node; the number of detection and observation results confirming the target node's abnormality is counted according to the total number of multiple fault migration controllers; when the number exceeds half of the total, the target node is determined to be faulty; when the number does not exceed half of the total, the target node is determined to be suspected of being faulty, and health probe and network probe are continued to be performed on the target node, and the fault confirmation window timer is restarted when the target node is confirmed to be unreachable again.

[0034] Specifically, when the primary controller confirms that a target node is unreachable and continues to reach the fault confirmation window, it requests the secondary controller to perform auxiliary observations on the same target node. The auxiliary observations performed by the secondary controller are the same as the health probes and network probes performed by the primary controller; that is, the secondary controller also accesses the target node's node agent and sends network probe packets to the target node. After completing the auxiliary observations, the secondary controller feeds back the observation results to the primary controller. The primary controller then aggregates its own probe results and the secondary controller's observation results, and counts the total number of probe and observation results confirming the target node's anomaly, based on the total number of fault migration controllers.

[0035] The majority arbitration logic works as follows: using the total number of failover controllers as a base, the number of detection and observation results confirming the target node's anomaly is counted. When this number exceeds half of the total, the target node is deemed faulty. For example, with three failover controllers, if the primary controller and at least one auxiliary controller confirm the target node's anomaly, the number of confirmed anomalies is two, exceeding half of the total of three, thus determining the target node's fault. With five failover controllers, if the primary controller and at least two auxiliary controllers confirm the target node's anomaly, the number of confirmed anomalies is three, exceeding half of the total of five, thus determining the target node's fault. Through majority arbitration, fault determination does not rely on the observation results of a single controller. Even if the primary controller misjudges the target node's anomaly due to its own network anomalies or observation bias, the migration process will not be triggered as long as the observation results of a majority of failover controllers do not confirm the anomaly, thereby reducing the risk of misjudgment by a single controller.

[0036] When the number of confirmed anomalies does not exceed half of the total, the main controller determines the target node as a suspected fault, continues to perform health and network probes on the target node, and restarts the fault confirmation window timer if the target node is confirmed to be unreachable again. This setup is based on the consideration that if less than half of the confirmed anomalies are detected, it means the target node's fault status has not yet been confirmed by a majority of fault migration controllers. In this case, migration should not be triggered hastily, but rather observation should continue. Restarting the fault confirmation window timer when subsequent probes confirm the target node's unreachability provides continuous monitoring of the fault confirmation process, further filtering out sporadic anomalies.

[0037] This embodiment, through the above-described scheme, implements a complete process in the fault confirmation stage, including pre-probe condition verification, dual-probe cross-validation, continuous observation of the fault confirmation window, and majority arbitration. By initiating probing only when a node is in a ready state and its lease status is normal, invalid probing is avoided when the node itself is abnormal. The fault confirmation window timing only begins after both health and network probes confirm unreachability, ensuring that nodes that become reachable within the fault confirmation window are considered network jitter and do not trigger migration, thus reducing false migrations caused by short-term network jitter. By counting the number of confirmed anomalies based on the total number of fault migration controllers and conducting majority arbitration, fault determination is not dependent on the observation results of a single controller, reducing the impact of unilateral observation bias on fault confirmation. When the majority vote is less than half, the node is considered a suspected fault and the timing is restarted, providing continuous observation capabilities for the fault confirmation process and further filtering out sporadic anomalies.

[0038] Example 3: like Figure 5 As shown, this embodiment, based on the above embodiments, explains the attribute definition of the node action lock and the idempotent takeover process after the master controller switch. The difference between this embodiment and Embodiments 1 and 2 is that it further defines the data object attributes, resource types, lock holder permissions of the node action lock, and the specific process of the new master controller taking over the incomplete migration action according to the node action lock after the master controller switch. The above steps are further explanations of the takeover part after the master controller switch in step S200 of Embodiment 1.

[0039] A node action lock is a data object with the target node as the granularity. It is used to indicate whether a fault migration controller is already processing the target node and to record the fault migration controller holding the lock.

[0040] Specifically, using the target node as the granularity means that the creation and holding of node action locks are done on a per-target-node basis. Different target nodes correspond to different node action locks. A fault migration controller can hold node action locks for multiple target nodes simultaneously, but for the same target node, only one fault migration controller can hold its node action lock at any given time. The node action lock indicates whether a fault migration controller is already processing the target node. That is, when a fault migration controller determines that a target node is faulty and prepares to execute a migration action, it first acquires the node action lock for that target node. If the acquisition is successful, it means that no other fault migration controller is currently processing the target node. If the acquisition fails, it means that another fault migration controller is already processing the target node, and the current fault migration controller will not re-initiate the migration action. The node action lock also records the fault migration controller holding the lock, that is, it records the identification information of the fault migration controller currently holding the node action lock, so that when a fault migration controller switch occurs, the new master controller can identify the holder of the node action lock and whether the holder is still alive.

[0041] The node action lock is carried by resources in the industrial cluster that have consistent update capabilities.

[0042] Specifically, resources with consistent update capabilities refer to storage resources in an industrial cluster that can guarantee consistency in concurrent read and write operations on the same resource by multiple failover controllers. For example, in a Kubernetes cluster, node action locks can be carried by native Kubernetes objects, such as ConfigMaps, Leases, or custom resource definition objects. These objects ensure the atomicity of concurrent updates through the consistent storage mechanism provided by the Kubernetes interface service. In other implementations, node action locks can also be carried by other storage resources with distributed consistent update capabilities, such as distributed key-value stores based on the Raft protocol, as long as it can guarantee that only one failover controller successfully acquires the node action lock at any given time. This application does not limit the specific resource type and field names carrying the node action lock, as long as the resource can record the stage of failover, the scope of completed failover, and the takeover status at the target node level, and supports consistent updates.

[0043] A failover controller that only holds a node action lock can perform failover actions on the target node.

[0044] Specifically, fault migration actions refer to a series of migration-related operations performed on the target node, including setting an unschedulable flag, filtering migrateable service instances, calling the eviction interface, recording migration progress, and cleaning up old service remnants. By limiting fault migration controllers to only holding node action locks to perform fault migration actions on target nodes, migration actions on the same target node remain mutually exclusive, avoiding migration conflicts caused by multiple fault migration controllers concurrently operating on the same target node. For example, when the primary controller loses contact with the secondary controller due to network partitioning but is still running, the secondary controller may be elected as the new primary controller due to lease timeout and attempt to perform a migration action on the same target node. In this case, because the original primary controller still holds the node action lock, the new primary controller cannot acquire the node action lock, thus preventing duplicate migration actions on the same target node until the original primary controller's node action lock becomes takeoverable due to lease timeout or active release.

[0045] like Figure 5 As shown, after the master controller is switched, the new master controller takes over the unfinished migration actions according to the node action lock, including the following steps.

[0046] In step S210, multiple failover controllers elect a master through a distributed lease. After the master controller stops renewing its lease, the distributed lease times out, and the auxiliary controllers compete for the distributed lease to become the new master controller.

[0047] Specifically, a distributed lease refers to a lease mechanism maintained among multiple failover controllers through consistent storage resources in an industrial cluster, such as a Lease object in a Kubernetes cluster. The process of multiple failover controllers electing a master through distributed leases is as follows: each failover controller competes for the same lease resource; the failover controller that successfully acquires the lease becomes the master controller, and the remaining failover controllers become auxiliary controllers. The master controller periodically renews its lease during operation to indicate that it is still alive. When the master node hosting the master controller crashes or the master controller process exits abnormally, the master controller stops renewing its lease, and the distributed lease is released after a timeout. When an auxiliary controller detects a distributed lease timeout, it competes for the lease; the auxiliary controller that wins the competition becomes the new master controller. For example, in a scenario where three failover controllers are deployed on three master nodes, if the original master node experiences a complete power outage, the original master controller stops renewing its lease. After the distributed lease times out, two auxiliary controllers simultaneously attempt to renew the lease; the auxiliary controller that successfully renews its lease first becomes the new master controller, and the other auxiliary controller continues to operate as an auxiliary controller.

[0048] In step S220, the new master controller does not inherit the memory state of the master controller. First, it confirms that the interface service of the industrial cluster is stable and available, and then rebuilds the node state, business state, scheduling restriction state and node action lock state of the target node from the interface service.

[0049] Specifically, the new master controller does not inherit the memory state of the original master controller. This means that after becoming the master controller, the new master controller neither reads nor relies on the node state, business state, migration progress, and other information cached in memory by the original master controller during its operation. This is because the original master controller may be unable to provide a reliable memory state due to the failure of its host node or abnormal process exit. If the new master controller attempts to obtain the memory state from the original master controller, it may fail because the original master controller is no longer accessible, or it may obtain an expired state that was not updated in time before the original master controller's abnormal exit, causing the takeover process to be executed based on incorrect state information. Therefore, the new master controller rereads and reconstructs the state from the industrial cluster's interface service to ensure state consistency during the takeover process.

[0050] Before rebuilding the state, the new master controller first verifies the stability and availability of the industrial cluster's interface services. Specifically, interface services refer to services within the industrial cluster that provide resource read / write interfaces, such as the APIServer in a Kubernetes cluster. Verifying the stability and availability of interface services can be done by sending interface requests to the service and confirming that it returns data normally, such as an HTTP API returning a 200 status code. This setup is based on the consideration that master controller switching scenarios often involve control plane instability. For example, a failure of the original master controller's node may lead to increased load on interface services or timeouts for some interface requests. If the new master controller rashly performs state rebuilding and migration takeover before the interface services are stable, state rebuilding errors may occur due to failed interface requests or incomplete data returns, leading to duplicate migrations or missed migrations. By verifying the stability and availability of interface services before performing state rebuilding, the takeover process is built on a stable control plane.

[0051] The new master controller reconstructs the target node's node state, service state, scheduling constraint state, and node action lock state from the interface service. Node state refers to the target node's reported operational status within the industrial cluster, such as the Ready state and lease renewal status of a Node object in a Kubernetes cluster. Service state refers to the operational status of service instances on the target node, such as whether a service instance is in a running / ready state. Scheduling constraint state refers to the scheduling constraint flags set on the target node, such as an unschedulable flag. The node action lock state refers to the fault migration stage, completed migration scope, and takeover status recorded in the node action lock. The new master controller reads the above state information from the interface service and reconstructs a state view locally that is consistent with the target node's current actual state, serving as the basis for subsequent takeover and migration actions.

[0052] Step S230: When the fault migration stage recorded by the node action lock is completed, skip the migration action; when the node action lock can take over, continue to execute the unfinished migration actions according to the fault migration stage and the scope of completed migration.

[0053] Specifically, after rebuilding, the new master controller reads the node action lock of the target node and determines subsequent actions based on the fault migration stage and takeover status recorded in the node action lock. When the fault migration stage recorded in the node action lock is "completed," it means that the migration action has been completed within the original master controller's term. The new master controller skips the migration action and does not perform repeated migration operations on the target node. When the node action lock is "takeover allowed," it means that the migration action has not yet been completed and the node action lock allows the new master controller to take over. The new master controller determines which stage of the current migration is in based on the fault migration stage and which service instances have been migrated based on the scope of completed migrations. It then skips the migrated service instances and continues to execute the incomplete migration actions. For example, if the fault migration stage recorded in the node action lock is "service eviction," and the scope of completed migrations includes service instances A and B, then the new master controller skips service instances A and B and continues to perform eviction and rebuild on service instances C and D. By continuing to execute unfinished migration actions based on the fault migration stage and completed migration range recorded by the node action lock, the migration actions after the master controller switchover can be idempotently continued, reducing the risk of repeated evictions and incomplete states.

[0054] This embodiment uses the above-described scheme to set node action locks at the target node level and restricts the ability of only the lock holder to perform fault migration actions, ensuring that migration actions on the same target node remain mutually exclusive. Distributed lease election enables the generation of a new master controller after a switch. The new master controller does not inherit memory state and rebuilds the state from the interface service, ensuring that the takeover process does not depend on the survival state of the original master controller. By skipping completed items and continuing to execute incomplete items according to the fault migration stage and completed migration range recorded by the node action lock, the migration action can be idempotently continued after a master controller switch, reducing the risks of repeated evictions, concurrent writes, and half-complete states.

[0055] Example 4: like Figure 6 As shown, this embodiment, based on the above embodiments, describes the specific implementation of local network isolation for the local fault handling service. The difference between this embodiment and the previous embodiments lies in that it further defines the local state information upon which the local fault handling service perceives a fault in this node, the method for identifying old service instances, the specific technical means of local network isolation, and the recorded content of the isolation status. The above steps are further explanations of the local network isolation part in step S300 of Embodiment 1.

[0056] Step S310: The local fault handling service detects faults in this node based on the network status, disk status, memory status of the local node's network card, and the heartbeat connection with the main controller.

[0057] Specifically, the local fault handling service runs locally on the compute node and starts as a system service with the operating system. Therefore, even if the local node experiences a network failure, power outage, or hardware failure but the operating system is still running, the local fault handling service can continue to run and perform local fault detection. The network status of the network interface card (NIC) refers to whether the NIC link on this node is connected and whether the NIC can send and receive data packets normally. This can be obtained, for example, by reading the link status register exposed by the NIC driver or by calling the network interface query command provided by the operating system. The disk status refers to whether the storage media on this node can be read and written normally. This can be obtained, for example, by checking whether disk I / O operations time out or return errors. The memory status refers to whether the memory on this node can be allocated and accessed normally. This can be obtained, for example, by checking whether memory allocation requests fail or memory verification occurs abnormally. The heartbeat connection with the main controller refers to the periodic exchange of liveness confirmation messages between the local fault handling service and the main controller. When the heartbeat connection does not receive a response from the main controller within a preset period, the local fault handling service determines that the communication link between this node and the main controller is abnormal. The local fault handling service integrates the above-mentioned various local status information to detect faults in this node, so that fault detection does not rely on the cluster-level controller's notification. Even if this node cannot communicate with the main controller due to network failure, the local fault handling service can still independently detect the occurrence of the fault based on the local status information.

[0058] Based on this, the local fault handling service can periodically collect the aforementioned local status information according to a preset polling cycle and compare the collected results with preset local isolation trigger conditions. Local isolation trigger conditions refer to a pre-defined combination of conditions used to determine whether local network isolation needs to be performed. For example, it could be set to a disconnected network interface card link and an interrupted heartbeat connection with the main controller, or an abnormal disk status and memory status. When the local fault handling service detects a fault in this node but the local isolation trigger conditions are not yet met, the local fault handling service continues to observe the local status and does not perform network isolation actions. This is because some local status anomalies may be short-lived or self-recoverable. If network isolation is performed before the trigger conditions are met, it may lead to the erroneous isolation of normally functioning business instances.

[0059] Step S320: When the preset local isolation triggering conditions are met, the local fault handling service obtains the container and business instance on the local node through the container runtime interface and determines the old business instance.

[0060] Specifically, the container runtime interface refers to the interface through which the local fault handling service interacts with the container runtime on the node. The container runtime is responsible for managing the lifecycle of containers on the node. For example, in a scenario where containerd is used as the container runtime, the container runtime interface can be the CRI interface provided by containerd or the native containerd interface. The local fault handling service uses this interface to query all currently running containers on the local node and their respective business instances. Old business instances refer to business instances that were already running on the local node before the fault occurred; these are business instances that need to be network isolated before the node recovers. After obtaining the containers and business instances on the local node through the container runtime interface, the local fault handling service can determine the old business instances that need to be isolated based on information such as the business instance's identifier, namespace, and running status. For example, the local fault handling service can obtain all running business instances on the local node and identify them as old business instances; for containers that are already terminated or have not started normally, network isolation is not required.

[0061] In step S330, the local fault handling service enters the network namespace of the old service instance through the container runtime interface and sets the container network card of the old service instance to an unavailable state to avoid address conflicts between the old service instance and the new service instance migrated to the healthy node when the target node recovers.

[0062] Specifically, a network namespace refers to a logical space created by the container runtime for each service instance to isolate network resources. Each service instance has an independent network interface card (NIC), network address, and routing table within its network namespace. After the local fault handling service enters the network namespace of the old service instance through the container runtime interface, it can directly manipulate the container NIC within that network namespace, rendering it unusable. For example, in a Linux operating system, the local fault handling service can obtain the network namespace file descriptor of the old service instance through the container runtime interface, enter that network namespace, and call network configuration commands to set the container NIC's status to down, preventing it from sending and receiving data packets. After rendering the container NIC unusable, although the old service instance is still running, it cannot interact with the industrial communication network through the container NIC, thus preventing it from occupying the original industrial communication address when the target node recovers.

[0063] The reason for performing network isolation locally rather than remotely by the main controller is that the local fault handling service runs on the faulty node and can still operate even if the node experiences a network failure or the communication link with the main controller is interrupted. The main controller, however, may be unable to access the node due to a network failure, and if the isolation action is performed remotely by the main controller, it may fail due to the inability to establish a remote connection. The reason for disabling the container's network interface card (NIC) instead of deleting the container is that deleting the container would result in the loss of the old service instance's running status and network configuration information, making it difficult to confirm the status of the old service instance and clean up any remnants during subsequent node recovery. Disabling the container's NIC, on the other hand, blocks the old service instance's communication capabilities while preserving the container's complete state, facilitating querying and cleanup during the subsequent recovery process.

[0064] In step S340, the local fault handling service records the isolation status to the local storage of this node. The isolation status includes the isolation round, business instance identifier, isolation result, and reason for the exception.

[0065] Specifically, local storage refers to storage media on the faulty node that can be directly read and written by the local fault handling service, such as the local disk. The local fault handling service can write the isolation status to the local disk in the form of a JSON file, or it can record it using other structured data formats. The isolation round number is used to identify the number of times the isolation action is executed. For example, when the local fault handling service re-executes the isolation action due to a restart, the isolation round number is incremented so that the subsequent recovery process can identify the latest isolation status. The business instance identifier is used to identify the isolated business instance, such as the business instance's name, number, or container identifier. The isolation result indicates whether the isolation action was successfully executed, for example, represented by a success or failure status value. The exception reason records the specific reason when the isolation action fails, such as the container runtime interface being inaccessible, the network namespace not existing, or the container network card operation failing. By recording the isolation status to local storage, the isolation action can be idempotently recovered after the local fault handling service restarts. That is, after the local fault handling service restarts, it can determine which business instances have been isolated based on the recorded isolation round number and business instance identifier, avoiding duplicate execution of the isolation action, and providing a status confirmation basis for the subsequent node recovery process.

[0066] In this embodiment, the local fault handling service perceives node faults based on various local status information such as network card status, disk status, memory status, and heartbeat connection, making fault perception independent of cluster-level controller notifications. It obtains containers and service instances on the local node through the container runtime interface and identifies the old service instance, ensuring that the identification of the old service instance is based on the node's actual local operating status. It enters the network namespace of the old service instance through the container runtime interface and sets the container network card to an unavailable state, preventing the old service instance from occupying industrial communication addresses after node recovery, thus reducing the risk of address conflicts between old and new service instances. By recording the isolation status, including isolation round, service instance identifier, isolation result, and anomaly cause, to local storage, the isolation action can be idempotently recovered after the local fault handling service restarts, providing a status confirmation basis for subsequent node recovery processes.

[0067] Example 5: like Figure 7 As shown, this embodiment, based on the above embodiments, describes the specific process of enabling the target node to re-participate in scheduling according to its isolation status after the target node has recovered and is ready. The difference between this embodiment and the previous embodiments lies in further defining the setting of the recovery stability window, the method of querying the isolation status through a private protocol, the logic for branch judgment based on the isolation status, and the process of cleaning up and scheduling the residual containers of old services. The above steps are further explanations of the node recovery part in step S300 of Embodiment 1.

[0068] In step S350, after the target node is restored to readiness, the main controller retains the unschedulable flag of the target node, and when the target node continues to be ready until the preset recovery stability window is reached, it queries the isolation status of the target node's local fault handling service through a network connection via a private protocol.

[0069] Specifically, target node readiness means that the target node, after experiencing a failure, re-reports its readiness status to the industrial cluster. For example, in a Kubernetes cluster, the Ready state of a Node object returns to normal, and the node agent resumes reporting. An unschedulable flag is a scheduling restriction flag set on a target node to prevent new service instances from being scheduled to that target node; for example, the NoSchedule taint in a Kubernetes cluster. After detecting that the target node has recovered to readiness, the master controller does not immediately remove the unschedulable flag but retains it, temporarily preventing the target node from participating in service scheduling.

[0070] The recovery stabilization window is a pre-set observation period used to confirm the continuous and stable operation of a target node. This window can be configured according to the industrial site's requirements for stability, determinism, and traceability, and is preferably 10 minutes. The main controller starts timing the recovery stabilization window after the target node has recovered and is ready, continuously monitoring the target node's readiness status during the timing period. When the target node maintains a ready state within the recovery stabilization window, the main controller confirms that the target node is operating stably and then executes subsequent isolation status queries. If the target node re-enters a non-ready state within the recovery stabilization window, it indicates that the target node has not yet truly stabilized. The main controller maintains the target node's fault handling state and does not execute subsequent scheduling and recovery procedures. This design is based on the consideration that node failures in industrial sites may be accompanied by repeated short-term recoveries and re-abnormalities. If the unschedulable flag is removed immediately after a node recovers and is ready, the node may resume service before its state is stable. If the node fails again, the recently migrated service will be interrupted again, resulting in a secondary failure. By setting a recovery stabilization window, nodes will only enter the subsequent recovery process after remaining continuously ready for a preset time. This filters out node status fluctuations and ensures that nodes are truly stable before resuming service.

[0071] After the target node has remained ready and reached a stable recovery window, the master controller queries the local fault handling service of the target node via a network connection using a private protocol. The private protocol refers to a pre-agreed communication protocol between the master controller and the local fault handling service that does not rely on the industrial cluster interface service, such as Protocol Buffers (PB) interacting via TCP. This setup is based on the fact that the local fault handling service runs locally on the target node, and its recorded isolation status is stored in the node's local storage. The industrial cluster interface service is unaware of this isolation status; therefore, the master controller needs to obtain the isolation status through direct communication with the local fault handling service. Using a private protocol instead of the industrial cluster interface service for querying ensures that the query process is independent of the availability and response speed of the interface service. Even under high load, the master controller can still directly establish a network connection with the local fault handling service and obtain the isolation status.

[0072] Specifically, the master controller sends an isolation status query request to the local fault handling service of the target node via a network connection using a proprietary protocol. Upon receiving the query request, the local fault handling service reads the isolation status recorded in its local storage and returns it to the master controller via the proprietary protocol. The master controller then determines whether there are any unresolved fault records on the target node based on the isolation status.

[0073] Step S360: When there are unresolved fault records in the isolation state, retain the unschedulable flag and wait for confirmation.

[0074] Specifically, unresolved fault records refer to records kept by the local fault handling service indicating that there are still unresolved or failed isolation records on the target node. For example, records where the isolation result is failed and the exception reason indicates that the container runtime interface is inaccessible, or records showing that the isolation action has not yet completed the isolation of all old service instances, all fall under the category of unresolved fault records. When unresolved fault records exist in the isolation state, it indicates that old service instances on the target node may still be occupying industrial communication addresses, or that the local fault handling service has not yet completed network isolation of all old service instances. In this case, the main controller retains the unschedulable flag of the target node, preventing the target node from participating in service scheduling, and logs it for confirmation. Waiting for confirmation may include waiting for manual intervention to confirm that the target node has fully recovered, or waiting for the local fault handling service to complete the isolation action and then query the isolation status again. This setting is based on the consideration that if the unschedulable flag is removed when unresolved fault records exist in the isolation state, the target node may resume carrying services, while the old service instances will still occupy the original communication addresses, leading to address conflicts between the old and new service instances and affecting the determinism of industrial communication.

[0075] Step S370: When there are no unresolved fault records in the isolation state, clean up the old service residual containers on the target node and remove the unschedulable flag, so that the target node can re-participate in scheduling.

[0076] Specifically, when there are no outstanding fault records in the isolation state, it indicates that the local fault handling service has completed network isolation of all old service instances on the target node, or that there are no old service instances on the target node that need to be isolated. At this point, the main controller cleans up residual old service containers on the target node. Residual old service containers refer to service instance containers on the target node that were not properly terminated by the industrial cluster due to abnormal exits, such as Pod containers that prevented the industrial cluster from exiting normally due to abnormal exits. These residual containers may still occupy resources or network configuration on the node; if not cleaned up, they may affect the normal operation of new service instances after the target node re-participates in scheduling. After cleaning up residual old service containers, the main controller removes the unschedulable flag on the target node, allowing the target node to re-participate in service scheduling.

[0077] In this embodiment, the main controller does not immediately remove the unschedulable flag after the target node recovers and becomes ready. Instead, it waits for the target node to remain ready until a recovery stabilization window is reached. Then, it queries the isolation status recorded in the local fault handling service via a private protocol. Based on the isolation status, it decides whether to retain the unschedulable flag and wait for confirmation, or to clean up the residual containers of old services and remove the unschedulable flag. By setting a recovery stabilization window, node state fluctuations are filtered out, ensuring that the node is truly stable before resuming service operation, reducing the risk of secondary service interruption due to node failure. By querying the isolation status instead of directly resuming scheduling, the node recovery decision is based on the actual isolation status of the local old service instance. The isolation status reflects whether the local old service instance still occupies the industrial communication address, which is a prerequisite for safe resumption of scheduling, thereby reducing the risk of address conflicts between the old service instance and the new service instance. By cleaning up the residual containers of old services, the residual containers of abnormal exits are cleared before the target node re-participates in scheduling, avoiding the impact of residual containers on the operation of the new service instance.

[0078] This embodiment implements the scheduling recovery process after a faulty node has recovered and is ready to resume operation through the above-described scheme. By retaining an unschedulable flag and setting a recovery stabilization window after the target node has recovered and is ready, the node will not immediately resume carrying services before its state has stabilized. By querying the isolation status of the local fault handling service through a private protocol after the stabilization window ends, the node recovery decision is based on the actual local isolation situation rather than solely on the node's ready state. By retaining the unschedulable flag and waiting for confirmation when there are unresolved fault records in the isolation state, address conflicts caused by re-including the node in the scheduling process while the isolation is not yet resolved are avoided. By cleaning up the old service remnant containers and removing the unschedulable flag when there are no unresolved fault records in the isolation state, the target node can re-participate in scheduling after safety is confirmed.

[0079] Example 6: like Figure 4 As shown, this embodiment, based on the above embodiments, describes the specific process of the main controller migrating migrateable service instances on the target node to a healthy node according to the node action lock, as well as the deletion and cleanup after migration timeout and the migration progress recording. The difference between this embodiment and the previous embodiments lies in further defining the setting of the unschedulable flag, the rules for obtaining and filtering the service instance list, the method for calling the eviction interface, the setting of the migration timeout, and the content and update method of the migration progress record. The above steps are further explanations of the service migration part in step S200 of Embodiment 1.

[0080] Step S240: Set an unschedulable flag for the target node and obtain a list of business instances on the target node. Exclude control plane component instances, business instances deployed as daemons, static business instances and their mirror objects directly managed by the node agent, and business instances fixed to a specific node through node selection or affinity from the list of business instances to obtain migrateable business instances.

[0081] Specifically, an unschedulable flag is a scheduling restriction flag set on a target node to prevent new service instances from being scheduled to that target node, such as the NoSchedule taint in a Kubernetes cluster. After acquiring the node action lock, the master controller first sets the unschedulable flag on the target node, preventing it from receiving new service instance scheduling requests. This is because if the target node has been determined to be faulty but has not yet completed its migration, and new service instances are still scheduled to that target node, it will increase the number of service instances that need to be migrated subsequently, and the newly scheduled service instances may not function properly due to the target node's failure.

[0082] The business instance list refers to the collection of all currently running business instances on the target node. The main controller queries the business instance list on the target node through the industrial cluster's interface service. For example, in a Kubernetes cluster, the main controller can query a list of all Pods on the target node through the API Server, which contains all business instances running on the target node.

[0083] After obtaining the list of business instances, the main controller excludes four types of business instances that should not be migrated from the list, thus obtaining the business instances that can be migrated.

[0084] Specifically, the first category is control plane component instances. Control plane components refer to the core components responsible for cluster management and control in an industrial cluster, such as kube-apiserver, etcd, kube-scheduler, and kube-controller-manager. Control plane components are typically deployed as static business instances on the master node, and their operational status directly affects the availability of the entire industrial cluster; therefore, they should not be migrated as migrateable business instances. Control plane component instances can be identified by their namespace. For example, in a Kubernetes cluster, control plane components typically run in the kube-system namespace, and the master controller can exclude business instances belonging to the kube-system namespace from the list of business instances.

[0085] The second category is business instances deployed as daemon processes. These are business instances in an industrial cluster that require a copy to run on each compute node, such as a DaemonSet type Pod in a Kubernetes cluster. The design intent of these instances is to run throughout the node's lifecycle and be bound to the node. If they are migrated to a healthy node, a similar instance may already exist on that node, leading to duplicate execution or resource conflicts. Business instances deployed as daemon processes can be identified by their deployment type. For example, in a Kubernetes cluster, if the Pod's ownerReferences field points to a DaemonSet object, then the Pod is a DaemonSet type Pod.

[0086] The third category consists of static business instances and their mirror objects directly managed by the node agent. Static business instances refer to business instances directly managed by the node agent without being scheduled through the industrial cluster interface service, such as Static Pods in a Kubernetes cluster. Mirror objects refer to the mirror records corresponding to static business instances in the industrial cluster interface service, such as Mirror Pods in a Kubernetes cluster. Static business instances are directly bound to their respective nodes, and their lifecycle is managed by the node agent. After migration to a healthy node, they cannot be properly taken over by the node agent of the healthy node. Static business instances and their mirror objects can be identified by the creation source of the business instance. For example, if a Pod's annotation field in a Kubernetes cluster contains static business instance configuration source information, then that Pod is a static business instance or its mirror object.

[0087] The fourth category consists of business instances that are fixed to a specific node through node selection or affinity. These instances specify the running node using a node selector or affinity rule, such as the `nodeSelector` or `affinity` field of a Pod in a Kubernetes cluster. The design intent of these instances is to run on a specific node. If they are migrated to a healthy node, they may fail to function properly because the healthy node does not meet their node selection or affinity requirements, or they may not be able to achieve their intended business functionality after migration. Business instances fixed to a specific node through node selection or affinity can be identified through their scheduling constraint fields.

[0088] By excluding the above four types of business instances, the migration scope is limited to migrateable business instances, avoiding migration operations on businesses that should not be migrated. For example, in an industrial cluster, migrating control plane component instances may cause cluster management functions to be interrupted; migrating business instances deployed in daemon mode may cause duplicate instances to appear on healthy nodes; migrating static business instances and their mirror objects may cause them to fail to operate normally due to the break in node proxy management relationships; and migrating fixed node business instances may cause them to remain in a pending scheduling state for a long time due to the inability to meet node selection or affinity constraints.

[0089] Step S250: Call the industrial cluster's eviction interface for the migrated service instance to trigger the rescheduling and reconstruction of the migrated service instance on a healthy node.

[0090] Specifically, the eviction interface refers to the interface provided by the industrial cluster for triggering the graceful termination of service instances and their rescheduling and reconstruction on other healthy nodes, such as the Eviction API in a Kubernetes cluster. The master controller calls the eviction interface for each migrateable service instance selected in step S240. After receiving the call request, the eviction interface first sends a termination signal to the migrateable service instance, causing it to execute the graceful termination process. Subsequently, the scheduling component of the industrial cluster reschedules the migrateable service instance to a healthy node and reconstructs it.

[0091] The reason for using the eviction interface instead of directly deleting migrateable service instances is that the eviction interface triggers a graceful termination and rescheduling process. Before termination, a migrateable service instance has the opportunity to complete its currently processed business requests, release occupied resources, and save necessary state information. Directly deleting a migrateable service instance, on the other hand, may lead to business interruption or data loss. For example, in industrial control scenarios, if a service instance executing control commands is directly deleted, it may cause the execution of control commands to be interrupted, affecting the operation of equipment in the industrial field. However, by triggering graceful termination through the eviction interface, the service instance can complete the current control command before termination, and then be rebuilt on a healthy node to continue executing subsequent control commands.

[0092] The main controller can perform eviction operations on migrateable service instances either serially or in parallel. Serial execution means the main controller calls the eviction interface sequentially for each migrateable service instance, processing the next instance only after the previous one has been rebuilt. Parallel execution means the main controller calls the eviction interface for multiple migrateable service instances simultaneously. Serial execution is suitable for scenarios where the migration order is critical, such as when multiple service instances with dependencies need to be migrated in that order. Parallel execution is suitable for scenarios where migration speed is critical, such as when there are many migrateable service instances with no dependencies on each other; parallel execution can shorten the overall migration time.

[0093] Once the rebuilt, migrateable business instances are ready, record the business instances that have completed the migration.

[0094] Specifically, after a migrateable service instance is rebuilt on a healthy node, the main controller monitors the running status of the rebuilt migrateable service instance. When the rebuilt migrateable service instance enters the ready state, it indicates that the migrateable service instance is running normally on the healthy node, and the main controller records the migrateable service instance as a service instance that has completed migration. The recording method is, for example, adding the service identifier of the migrateable service instance to the completed migration list, which is subsequently updated to the node action lock as the scope of completed migration.

[0095] Step S260: Starting from the invocation of the eviction interface, if the migrateable business instance is not terminated within the preset migration timeout period, perform a deletion cleanup on the migrateable business instance.

[0096] Specifically, the migration timeout period refers to a pre-set time threshold used to determine whether a migrateable service instance has terminated normally. This migration timeout period starts counting from the time the eviction interface is called and can be configured according to the type of service instance and the business recovery requirements of the industrial site, preferably 60 seconds. If a migrateable service instance is not terminated within the migration timeout period, it indicates that the eviction interface failed to trigger the graceful termination of the migrateable service instance. Possible reasons include an abnormal target node causing the migrated service instance to be unable to respond to the termination signal, or an abnormal internal process of the migrated service instance causing the graceful termination process to fail to complete. In this case, the main controller performs a deletion cleanup on the migrated service instance, that is, directly deletes the migrated service instance as a fallback cleanup method of the eviction interface.

[0097] It should be noted that the deletion and cleanup is performed only once. This means that the main controller will not repeatedly perform the deletion and cleanup operation on the same migrateable service instance to avoid operational conflicts caused by duplicate deletions. The reason for this setting is that the eviction interface can trigger the graceful termination of the migrateable service instance normally in most cases. If it is not terminated after a timeout, it is an abnormal situation. The remaining migrateable service instances can be cleared by a single deletion and cleanup, without the need for repeated execution.

[0098] Step S270: Record the migration progress, which includes the business identifier, processing action, processing result and failure reason, and update the migration progress as the scope of completed migration to the node action lock.

[0099] Specifically, migration progress refers to the data recorded by the main controller during the migration process, reflecting the migration processing status of each migrateable business instance. The business identifier identifies the migrateable business instance being processed, such as the instance's name, number, or container identifier. The processing action indicates the type of operation performed on the migrateable business instance, such as eviction or deletion / cleanup. The processing result indicates whether the operation was successfully executed, for example, represented by a success or failure status value. The failure reason records the specific reason for the operation failure, such as eviction API call failure, migration timeout failure, or deletion / cleanup failure.

[0100] The master controller updates the migration progress as part of the completed migration scope to the node action lock. Specifically, for migrated service instances that have completed migration, the master controller adds their service identifier to the completed migration scope of the node action lock; for migrated service instances that failed to migrate, the master controller records their service identifier and the reason for failure in the migration progress, so that after a master controller switchover, the new master controller can determine which service instances have completed migration and which need to be processed based on the migration progress. By updating the migration progress to the node action lock, after a master controller switchover, the new master controller can skip migrated service instances based on the completed migration scope recorded in the node action lock, and perform targeted processing on failed service instances based on the reason for failure recorded in the migration progress, thereby reducing the risk of duplicate evictions and incomplete migration states.

[0101] This embodiment achieves the screening, migration, timeout cleanup, and migration progress recording of migrateable business instances through the above-described scheme. By setting an unschedulable flag on the target node, new business instances are prevented from being scheduled to faulty nodes. By excluding control plane component instances, business instances deployed as daemons, static business instances and their mirror objects directly managed by node agents, and business instances fixed to specific nodes through node selection or affinity, the migration scope is limited to migrateable business instances, avoiding migration operations on services that should not be migrated. By calling the eviction interface to trigger the rescheduling and reconstruction of migrateable business instances on healthy nodes, the business migration process is completed by the industrial cluster's scheduling mechanism, and migrateable business instances have the opportunity to complete a graceful termination process before termination. A deletion cleanup is performed as a fallback operation if the migration timeout period is not terminated, ensuring that migrateable business instances that the eviction interface failed to terminate normally are cleaned up. By recording the migration progress, including business identifier, processing action, processing result, and failure reason, and updating it to the node action lock, the new master controller after a master controller switchover can skip completed items and continue processing unfinished items based on the migration progress, reducing the risk of duplicate evictions and incomplete states.

[0102] Example 7: like Figure 2As shown in the figure, this embodiment provides an industrial cluster node fault migration system. Multiple fault migration controllers are deployed on multiple master control nodes of the system, and local fault handling services are deployed on each computing node. The multiple fault migration controllers include master controllers and auxiliary controllers.

[0103] Specifically, an industrial cluster refers to a container orchestration cluster designed for industrial control, data acquisition, and general business operations, typically comprising multiple master nodes and multiple compute nodes. Fault migration controllers are deployed as independent processes on the master nodes, responsible for fault confirmation, business migration, and recovery processing at the cluster level. Local fault handling services are deployed as system services on each compute node, responsible for fault detection and local network isolation for that node. Multiple fault migration controllers elect a master through a distributed lease mechanism; at any given time, only one fault migration controller acts as the master controller to perform cluster write operations, while the others operate as auxiliary controllers. For example, an industrial cluster may include three master nodes, each with one fault migration controller deployed on it. The three fault migration controllers achieve master election and auxiliary observation through lease resources. It should be understood that although this embodiment uses three master nodes as an example, in other embodiments, the number of master nodes can be five or more, as long as the majority arbitration requirement is met.

[0104] The main controller performs health and network probes on the target node. When both health and network probes confirm that the target node is unreachable and continues to reach a preset fault confirmation window, it requests the auxiliary controller to perform auxiliary observations on the target node. When the combined observation results of the main controller and the auxiliary controller exceed half of the confirmed anomalies, the target node is determined to be faulty. The node action lock is acquired at the granularity of the target node. The node action lock records the stage of fault migration, the range of completed migration, and the takeover status.

[0105] Specifically, the target node refers to the computing node in the industrial cluster that needs to be monitored for failures. Health detection involves accessing the target node's node agent to confirm its normal response; the node agent is, for example, the kubelet process running on the node. Network detection involves sending network probe packets to the target node to confirm its network reachability; network probe packets are, for example, ICMP probe packets. The fault confirmation window is a pre-set observation period used to filter short-term network fluctuations. This window can be configured based on network latency and service recovery requirements in the industrial setting, preferably 10 seconds. When the main controller initiates the fault confirmation process, it first obtains the target node's node status and lease status. Only when the node status is ready and the lease status is normal will health detection and network detection be performed. If both health detection and network detection confirm that the target node is unreachable, it means the node agent is inaccessible and the network probe packet is unreachable; only when both fail simultaneously is the target node deemed unreachable. When the target node's unreachability persists for the entire fault confirmation window period, the main controller requests the auxiliary controller to perform auxiliary observations on the same target node. The auxiliary observations performed by the auxiliary controller are the same as those for health detection and network detection. The main controller aggregates its own detection results and the observation results of the auxiliary controllers, and counts the number of detection and observation results confirming the target node's anomaly based on the total number of fault migration controllers. When this number exceeds half of the total, the target node is determined to be faulty. For example, in the case of three fault migration controllers, if the main controller and at least one auxiliary controller confirm the target node's anomaly, the number of confirmed anomalies is two, which exceeds half of the total of three, and the target node is determined to be faulty. Through majority arbitration, fault determination does not depend on the observation results of a single controller, reducing the impact of one-sided observation bias on fault confirmation.

[0106] A node action lock is a data object at the target node level, used to indicate whether a fault migration controller is already processing the target node and to record the fault migration controller holding the lock. The node action lock records the stage of fault migration, the scope of completed migrations, and the takeover status. The stage of fault migration refers to the current stage of the migration action, such as migration start, service eviction, service rebuilding, or migration completion. The scope of completed migrations refers to the set of service instances that have completed migration, such as a list of service identifiers. The takeover status indicates whether the node action lock can be taken over by a new master controller, such as lock holding, takeover possible, or completed. The node action lock is carried by resources with consistent update capabilities in the industrial cluster, such as Kubernetes native objects or custom resources. Only the fault migration controller holding the node action lock can perform fault migration actions on the target node. By setting node action locks at the target node level and limiting the execution of fault migration actions to only the lock holder, migration actions on the same faulty node remain mutually exclusive, avoiding migration conflicts caused by multiple controller instances concurrently operating on the same node.

[0107] The master controller is also used to migrate migrateable service instances on the target node to healthy nodes based on node action locks, so that after a master controller switchover, the new master controller takes over the unfinished migration actions according to the node action locks.

[0108] Specifically, migrateable business instances refer to business instances on the target node that need to be migrated to a healthy node for re-running. A healthy node is a computing node in the industrial cluster that is in normal operating condition and capable of supporting the migrated business. During the migration process, the master controller continuously records the business instances that have been migrated to the completed migration range of the node action lock. When the master controller is switched, for example, if the master node where the master controller resides crashes or the master controller process exits abnormally, multiple fault migration controllers re-elect a master through a distributed lease, and the auxiliary controllers compete for the distributed lease to become the new master controller. The new master controller does not inherit the memory state of the original master controller, but instead reconstructs the node state, business state, scheduling constraint state, and node action lock state of the target node from the interface service of the industrial cluster. Then, based on the fault migration stage and completed migration range recorded by the node action lock, it skips the business instances that have been migrated and continues to execute the incomplete migration actions. The consideration for this setting is that the original master controller may have crashed, and its memory state is unreliable. Reconstructing the state from the interface service and taking over based on the node action lock ensures that the migration actions can continue idempotently after the master controller switch, reducing the risk of duplicate eviction and incomplete states.

[0109] After detecting a fault in the local node, the local fault handling service performs local network isolation on the old service instances on the local node and records the isolation status; after the target node recovers and is ready, the main controller enables the target node to re-participate in scheduling based on the isolation status.

[0110] Specifically, the local fault handling service runs locally on the compute node and can detect faults on the node even when it experiences network failures, power outages, or hardware failures while the operating system remains operational. The local fault handling service detects faults based on the network status of the node's network interface card (NIC), disk status, memory status, and heartbeat connection with the main controller. When preset local isolation trigger conditions are met, the local fault handling service obtains the containers and service instances on the node through the container runtime interface, identifying the old service instance. The old service instance refers to the service instance that was already running on the node before the fault occurred. The local fault handling service enters the network namespace of the old service instance through the container runtime interface and sets the container NIC of the old service instance to an unavailable state, for example, setting the container NIC to a down state. This setting is based on the consideration that dedicated networks carrying control, acquisition, or I / O service communication in industrial scenarios have high requirements for network address stability. If the old service instance still occupies the original communication address after the faulty node recovers, it may cause address conflicts with the new service instance migrated to the healthy node. Network isolation is performed locally rather than remotely by the main controller because the local fault handling service can still operate when a node fails, while the main controller may be unable to access the node due to network failure. Setting the container's network interface card to an unavailable state instead of deleting the container is to preserve the container for subsequent recovery and status confirmation. The local fault handling service records the isolation status to the local storage of this node. The isolation status includes the isolation round, the service instance identifier, the isolation result, and the reason for the exception. The isolation round indicates the number of times the isolation action was executed, the service instance identifier identifies the isolated service instance, the isolation result indicates whether the isolation was successful, and the reason for the exception records the reason for isolation failure. By recording the isolation status to local storage, the isolation action can be idempotently recovered after the local fault handling service restarts.

[0111] After the target node recovers and becomes ready, the main controller retains the unschedulable flag of the target node. While the target node remains ready for a preset recovery stabilization window, the main controller queries the local fault handling service of the target node via a private protocol over the network connection to check its isolation status. If there are unresolved fault records in the isolation status, the unschedulable flag is retained and confirmation is awaited. If there are no unresolved fault records in the isolation status, the old service remnants on the target node are cleaned up and the unschedulable flag is removed, allowing the target node to rejoin scheduling. By setting a recovery stabilization window, the node is prevented from immediately resuming service operations before its status has stabilized. By querying the isolation status of the local fault handling service, the node recovery decision is based on the actual local isolation situation rather than solely on the node's ready status, reducing the risk of address conflicts caused by re-including the node in scheduling while the isolation is not yet resolved.

[0112] like Figure 2 As shown in the system architecture of this embodiment, the data interaction paths between modules are as follows: The main controller and auxiliary controllers interact through distributed leases for master election. The main controller periodically renews the lease during operation. The auxiliary controller monitors the lease status and competes to become the new main controller after the lease expires. After confirming that the target node is unreachable and continuously reaches the fault confirmation window, the main controller sends an auxiliary observation request to the auxiliary controller. The auxiliary controller performs auxiliary observation on the same target node and feeds back the observation results to the main controller. The main controller interacts with the interface service of the industrial cluster for status read / write. The main controller reads the node status, lease status, business instance list, and node action lock status of the target node from the interface service, and writes the unschedulable flag, eviction interface call request, migration progress, and updated node action lock content to the interface service. The main controller interacts with the local fault handling service for isolation status query. After the target node continuously reaches the recovery stability window, the main controller sends an isolation status query request to the local fault handling service via a network connection through a private protocol. The local fault handling service reads the isolation status recorded in its local storage and returns it to the main controller through the private protocol.

[0113] This embodiment, through the above system architecture, enables the fault migration control plane to not rely on a single controller instance. Even when a single master node or a single controller instance malfunctions, the migration action can still continue through the master election mechanism. Through the coordination of node action locks and local network isolation, a complete closed loop is formed for service migration and node recovery.

[0114] Example 8: To more clearly illustrate this application, the following example demonstrates the complete application process of the industrial cluster node fault migration method provided in this application, using a worker node running industrial control services in an industrial cluster experiencing a complete power outage. The industrial cluster includes three master control nodes and multiple compute nodes. Each of the three master control nodes deploys a fault migration controller, which elects a master through a distributed lease. One fault migration controller operates as the master controller, while the other two operate as auxiliary controllers. Each compute node deploys a local fault handling service. The compute node that experienced the power outage is denoted as node N. Node N runs industrial control service instances, data acquisition service instances, and several system components.

[0115] The first phase involves fault confirmation and majority arbitration. The main controller periodically monitors the status of each computing node in the industrial cluster. First, the main controller reads the node status and lease status of node N from the industrial cluster's interface service, confirming that node N is in a ready state and its lease is normal. Then, it performs health probes and network probes on node N. Health probes involve accessing node N's node agent, while network probes involve sending network probe packets to node N. When node N experiences a complete power failure, the node agent on node N stops responding, and the network link is interrupted. Therefore, both health probes and network probes confirm that node N is unreachable. The main controller continuously observes node N's unreachable state until a preset fault confirmation window is reached. In this application scenario, this window is set to 10 seconds. The purpose of the 10-second fault confirmation window is to filter out short-term network jitter that may occur in the industrial environment, such as temporary unreachability caused by network device switching or momentary link congestion. If the node becomes reachable within 10 seconds, it is considered network jitter and migration is not triggered. In this application scenario, if node N fails to become reachable within 10 seconds due to a power outage, the main controller will confirm that node N remains unreachable.

[0116] Subsequently, the master controller requests the two auxiliary controllers to perform auxiliary observations on node N. The auxiliary observations performed by the auxiliary controllers are the same as the health probes and network probes performed by the master controller; that is, the auxiliary controllers also access node N's node agent and send network probe packets to node N. After completing the auxiliary observations, the two auxiliary controllers feed back the observation results to the master controller. The master controller summarizes its own probe results and the observation results of the two auxiliary controllers, and counts the number of probes and observations confirming node N's anomaly based on the total number of the three fault migration controllers. In this application scenario, the master controller's probe results confirm node N's anomaly, and the observation results of both auxiliary controllers also confirm node N's anomaly, with the number of confirmed anomalies being three, exceeding half of the total three. The master controller determines node N is faulty and initiates the migration process. Through majority arbitration, even if the master controller misjudges node N's anomaly due to its own network anomaly, the migration process will not be triggered as long as the observation results of the majority of fault migration controllers do not confirm anomalies, thus reducing the risk of misjudgment by a single controller.

[0117] The second phase involves node action lock acquisition and service migration. After the main controller determines that node N is faulty, it first acquires node action locks at the granularity of node N. A node action lock is a data object at the granularity of node N, used to indicate whether a fault migration controller is already processing node N, and to record the fault migration controller holding the lock. The node action lock records the stage of fault migration, the scope of completed migration, and the takeover status. In this application scenario, the main controller successfully acquires the node action lock for node N, sets the stage of fault migration to migration start, and the takeover status to lock holding.

[0118] After acquiring the node action lock, the main controller sets an unschedulable flag on node N, i.e., sets a NoSchedule taint, preventing new business instances from being scheduled to node N. Subsequently, the main controller obtains a list of business instances on node N from the industrial cluster's interface service. From this list, four types of business instances that should not be migrated are excluded, resulting in migrateable business instances. The first type is control plane component instances, such as kube-apiserver and etcd running in the kube-system namespace. The running status of these instances directly affects the availability of the entire industrial cluster and should not be migrated. The second type is business instances deployed as daemons, such as DaemonSet type Pods. These instances are designed to run with the node's lifecycle and are bound to the node. If migrated to a healthy node, similar business instances may already exist on the healthy node, leading to duplicate execution or resource conflicts. The third type is static business instances and their image objects directly managed by the node agent, such as Static Pods and Mirror Pods. These instances are directly bound to their respective nodes, and their lifecycle is managed by the node agent. After being migrated to a healthy node, they cannot be properly taken over by the healthy node agent. The fourth category consists of business instances that are fixed to a specific node through node selection or affinity. For example, business instances that specify the running node through the `nodeSelector` or `affinity` field. These business instances are designed to run on a specific node. If they are migrated to a healthy node, they may not function properly because the healthy node does not meet their node selection or affinity requirements. In this application scenario, the industrial control business instance and the data acquisition business instance on node N do not belong to the above four exclusion categories, and are therefore identified as migrateable business instances.

[0119] The main controller invokes the industrial cluster's eviction interface for each migrateable service instance, triggering its rescheduling and reconstruction on a healthy node. The eviction interface, rather than direct deletion, is used because it triggers a graceful termination and rescheduling process, allowing the migrated service instance to complete its currently processed business requests, release occupied resources, and save necessary state information before termination. For example, after receiving the termination signal from the eviction interface, an industrial control service instance can first complete its currently executing control command, then execute the graceful termination process, and subsequently be reconstructed on a healthy node to continue executing subsequent control commands. In this application scenario, the main controller invokes the eviction interface sequentially for both the industrial control and data acquisition service instances. The industrial cluster's scheduling component reschedules these two service instances to healthy nodes and reconstructs them. The main controller monitors the running status of the reconstructed service instances. When a reconstructed service instance enters the ready state, it is recorded as a migrated service instance, and its service identifier is added to the completed migration range of the node's action lock.

[0120] During the migration process, the main controller starts timing from the moment the eviction interface is invoked. If a migrateable service instance is not terminated within the preset migration timeout period, a deletion cleanup is performed on the migrateable service instance. The migration timeout is set to 60 seconds in this application scenario. The purpose of the 60-second migration timeout is to provide a reasonable time margin for the normal termination process of the eviction interface, while avoiding prolonged blocking of the overall migration process due to the inability of individual service instances to terminate normally. In this application scenario, the industrial control service instance completes graceful termination and is rebuilt and ready on a healthy node within approximately 15 seconds after invoking the eviction interface, and the data acquisition service instance completes graceful termination and is rebuilt and ready on a healthy node within approximately 20 seconds after invoking the eviction interface. Both terminate normally within the 60-second migration timeout period without triggering the deletion cleanup operation. The main controller records the migration progress, including the service identifier, processing action, processing result, and failure reason, and updates the node action lock with the migration progress as the completed migration scope. When all migrateable service instances have completed the migration, the main controller updates the fault migration stage of the node action lock to "completed".

[0121] The third stage involves local network isolation. During a system-wide power failure on node N, the local fault handling service on node N detects the fault before the power failure. The local fault handling service detects the fault based on the network status, disk status, memory status of the node's network interface card (NIC), and the heartbeat connection with the main controller. In this application scenario, before the system-wide power failure, the heartbeat connection between node N and the main controller is interrupted. The local fault handling service detects that the heartbeat connection has not received a response from the main controller within a preset period, and simultaneously, the NIC link status becomes disconnected, meeting the preset local isolation trigger conditions. The local fault handling service obtains the containers and service instances on the node through the container runtime interface to identify the old service instances. The container runtime interface can be, for example, the CRI interface provided by containerd or a native containerd interface. In this application scenario, the local fault handling service obtains the currently running industrial control service instances and data acquisition service instances on node N through the containerd interface and identifies them as the old service instances.

[0122] The local fault handling service accesses the network namespace of the old service instance through the container runtime interface and disables the container network interface card (NIC) of the old service instance. Specifically, the local fault handling service obtains the network namespace file descriptor of the old service instance through the containerd interface, enters the network namespace, and calls the network configuration command to set the status of the container NIC to down, preventing the container NIC from sending and receiving data packets. After disabling the container NIC, the old service instance is still running, but it cannot interact with the industrial communication network through the container NIC, thus preventing it from occupying the original industrial communication address when node N recovers. Network isolation is performed locally rather than remotely by the main controller because the local fault handling service runs on node N and can continue to operate even if node N experiences a network failure or the communication link with the main controller is interrupted. The main controller, however, may be unable to access node N due to a network failure; if the isolation action were performed remotely by the main controller, it might fail due to the inability to establish a remote connection. The reason for making the container network interface card unavailable instead of deleting the container is that deleting the container would result in the loss of the running status and network configuration information of the old service instance, making it difficult to confirm the status of the old service instance and clean up any remnants during subsequent node recovery. Making the container network interface card unavailable can block the communication capability of the old service instance while preserving the complete status of the container, which is convenient for querying and cleaning up in the subsequent recovery process.

[0123] The local fault handling service records the isolation status to the local storage of this node. The isolation status includes the isolation round, business instance identifier, isolation result, and reason for the exception. In this application scenario, the local fault handling service writes the isolation status to the local disk of node N in the form of a JSON file. The isolation round is recorded as the first round, the business instance identifier is recorded as the identifier of the industrial control business instance and the data acquisition business instance, the isolation result is recorded as successful, and the reason for the exception is recorded as empty. By recording the isolation status to local storage, the isolation action can be idempotently recovered after the local fault handling service restarts. That is, after restarting, the local fault handling service can determine which business instances have been isolated based on the recorded isolation round and business instance identifier, avoiding duplicate execution of the isolation action, and providing a status confirmation basis for subsequent node recovery processes.

[0124] The fourth phase involves faulty node recovery and scheduling restoration. After the power module of node N is replaced by maintenance personnel, it is powered on and restarted. The node agent on node N resumes operation and reports its ready status to the industrial cluster. The main controller detects that node N's node status has returned to the ready state. Upon detecting that node N has recovered and is ready, the main controller does not immediately remove the unschedulable flag from node N, but retains it, temporarily preventing node N from participating in service scheduling. Subsequently, the main controller starts a recovery stabilization window timer, which is set to 10 minutes in this application scenario. The purpose of the 10-minute recovery stabilization window is to filter node state fluctuations, ensuring that the node is truly stable before resuming service operation. Node failures in industrial settings may be accompanied by repeated short-term recovery and re-abnormalities. If the unschedulable flag is removed immediately after node recovery, the node may resume service operation before its state is stable. If the node fails again, the newly migrated service will be interrupted again, resulting in a secondary failure. In this application scenario, node N maintains its ready state for 10 minutes after recovery, and the main controller confirms that node N is running stably.

[0125] The main controller queries the local fault handling service of node N via a network connection using a proprietary protocol. This proprietary protocol, such as the PB protocol, interacts via a TCP connection. Using a proprietary protocol instead of the industrial cluster interface service for the query makes the query process independent of the availability and response speed of the interface service. Even when the interface service is under high load, the main controller can still directly establish a network connection with the local fault handling service and obtain the isolation status. Upon receiving the query request, the local fault handling service reads the isolation status recorded on its local disk and returns the isolation status to the main controller via the PB protocol. In this application scenario, the isolation status returned by the local fault handling service shows that the isolation round is the first round, and the isolation results for both the industrial control business instance and the data acquisition business instance are successful, with empty exception reasons, meaning there are no unresolved fault records in the isolation status.

[0126] The main controller determines that there are no outstanding fault records on node N based on the isolation status, and then cleans up residual containers of old services on node N. Residual containers of old services refer to service instance containers on node N that were not properly terminated by the industrial cluster due to abnormal exits, such as Pod containers whose abnormal exits prevented the industrial cluster from exiting normally. In this application scenario, when node N experiences a power outage, the containers of industrial control service instances and data acquisition service instances abnormally exit due to the power failure, and the industrial cluster fails to terminate these containers normally. The main controller performs a cleanup operation on the residual containers of old services on node N. After cleaning up the residual containers of old services, the main controller removes the unschedulable flag on node N, allowing node N to re-participate in service scheduling. Thus, the complete closed loop execution from fault confirmation, majority arbitration, node action lock acquisition, service migration, local network isolation to node recovery is completed.

[0127] In this application scenario, the fault confirmation window is set to 10 seconds, ensuring that fault confirmation meets the rapid recovery requirements of industrial scenarios while filtering out short-term network jitter. The migration timeout is set to 60 seconds, providing a reasonable time margin for the normal termination process of the eviction interface, while avoiding prolonged blockage of the overall migration process due to the inability of individual service instances to terminate normally. The recovery stabilization window is set to 10 minutes, filtering out node status jitter and ensuring that the node is truly stable before resuming service operation. Through the coordination of the above links, the industrial control service instances and data acquisition service instances on node N automatically migrate to healthy nodes and are rebuilt and ready after fault confirmation. After node N recovers, it rejoins scheduling after stabilization window confirmation and isolation status query, reducing the risk of address conflicts between old and new service instances and improving the requirements for stability, determinism, and traceability in industrial sites.

[0128] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An industrial cluster node failover method, a plurality of failover controllers are deployed on a plurality of master nodes of the industrial cluster, a local fault handling service is deployed on each computing node, the plurality of failover controllers include a primary controller and a secondary controller, characterized in that, The method includes: The main controller performs health detection and network detection on the target node. When both the health detection and the network detection confirm that the target node is unreachable and continues to reach a preset fault confirmation window, it requests the auxiliary controller to perform auxiliary observation on the target node. When the combined observation results of the main controller and the auxiliary controller exceed half of the confirmed anomalies, the target node is determined to be faulty. The main controller acquires node action locks at the granularity of the target node. The node action locks record the stage of the fault migration, the scope of the completed migration, and the takeover status. The main controller migrates the migrateable service instances on the target node to healthy nodes according to the node action locks, so that after the main controller is switched over, the new main controller takes over the incomplete migration actions according to the node action locks. After detecting a fault in the local node, the local fault handling service performs local network isolation on the old service instances on the local node and records the isolation status; after the target node recovers and becomes ready, the main controller enables the target node to re-participate in scheduling based on the isolation status.

2. The method of claim 1, wherein, The main controller performs health and network probes on the target node, including: Obtain the node status and lease status of the target node. When the node status is ready and the lease status is normal, perform the health probe and the network probe. The health probe involves accessing the node proxy of the target node, and the network probe involves sending a network probe message to the target node. When the node agent is inaccessible and the network probe packets are unreachable and continue to reach the fault confirmation window, the target node is confirmed to be unreachable. When the target node becomes reachable again within the fault confirmation window, the target node is determined to be experiencing network jitter, and the node status and lease status are reacquired.

3. The method of claim 1, wherein, The combined observations from the main controller and the auxiliary controller show that more than half of the data confirm anomalies, including: The auxiliary observations performed by the auxiliary controller on the target node are the same as those performed by the health detection and the network detection. The number of detection and observation results of the target node anomaly is confirmed by statistically analyzing the total number of the multiple fault migration controllers. When the number exceeds half of the total number, the target node is determined to be faulty; When the number does not exceed half of the total number, the target node is determined to be a suspected fault, and the health detection and network detection are continued to be performed on the target node. When the target node is confirmed to be unreachable again, the timer of the fault confirmation window is restarted.

4. The method of claim 1, wherein, The node action lock is a data object with the target node as the granularity, used to indicate whether the target node is already being processed by a fault migration controller, and to record the fault migration controller holding the lock; The node action lock is carried by resources in the industrial cluster that have consistent update capabilities; A fault migration controller that holds only the node action lock can perform a fault migration action on the target node.

5. The method of claim 1, wherein, After the master controller switches over, the new master controller takes over the unfinished migration actions according to the node action lock, including: The multiple failover controllers elect a master through a distributed lease. After the master controller stops renewing its lease, the distributed lease times out, and the auxiliary controllers compete for the distributed lease to become the new master controller. The new master controller does not inherit the memory state of the master controller. First, it confirms that the interface service of the industrial cluster is stable and available, and then reconstructs the node state, business state, scheduling restriction state and node action lock state of the target node from the interface service. When the fault migration stage recorded by the node action lock is completed, the migration action is skipped; when the node action lock can take over, the unfinished migration actions continue to be executed according to the fault migration stage and the completed migration range.

6. The method of claim 1, wherein, Upon detecting a fault on this node, the local fault handling service performs local network isolation on old service instances on this node and records the isolation status, including: The local fault handling service detects faults in this node based on the network status, disk status, memory status of the node's network card, and heartbeat connection with the main controller. When the preset local isolation triggering conditions are met, the local fault handling service obtains the container and business instance on the local node through the container runtime interface and determines the old business instance. The local fault handling service enters the network namespace of the old service instance through the container runtime interface and sets the container network card of the old service instance to an unavailable state to avoid address conflicts between the old service instance and the new service instance migrated to the healthy node when the target node recovers. The local fault handling service records the isolation status to the local storage of this node. The isolation status includes the isolation round, business instance identifier, isolation result, and reason for the exception.

7. The method according to claim 1, characterized in that, After the target node recovers and becomes ready, the main controller, based on the isolation status, re-enables the target node to participate in scheduling, including: After the target node recovers and becomes ready, the main controller retains the unschedulable flag of the target node, and when the target node continues to be ready until a preset recovery stabilization window is reached, it queries the isolation status of the target node's local fault handling service via a network connection using a private protocol. If there are unresolved fault records in the isolation state, retain the unschedulable flag and wait for confirmation; When there are no unresolved fault records in the isolation state, clean up the old service residual containers on the target node and remove the unschedulable flag, so that the target node can re-participate in scheduling.

8. The method according to claim 1, characterized in that, The main controller migrates the migrateable service instances on the target node to healthy nodes according to the node action lock, including: Set an unschedulable flag for the target node, and obtain a list of business instances on the target node. Exclude control plane component instances, business instances deployed as daemons, static business instances and their mirror objects directly managed by the node agent, and business instances fixed to a specific node through node selection or affinity from the list of business instances to obtain the migrateable business instances. The industrial cluster's eviction interface is invoked on the migrated service instance, triggering the rescheduling and reconstruction of the migrated service instance on a healthy node; Once the reconstructed migrateable service instance is ready, record the service instance that has completed the migration.

9. The method according to claim 8, characterized in that, Also includes: Starting from the invocation of the expulsion interface, if the migrateable service instance is not terminated within the preset migration timeout period, a deletion cleanup will be performed on the migrateable service instance. Record the migration progress, including business identifier, processing action, processing result and failure reason, and update the node action lock with the migration progress as the completed migration scope.

10. An industrial cluster node fault migration system, characterized in that, Multiple fault migration controllers are deployed on multiple master control nodes of the system, and local fault handling services are deployed on each computing node. The multiple fault migration controllers include master controllers and auxiliary controllers. The main controller is used to perform health detection and network detection on the target node. When both the health detection and the network detection confirm that the target node is unreachable and continues to reach a preset fault confirmation window, it requests the auxiliary controller to perform auxiliary observation on the target node. When the observation results of the main controller and the auxiliary controller together confirm an anomaly by more than half, the target node is determined to be faulty. Node action locks are obtained at the granularity of the target node. The node action locks record the stage of fault migration, the range of migration completed, and the takeover status. The main controller is also used to migrate the migrateable service instances on the target node to a healthy node according to the node action lock, so that after the main controller is switched, the new main controller takes over the unfinished migration action according to the node action lock. After detecting a fault in this node, the local fault handling service performs local network isolation on the old service instances on this node and records the isolation status. After the target node recovers and becomes ready, the main controller enables the target node to re-participate in scheduling based on the isolation status.