Node exception processing method and device, computer device, and storage medium

Abstract

The application relates to a node exception processing method and device, computer equipment and a storage medium. For each node in a cluster, the number of times that a target abnormal state of a work unit appears in each node in a sliding time window is acquired; a node whose number of times is greater than a first threshold value is determined as an abnormal node, and a node isolation operation is performed on the abnormal node; after the node isolation operation is performed, the abnormal node is determined for recovery based on a preset recovery condition, and the isolation state of the abnormal node is released in the case that the preset recovery condition is met. Therefore, an admission stage failure abnormal node can be accurately identified, and the problem that an existing node ready state signal cannot detect such a fault is solved; the isolation operation is performed on the abnormal node to prevent new work unit scheduling, automatic fusing is realized, and a large number of Pod reconstruction and control plane load surge caused by node abnormalities are prevented; the preset recovery condition is used to automatically release the isolation, and the stability and autonomy of the cluster are comprehensively improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a node anomaly handling method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In a Kubernetes container orchestration cluster, the kubelet component on a node is responsible for managing the entire lifecycle of a pod, including admission control for pod creation requests. When the kubelet encounters an exception during the admission phase, the pod is placed in a failed state due to an unexpected admission error or other admission phase errors. However, the node itself remains in a ready state and appears healthy to the outside world. In this state, the core controller on the control plane continuously rebuilds the failed pod, and the scheduler constantly attempts to schedule new pods to the failed node, creating a failure storm. This results in a large number of pod creation and state update operations, causing a large number of write operations to the cluster state storage system, a sharp increase in control interface service requests, and continuous retries by the controller increasing system load. In severe cases, it can lead to a degraded or even unstable performance of the entire control plane.

[0003] To address the aforementioned node-level failures, existing Kubernetes cluster scheduling and health management mechanisms primarily include: a node readiness detection mechanism, which determines node health by monitoring kubelet heartbeats and marks a node as unschedulable only when it stops reporting heartbeats; or, triggering alarms by collecting cluster metrics for manual handling by operations personnel; or, triggering alarms by listening for abnormal events in the cluster, allowing operations personnel to determine whether node isolation is necessary. These methods generally rely on node readiness assessments, external monitoring systems, or manual intervention, and cannot serve as an independent basis for determining node circuit breaker failures.

[0004] Therefore, how to automatically identify failure anomalies in the admission phase within a node and achieve automatic circuit breaking and self-healing of the node without manual intervention, so as to block the impact of failure storms on the control surface, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the embodiments of this application provide a node anomaly handling method, apparatus, computer equipment and storage medium.

[0006] In a first aspect, embodiments of this application provide a node anomaly handling method, including: For each node in the cluster, obtain the number of times the target abnormal state occurs in the work unit of each node within the sliding time window, where the work unit is the smallest running entity scheduled for execution in the cluster; Nodes whose frequency exceeds a first threshold are identified as abnormal nodes, and node isolation operations are performed on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. After performing the node isolation operation, the abnormal node is assessed for recovery based on preset recovery conditions, and the isolation status of the abnormal node is lifted if the preset recovery conditions are met.

[0007] In one possible implementation, obtaining the number of times the target abnormal state occurs in the working unit of each node within a sliding time window for each node in the cluster includes: The state change events of each node's working unit are monitored. If the state change event indicates that the working unit's state has changed to a failure state and the failure is due to an admission phase error, it is determined that the working unit has encountered a target abnormal state. Based on the node identification information of each node, the number of times each node experiences the target abnormal state within the sliding time window is counted.

[0008] In one possible implementation, performing node isolation operations on the abnormal node to prevent new work units from being scheduled to the abnormal node includes: Upon receiving an isolation command for the abnormal node, obtain the node identification information of the abnormal node; Based on the node identification information, a node taint addition instruction is generated, and a node taint is added to the abnormal node according to the node taint addition instruction, so as to prevent the scheduling system from scheduling new work units to the abnormal node when performing work unit scheduling, and to maintain the running status of existing work units on the abnormal node.

[0009] In one possible implementation, the step of determining the recovery of the abnormal node based on preset recovery conditions, and releasing the isolation state of the abnormal node if the preset recovery conditions are met, includes: Obtain the isolation duration of the abnormal node while it is in an isolated state; The isolation duration is compared with a preset cooldown time. If the isolation duration is greater than the preset cooldown time, it is determined that the preset recovery condition is met. Perform a node taint removal operation on the abnormal node to remove its isolation state and enable it to resume participation in work unit scheduling.

[0010] In one possible implementation, after the isolation state of the abnormal node is lifted, the method further includes: If the abnormal node re-enters the target abnormal state during operation and the node isolation operation is re-executed, record the number of times the abnormal node is isolated. The preset cooldown time is adjusted based on the number of isolation operations, so that the preset cooldown time increases with the number of isolation operations.

[0011] In one possible implementation, the method further includes: The abnormal state type of the work unit is determined based on the reasons for the failure; For each of the aforementioned abnormal state types, a corresponding trigger threshold is set, and a mapping relationship between abnormal state types and node isolation strategies is established. If the number of occurrences of any abnormal state type exceeds the corresponding trigger threshold, the node is determined to be an abnormal node. Based on the abnormal state type, a target node isolation strategy is determined from the mapping relationship, and node isolation operations are performed on the abnormal node according to the target node isolation strategy, including: In the case where the abnormal state type is an admission phase error, the target node isolation strategy is determined to be: add a node taint to the abnormal node to prevent new work units from being scheduled to the abnormal node; When the abnormal state type is a resource shortage error, the target node isolation strategy is determined to be: limit the number of work units scheduled to the abnormal node; When the abnormal state type is runtime error, the target node isolation strategy is determined to be: reduce the work unit scheduling priority of the abnormal node.

[0012] In one possible implementation, the method further includes: Obtain other nodes that have scheduling relationships with the abnormal node; If, within a preset time range, another node experiences a target abnormal state of the same type as the abnormal node, the scheduling priority of the work unit of that other node is reduced.

[0013] Secondly, embodiments of this application provide a node anomaly handling apparatus, comprising: The acquisition module is used to acquire, for each node in the cluster, the number of times the target abnormal state occurs in the work unit of each node within the sliding time window, wherein the work unit is the smallest running entity that is scheduled to be executed in the cluster; The first processing module is used to determine that nodes whose number of occurrences exceeds a first threshold are abnormal nodes, and to perform node isolation operations on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. The second processing module is used to determine the recovery of the abnormal node based on preset recovery conditions after performing the node isolation operation, and to release the isolation state of the abnormal node if the preset recovery conditions are met.

[0014] Thirdly, embodiments of this application provide a computer device, including: a processor and a memory, wherein the processor is configured to execute a node exception handling program stored in the memory to implement the node exception handling method described in any one of the first aspects above.

[0015] Fourthly, embodiments of this application provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the node exception handling method described in any of the first aspects above.

[0016] The node anomaly handling scheme provided in this application, for each node in the cluster, obtains the number of times the target abnormal state occurs in the work unit of each node within a sliding time window. The work unit is the smallest running entity scheduled for execution in the cluster. Nodes with the number of occurrences exceeding a first threshold are identified as abnormal nodes, and node isolation operations are performed on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. After performing the node isolation operation, the abnormal nodes are judged for recovery based on preset recovery conditions, and the isolation state of the abnormal nodes is lifted if the preset recovery conditions are met. Thus, by obtaining the number of times the target abnormal state occurs in the work unit of each node within a sliding time window, the failure of the admission phase abnormal nodes can be accurately identified, solving the problem that the existing node ready state signal cannot detect this type of failure; by performing isolation operations on abnormal nodes to prevent the scheduling of new work units, automatic circuit breaking is achieved, preventing a large number of Pod reconstructions and a surge in control plane load caused by node anomalies; by automatically lifting isolation through preset recovery conditions, a closed-loop self-healing mechanism is formed without manual intervention, thereby comprehensively improving the stability and autonomy of the cluster. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a node anomaly handling method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a node anomaly handling device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0020] Figure 1 This is a flowchart illustrating a node anomaly handling method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method specifically includes: S11. For each node in the cluster, obtain the number of times the target abnormal state occurs in the working unit of each node within the sliding time window, wherein the working unit is the smallest running entity scheduled for execution in the cluster.

[0021] The node anomaly handling method provided in this application can be applied to the production environment of container orchestration platforms, and is particularly suitable for large-scale Kubernetes clusters, critical business system operating environments, automated operation and maintenance scenarios, and multi-tenant cluster environments. In large-scale clusters, with a large number of nodes, manual inspection is difficult to detect node access anomalies in real time. This application can automatically identify and isolate faulty nodes, quickly trigger circuit breakers in the early stages of node anomalies to prevent the spread of faults and maintain business continuity, and provides node-level self-healing capabilities, reducing manual intervention and improving operation and maintenance efficiency. In multi-tenant environments, when an access anomaly of a certain node may affect the operation of multiple tenants' work units, this application can quickly isolate the abnormal node, ensuring fault isolation between tenants.

[0022] The execution entity of this application is a node monitoring controller, which is deployed in a computing device, including but not limited to servers, cloud hosts, virtual machines, containerized runtime environments, or any electronic device with computing capabilities. This node monitoring controller can run as an independent control component within the control plane of a Kubernetes cluster, obtaining the status of work units and executing node operations by accessing the cluster interface; it can also be deployed outside the cluster control plane, achieving the same monitoring and control functions by remotely calling the cluster interface; or it can be integrated into the scheduler or controller manager as an extension or plugin to existing Kubernetes control plane components. The node monitoring controller internally includes the following functional modules: a work unit status monitoring module, responsible for monitoring work unit status change events; an anomaly statistics module, responsible for counting the number of anomalies by node; a circuit breaker decision module, responsible for determining whether to trigger node isolation; a node control module, responsible for performing node taint addition and removal operations; and a cooldown recovery module, responsible for implementing an automatic recovery mechanism based on time thresholds. Through the collaborative work of these modules, this application can automatically complete the identification, isolation, and recovery of abnormal nodes without manual intervention.

[0023] In this embodiment, a work unit refers to the smallest running entity scheduled for execution in the cluster. In the Kubernetes container orchestration platform, a work unit corresponds to a container group (Pod), which is a combination of one or more containers. A sliding time window is a dynamic time-range statistical mechanism that moves continuously with the current time, always counting the number of events within the most recent time period. A target abnormal state refers to a specific state where a work unit is in a failed state and the failure is due to an admission phase error. An admission phase error refers to the error type in which a work unit is rejected or fails during the node admission control phase, including but not limited to unexpected admission errors in Kubernetes.

[0024] In practice, the node monitoring controller establishes a long-lived connection with the cluster control interface through the resource listening mechanism provided by the Kubernetes cluster (such as the Watch API) to monitor the state change events of all work units within the cluster in real time. When a state change of any work unit is detected, the node monitoring controller parses the state change event, extracts the work unit's state information (such as the Phase field) and failure reason information (such as the Reason field), and obtains the node identifier information (such as the NodeName field) of the work unit. When the state of the work unit changes to a preset target abnormal state, the state change event is recorded. Simultaneously, the node monitoring controller maintains an independent counter for each node, using a sliding time window method for statistical analysis.

[0025] Specifically, the node monitoring controller sets a preset time window length (e.g., 1 minute) and a preset anomaly trigger threshold (e.g., 10 times). Whenever an event indicating a target abnormal state is detected, the node monitoring controller locates the corresponding node's counter based on the node identifier information carried by the event, records the event in the counter, and records the event's timestamp. The node monitoring controller continuously calculates the total number of events recorded in each node's counter within a time window length (i.e., the current time minus the time window length) preceding the current time point. The time window slides continuously over time, always maintaining the number of abnormal events within the most recent time window, rather than a cumulative value over a fixed time period. The abnormal statistics for each node within the sliding time window are maintained in real time and used as input for subsequent circuit breaker decisions.

[0026] In one possible implementation, the state change events of each node's work unit are monitored. When the state change event indicates that the work unit's state has changed to a failure state and the failure is due to an admission phase error, it is determined that the work unit has encountered a target abnormal state. Based on the node identification information of each node, the number of times each node has encountered a target abnormal state within the sliding time window is counted.

[0027] In this embodiment, the state change events of the work units on each node are continuously monitored. State change events refer to event information generated by the cluster control plane that characterizes the state changes of the work unit's lifecycle, such as creation, scheduling, running, and failure state change events.

[0028] Upon receiving a state change event, the event content is parsed to determine whether the current state of the work unit has changed from a non-failure state to a failure state. A failure state refers to an abnormal termination state entered by a work unit due to its failure to successfully complete the creation or execution process. Simultaneously, the failure reason field is parsed to determine whether the failure belongs to an admission phase error. An admission phase error refers to a failure that occurs before or during the scheduling of a work unit to a node due to reasons such as insufficient resources (insufficient CPU or memory), unschedulable nodes, or mismatched node labels or affinity. When both the state change to a failure state and the failure reason is an admission phase error, the work unit is determined to have entered the target abnormal state.

[0029] Abnormal events are attributed to the node identification information (e.g., node name or node ID) of the node to which the work unit belongs. Using node identification information as the statistical dimension, a sliding time window mechanism is used to aggregate and statistically analyze abnormal events.

[0030] For example, in a cluster containing multiple nodes, if the sliding time window is set to 1 minute, and node A experiences 12 consecutive work unit creation failure events due to insufficient resources within a certain time window, then these 12 events are all identified as target abnormal states and are attributed to node A for cumulative statistics, thus obtaining the number of target abnormal states of node A within that time window as 12.

[0031] S12. Determine the nodes whose frequency exceeds the first threshold as abnormal nodes, and perform node isolation operations on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes.

[0032] In this embodiment, for each node, the number of target abnormal states within the current sliding time window is obtained, and this number is compared with a pre-set first threshold. The first threshold represents the upper limit of the number of abnormal states that a node can tolerate per unit time, and can be configured based on system operating experience or historical statistical data. When the number of target abnormal states corresponding to a node exceeds the first threshold, the node is determined to be an abnormal node.

[0033] Perform node isolation operations on abnormal nodes and apply scheduling restriction policies to abnormal nodes based on the cluster control interface to prevent new work units from being scheduled to the abnormal nodes.

[0034] In actual execution, it can be determined whether the node is already in an isolated state before performing the node isolation operation. If it is already in an isolated state, the repetitive execution is skipped to avoid repeatedly applying node taints or sending control commands, thereby reducing system overhead and ensuring the consistency of node state. If it is not in an isolated state, the node isolation operation is performed on the abnormal node.

[0035] In one possible implementation, a node isolation operation is performed on the abnormal node to prevent new work units from being scheduled to the abnormal node, including: Upon receiving an isolation instruction for an abnormal node, the system obtains the node identification information of the abnormal node; generates a node taint addition instruction based on the node identification information; and adds node taints to the abnormal node according to the node taint addition instruction, so as to prevent the scheduling system from scheduling new work units to the abnormal node when performing work unit scheduling, and to maintain the running status of existing work units on the abnormal node.

[0036] In this embodiment, after identifying an abnormal node, the system issues an isolation command for the abnormal node. Upon receiving the isolation command, the system parses the node identification information of the abnormal node carried in the isolation command. A node taint addition command is generated based on the node identification information. A node taint is a scheduling constraint mark applied to a node to influence the scheduling system's scheduling decisions for work units; a node taint typically includes a taint key, a taint value, and a taint effect.

[0037] Taint effects can be set to a "disallow scheduling" type, preventing work units that are not explicitly declared to tolerate the taint from being scheduled to that node. In practice, standardized taint key-value pairs can be pre-configured, such as taint keys identified by abnormal nodes, and corresponding node taint addition instructions can be generated by combining the taint effect indicating "disallow scheduling".

[0038] After generating the node taint addition instruction, the node taint is added to the abnormal node by calling the cluster control interface. Specifically, a node update request can be sent to the cluster control plane to write the node taint into the target node's attributes, thereby completing the taint marking of the abnormal node.

[0039] After node taints are added, the scheduling system automatically checks for node taints that are intolerable to the target work unit during work unit scheduling. When an abnormal node with a taint is detected, the scheduling system will skip that node and will not schedule new work units to it, thus isolating the abnormal node. Furthermore, since node taints only affect newly scheduled work units, they do not affect work units already running on the abnormal node, maintaining the existing work unit's operational status and avoiding interruptions to existing services.

[0040] For example, when node A is identified as an abnormal node, the system generates a node taint addition instruction based on node A's node identification information, which includes a preset taint key and value and a disabling scheduling effect, and adds the taint to node A. After that, the scheduling system will no longer select node A when allocating new work units, thereby achieving isolation control of node A, while the work units that were originally running on node A can continue to run.

[0041] S13. After performing node isolation operation, perform recovery judgment on abnormal nodes based on preset recovery conditions, and release the isolation status of abnormal nodes if the preset recovery conditions are met.

[0042] In this embodiment, after the node monitoring controller performs an isolation operation on an abnormal node, it monitors the node's status information and isolation time in real time. When the monitored status information and / or isolation time meet preset recovery conditions, the isolation status of the abnormal node is lifted. For example, the isolation start time is recorded, and the duration of the node's isolation is calculated. The node monitoring controller presets a cooldown time threshold (e.g., 5 minutes) and periodically checks the duration of the node's isolation. When the duration reaches or exceeds the cooldown time threshold, the node monitoring controller determines that the node meets the preset recovery conditions. A node taint removal command is generated, and the node taint is removed from the node through the cluster control interface. After the node taint is removed, the cluster scheduler resumes using the node as a schedulable target, allowing new work units to be scheduled to run on the node.

[0043] In one possible implementation, the isolation duration of the abnormal node in the isolation state is obtained; the isolation duration is compared with a preset cooldown time, and if the isolation duration is greater than the preset cooldown time, it is determined that the preset recovery condition is met; a node taint removal operation is performed on the abnormal node to release the isolation state of the abnormal node so that the abnormal node can resume participating in work unit scheduling.

[0044] In this embodiment, the isolation duration of an abnormal node in an isolated state is obtained. The isolation duration refers to the time interval from the moment the node isolation operation (i.e., adding a node taint) is performed on the abnormal node to the current moment. In specific implementation, the corresponding isolation start timestamp can be recorded when adding a node taint to the abnormal node, and the isolation duration can be calculated based on the current time and the isolation start timestamp during subsequent periodic tasks or event triggering processes.

[0045] The isolation duration is compared with a pre-set cooldown time. The pre-set cooldown time limits the minimum period during which an abnormal node must remain in isolation after being isolated, to prevent the node from frequently entering and exiting isolation and recovery states within a short period. The pre-set cooldown time can be a fixed value or dynamically adjusted based on the historical number of times the abnormal node has been isolated.

[0046] When the isolation duration exceeds the preset cooldown time, the abnormal node is determined to meet the preset recovery conditions. Further health status checks can be performed on the abnormal node, such as checking whether the node still exhibits the target abnormal state within the current or most recent sliding time window, or checking whether the node's resource usage status has returned to normal. Only when the above additional conditions are met is the abnormal node ultimately determined to meet the recovery conditions.

[0047] After confirming that the preset recovery conditions are met, a node taint removal operation is performed on the abnormal node to remove its isolation state. Specifically, this can be achieved by calling the cluster control interface to send a node update command to the abnormal node, removing the previously added node taint from its attributes, thereby removing the scheduling restrictions on that node. After the node taint removal is complete, the scheduling system will no longer be restricted by the node taint when scheduling work units, allowing new work units to be rescheduled to that node, thus enabling the abnormal node to resume participation in work unit scheduling.

[0048] In one possible implementation, after releasing the isolation status of the abnormal node, the method further includes: If an abnormal node re-enters the target abnormal state during operation and the node isolation operation is re-executed, the number of times the abnormal node is isolated is recorded; the preset cooldown time is adjusted based on the number of isolation operations so that the preset cooldown time increases with the number of isolation operations.

[0049] In this embodiment, when an abnormal node, after being released from isolation and resuming participation in work unit scheduling, experiences the target abnormal state again during subsequent operation and is subjected to node isolation operation again, the number of times the abnormal node is subjected to node isolation operation is recorded. The number of isolation operations is used to characterize the frequency with which the node is identified as an abnormal node and isolated within a certain time range.

[0050] In practical implementation, a corresponding isolation counter can be maintained for each node, and the counter can be incremented each time a node isolation operation is performed on that node. Simultaneously, a timestamp can be used to record the time of each isolation, allowing for periodic reset or decay of the isolation count when needed. The preset cooldown time is adjusted based on the number of isolation operations, gradually increasing with the number of isolation operations. The preset cooldown time limits the duration an abnormal node must remain isolated after each isolation. In one implementation, an exponential backoff strategy can be used to adjust the preset cooldown time; that is, based on the initial cooldown time, the next cooldown time is set to a preset multiple of the previous cooldown time, such as twice, each time a node is re-isolated. In another implementation, a linear growth strategy or a piecewise function strategy can be used to adjust the cooldown time to meet the stability requirements of different business scenarios. After adjustment, the updated preset cooldown time is applied to subsequent node recovery determination processes, requiring abnormal nodes with more isolation counts to have longer recovery waiting times, thereby reducing the system instability risk caused by their repeated participation in scheduling within a short period.

[0051] For example, in a scenario with an initial cooldown time of 5 minutes, when node A is first isolated, its cooldown time is 5 minutes; when node A recovers and is isolated again, its cooldown time is adjusted to 10 minutes; if it is isolated again, it is further adjusted to 20 minutes. This approach effectively suppresses frequent fluctuations in node performance under abnormal conditions, improving the overall stability of the cluster and the reliability of scheduling.

[0052] In one possible implementation, the abnormal state type of the work unit is determined based on the cause of failure; a corresponding trigger threshold is set for each abnormal state type, and a mapping relationship between the abnormal state type and the node isolation policy is established; when the occurrence frequency of any abnormal state type exceeds the corresponding trigger threshold, the node is determined to be an abnormal node; the target node isolation policy is determined from the mapping relationship according to the abnormal state type, and node isolation operation is performed on the abnormal node according to the target node isolation policy, including: when the abnormal state type is an admission phase error, the target node isolation policy is determined to be: adding node taints to the abnormal node to prevent new work units from being scheduled to the abnormal node; when the abnormal state type is a resource insufficiency error, the target node isolation policy is determined to be: limiting the number of work units scheduled to the abnormal node; when the abnormal state type is a runtime error, the target node isolation policy is determined to be: reducing the scheduling priority of the work units of the abnormal node.

[0053] In this embodiment, the abnormal states of work units are classified based on the reasons for their failure, resulting in multiple abnormal state types. The reason for failure refers to the error message or status code carried by the work unit when it enters a failure state during creation or operation; the abnormal state type is used to categorize different types of failure reasons.

[0054] Abnormal states can be categorized into admission phase errors, resource insufficiency errors, and runtime errors. Admission phase errors refer to failures during the work unit scheduling or creation phase due to unmet node conditions or scheduling constraints. Resource insufficiency errors refer to failures caused by insufficient CPU, memory, or other resources of the node to meet the operational requirements of the work unit (e.g., the node's CPU utilization rate is greater than a preset utilization rate threshold, or the remaining memory capacity is less than a preset memory threshold). Runtime errors refer to failures caused by program abnormalities or environmental problems after the work unit has been successfully scheduled and running.

[0055] For each abnormal state type, a corresponding trigger threshold is set, and a mapping relationship between the abnormal state type and the node isolation policy is established. In practice, the mapping relationship can be maintained through configuration files or policy tables. For example, a policy mapping table can be built using the abnormal state type as the key and the node isolation policy as the value.

[0056] During operation, when the number of occurrences of any abnormal state type within the current sliding time window exceeds its corresponding trigger threshold, the node corresponding to that abnormal state type is determined to be an abnormal node. Based on the abnormal state type, a target node isolation strategy is determined from the mapping relationship, and node isolation operations are performed on the abnormal node according to the target node isolation strategy.

[0057] In the case of an abnormal state type of admission phase error, the target node isolation strategy is determined to add a node taint to the abnormal node. By applying a node taint to the abnormal node, the scheduling system automatically avoids the node when scheduling work units, thereby preventing new work units from being scheduled to the abnormal node; When the abnormal status type is "resource insufficiency error," the target node isolation strategy is determined to be to limit the number of work units scheduled to the abnormal node. This can be achieved by adjusting scheduling weights, setting a maximum schedulable capacity for the node, or introducing rate limiting control strategies to reduce the number of work units scheduled to that node, thereby alleviating resource pressure. When the abnormal state type is runtime error, the target node isolation strategy is determined to reduce the scheduling priority of the abnormal node's work unit, so that the scheduling system will prioritize other healthy nodes when making scheduling decisions, thereby reducing the probability of new work units being scheduled to the abnormal node.

[0058] For example, in a scenario with a sliding time window of 1 minute, if node A experiences 12 admission phase errors within that time window, exceeding the corresponding threshold of 10 errors, then the isolation strategy of adding node taints to node A is determined through policy mapping. If node B experiences a large number of resource shortage errors, then a scheduling quantity limit strategy is implemented for node B. If node C frequently experiences runtime errors, then node C's scheduling priority is reduced. Through this approach, differentiated isolation measures can be taken for different types of anomalies, thereby ensuring system stability while avoiding the resource utilization degradation caused by excessive isolation.

[0059] In one possible implementation, other nodes that have a scheduling relationship with the abnormal node are obtained; if other nodes experience a target abnormal state of the same abnormal state type as the abnormal node within a preset time range, the scheduling priority of the work unit of the other nodes is reduced.

[0060] In this embodiment, after identifying a node as an abnormal node, other nodes with scheduling relationships with the abnormal node are obtained. Scheduling relationships characterize the correlation between nodes at the work unit scheduling level. Scheduling relationships can be determined based on historical scheduling information of work units. For example, by statistically analyzing work units scheduled to different nodes within a preset time range using the same scheduling strategy, the same business application, or the same task batch, a set of nodes sharing a common scheduling source can be identified. In another implementation, nodes with similar scheduling attributes can be identified based on node tags, node resource types, or the resource pool to which the node belongs, thus considering these nodes as having scheduling relationships.

[0061] Within a preset time frame, monitor the target abnormal status of other nodes. During the monitoring process, determine whether other nodes exhibit the same target abnormal status type as the abnormal node. Here, "same abnormal status type" means that the abnormal status type obtained based on the failure cause classification is consistent, for example, both are resource shortage errors or both are admission phase errors.

[0062] When any other node is detected to have the same target abnormal state as the abnormal node within a preset time range, a scheduling priority reduction operation is performed on that other node. Reducing the scheduling priority of a work unit means decreasing the probability of that node being selected when the scheduling system allocates work units. This can be achieved by adjusting node weight parameters, lowering node score values, or introducing penalty factors into the scheduling strategy, so that the scheduling system prioritizes other healthy nodes during the candidate node selection process. Therefore, preventative scheduling control can be implemented for potentially abnormal nodes before the anomaly spreads on a large scale, thereby reducing the risk of the anomaly propagating among multiple nodes and improving the overall stability of the cluster.

[0063] For example, in a cluster, if node A is identified as an abnormal node with an insufficient resource error, the system identifies nodes B and C as belonging to the same service scheduling batch as node A based on historical scheduling records, thus determining that nodes B and C are associated nodes. Within a preset time range of 1 minute thereafter, if node B is also detected to have an insufficient resource error, the scheduling priority of node B is reduced, so that the scheduling system prioritizes other nodes with sufficient resources, thereby preventing node B from becoming an abnormal node.

[0064] The node anomaly handling method provided in this application classifies and statistically analyzes the abnormal states of work units and monitors abnormal node behavior in real time based on a sliding time window, achieving rapid identification and accurate judgment of abnormal nodes. Simultaneously, by introducing differentiated isolation strategies such as node taints, scheduling priority adjustments, and scheduling quantity limits, it effectively blocks abnormal nodes from participating in new work unit scheduling, preventing anomaly propagation. Furthermore, by combining an adaptive cooling mechanism based on the number of isolation cycles and an exponential backoff strategy, it reduces system jitter caused by frequent node isolation and recovery, and proactively intervenes in potential abnormal nodes through an anomaly propagation suppression mechanism for associated nodes, thereby comprehensively improving the stability of cluster scheduling, resource utilization efficiency, and system reliability.

[0065] This application also provides a node anomaly handling system, including: a Pod status monitoring module, an anomaly statistics module, a circuit breaker decision module, a node control module, and a cooldown recovery module. These five modules work collaboratively in a pipelined manner: the Pod status monitoring module collects anomaly signals; the anomaly statistics module aggregates and identifies candidate anomaly nodes; the circuit breaker decision module comprehensively determines whether isolation should be triggered; the node control module executes the isolation operation; and the cooldown recovery module times the event and automatically releases the isolation after the cooldown period. If the node recovers and triggers isolation again, the cooldown recovery module dynamically adjusts the cooldown time using an exponential backoff strategy, forming a complete closed-loop self-healing mechanism.

[0066] The Pod status monitoring module is responsible for continuously monitoring Pod status change events in the Kubernetes cluster. This module obtains Pod creation events and status update events in real time through the resource monitoring mechanism provided by Kubernetes. When it detects that the Pod status has changed to a failed state and the failure reason is UnexpectedAdmissionError or other admission phase errors, the Pod status monitoring module will report the abnormal event to the exception statistics module, along with the corresponding node identification information.

[0067] The anomaly statistics module is responsible for aggregating and statistically analyzing anomaly events from the Pod status monitoring module. This module maintains anomaly count information at the node level and uses a sliding time window to count the number of anomalies occurring on a given node within a specified time range. For example, if the statistical time window is set to 1 minute and the anomaly trigger threshold is 10 times, when a node experiences more than the threshold number of anomaly Pods within that time window, that node will be marked as an anomaly node candidate, and this information will be passed to the circuit breaker decision module.

[0068] The circuit breaker decision module is responsible for determining whether to perform isolation operations on nodes. When it receives abnormal node information reported by the abnormal statistics module, this module executes the following judgment logic: determining whether the current node is already in an isolated state, determining whether the number of abnormal events exceeds a set threshold, and determining whether the abnormality belongs to the target abnormal type. When the above conditions are met, the circuit breaker decision module triggers the node control module to perform isolation operations on the target node.

[0069] The node control module is responsible for performing actual control operations on Kubernetes node objects. When it receives an isolation command from the circuit breaker decision module, this module adds a node taint to the target node, thereby preventing new Pods from being scheduled to that node. The purpose of node taints includes preventing new Pods from being scheduled to that node, keeping the running state of existing Pods unaffected, and preventing abnormal nodes from continuing to participate in scheduling. In this way, it quickly blocks the continuous scheduling failure problem caused by abnormal nodes.

[0070] The cooling-off recovery module is responsible for implementing the automatic recovery mechanism for nodes. When a node is tainted, it enters an isolated state, and the module records the isolation time and the corresponding number of isolations. During the isolation period, since new workloads will not be scheduled to that node, the system no longer judges the node status by adding new abnormal events, but instead adopts a time-window-based recovery strategy. The cooling-off recovery module periodically checks the node isolation time to determine if the current isolation duration has reached the preset cooldown time. If it has, the node taint is automatically removed, restoring the node's scheduling capability. After the node regains scheduling capability, if a similar abnormal event occurs again during subsequent operation and triggers node isolation again, the module records the number of isolations for that node and appropriately extends the next cooldown time.

[0071] Figure 2 This is a schematic diagram of a node anomaly handling device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the device specifically includes: The acquisition module 21 is used to acquire, for each node in the cluster, the number of times the target abnormal state occurs in the working unit of each node within the sliding time window, wherein the working unit is the smallest running entity that is scheduled to be executed in the cluster; The first processing module 22 is used to determine that the nodes whose number of occurrences exceeds a first threshold are abnormal nodes, and to perform node isolation operations on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. The second processing module 23 is used to determine the recovery of the abnormal node based on preset recovery conditions after performing the node isolation operation, and to release the isolation state of the abnormal node if the preset recovery conditions are met.

[0072] In one possible implementation, the acquisition module is specifically used to listen to the state change events of the working unit of each node, and when the state change event indicates that the state of the working unit has changed to a failure state and the reason for the failure is an admission phase error, the module determines that the working unit has a target abnormal state. Based on the node identification information of each node, the number of times each node experiences the target abnormal state within the sliding time window is counted.

[0073] In one possible implementation, the first processing module is specifically configured to obtain the node identification information of the abnormal node when it receives an isolation instruction for the abnormal node. Based on the node identification information, a node taint addition instruction is generated, and a node taint is added to the abnormal node according to the node taint addition instruction, so as to prevent the scheduling system from scheduling new work units to the abnormal node when performing work unit scheduling, and to maintain the running status of existing work units on the abnormal node.

[0074] In one possible implementation, the second processing module is specifically used to obtain the isolation duration of the abnormal node in the isolation state; The isolation duration is compared with a preset cooldown time. If the isolation duration is greater than the preset cooldown time, it is determined that the preset recovery condition is met. Perform a node taint removal operation on the abnormal node to remove its isolation state and enable it to resume participation in work unit scheduling.

[0075] In one possible implementation, the second processing module is further configured to record the number of times the abnormal node is isolated when the abnormal node re-enters the target abnormal state during operation and is re-performed the node isolation operation. The preset cooldown time is adjusted based on the number of isolation operations, so that the preset cooldown time increases with the number of isolation operations.

[0076] In one possible implementation, the first processing module is further configured to determine the abnormal state type of the work unit based on the failure reason; For each of the aforementioned abnormal state types, a corresponding trigger threshold is set, and a mapping relationship between abnormal state types and node isolation strategies is established. If the number of occurrences of any abnormal state type exceeds the corresponding trigger threshold, the node is determined to be an abnormal node. Based on the abnormal state type, a target node isolation strategy is determined from the mapping relationship, and node isolation operations are performed on the abnormal node according to the target node isolation strategy, including: In the case where the abnormal state type is an admission phase error, the target node isolation strategy is determined to be: add a node taint to the abnormal node to prevent new work units from being scheduled to the abnormal node; When the abnormal state type is a resource shortage error, the target node isolation strategy is determined to be: limit the number of work units scheduled to the abnormal node; When the abnormal state type is runtime error, the target node isolation strategy is determined to be: reduce the work unit scheduling priority of the abnormal node.

[0077] In one possible implementation, the first processing module is further configured to acquire other nodes that have a scheduling association with the abnormal node; If, within a preset time range, another node experiences a target abnormal state of the same type as the abnormal node, the scheduling priority of the work unit of that other node is reduced.

[0078] The node anomaly handling device provided in this embodiment can be as follows: Figure 2 The apparatus shown can perform, for example Figure 1 All steps of the method, thus achieving Figure 1 For details on the technical effects of the method shown, please refer to [link / reference]. Figure 1 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0079] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3 The computer device 300 shown includes at least one processor 301, a memory 302, at least one network interface 304, and other user interfaces 303. The various components in the computer device 300 are coupled together via a bus system 305. It is understood that the bus system 305 is used to implement communication between these components. In addition to a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general designated all buses as Bus System 305.

[0080] The user interface 303 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0081] It is understood that the memory 302 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 302 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0082] In some implementations, memory 302 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 3021 and application programs 3022.

[0083] The operating system 3021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 3022 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in application program 3022.

[0084] In this embodiment, by calling the program or instructions stored in memory 302, specifically the program or instructions stored in application program 3022, processor 301 executes the method steps provided in each method embodiment, including, for example: For each node in the cluster, obtain the number of times the target abnormal state occurs in the work unit of each node within the sliding time window, where the work unit is the smallest running entity scheduled for execution in the cluster; Nodes whose frequency exceeds a first threshold are identified as abnormal nodes, and node isolation operations are performed on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. After performing the node isolation operation, the abnormal node is assessed for recovery based on preset recovery conditions, and the isolation status of the abnormal node is lifted if the preset recovery conditions are met.

[0085] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 301. Processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 301 or by instructions in the form of software. The processor 301 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 302. The processor 301 reads the information in memory 302 and, in conjunction with its hardware, completes the steps of the above method.

[0086] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0087] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0088] The computer device provided in this embodiment may be as follows: Figure 3 The device shown can perform, for example Figure 1 All steps of the method, thus achieving Figure 1 For details on the technical effects of the method shown, please refer to [link / reference]. Figure 1 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0089] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0090] When one or more programs in the storage medium can be executed by one or more processors to achieve the above-described method of execution on the device side.

[0091] The processor is used to execute a program stored in memory to implement the steps of the method performed on the device side: For each node in the cluster, obtain the number of times the target abnormal state occurs in the work unit of each node within the sliding time window, where the work unit is the smallest running entity scheduled for execution in the cluster; Nodes whose frequency exceeds a first threshold are identified as abnormal nodes, and node isolation operations are performed on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. After performing the node isolation operation, the abnormal node is assessed for recovery based on preset recovery conditions, and the isolation status of the abnormal node is lifted if the preset recovery conditions are met.

[0092] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0094] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for handling node anomalies, characterized in that, include: For each node in the cluster, obtain the number of times the target abnormal state occurs in the work unit of each node within the sliding time window, where the work unit is the smallest running entity scheduled for execution in the cluster; Nodes whose frequency exceeds a first threshold are identified as abnormal nodes, and node isolation operations are performed on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. After performing the node isolation operation, the abnormal node is assessed for recovery based on preset recovery conditions, and the isolation status of the abnormal node is lifted if the preset recovery conditions are met.

2. The method according to claim 1, characterized in that, The step of obtaining the number of times the target abnormal state occurs in the working unit of each node within the sliding time window for each node in the cluster includes: The state change events of each node's working unit are monitored. If the state change event indicates that the working unit's state has changed to a failure state and the failure is due to an admission phase error, it is determined that the working unit has encountered a target abnormal state. Based on the node identification information of each node, the number of times each node experiences the target abnormal state within the sliding time window is counted.

3. The method according to claim 1, characterized in that, The step of performing node isolation operations on the abnormal node to prevent new work units from being scheduled to the abnormal node includes: Upon receiving an isolation command for the abnormal node, obtain the node identification information of the abnormal node; Based on the node identification information, a node taint addition instruction is generated, and a node taint is added to the abnormal node according to the node taint addition instruction, so as to prevent the scheduling system from scheduling new work units to the abnormal node when performing work unit scheduling, and to maintain the running status of existing work units on the abnormal node.

4. The method according to claim 1, characterized in that, The step of determining the recovery of the abnormal node based on preset recovery conditions, and releasing the isolation state of the abnormal node when the preset recovery conditions are met, includes: Obtain the isolation duration of the abnormal node while it is in an isolated state; The isolation duration is compared with a preset cooldown time. If the isolation duration is greater than the preset cooldown time, it is determined that the preset recovery condition is met. Perform a node taint removal operation on the abnormal node to remove its isolation state and enable it to resume participation in work unit scheduling.

5. The method according to claim 4, characterized in that, After the isolation status of the abnormal node is lifted, the method further includes: If the abnormal node re-enters the target abnormal state during operation and the node isolation operation is re-executed, record the number of times the abnormal node is isolated. The preset cooldown time is adjusted based on the number of isolation operations, so that the preset cooldown time increases with the number of isolation operations.

6. The method according to claim 2, characterized in that, The method further includes: The abnormal state type of the work unit is determined based on the reasons for the failure; For each of the aforementioned abnormal state types, a corresponding trigger threshold is set, and a mapping relationship between abnormal state types and node isolation strategies is established. If the number of occurrences of any abnormal state type exceeds the corresponding trigger threshold, the node is determined to be an abnormal node. Based on the abnormal state type, a target node isolation strategy is determined from the mapping relationship, and node isolation operations are performed on the abnormal node according to the target node isolation strategy, including: In the case where the abnormal state type is an admission phase error, the target node isolation strategy is determined to be: add a node taint to the abnormal node to prevent new work units from being scheduled to the abnormal node; When the abnormal state type is a resource shortage error, the target node isolation strategy is determined to be: limit the number of work units scheduled to the abnormal node; When the abnormal state type is runtime error, the target node isolation strategy is determined to be: reduce the work unit scheduling priority of the abnormal node.

7. The method according to claim 1, characterized in that, The method further includes: Obtain other nodes that have scheduling relationships with the abnormal node; If, within a preset time range, another node experiences a target abnormal state of the same type as the abnormal node, the scheduling priority of the work unit of that other node is reduced.

8. A node anomaly handling device, characterized in that, include: The acquisition module is used to acquire, for each node in the cluster, the number of times the target abnormal state occurs in the work unit of each node within the sliding time window, wherein the work unit is the smallest running entity that is scheduled to be executed in the cluster; The first processing module is used to determine that nodes whose number of occurrences exceeds a first threshold are abnormal nodes, and to perform node isolation operations on the abnormal nodes to prevent new work units from being scheduled to the abnormal nodes. The second processing module is used to determine the recovery of the abnormal node based on preset recovery conditions after performing the node isolation operation, and to release the isolation state of the abnormal node if the preset recovery conditions are met.

9. A computer device, characterized in that, include: A processor and a memory, the processor being configured to execute a node exception handling program stored in the memory to implement the node exception handling method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the node exception handling method according to any one of claims 1 to 7.

Images