Cluster fault prediction and self-healing method, device and equipment based on AI

Abstract

The embodiment of the invention relates to the technical field of operation and maintenance, and discloses an AI-based cluster fault prediction and self-healing method, device and equipment, and the method comprises the steps: obtaining metrics feature data, log feature data and event feature data; performing prediction processing on the metrics feature data by using the target time sequence prediction model to obtain a cluster resource prediction value and fault information; based on the fault information and the cluster resource prediction value, determining to use a target XGBoost to carry out root cause positioning processing on the metrics feature data, the log feature data and the event feature data to obtain a fault root cause; determining a fault self-healing strategy based on the fault root cause, the cluster resource prediction value, a preset static strategy library and a dynamic AI decision model; and executing a fault self-healing strategy to repair a fault corresponding to the fault information. And the target XGBoost model is used to carry out root cause analysis on the multi-dimensional data, so that the accuracy of fault root cause positioning is greatly improved. And a fault self-healing strategy for solving the fault is automatically generated and executed by utilizing a dynamic AI decision model, so that the fault repairing delay is remarkably shortened.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of operation and maintenance technology, and in particular to an AI-based method, apparatus and equipment for cluster fault prediction and self-healing. Background Technology

[0003] In existing operation and maintenance (O&M) technologies, root cause analysis primarily relies on the manual experience of O&M personnel for troubleshooting and judgment. This approach is not only highly subjective and lacks reusability, but also struggles to quickly and accurately pinpoint the root cause when dealing with large-scale, highly complex cloud-native systems, resulting in low accuracy in root cause identification. Furthermore, existing methods generally employ a passive response mechanism, initiating the diagnostic process only after a fault actually occurs and produces observable impacts, failing to provide early warnings and proactive intervention. In addition, the process from fault occurrence to final repair typically involves multiple sequential stages, including "alarm perception - log retrieval - indicator analysis - root cause inference - strategy formulation - manual execution," resulting in a lengthy process with low automation and significantly extending the mean time to repair (MTBL). In summary, existing technologies generally suffer from inaccurate root cause localization, high fault repair delays, and a lack of closed-loop self-healing capabilities, making it difficult to meet the urgent needs of modern distributed systems for high availability and intelligent O&M. Summary of the Invention

[0004] The purpose of this invention is to provide at least one AI-based cluster fault prediction and self-healing method, apparatus and equipment, which can at least solve the technical problems of inaccurate fault root cause location, high fault repair delay and lack of closed-loop self-healing capability that are common in the prior art. It can at least improve the accuracy of fault root cause location, shorten the fault repair delay and realize closed-loop self-healing capability.

[0005] To address the aforementioned technical problems, at least one embodiment of this application provides an AI-based cluster fault prediction and self-healing method, comprising: acquiring metrics feature data, log feature data, and event feature data; performing prediction processing on the metrics feature data using a target time-series prediction model to obtain cluster resource prediction values ​​and fault information; determining, based on the fault information and the cluster resource prediction values, performing root cause localization processing on the metrics feature data, log feature data, and event feature data using a target XGBoost model to obtain the fault root cause; determining a fault self-healing strategy based on the fault root cause, the cluster resource prediction values, a preset static strategy library, and a dynamic AI decision model; and executing the fault self-healing strategy to repair the fault corresponding to the fault information.

[0006] This solution significantly improves the intelligence level of fault handling by integrating metrics feature data, log feature data, and event feature data, and collaboratively applying a target time-series prediction model and a target XGBoost root cause localization model. Specifically, the time-series prediction model predicts cluster resource trends and identifies potential anomalies in advance based on metrics feature data, achieving proactive fault detection; the target XGBoost model combines metrics feature data, log feature data, and event feature data to perform high-precision root cause analysis, effectively overcoming the problems of high subjectivity and low accuracy in traditional manual diagnosis, and greatly improving the accuracy of fault root cause localization. On this basis, the system integrates fault root causes, resource prediction status, a preset static strategy library, and a dynamic AI decision model to automatically generate the optimal self-healing strategy adapted to the current context and automatically execute the repair operation, eliminating the need for manual intervention and significantly shortening fault repair delays. The entire solution forms a closed-loop process of "early fault prediction - fault root cause diagnosis - fault self-healing strategy decision - fault self-healing strategy execution", realizing the transformation from passive fault response and manual handling to active fault response and automated handling. It has end-to-end fault self-healing capability and effectively ensures system stability and business continuity in the cloud-native environment.

[0007] In some examples, the metrics feature data includes data related to CPU resources, memory, disk I / O, and network traffic. The step of using a target time-series prediction model to predict the metrics feature data to obtain cluster resource prediction values ​​and fault information includes: the target time-series prediction model using an additive attention mechanism to calculate the attention weights of the CPU resource-related data, the memory-related data, and the disk I / O-related data on the feature dimensions; weighting and fusing the data of each feature dimension in the metrics feature data using the attention weights of each feature dimension to generate an enhanced context representation; and the target time-series prediction model using an LSTM layer to capture long-term dependencies in the context representation and predicting cluster resource prediction values ​​and fault information for a specified time window based on the long-term dependencies.

[0008] In some examples, the step of determining the root cause of the fault based on the fault information and the predicted cluster resource value, and then using the target XGBoost to perform root cause localization processing on the metrics feature data, the log feature data, and the event feature data to obtain the root cause of the fault, includes: if the fault information and the predicted cluster resource value indicate that there is a fault in the cluster, then using the target XGBoost to perform root cause localization processing on the metrics feature data, the log feature data, and the event feature data, and outputting the predicted probability corresponding to each initial root cause of the fault; and taking the initial root cause of the fault corresponding to the highest predicted probability as the root cause of the fault.

[0009] In some examples, determining a fault self-healing strategy based on the root cause of the fault, the predicted value of cluster resources, a preset static strategy library, and a dynamic AI decision model includes: selecting a basic fault self-healing strategy that matches the root cause of the fault from the preset static strategy library; the dynamic AI decision model predicting the fault self-healing probability of executing the basic fault self-healing strategy based on the predicted value of cluster resources; if the fault self-healing probability is lower than a preset threshold, then optimizing the basic fault self-healing strategy based on the dynamic AI decision model and the predicted value of cluster resources to obtain a fault self-healing strategy.

[0010] In some examples, the method further includes: collecting real-time metrics data, real-time log data, and real-time event data of the cluster after a preset time interval from the execution of the fault self-healing strategy; if there is no abnormal data in the real-time metrics data, no abnormal logs in the real-time log data, and no abnormal events in the real-time event data, then it is determined that the fault corresponding to the fault information has been repaired, and self-healing success information is generated; otherwise, it is determined that the fault corresponding to the fault information has not been repaired, self-healing failure information is generated, and a secondary fault self-healing strategy is generated using a dynamic AI decision model, the cluster resource prediction value, and the self-healing failure information, so as to repair the fault corresponding to the fault information based on the secondary fault self-healing strategy.

[0011] In some examples, the method further includes: if the feedback information from executing the fault self-healing strategy is self-healing failure information, then using the self-healing failure information as a training sample; optimizing the preset static strategy library based on the training sample; and incrementally training the dynamic AI decision model based on the training sample to optimize the dynamic AI decision model.

[0012] In some examples, the cluster resource prediction value is the cluster resource prediction value for a preset time period. The method further includes: during the preset time period, collecting the cluster resource value in real time; if the deviation between the resource value and the cluster resource prediction value is greater than a preset threshold, then using the resource value, the metrics feature data, and the cluster resource prediction value as a model update sample; and incrementally training the target time series prediction model based on the model update sample to optimize the target time series prediction model.

[0013] At least one embodiment of this application also provides an AI-based cluster fault prediction and self-healing device, comprising: an acquisition unit for acquiring metrics feature data, log feature data, and event feature data; a processing unit for using a target time-series prediction model to perform prediction processing on the metrics feature data to obtain cluster resource prediction values ​​and fault information; the processing unit is further configured to determine, based on the fault information and the cluster resource prediction values, to use a target XGBoost to perform root cause localization processing on the metrics feature data, the log feature data, and the event feature data to obtain the fault root cause; a determination unit for determining a fault self-healing strategy based on the fault root cause, the cluster resource prediction values, a preset static strategy library, and a dynamic AI decision model; and an execution unit for executing the fault self-healing strategy to repair the fault corresponding to the fault information.

[0014] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described AI-based cluster fault prediction and self-healing method.

[0015] At least one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described AI-based cluster fault prediction and self-healing method. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0017] Figure 1 This is a flowchart of an AI-based cluster fault prediction and self-healing method provided in one embodiment of this application; Figure 2 This is a schematic diagram of an AI-based cluster fault prediction and self-healing device provided in another embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0019] It should be noted that the acquisition or use of data in the embodiments of this application requires the user's consent. The relevant data can only be obtained after the user's authorization, and the acquisition or use of the data complies with the provisions of relevant laws and regulations.

[0020] To facilitate understanding of the embodiments of this application, we will first introduce the relevant content of AI-based cluster fault prediction and self-healing methods.

[0021] In existing operation and maintenance (O&M) technologies, root cause analysis primarily relies on the manual experience of O&M personnel for troubleshooting and judgment. This approach is not only highly subjective and lacks reusability, but also struggles to quickly and accurately pinpoint the root cause when dealing with large-scale, highly complex cloud-native systems, resulting in low accuracy in root cause identification. Furthermore, existing methods generally employ a passive response mechanism, initiating the diagnostic process only after a fault actually occurs and produces observable impacts, failing to provide early warnings and proactive intervention. In addition, the process from fault occurrence to final repair typically involves multiple sequential stages, including "alarm perception - log retrieval - indicator analysis - root cause inference - strategy formulation - manual execution," resulting in a lengthy process with low automation and significantly extending the mean time to repair (MTBL). In summary, existing technologies generally suffer from inaccurate root cause localization, high fault repair delays, and a lack of closed-loop self-healing capabilities, making it difficult to meet the urgent needs of modern distributed systems for high availability and intelligent O&M.

[0022] To address the aforementioned existing technical problems, this invention proposes an AI-based cluster fault prediction and self-healing method. The implementation details of the AI-based cluster fault prediction and self-healing method in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0023] Example 1: The AI-based cluster fault prediction and self-healing method in this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 1 As shown, it includes: Step 110: Obtain metrics feature data, log feature data, and event feature data.

[0024] Among them, metrics feature data refers to feature data related to cluster resources; log feature data refers to feature data extracted based on cluster logs; and event feature data refers to feature data extracted based on cluster events.

[0025] Specifically, acquiring metrics feature data, log feature data, and event feature data includes: acquiring metrics data, log data, and event data; performing data cleaning processing on the metrics data, log data, and event data to obtain cleaned metrics data, cleaned log data, and cleaned event data; and performing feature extraction processing on the cleaned metrics data, cleaned log data, and cleaned event data to obtain metrics feature data, log feature data, and event feature data.

[0026] The metrics data includes resource usage data, hardware metrics data, and status metrics data. Hardware metrics data and status metrics data are resource-related. Resource usage data reflects the consumption of cluster resources, including CPU usage, memory usage, disk I / O throughput, CPU utilization, memory utilization, and disk I / O utilization. Hardware metrics data refers to monitoring data related to the health status of physical hardware, including CPU temperature, hard drive temperature, fan speed, and disk SMART information. Status metrics data describes the running status of services in the cluster, including the number of Pod (process instance) restarts, the number of Deployment (deployment controller) restarts, Pod ready status, and Deployment ready status. Log data includes container standard output logs (stdout, stderr), Kubernetes component logs (API Server, Scheduler logs), and application business logs (such as log4j logs for Java applications). Event data includes Kubernetes cluster events such as Pod creation, Pod deletion, Node going online, Node going offline, Deployment scaling up, and Deployment scaling down.

[0027] Furthermore, the metrics collection module collects metrics data at specified intervals (e.g., 10 seconds) and adds metadata to the collected metrics data, including cluster ID, Node IP, Pod name, and collection timestamp. For example, the metrics collection module relies on the metrics-server deployed on the Kubernetes cluster to collect resource usage data, collects hardware metrics data based on Node-exporter, and collects status metrics data based on kube-state-metrics. The log collection module collects log data at specified intervals (e.g., 5 seconds). For example, the log collection module deploys a custom DaemonSet based on Fluentd (one Fluentd Pod per Node) and reads container logs through the Container Runtime Interface (CRI). The event collection module obtains event data through the event interface of the Kubernetes API Server. For example, the event collection module listens for events using the Client-go Watch mechanism, capturing events in real time when they occur, rather than collecting event data based on polling, thus reducing API pressure.

[0028] Furthermore, data cleaning includes handling missing values, outliers, and redundant data. Metrics data, log data, and event data are cleaned to obtain cleaned metrics data, cleaned log data, and cleaned event data. This includes: if missing data exists in the metrics data, linear interpolation is used to fill in the missing data; if missing data exists in the log data and event data, the missing data is filled using a "missing data" identifier. Outliers in the metrics data are removed using the "3σ principle"; if garbled data exists in the log data and event data, the garbled data is converted to UTF-8 format using the iconv tool; if the conversion fails, the garbled data is deleted. Outliers and garbled data are considered abnormal values. For duplicate data in the metrics data, log data, and event data, multiple duplicate data entries are deleted and merged into a single data entry.

[0029] Furthermore, feature extraction processing includes feature extraction, time alignment, resource association, and data format conversion. Feature extraction processing is performed on the cleaned metrics data, cleaned log data, and cleaned event data to obtain metrics feature data, log feature data, and event feature data. This includes: performing statistical processing on the cleaned metrics data to extract statistical features; performing trend feature analysis on the cleaned metrics data to extract trend features; performing association analysis on the cleaned metrics data to extract association features between cleaned metrics data; and using the statistical features, trend features, and association features as metrics feature data. Keyword feature extraction is performed on the cleaned log data to obtain log keyword features; one-hot encoding is performed on the log levels and log error codes in the cleaned log data to obtain log encoding features; statistical processing is performed on the cleaned log data to obtain log statistical features; and the log keyword features, log encoding features, and log statistical features are used as log feature data. The cleaned event data undergoes key field extraction to obtain key event features; correlation features are also extracted to obtain event correlation features; these key and correlation features are then used as event feature data. Time alignment is performed on metrics, log, and event feature data to obtain new metrics, log, and event feature data; the new metrics, log, and event feature data are then converted to their respective formats and associated with a unique cluster identifier.

[0030] For example, the statistical features can be the mean, variance, maximum, and minimum values ​​of a preset sliding window, with the preset sliding window size being 5 minutes, 15 minutes, or other durations. Trend features can be first-order differences, second-order differences, and other features reflecting the rate of change of indicators; correlation features can be the ratio of "Pod CPU utilization" to "Node CPU utilization". TF-IDF can be used to extract keyword features from log data to extract log keyword features, such as "OutOfMemory" and "ConnectionRefused"; log encoding features can be ERROR=1, WARN=0.5, and INFO=0. Log statistical features are the frequency of a certain type of log occurrence within a unit of time, for example, the number of OOM logs per minute. Key event features include event type, associated resources, event cause, and message content; event type can be "PodEvent", "NodeEvent", or "DeploymentEvent". Event correlation features can be the correlation between different events, for example, the time difference between "Pod restart event" and "Node resource shortage event". The cluster unique identifier can be cluster ID + Namespace + Pod name. The format conversion transforms new metrics feature data, new log feature data, and new event feature data into a format that the model can recognize, such as TensorFlow's TFRecord format or PyTorch's Tensor format.

[0031] Step 120: Use the target time series prediction model to predict the metric feature data to obtain the cluster resource prediction value and fault information.

[0032] The cluster resource prediction values ​​are as follows: Cluster resource prediction values ​​are predicted for at least one specified time period. Each moment within a specified time period is later than or equal to the current moment. For example, cluster resource prediction values ​​could be predicted for a time period starting at the current moment and ending at a moment 15 minutes, 30 minutes, or other durations away from the current moment. The resource prediction values ​​are the resource values ​​for a future time period (the specified time period) predicted by processing historical metrics feature data using a target time-series prediction model. These resource values ​​include at least CPU usage, memory usage, disk I / O throughput, CPU utilization, memory utilization, and disk I / O utilization. Fault information is the fault information for at least one specified time period, including predicted faults and the probability of the predicted fault occurring.

[0033] Specifically, the metrics feature data includes data related to CPU resources, memory, disk I / O, and network traffic. In step 120, the target time series prediction model is used to predict the metrics feature data to obtain cluster resource prediction values ​​and fault information. This includes: the target time series prediction model uses an additive attention mechanism to calculate the attention weights of data related to CPU resources, memory, and disk I / O on the feature dimensions; the attention weights of each feature dimension are used to perform weighted fusion of the data in each feature dimension of the metrics feature data to generate an enhanced context representation; the target time series prediction model uses an LSTM layer to capture long-term dependencies in the context representation and predicts cluster resource prediction values ​​and fault information for a specified time window based on the long-term dependencies.

[0034] Among them, data related to CPU resources includes at least CPU usage and CPU utilization; data related to memory includes at least memory usage and memory utilization; and data related to disk I / O includes at least disk I / O throughput and disk I / O utilization.

[0035] For example, the target time series prediction model includes an input layer, an attention layer, an LSTM layer, and an output layer. The metrics feature data input to the input layer is multi-dimensional data, meaning it includes multiple feature dimensions. For instance, the metrics feature data contains 64 feature dimensions. By inputting the metrics feature data into the input layer, the additive attention mechanism of the attention layer is used to perform weighted fusion processing on the metrics feature data to calculate the attention weight of each feature dimension in the metrics feature data. This attention weight is then used to enhance the feature dimension data to obtain the corresponding enhanced context representation. The enhanced context representation is processed using a three-layer LSTM network within the LSTM layer to capture the long-term dependencies of the metrics feature data. Each LSTM layer contains 128 hidden units, and an activation function is connected after each LSTM layer to output smoothing information. Furthermore, the Dropout of each LSTM layer is set to 0.2 to randomly mask 20% of the neurons in each LSTM layer, preventing overfitting to noise. Finally, the fully connected layer of the output layer is used to predict long-term dependencies, outputting resource prediction values ​​for the next 15 minutes and 30 minutes, as well as the probability of Pod memory shortage failures being 92%.

[0036] In some examples, an initial time-series prediction model is trained based on cluster operation data collected over historical periods to obtain a target time-series prediction model. The cluster operation data includes metrics feature data and actual results. Actual results include real fault information and real resource values. During the training of the initial time-series prediction model, the initial learning rate of the Adam optimizer is set to 0.001, and the initial learning rate is decayed to 0.5 every 5 epochs. The batch size is set to 32, and the maximum number of iterations is set to 200. The prediction bias of the model is calculated using mean squared error, and the outlier prediction bias is calculated using cross-entropy loss. When both the prediction bias and outlier prediction bias of the model meet the requirements on the validation set, this model is used as the target time-series prediction model.

[0037] Here, the anomaly prediction bias represents the deviation between the failure probability predicted by the target time-series prediction model and the actual failure probability. The prediction bias represents the deviation between the cluster resource prediction value predicted by the target time-series prediction model and the actual cluster resource value.

[0038] For example, when the MSE (prediction bias) of the validation set is less than 5% and the abnormal prediction bias is less than 5%, it is considered that both the prediction bias and the abnormal prediction bias meet the requirements.

[0039] Alternatively, the total loss can be calculated based on prediction bias and anomalous prediction bias, and the initial time-series prediction model that meets the requirements for total loss can be used as the target time-series prediction model. The total loss calculated based on prediction bias and anomalous prediction bias can be found in the following formula: Total loss = 0.7MSE + 0.3 outlier prediction bias MSE is the preset deviation.

[0040] In some examples, the cluster resource prediction value is the cluster resource prediction value for a preset time period. The method also includes: collecting the cluster resource value in real time during the preset time period; if the deviation between the resource value and the cluster resource prediction value is greater than a preset threshold, then using the resource value, metrics feature data and cluster resource prediction value as a model update sample; and incrementally training the target time series prediction model based on the model update sample to optimize the target time series prediction model.

[0041] In this preset time period, each moment is later than or equal to the current moment.

[0042] Specifically, during a preset time period, the cluster's resource values ​​are collected in real time. If the deviation between the resource values ​​and the predicted cluster resource values ​​is greater than a preset threshold, the resource values, metrics feature data, and the predicted cluster resource values ​​are used as a model update sample. This includes: for each moment in the preset time period, collecting the cluster's resource values ​​in real time, obtaining the predicted resource values ​​belonging to that moment from the predicted cluster resource values; determining the deviation between the resource values ​​and the predicted resource values, and if the deviation is greater than a preset threshold, the resource values, metrics feature data, and the predicted resource values ​​are used as a model update sample.

[0043] In some cases, if the deviation between the resource value and the predicted cluster resource value is greater than a preset threshold for a certain number of consecutive preset times, the target time series prediction model will be optimized immediately based on the model update sample.

[0044] Step 130: Based on the fault information and cluster resource prediction values, the target XGBoost is used to perform root cause localization processing on the metrics feature data, log feature data and event feature data to obtain the root cause of the fault.

[0045] Among them, the target XGBoost is used to locate the root cause of the fault. The root cause of the fault refers to the fundamental reason for the fault in the fault information.

[0046] Specifically, in step 130 above, based on the fault information and cluster resource prediction values, the target XGBoost is used to perform root cause localization processing on the metrics feature data, log feature data, and event feature data to obtain the fault root cause. This includes: if the fault information and cluster resource prediction values ​​indicate that there is a fault in the cluster, the target XGBoost is used to perform root cause localization processing on the metrics feature data, log feature data, and event feature data, and the predicted probability corresponding to each initial fault root cause is output; the initial fault root cause corresponding to the highest predicted probability is taken as the fault root cause.

[0047] The predicted probability is used to indicate the degree of confidence that the failure is caused by the corresponding initial root cause. The initial root cause is a preset root cause, specifically including eight types of initial root causes: memory leak, CPU overload, network failure, storage failure, image compatibility issues, configuration errors, hardware failure, and unknown failure.

[0048] Furthermore, when the probability of a fault occurring in the fault information is greater than a preset fault probability threshold, or when the cluster resource prediction value is greater than a preset resource threshold, it is considered that the fault information and the cluster resource prediction value indicate that the cluster has a fault.

[0049] For example, the target XGBoost includes an input layer, a decision tree layer, and an output layer. Metrics feature data, log feature data, and event feature data are used as a multimodal fusion feature vector. This multimodal fusion feature vector is input to the decision tree layer, which uses 100 decision trees to analyze and process the multimodal fusion feature vector to obtain the predicted probability corresponding to the initial root cause of the failure. The output layer outputs the initial root cause of the failure corresponding to the highest predicted probability, which is then used as the root cause of the failure.

[0050] Further, in step 130 above, taking the initial root cause of the fault corresponding to the highest predicted probability as the root cause of the fault includes: if the highest predicted probability is greater than the first confidence level, then taking the initial root cause of the fault corresponding to the highest predicted probability as the root cause of the fault; if the highest predicted probability is greater than the second confidence level and less than the first confidence level, then taking the initial root cause of the fault corresponding to the highest predicted probability as the root cause of the fault, and marking the root cause of the fault as requiring manual processing, so as to further determine a new root cause of the fault based on manual processing, and taking the new root cause of the fault as the root cause of the fault; if the highest predicted probability is less than the second confidence level, then taking the unknown fault as the root cause of the fault.

[0051] In some examples, multiple historical multimodal fusion feature vectors containing historical failure root causes are obtained. These vectors include historical metrics feature data, historical log feature data, and historical event feature data. The initial XGBoost is trained based on these historical multimodal fusion feature vectors to obtain the target XGBoost. The historical failure root causes include each initial failure root cause. Specifically, in the actual training process, training parameters of a maximum depth of 8, a learning rate of 0.1, and a minimum number of sample splits of 20 are used. The trained initial XGBoost is validated using 5-fold cross-validation. The Gini coefficient is used as the feature importance evaluation metric, prioritizing features with high discriminative power (such as OOM log frequency and memory usage trend) to train the model and obtain the target XGBoost.

[0052] In some examples, the trained target XGBoost is saved as a .model file; the trained target time series prediction model is saved as a .h5 file; and the .model target XGBoost, the .h5 target time series prediction model, and their corresponding parameter configurations are stored in etcd (a distributed key-value store). The models and their parameter configurations in etcd are dynamically updated based on feedback from operations personnel and automatic feedback. Simultaneously, the models and their parameter configurations can be retrieved from etcd in real time, and model accuracy is verified in a validation environment using a canary release approach. The validated models replace the corresponding models in the production environment. The model accuracy of each model in the production environment is monitored in real time. If the model accuracy decreases by more than a preset value for a consecutive preset duration, the historical best model corresponding to that model is automatically retrieved from etcd and replaced with it. Among multiple historical models with the same functionality, the model with the highest accuracy is selected.

[0053] Furthermore, when utilizing the target XGBoost and the target time series prediction model, the target XGBoost and the target time series prediction model that are closest to the current time in terms of model update time can be obtained from etcd.

[0054] In some cases, if a new root cause of failure exists, the target XGBoost is optimized by adding a new decision tree based on the new root cause and the corresponding multimodal fusion feature vector, so that the target XGBoost can identify the new root cause of failure.

[0055] Step 140: Based on the root cause of the fault, the predicted value of cluster resources, the preset static strategy library, and the dynamic AI decision model, determine the fault self-healing strategy.

[0056] Specifically, in step 140 above, a fault self-healing strategy is determined based on the root cause of the fault, the predicted value of cluster resources, the preset static strategy library, and the dynamic AI decision model. This includes: selecting a basic fault self-healing strategy that matches the root cause of the fault from the preset static strategy library; the dynamic AI decision model predicts the fault self-healing probability of executing the basic fault self-healing strategy based on the predicted value of cluster resources; if the fault self-healing probability is lower than a preset threshold, the basic fault self-healing strategy is optimized based on the dynamic AI decision model and the predicted value of cluster resources to obtain the fault self-healing strategy.

[0057] The system includes a pre-defined static policy library that stores basic self-healing policies corresponding to various root causes of failures. Specifically, it provides customized basic self-healing policies for different root causes based on Kubernetes CRDs, and supports configuration by operations personnel via YAML files. Furthermore, it allows modification and deletion of these basic self-healing policies using the kubectl command. These basic self-healing policies include one or more of the following: resource scaling, Pod restart, cluster scaling, resource adjustment, Pod migration, and Node drainage.

[0058] Specifically, if the fault self-healing probability is not lower than a preset threshold, the basic fault self-healing strategy will be used as the fault self-healing strategy.

[0059] Furthermore, the basic fault self-healing strategy is optimized based on the dynamic AI decision model and the cluster resource prediction value to obtain the fault self-healing strategy, including: the dynamic AI decision model selects the auxiliary self-healing strategy corresponding to the cluster resource prediction value based on the cluster resource prediction value; and the fault self-healing strategy is obtained based on the auxiliary self-healing strategy and the basic fault self-healing strategy.

[0060] Furthermore, based on the auxiliary self-healing strategy and the basic fault self-healing strategy, a fault self-healing strategy is obtained, including: forming multiple sub-self-healing strategies based on the auxiliary self-healing strategy and the basic fault self-healing strategy; predicting the sub-self-healing strategy based on the dynamic AI decision model and cluster resource prediction value for each sub-self-healing strategy to obtain the fault self-healing sub-probability; and taking the sub-self-healing strategy corresponding to the largest fault self-healing sub-probability as the fault self-healing strategy.

[0061] For example, if the basic fault self-healing strategy is to restart the Pod first and then adjust resources; and the cluster resource prediction value is a CPU utilization rate of 80%, then an auxiliary self-healing strategy corresponding to a CPU utilization rate of 80% is obtained, which is to first expand resources and then restart the Pod; based on the auxiliary self-healing strategy and the basic fault self-healing strategy, multiple sub-self-healing strategies are generated, including sub-self-healing strategy 1 for Pod restart, sub-self-healing strategy 2 for resource expansion, sub-self-healing strategy 3 for resource adjustment, sub-self-healing strategy 4 for expanding resources first and then restarting the Pod, and sub-self-healing strategy 5 for restarting the Pod first and then adjusting resources. The sub-self-healing strategies are as follows: Sub-self-healing strategy 6 is to adjust resources first and then restart the Pod; Sub-self-healing strategy 7 is to restart the Pod first, then adjust resources, and finally expand resources; Sub-self-healing strategy 8 is to expand resources first, then restart the Pod, and finally adjust resources; Sub-self-healing strategy 9 is to restart the Pod first, then expand resources, and finally adjust resources; Sub-self-healing strategy 10 is to expand resources first, then adjust resources, and finally restart the Pod; Sub-self-healing strategy 11 is to adjust resources first, then restart the Pod, and finally expand resources; Sub-self-healing strategy 12 is to adjust resources first, then expand resources, and finally restart the Pod. By predicting the sub-resource strategies, the highest probability of fault self-healing is found to be 80%, which corresponds to sub-self-healing strategy 4, which is to expand resources first and then restart the Pod. Therefore, the fault self-healing strategy of expanding resources first and then restarting the Pod is obtained.

[0062] Step 150: Execute the fault self-healing strategy to repair the faults corresponding to the fault information.

[0063] In some examples, the method further includes: collecting real-time metrics data, real-time log data, and real-time event data of the cluster after a preset time interval from the execution of the fault self-healing strategy; if there is no abnormal data in the real-time metrics data, no abnormal logs in the real-time log data, and no abnormal events in the real-time event data, then it is determined that the fault corresponding to the fault information has been repaired, and a self-healing success message is generated; otherwise, it is determined that the fault corresponding to the fault information has not been repaired, a self-healing failure message is generated, and a secondary fault self-healing strategy is generated using a dynamic AI decision model, cluster resource prediction values, and the self-healing failure message, so as to repair the fault corresponding to the fault information based on the secondary fault self-healing strategy.

[0064] The preset duration can be set according to actual needs, such as 5 minutes, 10 minutes, or other durations. Self-healing failure information includes the executed fault self-healing strategy and a self-healing failure flag. Specifically, if real-time metrics contain data that does not meet preset data requirements, it is considered that there is abnormal data in the real-time metrics data; if there is no data in the real-time metrics data that does not meet preset data requirements, it is considered that there is no abnormal data in the real-time metrics data. For example, if the memory usage rate in the real-time metrics data is 82%, and the preset data requirement for memory usage rate is less than 80%, then the memory usage rate in the metrics data is greater than 80%, which means that there is abnormal data in the real-time metrics data.

[0065] Specifically, if there are no exception markers in the real-time log data, it is considered that there are no exception logs in the real-time log data. The exception marker can be OOM. If there are no exception event markers in the real-time event data, it is considered that there are no exception events in the real-time event data. The exception event markers can be PodFailed, NodeNotReady, or Eviction.

[0066] For example, if a self-healing failure is obtained after executing a fault self-healing strategy of first expanding resources and then restarting the Pod, an auxiliary self-healing strategy corresponding to the predicted cluster resources is selected using a dynamic AI decision model. Multiple sub-self-healing strategies are formed based on the auxiliary self-healing strategy and the self-healing failure information. The sub-self-healing strategies do not include the fault self-healing strategy in the self-healing failure information. For each sub-self-healing strategy, the sub-self-healing strategy is predicted based on the dynamic AI decision model and the predicted cluster resources to obtain the fault self-healing sub-probability. The sub-self-healing strategy corresponding to the highest fault self-healing sub-probability is taken as the secondary fault self-healing strategy.

[0067] In some examples, the method also includes: if the feedback information from executing the fault self-healing strategy is self-healing failure information, then using the self-healing failure information as a training sample; optimizing the pre-set static strategy library based on the training sample; and incrementally training the dynamic AI decision model based on the training sample to optimize the dynamic AI decision model.

[0068] Therefore, incremental training, rather than full training, reduces the cost of model iteration.

[0069] Specifically, training samples are stored in a feedback storage area, and multiple training samples are periodically retrieved from the feedback storage area to optimize the pre-defined static policy library based on each training sample. Incremental training is then performed on the dynamic AI decision-making model based on each training sample to optimize the dynamic AI decision-making model. Simultaneously, the updated dynamic AI decision-making model and its model parameters are stored in etcd.

[0070] Furthermore, training samples from a specified historical period are obtained, and the average repair time and average business impact range corresponding to each fault self-healing strategy are statistically analyzed based on the training samples. A first sub-strategy with a repair time faster than the average repair time is selected, and a second sub-strategy with a business impact range smaller than the average business impact range is selected. The first and second sub-strategies are used as new fault self-healing strategies.

[0071] For example, within a preset time period, the self-healing strategies for memory leaks are Pod restart and Node drainage. The average repair time is two minutes and the average business impact is relatively wide. Since Pod restart takes a long time, although it can deal with memory leaks, the response time is too long. Therefore, Pod migration, which can solve memory shortages and has a shorter response time than Pod restart, is selected. Since Node drainage will cause business interruption and expand the business impact, resource expansion, which can solve memory leaks and has a small business impact, is selected. Finally, Pod migration and resource expansion are used as new self-healing strategies for memory leaks.

[0072] In summary, this solution acquires metrics feature data, log feature data, and event feature data; it uses a target time-series prediction model to predict the metrics feature data, obtaining predicted cluster resource values ​​and fault information; based on the fault information and predicted cluster resource values, it uses target XGBoost to perform root cause localization processing on the metrics feature data, log feature data, and event feature data to obtain the root cause of the fault; based on the root cause, predicted cluster resource values, a preset static policy library, and a dynamic AI decision model, it determines a fault self-healing strategy; and it executes the fault self-healing strategy to repair the fault corresponding to the fault information. By integrating metrics feature data, log feature data, and event feature data, and synergistically applying the target time-series prediction model and the target XGBoost root cause localization model, the intelligence level of fault handling is significantly improved. Specifically, the time-series prediction model predicts cluster resource trends and identifies potential anomalies in advance based on metrics feature data, achieving proactive fault detection; the target XGBoost model combines metrics feature data, log feature data, and event feature data for high-precision root cause analysis, effectively overcoming the problems of strong subjectivity and low accuracy in traditional manual diagnosis, and significantly improving the accuracy of fault root cause localization. Based on this, the system integrates the root cause of the fault, the predicted resource status, the preset static strategy library, and the dynamic AI decision model to automatically generate the optimal self-healing strategy adapted to the current context and automatically execute the repair operation, eliminating the need for manual intervention and significantly shortening the fault repair delay. The entire solution forms a closed-loop process of "early fault prediction - fault root cause diagnosis - fault self-healing strategy decision - fault self-healing strategy execution," realizing the transformation from passive fault response and manual handling to proactive fault response and automated handling, possessing end-to-end fault self-healing capabilities, and effectively ensuring the system stability and business continuity in the cloud-native environment.

[0073] Example 2: Another embodiment of this application relates to an AI-based cluster fault prediction and self-healing device. The implementation details of this embodiment are described below. The following content is for ease of understanding and is not essential for implementing this solution. A schematic diagram of the AI-based cluster fault prediction and self-healing device 20 in this embodiment can be seen as follows: Figure 2 As shown, it includes an acquisition unit 201, a processing unit 202, a determination unit 203, and an execution unit 204.

[0074] The acquisition unit 201 is used to acquire metrics feature data, log feature data, and event feature data.

[0075] The processing unit 202 is used to perform prediction processing on the metric feature data using the target time series prediction model to obtain cluster resource prediction values ​​and fault information.

[0076] The processing unit 202 is further configured to determine the root cause of the fault by using the target XGBoost to perform root cause localization processing on the metrics feature data, the log feature data, and the event feature data based on the fault information and the cluster resource prediction value.

[0077] The determining unit 203 is used to determine a fault self-healing strategy based on the root cause of the fault, the predicted value of the cluster resources, the preset static strategy library, and the dynamic AI decision model.

[0078] The execution unit 204 is used to execute the fault self-healing strategy to repair the fault corresponding to the fault information.

[0079] In some examples, the metrics feature data includes data related to CPU resources, memory, disk I / O, and network traffic. When processing unit 202 performs the prediction processing on the metrics feature data using the target time-series prediction model to obtain cluster resource prediction values ​​and fault information, specifically: the target time-series prediction model uses an additive attention mechanism to calculate the attention weights of the CPU resource-related data, the memory-related data, and the disk I / O-related data on the feature dimensions; it uses the attention weights of each feature dimension to perform weighted fusion of the data in each feature dimension of the metrics feature data to generate an enhanced context representation; the target time-series prediction model uses an LSTM layer to capture long-term dependencies in the context representation and predicts cluster resource prediction values ​​and fault information for a specified time window based on the long-term dependencies.

[0080] In some examples, when processing unit 202 is used to determine, based on the fault information and the cluster resource prediction value, to perform root cause localization processing on the metrics feature data, the log feature data, and the event feature data using target XGBoost to obtain the fault root cause, it specifically performs the following: if the fault information and the cluster resource prediction value indicate that there is a fault in the cluster, then it uses target XGBoost to perform root cause localization processing on the metrics feature data, the log feature data, and the event feature data, and outputs the predicted probability corresponding to each initial fault root cause; the initial fault root cause corresponding to the highest predicted probability is taken as the fault root cause.

[0081] In some examples, when determining a fault self-healing strategy based on the root cause of the fault, the predicted value of cluster resources, a preset static strategy library, and a dynamic AI decision model, the determining unit 203 specifically performs the following steps: selecting a basic fault self-healing strategy that matches the root cause of the fault from the preset static strategy library; predicting the fault self-healing probability of executing the basic fault self-healing strategy based on the predicted value of cluster resources using the dynamic AI decision model; and optimizing the basic fault self-healing strategy based on the dynamic AI decision model and the predicted value of cluster resources to obtain a fault self-healing strategy if the fault self-healing probability is lower than a preset threshold.

[0082] In some examples, the determining unit 203 is further configured to: collect real-time metrics data, real-time log data, and real-time event data of the cluster after a preset time interval from the execution of the fault self-healing strategy; if there is no abnormal data in the real-time metrics data, no abnormal logs in the real-time log data, and no abnormal events in the real-time event data, then determine that the fault corresponding to the fault information has been repaired and generate self-healing success information; otherwise, determine that the fault corresponding to the fault information has not been repaired, generate self-healing failure information, and trigger the generation of a secondary fault self-healing strategy using a dynamic AI decision model, the cluster resource prediction value, and the self-healing failure information, so as to repair the fault corresponding to the fault information based on the secondary fault self-healing strategy.

[0083] In some examples, the execution unit 204 is further configured to: if the feedback information of the fault self-healing strategy is self-healing failure information, then use the self-healing failure information as a training sample; optimize the preset static strategy library based on the training sample; and incrementally train the dynamic AI decision model based on the training sample to optimize the dynamic AI decision model.

[0084] In some examples, the cluster resource prediction value is the cluster resource prediction value for a preset time period. The execution unit 204 is further configured to: collect the cluster resource value in real time during the preset time period; if the deviation between the resource value and the cluster resource prediction value is greater than a preset threshold, use the resource value, the metrics feature data, and the cluster resource prediction value as a model update sample; and perform incremental training on the target time series prediction model based on the model update sample to optimize the target time series prediction model.

[0085] It is worth mentioning that all units involved in this embodiment are logical units. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0086] Example 3: Another embodiment of this application relates to an electronic device, such as... Figure 3 As shown, it includes: at least one processor 901; and a memory 902 communicatively connected to the at least one processor 901; wherein the memory 902 stores instructions executable by the at least one processor 901, the instructions being executed by the at least one processor 901 to enable the at least one processor 901 to execute the AI-based cluster fault prediction and self-healing methods in the above embodiments.

[0087] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0088] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0089] Example 4: Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0090] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. An AI-based cluster fault prediction and self-healing method, characterized in that, The method comprises: acquiring metrics feature data, log feature data, and event feature data; performing prediction processing on the metrics feature data by using a target time series prediction model to obtain cluster resource prediction values and fault information; based on the fault information and the cluster resource prediction values, performing root cause positioning processing on the metrics feature data, the log feature data, and the event feature data by using a target XGBoost to obtain a fault root cause; based on the fault root cause, the cluster resource prediction values, a preset static policy library, and a dynamic AI decision model, determining a fault self-healing strategy; executing the fault self-healing strategy to repair the fault corresponding to the fault information. 2.The AI-based cluster fault prediction and self-healing method according to claim 1, characterized in that, The metrics feature data includes data related to CPU resources, data related to memory, data related to disk I / O, and data related to network traffic. The prediction processing on the metrics feature data by using the target time series prediction model to obtain the cluster resource prediction values and the fault information comprises: The target time series prediction model uses an additive attention mechanism to calculate the attention weights of the data related to CPU resources, the data related to memory, and the data related to disk I / O in the feature dimension; using the attention weights of each feature dimension to weight and fuse each feature dimension data in the metrics feature data to generate an enhanced context representation; The target time series prediction model uses an LSTM layer to capture long-term dependencies in the context representation, and based on the long-term dependencies, predicts the cluster resource prediction values and the fault information in a specified time window. 3.The AI-based cluster fault prediction and self-healing method according to claim 1, characterized in that, Based on the fault information and the cluster resource prediction values, the target XGBoost is used to perform root cause positioning processing on the metrics feature data, the log feature data, and the event feature data to obtain a fault root cause, which comprises: If the fault information and the cluster resource prediction values indicate that the cluster has a fault, the target XGBoost is used to perform root cause positioning processing on the metrics feature data, the log feature data, and the event feature data to output the prediction probability corresponding to each initial fault root cause; the initial fault root cause corresponding to the maximum prediction probability is taken as the fault root cause. 4.The AI-based cluster fault prediction and self-healing method according to claim 1, characterized in that, Based on the fault root cause, the cluster resource prediction values, the preset static policy library, and the dynamic AI decision model, the fault self-healing strategy is determined, which comprises: selecting a basic fault self-healing strategy matching the fault root cause from the preset static policy library; the dynamic AI decision model predicts the fault self-healing probability of executing the basic fault self-healing strategy based on the cluster resource prediction values; if the fault self-healing probability is lower than a preset threshold, the basic fault self-healing strategy is optimized based on the dynamic AI decision model and the cluster resource prediction values to obtain the fault self-healing strategy. 5.The AI-based cluster fault prediction and self-healing method according to claim 1, characterized in that, The method further comprises: acquire real-time metrics data, real-time log data, and real-time event data of the cluster at a preset time length after the fault self-healing strategy is executed; if there is no abnormal data in the real-time metrics data, no abnormal log in the real-time log data, and no abnormal event in the real-time event data, it is determined that the fault corresponding to the fault information has been repaired, and self-healing success information is generated; otherwise, it is determined that the fault corresponding to the fault information has not been repaired, self-healing failure information is generated, and a secondary fault self-healing strategy is generated by using a dynamic AI decision model, the cluster resource prediction value, and the self-healing failure information, so as to repair the fault corresponding to the fault information based on the secondary fault self-healing strategy. 6.The AI-based cluster fault prediction and self-healing method according to claim 5, characterized in that, The method further comprises: if the feedback information of the execution of the fault self-healing strategy is self-healing failure information, the self-healing failure information is taken as a training sample; based on the training sample, the preset static strategy library is optimized; based on the training sample, the dynamic AI decision model is incrementally trained to optimize the dynamic AI decision model.

7. The AI-based cluster fault prediction and self-healing method according to claim 1, characterized in that, The cluster resource prediction value is a cluster resource prediction value in a preset time period, and the method further comprises: at the preset time period, the resource value of the cluster is acquired in real time, and if the deviation of the resource value from the cluster resource prediction value is greater than a preset threshold, the resource value, the metrics feature data, and the cluster resource prediction value are taken as a model update sample; based on the model update sample, the target time series prediction model is incrementally trained to optimize the target time series prediction model.

8. An AI-based cluster fault prediction and self-healing device, characterized by, comprises: an acquisition unit configured to acquire metrics feature data, log feature data, and event feature data; a processing unit configured to perform prediction processing on the metrics feature data by using a target time series prediction model to obtain a cluster resource prediction value and fault information; the processing unit is further configured to perform root cause positioning processing on the metrics feature data, the log feature data, and the event feature data by using a target XGBoost based on the fault information and the cluster resource prediction value to obtain a fault root cause; a determination unit configured to determine a fault self-healing strategy based on the fault root cause, the cluster resource prediction value, a preset static strategy library, and a dynamic AI decision model; an execution unit configured to execute the fault self-healing strategy to repair the fault corresponding to the fault information.

9. An electronic device, comprising: comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the AI-based cluster fault prediction and self-healing method of any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the AI-based cluster fault prediction and self-healing method of any one of claims 1 to 7. The computer program is executed by the processor to implement the AI-based cluster fault prediction and self-healing method of any one of claims 1 to 7.

Images