Intelligent fault self-healing method based on multi-dimensional indexes

Abstract

The invention provides an intelligent fault self-healing method based on a multi-dimensional index, and relates to the technical field of fault detection and repair, and the method comprises the steps: collecting multi-source data, carrying out the feature engineering processing, carrying out the fault classification through employing an XGBoost classifier, and selecting a combined repair strategy according to the fault type in combination with the controller type and service level features. And the repair operation is executed after the service dependency graph is optimized. The fault diagnosis accuracy can be improved, the fault repair time is shortened, the manual intervention requirement is reduced, and the cloud platform service reliability is improved.

Description

Technical Field

[0001] The present invention relates to fault detection and repair technology, and in particular to an intelligent fault self-healing method based on multi-dimensional indicators. Background Art

[0002] With the widespread adoption of cloud-native architectures, Kubernetes-based container orchestration platforms have become core components of enterprise IT infrastructure. Application failures are inevitable in complex distributed systems, potentially stemming from a variety of factors, including resource contention, configuration errors, and dependent service anomalies. Traditional troubleshooting often relies on manual intervention, requiring operations personnel to collect logs, analyze metrics, and manually perform remediation operations. This approach is inefficient and error-prone for large-scale clusters.

[0003] To improve system availability and reduce maintenance costs, the industry has begun exploring automated fault detection and repair solutions. Existing technologies primarily employ fault detection mechanisms based on single indicators or simple rules. For example, these mechanisms trigger alarms based on thresholds set based on CPU utilization or memory usage, and then combine these with predefined simple repair strategies to execute automated actions. However, these approaches have significant limitations in practical applications: Existing fault detection mechanisms rely too heavily on single-dimensional metrics and lack the ability to comprehensively analyze heterogeneous data from multiple sources. A single metric can't fully reflect the health of a system, leading to misjudgments or missed detections. This is especially true for complex distributed system failures, where a single metric often can't accurately identify the root cause.

[0004] Existing fault repair strategies lack flexibility and context-awareness. Most automated repair solutions use static, predefined strategies that are unable to dynamically adjust repair strategies based on factors such as workload type and service importance. This can result in repair operations being unsuitable for specific scenarios and even triggering chain reactions, further deteriorating system status.

[0005] Existing technologies often overlook the impact of inter-service dependencies on fault remediation. In a microservices architecture, services have complex dependencies, and an anomaly in one service can impact multiple related services. Existing remediation mechanisms lack awareness and consideration of these dependencies, making it difficult to achieve globally optimal remediation decisions. This can easily lead to remediation operations causing new problems or unsatisfactory remediation results. Summary of the Invention

[0006] The embodiments of the present invention provide an intelligent fault self-healing method based on multi-dimensional indicators, which can solve the problems in the prior art.

[0007] A first aspect of an embodiment of the present invention provides an intelligent fault self-healing method based on multi-dimensional indicators, including: Collect Pod restart data through the Kubernetes API, CPU utilization and memory usage data through Prometheus, collect and parse file damage keyword data through the ELK Stack, and collect error rate data through custom application monitoring to form multi-source data. Performing feature engineering on the multi-source data, including feature encoding the controller type to which the Pod belongs to obtain controller type features, feature encoding the service level in the Pod label to obtain service level features, and generating a multi-dimensional feature vector; Inputting the multi-dimensional feature vector into the XGBoost classifier to classify the fault to obtain a fault type; selecting a basic strategy template according to the fault type; Combining the basic policy templates based on the controller type characteristics and the service level characteristics to form a combined repair strategy; constructing a service dependency graph to optimize the combined repair strategy; The optimized combined repair strategy is sent to the Kubernetes control plane, and the corresponding repair operation is performed by calling the Kubernetes API. The repair operation includes one or more combinations of Pod restart operation, node expansion operation, and node eviction operation.

[0008] In an optional embodiment, Performing feature engineering on the multi-source data includes: Perform one-hot encoding on the controller type to which the Pod belongs to generate a controller type feature, perform normalization on the CPU utilization and memory usage data to generate a resource usage feature, perform raw count on the Pod restart count data to generate a restart count feature, perform Boolean feature conversion on the file damage keyword data to generate a log anomaly feature, perform ordinal encoding on the service level in the Pod label to generate a service level feature, and the service level is divided into multiple levels from high to low according to the business importance, perform Boolean feature conversion on the database storage type in the Pod storage volume declaration configuration to generate a storage volume feature, and perform time window statistics on the error rate data to generate a service call feature; Calculate the contribution of each feature to fault classification based on information gain, calculate the correlation between features based on the Pearson correlation coefficient, and screen and obtain an optimal feature set based on the contribution and correlation; Based on the preferred feature set, the controller type feature and the service level feature are cross-combined to generate a service attribute feature, the resource usage feature and the restart number feature are weightedly combined to generate a stability feature, the log anomaly feature and the storage volume feature are associated and combined to obtain an initial dependency feature, and the service call feature and the initial dependency feature are associated and combined to generate an enhanced dependency feature; the service attribute feature, the stability feature, and the enhanced dependency feature are spliced ​​to generate a multi-dimensional feature vector.

[0009] In an optional embodiment, Inputting the multi-dimensional feature vector into the XGBoost classifier for fault classification includes: Performing wavelet transform on the resource usage features to obtain time-frequency features, performing BERT encoding on the log anomaly features to obtain semantic features, concatenating the time-frequency features and the semantic features into the multi-dimensional feature vector to form an enhanced feature vector; performing mean-variance normalization on the enhanced feature vector to obtain a standardized feature vector; Perform sample balancing on historical fault annotated data to obtain training data, and select the training data based on annotation consistency and repair effect to form a core training set; The core training set is stratified according to the fault type and the service level feature, and the standardized feature vector is used to train the XGBoost classifier to obtain a hierarchical fault classification model; when the repair strategy corresponding to the fault type fails to execute, the layer to which the failure case belongs is determined, and the failure case is added to the negative sample pool of the corresponding layer; the hierarchical fault classification model of the corresponding layer is incrementally trained based on the time decay weights of the samples in the negative sample pool; Calculate the predicted probability distribution of the XGBoost classifier to obtain the uncertainty index of the sample classification, and manually confirm the samples whose uncertainty index exceeds the preset index threshold and add them to the core training set; An evaluation index system including accuracy, recall rate, and F1 score is constructed to evaluate the XGBoost classifier. When the evaluation index is lower than the preset evaluation threshold, the model parameters are optimized based on the consistency between the predicted results and the actual repair effect.

[0010] In an optional embodiment, The core training set is trained in layers and incremental training is performed based on failed samples. The core training set is allocated according to fault type and service level to obtain an initial stratified sample set. The sample sharing coefficient is calculated by calculating the ratio of the number of samples in each initial stratified sample set to the maximum number of samples in all initial stratified sample sets. When the sample sharing coefficient is less than a preset sample threshold, the cosine similarity between the adjacent level samples and the current level class center is calculated. The adjacent level samples whose cosine similarity is greater than the preset similarity threshold are added to the current level sample set to form an expanded stratified sample set. The samples in the expanded stratified sample set are trained to obtain a stratified fault classification model. Receive failed samples from a negative sample pool, calculate a time decay factor based on the time interval of the failed samples, calculate a degree of repair failure based on the ratio of the number of retries of the failed samples during the repair process to a preset maximum number of retries, calculate a sample importance coefficient based on the degree of repair failure, and use the product of the time decay factor and the sample importance coefficient as a time decay weight; The ratio of the number of samples in the negative sample pool to the preset maximum number of samples is calculated to obtain the sample capacity ratio, and the product of the difference between the preset benchmark learning rate and the sample capacity ratio is used as the adaptive learning rate; the ratio of the classification accuracy change to the time interval is calculated to obtain the accuracy change ratio, and the product of the preset benchmark update threshold and the accuracy change ratio is used as the dynamic update threshold; when the sum of the time decay weights is greater than the dynamic update threshold, the hierarchical fault classification model is incrementally trained using the adaptive learning rate to obtain an optimized hierarchical fault classification model.

[0011] In an optional embodiment, The basic policy template includes: A first basic policy template for data-type failures, which includes basic operations such as traffic suspension through the Ingress API, data backup through the CSI snapshot interface, and safe restart through the Kubernetes API; The second basic policy template for resource-based failures includes basic operations such as current limiting and degradation through the HPA interface, resource restriction through the Cgroup interface, and node eviction through the Kubernetes drain interface. The third basic policy template for dependency-type failures includes basic operations of performing dependency chain traversal through the dependency graph interface and performing circuit breaking and degradation through the Hystrix API.

[0012] In an optional embodiment, Combining the basic policy templates based on the controller type characteristics and the service level characteristics to form a combined repair policy includes: When the controller type feature is StatefulSet and the fault type is a data fault, the operations in the first basic policy template are sorted based on the historical repair success rate, and the traffic suspension operation and data backup operation with the highest repair success rate are selected. The storage volume retention operation is added based on the storage volume status evaluation, and the image switching operation is added based on the available image version evaluation to generate a first combined policy; When the controller type feature is Deployment and the fault type is a resource-based fault, the current limiting degradation threshold is determined based on the resource usage feature, the current limiting degradation operation is selected, and the Pod restart operation is added based on the Pod restart history evaluation, the number of replica expansion operations is increased based on the cluster resource margin evaluation, and the connection pool fuse parameter configuration is increased based on the service call volume evaluation to generate a second combination strategy; When the service level characteristic is higher than the preset level threshold and the fault type is a resource-based fault, the restriction parameters of the resource restriction operation are selected based on the multi-dimensional resource profile, and the optimal migration target node is selected based on the target node health score to generate a third combination strategy; According to different service level characteristics, different execution priorities are set for the first combination strategy, the second combination strategy or the third combination strategy.

[0013] In an optional embodiment, Building a service dependency graph to optimize the combined repair strategy includes: Construct a service dependency graph, which includes service node information, service call relationship edge information, and storage dependency edge information. The service node information includes Pod name, controller type characteristics, and service level characteristics. The service call relationship edge information records the direction and error rate data of request calls between services. The storage dependency edge information records the association between services and the storage volumes they declare to use. Based on the service dependency graph, the combined repair strategy is optimized, the error rate data propagation path in the service call relationship side information is analyzed, the storage volume access status in the storage dependency side information is analyzed, and the root cause of the fault and the propagation link are determined in combination with the resource usage characteristics; based on the controller type characteristics and service level characteristics in the service node information, the fault impact range is determined, and under the premise of maintaining the service level priority constraint, the execution order of the first combined strategy, the second combined strategy or the third combined strategy is optimized according to the root cause of the fault and the propagation link to generate an optimized repair strategy.

[0014] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0016] By constructing a multi-dimensional data collection and feature engineering system, the present invention achieves accurate classification of fault types in the Kubernetes environment, improves the accuracy of fault identification, and reduces the misjudgment rate, thereby making the self-healing process more accurate and efficient.

[0017] The present invention combines controller type and service level characteristics to optimize the basic policy templates in combination, and can formulate differentiated repair strategies for different types of applications and services. At the same time, it further optimizes the repair strategy through the service dependency graph to ensure that the repair operation complies with business logic and dependency relationships, thereby improving the accuracy and effectiveness of the repair strategy.

[0018] This paper automatically performs repair operations through the Kubernetes API, realizing full process automation from fault detection and classification to repair, significantly reducing manual intervention, shortening fault recovery time, and improving system availability. It has important practical value for large-scale cluster management. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an intelligent fault self-healing method based on multi-dimensional indicators according to an embodiment of the present invention; Figure 2 A flowchart illustrating the optimization of the combined repair strategy for building a service dependency graph. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 FIG. 1 is a flow chart of an intelligent fault self-healing method based on multi-dimensional indicators according to an embodiment of the present invention. Figure 1 As shown, the method includes: Collect Pod restart data through the Kubernetes API, CPU utilization and memory usage data through Prometheus, collect and parse file damage keyword data through the ELK Stack, and collect error rate data through custom application monitoring to form multi-source data. Performing feature engineering on the multi-source data, including feature encoding the controller type to which the Pod belongs to obtain controller type features, feature encoding the service level in the Pod label to obtain service level features, and generating a multi-dimensional feature vector; Inputting the multi-dimensional feature vector into the XGBoost classifier to classify the fault to obtain a fault type; selecting a basic strategy template according to the fault type; Combining the basic policy templates based on the controller type characteristics and the service level characteristics to form a combined repair strategy; constructing a service dependency graph to optimize the combined repair strategy; The optimized combined repair strategy is sent to the Kubernetes control plane, and the corresponding repair operation is performed by calling the Kubernetes API. The repair operation includes one or more combinations of Pod restart operation, node expansion operation, and node eviction operation.

[0023] In an optional embodiment, performing feature engineering on the multi-source data includes: Perform one-hot encoding on the controller type to which the Pod belongs to generate a controller type feature, perform normalization on the CPU utilization and memory usage data to generate a resource usage feature, perform raw count on the Pod restart count data to generate a restart count feature, perform Boolean feature conversion on the file damage keyword data to generate a log anomaly feature, perform ordinal encoding on the service level in the Pod label to generate a service level feature, and the service level is divided into multiple levels from high to low according to the business importance, perform Boolean feature conversion on the database storage type in the Pod storage volume declaration configuration to generate a storage volume feature, and perform time window statistics on the error rate data to generate a service call feature; Calculate the contribution of each feature to fault classification based on information gain, calculate the correlation between features based on the Pearson correlation coefficient, and screen and obtain an optimal feature set based on the contribution and correlation; Based on the preferred feature set, the controller type feature and the service level feature are cross-combined to generate a service attribute feature, the resource usage feature and the restart number feature are weightedly combined to generate a stability feature, the log anomaly feature and the storage volume feature are associated and combined to obtain an initial dependency feature, and the service call feature and the initial dependency feature are associated and combined to generate an enhanced dependency feature; the service attribute feature, the stability feature, and the enhanced dependency feature are spliced ​​to generate a multi-dimensional feature vector.

[0024] Exemplarily, this embodiment obtains Pod-related data in the Kubernetes cluster, including controller type, CPU utilization, memory usage, number of restarts, log keywords, Pod labels, and storage volume configuration, and then converts it into a feature vector that can be used by the machine learning model through feature engineering.

[0025] When generating controller type features using one-hot encoding for a pod's controller type, controller types such as Deployment, StatefulSet, and DaemonSet are mapped to binary vectors. For example, if there are three controller types and a pod belongs to a Deployment, a vector of [1,0,0] is generated; if it belongs to a StatefulSet, a vector of [0,1,0] is generated. This encoding method effectively preserves the categorical information of the controller type and avoids introducing inappropriate numerical ordering.

[0026] When normalizing CPU utilization and memory usage data to generate resource usage features, we use the min-max normalization method to convert CPU utilization and memory usage to a range of 0-1. Specifically, for the original value x, the normalized value is (x-min) / (max-min), where min and max are the minimum and maximum values ​​of the metric, respectively. For example, if the CPU utilization is 75%, the minimum is 10%, and the maximum is 90%, the normalized value is (75-10) / (90-10) = 0.8125. This process allows resource metrics of different scales to be compared on the same scale.

[0027] When generating the restart count feature from raw counts of Pod restart data, use the integer count directly. For example, if a Pod restarted three times during the observation period, the restart count feature value would be 3. Frequent Pod restarts are often a significant sign of stability issues, and retaining the raw count directly reflects this characteristic.

[0028] When performing Boolean feature conversion on file corruption keyword data to generate log anomaly features, we check whether the log contains keywords such as "file corruption" and "checksum error." If so, the feature value is 1; otherwise, it is 0. For example, if the words "data file checksum mismatch" appear in a pod's log, the log anomaly feature value is 1, indicating a file corruption issue.

[0029] When ordinal encoding the service levels in pod labels to generate service-level signatures, service levels are mapped to integers based on business importance, from high to low. For example, core services, critical services, standard services, and non-critical services are coded as 3, 2, 1, and 0, respectively. If a pod's label contains "service-level: critical," its service-level signature value is 2.

[0030] When generating a volume signature by converting the database storage type in a Pod's volume declaration configuration to a Boolean signature, the system checks whether the volume is a database-specific type. If so, the signature value is 1; otherwise, it is 0. For example, if a Pod's volume declaration specifies the "database-storage" type, the value of its volume signature is 1, indicating that the Pod is running a database service.

[0031] When generating service call features by performing time window statistics on error rate data, the service call error rate is calculated within a 5-minute sliding window. For example, if a pod received 100 requests in the last 5 minutes, 5 of which returned errors, the error rate is 5%. To capture the changing trends of the error rate, you can calculate the error rates for multiple consecutive windows, such as [3%, 4%, 5%, 4%, 6%], to form an error rate sequence as the service call feature.

[0032] When calculating feature contribution based on information gain, the correlation between the distribution of each feature and the fault type is statistically analyzed, and continuous features are discretized and divided into multiple intervals. Then, the ability of each feature value to distinguish fault classification is calculated. The higher the value, the greater the contribution of the feature to the classification.

[0033] When assessing feature redundancy based on the Pearson correlation coefficient, the linear correlation strength between each pair of features is calculated, with values ​​ranging from -1 to 1. This calculation requires normalizing the raw data and then calculating the ratio of the covariance between the feature pairs to their standard deviation. Values ​​close to ±1 indicate strong correlation, while values ​​close to 0 indicate weak correlation. A redundancy threshold of 0.8 is set to filter out highly overlapping features.

[0034] The optimal feature set selection process uses a two-step approach: first, all features are sorted in descending order of information gain, retaining the top 80% of high-contribution features. Then, the correlation coefficient of feature pairs is tested, and features with higher information gain are retained when they exceed 0.8. For example, the correlation coefficient between Pod status code and restart count is 0.87, so the restart count feature with higher information gain (0.33) is retained.

[0035] Based on the preferred feature set, the controller type feature and the service level feature are cross-combined to generate service attribute features using vector outer product or eigenvalue concatenation. For example, if the controller type is [1,0,0] and the service level is 2, the service attribute feature [2,0,0] can be generated, indicating that the pod is a deployment running critical services.

[0036] When weighted combining resource usage and restart count features to generate a stability feature, the normalized CPU utilization, memory utilization, and restart count are weighted and summed. For example, if CPU utilization is 0.8, memory utilization is 0.7, and the normalized restart count is 0.2, and the weights are 0.4, 0.4, and 0.2, respectively, the stability feature value is 0.8 × 0.4 + 0.7 × 0.4 + 0.2 × 0.2 = 0.64.

[0037] When correlating log anomaly signatures with storage volume signatures to generate the initial dependency signature, the combination of log anomalies and storage volume types is detected. For example, if both the log anomaly signature and the storage volume signature are 1, the initial dependency signature is 1, indicating a database storage-related failure; for other combinations, the initial dependency signature is 0.

[0038] When correlating service call features with initial dependency features to generate enhanced dependency features, the error rate trend is combined with the initial dependency features. For example, if the recent error rate shows an increasing trend of [3%, 4%, 5%, 7%, 10%] and the initial dependency feature is 1, the enhanced dependency feature can be represented as [0.03, 0.04, 0.05, 0.07, 0.10, 1], capturing the combined information of service call anomalies and dependency relationships.

[0039] Finally, the service attribute features, stability features, and enhanced dependency features are concatenated to generate a multi-dimensional feature vector. For example, if the service attribute features are [2, 0, 0], the stability feature is 0.64, and the enhanced dependency feature is [0.03, 0.04, 0.05, 0.07, 0.10, 1], the resulting feature vector is [2, 0, 0, 0.64, 0.03, 0.04, 0.05, 0.07, 0.10, 1]. This multi-dimensional feature vector comprehensively considers the pod's service attributes, operational stability, and dependencies, providing a rich information foundation for subsequent fault detection and classification.

[0040] This invention achieves comprehensive extraction and optimized combination of fault-related data through multi-dimensional feature engineering and feature combination, improving the expressiveness and discriminability of features. A feature screening mechanism based on information gain and correlation ensures feature validity, while cross-feature combination enhances the correlation between features, thereby improving the accuracy of subsequent fault classification.

[0041] In an optional embodiment, inputting the multi-dimensional feature vector into an XGBoost classifier for fault classification includes: Performing wavelet transform on the resource usage features to obtain time-frequency features, performing BERT encoding on the log anomaly features to obtain semantic features, concatenating the time-frequency features and the semantic features into the multi-dimensional feature vector to form an enhanced feature vector; performing mean-variance normalization on the enhanced feature vector to obtain a standardized feature vector; Perform sample balancing on historical fault annotated data to obtain training data, and select the training data based on annotation consistency and repair effect to form a core training set; The core training set is stratified according to the fault type and the service level feature, and the standardized feature vector is used to train the XGBoost classifier to obtain a hierarchical fault classification model; when the repair strategy corresponding to the fault type fails to execute, the layer to which the failure case belongs is determined, and the failure case is added to the negative sample pool of the corresponding layer; the hierarchical fault classification model of the corresponding layer is incrementally trained based on the time decay weights of the samples in the negative sample pool; Calculate the predicted probability distribution of the XGBoost classifier to obtain the uncertainty index of the sample classification, and manually confirm the samples whose uncertainty index exceeds the preset index threshold and add them to the core training set; An evaluation index system including accuracy, recall rate, and F1 score is constructed to evaluate the XGBoost classifier. When the evaluation index is lower than the preset evaluation threshold, the model parameters are optimized based on the consistency between the predicted results and the actual repair effect.

[0042] Exemplarily, the step of inputting the multi-dimensional feature vector into the XGBoost classifier for fault classification includes performing wavelet transform on resource usage features, performing BERT encoding on log anomaly features, feature splicing processing, and model training and optimization.

[0043] To characterize resource usage, we use the DB4 wavelet basis to perform a three-layer decomposition of time series data for resource metrics such as CPU utilization, memory usage, and disk I / O. Low-frequency coefficients are extracted as stable trend features, and high-frequency coefficients as fluctuation features. For example, we perform wavelet decomposition on the CPU utilization curve of a server 30 minutes before a failure, obtaining four sets of coefficients: low-frequency approximate coefficients [0.82, 0.85, 0.79, 0.91, 0.95] and three sets of high-frequency detail coefficients [0.03, -0.02, 0.04, -0.05, 0.01], [0.01, -0.03, 0.02, -0.01, 0.02], and [0.005, 0.002, -0.004, 0.003, -0.001]. These coefficients together constitute the time-frequency feature vector.

[0044] To process log anomaly features, key logs are preprocessed, including timestamp extraction, variable replacement, and error code standardization. Semantic representations are extracted from the processed logs using a pretrained BERT-Base model (12 layers, 768 hidden layers, and 12 attention heads). For example, encoding the error log "Connection timeout while accessing database server" yields a 768-dimensional semantic vector, with the first five dimensions being [0.153, -0.278, 0.421, -0.087, 0.309].

[0045] Concatenate the time-frequency features and semantic features to form an enhanced feature vector. Assuming the time-frequency feature dimension is 120 and the semantic feature dimension is 768, this concatenation results in an 888-dimensional enhanced feature vector. This feature vector is normalized by mean-variance normalization. The mean and standard deviation of each feature dimension in the training set are calculated. The mean is subtracted from the feature value and then divided by the standard deviation. For example, if the original value of the first dimension is 0.82, the mean of this dimension in the training set is 0.75, and the standard deviation is 0.08, the normalized value is (0.82 - 0.75) / 0.08 = 0.875.

[0046] When training data is generated by performing sample balancing on historical fault annotated data, the number of samples for each fault type is counted. For rare fault types with fewer than 100 samples (such as network partitions), the SMOTE algorithm is used to generate synthetic samples. For example, if a rare fault type has only 20 samples, the SMOTE algorithm interpolates the values ​​in the feature space to generate 80 synthetic samples, bringing the total number of samples for that category to 100. The sampling ratio is typically set to 80% of the overall average number of samples. When selecting training data based on annotation consistency, the success rate of repair strategy execution for each fault sample is calculated, and samples with a success rate greater than 85% are selected as the core training set.

[0047] The core training set was stratified based on fault type and service level characteristics. For example, faults were categorized into four types: "network fault," "disk fault," "memory fault," and "CPU fault." Service levels were divided into three levels: "critical business," "important business," and "normal business," resulting in a total of 12 sub-training sets. An XGBoost classifier was trained using standardized feature vectors to generate a hierarchical fault classification model. Different model parameters were configured for each layer: a deeper tree depth (maximum depth 8) and a smaller learning rate (0.01) were set for the critical business layer to ensure classification accuracy; a shallower tree depth (maximum depth 5) and a larger learning rate (0.05) were set for the normal business layer to improve training speed.

[0048] When the repair strategy for a fault type fails, the tier to which the failure case belongs is determined and added to the corresponding tier's negative sample pool. Time-decay-based weights are assigned to the samples in the negative sample pool. The hierarchical fault classification model for the corresponding tier is incrementally trained based on the time-decay weights of the samples in the negative sample pool. Incremental updates are performed weekly, with each update using negative samples from the last 30 days.

[0049] The XGBoost classifier's predicted probability distribution is calculated to obtain an uncertainty indicator for sample classification, using the predicted probability entropy as the uncertainty metric. For example, for a fault sample with a predicted probability distribution of [0.6, 0.3, 0.05, 0.05], the calculated entropy is 1.05. A preset threshold of 0.9 is set. Samples with an entropy exceeding 0.9 are manually confirmed and added to the core training set. During the manual confirmation phase, experts vote to determine the fault type, and the label is updated after reaching consensus.

[0050] An evaluation metric system consisting of accuracy, recall, and F1 score was constructed to evaluate the XGBoost classifier, and these metrics were regularly calculated on the test set. When an evaluation metric fell below a preset threshold (accuracy <0.85, recall <0.80, or F1 score <0.82), model parameters were optimized based on the consistency between the predicted results and the actual repair results. Parameter optimization included adjusting the number of trees (ranging from 50 to 500), maximum depth (ranging from 3 to 10), learning rate (ranging from 0.01 to 0.2), and sampling ratio (ranging from 0.6 to 0.9). A grid search was performed to determine the optimal parameter combination and applied it to the next round of model training, ensuring that the model maintained high classification performance in new fault scenarios.

[0051] Traditional Kubernetes fault detection mainly relies on threshold rules and simple statistical methods, which are difficult to cope with the complex and changeable cloud native environment, especially when the fault manifestations are diverse and interrelated. The present invention introduces a hierarchical XGBoost classification architecture, and stratifies the model according to the service level and fault type, solving the practical problem that services of different importance levels require different accuracy requirements. At the same time, a negative sample incremental learning mechanism based on time-decaying weights is designed, which enables the model to continuously learn from failure cases and self-optimize. Through sample balancing and incremental training mechanisms, the problem of uneven distribution of fault samples is solved, and the manual confirmation mechanism based on uncertainty indicators improves the reliability and adaptability of the model.

[0052] In an optional embodiment, performing layered training on the core training set and performing incremental training based on the failure samples includes: The core training set is allocated according to fault type and service level to obtain an initial stratified sample set. The sample sharing coefficient is calculated by calculating the ratio of the number of samples in each initial stratified sample set to the maximum number of samples in all initial stratified sample sets. When the sample sharing coefficient is less than a preset sample threshold, the cosine similarity between the adjacent level samples and the current level class center is calculated. The adjacent level samples whose cosine similarity is greater than the preset similarity threshold are added to the current level sample set to form an expanded stratified sample set. The samples in the expanded stratified sample set are trained to obtain a stratified fault classification model. Receive failed samples from a negative sample pool, calculate a time decay factor based on the time interval of the failed samples, calculate a degree of repair failure based on the ratio of the number of retries of the failed samples during the repair process to a preset maximum number of retries, calculate a sample importance coefficient based on the degree of repair failure, and use the product of the time decay factor and the sample importance coefficient as a time decay weight; The ratio of the number of samples in the negative sample pool to the preset maximum number of samples is calculated to obtain the sample capacity ratio, and the product of the difference between the preset benchmark learning rate and the sample capacity ratio is used as the adaptive learning rate; the ratio of the classification accuracy change to the time interval is calculated to obtain the accuracy change ratio, and the product of the preset benchmark update threshold and the accuracy change ratio is used as the dynamic update threshold; when the sum of the time decay weights is greater than the dynamic update threshold, the hierarchical fault classification model is incrementally trained using the adaptive learning rate to obtain an optimized hierarchical fault classification model.

[0053] For example, fault types are categorized into multiple categories, such as network faults, hardware faults, and software faults, and service levels are categorized into high, medium, and low. For example, given a core training set containing 10,000 fault samples, multiple initial stratified sample sets, such as network fault-high, network fault-medium, network fault-low, and hardware fault-high, can be obtained based on the combination of fault type and service level.

[0054] Taking the sample distribution of the initial stratified sample set as an example, there are 500 samples for network faults (high level), 2000 samples for hardware faults (medium level), and 300 samples for software faults (low level). Assuming the maximum number of samples is 2000, the sample sharing coefficient for network faults (high level) is 500 / 2000 = 0.25, and the sample sharing coefficient for software faults (low level) is 300 / 2000 = 0.15. When the sample sharing coefficient is less than the preset sample threshold, sample expansion is required. If the preset sample threshold is set to 0.2, the sample sharing coefficient for software faults (low level) of 0.15 is less than the preset sample threshold, requiring sample expansion. Calculate the cosine similarity between samples from adjacent levels and the cluster center of the current level. For software faults (low level), its adjacent levels are software faults (medium level) and other fault types (low level). Through feature vector calculation, assuming that the cosine similarity between 30 software fault-medium level samples and the class center of software fault-low level is greater than the preset similarity threshold of 0.8, these 30 samples are added to the software fault-low level sample set to form an expanded high-level sample set. After expansion, the number of samples reaches 330.

[0055] The random forest algorithm is used to train the classification model for the samples in the expanded stratum sample set. For example, for the expanded stratum sample set of software fault-low level, 330 samples are used for training to obtain a stratified fault classification model with a classification accuracy of 92%.

[0056] Receive failure samples from the negative sample pool. These failure samples are samples that the classification model cannot correctly classify. Suppose that after one week of operation, the negative sample pool has accumulated 50 failure samples. Calculate the time decay factor based on the time interval between failure samples. For example, for failure samples from 3 days ago, set the time decay factor to 0.7; for failure samples from 1 day ago, set the time decay factor to 0.9.

[0057] Assuming the preset maximum number of retries is 10, and a failed sample remains unrepaired after 7 retries, its repair failure degree is 7 / 10 = 0.7. The sample importance coefficient is calculated based on the repair failure degree. The repair failure degree can be used directly as the sample importance coefficient, or it can be converted using a mapping function. For example, the sample importance coefficient = 1 - e^(-repair failure degree), resulting in a corresponding sample importance coefficient of 0.5. The product of the time decay factor and the sample importance coefficient is used as the time decay weight. For this failed sample, the time decay weight = 0.7 × 0.5 = 0.35.

[0058] Assuming the preset maximum number of samples is 200 and the current negative sample pool has 50 samples, the sample size ratio = 50 / 200 = 0.25. The adaptive learning rate is calculated by multiplying the difference between the preset baseline learning rate and the sample size ratio. Assuming the preset baseline learning rate is 0.1, the adaptive learning rate = 0.1 × (1 - 0.25) = 0.075. Assuming the classification accuracy was 92% a week ago and is now 90%, with a time interval of 7 days, the accuracy change ratio = (92% - 90%) / 7 = 0.286% / day. The dynamic update threshold is calculated by multiplying the preset baseline update threshold and the accuracy change ratio. Assuming the preset baseline update threshold is 10, the dynamic update threshold = 10 × 0.286% = 2.86.

[0059] The sum of the time-decayed weights of all failure samples is calculated, assuming it is 15.5. When the sum of the time-decayed weights, 15.5, is greater than the dynamic update threshold of 2.86, the hierarchical fault classification model is incrementally trained using an adaptive learning rate of 0.075 to obtain an optimized hierarchical fault classification model.

[0060] The existing system uses a static batch training mode, which cannot adapt to newly emerging failure modes in a timely manner. When facing a continuously changing cloud environment, the model accuracy will drop significantly over time and requires frequent retraining. The cross-level sample supplementation strategy based on the sample sharing coefficient of the present invention solves the problem of sample scarcity at a specific level; a method for selecting adjacent level samples based on cosine similarity is designed to ensure that the supplemented samples have semantic relevance to the target level. The core innovation lies in the introduction of a multi-dimensional adaptive incremental learning mechanism, which captures the timeliness of faults through a time decay factor, quantifies the importance of samples through the degree of repair failure, and dynamically adjusts the learning parameters based on changes in sample capacity and accuracy to achieve an intelligent update strategy for the model. This method is particularly suitable for scenarios where real-time fault diagnosis is required, such as network equipment and cloud computing platforms, and can significantly improve reliability and service quality.

[0061] In an optional implementation, the basic policy template includes: A first basic policy template for data-type failures, which includes basic operations such as traffic suspension through the Ingress API, data backup through the CSI snapshot interface, and safe restart through the Kubernetes API; The second basic policy template for resource-based failures includes basic operations such as current limiting and degradation through the HPA interface, resource restriction through the Cgroup interface, and node eviction through the Kubernetes drain interface. A third basic policy template for dependency failures, including basic operations such as performing dependency chain traversal through the dependency graph interface and performing circuit breaking and degradation through the Hystrix API; A basic policy template is selected according to the fault type.

[0062] By way of example, this embodiment describes in detail a method for selecting an adaptive strategy based on fault types. This method achieves rapid response and recovery to faults by presetting basic strategy templates for different fault types.

[0063] Applications deployed in a Kubernetes environment face various types of failures, including data failures, resource failures, and dependency failures. After observing a failure, administrators need to determine the failure type based on the failure characteristics and select an appropriate basic policy template to perform recovery actions based on the failure type. In this implementation, a basic policy template is a collection of templates composed of multiple basic operations tailored to specific failure types.

[0064] The first basic policy template for data-related failures includes three core operations. When a data-related failure, such as database persistent volume corruption or data inconsistency, is detected, traffic is first suspended through the Ingress API. Specifically, the Kubernetes Ingress API is called to modify routing rules, temporarily routing user requests to the fault prompt page to prevent new requests from being written to the damaged database. For example, you can call the command "kubectl patch ingress database-ingress -p '{"spec":{"rules":[{"http":{"paths":[{"path":" / api","pathType":"Prefix","backend":{"service":{"name":"maintenance-service","port":{"number":80}}}}]}}]}}'" to redirect traffic originally destined for the database service to the maintenance service.

[0065] After traffic is suspended, data is backed up using the CSI snapshot interface. The Kubernetes Container Storage Interface (CSI) snapshot function is called to create a persistent volume data snapshot for subsequent data recovery. This is accomplished by creating a VolumeSnapshot resource, such as "kubectl create -f snapshot.yaml." The snapshot.yaml file defines the volume snapshot configuration, including information such as the snapshot name and data source. Data backups must be performed promptly before damage spreads. A typical backup timeout is 30 seconds to ensure critical data protection even under high load.

[0066] After the data backup is complete, a safe restart is performed through the Kubernetes API. This recreates the pod instance, allowing the application to load the backed-up data or repair minor data inconsistencies. This is accomplished by calling the "kubectl delete poddatabase-pod-name" and "kubectl apply -f database-deployment.yaml" commands to ensure that the data service restarts in a healthy state. For database instances with a large number of instances, a rolling restart parameter, maxUnavailable=25%, is set to ensure service availability.

[0067] The second basic policy template for resource-based failures also includes three core operations. When resource-based failures such as CPU usage exceeding 90%, memory usage exceeding 85%, or disk I / O saturation are detected, current limiting and degradation are first performed through the HPA (Horizontal Pod Autoscaler) interface. Create or modify HPA resources, adjust the number of replicas of application instances, and distribute load pressure. For example, for CPU-intensive applications, call "kubectl apply -f hpa-config.yaml" to create an HPA configuration, set the CPU target utilization to 60%, and the maximum number of replicas to 10, so that it can automatically scale according to the load.

[0068] When rate limiting and degradation fail to immediately alleviate resource pressure, resource restrictions are implemented through the Cgroup interface. Modify the Pod's resource configuration to limit the Pod's CPU and memory usage to prevent a single service from overconsuming cluster resources. This is achieved by calling "kubectl patch deployment resource-intensive-app-p '{"spec" :{ "template" :{ "spec" : { "containers" :[{ "name" : "main-container" , "resources":{ "limits" :{ "cpu" : "200m" , "memory" : "512Mi"}}}]}}}}'" to adjust the resource limits of the problematic container to a lower, safer value.

[0069] If resource pressure remains severe, perform a node eviction operation through the Kubernetes drain interface. This identifies nodes with resource anomalies and safely migrates pods from them to healthy nodes. This is accomplished by calling "kubectl drainnode-name --ignore-daemonsets --delete-local-data" to ensure safe migration of workloads on the node. In practice, the node eviction operation sends a SIGTERM signal to the application 15 seconds in advance, giving the application sufficient time to save state and close connections.

[0070] The third basic policy template for dependency failures includes two core operations. When dependency failures such as service call timeouts and third-party API response exceptions are detected, the dependency chain is first traversed through the dependency graph interface. The service dependency graph is queried to identify the source of the failure and the scope of the affected services. This is achieved by calling the dependency graph API "GET / api / v1 / dependencies / service-name" to obtain a complete list of dependencies and analyze the failed dependent services and their impact links. In actual applications, the traversal depth is limited to 5 layers to avoid recovery delays caused by excessive analysis.

[0071] After determining the scope of the fault, we use the Hystrix API to perform circuit breaking and downgrade operations. We configure circuit breaking policies for the faulty dependent services, blocking unstable calls and returning downgraded results or calling a backup service. This is achieved by calling the Hystrix configuration API to set circuit breaking parameters, such as a 50% error rate threshold, a 5000 millisecond wait time after circuit breaking, and a sampling window of 10 requests. This ensures that when a dependent service fails, abnormal calls are quickly cut off, maintaining core functionality.

[0072] After identifying the fault type, the corresponding basic policy template is selected based on the fault type. The fault classification result is associated with a predefined policy template through a policy mapping mechanism. A policy mapping table is maintained, where the key is the fault type identifier and the value is the corresponding basic policy template reference. For example, when the XGBoost classifier outputs the fault type "DataCorruption," the policy mapping table is queried to obtain the corresponding first basic policy template; when the fault type is "ResourceExhaustion," the second basic policy template is obtained; and when the fault type is "DependencyFailure," the third basic policy template is obtained. The policy selection logic is executed by the StrategySelector component, which receives the fault type as a parameter and returns the corresponding policy template instance. For example, if "checksumfailed" is detected in the database pod log and the I / O error rate exceeds the threshold, the fault is classified as a data-related fault and the first basic policy template is automatically selected. If the node CPU usage is detected to be continuously above 90%, the fault is classified as a resource-related fault and the second basic policy template is selected. The policy selection process also considers the severity of the fault. For faults that reach the "Critical" level, the policy template's accelerated execution mode will be enabled to shorten the operation interval and improve the repair response speed.

[0073] This invention establishes specialized basic policy templates for different fault types, enabling precise execution of fault repairs through combined API calls. This templated design improves the reusability of repair strategies while ensuring the standardization and controllability of repair operations. It can implement appropriate recovery measures for different fault types, significantly improving service availability and fault recovery efficiency.

[0074] In an optional implementation, combining the basic policy templates based on the controller type feature and the service level feature to form a combined repair policy includes: When the controller type feature is StatefulSet and the fault type is a data fault, the operations in the first basic policy template are sorted based on the historical repair success rate, and the traffic suspension operation and data backup operation with the highest repair success rate are selected. The storage volume retention operation is added based on the storage volume status evaluation, and the image switching operation is added based on the available image version evaluation to generate a first combined policy; Normalizing the CPU utilization and memory utilization data to generate resource usage characteristics; when the controller type characteristic is Deployment and the fault type is a resource-based fault, determining the current limiting degradation threshold based on the resource usage characteristics, selecting the current limiting degradation operation, and adding a Pod restart operation based on the Pod restart history evaluation, increasing the number of replica expansion operations based on the cluster resource margin evaluation, and adding a connection pool fuse parameter configuration based on the service call volume evaluation to generate a second combination strategy; When the service level characteristic is higher than the preset level threshold and the fault type is a resource-based fault, the restriction parameters of the resource restriction operation are selected based on the multi-dimensional resource profile, and the optimal migration target node is selected based on the target node health score to generate a third combination strategy; According to different service level characteristics, different execution priorities are set for the first combination strategy, the second combination strategy or the third combination strategy.

[0075] For example, when a StatefulSet controller is detected and the fault type is a data fault, the repair operations and their success rates for this type of fault are first extracted from the historical repair record database. For example, for a StatefulSet controller for a database service, historical records show a 92.3% success rate for traffic pause operations, an 87.6% success rate for data backup operations, a 65.2% success rate for volume rebuild operations, and a 58.7% success rate for container restart operations. Based on this data, the operations in the first basic policy template are sorted, prioritizing the traffic pause and data backup operations, which have the highest success rates.

[0076] Obtain the current storage volume status information through API calls, including the storage volume health status, data integrity check results, and mount status. Calculate the storage volume retention necessity score based on the preset storage volume evaluation rules. When the score exceeds the threshold of 75 points (out of 100 points), add a storage volume retention operation to the combined policy to ensure that the faulty data is not cleared for subsequent analysis. At the same time, search the image repository to obtain a list of available application image versions, and select the latest image version with a score of not less than 80 points based on the version stability score (calculated through historical operation data). If there is a mirror that meets the conditions, add a mirror switching operation to the combined policy and specify the target version to switch to. Through the above steps, the first combined policy is generated, which includes operations such as traffic suspension, data backup, storage volume retention, and mirror switching. Each operation contains detailed parameter configurations.

[0077] In scenarios where a Deployment controller has a resource failure, we first collect recent (e.g., 30 minutes) CPU and memory usage data. For a web service, for example, the raw data shows CPU utilization fluctuating between 60% and 95%, and memory utilization between 75% and 88%. We then process this data using the maximum-minimum normalization method, mapping the range to a range of 0-1 to obtain normalized resource usage characteristic values. Based on these characteristic values, we determine the trigger thresholds for throttling and degradation. For example, throttling is triggered when the normalized CPU utilization exceeds 0.85 and the normalized memory utilization exceeds 0.8. The degradation ratio is set to 30%, meaning that only 70% of the normalized request volume is accepted.

[0078] At the same time, the Pod restart history is analyzed. If it is found that the number of Pod restarts in the past 24 hours is less than 3 and the service returns to normal after each restart, a Pod restart operation is added to the second combination strategy. The resource margin data of each node in the cluster is also obtained through the cluster resource scheduler API to calculate the upper limit of resources that can be used for capacity expansion. When the node CPU allocatable resources exceed 25% of the total and the memory allocatable resources exceed 20%, a replica expansion operation is added to the strategy, and the optimal expansion number is calculated to be 50% of the current number of replicas (rounded up). Based on the service call volume data such as the recent average number of requests per second and average response time, the connection pool fuse parameters are configured, including setting the maximum number of connections to 85% of the current value, the timeout threshold to 200% of the average response time, and the circuit breaker closing time to 30 seconds. Through these operations, the second combination strategy is formed.

[0079] For scenarios where service-level characteristics exceed preset thresholds (e.g., SLA availability requirement ≥ 99.9%) and the failure type is resource-related, resource limit operating parameters are selected based on a multi-dimensional resource profile. This profile includes dimensions such as historical CPU usage distribution, memory usage trends, disk I / O frequency, and network throughput. For a key microservice, for example, resource usage patterns were analyzed and CPU limits were set to 150% of the requested values ​​and memory limits to 125% of the requested values, to avoid performance fluctuations caused by excessive resource competition.

[0080] The health score of each node in the cluster is also calculated. Scoring metrics include node resource utilization, recent failure counts, network latency, and disk health. For example, if node A scores 85, node B scores 92, and node C scores 78, node B, with the highest score, is selected as the migration target node. Precise migration parameters are then set in the third combined strategy, including the migration time window, resource reservation ratio, and downtime tolerance.

[0081] Execution priorities are set for different policy combinations based on service level characteristics. The third policy combination for high-level services (SLA ≥ 99.99%) receives the highest execution priority, "P0." The first policy combination for medium-level services (99.9% ≤ SLA < 99.99%) receives "P1." The second policy combination for low-level services (SLA < 99.9%) receives "P2." Priorities determine the order in which policies are executed in resource contention situations, ensuring that critical services are repaired first.

[0082] The present invention's controller type and service level-based policy combination mechanism enables scenario-based customization of repair strategies. Multi-dimensional evaluation and parameter adjustment improve the accuracy and adaptability of repair strategies, while a service level-based priority mechanism ensures priority protection for critical services.

[0083] In an optional implementation, constructing a service dependency graph to optimize the combined repair strategy includes: Construct a service dependency graph, which includes service node information, service call relationship edge information, and storage dependency edge information. The service node information includes Pod name, controller type characteristics, and service level characteristics. The service call relationship edge information records the direction and error rate data of request calls between services. The storage dependency edge information records the association between services and the storage volumes they declare to use. Based on the service dependency graph, the combined repair strategy is optimized, the error rate data propagation path in the service call relationship side information is analyzed, the storage volume access status in the storage dependency side information is analyzed, and the root cause of the fault and the propagation link are determined in combination with the resource usage characteristics; based on the controller type characteristics and service level characteristics in the service node information, the fault impact range is determined, and under the premise of maintaining the service level priority constraint, the execution order of the first combined strategy, the second combined strategy or the third combined strategy is optimized according to the root cause of the fault and the propagation link to generate an optimized repair strategy.

[0084] Figure 2A schematic diagram of the process for optimizing the combined repair strategy for building a service dependency graph. For example, during the service dependency graph construction process, service node information is first collected, including pod names, controller type characteristics, and service level characteristics. Pod names are directly obtained from the cluster manager, such as "payment-service-5f7d9c5b5b-2xvzp." Controller type characteristics include deployment methods (Deployment, StatefulSet, DaemonSet, etc.). For example, "payment-service" belongs to the "Deployment" controller type. Service level characteristics are categorized into three levels based on service importance: critical services, core services, and general services. For example, the payment service is marked as "critical service" and the log service is marked as "general service."

[0085] Service call relationship edge information is constructed by analyzing telemetry data from the service mesh. Information such as the source service, target service, request success rate, and error type is extracted from request logs collected by the service mesh proxy. For example, the edge information for an order service calling a payment service is recorded as follows: source node "order-service", target node "payment-service", total request volume 1000 times / minute, error rate 4.5%, and error type primarily "timeout error". The direction of the edge indicates the request flow, from the caller to the callee.

[0086] Storage dependency edge information is obtained by parsing the volume mount configuration declared by the Pod. The volume declaration information in the Pod configuration is extracted to establish an association between the Pod and the storage volume. For example, a storage dependency edge is established between the database service Pod "mysql-0" and the persistent volume "mysql-pv-data", recording the access mode as "ReadWriteOnce" and the status as "Bound".

[0087] After building a complete service dependency graph, we optimize and combine repair strategies based on this graph. The optimization process first analyzes the error rate data propagation paths along the service call relationship edges to identify the direction of the error rate anomaly. For example, if the payment service experiences a 60% error rate, while the order service that calls the payment service has a 45% error rate and the inventory service has a 50% error rate, the error rate transmission chain indicates that the payment service is the root cause.

[0088] Storage dependency side information analysis involves checking storage volume access status, such as "Available," "Bound," or "Pending." When a storage volume status anomaly is detected, such as when the "mysql-pv-data" status changes to "Failed," the associated database service is marked as a potential source of failure. Resource usage characteristics, including abnormal patterns such as a sudden increase in CPU usage from a normal 30% to 95%, a jump in memory usage from 50% to 88%, and a sudden increase in network traffic by 200%, are also analyzed to further identify the root cause of the failure and its propagation path.

[0089] When determining the scope of a fault's impact, use the controller type characteristics of the service node to assess the potential for fault spread. For example, a failure of a database service managed by a StatefulSet controller could lead to data consistency issues, while a failure of a stateless service managed by a Deployment controller would have a more manageable impact. Service level characteristics are used to determine the severity of the business impact, with failures of critical services taking priority over those of common services.

[0090] When generating optimized repair strategies, maintain service-level priority constraints, prioritizing critical services over core services, which in turn prioritize core services over general services. Based on the identified root causes and propagation paths, optimize the execution order of the first combination strategy (for pod-level failures), the second combination strategy (for node-level failures), or the third combination strategy (for cluster-level failures).

[0091] For example, when a memory leak in the payment service (a critical service) pod was detected, causing multiple service call failures, the optimized first combination strategy was implemented, restarting the payment service pod first, rather than simultaneously processing the affected order service. The specific command executed was "kubectl rollout restart deployment payment-service." After the restart, the monitored memory usage dropped below 40% of the normal level, and the error rate returned to below 0.5%.

[0092] If a storage failure causes multiple service anomalies, adjust the policy combination order. For example, first perform the storage repair operation "kubectl patch pv mysql-pv-data -p '{"spec":{"claimRef":null}}'" to unbind the abnormal storage volume, and then reschedule the relevant pods. Throughout the repair process, continuously update the service status and error rate information in the service dependency map, and adjust the priority and combination order of the repair policies in real time.

[0093] The optimized repair strategy is converted into a sequence of Kubernetes API calls. For pod restarts, a "DELETE / api / v1 / namespaces / {namespace} / pods / {name}" request is constructed with the "gracePeriodSeconds" parameter set to 30 seconds to ensure sufficient time for application state preservation. For node scaling, a "PATCH / apis / apps / v1 / namespaces / {namespace} / deployments / {name}" request is sent by modifying the deployment configuration, increasing the "spec.replicas" value, and setting the "maxSurge" parameter to ensure a smooth scaling process. For node evictions, a "POST / api / v1 / nodes / {name} / drain" request is constructed with the "--delete-local-data" and "--ignore-daemonsets" flags to ensure safe migration of pods to healthy nodes. A transactional execution model is implemented with timeouts and retry mechanisms for each API operation. If a single operation fails, the executed repair steps are automatically rolled back to ensure cluster consistency. After execution is completed, the repair operation log and execution results are recorded for subsequent effect evaluation and model optimization.

[0094] Traditional Kubernetes fault repair methods usually adopt an isolation strategy and lack an understanding of the complex dependencies between services, which may cause the repair operation to trigger a chain reaction - repairing one component may damage other services that depend on it. The present invention introduces multi-dimensional service dependency graph technology, breaking through the limitations of traditional single-dimensional topology graphs, and integrating three types of key information: service node attributes (including controller type and business level), service call relationships (including direction and error rate), and storage dependencies. The core innovation lies in the graph-based policy optimization mechanism, which identifies the root cause of the fault by analyzing the error rate propagation path, evaluates the data impact through storage access status, and determines the priority based on service attributes. Unlike existing technologies, the present invention realizes the optimization of repair strategies that take into account global dependency constraints, ensuring that repair operations are performed in the optimal order in complex microservice environments.

[0095] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0096] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0097] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent fault self-healing method based on multi-dimensional indicators, characterized by: include: Collect Pod restart data through the Kubernetes API, CPU utilization and memory usage data through Prometheus, collect and parse file damage keyword data through the ELK Stack, and collect error rate data through custom application monitoring to form multi-source data. Performing feature engineering on the multi-source data, including feature encoding the controller type to which the Pod belongs to obtain controller type features, feature encoding the service level in the Pod label to obtain service level features, and generating a multi-dimensional feature vector; Inputting the multi-dimensional feature vector into the XGBoost classifier to classify the fault to obtain a fault type; selecting a basic strategy template according to the fault type; Combining the basic policy templates based on the controller type characteristics and the service level characteristics to form a combined repair strategy; constructing a service dependency graph to optimize the combined repair strategy; The optimized combined repair strategy is sent to the Kubernetes control plane, and the corresponding repair operation is performed by calling the Kubernetes API. The repair operation includes one or more combinations of Pod restart operation, node expansion operation, and node eviction operation.

2. The method according to claim 1, characterized in that Performing feature engineering on the multi-source data includes: Perform one-hot encoding on the controller type to which the Pod belongs to generate a controller type feature, perform normalization on the CPU utilization and memory usage data to generate a resource usage feature, perform raw count on the Pod restart count data to generate a restart count feature, perform Boolean feature conversion on the file damage keyword data to generate a log anomaly feature, perform ordinal encoding on the service level in the Pod label to generate a service level feature, and the service level is divided into multiple levels from high to low according to the business importance, perform Boolean feature conversion on the database storage type in the Pod storage volume declaration configuration to generate a storage volume feature, and perform time window statistics on the error rate data to generate a service call feature; Calculate the contribution of each feature to fault classification based on information gain, calculate the correlation between features based on the Pearson correlation coefficient, and screen and obtain an optimal feature set based on the contribution and correlation; Based on the preferred feature set, the controller type feature and the service level feature are cross-combined to generate a service attribute feature, the resource usage feature and the restart number feature are weightedly combined to generate a stability feature, the log anomaly feature and the storage volume feature are associated and combined to obtain an initial dependency feature, and the service call feature and the initial dependency feature are associated and combined to generate an enhanced dependency feature; the service attribute feature, the stability feature, and the enhanced dependency feature are spliced ​​to generate a multi-dimensional feature vector.

3. The method according to claim 2, characterized in that Inputting the multi-dimensional feature vector into the XGBoost classifier for fault classification includes: Performing wavelet transform on the resource usage features to obtain time-frequency features, performing BERT encoding on the log anomaly features to obtain semantic features, concatenating the time-frequency features and the semantic features into the multi-dimensional feature vector to form an enhanced feature vector; performing mean-variance normalization on the enhanced feature vector to obtain a standardized feature vector; Perform sample balancing on historical fault annotated data to obtain training data, and select the training data based on annotation consistency and repair effect to form a core training set; The core training set is stratified according to the fault type and the service level feature, and the standardized feature vector is used to train the XGBoost classifier to obtain a hierarchical fault classification model; when the repair strategy corresponding to the fault type fails to execute, the layer to which the failure case belongs is determined, and the failure case is added to the negative sample pool of the corresponding layer; the hierarchical fault classification model of the corresponding layer is incrementally trained based on the time decay weights of the samples in the negative sample pool; Calculate the predicted probability distribution of the XGBoost classifier to obtain the uncertainty index of the sample classification, and manually confirm the samples whose uncertainty index exceeds the preset index threshold and add them to the core training set; An evaluation index system including accuracy, recall rate, and F1 score is constructed to evaluate the XGBoost classifier. When the evaluation index is lower than the preset evaluation threshold, the model parameters are optimized based on the consistency between the predicted results and the actual repair effect.

4. The method according to claim 3, characterized in that The core training set is trained in layers and incremental training is performed based on failed samples. The core training set is allocated according to fault type and service level to obtain an initial stratified sample set. The sample sharing coefficient is calculated by calculating the ratio of the number of samples in each initial stratified sample set to the maximum number of samples in all initial stratified sample sets. When the sample sharing coefficient is less than a preset sample threshold, the cosine similarity between the adjacent level samples and the current level class center is calculated. The adjacent level samples whose cosine similarity is greater than the preset similarity threshold are added to the current level sample set to form an expanded stratified sample set. The samples in the expanded stratified sample set are trained to obtain a stratified fault classification model. Receive failed samples from a negative sample pool, calculate a time decay factor based on the time interval of the failed samples, calculate a degree of repair failure based on the ratio of the number of retries of the failed samples during the repair process to a preset maximum number of retries, calculate a sample importance coefficient based on the degree of repair failure, and use the product of the time decay factor and the sample importance coefficient as a time decay weight; The ratio of the number of samples in the negative sample pool to the preset maximum number of samples is calculated to obtain the sample capacity ratio, and the product of the difference between the preset benchmark learning rate and the sample capacity ratio is used as the adaptive learning rate; the ratio of the classification accuracy change to the time interval is calculated to obtain the accuracy change ratio, and the product of the preset benchmark update threshold and the accuracy change ratio is used as the dynamic update threshold; when the sum of the time decay weights is greater than the dynamic update threshold, the hierarchical fault classification model is incrementally trained using the adaptive learning rate to obtain an optimized hierarchical fault classification model.

5. The method according to claim 1, wherein The basic policy template includes: A first basic policy template for data-type failures, which includes basic operations such as traffic suspension through the Ingress API, data backup through the CSI snapshot interface, and safe restart through the Kubernetes API; The second basic policy template for resource-based failures includes basic operations such as current limiting and degradation through the HPA interface, resource restriction through the Cgroup interface, and node eviction through the Kubernetes drain interface. The third basic policy template for dependency-type failures includes basic operations of performing dependency chain traversal through the dependency graph interface and performing circuit breaking and degradation through the Hystrix API.

6. The method according to claim 5, characterized in that Combining the basic policy templates based on the controller type characteristics and the service level characteristics to form a combined repair policy includes: When the controller type feature is StatefulSet and the fault type is a data fault, the operations in the first basic policy template are sorted based on the historical repair success rate, and the traffic suspension operation and data backup operation with the highest repair success rate are selected. The storage volume retention operation is added based on the storage volume status evaluation, and the image switching operation is added based on the available image version evaluation to generate a first combined policy; When the controller type feature is Deployment and the fault type is a resource-based fault, the current limiting degradation threshold is determined based on the resource usage feature, the current limiting degradation operation is selected, and the Pod restart operation is added based on the Pod restart history evaluation, the number of replica expansion operations is increased based on the cluster resource margin evaluation, and the connection pool fuse parameter configuration is increased based on the service call volume evaluation to generate a second combination strategy; When the service level characteristic is higher than the preset level threshold and the fault type is a resource-based fault, the restriction parameters of the resource restriction operation are selected based on the multi-dimensional resource profile, and the optimal migration target node is selected based on the target node health score to generate a third combination strategy; According to different service level characteristics, different execution priorities are set for the first combination strategy, the second combination strategy or the third combination strategy.

7. The method according to claim 6, characterized in that Building a service dependency graph to optimize the combined repair strategy includes: Construct a service dependency graph, which includes service node information, service call relationship edge information, and storage dependency edge information. The service node information includes Pod name, controller type characteristics, and service level characteristics. The service call relationship edge information records the direction and error rate data of request calls between services. The storage dependency edge information records the association between services and the storage volumes they declare to use. Based on the service dependency graph, the combined repair strategy is optimized, the error rate data propagation path in the service call relationship side information is analyzed, the storage volume access status in the storage dependency side information is analyzed, and the root cause of the fault and the propagation link are determined in combination with the resource usage characteristics; based on the controller type characteristics and service level characteristics in the service node information, the fault impact range is determined, and under the premise of maintaining the service level priority constraint, the execution order of the first combined strategy, the second combined strategy or the third combined strategy is optimized according to the root cause of the fault and the propagation link to generate an optimized repair strategy.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images