Knowledge base based microservice failure detection method

Abstract

The present application relates to the technical field of microservice architecture, in particular to a microservice fault detection method based on a knowledge base. The method comprises the following steps: S1, collecting log data and performance monitoring data of microservices, and preprocessing the collected data; S2, detecting abnormal conditions in the data through a machine learning model; S3, according to the results of the anomaly detection, finding matching known problems from the knowledge base, providing solutions, and automatically performing repair operations for common faults; S4, constructing a fault prediction model based on the data collected in S1 and the known fault patterns in the knowledge base, for identifying potential fault points. The method realizes comprehensive detection and prediction of various faults under the microservice architecture, covers data collection, preprocessing, anomaly detection, fault matching and diagnosis, fault prediction and preventive measures, forms a closed-loop fault handling process, and significantly improves the efficiency and accuracy of fault handling.

Description

Technical Field

[0001] This invention relates to the field of microservice architecture technology, and more specifically, to a microservice fault detection method based on a knowledge base. Background Technology

[0002] With the widespread adoption of cloud computing and microservice architectures, modern software systems are becoming increasingly complex, consisting of multiple interdependent services. This architectural pattern enhances system flexibility and scalability, but also introduces new challenges, particularly in fault detection and prevention. Traditional fault detection methods often struggle to cope with the dynamic changes and high concurrency requests in microservice environments, thus requiring more intelligent approaches to address these issues.

[0003] In summary, developing a knowledge-based microservice fault detection method, combined with intelligent anomaly detection, prediction, and automated processing mechanisms, is of great significance for the efficient operation and maintenance of modern microservice architectures. Summary of the Invention

[0004] The purpose of this invention is to provide a knowledge base-based microservice fault detection method to solve the problem that traditional fault detection methods mentioned in the background art are unable to cope with dynamic changes and high concurrency requests in a microservice environment.

[0005] To achieve the above objectives, the present invention aims to provide a knowledge-based microservice fault detection method, comprising the following steps:

[0006] S1. Collect log data and performance monitoring data from microservices, and preprocess the collected data;

[0007] S2. Detect anomalies in the data using machine learning models;

[0008] S3. Based on the results of anomaly detection, search for matching known issues from the knowledge base, provide solutions, and automatically perform repair operations for common faults;

[0009] S4. Based on the data collected in S1 and the known fault modes in the knowledge base, construct a fault prediction model to identify potential fault points.

[0010] As a further improvement to this technical solution, the specific steps in S1 are as follows:

[0011] S1.1 Use log management tools to collect log data from each microservice and use monitoring tools to obtain the performance metrics of the microservices;

[0012] S1.2 Remove irrelevant data, delete and correct outliers and missing values ​​in the data, and standardize the format of logs and performance data;

[0013] S1.3 Store the processed data in the data storage system;

[0014] S1.4. Associate log data with performance monitoring data and integrate real-time and historical data.

[0015] As a further improvement to this technical solution, the specific steps of S2 are as follows:

[0016] S2.1 Extract meaningful features from the data in S1.4, select the feature subset most relevant to fault detection, and reduce redundant features;

[0017] S2.2. Use the extracted features to construct a training dataset, train the machine learning model, find the optimal weight vector w and bias term b, and an appropriate slack variable ξ. i ;

[0018] S2.3. Input the new data sample into the trained machine learning model to perform anomaly detection. Sample x i Satisfy y i (w T φ(x i If b) < 1, then the sample is considered to represent an anomaly.

[0019] S2.4. Use cross-validation techniques to evaluate the model's performance, including at least calculating accuracy, recall, and F1 score, and adjusting model parameters to optimize performance.

[0020] As a further improvement to this technical solution, the feature extraction method in S2.1 is specifically as follows:

[0021]

[0022] Among them, X i Y represents the feature value of the i-th sample; i This represents the target value of the i-th sample; and The mean of the feature and the target variable are respectively, and vn represents the total number of samples.

[0023] As a further improvement to this technical solution, the machine learning model in S2.2 is specifically as follows:

[0024]

[0025] subject to y i (w T φ(x i )+b)≥1-ξ i ξ i ≥0, i=1,...,n,

[0026] Where w represents the weight vector, which determines the direction of the decision boundary; b represents the bias term, which determines the displacement of the decision boundary; ξ i y represents the slack variable; C represents the regularization parameter, which controls the degree of penalty for misclassification; i The class label representing the i-th sample; x i The feature vector representing the i-th sample includes log data and performance monitoring data; φ(x i ) represents the feature mapping function; n represents the total number of samples.

[0027] As a further improvement to this technical solution, the specific steps of S3 are as follows:

[0028] S3.1 Build a knowledge base, which includes descriptions of known failure modes, common failures, possible causes, and corresponding solutions;

[0029] S3.2 When the machine learning model detects an anomaly, it searches for a matching fault description in the knowledge base based on the anomaly type and context information.

[0030] S3.3 If there is no directly matching fault description in the knowledge base, text similarity calculation methods can be used to find the closest fault case.

[0031] S3.4 Based on the matched fault cases, provide one or more solutions. For common faults, there are pre-configured automated repair scripts and programs in the knowledge base, which can be directly executed to solve the problem.

[0032] S3.5 If automated repair fails, notify manual inspection and repair. After repair is completed, record the abnormal situation and repair procedure into the knowledge base.

[0033] As a further improvement to this technical solution, the specific steps of S4 are as follows:

[0034] S4.1 uses real-time and historical data integrated from S1.4, including preprocessed log data and performance monitoring data;

[0035] S4.2. Further select those features from the features extracted in S2.1 that are helpful in predicting future performance trends;

[0036] S4.3 Construct a fault prediction model;

[0037] S4.4 Introduce seasonal components into the fault prediction model and train the fault prediction model using historical data;

[0038] S4.5 Apply the fault prediction model to the latest data to predict the performance trend in the future. Use known fault modes in the knowledge base to compare with the predicted trend and use cosine similarity to find the matching fault modes.

[0039] S4.6. Based on the matched failure mode, automatically execute the corresponding preventive measures, which include at least adjusting resource allocation, optimizing configuration parameters, and deploying backup resources in advance.

[0040] S4.7 If a new failure mode is predicted but not covered by the knowledge base, then manual detection and prevention should be initiated, and the knowledge base should be updated to include the new failure mode and its solution.

[0041] As a further improvement to this technical solution, the fault prediction model in S4.3 is specifically as follows:

[0042]

[0043] in, y represents the predicted value at time t; t-i Represents the residual at time ti; e t-j φ represents the residual at time tj. i and θ j represents the parameters of the autoregression and moving average, respectively; c represents the constant term; p and q represent the number of terms in the autoregression and moving average, respectively.

[0044] As a further improvement to this technical solution, the algorithm formula in S4.4 after introducing seasonal components into the fault prediction model is as follows:

[0045]

[0046] Among them, Y t-k E represents the seasonal autoregressive term at time tk; t-l Φ represents the seasonal moving average term at time tl, where l is a multiple of the seasonal period; k Represents the seasonal autoregressive parameter; Θ L P represents the seasonal moving average parameter; Q represents the number of seasonal autoregressive terms; s represents the seasonal moving average term length.

[0047] As a further improvement to this technical solution, the cosine similarity in S4.5 is specifically as follows:

[0048]

[0049] Where cos(θ) represents the cosine value; n represents the length of the sequence; y represents the value of the predicted trend sequence at the i-th time point; fault，i This represents the value of the fault mode sequence in the knowledge base at the i-th time point.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] 1. By utilizing machine learning models for feature extraction and anomaly detection on preprocessed data, abnormal situations can be identified quickly and accurately. The trained machine learning models can effectively identify potential fault points and reduce false alarms and missed alarms. Introducing a fault prediction model based on seasonal factors, combined with historical fault patterns in a knowledge base, can predict potential fault points in advance, providing early warning information to the operations and maintenance team. Through preventative measures such as adjusting resource allocation and optimizing configuration parameters, the likelihood of faults is significantly reduced, improving system stability and availability.

[0052] 2. By recording the experience and solutions from each fault handling process into a knowledge base, the system can continuously learn and optimize, improving the efficiency of fault matching and diagnosis. For newly emerging fault modes, the system can also quickly respond and update the knowledge base through manual intervention, enhancing the system's adaptability and intelligence. For common faults, the method supports automatic execution of repair scripts to quickly restore system operation. For complex or unknown faults, by notifying manual intervention and combining intelligent matching from the knowledge base, precise repair guidance is provided, achieving efficient collaboration between automation and human intervention, significantly shortening fault handling time. The knowledge base-based microservice fault detection method proposed in this patent, through its comprehensiveness, accuracy, predictiveness, dynamic learning capabilities, and automated repair and human collaboration mechanisms, significantly improves the efficiency and accuracy of fault detection and handling under microservice architecture, reduces system operation and maintenance costs, enhances system stability and availability, and provides strong technical support for the efficient operation and maintenance of microservice architecture. Attached Figure Description

[0053] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1:

[0056] Please see Figure 1As shown, this embodiment provides a microservice fault detection method based on a knowledge base, including the following steps:

[0057] 1. A knowledge-based microservice fault detection method, characterized by the following steps:

[0058] S1. Collect log data and performance monitoring data from microservices, and preprocess the collected data.

[0059] The specific steps in S1 are as follows:

[0060] S1.1 Use log management tools (such as Logstash, Fluentd, etc.) to collect log data from each microservice. The log data should include, but is not limited to, error logs, warning logs, debug logs, etc., to record the service runtime status and exception information. Use monitoring tools (such as Prometheus, Tel egraf, etc.) to obtain the microservice performance metrics, such as CPU utilization, memory usage, disk I / O, network bandwidth, response time, error rate, etc.

[0061] S1.2 Remove irrelevant data, delete and correct outliers and missing values ​​in the data, and standardize the format of logs and performance data.

[0062] S1.3 Store the processed data in a data storage system, such as a time-series database (InfluxDB), a relational database (MySQL), or a NoSQL database (MongoDB);

[0063] S1.4. Associate log data with performance monitoring data and integrate real-time and historical data.

[0064] S2. Detect anomalies in the data using machine learning models.

[0065] The specific steps of S2 are as follows:

[0066] S2.1 Extract meaningful features from the data in S1.4. For example, features such as error frequency and the number of times a specific error code appears can be extracted from log data; peak CPU usage and peak memory usage can be extracted from performance monitoring data. Select the feature subset most relevant to fault detection, reduce redundant features, and improve the efficiency and accuracy of the model.

[0067] S2.2. Use the extracted features to construct a training dataset, train the machine learning model, find the optimal weight vector w and bias term b, and an appropriate slack variable ξ. i ;

[0068] S2.3. Input the new data sample into the trained machine learning model to perform anomaly detection. Sample xi Satisfy y i (w T φ(x i If b) < 1, then the sample is considered to represent an abnormal situation. If an abnormal situation is detected, the alarm device will send the abnormal situation to the staff.

[0069] S2.4. Use cross-validation techniques to evaluate the model's performance, including at least calculating accuracy, recall, and F1 score, and adjusting model parameters to optimize performance.

[0070] The feature extraction method in S2.1 is as follows:

[0071]

[0072] Among them, X i Y represents the feature value of the i-th sample; i This represents the target value for the i-th sample, either +1 or -1. and represents the mean of the feature and the target variable, respectively; n represents the total number of samples; by calculating the correlation coefficient between each feature and the target variable, the features with higher absolute values ​​of the correlation coefficient are selected as the final feature subset, which can reduce redundant features and improve the efficiency and accuracy of the model.

[0073] The machine learning model in S2.2 is specifically as follows:

[0074]

[0075] subject to y i (w T φ(x i )+b)≥1-ξ i ξ i ≥0, i=1,...,n,

[0076] Where w represents the weight vector, which determines the direction of the decision boundary; b represents the bias term, which determines the displacement of the decision boundary; ξ i represents the slack variable, allowing some data points to lie on the wrong side or within the boundary; C represents the regularization parameter, controlling the degree of penalty for misclassification, a larger CC indicates a lower tolerance for misclassification; y i The class label for the i-th sample, with a value of -1 or +1; x i The feature vector representing the i-th sample includes log data and performance monitoring data; φ(x i() represents the feature mapping function, which maps data in the original feature space to a higher-dimensional space, making linearly inseparable data linearly separable; n represents the total number of samples; in microservice fault detection, we can use collected log data and performance monitoring data as feature vectors x. i Label normal and abnormal data as y. i =+1 and y i =-1. By training a machine learning model, we can find a decision boundary to distinguish between normal and abnormal data.

[0077] S3. Based on the results of anomaly detection, search for matching known issues from the knowledge base, provide solutions, and automatically perform repair operations for common faults.

[0078] The specific steps of S3 are as follows:

[0079] S3.1 Build a knowledge base. The knowledge base contains descriptions of known failure modes, common failures, possible causes, and corresponding solutions. The knowledge base can be a structured database or a collection of documents.

[0080] S3.2 When the machine learning model detects an anomaly, it searches for a matching fault description in the knowledge base based on the anomaly type and context information (this function can be achieved using keyword matching, pattern matching, or natural language processing techniques).

[0081] S3.3 If there is no directly matching fault description in the knowledge base, text similarity calculation methods can be used to find the closest fault case. Common similarity calculation methods include cosine similarity, Jaccard similarity, etc.

[0082] S3.4 Based on the matched fault cases, provide one or more solutions. The solutions may include detailed step guides, scripts or links to external resources. For common faults, there are pre-configured automated repair scripts and programs in the knowledge base, which can be executed directly to solve the problem.

[0083] S3.5 If automated repair fails, notify manual inspection and repair. After repair is completed, record the abnormal situation and repair procedure into the knowledge base.

[0084] S4. Based on the data collected in S1 and the known fault modes in the knowledge base, construct a fault prediction model to identify potential fault points.

[0085] The specific steps of S4 are as follows:

[0086] S4.1 uses real-time and historical data integrated from S1.4, including preprocessed log data and performance monitoring data;

[0087] S4.2. Further select those features from the features extracted in S2.1 that are helpful in predicting future performance trends;

[0088] S4.3 Construct a fault prediction model;

[0089] S4.4 Introduce seasonal components into the fault prediction model and train the fault prediction model using historical data;

[0090] S4.5 Apply the fault prediction model to the latest data to predict the performance trend in the future. Use known fault modes in the knowledge base to compare with the predicted trend and use cosine similarity to find the matching fault modes.

[0091] S4.6. Based on the matched failure mode, automatically execute the corresponding preventive measures, which include at least adjusting resource allocation, optimizing configuration parameters, and deploying backup resources in advance.

[0092] S4.7 If a new failure mode is predicted but not covered by the knowledge base, then manual detection and prevention should be initiated, and the knowledge base should be updated to include the new failure mode and its solution.

[0093] The fault prediction model in S4.3 is specifically as follows:

[0094]

[0095] in, y represents the predicted value at time t; t-i Represents the residual at time ti; e t-j φ represents the residual at time tj. i and θ j represents the parameters of the autoregression and moving average, respectively; c represents the constant term; p and q represent the number of autoregression and moving average terms, respectively; this model can predict performance trends over a future period and thus identify potential failure points.

[0096] The algorithm formula in S4.4 after incorporating seasonal components into the fault prediction model is as follows:

[0097]

[0098] Among them, Y t-k E represents the seasonal autoregressive term at time tk, where k is a multiple of the seasonal period; t-l Φ represents the seasonal moving average term at time tl, where l is a multiple of the seasonal period; k Represents the seasonal autoregressive parameter; Θ Lrepresents the seasonal moving average parameter; P represents the number of seasonal autoregressive terms; Q represents the number of seasonal moving average terms; s represents the seasonal cycle length; by introducing seasonal components, the model can better capture the periodic patterns in the data, thereby improving the accuracy of predictions.

[0099] The cosine similarity in S4.5 is specifically as follows:

[0100]

[0101] Where cos(θ) represents the cosine value, which is the cosine of the angle between two vectors; n represents the length of the sequence, that is, the number of elements in the sequence; y represents the value of the predicted trend sequence at the i-th time point; fault，i This represents the value of the fault mode sequence in the knowledge base at time point i. In fault detection and prediction, by calculating the cosine similarity between the predicted trend sequence and the known fault modes in the knowledge base, it can be determined whether the predicted trend matches the known fault modes. If the cosine similarity between the predicted trend and a certain fault mode is higher than a certain threshold (e.g., 0.7 or higher), then the two sequences are considered similar, and there may be potential fault points. In this way, potential faults can be identified and prevented in advance, improving the stability and availability of the system.

[0102] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A knowledge base based microservice failure detection method, characterized in that: The method comprises the following steps: S1, collecting log data and performance monitoring data of microservices, and preprocessing the collected data; S2, detecting abnormal conditions in the data through a machine learning model; S3, according to the results of anomaly detection, finding matching known problems from the knowledge base, providing solutions, and automatically performing repair operations for common faults; S4, according to the data collected in S1 and the known fault patterns in the knowledge base, a fault prediction model is constructed to identify potential fault points; The specific steps of S4 are: S4.1, using real-time and historical data integrated from S1.4, including preprocessed log data and performance monitoring data; S4.2, further selecting features from the features extracted in S2.1 that are helpful in predicting future performance trends; S4.3, constructing a fault prediction model; S4.4, introducing seasonal components into the fault prediction model and training the fault prediction model using historical data; S4.5, applying the fault prediction model to the latest data to predict performance trends in the future, comparing the predicted trends with the known fault patterns in the knowledge base, and finding matching fault patterns using cosine similarity; S4.6, according to the matched fault patterns, automatically performing corresponding preventive measures, including at least adjusting resource allocation, optimizing configuration parameters, and deploying standby resources in advance; S4.7, if the predicted new fault pattern is not covered by the knowledge base, notify human to detect and prevent, and update the knowledge base to include the new fault pattern and its solution. 2.The knowledge base based microservice fault detection method of claim 1, wherein: The specific steps in S1 are as follows: S1.1, use log management tools to collect log data from each microservice, and use monitoring tools to obtain performance indicators of microservices; S1.2, remove irrelevant data, delete and correct outliers and missing values in the data, and unify the format of log and performance data; S1.3, store the processed data in a data storage system; S1.4, associate log data with performance monitoring data, and integrate real-time and historical data. 3.The knowledge base based microservice fault detection method of claim 1, wherein: The specific steps of S2 are as follows: S2.1, extract meaningful features from S1.4 data, select the most relevant feature subset for fault detection, and reduce redundant features; S2.2, using the extracted features to construct a training dataset, train a machine learning model, find the optimal weight vector and bias terms , and appropriate slack variables ; S2.3, inputting the new data sample to the trained machine learning model, performing anomaly detection, the sample satisfies then the sample is considered to represent an anomaly; S2.4, use cross-validation techniques to evaluate the performance of the model, including at least calculating accuracy, recall, and F1 score indicators, and adjusting model parameters to optimize performance. 4.The knowledge base based microservice fault detection method of claim 3, wherein: The feature extraction method in S2.1 is as follows: ； wherein, represent the feature values of the first sample; represent the target values of the first sample; and represent the mean values of the feature and target variables, respectively; represents the total number of samples.

5. The knowledge base based microservice fault detection method of claim 3, wherein: The machine learning model in S2.2 is as follows: ； in, This represents the weight vector, which determines the direction of the decision boundary; The bias term determines the displacement of the decision boundary; Represents slack variables; This represents the regularization parameter, which controls the degree of penalty for misclassification; Representing the The category labels of each sample; Representing the The feature vector of each sample, including log data and performance monitoring data; Represents the feature mapping function; Represents the total number of samples. 6.The knowledge base based microservice fault detection method of claim 1, wherein: The specific steps of S3 are as follows: S3.1, build a knowledge base containing known fault patterns, descriptions of common faults, possible causes, and corresponding solutions; S3.2, when the machine learning model detects an anomaly, find a matching fault description in the knowledge base according to the anomaly type and context information; S3.3, if there is no direct matching fault description in the knowledge base, use text similarity calculation methods to find the closest fault case; S3.4, according to the matched fault case, one or more solutions are provided, for common faults, the knowledge base has pre-configured automated repair scripts and programs, which can be directly executed to solve the problem; S3.5, if the automated repair fails, notify the human to check and repair, and after the repair is completed, the abnormal situation and the repair procedure are entered into the knowledge base.

7. The knowledge base based microservice fault detection method of claim 1, wherein: The fault prediction model in S4.3 is specifically: ； wherein, represents a prediction value of the time instant; represents a residual of the time instant; represents a residual of the time instant; and represent the parameters of the autoregressive and moving average, respectively; represents a constant term; and represent the number of autoregressive and moving average terms, respectively.

8. The knowledge base based microservice fault detection method of claim 7, wherein: The algorithm formula after introducing the seasonal component into the fault prediction model in S4.4 is: ； wherein represents a seasonal autoregressive term at time ; represents a seasonal moving average term at time , wherein is a multiple of the seasonal period; represents a seasonal autoregressive parameter; represents a seasonal moving average parameter; represents the number of seasonal autoregressive terms; represents the number of seasonal moving average terms; represents the length of the seasonal period. 9.The knowledge base based microservice fault detection method of claim 1, wherein: The cosine similarity in S4.5 is specifically: ； wherein, represents a cosine value; represents a length of the sequence; represents a value of the predicted trend sequence at the time point; represents a value of the failure mode sequence in the knowledge base at the time point.

Images