Microservice fault detection method based on knowledge base

Abstract

The invention relates to the technical field of micro-service architecture, in particular to a micro-service fault detection method based on a knowledge base. The method comprises the following steps: S1, collecting log data and performance monitoring data of a micro-service, and preprocessing the collected data; s2, detecting abnormal conditions in the data through a machine learning model; s3, according to the result of the anomaly detection, searching for a matched known problem from a knowledge base, providing a solution, and automatically executing a repair operation for a common fault; and S4, according to the data collected in the S1 and a known fault mode of the knowledge base, constructing a fault prediction model for identifying potential fault points. According to the method, comprehensive detection and prediction of various faults under the micro-service architecture are realized, the method covers data collection, preprocessing, anomaly detection, fault matching and diagnosis, and fault prediction and prevention measures, a closed-loop fault processing flow is formed, and the efficiency and accuracy of fault processing are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of microservice architecture technology, and more specifically, to a microservice fault detection method based on a knowledge base. Background Technology

[0002] With the widespread adoption of cloud computing and microservice architectures, modern software systems are becoming increasingly complex, consisting of multiple interdependent services. This architectural pattern enhances system flexibility and scalability, but also introduces new challenges, particularly in fault detection and prevention. Traditional fault detection methods often struggle to cope with the dynamic changes and high concurrency requests in microservice environments, thus requiring more intelligent approaches to address these issues.

[0003] In summary, developing a knowledge-based microservice fault detection method, combined with intelligent anomaly detection, prediction, and automated processing mechanisms, is of great significance for the efficient operation and maintenance of modern microservice architectures. Summary of the Invention

[0004] The purpose of this invention is to provide a knowledge base-based microservice fault detection method to solve the problem that traditional fault detection methods mentioned in the background art are unable to cope with dynamic changes and high concurrency requests in a microservice environment.

[0005] To achieve the above objectives, the present invention aims to provide a knowledge-based microservice fault detection method, comprising the following steps:

[0006] S1. Collect log data and performance monitoring data from microservices, and preprocess the collected data;

[0007] S2. Detect anomalies in the data using machine learning models;

[0008] S3. Based on the results of anomaly detection, search for matching known issues from the knowledge base, provide solutions, and automatically perform repair operations for common faults;

[0009] S4. Based on the data collected in S1 and the known fault modes in the knowledge base, construct a fault prediction model to identify potential fault points.

[0010] As a further improvement to this technical solution, the specific steps in S1 are as follows:

[0011] S1.1 Use log management tools to collect log data from each microservice and use monitoring tools to obtain the performance metrics of the microservices;

[0012] S1.2 Remove irrelevant data, delete and correct outliers and missing values ​​in the data, and standardize the format of logs and performance data;

[0013] S1.3. Store the processed data in the data storage system;

[0014] S1.4. Associate log data with performance monitoring data and integrate real-time and historical data.

[0015] As a further improvement to this technical solution, the specific steps of S2 are as follows:

[0016] S2.1 Extract meaningful features from the data in S1.4, select the feature subset most relevant to fault detection, and reduce redundant features;

[0017] S2.2. Use the extracted features to construct a training dataset, train the machine learning model, find the optimal weight vector w and bias term b, and an appropriate slack variable ξ. i ;

[0018] S2.3. Input the new data sample into the trained machine learning model to perform anomaly detection. Sample x i Satisfy y i (w T φ(x i If b) < 1, then the sample is considered to represent an anomaly.

[0019] S2.4. Use cross-validation techniques to evaluate the model's performance, including at least calculating accuracy, recall, and F1 score, and adjusting model parameters to optimize performance.

[0020] As a further improvement to this technical solution, the feature extraction method in S2.1 is specifically as follows:

[0021]

[0022] Among them, X i Y represents the feature value of the i-th sample; i This represents the target value of the i-th sample; and The mean of the feature and the target variable are respectively, and vn represents the total number of samples.

[0023] As a further improvement to this technical solution, the machine learning model in S2.2 is specifically as follows:

[0024]

[0025] subject to y i (w T φ(x i )+b)≥1-ξ i ξ i ≥0, i=1,...,n,

[0026] Where w represents the weight vector, which determines the direction of the decision boundary; b represents the bias term, which determines the displacement of the decision boundary; ξ i y represents the slack variable; C represents the regularization parameter, which controls the degree of penalty for misclassification; i The class label representing the i-th sample; x i The feature vector representing the i-th sample includes log data and performance monitoring data; φ(x i ) represents the feature mapping function; n represents the total number of samples.

[0027] As a further improvement to this technical solution, the specific steps of S3 are as follows:

[0028] S3.1 Build a knowledge base, which includes descriptions of known failure modes, common failures, possible causes, and corresponding solutions;

[0029] S3.2 When the machine learning model detects an anomaly, it searches for a matching fault description in the knowledge base based on the anomaly type and context information.

[0030] S3.3 If there is no directly matching fault description in the knowledge base, text similarity calculation methods can be used to find the closest fault case.

[0031] S3.4 Based on the matched fault cases, provide one or more solutions. For common faults, there are pre-configured automated repair scripts and programs in the knowledge base, which can be directly executed to solve the problem.

[0032] S3.5 If automated repair fails, notify manual inspection and repair. After repair is completed, record the abnormal situation and repair procedure into the knowledge base.

[0033] As a further improvement to this technical solution, the specific steps of S4 are as follows:

[0034] S4.1 uses real-time and historical data integrated from S1.4, including preprocessed log data and performance monitoring data;

[0035] S4.2. Further select those features from the features extracted in S2.1 that are helpful in predicting future performance trends;

[0036] S4.3 Construct a fault prediction model;

[0037] S4.4 Introduce seasonal components into the fault prediction model and train the fault prediction model using historical data;

[0038] S4.5 Apply the fault prediction model to the latest data to predict the performance trend in the future. Use known fault modes in the knowledge base to compare with the predicted trend and use cosine similarity to find the matching fault modes.

[0039] S4.6. Based on the matched failure mode, automatically execute the corresponding preventive measures, which include at least adjusting resource allocation, optimizing configuration parameters, and deploying backup resources in advance.

[0040] S4.7 If a new failure mode is predicted but not covered by the knowledge base, then manual detection and prevention should be initiated, and the knowledge base should be updated to include the new failure mode and its solution.

[0041] As a further improvement to this technical solution, the fault prediction model in S4.3 is specifically as follows:

[0042]

[0043] in, y represents the predicted value at time t; t-i Represents the residual at time ti; e t-j φ represents the residual at time tj; i and θ j represents the parameters of the autoregression and moving average, respectively; c represents the constant term; p and q represent the number of terms in the autoregression and moving average, respectively.

[0044] As a further improvement to this technical solution, the algorithm formula in S4.4 after introducing seasonal components into the fault prediction model is as follows:

[0045]

[0046] Among them, Y t-k E represents the seasonal autoregressive term at time tk; t-l Φ represents the seasonal moving average term at time tl, where l is a multiple of the seasonal period; k Represents the seasonal autoregressive parameter; Θ L P represents the seasonal moving average parameter; Q represents the number of seasonal autoregressive terms; s represents the seasonal moving average term length.

[0047] As a further improvement to this technical solution, the cosine similarity in S4.5 is specifically as follows:

[0048]

[0049] Where cos(θ) represents the cosine value; n represents the length of the sequence; y represents the value of the predicted trend sequence at the i-th time point; fault，i This represents the value of the fault mode sequence in the knowledge base at the i-th time point.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] 1. By utilizing machine learning models for feature extraction and anomaly detection on preprocessed data, abnormal situations can be identified quickly and accurately. The trained machine learning models can effectively identify potential fault points and reduce false alarms and missed alarms. Introducing a fault prediction model based on seasonal factors, combined with historical fault patterns in a knowledge base, can predict potential fault points in advance, providing early warning information to the operations and maintenance team. Through preventative measures such as adjusting resource allocation and optimizing configuration parameters, the likelihood of faults is significantly reduced, improving system stability and availability.

[0052] 2. By recording the experience and solutions from each fault handling process into a knowledge base, the system can continuously learn and optimize, improving the efficiency of fault matching and diagnosis. For newly emerging fault modes, the system can also quickly respond and update the knowledge base through manual intervention, enhancing the system's adaptability and intelligence. For common faults, the method supports automatic execution of repair scripts to quickly restore system operation. For complex or unknown faults, by notifying manual intervention and combining intelligent matching from the knowledge base, precise repair guidance is provided, achieving efficient collaboration between automation and human intervention, significantly shortening fault handling time. The knowledge base-based microservice fault detection method proposed in this patent, through its comprehensiveness, accuracy, predictiveness, dynamic learning capabilities, and automated repair and human collaboration mechanisms, significantly improves the efficiency and accuracy of fault detection and handling under microservice architecture, reduces system operation and maintenance costs, enhances system stability and availability, and provides strong technical support for the efficient operation and maintenance of microservice architecture. Attached Figure Description

[0053] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1:

[0056] Please see Figure 1As shown, this embodiment provides a microservice fault detection method based on a knowledge base, including the following steps:

[0057] 1. A knowledge-based microservice fault detection method, characterized by the following steps:

[0058] S1. Collect log data and performance monitoring data from microservices, and preprocess the collected data.

[0059] The specific steps in S1 are as follows:

[0060] S1.1 Use log management tools (such as Logstash, Fluentd, etc.) to collect log data from each microservice. The log data should include, but is not limited to, error logs, warning logs, debug logs, etc., to record the service runtime status and exception information. Use monitoring tools (such as Prometheus, Tel egraf, etc.) to obtain the microservice performance metrics, such as CPU utilization, memory usage, disk I / O, network bandwidth, response time, error rate, etc.

[0061] S1.2 Remove irrelevant data, delete and correct outliers and missing values ​​in the data, and standardize the format of logs and performance data.

[0062] S1.3 Store the processed data in a data storage system, such as a time-series database (InfluxDB), a relational database (MySQL), or a NoSQL database (MongoDB);

[0063] S1.4. Associate log data with performance monitoring data and integrate real-time and historical data.

[0064] S2. Detect anomalies in the data using machine learning models.

[0065] The specific steps of S2 are as follows:

[0066] S2.1 Extract meaningful features from the data in S1.4. For example, features such as error frequency and the number of times a specific error code appears can be extracted from log data; peak CPU usage and peak memory usage can be extracted from performance monitoring data. Select the feature subset most relevant to fault detection, reduce redundant features, and improve the efficiency and accuracy of the model.

[0067] S2.2. Use the extracted features to construct a training dataset, train the machine learning model, find the optimal weight vector w and bias term b, and an appropriate slack variable ξ. i ;

[0068] S2.3. Input the new data sample into the trained machine learning model to perform anomaly detection. Sample xi Satisfy y i (w T φ(x i If b) < 1, then the sample is considered to represent an abnormal situation. If an abnormal situation is detected, the alarm device will send the abnormal situation to the staff.

[0069] S2.4. Use cross-validation techniques to evaluate the model's performance, including at least calculating accuracy, recall, and F1 score, and adjusting model parameters to optimize performance.

[0070] The feature extraction method in S2.1 is as follows:

[0071]

[0072] Among them, X i Y represents the feature value of the i-th sample; i This represents the target value for the i-th sample, either +1 or -1. and represents the mean of the feature and the target variable, respectively; n represents the total number of samples; by calculating the correlation coefficient between each feature and the target variable, the features with higher absolute values ​​of the correlation coefficient are selected as the final feature subset, which can reduce redundant features and improve the efficiency and accuracy of the model.

[0073] The machine learning model in S2.2 is specifically as follows:

[0074]

[0075] subject to y i (w T φ(x i )+b)≥1-ξ i ξ i ≥0, i=1,...,n,

[0076] Where w represents the weight vector, which determines the direction of the decision boundary; b represents the bias term, which determines the displacement of the decision boundary; ξ i represents the slack variable, allowing some data points to lie on the wrong side or within the boundary; C represents the regularization parameter, controlling the degree of penalty for misclassification; a larger CC indicates a lower tolerance for misclassification; y i The class label for the i-th sample, with a value of -1 or +1; x i The feature vector representing the i-th sample includes log data and performance monitoring data; φ(x i() represents the feature mapping function, which maps data in the original feature space to a higher-dimensional space, making linearly inseparable data linearly separable; n represents the total number of samples; in microservice fault detection, we can use collected log data and performance monitoring data as feature vectors x. i Label normal and abnormal data as y. i =+1 and y i =-1. By training a machine learning model, we can find a decision boundary to distinguish between normal and abnormal data.

[0077] S3. Based on the results of anomaly detection, search for matching known issues from the knowledge base, provide solutions, and automatically perform repair operations for common faults.

[0078] The specific steps of S3 are as follows:

[0079] S3.1 Build a knowledge base. The knowledge base contains descriptions of known failure modes, common failures, possible causes, and corresponding solutions. The knowledge base can be a structured database or a collection of documents.

[0080] S3.2 When the machine learning model detects an anomaly, it searches for a matching fault description in the knowledge base based on the anomaly type and context information (this function can be achieved using keyword matching, pattern matching, or natural language processing techniques).

[0081] S3.3 If there is no directly matching fault description in the knowledge base, text similarity calculation methods can be used to find the closest fault case. Common similarity calculation methods include cosine similarity, Jaccard similarity, etc.

[0082] S3.4 Based on the matched fault cases, provide one or more solutions. The solutions may include detailed step guides, scripts or links to external resources. For common faults, there are pre-configured automated repair scripts and programs in the knowledge base, which can be executed directly to solve the problem.

[0083] S3.5 If automated repair fails, notify manual inspection and repair. After repair is completed, record the abnormal situation and repair procedure into the knowledge base.

[0084] S4. Based on the data collected in S1 and the known fault modes in the knowledge base, construct a fault prediction model to identify potential fault points.

[0085] The specific steps of S4 are as follows:

[0086] S4.1 uses real-time and historical data integrated from S1.4, including preprocessed log data and performance monitoring data;

[0087] S4.2. Further select those features from the features extracted in S2.1 that are helpful in predicting future performance trends;

[0088] S4.3 Construct a fault prediction model;

[0089] S4.4 Introduce seasonal components into the fault prediction model and train the fault prediction model using historical data;

[0090] S4.5 Apply the fault prediction model to the latest data to predict the performance trend in the future. Use known fault modes in the knowledge base to compare with the predicted trend and use cosine similarity to find the matching fault modes.

[0091] S4.6. Based on the matched failure mode, automatically execute the corresponding preventive measures, which include at least adjusting resource allocation, optimizing configuration parameters, and deploying backup resources in advance.

[0092] S4.7 If a new failure mode is predicted but not covered by the knowledge base, then manual detection and prevention should be initiated, and the knowledge base should be updated to include the new failure mode and its solution.

[0093] The fault prediction model in S4.3 is specifically as follows:

[0094]

[0095] in, y represents the predicted value at time t; t-i Represents the residual at time ti; e t-j φ represents the residual at time tj; i and θ j represents the parameters of the autoregression and moving average, respectively; c represents the constant term; p and q represent the number of autoregression and moving average terms, respectively; this model can predict performance trends over a future period and thus identify potential failure points.

[0096] The algorithm formula in S4.4 after incorporating seasonal components into the fault prediction model is as follows:

[0097]

[0098] Among them, Y t-k E represents the seasonal autoregressive term at time tk, where k is a multiple of the seasonal period; t-l Φ represents the seasonal moving average term at time tl, where l is a multiple of the seasonal period; k Represents the seasonal autoregressive parameter; Θ Lrepresents the seasonal moving average parameter; P represents the number of seasonal autoregressive terms; Q represents the number of seasonal moving average terms; s represents the seasonal cycle length; by introducing seasonal components, the model can better capture the periodic patterns in the data, thereby improving the accuracy of predictions.

[0099] The cosine similarity in S4.5 is specifically as follows:

[0100]

[0101] Where cos(θ) represents the cosine value, which is the cosine of the angle between two vectors; n represents the length of the sequence, that is, the number of elements in the sequence; y represents the value of the predicted trend sequence at the i-th time point; fault，i This represents the value of the fault mode sequence in the knowledge base at time point i. In fault detection and prediction, by calculating the cosine similarity between the predicted trend sequence and the known fault modes in the knowledge base, it can be determined whether the predicted trend matches the known fault modes. If the cosine similarity between the predicted trend and a certain fault mode is higher than a certain threshold (e.g., 0.7 or higher), then the two sequences are considered similar, and there may be potential fault points. In this way, potential faults can be identified and prevented in advance, improving the stability and availability of the system.

[0102] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A knowledge-based microservice fault detection method, characterized by: Includes the following steps: S1. Collect log data and performance monitoring data from microservices, and preprocess the collected data; S2. Detect anomalies in the data using machine learning models; S3. Based on the results of anomaly detection, search for matching known issues from the knowledge base, provide solutions, and automatically perform repair operations for common faults; S4. Based on the data collected in S1 and the known fault modes in the knowledge base, construct a fault prediction model to identify potential fault points.

2. The knowledge-based microservice fault detection method according to claim 1, characterized in that: The specific steps in S1 are as follows: S1.1 Use log management tools to collect log data from each microservice and use monitoring tools to obtain the performance metrics of the microservices; S1.2 Remove irrelevant data, delete and correct outliers and missing values ​​in the data, and standardize the format of logs and performance data; S1.

3. Store the processed data in the data storage system; S1.

4. Associate log data with performance monitoring data and integrate real-time and historical data.

3. The microservice fault detection method based on a knowledge base according to claim 1, characterized in that: The specific steps of S2 are as follows: S2.1 Extract meaningful features from the data in S1.4, select the feature subset most relevant to fault detection, and reduce redundant features; S2.

2. Use the extracted features to construct a training dataset, train the machine learning model, find the optimal weight vector w and bias term b, and an appropriate slack variable ξ. i ; S2.

3. Input the new data sample into the trained machine learning model to perform anomaly detection. Sample x i Satisfy y i (w T φ(x i If b) < 1, then the sample is considered to represent an anomaly. S2.

4. Use cross-validation techniques to evaluate the model's performance, including at least calculating accuracy, recall, and F1 score, and adjusting model parameters to optimize performance.

4. The knowledge-based microservice fault detection method according to claim 3, characterized in that: The feature extraction method in S2.1 is as follows: Among them, X i Y represents the feature value of the i-th sample; i This represents the target value of the i-th sample; and represents the mean of the feature and the target variable, respectively; n represents the total number of samples.

5. The knowledge-based microservice fault detection method according to claim 3, characterized in that: The machine learning model in S2.2 is specifically as follows: subject to y i (w T φ(x i )+b)≥1-ξ i ,ξ i ≥0,i=1,...,n, Where w represents the weight vector, which determines the direction of the decision boundary; b represents the bias term, which determines the displacement of the decision boundary; ξ i y represents the slack variable; C represents the regularization parameter, which controls the degree of penalty for misclassification; i The class label representing the i-th sample; x i The feature vector representing the i-th sample includes log data and performance monitoring data; φ(x i ) represents the feature mapping function; n represents the total number of samples.

6. The microservice fault detection method based on a knowledge base according to claim 1, characterized in that: The specific steps of S3 are as follows: S3.1 Build a knowledge base, which includes descriptions of known failure modes, common failures, possible causes, and corresponding solutions; S3.2 When the machine learning model detects an anomaly, it searches for a matching fault description in the knowledge base based on the anomaly type and context information. S3.3 If there is no directly matching fault description in the knowledge base, the text similarity calculation method can be used to find the closest fault case. S3.4 Based on the matched fault cases, provide one or more solutions. For common faults, there are pre-configured automated repair scripts and programs in the knowledge base, which can be directly executed to solve the problem. S3.5 If automated repair fails, notify manual inspection and repair. After repair is completed, record the abnormal situation and repair procedure into the knowledge base.

7. The knowledge-based microservice fault detection method according to claim 1, characterized in that: The specific steps of S4 are as follows: S4.1 uses real-time and historical data integrated from S1.4, including preprocessed log data and performance monitoring data; S4.

2. Further select those features from the features extracted in S2.1 that are helpful in predicting future performance trends; S4.3 Construct a fault prediction model; S4.4 Introduce seasonal components into the fault prediction model and train the fault prediction model using historical data; S4.5 Apply the fault prediction model to the latest data to predict the performance trend in the future. Use known fault modes in the knowledge base to compare with the predicted trend and use cosine similarity to find the matching fault modes. S4.

6. Based on the matched failure mode, automatically execute the corresponding preventive measures, which include at least adjusting resource allocation, optimizing configuration parameters, and deploying backup resources in advance. S4.7 If a new failure mode is predicted but not covered by the knowledge base, then manual detection and prevention should be initiated, and the knowledge base should be updated to include the new failure mode and its solution.

8. The knowledge-based microservice fault detection method according to claim 7, characterized in that: The fault prediction model in S4.3 is specifically as follows: in, y represents the predicted value at time t; t-i Represents the residual at time ti; e t-j φ represents the residual at time tj. i and θ j represents the parameters of the autoregression and moving average, respectively; c represents the constant term; p and q represent the number of terms in the autoregression and moving average, respectively.

9. The knowledge-based microservice fault detection method according to claim 7, characterized in that: The algorithm formula in S4.4 after incorporating seasonal components into the fault prediction model is as follows: Among them, Y t-k E represents the seasonal autoregressive term at time tk; t-l Φ represents the seasonal moving average term at time tl, where l is a multiple of the seasonal period; k Represents the seasonal autoregressive parameter; Θ L P represents the seasonal moving average parameter; Q represents the number of seasonal autoregressive terms; s represents the seasonal moving average term length.

10. The microservice fault detection method based on a knowledge base according to claim 1, characterized in that: The cosine similarity in S4.5 is specifically as follows: Where cos(θ) represents the cosine value; n represents the length of the sequence; y represents the value of the predicted trend sequence at the i-th time point; fault，i This represents the value of the fault mode sequence in the knowledge base at the i-th time point.

Images