Micro-service fault processing method based on large model

Abstract

The invention relates to the technical field of cloud computing and micro-service architecture, in particular to a micro-service fault processing method based on a large model. The method comprises the following steps: S1, collecting runtime data from a micro-service system, and preprocessing the data; s2, constructing an anomaly detection model, and performing real-time analysis on the collected data to identify abnormal behaviors and potential faults; s3, on the basis of the output of the anomaly detection model, dynamically adjusting the threshold value of anomaly detection according to historical data and the current load condition so as to reduce false alarms and missing alarms; and S4, constructing a fault library, searching a matched known problem from the fault library according to an anomaly detection result, and automatically executing a repair operation for a known fault mode. According to the method, through real-time fault detection and prediction, dynamic threshold adjustment, automatic fault processing and strong learning ability, the stability and the operation and maintenance efficiency of the micro-service system are effectively improved, the operation and maintenance cost is reduced, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and microservice architecture technology, and more specifically, to a microservice fault handling method based on a large model. Background Technology

[0002] The widespread adoption of microservice architecture has brought enterprises high flexibility, scalability, and independent deployment capabilities, but it has also introduced challenges in operational complexity and fault detection. The independence, dynamic configuration, and frequent interactions between microservices make system fault detection and handling exceptionally complex. Traditional fault detection technologies, often based on static rules and thresholds, struggle to adapt to the dynamically changing system states in a microservice environment. This leads to high false positive and false negative rates, severely impacting system stability and user experience. Furthermore, traditional fault detection technologies lack automatic fault repair capabilities, relying entirely on manual repair, which is costly.

[0003] In summary, current microservice fault handling technologies face challenges such as effectively detecting anomalies, adapting to dynamic system changes, and automatically repairing faults. Therefore, developing a large-model-based microservice fault handling method capable of real-time analysis, dynamic threshold adjustment, and automatic repair is of great significance for improving the stability and operational efficiency of microservice systems. Summary of the Invention

[0004] The purpose of this invention is to provide a microservice fault handling method based on a large model, so as to solve the problems faced by the microservice fault handling technology mentioned in the background art, such as how to effectively detect anomalies, adapt to dynamic changes in the system, and automatically repair faults.

[0005] To achieve the above objectives, the present invention aims to provide a microservice fault handling method based on a large model, comprising the following steps:

[0006] S1. Collect runtime data from the microservice system and preprocess the data;

[0007] S2. Construct an anomaly detection model to analyze the collected data in real time in order to identify abnormal behavior and potential faults;

[0008] S3. Based on the output of the anomaly detection model, dynamically adjust the anomaly detection threshold according to historical data and current load conditions to reduce false alarms and false negatives.

[0009] S4. Build a fault database. Based on the results of anomaly detection, search for matching known problems in the fault database. For known fault modes, automatically perform repair operations.

[0010] As a further improvement to this technical solution, the specific steps of S1 are as follows:

[0011] S1.1 Collect log files and performance metrics from various microservices, collect call chain data from the distributed tracing system, and record the complete path of each request and the response time of each service node;

[0012] S1.2 Remove outliers and duplicate data from the collected data, transform unstructured data into structured data, standardize the collected data, and eliminate the influence of units.

[0013] S1.3 Extract real-time key features from the preprocessed data;

[0014] S1.4 Integrate data from different sources into a unified data warehouse.

[0015] As a further improvement to this technical solution, feature extraction in step S1.3 is performed using the standard deviation method, which specifically involves:

[0016]

[0017] Where σ represents the standard deviation; N represents the total number of sample data; x i This represents the value of the i-th data point; This represents the average of all data points, calculated by dividing the sum of all data points by the total number of data points.

[0018] As a further improvement to this technical solution, the specific steps of S2 are as follows:

[0019] S2.1 Construct an anomaly detection model;

[0020] S2.2. Train the anomaly detection model using historical data, optimize the performance of the anomaly detection model by adjusting hyperparameters, and use cross-validation technology to evaluate the generalization ability of the model to ensure that the anomaly detection model performs well on unseen data.

[0021] S2.3 Input the key feature stream extracted in real time in S1.3 into the trained anomaly detection model to detect anomalies in the data and predict potential faults.

[0022] As a further improvement to this technical solution, the anomaly detection model in S2.1 includes an input layer, an LSTM layer, a Dropout layer, an output layer, and a loss function;

[0023] Specifically, the input layer is:

[0024] X = {x1, x2, ..., x} t}

[0025] Where X represents the input data, which is a sequence; xt The input data vector representing time t;

[0026] Specifically, the LSTM layer is as follows:

[0027] i t =σ(W xi x t +W hi h t-1 +b i )

[0028] f t =σ(W xf x t +W hf h t-1 +b f )

[0029] C t =f t ⊙C t-1 +i t ⊙tanh(W xc x t +W hc h t-1 +b c )

[0030] o t =σ(W xo x t +W ho h t-1 +b o )

[0031] h t =o t ⊙tanh(C t )

[0032] Among them, i t f represents the activation value of the input gate; t The activation value representing the forgetting gate; C t Represents cell state; h t Represents the hidden state, which is also the output of the LSTM; σ represents the Sigmoid activation function; W xi W hi W xf W hf W xc W hc W xo W ho These represent the weight matrices of the corresponding gates; b i b f b c b o These represent the bias terms of the corresponding gates;

[0033] Specifically, the Dropout layer is as follows:

[0034]

[0035] Among them, h t This represents the output of the LSTM; p represents the dropout probability. Represents the output of the Dropout layer;

[0036] Specifically, the output layer is:

[0037]

[0038]

[0039] Among them, P anomaly Represents the probability of an anomaly; y prediction W represents the predicted performance metric value. ah W ph These represent the weight matrices of the corresponding output layer; b a b p These represent the bias terms of the corresponding output layer;

[0040] The loss function is specifically as follows:

[0041] L = L anomaly +L prediction

[0042]

[0043] Where N represents the number of samples; L anomaly The binary cross-entropy loss represents anomaly detection; L prediction The mean squared error loss representing fault prediction; Y prediction p represents the actual value of future performance indicators; anomaly y represents the probability of anomalies predicted by the model; prediction This represents the performance metric value predicted by the model.

[0044] As a further improvement to this technical solution, the specific steps of S3 are as follows:

[0045] S3.1 Initialize the threshold. The initial threshold can be determined based on statistical analysis of historical data.

[0046] S3.2 Monitor and record the current system load;

[0047] S3.3. Use an anomaly detection model to predict real-time data and obtain the anomaly probability p. anomaly and predicted performance metrics y prediction ;

[0048] S3.4. Evaluate the difference between the anomaly detection model output and the actual labels, and calculate the false positive rate and false negative rate;

[0049] S3.5. Based on the false alarm rate and false negative rate, a dynamic adjustment model is used to adjust the anomaly detection threshold.

[0050] As a further improvement to this technical solution, the dynamic adjustment model in S3.5 is specifically as follows:

[0051] θ new =θ t-1 ×(1+β×(FPR-FNR)+γ×(L t -L avg ))

[0052] Where, θ t The anomaly detection threshold representing time t; θ t-1 θ represents the anomaly detection threshold of the previous time step; new This represents the new threshold calculated based on current load and historical data; α represents the smoothing factor, used to balance the impact of the old and new thresholds; β represents the trade-off coefficient between false alarm rate and false negative rate; γ represents the impact coefficient of load changes; L avg Represents average load level; FPR represents false alarm rate; FNR represents false negative rate; L t This represents the load level.

[0053] As a further improvement to this technical solution, the specific steps of S4 are as follows:

[0054] S4.1 Build a fault database, which contains known fault modes and their repair scripts and programs;

[0055] S4.2 When the anomaly detection model detects an anomaly, it uses text similarity calculation to find the corresponding fault and its repair script and program in the fault database;

[0056] S4.3 After matching the corresponding fault, automatically execute the repair script and program to solve the problem;

[0057] S4.4 If the repair fails or no corresponding fault is found, then notify the operator to check and repair it manually. After the repair is completed, the abnormal situation and repair procedure will be entered into the fault database.

[0058] As a further improvement to this technical solution, the text similarity calculation method in S4.2 is specifically as follows:

[0059]

[0060] Here, A and B represent two different sets of keywords.

[0061] As a further improvement to this technical solution, when the anomaly detection model in S2 predicts a potential fault, it notifies a human to conduct an inspection and prevention.

[0062] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0063] 1. This method constructs an anomaly detection model that can analyze runtime data of microservice systems in real time, effectively identify abnormal behaviors and potential faults. Utilizing deep learning technology, especially the LSTM layer, the model can capture long-term dependencies in time series, improving the accuracy of fault prediction and thus enabling early warning of faults. By dynamically adjusting the anomaly detection threshold, the threshold setting can be automatically optimized based on the current system load and historical data, effectively reducing false alarms and missed alarms, and significantly improving the accuracy and reliability of fault detection. The dynamic model takes into account the impact of system load and false alarm / missed alarm rates, ensuring the rationality of threshold adjustment.

[0064] 2. The fault database, combined with the detection results of the anomaly detection model, enables automatic repair of known fault modes. Through text similarity calculation, it can quickly match repair scripts and programs in the fault database, realize automated fault repair, reduce manual intervention, and when the anomaly detection model predicts a potential fault, it will promptly notify the operation and maintenance personnel to check and prevent it, improve system operation and maintenance efficiency, and significantly reduce operation and maintenance costs. Attached Figure Description

[0065] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1:

[0068] Please see Figure 1 As shown, this embodiment provides a microservice fault handling method based on a large model, including the following steps:

[0069] 1. A microservice fault handling method based on a large model, characterized by the following steps:

[0070] S1. Collect runtime data from the microservice system and preprocess the data, including logs, monitoring metrics, call chains, etc.

[0071] The specific steps of S1 are as follows:

[0072] S1.1 Collect log files and performance metrics from various microservices, collect call chain data from the distributed tracing system, record the complete path of each request and the response time of each service node. Log files include, but are not limited to, application logs and system logs. Performance metrics of microservices include CPU utilization, memory utilization, disk I / O, network traffic, etc.

[0073] S1.2 Remove outliers and duplicate data from the collected data, transform unstructured data into structured data, standardize the collected data, and eliminate the influence of units.

[0074] S1.3 Extract real-time key features from the preprocessed data. Key features include service request response time, CPU and memory usage, disk I / O and network traffic, call chain response time analysis, error rate and exception logs, inter-service correlation features (frequency of inter-service calls, dependency strength, etc.) and load and request volume, etc.

[0075] S1.4 Integrate data from different sources into a unified data warehouse.

[0076] In S1.3, the standard deviation method is used for feature extraction. The standard deviation method is as follows:

[0077]

[0078] Where σ represents the standard deviation, used to quantify the volatility or dispersion of the data; N represents the total number of sample data; x i This represents the value of the i-th data point; This represents the average of all data points, calculated by dividing the sum of all data points by the total number of data points. The standard deviation method can help identify features that fluctuate significantly in data, which may contain important information.

[0079] S2. Construct an anomaly detection model to analyze the collected data in real time in order to identify abnormal behavior and potential faults.

[0080] The specific steps of S2 are as follows:

[0081] S2.1 Construct an anomaly detection model;

[0082] S2.2. Train the anomaly detection model using historical data, optimize the performance of the anomaly detection model by adjusting hyperparameters, and use cross-validation technology to evaluate the generalization ability of the model to ensure that the anomaly detection model performs well on unseen data.

[0083] S2.3 Input the key feature stream extracted in real time in S1.3 into the trained anomaly detection model to detect anomalies in the data and predict potential faults.

[0084] The anomaly detection model in S2.1 includes an input layer, an LSTM layer, a Dropout layer, an output layer, and a loss function.

[0085] Specifically, the input layer is:

[0086] X = {x1, x2, ..., x} t}

[0087] Where X represents the input data, which is a sequence; x t The input data vector representing time t can contain values ​​of multiple performance metrics;

[0088] Specifically, the LSTM layer is as follows:

[0089] i t =σ(W xi x t +W hi h t-1 +b i )

[0090] f t =σ(W xf x t +W hf h t-1 +b f )

[0091] C t =f t ⊙C t-1 +i t ⊙tanh(W xc x t +W hc h t-1 +b c )

[0092] o T =σ(W xo x t +W ho h t-1 +b o )

[0093] h t =o T ⊙tanh(C t )

[0094] Among them, i t f represents the activation value of the input gate; t The activation value representing the forgetting gate; Ct Represents cell state; h t Represents the hidden state, which is also the output of the LSTM; σ represents the Sigmoid activation function; W xi W hi W xf W hf W xc W hc W xo W ho These represent the weight matrices of the corresponding gates; b i b f b c b o These represent the bias terms of the corresponding gates; the LSTM layer is used to capture long-term dependencies in time series.

[0095] Specifically, the Dropout layer is as follows:

[0096]

[0097] Among them, h t This represents the output of the LSTM; p represents the dropout probability. This represents the output of the Dropout layer; the Dropout layer is used to reduce overfitting.

[0098] Specifically, the output layer is:

[0099]

[0100]

[0101] Where, p anomaly Represents the probability of an anomaly; y prediction W represents the predicted performance metric value. ah W ph These represent the weight matrices of the corresponding output layer; b a b p These represent the bias terms of the corresponding output layer;

[0102] The loss function is specifically as follows:

[0103] L = L anomaly +L prediction

[0104]

[0105] Where N represents the number of samples; L anomaly The binary cross-entropy loss represents anomaly detection; L prediction The mean squared error loss representing fault prediction; Y predictionp represents the actual value of future performance indicators; anomaly y represents the probability of anomalies predicted by the model; prediction The value represents the performance metric predicted by the model; the loss function is used to evaluate the gap between the model's predictions and the actual results, and to guide the direction of model optimization.

[0106] S3. Based on the output of the anomaly detection model, dynamically adjust the anomaly detection threshold according to historical data and current load conditions to reduce false alarms and missed alarms.

[0107] The specific steps for S3 are as follows:

[0108] S3.1 Initialize the threshold. The initial threshold can be determined based on statistical analysis of historical data, such as using the standard deviation method or based on the initial prediction results of the anomaly detection model.

[0109] S3.2 Monitor and record the current system load, such as CPU utilization, memory utilization, etc.

[0110] S3.3. Use an anomaly detection model to predict real-time data and obtain the anomaly probability p. anomaly and predicted performance metrics y prediction ;

[0111] S3.4. Evaluate the difference between the anomaly detection model output and the actual labels, and calculate the false positive rate and false negative rate;

[0112] S3.5. Based on the false alarm rate and false negative rate, a dynamic adjustment model is used to adjust the anomaly detection threshold.

[0113] The dynamic adjustment model in S3.5 is as follows:

[0114] θ new =θ t-1 ×(1+β×(FPR-FNR)+γ×(Lt-L avg ))

[0115] Where, θ t The anomaly detection threshold representing time t; θ t-1 θ represents the anomaly detection threshold of the previous time step; new This represents the new threshold calculated based on current load and historical data; α represents the smoothing factor, used to balance the impact of the old and new thresholds, and is usually set to a small value to keep the threshold change smooth; β represents the trade-off coefficient between false alarm rate and false negative rate; γ represents the impact coefficient of load change; L avg Represents average load level; FPR represents false alarm rate; FNR represents false negative rate; L tThis represents the load level; the formula is used to dynamically adjust the anomaly detection threshold, gradually adjusting the threshold based on the current load and historical data to adapt to changes in the system.

[0116] S4. Build a fault database containing known fault modes and corresponding solutions. Based on the results of anomaly detection, search for matching known problems in the fault database and automatically perform repair operations for known fault modes.

[0117] The specific steps of S4 are as follows:

[0118] S4.1 Build a fault database, which contains known fault modes and their repair scripts and programs;

[0119] S4.2 When the anomaly detection model detects an anomaly, it uses text similarity calculation to find the corresponding fault and its repair script and program in the fault database;

[0120] S4.3 After matching the corresponding fault, automatically execute the repair script and program to solve the problem;

[0121] S4.4 If the repair fails or no corresponding fault is found, then notify the operator to check and repair it manually. After the repair is completed, the abnormal situation and repair procedure will be entered into the fault database.

[0122] The specific method for calculating text similarity in S4.2 is as follows:

[0123]

[0124] Here, A and B represent two different sets of keywords. For example, A could be a set of all keywords extracted from the exception log text, while B could be a set of all keywords extracted from the fault description text.

[0125] When the anomaly detection model in S2 predicts a potential fault, it notifies humans to conduct inspections and preventative measures.

[0126] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A microservice fault handling method based on a large model, characterized by: Includes the following steps: S1. Collect runtime data from the microservice system and preprocess the data; S2. Construct an anomaly detection model to analyze the collected data in real time in order to identify abnormal behavior and potential faults; S3. Based on the output of the anomaly detection model, dynamically adjust the anomaly detection threshold according to historical data and current load conditions to reduce false alarms and false negatives. S4. Build a fault database. Based on the results of anomaly detection, search for matching known problems in the fault database. For known fault modes, automatically perform repair operations.

2. The microservice fault handling method based on a large model according to claim 1, characterized in that: The specific steps of S1 are as follows: S1.1 Collect log files and performance metrics from various microservices, collect call chain data from the distributed tracing system, and record the complete path of each request and the response time of each service node; S1.2 Remove outliers and duplicate data from the collected data, transform unstructured data into structured data, standardize the collected data, and eliminate the influence of units. S1.3 Extract real-time key features from the preprocessed data; S1.4 Integrate data from different sources into a unified data warehouse.

3. The microservice fault handling method based on a large model according to claim 2, characterized in that: In step S1.3, the standard deviation method is used for feature extraction. Specifically, the standard deviation method is as follows: Where σ represents the standard deviation; N represents the total number of sample data; x i This represents the value of the i-th data point; This represents the average of all data points, calculated by dividing the sum of all data points by the total number of data points.

4. The microservice fault handling method based on a large model according to claim 1, characterized in that: The specific steps of S2 are as follows: S2.1 Construct an anomaly detection model; S2.

2. Train the anomaly detection model using historical data, optimize the performance of the anomaly detection model by adjusting hyperparameters, and use cross-validation technology to evaluate the generalization ability of the model to ensure that the anomaly detection model performs well on unseen data. S2.3 Input the key feature stream extracted in real time in S1.3 into the trained anomaly detection model to detect anomalies in the data and predict potential faults.

5. The microservice fault handling method based on a large model according to claim 4, characterized in that: The anomaly detection model in S2.1 includes an input layer, an LSTM layer, a Dropout layer, an output layer, and a loss function. Specifically, the input layer is: X={x1,x2,…,x t } Where X represents the input data, which is a sequence; x t The input data vector representing time t; Specifically, the LSTM layer is as follows: i t =σ(W xi x t +W hi h t-1 +b i ) f t =σ(W xf x t +W hf h t-1 +b f ) C t =f t ⊙C t-1 +i t ⊙tanh(W xc x t +W hc h t-1 +b c ) o t =σ(W xo x t +W ho h t-1 +b o ) h t =o t ⊙tanh(C t ) Among them, i t f represents the activation value of the input gate; t The activation value representing the forgetting gate; C t Represents cell state; h t Represents the hidden state, which is also the output of the LSTM; σ represents the Sigmoid activation function; W xi W hi W xf W hf W xc W hc W xo W ho These represent the weight matrices of the corresponding gates; b i b f b c b o These represent the bias terms of the corresponding gates; Specifically, the Dropout layer is as follows: Among them, h t This represents the output of the LSTM; p represents the dropout probability. Represents the output of the Dropout layer; Specifically, the output layer is as follows: Where, p anomaly Represents the probability of an anomaly; y prediction W represents the predicted performance metric value. ah W ph These represent the weight matrices of the corresponding output layer; b a b p These represent the bias terms of the corresponding output layer; The loss function is specifically as follows: L=L anomaly +L prediction Where N represents the number of samples; L anomaly The binary cross-entropy loss represents anomaly detection; L prediction The mean squared error loss representing fault prediction; Y prediction p represents the actual value of future performance indicators; anomaly y represents the probability of anomalies predicted by the model; prediction This represents the performance metric value predicted by the model.

6. The microservice fault handling method based on a large model according to claim 1, characterized in that: The specific steps of S3 are as follows: S3.1 Initialize the threshold. The initial threshold can be determined based on statistical analysis of historical data. S3.2 Monitor and record the current system load; S3.

3. Use an anomaly detection model to predict real-time data and obtain the anomaly probability p. anomaly and predicted performance metrics y prediction ; S3.

4. Evaluate the difference between the anomaly detection model output and the actual labels, and calculate the false positive rate and false negative rate; S3.

5. Based on the false alarm rate and false negative rate, a dynamic adjustment model is used to adjust the anomaly detection threshold.

7. The microservice fault handling method based on a large model according to claim 6, characterized in that: The dynamic adjustment model in S3.5 is specifically as follows: i new =θ t-1 ×(1+β×(FPR-FNR)+γ×(L t -L avg )) Where, θ t The anomaly detection threshold representing time t; θ t-1 θ represents the anomaly detection threshold of the previous time step; new This represents the new threshold calculated based on current load and historical data; α represents the smoothing factor, used to balance the impact of the old and new thresholds; β represents the trade-off coefficient between false alarm rate and false negative rate; γ represents the impact coefficient of load changes; L avg Represents average load level; FPR represents false alarm rate; FNR represents false negative rate; L t This represents the load level.

8. The microservice fault handling method based on a large model according to claim 1, characterized in that: The specific steps of S4 are as follows: S4.1 Build a fault database, which contains known fault modes and their repair scripts and programs; S4.2 When the anomaly detection model detects an anomaly, it uses text similarity calculation to find the corresponding fault and its repair script and program in the fault database; S4.3 After matching the corresponding fault, automatically execute the repair script and program to solve the problem; S4.4 If the repair fails or no corresponding fault is found, then notify the operator to check and repair it manually. After the repair is completed, the abnormal situation and repair procedure will be entered into the fault database.

9. The microservice fault handling method based on a large model according to claim 8, characterized in that: The text similarity calculation method in S4.2 is as follows: Here, A and B represent two different sets of keywords.

10. The microservice fault handling method based on a large model according to claim 1, characterized in that: When the anomaly detection model in S2 predicts a potential fault, it notifies a human to conduct an inspection and preventative measure.

Images