Unsupervised micro-service system fault positioning method based on multi-modal data

Abstract

The invention relates to an unsupervised micro-service system fault positioning method based on multi-modal data, and belongs to the field of service calculation. The method comprises the following steps: S1, constructing an unsupervised micro-service system fault positioning framework model, wherein the model is divided into an offline training stage and an online fault positioning stage; s2, in an off-line stage, micro-service system data and the like are collected in a fault-free stage for preprocessing, a model is pre-trained, and a self-encoder model containing self-attention is obtained; s3, in an online fault positioning stage, acquiring data in a t time window before an exception occurs, inputting the data into the model for reasoning, performing data layering by comparing differences between output and input of the model, and separating out suspected fault data; and S4, in the layered fault data, constructing a weighted state transition probability matrix to carry out final fault positioning. Based on the multi-modal data and the data layering thought, the reliability and the positioning accuracy can be ensured to be kept when the state transition probability matrix is constructed.

Description

Technical Field

[0001] The present invention belongs to the field of service computing and relates to an unsupervised microservice system fault location method based on multimodal data. Background Art

[0002] The microservices architecture is a software development and deployment model that divides complex applications into several small, autonomous service units that collaborate and interact with each other through lightweight communication mechanisms. In a microservices architecture, each service unit has its own business logic and can be independently deployed, scaled, and maintained. Service units communicate through methods such as API calls and messaging to achieve more efficient decoupling and flexibility. Each service unit can be packaged as a container image and managed and scheduled using container orchestration tools. This simplifies deployment and scaling and improves system scalability and resilience. The main goals of the microservices architecture are to improve application maintainability, scalability, and reliability.

[0003] Microservices, with their advantages of flexible scalability and rapid delivery, have become a popular software architecture in the modern IT industry. With the development of distributed systems and cloud computing, microservices have become a widely adopted software architectural style. While its implementation can improve software scalability and flexibility, microservices architectures also bring challenges such as increased complexity in dependencies between multiple services and network latency.

[0004] In a microservices architecture, fault location is a crucial issue. Because microservices typically consist of many services, faults can affect multiple services. When a fault occurs in a single microservice instance, it propagates upstream, impacting the normal operation of several other microservices. The complex interoperability between microservices in a microservices system makes it difficult to accurately locate the faulty microservice. When a fault occurs, failing to quickly and accurately determine the root cause can lead to slower response times, a degraded user experience, and ultimately disrupt the normal operation of the business. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a microservice system fault location method based on multimodal data and unsupervised learning, which can achieve rapid location by effectively fusing multimodal data in the microservice system and applying autoencoders to stratify fault data and normal data, and accurately locate faults by comprehensively calculating the fault transfer probability between nodes.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A microservice system fault location method based on multimodal data and unsupervised learning, the method comprising the following steps:

[0008] S1: Build an unsupervised microservice system fault location framework model. The model includes modules such as data preprocessing module, anomaly detector module, and autoencoder with self-attention mechanism. The model is divided into offline training phase and online fault location phase.

[0009] S2: In the offline phase, the microservice system logs and measurement data collected during the fault-free phase are used for preprocessing, and the model of step S2 is pre-trained to obtain the autoencoder model parameters including self-attention.

[0010] S3: During the online fault location phase, when the anomaly detector module detects an anomaly in the microservice system, it triggers the fault location process. It obtains log data, metric data, and trace data within the time window t before the anomaly occurs. Following the data preprocessing method in step S2, it obtains a predicted data sample containing log features and metric features. This data sample is input into the autoencoder in step S1 for inference. By comparing the model output with the original sample, the data is stratified to separate the suspected fault data.

[0011] S4: In the layered fault data, the autoencoder model is used to obtain the feature vector of each span, and the microservice topology is extracted from the fault trace data. Based on this, the weighted state transition probability matrix is ​​calculated, and the final fault location is performed using the random walk algorithm.

[0012] Furthermore, the step S1 specifically includes the following steps:

[0013] S11: In the data preprocessing module, feature engineering is performed on the log data and metric data in the original dataset to obtain log features and metric features, which are then combined to form a complete training sample.

[0014] S12: In the anomaly detector module, the anomaly detector maintains the measurement data within the t time window in real time, analyzes and processes the measurement data within the time window, and performs SLO violation detection. If it is determined that the system is abnormal, it enters the fault location stage.

[0015] S13: In the autoencoder module of the self-attention mechanism, the input of the model is a feature vector formed by the log feature and metric feature of each span. The output has the same dimension as the input vector. The goal of the model is to restore the input sample as much as possible.

[0016] Furthermore, step S2 specifically includes the following steps:

[0017] S21: During data preprocessing, the Drain3 algorithm is used to extract log templates from log data during the trouble-free phase. Log features such as log type are collected, and a corresponding log feature vector is constructed for each span. After obtaining the feature vector, the PCA algorithm is used to perform feature dimensionality reduction. For the measurement data during the trouble-free phase, the measurement data within each span's time window is collected and averaged. Each type of measurement data is then standardized on the training set. The measurement features and log features are combined to form a complete training sample.

[0018] S22: The first module of the autoencoder model is a self-attention mechanism module, with a 29-dimensional input vector. This allows the model to adaptively focus on important parts of the input. The second module of the autoencoder model is the autoencoder module, which is used to learn the patterns and data distribution of fault-free data. Both the encoder and decoder are two fully connected layers, using RELU as the activation function. The encoder output vector is 16-dimensional and serves as the feature vector of each encoded sample. The decoder output is 29-dimensional.

[0019] Furthermore, step S3 specifically includes the following steps:

[0020] S31: After the anomaly detector detects that an anomaly has occurred in the microservice system, the log data, measurement data, and trace data of the minute before the anomaly occurs are collected. For each span in each trace, a corresponding test data sample is constructed according to the data preprocessing method in claim 1.

[0021] S32: The obtained test data sample is verified by the autoencoder. If the restoration degree of the autoencoder for the test sample is lower than the set threshold, it is determined that the span has an abnormality. If all spans in a trace are normal, it can be determined that the trace is fault-free, and the faulty data and normal data are separated accordingly.

[0022] Furthermore, step S4 specifically includes the following steps:

[0023] S41: In the fault data, all trace data is collected, the microservice topology within the time window is extracted, the spatial relationship between the nodes is obtained, and the interaction frequency between each microservice node and the activity of the node itself are calculated as part of the calculation of the fault transfer probability between nodes.

[0024] S42: For each type of span, the autoencoder restoration degree corresponding to each trace in the fault data is calculated. This restoration degree is used as part of the indicator to measure the fault level. The lower the restoration degree, the greater the deviation between the span data and the fault-free data.

[0025] S43: For each span data sample, the latent space vector of the encoder is obtained through the autoencoder as the feature vector of the data sample. The distribution trend between the feature vectors of each type of span within the fault time window is calculated, and the KL divergence is calculated as part of the basis for measuring the correlation between the two microservice nodes.

[0026] S44: The multi-part data obtained in S31, S32, and S33 are weighted and fused to obtain the final state transition probability matrix, where each element represents the probability of a node failure transitioning to another node failure. A random walk algorithm is then used to obtain the final list of fault root nodes.

[0027] The beneficial effects of the present invention are:

[0028] 1) Make full use of the multimodal data of logs, metrics, and traces generated in the microservice system. Through effective data fusion, we can comprehensively evaluate the status of the microservice system and the transfer probability between computing nodes.

[0029] 2) By establishing an autoencoder based on fault-free historical data in the offline stage, the autoencoder is used to quickly stratify fault data and abnormal data in the online positioning stage, reducing the scope of fault positioning, and quickly constructing the state transfer matrix and iterating the random walk algorithm.

[0030] 3) Comprehensively calculate the failure transfer probability from multiple perspectives, taking into account the topology structure, the interaction relationship between nodes, the status of each node itself, etc., to more comprehensively evaluate the status of the entire system and calculate the transfer probability more accurately.

[0031] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0033] Figure 1 Flowchart for the construction of the offline autoencoder

[0034] Figure 2 Flowchart for online fault location

[0035] Figure 3 The detailed structure and principle diagram of the autoencoder with self-attention mechanism. DETAILED DESCRIPTION

[0036] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0037] Figure 1 This is the flowchart for building the autoencoder in the offline stage. Figure 2 This is the operation flow chart of online fault location. Figure 3 The detailed structure and principle diagram of the autoencoder with self-attention mechanism.

[0038] The present invention provides a method for locating faults in an unsupervised microservice system, which mainly includes the following steps:

[0039] Step 1: Collect log and metric data from the microservice system during the trouble-free phase. Preprocess and feature engineer the raw data to form a training set for the offline phase.

[0040] Step 2: Using a training set preprocessed with historical data from the fault-free phase, build and train an autoencoder model with a self-attention mechanism, using data from each span (an operation on a service in a trace) as a sample.

[0041] Step 3: In the online positioning phase, when the anomaly detector detects an anomaly in the microservice system, it triggers a specific positioning process, collects historical data from one minute before the anomaly occurs, including log data, measurement data, and trace data within this time window, and processes the collected raw data to form a test sample in accordance with the data preprocessing method in claim 1. The autoencoder pre-trained in the offline phase is used for inference, and data is stratified by comparing the differences between the model output and the original sample to separate the fault data.

[0042] Step 4: Use the autoencoder model to obtain the feature vector of each span in the layered fault data, and extract the microservice topology from the fault trace data. Based on this, calculate the weighted state transition probability matrix and perform the final fault location using the random walk algorithm.

[0043] The steps for extracting sample data in step 1 of the offline phase are as follows:

[0044] (1) In the fault-free training data, taking each span as the benchmark, first extract the start time and end time of the span, and then filter out the log data belonging to the span and the measurement data of the node within the time window.

[0045] (2) For log data, first apply the Drain3 algorithm to all logs in the entire training set to obtain the log template in the entire training set. For each Span, all logs belonging to this Sapn construct the following feature vector L i =(c1,c2,c3,t1,t2,t3...t n ), where c1 indicates whether to include info type logs, c2 indicates whether to include debug type logs, and c3 indicates whether to include error type logs. i Indicates the number of the i-th log template in the span. Due to the large number of log templates, the log feature L in the training set will be a sparse matrix. Therefore, the PCA algorithm is used for dimensionality reduction. The specific calculation process is as follows:

[0046] Calculate the mean of each column of matrix L Decenter the original matrix according to the mean of each column. Calculate the covariance matrix of the decentralized matrix Perform matrix eigendecomposition on the covariance matrix, according to Cv k =λ k v k Find the eigenvalue λ k and the eigenvector v k , arrange the eigenvectors into a matrix in descending order from left to right according to the corresponding eigenvalues, and take the first k columns to form the matrix W nxk ,pass Get the log features after dimensionality reduction.

[0047] (3) For the measurement data, extract all the measurement data in the service node time window corresponding to the Span. Since the measurement data is time series data, the mean is taken for each measurement data. Finally, the following feature vector is formed for the measurement data: M i =(m1,m2,m3...m n ), where m i represents the mean of the i-th metric. This feature vector primarily includes data such as CPU usage, disk usage, and network traffic transmission. After obtaining the metric features of all spans, the metric data of the entire training set is standardized to eliminate the impact of dimensional differences between different types of metric data on the model.

[0048] In step 2, we use the constructed training set to complete the model of the autoencoder with the self-attention mechanism, as follows:

[0049] (1) Integrate log data features and measurement data features to form the final training sample X i =(L i ,Mi ), where L i Indicates the log feature corresponding to the i-th Span, M i Represents the metric feature corresponding to the i-th Span. Figure 3 As shown in the figure, the sorted training set is obtained and sent to the autoencoder model for training, using the mean square error as the loss function.

[0050] (2) The first layer uses a self-attention layer to better integrate log data and measurement data. The self-attention mechanism is calculated as follows:

[0051] Q=XW Q

[0052] K=XW K

[0053] V=XW V

[0054]

[0055] Where X is the training sample, d k is the dimension size of K.

[0056] (3) The encoder module follows the self-attention layer, and the specific calculation is as follows:

[0057] Z = RELU(RELU(X'w1+b1)w2+b2)

[0058] Where Z is the feature vector obtained from the latent space after the encoder, which is used to calculate the subsequent state transition probability. The specific calculation of the decoder part is as follows:

[0059] y'=RELU(Zw3+b3)w4+b4

[0060] Where y' is the final output restoration vector of the decoder, and the loss function uses MSE, which is calculated as follows:

[0061]

[0062] In step 3, the trained autoencoder is used to separate the fault data from the normal data, as follows:

[0063] (1) The anomaly detector maintains the metric data within the time window t in real time and checks for SLO violations by using the k-σ rule. If the success rate is less than μ-kσ or the P90 latency is greater than μ+kσ (k=3 by default), the anomaly detector declares the current time window anomaly.

[0064] (2) Collect system data in the one-minute window before the exception occurs, construct test samples for all spans, and form a test data set X t , classify the Span samples in units of Trace, and form a sub-dataset X for each Trace trace i All Span samples in are input into the autoencoder model to obtain the decoding result of the decoder in m is the number of spans in the i-th trace. The decoding result of the k-th Span in the i-th trace.

[0065] (3) By calculating the mean square error between the decoding result and the input sample, it is determined whether the current span is faulty data. This is achieved by comparing the fault degree of the decoder to the input sample. The specific calculation is as follows:

[0066]

[0067] If the calculated failure degree is greater than the threshold, then it can be determined that the span has failed. Then all spans in the trace where the span is located should be transferred to the failure sub-dataset X. E Among them

[0068] In step 4, the autoencoder model is used to obtain the feature vector of each span, and the microservice topology is extracted from the fault trace data. Based on this, the weighted state transition probability matrix is ​​calculated, and the final fault location is performed using the random walk algorithm. The specific calculation process is as follows:

[0069] (1) For sub-dataset X E , extract the topological structure of the data set, the topological structure is expressed as the degree matrix M T Indicates that m i,j Indicates the number of times node i calls node j. Get M T Afterwards, the partial transition probability matrix P obtained from the interaction information between the topology structure and microservices is calculated T , the specific calculation is as follows:

[0070]

[0071] (2) For each microservice node S i , in the fault dataset X E There are several Span data X Si =(X1,X2,...,X m ), for Si , calculate X Si The fault degree of each sample in the autoencoder is used to obtain the node’s own fault degree vector V i =(D1,D2,...D m ), normalize the vector to get VE i , and calculate the mean Mean(V i ), construct the matrix P D , as follows:

[0072]

[0073] (3) For any two VEs i , VE j , taking these two vectors as the probability density distribution of the fault degree of the two microservice nodes, and using KL divergence to calculate the similarity of the two distributions. The higher the similarity of the two distributions, the stronger the correlation between the fault propagation between the two nodes. The specific calculation is as follows:

[0074]

[0075] According to this, we can get P K , as follows:

[0076]

[0077] (4) Through P T , P D , P K The final state transition probability matrix P is obtained by weighted fusion. The specific calculation is as follows:

[0078] P=α*P T +β*P D +(1-α-β)*P K ,where α, β<1 and α+β≤1.

[0079] (5) After obtaining the state transition probability matrix of the microservice system, the back propagation algorithm is used to further locate the fine-grained root node. The specific calculation process is as follows:

[0080] Initialize SV=[1,...,1], where SV i represents the probability that node i is the root fault node, Set a threshold ξ, if Sum(SV'-SV)≥ξ, then set SV=SV', repeat Until convergence, the index subscript with the largest value in the SV vector corresponds to the fault root node with the highest probability, and the fault root node is located.

Claims

1. An unsupervised microservice system fault location method based on multimodal data, characterized in that: The method comprises the following steps: S1: Build an unsupervised microservice system fault location framework model, which includes data preprocessing module, anomaly detector module, self-attention mechanism autoencoder and other modules. The model is divided into offline training stage and online fault location stage; S2: In the offline phase, the microservice system logs and measurement data collected in the fault-free phase are used for preprocessing, and the model of step S2 is pre-trained to obtain the autoencoder model parameters including self-attention; S3: In the online fault location phase, when the anomaly detector module detects an anomaly in the microservice system, the fault location process is triggered, and the log data, measurement data, and trace data in the time window t before the anomaly occurs are obtained. According to the data preprocessing method of step S2, the predicted data samples containing log features and measurement features are obtained, and input into the autoencoder of step S1 for inference. By comparing the difference between the output of the model and the original sample, the data is stratified to separate the suspected fault data; S4: Using the layered suspected fault data, the autoencoder model is used to obtain the feature vector of each span, and the microservice topology structure is extracted from the fault trace data. Based on this, the weighted state transition probability matrix is ​​calculated, and the final fault location is performed through the random walk algorithm.

2. The training set for constructing an offline phase according to claim 1, characterized in that: In step S1, it specifically includes: S11: In the data preprocessing module, feature engineering is performed on the log data and metric data in the original data set to obtain log features and metric features, and these two parts are combined to form a complete training sample. S12: In the anomaly detector module, the anomaly detector maintains the measurement data within the t time window in real time, analyzes and processes the measurement data within the time window, and performs SLO violation detection. If it is determined that the system is abnormal, it enters the fault location stage. S13: In the autoencoder module of the self-attention mechanism, the input of the model is a feature vector formed by the log features and metric features of each span. The output has the same dimension as the input vector. The goal of the model is to restore the input sample as much as possible.

3. The method of constructing and training an autoencoder model including a self-attention mechanism according to claim 2, wherein: In step S2, it specifically includes: S21: In data preprocessing, for the log data in the fault-free phase, the Drain3 algorithm is used to extract the log template, and the log features such as log type are collected to construct the corresponding log feature vector for each span. After obtaining the feature vector, the PCA algorithm is used to reduce the feature dimension. For the measurement data in the fault-free phase, for each span, the measurement data within the span time window is collected and the average is taken, and then each type of measurement data is standardized on the training set. The measurement features and log features are combined to form a complete training sample. S22: The first module of the autoencoder model is the self-attention mechanism module, and the input vector dimension is 29 dimensions. This allows the model to adaptively focus on important input parts. The second module of the autoencoder model is the autoencoder module, which is used to learn the pattern and data distribution of fault-free data. Both the encoder and decoder parts are two layers of fully connected layers, using RELU as the activation function. The output vector of the encoder is 16 dimensions, and it is used as the feature vector of each sample after encoding. The output of the decoder is 29 dimensions.

4. The method for separating normal data and fault data using an autoencoder trained in an offline stage according to claim 3 is characterized in that: In step S3, it specifically includes: S31: After the anomaly detector detects that an anomaly has occurred in the microservice system, the log data, measurement data, and trace data of the minute before the anomaly occurs are collected, and for each Span in each trace, a corresponding test data sample is constructed according to the data preprocessing method in claim 1. S32: The obtained test data sample is verified by the autoencoder. If the restoration degree of the autoencoder for the test sample is lower than the set threshold, it is determined that the span is abnormal; if all spans in a trace are normal, it can be determined that the trace is fault-free, and the faulty data and normal data are separated accordingly.

5. The method for performing random walk based on calculating weighted state transition probability matrix based on fusion multimodal data according to claim 4, characterized in that: In step S4, it specifically includes: S41: In the fault data, all trace data is collected, the microservice topology within the time window is extracted, the spatial relationship between the nodes is obtained, and the interaction frequency between each microservice node and the activity of the node itself are calculated as part of the probability of fault transfer between the nodes. S42: For each type of span, the autoencoder restoration degree corresponding to each trace is calculated in the fault data. The restoration degree is used as a partial indicator to measure the fault level. The lower the restoration degree, the greater the deviation between the span data and the fault-free data. S43: For each Span data sample, the latent space vector of the encoder is obtained through the autoencoder as the feature vector of the data sample. The distribution trend between the feature vectors of each type of Span within the fault time window is calculated, and the KL divergence is calculated as part of the basis for measuring the correlation between two microservice nodes. S44: The multi-part data obtained by S31, S32, and S33 are weightedly fused to obtain the final state transition probability matrix, where each element represents the probability of a node failure transferring to another node failure. The final list of fault root nodes is then obtained through the random walk algorithm.

Images