Fault detection method and device for micro-service instance

Abstract

The embodiment of the invention provides a micro-service instance fault detection method and device.The method comprises the steps that multi-modal data of a current time window of a micro-service instance are obtained, and the multi-modal data comprise index data, log data and call chain data; determining an observation data graph according to the multi-modal data; determining vector data according to the observation data graph and a pre-trained graph neural network model; and determining a fault detection result by using a pre-trained large language model according to the vector data. According to the method and the device, the technical problems that single-mode data can cause false report and missing report of faults and detection and diagnosis of the faults are not interpretable are solved, and the effects of improving the quality of fault detection and helping operation and maintenance personnel to understand the problems are achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a method and device for detecting faults in a microservice instance. Background Art

[0002] As Internet applications become larger and more complex, engineers break down complete applications into many independent service instances to achieve flexible modification and deployment of applications, forming the so-called microservice system architecture. In a microservice system, service instances are deployed on many physical hosts using a container environment, and instances are connected to each other through service calls, forming a complex and dynamic interconnected system.

[0003] Timely diagnosis, troubleshooting, and repair of system failures are the core tasks of Internet application operation and maintenance, which can reduce the economic losses caused by system failures. With the increase in business volume and the diversification of business content, the instance scale of microservice systems has expanded accordingly, and the instance interconnection structure has become more complex. This requires the development of automated system fault detection and diagnosis methods and technologies to assist operation and maintenance personnel in timely detecting faults, quickly determining the location and cause of faults, and realizing intelligent operation and maintenance. The detection and diagnosis of system faults depends on real-time observation of microservice systems. The observability of the system is generally reflected in multimodal system observation data, including indicators, logs, and call chain data.

[0004] However, in the existing technology, there are many challenges in efficiently and accurately detecting and diagnosing faults in microservice systems: microservice system faults are diverse, with different causes and manifestations. Relying on data from a single modality will lead to false positives and false negatives of faults, and fault detection and diagnosis are not explainable. Summary of the invention

[0005] The embodiments of the present application provide a method and device for fault detection of a microservice instance, so as to at least solve the problem in the related art that reliance on data of a single modality may lead to false positives and false negatives of faults, and at the same time, the detection and diagnosis of faults are not explainable.

[0006] According to an embodiment of the present application, a fault detection method for a microservice instance is provided, which is applied to a microservice system, wherein the microservice system includes multiple microservice instances, and the method includes:

[0007] Obtain multimodal data of the current time window of the microservice instance, where the multimodal data includes indicator data, log data, and call chain data; determine the observation data graph based on the multimodal data; determine the vector data based on the observation data graph and the pre-trained graph neural network model; determine the fault detection result based on the vector data using the pre-trained large language model.

[0008] In an exemplary embodiment, determining an observation data graph based on multimodal data includes: determining fused text sequence data based on the multimodal data, wherein the text sequence data is used to characterize the state evolution of the microservice; and determining an observation data graph based on the fused text sequence data.

[0009] In an exemplary embodiment, fused text sequence data is determined based on multimodal data, including: determining an indicator data event sequence based on indicator data; determining a log data event sequence based on log data; determining a call chain data event sequence based on call chain data; and utilizing timestamps to fuse the indicator data event sequence, the log data event sequence, and the call chain data event sequence into fused text sequence data.

[0010] In an exemplary embodiment, an observation data graph is determined based on fused text sequence data, including: constructing a call relationship graph of multiple microservice instances based on call chain data; and determining an observation data graph based on the call relationship graph and the fused text sequence.

[0011] In an exemplary embodiment, the vector data includes a multimodal data vector and a similarity vector. The vector data is determined based on an observed data graph and a pre-trained graph neural network model, including: determining the multimodal data vector based on the observed data graph and the pre-trained graph neural network model; determining the similarity vector based on the multimodal data vector and a pre-built vector database.

[0012] In an exemplary embodiment, the training of the graph neural network model includes: obtaining a sample atlas, wherein the sample atlas is constructed based on historical multimodal data; dividing the sample atlas into a training set and a verification set; and training and verifying a preset initial model based on preset model hyperparameters, the training set, and the verification set to obtain a graph neural network model.

[0013] In an exemplary embodiment, the construction of a vector database includes: determining multiple feature vectors corresponding to multiple microservices based on a historical observation data graph; normalizing the multiple feature vectors to obtain multiple normalized vectors; using a preset indexing algorithm to construct indexes for the multiple normalized vectors to obtain multiple indexed vectors; and constructing a vector database based on the multiple indexed vectors.

[0014] In an exemplary embodiment, a fault detection result is determined based on vector data using a pre-trained large language model, including: determining a prompt word template based on similarity vectors, multimodal data vectors and preset system data; and determining the fault detection result based on the prompt word template, multimodal data vectors and similarity vectors using a pre-trained large language model.

[0015] In an exemplary embodiment, the preset system data includes at least one of the following: preset system configuration, system service information, fault manual and historical fault troubleshooting records.

[0016] In an exemplary embodiment, after determining the fault detection result based on the vector data using a pre-trained large language model, the fault detection method of the microservice instance further includes: in response to the fault detection result indicating that there is no abnormality in the multimodal data corresponding to the current time window, obtaining the multimodal data of the next time window.

[0017] In an exemplary embodiment, the fault detection method of the microservice instance further includes: in response to the fault detection result indicating that the multimodal data corresponding to the current time window is abnormal, outputting abnormal prompt information.

[0018] According to another embodiment of the present application, a fault detection device for a microservice instance is provided, which is applied to a microservice system, wherein the microservice system includes multiple microservice instances, and the device includes:

[0019] An acquisition module is used to acquire multimodal data of a microservice instance, wherein the multimodal data includes indicator data, log data, and call chain data; a first determination module is used to determine an observation data graph based on the multimodal data; a second determination module is used to determine vector data based on the observation data graph and a pre-trained graph neural network model; and an output module is used to determine a fault detection result based on the vector data using a pre-trained large language model.

[0020] In an exemplary embodiment, the first determination module is also used to: determine fused text sequence data based on the multimodal data, wherein the text sequence data is used to characterize the state evolution of the microservice; and determine an observation data graph based on the fused text sequence data.

[0021] In an exemplary embodiment, the first determination module is also used to: determine the indicator data event sequence based on the indicator data; determine the log data event sequence based on the log data; determine the call chain data event sequence based on the call chain data; and use timestamps to fuse the indicator data event sequence, log data event sequence, and call chain data event sequence into fused text sequence data.

[0022] In an exemplary embodiment, the first determination module is further used to: construct a call relationship graph of multiple microservice instances based on the call chain data; and determine an observation data graph based on the call relationship graph and the fused text sequence.

[0023] In an exemplary embodiment, the vector data includes a multimodal data vector and a similarity vector, and the second determination module 203 is further used to: determine the multimodal data vector based on the observed data graph and the pre-trained graph neural network model; determine the similarity vector based on the multimodal data vector and the pre-built vector database.

[0024] In an exemplary embodiment, the second determination module is also used to: obtain a sample atlas, wherein the sample atlas is constructed based on historical multimodal data; divide the sample atlas into a training set and a verification set; train and verify the preset initial model according to the preset model hyperparameters, training set and verification set to obtain a graph neural network model.

[0025] In an exemplary embodiment, the second determination module is also used to: determine multiple feature vectors corresponding to multiple microservices based on a historical observation data graph; normalize the multiple feature vectors to obtain multiple normalized vectors; use a preset indexing algorithm to construct indexes for the multiple normalized vectors to obtain multiple indexed vectors; and construct a vector database based on the multiple indexed vectors.

[0026] In an exemplary embodiment, the output module is also used to: determine a prompt word template based on the similarity vector, the multimodal data vector and the preset system data; and determine the fault detection result using a pre-trained large language model based on the prompt word template, the multimodal data vector and the similarity vector.

[0027] In an exemplary embodiment, the preset system data in the output module includes at least one of the following: preset system configuration, system service information, fault manual and historical fault troubleshooting records.

[0028] In an exemplary embodiment, the output module is further used to: in response to the fault detection result indicating that the multimodal data corresponding to the current time window has no abnormality, obtain the multimodal data of the next time window.

[0029] In an exemplary embodiment, the output module is further used to: in response to the fault detection result indicating that the multimodal data corresponding to the current time window is abnormal, output abnormal prompt information.

[0030] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0031] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0032] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0033] Through this application, the multimodal data of the current time window of the microservice instance is obtained, wherein the multimodal data includes indicator data, log data and call chain data; based on the multimodal data, the observation data graph is determined; based on the observation data graph and the pre-trained graph neural network model, the vector data is determined; based on the vector data, the fault detection result is determined using the pre-trained large language model. When performing fault detection, multimodal data is obtained, and the large language model determines the fault detection result based on the vector data corresponding to the multimodal data. Using multimodal data as the basis for detection is more comprehensive and accurate than the inspection results of single-modal data. At the same time, the detection results output by the large language model have the interactivity of natural language. Therefore, it can solve the technical problems that single-modal data will lead to false alarms and omissions of faults and the detection and diagnosis of faults are not explainable, so as to improve the quality of fault detection and help operation and maintenance personnel understand the problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a hardware structure block diagram of a server device of a fault detection method of a microservice instance of an embodiment of the present application;

[0035] Figure 2 is a flowchart of a fault detection method for a microservice instance according to an embodiment of the present application;

[0036] Figure 3 is a structural block diagram of a fault detection device for a microservice instance according to an embodiment of the present application; DETAILED DESCRIPTION

[0037] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0039] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a method for fault detection of a microservice instance in an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0040] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the fault detection method of the microservice instance in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] In this embodiment, a method for detecting a fault of a microservice instance is provided, which is applied to a microservice system including multiple microservice instances. Figure 2 is a flowchart of a fault detection method for a microservice instance according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0043] Step S102, obtaining multimodal data of the current time window of the microservice instance, wherein the multimodal data includes indicator data, log data, and call chain data.

[0044] Specifically, in step S102, the system collects multimodal data of the microservice instance within a specific time window, including indicator data, log data, and call chain data. Among them, indicator data refers to the performance indicators of the microservice instance during operation, such as CPU usage, memory usage, disk I / O, etc. Log data contains log entries generated during the operation of the system, which record the operating status and events of the system. Call chain data is used to record service call information between microservice instances, including call delay, call frequency, etc.

[0045] It can be understood that multimodal data is used to characterize the operating status of the system in different aspects and is crucial for a comprehensive assessment of the system health status.

[0046] Step S104, determining an observation data graph based on the multimodal data.

[0047] Specifically, after obtaining the multimodal data, an observation data graph may be constructed, in which nodes have text attributes, and the text attributes are associated with the multimodal data.

[0048] For example, the observed data is expressed as ,in yes The set of merged sequence data corresponding to the multimodal data, express A collection of microservice instances. Represents an edge constructed from call data.

[0049] Step S106, determining vector data according to the observed data graph and the pre-trained graph neural network model;

[0050] Specifically, after the observation data graph is determined to be obtained, in order to facilitate effective retrieval and comparison, the observation data graph is converted into vector data for effective retrieval and comparison.

[0051] Step S108, determining the fault detection result based on the vector data using the pre-trained large language model.

[0052] Specifically, after obtaining the vector data, the vector data is input into the vector database, and the historical data most similar to the current vector data is found through the retrieval mechanism. This helps to learn the fault mode from history, and integrates the retrieved similar historical data, the vector data of the current time window, the system configuration information, business knowledge, etc. into the input prompt words of the large language model. The constructed prompt words are input into the large language model, and the model outputs the fault detection results of the current time window, including whether the fault is detected, the fault location, and the possible causes of the fault. In addition, the model will also provide the basis for its judgment to enhance the interpretability of the fault detection results.

[0053] Obtain multimodal data of the current time window of the microservice instance, where multimodal data includes indicator data, log data, and call chain data; determine the observation data graph based on the multimodal data; determine the vector data based on the observation data graph and the pre-trained graph neural network model; and determine the fault detection result based on the vector data using the pre-trained large language model. When performing fault detection, multimodal data is obtained, and the large language model determines the fault detection result based on the vector data corresponding to the multimodal data. Using multimodal data as the basis for detection is more comprehensive and accurate than the inspection results of single-modal data. At the same time, the detection results output by the large language model have the interactivity of natural language. Therefore, it can solve the technical problems that single-modal data will lead to false positives and omissions of faults and the detection and diagnosis of faults are not explainable, thereby improving the quality of fault detection and helping operation and maintenance personnel understand the problem.

[0054] The execution subject of the above steps may be a server, a terminal, etc., but is not limited thereto.

[0055] In an exemplary embodiment, determining an observation data graph based on multimodal data includes: determining fused text sequence data based on the multimodal data, wherein the text sequence data is used to characterize the state evolution of the microservice; and determining an observation data graph based on the fused text sequence data.

[0056] Specifically, in a microservice system, multimodal data includes indicator data, log data, and call chain data. In order to better characterize the state evolution of the microservice system within a time window, the multimodal data is converted and fused into unified text sequence data. After obtaining the text sequence data, the observation data graph can be directly constructed based on the text sequence data.

[0057] It is understandable that by converting multimodal data into fused text sequence data, the state changes of microservice instances can be more intuitively displayed, facilitating subsequent fault detection and analysis.

[0058] In an exemplary embodiment, fused text sequence data is determined based on multimodal data, including: determining an indicator data event sequence based on indicator data; determining a log data event sequence based on log data; determining a call chain data event sequence based on call chain data; and utilizing timestamps to fuse the indicator data event sequence, the log data event sequence, and the call chain data event sequence into fused text sequence data.

[0059] Specifically, in the process of determining the indicator data event sequence, for the microservice instance Multivariate indicator data , this technology is simple but efficient The technology extracts abnormal events in time series. Specifically, for the first Time series , Represents the length of the time window. The present invention first calculates the mean of the sequence and variance If the window Moments, values ​​of time series or , then it is considered that an abnormal warning event has occurred in the time series and is recorded with a timestamp The event "or" ",in Indicates the index sequence The name of the microservice instance is represented by a natural language string, such as "CPU usage", "Disk IO", etc. Multivariate indicator data Can be processed as natural language sequence data with timestamps , Represents the total number of events extracted within this time window.

[0060] In the process of determining the log data event sequence, we first use the log parsing tool to extract the templates of all historical log data. We count the frequency of each template, discard the log entries corresponding to the 50% most frequently occurring templates, and use the remaining log templates to extract log events. This can avoid data noise caused by redundant logs. Log sequence data within a time window , in is the number of log entries collected in the window. The log parsing tool maps each log entry to its corresponding log template and extracts the corresponding log timestamp. Log sequence data Can be processed as a sequence with timestamps ,in Indicates the number of times extracted in this time window. Note that the length of this time series will be significantly smaller than length, because common logs are discarded.

[0061] In the process of determining the call chain data event sequence, for the microservice instance Call chain sequence data within a time window The present invention also utilizes Specifically, the present invention calculates and counts the average and method of the call delay (end time minus start time), and counts the call delay that exceeds The call delay is regarded as an abnormal call event, the corresponding call data is extracted as an (invoke delay) event, and the timestamp of the event is set to start_time. Call chain sequence data within a time window Can be processed as a sequence with timestamps ,in Indicates the number of extracted call chain times.

[0062] In the fusion process, the three sequence data are sequentially merged into one text sequence data using timestamps to represent the state evolution of the system. For example, suppose the merged sequence data is:

[0063]

[0064] The sequence data is represented in a time window. The following events have been experienced: call delay, indicator Abnormal rise, indicators Abnormal drop occurs and is recorded in the log template The advantage of this representation is that the observation data of the three modalities can be represented by a unified text modality, which is conducive to subsequent embedding and retrieval.

[0065] It can be understood that the timestamp fusion method can ensure the timeliness and relevance of data, and has significant advantages in quickly responding to and handling transient faults.

[0066] In an exemplary embodiment, an observation data graph is determined based on fused text sequence data, including: constructing a call relationship graph of multiple microservice instances based on call chain data; and determining an observation data graph based on the call relationship graph and the fused text sequence.

[0067] Specifically, in the process of determining the observation data graph, we first use the call chain data set of all microservice instances in a single time window , build a call relationship graph between instances ,in express A collection of microservice instances. Indicates the edge constructed by the call data. If and only if the call chain data and Contains examples and Then, combined with the call relationship graph, the system observation data within a time window is represented as a graph with text attributes on the nodes. ,in yes A collection of merged sequence data.

[0068] In an exemplary embodiment, the vector data includes a multimodal data vector and a similarity vector. The vector data is determined based on an observed data graph and a pre-trained graph neural network model, including: determining the multimodal data vector based on the observed data graph and the pre-trained graph neural network model; determining the similarity vector based on the multimodal data vector and a pre-built vector database.

[0069] Specifically, first, the text sequence of each node Input the pre-trained BERT model to generate nodes (service instances) After the text sequence is embedded, we get a set of vectors ,in Represents each event The embedding of . Taking the average of these embedding vectors gives the node The initial characteristics . Input the initial features into the pre-trained graph neural network model to generate node embedding vectors ,in Represents the number of nodes in the time window. Then, based on the multimodal data vector, similar vectors are matched from the pre-built vector database.

[0070] Exemplarily, the encoder is used to obtain a vector representation of the new data , that is, the multimodal data vector. Most similar Vector , return here The original graph data set corresponding to the vector is expressed as , that is, similarity vector, where , .

[0071] In an exemplary embodiment, the training of the graph neural network model includes: obtaining a sample atlas, wherein the sample atlas is constructed based on historical multimodal data; dividing the sample atlas into a training set and a verification set; and training and verifying a preset initial model based on preset model hyperparameters, the training set, and the verification set to obtain a graph neural network model.

[0072] Specifically, by using the collected historical data, the present invention constructs a large number of unlabeled training data sets by dividing the time window, that is, obtaining a graph set representing the multimodal data of the microservice system. According to the GAE model structure designed above, the model training is completed through the following steps: configure the model's hyperparameters, including learning rate and batch training size, divide the graph collection into training set and verification set. Output the graph collection to the GAE model in batches and calculate the loss function ,The model parameters are updated through gradient descent. When the loss value of the validation set stops decreasing, the model training ends to avoid overfitting. Output the encoder of the GAE model.

[0073] It should be noted that the graph neural network model mainly consists of two parts: encoder and decoder. The encoder uses GAT to map node features to latent space and reconstructs the structure and feature mask. The graph attention encoder uses the GAT model to map node features to latent space. Convert to embedded representation GAT uses the adjacency matrix A and the attention weights between nodes Indicates the importance of neighbor nodes. The embedding representation of is calculated as: ,in is the attention weight. The decoder is used to reconstruct the structure of the graph and the mask features of the nodes. First, the adjacency matrix is ​​reconstructed through the structural reconstruction loss function The structure reconstruction loss is defined as: . The node embedding at the encoder input is Randomly mask some features, and then require the decoder to reconstruct the masked features, defined as .in represents the mapping function of the decoder to reconstruct the mask features, is the masked feature index. The GAE model is trained by jointly optimizing the structure reconstruction loss and the feature mask reconstruction loss: .in Weight hyperparameters for the feature mask reconstruction loss.

[0074] In an exemplary embodiment, the construction of a vector database includes: determining multiple feature vectors corresponding to multiple microservices based on a historical observation data graph; normalizing the multiple feature vectors to obtain multiple normalized vectors; using a preset indexing algorithm to construct indexes for the multiple normalized vectors to obtain multiple indexed vectors; and constructing a vector database based on the multiple indexed vectors.

[0075] For example, the construction of the vector database requires obtaining the historical observation data graph. For the historical observation data graph, the trained GAE encoder is used to generate the node embedding vector (feature vector) ,in Represents the number of nodes in the time window. The generated embedding vector is normalized to ensure that the similarity calculation result during retrieval is more accurate. The normalization formula is: . Build index: embedding vector of each time window Pooling, as the only representation vector for this time window , where the Pooling function can be any function with permutation invariance, such as max, min, mean, etc. Then Store in a vector database and accelerate index construction through index structures (such as FAISS or HNSW). This enables efficient approximate nearest neighbor (ANN) search. Retrieval configuration: Set the retrieval parameters of the vector database (such as distance measurement method and retrieval threshold) to facilitate the subsequent rapid recall of similar data for new observations.

[0076] In an exemplary embodiment, a fault detection result is determined based on vector data using a pre-trained large language model, including: determining a prompt word template based on similarity vectors, multimodal data vectors and preset system data; and determining the fault detection result based on the prompt word template, multimodal data vectors and similarity vectors using a pre-trained large language model.

[0077] Specifically, the constructed prompt word template is passed as input to the pre-trained LLM (Large Language Model). The information in the template is intended to guide the model to focus on the data in the current time window, while taking into account historical similar data (similar vectors) and system preset information. LLM performs reasoning based on the input prompt information, analyzes the similarity between current data (multimodal data vectors) and historical data (similar vectors), combines business rules and fault definitions, and outputs a judgment on the current system status, including whether a fault is detected and the possible root cause of the fault.

[0078] It should be noted that the output of LLM not only includes the results of fault detection and diagnosis, but also includes the explanation of the results, that is, how LLM draws the basis for judgment from the data, which provides the interpretability of the fault detection results, facilitates the understanding of the decision logic of LLM and performs subsequent troubleshooting.

[0079] It is understandable that fault detection and diagnosis through LLM can not only respond quickly to system state changes, but also provide accurate fault identification without a large amount of labeled data, while enhancing the credibility and practicality of the results through natural language interpretation. This technical route combines the efficient retrieval of deep learning and the reasoning ability of large language models, providing a new intelligent solution for the operation and maintenance of microservice systems.

[0080] In an exemplary embodiment, the preset system data includes at least one of the following: preset system configuration, system service information, fault manual and historical fault troubleshooting records.

[0081] In an exemplary embodiment, after determining the fault detection result based on the vector data using a pre-trained large language model, the fault detection method of the microservice instance further includes: in response to the fault detection result indicating that there is no abnormality in the multimodal data corresponding to the current time window, obtaining the multimodal data of the next time window.

[0082] Specifically, if the fault detection result shows that the multimodal data corresponding to the current time window is normal, the multimodal data of the next time window is obtained, and the data of the next time window is further fault detected. This continuous detection mechanism can ensure the long-term stable operation of the microservice system and is suitable for microservice architecture scenarios that require continuous monitoring.

[0083] It should be noted that the next time window is a data acquisition window adjacent to the current time window.

[0084] In an exemplary embodiment, the fault detection method of the microservice instance further includes: in response to the fault detection result indicating that the multimodal data corresponding to the current time window is abnormal, outputting abnormal prompt information.

[0085] Exemplarily, if LLM determines that there is a fault, the LLM result is returned to the operation and maintenance personnel through the interactive UI (optional configuration), including information on whether a fault has occurred, possible root causes of the fault, and an explanation of such determination, for reference by the operation and maintenance personnel.

[0086] In an exemplary embodiment, for single server fault management, the call chain mode in the above fault detection method is removed, and a customized autoencoder (instead of GAE) is used to implement the retrieval and recall of historical data. The exemplary steps include:

[0087] 1. Data preprocessing (after adjustment):

[0088] Target data: indicator data and log data of a stand-alone server, excluding call chain data, because there is no call relationship between services in a stand-alone scenario.

[0089] Metrics data processing: Similar to the multi-service scenario, use predefined event templates and anomaly detection algorithms to extract abnormal events from the metrics data.

[0090] Log data processing: Similarly, use log parsing tools (such as Drain) and abnormal event templates to identify and extract abnormal events from log data.

[0091] 2. Construct fused text sequence data:

[0092] Event fusion: The extracted indicator abnormal events and log abnormal events are fused in timestamp order to form fused text sequence data that describes the server status changes.

[0093] Conversion of time series to text sequences: Ensure that all abnormal events are converted into easy-to-handle text form for subsequent embedding and retrieval.

[0094] 3. Training of custom autoencoders (instead of GAE):

[0095] Model selection: Choose an autoencoder model that is more suitable for unimodal or bimodal data, such as LSTM-based Autoencoder or Simple Feedforward Neural Network Autoencoder.

[0096] Data embedding: The fused text sequence data is input into a customized autoencoder and the model is trained for data embedding. The goal is to convert the text sequence into a compact vector representation and retain the key information on the time series.

[0097] Training process: Similar to GAE, the customized autoencoder also needs to be trained through self-supervised learning (such as data reconstruction or feature masking), but the graph structure information between microservices is no longer considered.

[0098] 4. Build vector database:

[0099] Vector storage: Once the autoencoder is trained, the model is used to convert each time window data in the historical dataset into a vector and stored in a vector database.

[0100] Index construction: Similar to the GAE solution, the efficient retrieval mechanism of the vector database (such as FAISS or HNSW) is used to build an index to speed up the retrieval and recall of historical data.

[0101] 5. Fault detection and diagnosis:

[0102] Vector representation of new data: When new time window data needs to be detected and diagnosed, it is converted into a vector representation using a trained customized autoencoder.

[0103] Retrieve similar historical data: Input the vector of new data into the vector database to retrieve the most similar historical data vector.

[0104] Construct prompt words: Similar to the prompt word template in the multi-service scenario, the current data vector and the most similar historical data retrieved, as well as the system configuration information and business rules, are integrated to construct prompt words for LLM.

[0105] Fault detection and explanation: Input prompt words into LLM, and output the fault detection results of the current time window, including fault diagnosis explanation, to help operation and maintenance engineers understand the specific cause and location of the fault.

[0106] 6. Targeted adjustments:

[0107] Targeted data cleaning: Since the amount and type of data on a single server are relatively simple, more sophisticated data cleaning and preprocessing strategies may be required to improve the accuracy and speed of anomaly detection.

[0108] Model performance optimization: Customized autoencoders may need to be adjusted or optimized in terms of model structure to adapt to the characteristics of single-server data, such as smaller input size and simpler sequence dependencies.

[0109] Through the above process, the core technology of the present invention, fault detection and diagnosis based on multimodal RAG and LLM is adjusted to be more suitable for fault management scenarios of single servers, effectively removing the call chain modal data unique to the microservice architecture, and at the same time using more matching encoder technology to achieve efficient historical data retrieval and fault judgment, thereby improving the operation and maintenance efficiency and fault handling capabilities of single-machine servers.

[0110] In an exemplary embodiment, the definition of abnormal events is not only through preset templates, but abnormal events can be automatically generated and discovered, using statistical methods, deep learning models or rule engines to automatically identify abnormal behavior patterns in microservice systems, thereby dynamically generating event templates. This will reduce manual intervention and improve the system's adaptability and intelligent operation and maintenance capabilities.

[0111] In an exemplary embodiment, the extraction of abnormal events does not rely solely on fixed thresholds (such as variance multiples of indicator data). The threshold of abnormality detection can be dynamically adjusted according to the system operation status and historical data statistics to adapt to the characteristics of different microservice instances and time windows, reducing false positives and false negatives.

[0112] Based on the description of the above embodiments, it can be understood that the technical solution provided by the present invention can achieve the effect of helping to improve the intelligent operation and maintenance of microservice systems. Specifically, the present invention designs an embedding, retrieval and recall module for multimodal data, and uses multimodal RAG technology to achieve detection and diagnosis of microservice system faults with the help of LLM, and provides interpretability of the results. On the one hand, the technical solution of the present invention does not rely on the annotation information of historical data, expands the application scope of intelligent operation and maintenance technology, and is suitable for scenarios where data is difficult to annotate. On the other hand, the technical solution of the present invention utilizes multimodal data to more comprehensively characterize the state of the system and improve the accuracy of intelligent operation and maintenance. Finally, the technical solution of the present invention provides a natural interaction function with operation and maintenance engineers, which can explain the basis for judgment and reduce the risk of operation and maintenance.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0114] In this embodiment, a fault detection device for a microservice instance is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0115] Figure 3 is a structural block diagram of a fault detection device for a microservice instance according to an embodiment of the present application, such as Figure 3 As shown, the device includes: an acquisition module 201, used to acquire multimodal data of a microservice instance, wherein the multimodal data includes indicator data, log data, and call chain data; a first determination module 202, used to determine an observation data graph based on the multimodal data; a second determination module 203, used to determine vector data based on the observation data graph and a pre-trained graph neural network model; an output module 204, used to determine a fault detection result based on the vector data using a pre-trained large language model.

[0116] In an exemplary embodiment, the first determination module 202 is further used to: determine fused text sequence data based on the multimodal data, wherein the text sequence data is used to characterize the state evolution of the microservice; and determine an observation data graph based on the fused text sequence data.

[0117] In an exemplary embodiment, the first determination module 202 is also used to: determine the indicator data event sequence based on the indicator data; determine the log data event sequence based on the log data; determine the call chain data event sequence based on the call chain data; and use timestamps to fuse the indicator data event sequence, the log data event sequence, and the call chain data event sequence into fused text sequence data.

[0118] In an exemplary embodiment, the first determination module 202 is further used to: construct a call relationship graph of multiple microservice instances according to the call chain data; and determine an observation data graph according to the call relationship graph and the fused text sequence.

[0119] In an exemplary embodiment, the vector data includes a multimodal data vector and a similarity vector, and the second determination module 203 is further used to: determine the multimodal data vector based on the observed data graph and the pre-trained graph neural network model; determine the similarity vector based on the multimodal data vector and the pre-built vector database.

[0120] In an exemplary embodiment, the second determination module 203 is also used to: obtain a sample atlas, wherein the sample atlas is constructed based on historical multimodal data; divide the sample atlas into a training set and a verification set; train and verify the preset initial model according to the preset model hyperparameters, the training set and the verification set to obtain a graph neural network model.

[0121] In an exemplary embodiment, the second determination module 203 is also used to: determine multiple feature vectors corresponding to multiple microservices based on a historical observation data graph; normalize the multiple feature vectors to obtain multiple normalized vectors; use a preset indexing algorithm to construct indexes for the multiple normalized vectors to obtain multiple indexed vectors; and construct a vector database based on the multiple indexed vectors.

[0122] In an exemplary embodiment, the output module 204 is also used to: determine a prompt word template based on the similarity vector, the multimodal data vector and the preset system data; and determine the fault detection result using a pre-trained large language model based on the prompt word template, the multimodal data vector and the similarity vector.

[0123] In an exemplary embodiment, the preset system data in the output module 204 includes at least one of the following: preset system configuration, system service information, fault manual and historical fault troubleshooting records.

[0124] In an exemplary embodiment, the output module 204 is further used to: in response to the fault detection result indicating that the multimodal data corresponding to the current time window has no abnormality, obtain the multimodal data of the next time window.

[0125] In an exemplary embodiment, the output module 204 is further configured to: in response to the fault detection result indicating that the multimodal data corresponding to the current time window is abnormal, output abnormal prompt information.

[0126] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0127] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0128] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0129] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0130] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0131] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0132] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0133] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0134] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0135] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0136] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting a fault in a microservice instance, applied to a microservice system, wherein: The microservice system includes multiple microservice instances, characterized in that: include: Obtain multimodal data of the current time window of the microservice instance, wherein the multimodal data includes indicator data, log data, and call chain data; Determining an observation data graph based on the multimodal data; Determining vector data according to the observed data graph and the pre-trained graph neural network model; Based on the vector data, a fault detection result is determined using a pre-trained large language model.

2. The method according to claim 1, characterized in that Determining an observation data graph according to the multimodal data includes: Determine fused text sequence data according to the multimodal data, wherein the text sequence data is used to characterize the state evolution of the microservice instance; An observation data graph is determined based on the fused text sequence data.

3. The method according to claim 2, characterized in that Determining fused text sequence data according to the multimodal data includes: Determining an indicator data event sequence according to the indicator data; Determine a log data event sequence based on the log data; Determine a call chain data event sequence according to the call chain data; The indicator data event sequence, the log data event sequence and the call chain data event sequence are fused into the fused text sequence data by using timestamps.

4. The method according to claim 2, characterized in that: Determining the observation data graph according to the fused text sequence data includes: Constructing a call relationship graph of multiple microservice instances according to the call chain data; The observation data graph is determined according to the call relationship graph and the fused text sequence.

5. The method according to claim 1, characterized in that The vector data includes a multimodal data vector and a similarity vector, and the determining of the vector data according to the observed data graph and the pre-trained graph neural network model includes: Determining the multimodal data vector according to the observed data graph and the pre-trained graph neural network model; The similarity vector is determined according to the multimodal data vector and a pre-constructed vector database.

6. The method according to claim 1, characterized in that The training of the graph neural network includes: Acquire a sample atlas, wherein the sample atlas is constructed based on historical multimodal data; Dividing the sample atlas into a training set and a verification set; According to the preset model hyperparameters, the training set and the verification set, the preset initial model is trained and verified to obtain the graph neural network model.

7. The method according to claim 5, characterized in that The construction of the vector database includes: Determine, according to the historical observation data graph, a plurality of feature vectors corresponding to the plurality of microservice instances; Normalizing the multiple feature vectors to obtain multiple normalized vectors; Using a preset indexing algorithm, indexing the multiple normalized vectors to obtain multiple indexed vectors; The vector database is constructed according to the multiple indexed vectors.

8. The method according to claim 5, characterized in that Determining the fault detection result based on the vector data using a pre-trained large language model includes: Determining a prompt word template according to the similarity vector, the multimodal data vector and preset system data; A fault detection result is determined using a pre-trained large language model according to the prompt word template, the multimodal data vector and the similarity vector.

9. The method according to claim 8, characterized in that The preset system data includes at least one of the following: preset system configuration, system service information, fault manual and historical fault troubleshooting records.

10. The method according to claim 1, characterized in that After determining the fault detection result according to the vector data by using the pre-trained large language model, the fault detection method of the microservice instance further includes: In response to the fault detection result indicating that the multimodal data corresponding to the current time window has no abnormality, multimodal data of a next time window is acquired.

11. The method according to claim 10, characterized in that The fault detection method of the microservice instance also includes: In response to the fault detection result indicating that the multimodal data corresponding to the current time window is abnormal, outputting abnormal prompt information.

12. A fault detection device for a microservice instance, characterized in that: include: An acquisition module, used to acquire multimodal data of the microservice instance, wherein the multimodal data includes indicator data, log data and call chain data; A first determination module, configured to determine an observation data graph based on the multimodal data; A second determination module, used to determine vector data according to the observed data graph and the pre-trained graph neural network model; The output module is used to determine the fault detection result based on the vector data using a pre-trained large language model.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 11 when executed by a processor.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

Images