Fault detection method based on large model

Abstract

The invention relates to a fault detection method based on a large model, and belongs to the field of big data. According to the method, an improved model is adopted for fault detection, the improved model comprises an encoder-dyLLM, a sound embedding module and a word embedding module, and the improved model comprises the following processes that text and sound data are divided into n segments according to timestamps, in kth reasoning, a kth text data segment generates a text vector through the word embedding module, and the text vector is used for generating a text vector; the kth time of sound data generates a sound vector through a sound embedding module, the jth time of encoder-dyLLM reasoning outputs a cache vector memj, the text vector, the sound vector and the cache vector are subjected to concat and sent to encoder-dyLLM reasoning, and kth time of improved model reasoning is completed; the classification result output by the encoder-dyllama comprises a normal result, an IO (Input / Output) fault and a system fault. According to the method, the decoder module is transformed, the global cache is introduced, and sound and ECC data are introduced, so that fusion of various data is realized, and the robustness of the model to fault detection is improved.

Description

Technical Field

[0001] The present invention belongs to the field of big data, and in particular relates to a fault detection method based on a large model. Background Art

[0002] Big data, cloud computing, and intelligent data mining have become the main directions of the Internet industry. In response to these directions, companies have established server cluster centers, which have a large number of servers and provide services to users for a long time. Once a server cluster fails, it needs to be repaired immediately, otherwise it will cause losses to the company. Faced with such a large amount of data, traditional manual methods to analyze the cause of the failure are no longer sufficient. At present, a variety of automated fault analysis tools have been developed. For example, RNN, LSTM, BERT and other methods, but such methods require data preprocessing according to rules. Rule-based methods need to process data in advance according to certain conditions. This process will lose a lot of information, resulting in weak robustness of fault detection. In the process of writing code, program developers will add log printing to the corresponding position according to the logical relationship. Among these logs, except for a small number of intuitive information such as "FAILD" and "ERROR" that can be used to judge errors, in some cases "FAILD" does not appear as a fault, such as Figure 1 As shown, the semantic relationship can be used to determine whether the log is faulty or normal.

[0003] In view of the shortcomings of traditional methods, the LLM model can well understand text semantics and perform fault detection through logs. At present, many companies mainly use decoder mode LLM to analyze logs, which has achieved good results compared with traditional methods. However, there are still problems to be solved based on mainstream LLM:

[0004] 1) Mainstream LLM adopts the decoder mode mainly because it is oriented to dialogue tasks. In dialogue, it is impossible to obtain all the context at once, and it is necessary to infer the following context based on the previous context. Fault detection based on logs is a text understanding task, which is based on context. In this case, it is necessary to change the one-way attention to two-way attention to better understand the context.

[0005] 2) Since the time complexity of LLM reasoning is the square of the input token length n, it is not possible to input too long tokens. However, the logs generated by the server cluster are often too large to be input into LLM at one time. If the input log is divided into k paragraphs and reasoned separately, the output of these k paragraphs will be independent and the full text cannot be understood.

[0006] 3) In addition to the important role of logs in server fault diagnosis, the error correction code (ECC) data on the memory chip and the fan rotation frequency also play a certain role in fault diagnosis. The ECC data can reflect the current memory status, and the fan frequency reflects the current chip workload status. If the chip load status is inconsistent with the program running status reflected in the log, a fault may occur. Therefore, compared with pure log detection, adding ECC data and fan data can better detect faults.

[0007] 4) Currently, the attention distance between tokens is realized through position embedding in LLM. Formula 1 is a set of sin-cos in the rotation position encoding (rope), where k is the dimension of the token word vector, d is the total dimension of the embedded word vector, b is the cardinality, and p is the coefficient. In mainstream LLM, the cardinality b is a hyperparameter set in advance by the developer, but it cannot be determined to meet most of the situations in the log.

[0008] Summary of the invention

[0009] 1. Technical issues to be resolved

[0010] The technical problem to be solved by the present invention is how to provide a large model-based fault detection method to solve the above-mentioned problems existing in the decoder mode LLM.

[0011] (II) Technical solution

[0012] In order to solve the above technical problems, the present invention proposes a large model-based fault detection method, which adopts an improved model for fault detection. The improved model includes: encoder-dyLLM, a sound embedding module and a word embedding module, wherein the improved model includes the following process:

[0013] S1. The system tool collects memory-related ECC and LOG, writes them into text data through the preprocessing program, and collects sound data through the sound collection program, and then inputs the improved model through the prompt word interface;

[0014] S2. Divide the text and sound data into n segments according to the timestamp. In the kth reasoning, the kth text data segment generates a text vector through the word embedding module, and the kth sound data generates a sound vector through the sound embedding module. The jth encoder-dyLLM reasoning outputs the cache vector memj. The text vector, sound vector and cache vector are concat and sent to the encoder-dyLLM reasoning to complete the kth improved model reasoning, where j=k-1.

[0015] S3, encoder-dyllama output includes: detection output vector and cache vector. The detection output vector is input into softmax after linear processing, and the fault is classified. The classification results include: normal, IO fault and system fault; the cache vector enters the next inference.

[0016] (III) Beneficial effects

[0017] The present invention proposes a fault detection method based on a large model. Compared with log fault detection based on the mainstream LLM, this model realizes the following functions:

[0018] 1) By modifying the decoder module, the bidirectional attention mechanism is introduced into fault detection. Compared with the mainstream LLM, it focuses more on text understanding tasks and improves the model's understanding ability;

[0019] 2) By modifying the rope module in the decoder and changing the base from a hyperparameter to a model parameter for training, the model's ability to capture text context semantics is improved;

[0020] 3) By introducing a global cache mechanism, Figure 2 memj and memk (summaries of the first paragraph to the jth paragraph and the kth paragraph) ensure that the model can perform reasoning based on historical paragraph summaries and the current paragraph, and solve the problem that the log is too long to be input into LLM reasoning;

[0021] 4) In addition to logs, sound and ECC data are also introduced to achieve multiple data fusion, which improves the robustness of the model for fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of a fault log in the prior art;

[0023] Figure 2 It is a schematic diagram of a decoder module in the prior art;

[0024] Figure 3 This is a structural diagram of the improved model of the present invention;

[0025] Figure 4 A schematic diagram of prompt words for data input;

[0026] Figure 5 This is a schematic diagram of the encoder-dyLLM structure;

[0027] Figure 6 This is a schematic diagram of the sound embedding module structure;

[0028] Figure 7 This is a schematic diagram of data splicing. DETAILED DESCRIPTION

[0029] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0030] In view of the above problems, the present invention proposes a large model-based fault detection method, which uses an improved model for fault detection. Figure 3 The improved model of the present invention includes: encoder-dyLLM, sound embedding module and word embedding module; the processing of the model includes the following process:

[0031] S1. The system tool collects memory-related ECC and LOG, and uses the preprocessor to separate the two into Figure 4 The data is written into text format, and the sound data is collected through the sound collection program, and then input into the model through the prompt word interface;

[0032] S2. Divide the text and sound data into n segments according to the timestamp. In the kth reasoning, the kth text data segment generates a text vector through the word embedding module, and the kth sound data generates a sound vector through the sound embedding module. The jth encoder-dyLLM reasoning outputs the cache vector memj. The text vector, sound vector and cache vector are calculated according to Figure 3 The format is concat and sent to encoder-dyLLM reasoning to complete the k-th model reasoning, where j = k-1.

[0033] S3, encoder-dyllama output includes: detection output vector and cache vector. The detection output vector is input into softmax after linear processing, and the fault is classified. The classification results include: normal, IO fault and system fault; the cache vector enters the next inference.

[0034] Furthermore, in S1, ECC and LOG data information are input in text form, and the present invention inputs data by constructing prompt words, such as Figure 4 As shown, the prompt words of the present invention include: ECC data, LOG data, cache vector (global data) and sound data, and the user can input corresponding data according to the prompts.

[0035] Furthermore, for the sound part, the collected signal contains noise, which needs to be subjected to noise reduction processing. At the same time, sound data has local features and global features like graphics, and convolution has good extraction performance for local features. Therefore, in S2, the present invention adopts Figure 6The sound embedding module network processes sound data. First, several layers of one-dimensional CNN networks are used to filter the sound sequence and extract local features. For long time series features, the network memory function needs to be used to save the above important information. Therefore, a multi-layer LSTM network is used to extract the overall features of the sound sequence. Finally, each time dimension is spliced ​​and fully connected to output the sound vector.

[0036] Figure 6 It is the network structure of the sound embedding module, in which the CNN network includes convolution, batch normalization, and activation function. Figure 6 As shown, in one embodiment, the sound embedding module includes a 5-layer CNN network and a 10-layer LSTM network.

[0037] Furthermore, in S2, the objects to be concat also include special tokens, which include ### and ***. ### in the prompt word indicates the separator of different types of data, and *** in the prompt word indicates the end of input. Special tokens are concat after word embedding. Therefore, when inputting data, you should follow Figure 7 The cache vector here corresponds to the summary vector output by the encoder-dyllama processing the previous segment. This vector code will use a global persistent space to save it to prevent the storage space from being released after the model inference ends.

[0038] Furthermore, if Figure 5 As shown, the main module in the improved model of the present invention is encoder-dyLLM, which is cascaded by 32 encoder modules.

[0039] The improved model of the present invention adopts the llama3 in the prior art as the basic model and makes corresponding changes to it. The main structure of the llama3 model is formed by stacking decoder modules. Figure 2 As shown in the figure, the decoder is mainly a one-way attention mechanism. The current mainstream model uses the decoder because the conversational robot is prohibited from paying attention to future information. Figure 2 CauselMask is a lower triangular matrix. When training the model through this Mask, future information can be shielded so that the model can complete the above attention mechanism training. Since the present invention needs to pay attention to context information, the use of CauselMask will only pay attention to the above information, so the present invention deletes it to form the encoder module of the present invention, while retaining the remaining parameters and structural information of the llama3 model.

[0040] At the same time, the present invention also replaces the ROPE module of the decoder module with the DYROPE module in the encoder module. The changes made to the DYROPE module relative to the ROPE are: the cardinality b is changed to a model parameter for training and dynamic parameter adjustment. As described in the background technology of the present invention, the cardinality b in the ROPE module is usually a hyperparameter set in advance by the developer, but it cannot be guaranteed to be up to date with the current log, so it is adjusted to a variable for training here. The present invention can improve the convergence speed and accuracy of model training by setting the cardinality b as a parameter and training it together with the model parameter.

[0041] At this time, the present invention replaces the decoder module of the llama main structure with the encoder module to complete the main construction, which is called the encoder-dyLLM model.

[0042] Compared with log fault detection based on mainstream LLM, this model implements the following functions:

[0043] 1) By modifying the decoder module, the bidirectional attention mechanism is introduced into fault detection. Compared with the mainstream LLM, it focuses more on text understanding tasks and improves the model's understanding ability;

[0044] 2) By modifying the rope module in the decoder and changing the base from a hyperparameter to a model parameter for training, the model's ability to capture text context semantics is improved;

[0045] 3) By introducing a global cache mechanism, Figure 2 memj and memk (summaries of the first paragraph to the jth paragraph and the kth paragraph) ensure that the model can perform reasoning based on historical paragraph summaries and the current paragraph, and solve the problem that the log is too long to be input into LLM reasoning;

[0046] 4) In addition to logs, sound and ECC data are also introduced to achieve multiple data fusion, which improves the robustness of the model for fault detection.

[0047] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fault detection method based on a large model, characterized in that: The method adopts an improved model for fault detection, and the improved model includes: encoder-dyLLM, sound embedding module and word embedding module, wherein the improved model includes the following process: S1. The system tool collects memory-related ECC and LOG, writes them into text data through the preprocessing program, and collects sound data through the sound collection program, and then inputs the improved model through the prompt word interface; S2. Divide the text and sound data into n segments according to the timestamp. In the kth reasoning, the kth text data segment generates a text vector through the word embedding module, and the kth sound data generates a sound vector through the sound embedding module. The jth encoder-dyLLM reasoning outputs the cache vector memj. The text vector, sound vector and cache vector are concat and sent to the encoder-dyLLM reasoning to complete the kth improved model reasoning, where j=k-1. S3, encoder-dyllama output includes: detection output vector and cache vector. The detection output vector is input into softmax after linear processing, and the fault is classified. The classification results include: normal, IO fault and system fault; the cache vector enters the next inference.

2. The large model-based fault detection method according to claim 1, characterized in that: In S1, data is input by constructing prompt words, and the prompt words include: ECC data, LOG data, cache vector and sound data.

3. The large model-based fault detection method according to claim 1, characterized in that: In S2, the sound data generates a sound vector through a sound embedding module, including: first, using several layers of one-dimensional CNN networks to filter the sound sequence and extract local features at the same time, then using a multi-layer LSTM network to extract the overall features of the sound sequence, and finally splicing each time dimension to fully connect and output the sound vector.

4. The large model-based fault detection method according to claim 3, characterized in that: The CNN network includes convolution, batch normalization, and activation function.

5. The large model-based fault detection method according to claim 3, characterized in that: The sound embedding module includes a 5-layer CNN network and a 10-layer LSTM network.

6. The large model-based fault detection method according to claim 1, characterized in that: In S2 shown, the objects of concat also include: special tokens, and the special tokens include: ### and ***, where ### represents the separator of different types of data in the prompt word, and *** represents the end of input in the prompt word. The special tokens are concat after word embedding.

7. The large model-based fault detection method according to claim 1, characterized in that: The cache vector corresponds to the summary vector output by the encoder-dyllama processing of the previous fragment. This vector code will use a global persistent space to save.

8. The large model-based fault detection method according to any one of claims 1 to 7, characterized in that: The encoder-dyLLM is formed by cascading encoder modules. The encoder-dyLLM uses llama3 as the backbone model and deletes the Causel Mask in the decoder module in the llama3 model to obtain the encoder module.

9. The large model-based fault detection method according to claim 8, characterized in that: The encoder-dyLLM consists of 32 encoder modules cascaded.

10. The large model-based fault detection method according to claim 8, characterized in that: The ROPE module of the decoder module is replaced by the DYROPE module in the encoder module. The changes made to the DYROPE module compared with ROPE are: the base b is changed to a model parameter for training and dynamic parameter adjustment.

Images