A large model-based fault detection method

Abstract

The present application relates to a kind of based on large model fault detection method, belong to big data field.The method of the present application carries out fault detection using improved model, and the improved model includes: encoder-dyLLM, sound embedding module and word embedding module, wherein the improved model includes the following processes: text and sound data are segmented into n sections according to timestamp, in the kth inference, the kth text data paragraph is generated by word embedding module Text vector, the kth sound data is generated by sound embedding module Sound vector, the jth encoder-dyLLM inference output buffer vector memj, text vector, sound vector and buffer vector are concatenated, input into encoder-dyLLM inference, complete the kth improved model inference;The classification result output by encoder-dyllama includes: normal, IO fault and system fault.The present application realizes the fusion of multiple data by reforming decoder module, introducing global cache, introducing sound, ECC data, and improves the robustness of model for fault detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of big data, and particularly relates to a big model fault detection method. BACKGROUND

[0002] Big data, cloud computing, and intelligent data mining have become the main direction of the Internet industry. Enterprises have established server cluster centers for these directions. These server cluster centers have a large number of servers and provide services to users for a long time. Once the server cluster fails, it needs to be repaired immediately, otherwise it will cause losses to the enterprise. In the face of such a large amount of data, traditional manual means to analyze the fault reason is completely not enough. At present, various automatic fault analysis tools have been developed. For example, rnn, lstm, bert and other methods, but such methods need to follow rules for data preprocessing, and the rule-based method needs to process the data in advance according to certain conditions. This process will lose a lot of information, resulting in poor robustness of fault detection. Program developers will add log printing to the corresponding position according to the logical relationship during code development. In these logs, in addition to a small part of intuitive information such as "FAILD" and "ERROR" that can be used to judge errors, some cases appear "FAILD" and are not faults, such as Figure 1 as shown, but through semantic relationship, it can be determined whether the log is a fault or normal.

[0003] In view of the shortcomings of the traditional method, the LLM model can well understand the text semantics and detect faults through logs. At present, many enterprises mainly use decoder mode LLM to analyze logs, which has achieved good results compared with traditional methods. However, there are still problems to be solved based on mainstream LLM:

[0004] 1) The mainstream LLM adopts the decoder mode mainly because it is aimed at the dialogue task. The dialogue cannot get all the context at once and needs to infer the next context based on the previous context. Fault detection based on logs is a text understanding task, which is based on context text understanding. At this time, the one-way attention needs to be changed to bidirectional attention, which can better understand the context.

[0005] 2) Since the inference time complexity of LLM is in quadratic relationship with the length n of input tokens, it cannot input too long tokens. However, the logs generated by the server cluster are often too large to be input to LLM at once. If the input log is divided into k paragraphs and inference is performed respectively, the outputs of the k paragraphs will be independent and cannot understand the whole text.

[0006] 3) In addition to the log playing an important role in server failure judgment, the error correction code (ECC) data on the memory chip and the fan rotation frequency also have a certain effect on the failure judgment. The ECC data can reflect the current memory state, and the fan frequency reflects the current chip workload state. If the chip load state is inconsistent with the program running state reflected in the log, a failure may occur. Therefore, compared with pure log detection, adding ECC data and fan data can better detect failures.

[0007] 4) Currently, the attention distance between tokens in LLM is realized through position embedding. Formula 1 is a set of sin-cos in the rotation position encoding (rope), where k is the dimension of the token vector, d is the total dimension of the embedded word vector, b is the base, and p is the coefficient. In the mainstream LLM, the base b is a hyperparameter set by the developer in advance, but it cannot be determined to meet most cases in the log.

[0008] SUMMARY

[0009] (1) Technical problems to be solved

[0010] The technical problem to be solved by the present application is how to provide a large model fault detection method to solve the above problems existing in the decoder mode LLM.

[0011] (2) Technical solutions

[0012] In order to solve the above technical problems, the present application provides a large model fault detection method, which adopts an improved model for fault detection. The improved model comprises an encoder-dyLLM, a sound embedding module and a word embedding module, wherein the improved model comprises the following processes:

[0013] S1, the system tool collects memory related ECC and LOG, and writes them into text data through a preprocessing program. Meanwhile, sound data is collected through a sound collection program, and then an improved model is input through a prompt word interface;

[0014] S2, the text and sound data are divided into n segments according to the time stamp. In the kth inference, the kth text data segment generates a text vector through the word embedding module, the kth sound data generates a sound vector through the sound embedding module, and the jth encoder-dyLLM inference outputs a cache vector memj. The text vector, the sound vector and the cache vector are concatenated and input into the encoder-dyLLM inference to complete the kth improved model inference, wherein j=k-1;

[0015] S3, the encoder-dyllama output includes: a detection output vector and a cache vector, wherein the detection output vector is input into softmax after linear processing, and the fault is classified, and the classification result includes: normal, IO fault and system fault; the cache vector enters the next inference.

[0016] (III) Advantages

[0017] The present application proposes a large model fault detection method, compared with the log fault detection based on the mainstream LLM, the model realizes the following functions:

[0018] 1) By modifying the decoder module, the bidirectional attention mechanism is introduced into fault detection, which is more focused on text understanding task than the mainstream LLM, and improves the model understanding ability;

[0019] 2) By modifying the rope module in the decoder, the base radix is changed from a hyperparameter to a model parameter for training, which improves the model's ability to capture text context semantics;

[0020] 3) By introducing a global cache mechanism, Figure 2 memj, memk (summary of the first paragraph to the jth paragraph, to the kth paragraph), which ensures that the model can summarize based on historical paragraphs and current paragraphs for inference, and solves the problem that the log is too long to be input into the LLM inference;

[0021] 4) In addition to the log, sound and ECC data are introduced to realize multi-data fusion, which improves the robustness of the model for fault detection. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 It is a fault log diagram in the prior art;

[0023] Figure 2 It is a decoder module diagram in the prior art;

[0024] Figure 3 It is an improved model structure diagram of the present application;

[0025] Figure 4 It is a data input prompt word diagram;

[0026] Figure 5 It is an encoder-dyLLM structure diagram;

[0027] Figure 6 It is a sound embedding module structure diagram;

[0028] Figure 7 It is a data splicing diagram. DETAILED DESCRIPTION

[0029] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0030] To address the above problems, this invention proposes a fault detection method based on a large model. This method employs an improved model for fault detection. Figure 3 The improved model of this invention includes: an encoder-dyLLM, a sound embedding module, and a word embedding module; the processing of this model includes the following steps:

[0031] S1. System tools collect memory-related ECC and LOG data, and preprocess them according to... Figure 4 The format is written as text data, and sound data is collected through a sound acquisition program. Then, the data is input into the model through a prompt word interface.

[0032] S2. Segment the text and audio data into n segments according to timestamps. In the k-th inference, the k-th text data segment generates a text vector through the word embedding module, and the k-th audio data segment generates an audio vector through the audio embedding module. The j-th encoder-dyLLM inference outputs a cache vector memj. The text vector, audio vector, and cache vector are then processed according to... Figure 3 The format is concatted and fed into encoder-dyLLM inference to complete the k-th model inference, where j = k-1.

[0033] S3, the encoder-dyllama output includes: detection output vector and cache vector. The detection output vector is linearly processed and then input into softmax to classify faults. The classification results include: normal, IO fault and system fault. The cache vector is used for the next inference.

[0034] Furthermore, in step S1, ECC and LOG data information are input in text form. This invention uses prompt words to input the data, such as... Figure 4 As shown, the prompts in this invention include: ECC data, LOG data, cache vector (global data), and sound data. Users can input the corresponding data according to the prompts.

[0035] Furthermore, for the audio portion, the acquired signal contains noise, requiring noise reduction processing. Simultaneously, audio data, like graphics, possesses both local and global features. Convolution has excellent extraction performance for local features; therefore, in step S2, this invention employs... Figure 6The sound embedding module network processes sound data. First, a multi-layer one-dimensional CNN network is used to filter the sound sequence and extract local features. For long-term sequence features, the network memory function needs to be used to save important information from the previous context. Therefore, a multi-layer LSTM network is used to extract the overall features of the sound sequence. Finally, each time dimension is concatenated and a fully connected output sound vector is generated.

[0036] Figure 6 This is the network structure for the sound embedding module, where the CNN network includes convolution, batch normalization, and activation functions. For example... Figure 6 As shown, in one embodiment, the sound embedding module includes a 5-layer CNN network and a 10-layer LSTM network.

[0037] Furthermore, in S2, the objects to be concatted also include special tokens, which include ### and ***. ### represents a separator between different data types in the prompt, and *** indicates the end of input. These special tokens are concatted after word embedding. Therefore, when inputting data, it is necessary to follow... Figure 7 The prompt word format is concatenated. The cached vector here corresponds to the summary vector output by encoder-dyllama after processing the previous segment. This vector will be stored in a global persistent space in the code to prevent the storage space from being released when the model inference ends.

[0038] Furthermore, such as Figure 5 As shown, the main module in the improved model of this invention is encoder-dyLLM, which consists of 32 cascaded encoder modules.

[0039] The improved model of this invention uses the existing llama3 model as its base model and modifies it accordingly. The main structure of the llama3 model is formed by stacking decoder modules. Figure 2 As shown. The decoder primarily uses a unidirectional attention mechanism. Current mainstream models employ decoders because chatbots are prohibited from focusing on future information. Figure 2 The CauselMask is a lower triangular matrix. When training the model, future information can be masked using this mask, allowing the model to complete the training of the context-aware attention mechanism. Since this invention requires attention to contextual information, and the CauselMask only focuses on the contextual information, it is removed in this invention to form the encoder module, while retaining the remaining parameters and structural information of the llama3 model.

[0040] Furthermore, this invention replaces the ROPE module in the decoder module with the DYROPE module in the encoder module. The DYROPE module differs from ROPE in that it replaces the cardinality b with model parameters for dynamic parameter tuning during training. As described in the background section of this invention, the cardinality b in the ROPE module is usually a hyperparameter pre-set by the developer, but it cannot be guaranteed to be up-to-date with the current logs. Therefore, this invention adjusts it as a variable for training. By setting the cardinality b as a parameter and training it together with the model parameters, this invention can improve the model training convergence speed and accuracy.

[0041] At this point, the present invention replaces the decoder module of the main structure of LLAM with the encoder module to complete the main structure construction, and it is called the encoder-dyLLM model.

[0042] Compared to log fault detection based on mainstream LLM, this model achieves the following functions:

[0043] 1) By modifying the decoder module to implement a bidirectional attention mechanism for fault detection, it focuses more on text understanding tasks than mainstream LLMs, thus improving the model's understanding ability;

[0044] 2) By modifying the rope module in the decoder and changing the base from a hyperparameter to a model parameter for training, the model's ability to capture the semantic context of the text was improved.

[0045] 3) By introducing a global caching mechanism, Figure 2 The `memj` and `memk` (summaries from the 1st paragraph to the jth paragraph and from the 1st paragraph to the kth paragraph) ensure that the model can perform inference based on historical paragraph summaries and the current paragraph, and also solve the problem that logs that are too long cannot be input into LLM inference.

[0046] 4) In addition to logs, sound and ECC data are also introduced to achieve multiple data fusion, which improves the robustness of the model for fault detection.

[0047] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A fault detection method based on a large model, characterized in that, This method employs an improved model for fault detection, which includes an encoder-dyLLM, a sound embedding module, and a word embedding module. The improved model comprises the following steps: S1. The system tools collect memory-related ECC and LOG data, and use a preprocessing program to compile both into text data. At the same time, the sound data is collected through a sound acquisition program, and then the improved model is input through the prompt word interface. S2. Divide the text and audio data into n segments according to the timestamp. In the kth inference, the kth text data segment generates a text vector through the word embedding module, and the kth audio data generates an audio vector through the audio embedding module. The jth encoder-dyLLM inference outputs a cache vector memj. The text vector, audio vector and cache vector are concatted and fed into the encoder-dyLLM inference to complete the kth improved model inference, where j=k-1. S3, the encoder-dyllama output includes: a detection output vector and a cache vector. The detection output vector is linearly processed and then input into the softmax to classify the faults. The classification results include: normal, IO fault, and system fault. The cache vector is used for the next inference. in, encoder-dyLLM is composed of cascaded encoder modules. encoder-dyLLM uses llama3 as the base model and removes the Causel Mask from the decoder module in the llama3 model to obtain the encoder module. The ROPE module in the decoder module has been replaced with the DYROPE module in the encoder module. The changes made to the DYROPE module compared to ROPE are: the cardinality b has been changed to the model parameters for training and dynamic parameter tuning.

2. The fault detection method based on a large model as described in claim 1, characterized in that, In step S1, input data is obtained by constructing prompt words, which include: ECC data, LOG data, cache vectors, and sound data.

3. The fault detection method based on a large model as described in claim 1, characterized in that, In step S2, the sound data is used to generate a sound vector through the sound embedding module. This includes: first, using a multi-layer one-dimensional CNN network to filter the sound sequence and extract local features; next, using a multi-layer LSTM network to extract the overall features of the sound sequence; and finally, concatenating each time dimension and performing a fully connected output sound vector.

4. The fault detection method based on a large model as described in claim 3, characterized in that, The CNN network includes convolution, batch normalization, and activation functions.

5. The fault detection method based on a large model as described in claim 3, characterized in that, The sound embedding module includes a 5-layer CNN network and a 10-layer LSTM network.

6. The fault detection method based on a large model as described in claim 1, characterized in that, In S2, the concat object also includes special tokens, which include: ### and In the prompt, ### represents a separator for different data types. The prompt indicates the end of input, and special tokens are concatted after word embedding.

7. The fault detection method based on a large model as described in claim 1, characterized in that, The cache vector corresponds to the summary vector output by encoder-dyllama after processing the previous segment. This vector will be stored in a global persistent space in the code.

8. The fault detection method based on a large model as described in claim 1, characterized in that, encoder-dyLLM consists of 32 cascaded encoder modules.

Images