A log language model training method and system

Abstract

The application provides a log language model training method, which accepts log input and constructs a corresponding log corpus, constructs a data set to train a corresponding log language model, uses the log language model to perform vector representation on a log template, applies the obtained log template vector to downstream task training of abnormality detection, intercepts a log paragraph with a fixed sequence length to construct a sentence pair, that is, randomly divides the log paragraph with the fixed length into two long sentences, namely, sentence A and sentence B, and repeatedly constructs the log corpus ten times to form a data set. The application can train corresponding log language models for logs of different systems, improve the semantic expression capability of log word vectors, and input the generated log template vector into downstream task models such as log abnormality detection and log classification for training, thereby improving the effect of the models.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a log language model training method and system. BACKGROUND

[0002] As a kind of unstructured time series text data composed of timestamp and text information, log records the state information of software system running in real time, and is an important basis for operation and maintenance personnel to analyze abnormal behavior and errors of system. In recent years, with the continuous development of artificial intelligence, statistics, data mining and machine learning technologies are widely used in automatic analysis of logs to assist and promote application monitoring, management and troubleshooting.

[0003] In the field of AIOps (Intelligent Operation and Maintenance), log anomaly detection refers to using machine learning methods to identify system abnormal patterns that do not conform to expected behavior in log data, and to explore potential software faults behind abnormal patterns. Compared with traditional manual processing of log data methods, this research can help application systems further realize automated operation and maintenance, reduce the burden of human resources, and also help application systems quickly discover abnormal situations, so as to respond in the shortest time and reduce the loss caused by abnormal situations.

[0004] The general steps of current log anomaly detection methods are as follows: first, perform word segmentation on logs according to standards (such as time or identifier), then perform log parsing to parse log messages into log templates, next extract features from parsed logs, convert log templates into corresponding template vectors, and finally use different models for anomaly detection. Among them, feature extraction is the key to anomaly detection, and semantic vector is the most commonly used feature in anomaly detection, and the selection of different semantic vector extraction methods will directly affect the effect of anomaly detection model.

[0005] In recent years, pre-training language models represented by BERT have developed rapidly, and their representation ability for different languages far exceeds all previous language models. Current research in this field mainly falls into two categories: one is the improvement of language model structure itself, such as changing pre-training tasks, adding new features in word embedding layer, optimizing self-attention mechanism, etc.; the other is designing different fine-tuning schemes for different downstream tasks. These researches are all around natural language text, and no one has researched a set of strong usability log language model training scheme in the log field.

[0006] Since the log statements are semi-structured and different from the natural language expression, if the language model trained by the data in other fields is directly used in the log field, the representation ability of the log semantic information is weak, which restricts the performance of the downstream task (such as anomaly detection). Therefore, it is necessary to propose a log language model training method, which can more accurately represent the log language, thereby improving the effect of the downstream task such as anomaly detection. SUMMARY

[0007] In view of the above-mentioned disadvantages of the prior art, the purpose of the present application is to provide a log language model training method and system for solving the problem of inaccurate log language representation and poor anomaly detection effect in the prior art.

[0008] To achieve the above object and other related objects, the present application provides a log language model training method, which accepts the input of logs and constructs a corresponding log corpus, constructs a data set to train a corresponding log language model, uses the log language model to perform vector representation on the log template, and applies the obtained log template vector to the downstream task training of anomaly detection, wherein the specific steps of constructing the data set are as follows:

[0009] S1, word segmentation is performed on the log corpus in the log corpus, and then the log corpus is traversed;

[0010] S2, a fixed sequence length of log paragraph is intercepted for constructing a sentence pair, that is, the fixed length of the log paragraph is randomly divided into two long sentences, which are sentence A and sentence B;

[0011] S3, the probability is calculated to generate a random number between 0 and 1, when the value is less than 0.5, the sentence B is replaced by a sentence in other log paragraphs, otherwise the sentence B is maintained unchanged; sentence A and sentence B are spliced and [CLS] and [SEP] identifiers are added to form positive / negative examples of NSP task respectively;

[0012] S4, the positions corresponding to each word in the sentence pair in step S3 are disturbed, and the top 15% of the sentence pairs in the word sequence are selected for masking, each word has an 80% probability of being replaced by a [mask] label, a 10% probability of being replaced by any word in the word table, and a 10% probability of remaining unchanged; the index of the replaced word in the word table is used as a label for the training of the MLM task;

[0013] S5, the log corpus is repeatedly constructed ten times to form a data set.

[0014] In an embodiment of the present application, language model training after step S5 is further included, and the specific steps are as follows:

[0015] S6, the parameters of the pre-trained language model are fixed based on the general pre-trained language model, so that they do not participate in gradient calculation;

[0016] S7, three adapters are constructed to receive output vectors of the 0th, 6th and 11th layers of the pre-trained language model and the output vector of the previous adapter;

[0017] S8, two initial output vectors with a dimension of 768 are obtained, the two initial output vectors are connected and input to a fully connected layer for dimension reduction, and finally a vector with a dimension of 768 is obtained as a final output vector of the language model, and the log template vector is formed by the final output vector.

[0018] The above embodiment has the beneficial effect that all parameters of the pre-trained model are fixed, and an insertable adapter is constructed outside the model to receive the output of part of the layers in the model, and the required log language model is obtained by adjusting the parameters of the adapter, the number of adjustable parameters during model training is reduced to 40% of the original on the basis of maintaining the semantic extraction capability of the model, the parameter utilization rate of the model is improved, and the training time of the model is reduced.

[0019] In an embodiment of the present application, the pre-trained language model uses a cross-entropy function to calculate the loss during training, AdamW is the optimizer, and the adapter hidden layer dimension is 768.

[0020] In an embodiment of the present application, the corresponding log corpus is constructed, including step S9, log preprocessing, including log structure and variable type identification.

[0021] In an embodiment of the present application, it further includes step S10 after step S9, log parsing, using the regular expression corresponding to the log structure and the variable type obtained in step S9, using a log parsing algorithm to extract the log template.

[0022] In an embodiment of the present application, the log parsing algorithm is Drain, Spell or LogMine.

[0023] The above embodiment has the beneficial effect that multiple log parsing algorithms can meet multiple application scenarios and enhance applicability.

[0024] In an embodiment of the present application, the mask in step S4 adopts a bert-base-uncased, bert-large or RoBERTa mask type pre-trained model.

[0025] To achieve the above object and other related objects, the present application also provides a log language model training system based on the above log language model training method, comprising a data set construction module, which is used for segmenting log corpus in a log corpus and then traversing the log corpus; intercepting a log paragraph with a fixed sequence length to construct a sentence pair, i.e. randomly dividing the log paragraph with the fixed length into two long sentences, which are sentence A and sentence B respectively; calculating a probability and generating a random number between 0 and 1, when the value is less than 0.5, replacing sentence B with a sentence in other log paragraphs, otherwise maintaining sentence B unchanged; splicing sentence A and sentence B and adding [CLS] and [SEP] identifiers to form a positive / negative example of an NSP task respectively; scrambling the position corresponding to each word in the sentence pair, selecting the top 15% of the sentence pairs in word sequence order to mask, each word has an 80% probability of being replaced with a [mask] label, a 10% probability of being replaced with any word in the word table, and a 10% probability of remaining unchanged; using the index of the replaced word in the word table as a label for training the MLM task; and repeating the construction of the log corpus ten times to form a data set.

[0026] The language model training module is further used for fixing the parameters of the pre-trained language model based on the general pre-trained language model, so that the parameters do not participate in gradient calculation; three adapters are constructed to receive the output vectors of the 0th, 6th and 11th layers of the pre-trained language model and the output vector of the previous adapter; two initial output vectors with a dimension of 768 are obtained, the two initial output vectors are connected and input to a fully connected layer for dimension reduction, and finally a vector with a dimension of 768 is obtained as the final output vector of the language model, and the log template vector is formed by the final output vector.

[0027] The above embodiment has the beneficial effect that all the parameters of the pre-trained model are fixed, an insertable adapter is constructed outside the model to receive the output of part of the layers in the model, the required log language model is obtained by adjusting the parameters of the adapter, and the time consumption can be significantly shortened and the efficiency can be improved.

[0028] As described above, the log language model training method and system of the present application have the following beneficial effects: the present application can train corresponding log language models for different system logs, improve the semantic expression ability of log word vectors, input the generated log template vector into downstream task models such as log anomaly detection and log classification for training, and improve the effect of the models. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0030] Figure 1 The whole flowchart of the log language model training method of the present application is shown in the figure.

[0031] Figure 2 The MLM data set construction flowchart of the log language model training method of the present application is shown in the figure.

[0032] Figure 3 The NSP data set construction flowchart of the log language model training method of the present application is shown in the figure.

[0033] Figure 4 The language model training framework diagram of the log language model training method of the present application is shown in the figure.

[0034] Figure 5 The adapter structure diagram of the log language model training method of the present application is shown in the figure.

[0035] Figure 6 The HDFS log corpus diagram of the log language model training method of the present application is shown in the figure.

[0036] Figure 7 The HDFS log anomaly detection result diagram of the log language model training method of the present application is shown in the figure. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied through other different specific embodiments, and each detail in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0038] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component can be randomly changed in terms of type, number and proportion, and the component layout pattern can also be more complex.

[0039] Terminology explanation:

[0040] AIOps: Artificial Intelligence for IT Operations

[0041] Log parsing: converting unstructured logs into structured logs for log mining, such as log anomaly detection, fault prediction, etc.

[0042] Pre-trained language model: a pre-trained language model is a multi-layer network structure that is pre-trained to initialize the multi-layer network structure of the downstream task model, which can learn both shallow and deep information. This model is a dynamic text representation method that dynamically adjusts text representation according to the current context. The adjusted text representation can better express the specific meaning of words in this context and effectively handle the problem of polysemy.

[0043] OOV: Out of Value, a common problem in natural language processing, which occurs when the vocabulary in the sequence exceeds the vocabulary of the language model, causing the language model to fail to encode the vocabulary.

[0044] Bert-uncase-based: HuggingFace uses a masked language model (MLM) to pre-train the English language model.

[0045] MLM: Masked Language Modeling, a method of replacing some words in the corpus with other words to make the language model learn semantic information in the corpus by restoring the replaced words, which is one of the common pre-training methods for BERT models.

[0046] NSP: Next Sentence Prediction, a method of grouping adjacent or non-adjacent sentences in the corpus into sentence pairs, and the language model learns the semantic information of the two sentences to determine whether the sentence pair is adjacent, which is one of the common pre-training methods for BERT models.

[0047] NLP: Natural Language Processing, natural language processing.

[0048] Log tokenization: English translation for tokenization, refers to the decomposition of original text into small pieces. Tokenization decomposes the original text into words, which is called tokenized sentences. These tokens help understand the context or develop NLP models.

[0049] Sentence pair: a common term in the BERT model pre-training data set. The sentence pair in the noun refers to a sequence of words separated by [CLS] and [SEP] tags, which may be composed of multiple sentences in the corpus.

[0050] Embodiment one

[0051] Referring to Figures 1 to 7 The application provides a log language model training method, which accepts log input and constructs a corresponding log corpus, constructs a data set to train a corresponding log language model, uses the log language model to perform vector representation on a log template, and applies the obtained log template vector to downstream task training of anomaly detection, wherein the specific steps of data set construction are as follows: S1, performing word segmentation on log corpus in the log corpus, and then traversing the log corpus; S2, intercepting a log paragraph of a fixed sequence length for constructing a sentence pair, that is, randomly dividing the log paragraph of the fixed length into two long sentences, namely sentence A and sentence B; S3, calculating a probability, when a 0-1 random number is generated, if the value is less than 0.5, replacing sentence B with a sentence in another log paragraph, otherwise maintaining sentence B unchanged; concatenating sentence A and sentence B and adding [CLS] and [SEP] identifiers to form a positive / negative example of an NSP task; S4, the position corresponding to each word in the sentence pair in step S3 is disturbed, and the top 15% of the sentence pairs in word sequence sorting are selected for masking, each word has an 80% probability of being replaced with a [mask] label, a 10% probability of being replaced with any word in a word table, and a 10% probability of remaining unchanged; the index of the replaced word in the word table is used as a label for training of an MLM task; S5, the log corpus is repeatedly constructed ten times to form a data set.

[0052] Referring to Figure 2 and Figure 3 Since the sentence pairs and the masked words are different each time, the model can more fully learn the semantic information of the log sentences.

[0053] In another feasible embodiment, when constructing the data set for the MLM and NSP tasks, the proportion of the masked words can also be other than 15%. When the words are replaced according to the three schemes in a proportion of 8:1:1, other proportions can also be used, or only part of the replacement methods are used, such as replacing 90% of the words with other words in the word table and keeping 10% unchanged.

[0054] The language model training after step S5 is also included, and the specific steps are as follows: S6, fixing the parameters of the pre-training language model on the basis of the general pre-training language model, so that they do not participate in gradient calculation; S7, constructing three adapters for receiving the output vectors of the 0th, 6th and 11th layers of the pre-training language model and the output vector of the previous adapter; for details, please refer to Figure 5The adapter consists of a lower projection layer, two Transformer encoders and an upper projection layer. The output vectors of layers 0, 6 and 11 and the output vector of the previous adapter are added together and used for training. This can reduce the number of trainable parameters of the model while incorporating general semantic information from the pre-trained model.

[0055] S8. Obtain two initial output vectors with a dimension of 768. Concatenate the two initial output vectors and input them into a fully connected layer for dimensionality reduction. Finally, obtain a vector with a dimension of 768 as the final output vector of the language model. The log template vector is formed from the final output vector.

[0056] In short, the language model training of this invention involves fixing all parameters of a pre-trained model, building an insertable adapter outside the model to receive the output of some layers in the model, and obtaining the required log language model by adjusting the parameters of the adapter.

[0057] In this embodiment, the downstream task training specifically involves: the trained log language model can be used to encode log templates, obtaining template vectors corresponding to the log templates. Inputting these vectors into downstream task models such as log anomaly detection and log classification for training will improve the model's performance. Taking the log anomaly detection model as an example, when using LogAnomaly and LogRobust models for anomaly detection, compared to the currently most commonly used FastText pre-trained word vector method, the anomaly detection model trained using log template vectors generated by the log language model shows a 0.5%-3% improvement in accuracy, recall, and F1 score across multiple datasets.

[0058] In detail, the pre-trained language model uses the cross-entropy function to calculate the loss during training, AdamW is used as the optimizer, and the adapter hidden layer has a dimension of 768. Specifically, the output vector corresponding to [CLS] is used as the representation vector of the entire sentence pair and input into the NSP task classifier to predict whether the two sentences are adjacent and calculate the cross-entropy loss. The loss in the entire fine-tuning stage is the sum of the two losses. The AdamW optimizer is used to update the model parameters, and finally the training of the log language model is completed.

[0059] The construction of the corresponding log corpus includes step S9, log preprocessing, which involves identifying the log structure and variable types to facilitate structural decomposition and variable substitution of log statements during log parsing. Logs typically consist of a log header and a log message. The log header contains information such as timestamps and log levels, while the log message contains rich semantic information such as the operations performed during program execution and their corresponding results. Therefore, the key to structural decomposition is extracting the log message from the log statements. The extracted log message often contains rich variable information, such as numbers, IP addresses, URLs, and file paths. During preprocessing, corresponding regular expressions need to be written to identify and replace all variable information.

[0060] It also includes step S10 after step S9, log parsing, which uses a log parsing algorithm to extract log templates based on the log structure and the regular expressions corresponding to the variable types obtained in step S9.

[0061] The log parsing algorithm is Drain, Spell, or LogMine.

[0062] The mask in step S4 uses a bert-base-uncased, bert-large, or RoBERTa mask-based pre-trained model. It is understood that when training a log language model with an adapter, in addition to the adapter structure mentioned above, simple network structures such as feedforward neural networks can also be used instead.

[0063] The embodiments of the present invention take the HDFS log anomaly detection dataset as an example. The specific implementation steps of the present invention are as follows:

[0064] The first stage is to build a log corpus.

[0065] The Drain parsing method is used to parse HDFS logs into log templates. Variable types that need to be identified and replaced include block IDs, numbers, and IP addresses. The variable identifiers '<*>' in the log templates are removed, and the remaining words form the corresponding log statements. Using the block ID as an identifier, the log statements are grouped into contextually related log segments. After deduplication, these segments are stored in a file as an HDFS log corpus, such as... Figure 6 As shown.

[0066] The second stage is language model training.

[0067] The BERT-based uncased model, pre-trained on English corpora, was chosen as the base model. This model consists of 12 Transformer Encoder layers, generating word vectors with a dimension of 768, and its corresponding vocabulary size is 30522. The log corpus was segmented using the vocabulary, converting log statements into a set of indexed sequences with an index size between [0, 30521]. Log statements were traversed, and a set of sentences of length 512 was extracted to construct sentence pairs (512 is the maximum sequence length acceptable to the BERT model). This set of sentences was randomly divided into two longer sentences, A and B. Random numbers were generated between 0 and 1; if the result was less than 0.5, a set of sentences was selected from other segments to replace sentence B; otherwise, it remained unchanged. Sentences A and B were merged and their lengths trimmed, ultimately converting the log corpus into a set of sentence pairs, each with a length of 512.

[0068] In each sentence pair, 15% of the words are selected for replacement, using three word replacement methods in an 8:1:1 ratio. The indices of the replaced words are used as labels for the MLM task. This constructs the dataset needed for fine-tuning the language model.

[0069] Construct a log language model with adapters. First, the parameters of the pre-trained language model are fixed. Then, three adapters are built to receive the 0th, 6th, and 11th output vectors of the BERT model, as well as the output vector of the previous adapter. The final model outputs two 768-dimensional output vectors. These two vectors are concatenated and fed into a fully connected layer for dimensionality reduction, resulting in a 768-dimensional vector as the output vector of the language model. During model training, the cross-entropy function is used to calculate the loss, AdamW is used as the optimizer, and the adapter hidden layers have a dimension of 768.

[0070] Encoding Log Templates. After constructing the log language model, the log template statements are input into the language model for encoding. Taking HDFS as an example, all 48 log templates in the dataset ultimately generated log template vectors of length 768.

[0071] Phase Three, Downstream Tasks

[0072] The anomaly detection models LogAnomaly and LogRobust were used for validation. Both models take as input a sequence of template vectors corresponding to the log template sequence and output whether the sequence is anomaly. Therefore, this patent uses the log template vectors generated in the second stage as feature inputs to train the anomaly detection model. Simultaneously, the FastText pre-trained word vectors (300 dimensions) initially used by the model were selected as the experimental baseline.

[0073] All models were trained for 50 epochs and then tested using the same test set to obtain the accuracy, recall, and F1 score for each model. The results are as follows: Figure 7 As shown in the results, the log template vectors generated by the log language model significantly improve the performance of the anomaly detection model. Therefore, it can be concluded that the fine-tuning-based log language model proposed in this patent is superior to existing pre-trained word vector methods in terms of semantic extraction capability.

[0074] Example 2

[0075] This invention provides a log language model training system based on the log language model training method according to any one of claims 1-7. The system includes a dataset construction module for segmenting log data within a log corpus and then traversing the log data; extracting log segments of fixed sequence length to construct sentence pairs, i.e., randomly dividing a fixed-length log segment into two long sentences, namely sentence A and sentence B; calculating probabilities and generating random numbers from 0 to 1; when the value is less than 0.5, replacing sentence B with a sentence from another log segment, otherwise keeping sentence B unchanged; concatenating sentence A and sentence B and adding [CLS] and [SEP] identifiers to form positive / negative examples for the NSP task, respectively; shuffling the positions corresponding to each word in the sentence pair; selecting the top 15% of sentence pairs in the word sequence sorting for masking; each word has an 80% probability of being replaced with the [mask] label, a 10% probability of being replaced with any word in the vocabulary, and a 10% probability of keeping the original word unchanged; using the index of the replaced word in the vocabulary as a label for training the MLM task; and repeating the log data construction ten times to form a dataset.

[0076] It also includes a language model training module, which fixes the parameters of the pre-trained language model based on a general pre-trained language model so that it does not participate in gradient calculation; it constructs three adapters to receive the output vectors of layers 0, 6 and 11 of the pre-trained language model and the output vector of the previous adapter; it obtains two initial output vectors with a dimension of 768, concatenates the two initial output vectors and inputs them into a fully connected layer for dimensionality reduction, and finally obtains a vector with a dimension of 768 as the final output vector of the language model, which forms the log template vector.

[0077] In summary, the log language model training method and system of the present invention can train corresponding log language models for logs from different systems, improving the semantic expressive power of log word vectors. The generated log template vectors are input into downstream task models such as log anomaly detection and log classification for training, improving the model performance. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0078] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for training a log language model, the method comprising: The input of the log is accepted and the corresponding log corpus is constructed, the data set is constructed to train the corresponding log language model, the log template is vectorized using the log language model, and the obtained log template vector is applied to the downstream task training of anomaly detection, wherein the specific steps of constructing the data set are as follows: S1, word segmentation is performed on the log corpus in the log corpus, and then the log corpus is traversed; S2, a fixed sequence length of log paragraph is intercepted for constructing a sentence pair, that is, the fixed length of the log paragraph is randomly divided into two long sentences, which are sentence A and sentence B; S3, the probability is calculated, a random number between 0 and 1 is generated, when the value is less than 0.5, sentence B is replaced with a sentence in other log paragraphs, otherwise sentence B remains unchanged; sentence A and sentence B are spliced and CLS and SEP identifiers are added to form positive and negative examples of the NSP task, NSP is a pre-training method for judging whether the sentence pair is adjacent by learning the semantic information of the two sentences; S4, the position corresponding to each word in the sentence pair in step S3 is disturbed, and the top 15% of the sentence pair in the word sequence is selected for masking, each word has an 80% probability of being replaced with a mask label, a 10% probability of being replaced with any word in the word table, and a 10% probability of remaining unchanged; the index of the replaced word in the word table is used as a label for training the MLM task, and MLM is a model for learning the semantic information in the pre-linguistic model by restoring the replaced words; S5, the log corpus is repeatedly constructed ten times to form a data set; The language model training after step S5 is also included, and the specific steps are as follows: S6, fix the parameters of the pre-trained language model based on the general pre-trained language model, so that it does not participate in gradient calculation; S7, three adapters are constructed to receive the output vectors of the 0th, 6th and 11th layers of the pre-trained language model and the output vectors of the previous adapter; S8, two initial output vectors with a dimension of 768 are obtained, the two initial output vectors are connected and input to a fully connected layer for dimension reduction, and finally a vector with a dimension of 768 is obtained as the final output vector of the language model, and the final output vector forms the log template vector. 2.The log language model training method of claim 1, wherein, The pre-trained language model uses the cross-entropy function to calculate the loss during training, AdamW is the optimizer, and the adapter hidden layer dimension is 768. 3.The log language model training method of claim 1, wherein, The corresponding log corpus includes step S9, log preprocessing, including log structure and variable type identification. 4.The log language model training method of claim 3, wherein, Step S10 after step S9 is also included, log parsing, using the regular expression corresponding to the log structure and variable type obtained in step S9, using the log parsing algorithm, extracting the log template. 5.The log language model training method of claim 4, wherein, The log parsing algorithm is Drain, Spell or LogMine. 6.The log language model training method of claim 1, wherein, The mask in step S4 uses a bert-base-uncased, bert-large or RoBERTa mask pre-training model.

7. A log language model training system based on the log language model training method of any one of claims 1-6. The data set construction module is configured to perform word segmentation on the log corpus in the log corpus library, and then traverse the log corpus; intercept a log paragraph with a fixed sequence length to construct a sentence pair, i.e., randomly divide the log paragraph with the fixed length into two long sentences, which are sentence A and sentence B; Calculate the probability to generate a random number between 0 and 1, and if the number is less than 0.5, replace sentence B with a sentence in another log paragraph, otherwise maintain sentence B unchanged; Splice sentence A and sentence B and add CLS and SEP identifiers to form a positive example / negative example of an NSP task; Shuffle the positions of each word in the sentence pair, select the top 15% of the sentence pair in the word sequence order, and replace each word with a mask label with a probability of 80%, an arbitrary word in the word table with a probability of 10%, and the original word with a probability of 10%; the index of the replaced word in the word table is used as a label for training the MLM task; the log corpus is repeatedly constructed ten times to form a data set; The language model training module is configured to fix the parameters of the pre-trained language model on the basis of the general pre-trained language model, so that the pre-trained language model does not participate in gradient calculation; construct three adapters to receive the output vectors of the 0th, 6th and 11th layers of the pre-trained language model and the output vector of the previous adapter; obtain two initial output vectors with a dimension of 768, connect the two initial output vectors, input them into a fully connected layer for dimension reduction, and finally obtain a vector with a dimension of 768 as the final output vector of the language model, and form the log template vector from the final output vector.

Images