Log anomaly detection model training method, device and equipment

Abstract

A log anomaly detection model training method, device and equipment are disclosed, applied to the field of artificial intelligence, the method comprising: obtaining a first log sample set, and pre-training an initial log anomaly detection model through the first log sample set to obtain a pre-trained log anomaly detection model, then obtaining a second log sample, and fine-tuning the pre-trained log anomaly detection model through the second log sample set, so as to obtain a trained log anomaly detection model, wherein the first log sample set is obtained by processing log data of a target object, the second log sample set is obtained by processing log data of a target sub-object, and the target sub-object belongs to the target object. This method can solve the problem of low generalization ability of the log anomaly detection model obtained by the prior art, which leads to the need to train a new model when a user implements anomaly detection on logs generated by similar objects, resulting in high labor and time costs and low efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus and device for training a log anomaly detection model. Background Technology

[0002] Hard disk drives (HDDs), network devices (such as routers and switches), and processors all generate various logs during operation to record their own status and important events. Logs contain rich dynamic information about operation, and abnormal information can be expressed through logs. Therefore, logs can be used for anomaly detection and fault diagnosis.

[0003] Currently, such as Figure 1 As shown, if a user wants to perform anomaly detection on logs generated by a specific object (such as a certain model of memory produced by a certain manufacturer), the main approach is to obtain historical logs generated by that specific object as training samples, train the initial log anomaly detection model, obtain a well-trained log anomaly detection model that has a good detection effect on logs generated by that specific object, and then use the trained model to perform anomaly detection on logs generated by that specific object.

[0004] However, the log anomaly detection model trained by the above method has the problem of low generalization ability. As a result, when users implement anomaly detection for logs generated by similar specific objects (such as memory of another model produced by another manufacturer), they cannot use the log anomaly detection model that has already been trained. They can only obtain historical logs generated by similar specific objects as training samples to train the initial log anomaly detection model to obtain a log anomaly detection model that has a better detection effect for logs generated by similar specific objects. The process of retraining a new model for different specific objects usually consumes a lot of manpower and time costs and is inefficient. Summary of the Invention

[0005] This application provides a method, apparatus, and device for training a log anomaly detection model, which can solve the problem that the generalization ability of the log anomaly detection model trained in the prior art is low. As a result, when users implement anomaly detection for logs generated by similar specific objects, they need to train a new log anomaly detection model for that specific object, which consumes a lot of manpower and time and is inefficient.

[0006] Firstly, a method for training a log anomaly detection model is provided, the method comprising:

[0007] Obtain a first log sample set, wherein the first log sample set is obtained by processing the log data of the target object;

[0008] The initial log anomaly detection model is pre-trained using the first log sample set to obtain a pre-trained log anomaly detection model.

[0009] Obtain a second log sample set, wherein the second log sample set is obtained by processing the log data of the target sub-object, and the target sub-object belongs to the target object;

[0010] The pre-trained log anomaly detection model is fine-tuned using the second log sample set to obtain a trained log anomaly detection model.

[0011] In the above scheme, the initial log anomaly detection model is pre-trained using a first log sample set from the target object. This yields a pre-trained log anomaly detection model with high-quality model parameters and strong generalization ability. When the user obtains a pre-trained log anomaly detection model for the target sub-object, a second log sample set from the target sub-object is used to fine-tune the pre-trained model with high-quality model parameters and strong generalization ability. Compared with existing technologies, the model training method provided in this application can solve the problem that existing technologies have low generalization ability of the trained log anomaly detection model, which requires users to retrain a new log anomaly detection model for the specific object when performing anomaly detection on logs generated by similar specific objects. This results in a large amount of manpower and time costs and low efficiency.

[0012] In one possible implementation, the target object includes at least one of the following sub-objects: hard disk, memory, flash memory, network device, and processor, wherein the target sub-object is any type of sub-object among the target objects.

[0013] In one possible implementation, the first log sample set includes m log samples, where m is a natural number greater than 1. The step of pre-training the initial log anomaly detection model using the first log sample set to obtain a pre-trained log anomaly detection model includes:

[0014] Each of the m log samples is segmented into words to obtain a sequence of m words corresponding to the m log samples;

[0015] The initial log anomaly detection model is pre-trained using the m word sequences to obtain a pre-trained log anomaly detection model.

[0016] In one possible implementation, the step of pre-training the initial log anomaly detection model using the m word sequences to obtain a pre-trained log anomaly detection model includes:

[0017] Each word in the m word sequence is masked according to a predetermined proportion to obtain a masked m word sequence;

[0018] The initial log anomaly detection model is pre-trained using the m-word sequence after masking to obtain the pre-trained log anomaly detection model.

[0019] In the above scheme, the initial log anomaly detection model is pre-trained using the masked m-word sequence to obtain the pre-trained log anomaly detection model. This allows the model to better learn the contextual information of the masked words, thereby enabling the pre-trained log anomaly detection model to learn the semantic information of each word sequence. This facilitates the subsequent trained log anomaly detection model to detect whether the log to be detected is abnormal based on the semantic information of the log to be detected.

[0020] In one possible implementation, the step of pre-training the initial log anomaly detection model using the m-word sequence after masking to obtain a pre-trained log anomaly detection model includes:

[0021] Obtain the word embedding vector and position embedding vector corresponding to each word in the m word sequences after masking, wherein the word embedding vector corresponding to each word is a multi-dimensional vector used to represent each word, and the position embedding vector corresponding to each word represents the position of each word in its word sequence;

[0022] Based on the word embedding vector and position embedding vector corresponding to each word in the masked m-word sequence, obtain the m first row vectors corresponding to the masked m-word sequence;

[0023] Using the m first row vectors, the initial log anomaly detection model is pre-trained to obtain the pre-trained log anomaly detection model.

[0024] In one possible implementation, the step of pre-training the initial log anomaly detection model using the m first row vectors to obtain the pre-trained log anomaly detection model includes:

[0025] Each of the m first row vectors is input into the initial log anomaly detection model for training, resulting in m second row vectors.

[0026] Obtain the loss from the m second row vectors to the initial cluster centers;

[0027] Based on the loss from the m second row vectors to the initial cluster center, train the initial log anomaly detection model to obtain the pre-trained log anomaly detection model and the target cluster center.

[0028] In one possible implementation, the method further includes:

[0029] Obtain the percentiles of the losses from the m second row vectors to the target cluster center;

[0030] The classification threshold is determined based on the percentiles corresponding to the losses from the m second row vectors to the target cluster center. The classification threshold is used by the trained log anomaly detection model to perform anomaly detection on the log to be detected and obtain the detection result.

[0031] In one possible implementation, the formula for obtaining the loss from the m second row vectors to the initial cluster centers is:

[0032]

[0033] Among them, V i Let represent the i-th second row vector among the m second row vectors, and let c represent the initial cluster center. loss(c, V) i ) represents the loss from the i-th second row vector to the initial cluster center, where i is a natural number.

[0034] Secondly, a log anomaly detection model training device is provided, the device comprising:

[0035] The acquisition module is used to acquire a first log sample set, wherein the first log sample set is obtained by processing the log data of the target object;

[0036] The training module pre-trains the initial log anomaly detection model using the first log sample set to obtain a pre-trained log anomaly detection model.

[0037] The acquisition module is further configured to acquire a second log sample set, wherein the second log sample set is obtained by processing the log data of the target sub-object, and the target sub-object belongs to the target object;

[0038] The training module is further configured to fine-tune the pre-trained log anomaly detection model using the second log sample set to obtain a trained log anomaly detection model.

[0039] In one possible implementation, the target object includes at least one of the following sub-objects: hard disk, memory, flash memory, network device, and processor, wherein the target sub-object is any type of sub-object among the target objects.

[0040] In one possible implementation, the first log sample set includes m log samples, where m is a natural number greater than 1, and the training module is specifically used for:

[0041] Each of the m log samples is segmented into words to obtain a sequence of m words corresponding to the m log samples;

[0042] The initial log anomaly detection model is pre-trained using the m word sequences to obtain a pre-trained log anomaly detection model.

[0043] In one possible implementation, the training module is specifically used for:

[0044] Each word in the m word sequence is masked according to a predetermined proportion to obtain a masked m word sequence;

[0045] The initial log anomaly detection model is pre-trained using the m-word sequence after masking to obtain the pre-trained log anomaly detection model.

[0046] In one possible implementation, the training module is specifically used for:

[0047] Obtain the word embedding vector and position embedding vector corresponding to each word in the m word sequences after masking, wherein the word embedding vector corresponding to each word is a multi-dimensional vector used to represent each word, and the position embedding vector corresponding to each word represents the position of each word in its word sequence;

[0048] Based on the word embedding vector and position embedding vector corresponding to each word in the masked m-word sequence, obtain the m first row vectors corresponding to the masked m-word sequence;

[0049] Using the m first row vectors, the initial log anomaly detection model is pre-trained to obtain the pre-trained log anomaly detection model.

[0050] In one possible implementation, the training module is specifically used for:

[0051] The m first row vectors are respectively input into the initial log anomaly detection model for training to obtain m second row vectors. The m second row vectors have a one-to-one correspondence with the m word sequences after masking. Each of the m second row vectors includes the semantic information of the word sequence after masking.

[0052] Obtain the loss from the m second row vectors to the initial cluster centers;

[0053] Based on the loss from the m second row vectors to the initial cluster center, train the initial log anomaly detection model to obtain the pre-trained log anomaly detection model and the target cluster center.

[0054] In one possible implementation, the training module is further configured to:

[0055] Obtain the percentiles of the losses from the m second row vectors to the target cluster center;

[0056] The classification threshold is determined based on the percentiles corresponding to the losses from the m second row vectors to the target cluster center. The classification threshold is used by the trained log anomaly detection model to perform anomaly detection on the log to be detected and obtain the detection result.

[0057] In one possible implementation, the formula for obtaining the loss from the m second row vectors to the initial cluster centers is:

[0058]

[0059] Among them, V i Let represent the i-th second row vector among the m second row vectors, and let c represent the initial cluster center. loss(c, V) i ) represents the loss from the i-th second row vector to the initial cluster center, where i is a natural number.

[0060] Thirdly, a non-transient computer-readable storage medium is provided, the non-transient computer-readable medium storing instructions for implementing the method provided as described in the first aspect or any possible implementation thereof.

[0061] Fourthly, a computing device is provided, the computing device including a processor and a memory; the processor is configured to execute instructions stored in the memory, causing the computing device to implement the method provided by the first aspect or any possible implementation thereof.

[0062] Fifthly, a computer program product is provided, comprising a computer program that, when read and executed by a computing device, causes the computing device to perform the method provided as described in the first aspect or any possible implementation thereof. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0064] Figure 1 This is a schematic diagram of a prior art involved in this application;

[0065] Figure 2 This is a schematic diagram illustrating how a masked language model (MLM) method, as described in this application, masks words in an input sequence.

[0066] Figure 3This is a schematic diagram illustrating the principle of a log anomaly detection model training method provided in this application;

[0067] Figure 4 This is a flowchart illustrating a log anomaly detection model training method provided in this application;

[0068] Figure 5 This is a schematic diagram of obtaining the word sequence corresponding to the i-th first log sample provided in this application;

[0069] Figure 6 This is a schematic diagram of obtaining the first row vector and the second row vector corresponding to the masked word sequence provided in this application;

[0070] Figure 7 This is a schematic diagram illustrating an exemplary word embedding vector and position embedding vector provided in this application;

[0071] Figure 8 This is a schematic diagram of an exemplary word vector provided in this application;

[0072] Figure 9 This is a flowchart illustrating the process of training a pre-trained log anomaly detection model provided in this application.

[0073] Figure 10 This is a schematic diagram illustrating the use of m first row vectors to pre-train an initial log anomaly detection model to obtain m second row vectors and the target cluster center, as per this application.

[0074] Figure 11 This is a flowchart illustrating a log anomaly detection method involved in this application;

[0075] Figure 12 This is a schematic diagram illustrating the anomaly detection result obtained by performing anomaly detection on the first row vector x corresponding to the sequence to be detected, as described in this application.

[0076] Figure 13 This is a schematic diagram of the structure of a log anomaly detection model training device provided in this application;

[0077] Figure 14 This is a schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0078] The embodiments of this application are described below with reference to the accompanying drawings.

[0079] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0080] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0081] To facilitate understanding of the embodiments of this application by those skilled in the art, the relevant concepts or terms involved in this application will be introduced first.

[0082] (1) A log is a record generated by objects such as hard drives, network devices, and processors to indicate the status of the hard drive, network device, processor, etc., and what events have occurred. For example, a hard drive generates a log when a failure occurs or when a failure is expected to occur.

[0083] Logs are typically stored in devices as log files. These files can be directly readable text files, machine-readable binary files, or other file formats; this application does not impose any specific limitations. Each log file consists of lines of log records. One or several consecutive records describe an independent event, and a log record describing an independent event can be called a log entry. A log file includes multiple log entries. A log entry typically includes the event's occurrence time, event content, event type, event level, etc.

[0084] Typically, different objects generate log entries in different formats. For example, the format of a log entry generated by device A is: the time of the event, the identifier of the device accessing device A, and the event content. A log entry generated by device A contains 20 characters. The format of a log entry generated by device B is: the identifier of the device accessing device B, the time of the event, and the event content. A log entry generated by device B contains 50 characters.

[0085] (2) A neural network can be composed of neural units (also called neurons). A neural unit can refer to a variable x. s The operation unit takes the intercept b as input, and the output of this operation unit can be:

[0086]

[0087] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function or other functions, which are not limited here. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0088] (3) Loss Function: During model training, to ensure the model's output is as close as possible to the desired predicted value, we compare the current model's predicted value with the desired target value and update the weight vector of each layer of the neural network based on the difference. (Of course, there is usually an initialization process before the first update, i.e., pre-configuring parameters for each layer in the model). For example, if the model's predicted value is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the model can predict the desired target value or a value very close to it. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important functions used to measure the difference between the predicted value and the target value. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so the model training process becomes a process of minimizing this loss as much as possible.

[0089] (4) Pre-training refers to the process of training an initial model using a large dataset, enabling the initial model to learn to recognize the general features in the large dataset. The pre-trained model (hereinafter referred to as the pre-trained model) has strong generalization ability and can provide high-quality model parameters for subsequent model training on specific datasets, and can adapt to a variety of specific datasets.

[0090] (5) Fine-tuning refers to the process of further training a pre-trained model using a specific dataset to obtain a trained model applicable to that specific dataset. Typically, the amount of data in the specific dataset used in the fine-tuning stage is less than the amount of data in the large dataset used in the pre-training stage. For example, the ratio of the amount of data in the specific dataset used in the fine-tuning stage to the amount of data in the large dataset used in the pre-training stage is 1:100, 1:500, 1:1000, etc., without specific limitations. The number of times the pre-trained model is trained using the specific dataset in the fine-tuning stage is less than the number of times the initial model is trained using the large dataset in the pre-training stage. For example, to obtain the pre-trained model, the number of times the initial model is trained using the large dataset is set to 100 times in the pre-training stage; to obtain the trained model, the number of times the pre-trained model is trained using the specific dataset is set to 20 times in the fine-tuning stage.

[0091] (6) MLM method, a model pre-training method, trains a model that can learn the semantic information of the input sequence, which is stored in the vector output by the model corresponding to the word "CLS" in the input sequence. This is because the masking strategy of MLM for the input sequence is different from the masking strategy of traditional pre-training methods (such as masked sequence to sequence (MASS)). In MLM, 15% of the words in the input sequence are randomly selected, 80% of the words in the randomly selected 15% are replaced with the special symbol MASK, 10% of the words in the randomly selected 15% are replaced with random words, and the remaining 10% are not replaced.

[0092] For example, such as Figure 2 As shown, the input sequence is <cls>The input sequence, "<Raise><Head><Look><Bright><Moon><Lower><Head><Think><Home><Home>", consists of 11 words. MLM randomly selects 15% of the words from the input sequence, which are the third word "head" and the seventh word "low". MLM replaces the third word "head" in the randomly selected input sequence with the special symbol MASK and replaces the seventh word "low" in the randomly selected input sequence with the random word "person".

[0093] When pre-training a model, masking the sequences input into the model for training can improve the fault tolerance and inference accuracy of the pre-trained model.

[0094] For example, if the first input sequence during model pre-training is... <cls>"Raise the head, look at the bright moon, lower the head, think of hometown", that is, the sequence without masking processing. The model is used to learn this first sequence. Since each word in this first sequence is not masked, that is to say, each word in the first sequence is known. Therefore, the model only needs to learn the words in the first sequence and does not need to learn the context of the words in the first sequence to learn that the semantic information of the first sequence is "Raise the head, look at the bright moon, lower the head, think of hometown". Therefore, the obtained pre-trained model usually does not have the ability to infer the semantic information of the sequence based on the context of the words in the sequence. If the pre-trained model is subsequently used for the second sequence <cls>By reasoning about the semantic information of "<Raise head><Look at bright moon><Lower head><Think of hometown>", it can be seen that compared with the first sequence, the second sequence lacks the three words "<Head><Lower><Hometown>". Since the pre-trained model does not have the ability to reason about the semantic information of the sequence based on the context of the words in the sequence, the model is less likely to infer that the semantic information of the second sequence is "Raise head, look at bright moon, lower head, think of hometown", and it is more likely to infer that the semantic information of the second sequence is "Missing hometown" or "Looking at bright moon and missing hometown", resulting in lower fault tolerance and accuracy of the model.

[0095] If during the pre-training of the model, the input first sequence is <cls><For example> <mask><Hope> <mask><Month><Lower><Head><Think><Of><Hometown> <mask>, that is, the masked sequence. The model is used to learn this first sequence. Since some words in this first sequence (i.e., <head><bright><hometown>) are masked, that is to say, some words in the first sequence are unknown and the other part of the words are known. Therefore, the model not only needs to learn the known words in the first sequence, but also needs to learn the masked words according to the context of the masked words in the first sequence. After learning the masked words, it can learn that the semantic information of the first sequence is "Raising my head, I look at the bright moon; lowering my head, I am thinking of my hometown". Therefore, the obtained pre-trained model usually has the ability to infer the semantic information of the sequence according to the context of the words in the sequence. If the pre-trained model is subsequently used for the second sequence <cls>By reasoning about the semantic information of "<Raise head><Look at><Bright moon><Lower head><Think of><Hometown>", it can be seen that compared with the first sequence, the second sequence lacks the two words "<Lower><Old>", and has the two words "<Bright><Hometown>". Since the pre-trained model has the ability to reason about the semantic information of the sequence based on the context of the words in the sequence, therefore, it is more likely that the semantic information of the second sequence inferred by the model is "Raise head, look at the bright moon, lower head, think of hometown", and it is less likely that the semantic information of the second sequence inferred by it is "Think of hometown" or "Look at the bright moon and think of hometown". The fault tolerance ability and accuracy of the model are relatively high.

[0096] (7) Log anomaly detection refers to detecting the event information included in the log entries to determine whether the event information included in the log entries is the information of the device anomaly that generated the log entries. In the case where it is determined that the event information included in the log entries is the information of the device anomaly, it is determined that the device has an anomaly.

[0097] (8) Initial log anomaly detection model refers to the model (which can also be called an algorithm) before being trained using log training samples. The purpose of training the initial log anomaly detection model using log training samples is to obtain a model that can perform anomaly detection on logs.

[0098] As the scale of the system continues to increase, it is difficult to ensure the stability and reliability of large-scale systems. In addition, the network environment is becoming increasingly complex, and various new types of attacks are emerging continuously. And anomaly detection is one of the supporting technologies for ensuring system security. During the operation of the system, the hard disk, processor of the system, and network devices providing network services for the system, etc. will generate various log files to record the running state and events that occur in the system. Logs contain rich information, and a large amount of log data contains huge amounts of information, providing a way for the anomaly detection of the system, making log anomaly detection a research hotspot in the field of anomaly detection. Among them, using the trained log anomaly detection model obtained by training with log training samples to perform anomaly detection on logs is a relatively popular log anomaly detection method currently.

[0099] However, the log anomaly detection model obtained by training with the existing training method of the log anomaly detection model has the problem of low generalization ability. When users implement anomaly detection on the logs generated by similar specific objects, users cannot use the already trained log anomaly detection model, and can only re-obtain the historical logs generated by similar specific objects as training samples to re-train the initial log anomaly detection model to obtain a log anomaly detection model with better detection effect for the logs generated by similar specific objects. The process of training the new model usually consumes a large amount of human and time costs, and the efficiency is low.

[0100] For example, if a user has already trained a well-developed log anomaly detection model A that performs well in detecting logs generated from the hard drive, but wants to perform anomaly detection on logs generated from memory, although model A performs well in detecting logs generated from the hard drive, it usually performs poorly in detecting logs generated from memory. Even if the user continues to train model A using historical logs generated from memory as training samples, the low generalization ability of model A usually prevents the user from obtaining a well-developed log anomaly detection model that performs well in detecting logs generated from memory. The user can only obtain historical logs generated from memory again as training samples and use the newly obtained training samples to train the initial log anomaly detection model to obtain a well-developed log anomaly detection model that performs well in detecting logs generated from memory.

[0101] For example, if a user has already trained a well-developed log anomaly detection model B (hereinafter referred to as Model B) that performs well in detecting logs generated by hard drives of model B manufactured by manufacturer B (hereinafter referred to as Hard Drive B), but wants to perform anomaly detection on logs generated by hard drives of model C manufactured by manufacturer C (hereinafter referred to as Hard Drive C), although Model B performs well in detecting logs generated by Hard Drive B, it usually performs poorly in detecting logs generated by Hard Drive C. Even if the user continues to train Model B using historical logs generated by Hard Drive C as training samples, due to Model B's low generalization ability, it usually cannot obtain a log anomaly detection model that performs well in detecting logs generated by Hard Drive C. The user can only obtain historical logs generated by Hard Drive C again as training samples and use the newly obtained training samples to train the initial log anomaly detection model to obtain a well-developed log anomaly detection model that performs well in detecting logs generated by Hard Drive C.

[0102] To address the problems existing in the training methods for existing log anomaly detection models, this application provides a training method for log anomaly detection models, such as... Figure 3 As shown, the model training method provided in this application includes two stages: pre-training and fine-tuning. In the pre-training stage, a first log sample set from the target object (which includes multiple target sub-objects) is used to pre-train the initial log anomaly detection model, resulting in a model with high-quality model parameters and strong generalization ability. In the fine-tuning stage, a second log sample set from the target sub-object (which belongs to the target object) is used to fine-tune the pre-trained log anomaly detection model, thus obtaining a well-trained log anomaly detection model for the target sub-object. Compared with existing technologies, the model training method provided in this application can solve the problem of low generalization ability of the log anomaly detection model obtained by existing technologies, thereby improving the efficiency of model training.

[0103] The target object includes, but is not limited to, at least one of the following sub-objects: hard disk, memory, flash memory, network device, and processor; the target sub-object can be any type of sub-object among the target objects. It is readily understood that the above target object and target sub-object are merely exemplary examples, and this application does not impose specific limitations.

[0104] Taking the target object as an example where the target object only includes the hard disk as a seed object, the target sub-object can be a hard disk of different models.

[0105] Taking a target object that includes both hard disk and memory sub-objects as an example, the target sub-object can be either hard disk or memory.

[0106] It is understandable that when training a model, the more training samples used, the better the parameters of the trained model tend to be, and the wider the source of the training samples used, the stronger the generalization ability of the trained model tends to be. Therefore, in this embodiment, the target object can include as many sub-objects as possible, and the amount of data in the first log sample set obtained from the target object can also be as large as possible. In this way, the parameters of the pre-trained log anomaly detection model will be better, and the generalization ability of the model will be stronger.

[0107] To facilitate a clearer understanding of the log anomaly detection model training method provided in this application, the following section combines... Figure 4 The flowchart shown below provides a detailed description of the method provided in this application, such as... Figure 4 As shown, the method includes the following steps:

[0108] S401. The computing device acquires a first log sample set including m first log samples. The first log sample set is obtained by processing the log data of the target object.

[0109] As described above, log entries include event occurrence time, event content, event type, event level, etc. Among them, event content reflects the event that occurred. By analyzing the event content, it can be determined whether the event was abnormal. However, event occurrence time, event type, and event level have less effect on determining whether an event is abnormal.

[0110] Therefore, in one possible implementation, the computing device can extract m first event contents as m first log samples from a large number of log entries included in the log data of a large number of target objects, and delete other parts of the large number of log entries besides the event content, such as event occurrence time, event type, and event level. The log data of the large number of target objects can be obtained by the computing device from web crawlers on the Internet or manually collected from the target objects; there are no restrictions on this.

[0111] For example, suppose the target object's log data includes log entry A: 2021 / 06 / 03Thu 18:18:33PD_Vendor Done Check done,0xd ms.Flag 8ALL, where 2021 / 06 / 03Thu 18:18:33 is the event occurrence time, PD_Vendor Done Check done,0xd ms.Flag 8 is the event content, and ALL is the event level. Then, the first log sample obtained by the computing device from log entry A is PD_Vendor Done Check done,0xdms.Flag 8.

[0112] It is understandable that by obtaining m first event contents from the log data of a large number of target objects as m first log samples, and deleting other parts of the log data of a large number of target objects other than the event content, such as the event occurrence time, event type, and event level, the differences in format between the m first log samples can be masked, thereby increasing the number of first log samples in the first log sample set.

[0113] It is understandable that the larger the value of m, the better the parameters of the pre-trained log anomaly detection model, and the stronger the model's generalization ability. Therefore, in practical implementation, the computing device can acquire as many first log samples as possible and add them to the first log sample set. Here, m is a natural number greater than 1.

[0114] It should be noted that the above example of obtaining m first event entries from a large amount of target object log data as m first log samples is merely an example and should not be considered a specific limitation. In a concrete implementation, the computing device may also select m log entries with the same format from a large amount of target object log data and directly use them as m first log samples, or obtain m preset contents from a large amount of target object log data as m first log samples. The preset contents include not only the event content in the log entries but also other content in the log entries besides the event content, such as the event level and / or event type.

[0115] S402, The computing device performs word segmentation on each of the m first log samples to obtain the m word sequences corresponding to the m log samples.

[0116] Taking the word segmentation of the i-th first log sample out of m first log samples by a computing device as an example, the process by which the computing device obtains the word sequence corresponding to the i-th first log sample includes:

[0117] S4021. Segment the i-th first log sample to obtain the first word sequence.

[0118] Continuing with the example of the i-th first log sample, PD_Vendor Done Check done, 0xd ms.Flag 8, the computing device performs word segmentation on this log sample, resulting in the following first word sequence:

[0119] <PD_VendorDone> <check> <done> <0xd> <flag><8> ,like Figure 5 As shown.

[0120] S4022. If the first word sequence includes a mixed word composed of numbers and characters, replace the mixed word composed of numbers and characters in the first word sequence with the word "number" to obtain the second word sequence. If the first word sequence does not include a mixed word, no replacement operation is required, and the first word sequence is directly determined as the second word sequence.

[0121] Continuing with the example given in S4021, it can be seen that the first word sequence<PD_VendorDone> <check> <done> <0xd> <flag><8> Including the mixed word 0xd, the computing device processes the first word sequence.<PD_VendorDone> <check> <done> <0xd> <flag><8> The second word sequence obtained after replacing the mixed word 0xd in the sequence is:

[0122] <PD_VendorDone> <check> <done> <number> <flag><8> ,like Figure 5 As shown.

[0123] It should be noted that the above example of replacing the mixed word with "number" is just one example. In a specific implementation, other words such as "num" or "sep" can also be used to replace the mixed word. No specific limitation is made here.

[0124] S4023. After obtaining the second word sequence, add a special classification marker, namely the CLS marker, to the beginning of the sentence in the second word sequence to obtain the third word sequence.

[0125] The CLS marker at the beginning of the third sequence sentence marks the start of the third word sequence.

[0126] Continuing with the example given in S4022, the computing device in the second word sequence<PD_VendorDone> <check> <done> <number> <flag><8> Adding a CLS marker to the beginning of the sentence results in the following third-word sequence:

[0127] <cls><PD_VendorDone> <check> <done> <number> <flag><8> ,like Figure 5 As shown in S4024, after obtaining the third word sequence, if the number of words included in the third word sequence is less than a preset threshold, add a pad marker to the end of the sentence of the third word sequence to obtain the fourth word sequence, so that the number of words included in the fourth word sequence is equal to the preset threshold. If the number of words included in the third word sequence is equal to the preset threshold, directly determine the third word sequence as the fourth word sequence. If the number of words included in the third word sequence is greater than the preset threshold, truncate the words at the end of the sentence of the third word sequence to obtain the fourth word sequence, so that the number of words included in the fourth word sequence is equal to the preset threshold.

[0128] Continuing with the example in S4023, assuming a preset threshold of 10, it can be seen that the third word order... <cls><PD_VendorDone> <check> <done> <number> <flag><8> If the number of words included is 8, which is less than the preset threshold of 10, then the computing device will calculate the third word sequence. <cls><PD_VendorDone> <check> <done> <number> <flag><8> Adding two pad marks to the end of the sentence results in the following fourth word sequence:

[0129] <cls><PD_VendorDone> <check> <done> <number> <flag> <8> <pad> <pad>,like Figure 5 As shown.

[0130] It should be noted that the above example of replacing mixed words with "pad" is merely an example. In specific implementations, other words such as "PAD" or "pa" can also be used to replace mixed words, and no specific limitation is made here. The preset threshold of 10 is merely an example. In specific implementations, it can also be 20, 50, etc., and no specific limitation is made here.

[0131] S4025. After obtaining the fourth word sequence, use a preset dictionary to transform each word in the fourth word sequence to obtain the fifth word sequence, which is the word sequence corresponding to the i-th first log sample.

[0132] The preset dictionary includes a large number of words and the correspondence between the words and their corresponding tokens (identifications, or IDs). For example, the correspondence between the word "Check" and token ID "6".

[0133] Continuing with the example in S4024, assuming the preset dictionary includes token ID 1 for CLS, token ID 0 for pad, token ID 5 for PD_VendorDone, token ID 6 for Check, token ID 7 for done, token ID 4 for number, token ID 8 for ms, token ID 9 for Flag, token ID 10 for 8, and token ID 0 for pad, then the computing device uses the preset dictionary to process the fourth word sequence. <cls><PD_VendorDone> <check> <done> <number> <flag> <8> <pad> <pad>After conversion, the fifth word sequence is:

[0134] <1> <5> <6> <7> <4> <8> <9> <10> <0> <0> ,like Figure 5 As shown.

[0135] In a specific embodiment of this application, when using a preset dictionary to convert words in the fourth word sequence, if a word in the fourth word sequence and its corresponding token ID do not exist in the preset dictionary, the word and its corresponding token ID can be added to the preset dictionary.

[0136] For example, if the largest token ID included in the preset dictionary is "100000", and the preset dictionary does not include the token ID of the word "identification", then the word "identification" and its corresponding token ID "100001" or "100008" can be added to the preset dictionary.

[0137] It should be noted that the above process of obtaining the word sequence corresponding to the i-th first log sample is only an example. In the specific implementation, S4023 can be executed before S4022, or S4024 can be executed before S4022. No specific limitation is made here.

[0138] It is understandable that if the computing device performs steps S4021 to S4025 on each of the m first log samples, the computing device can obtain m word sequences, and the number of words included in the m word sequences is equal.

[0139] S403. The computing device pre-trains the initial log anomaly detection model using m word sequences to obtain a pre-trained log anomaly detection model.

[0140] In a specific embodiment of this application, the computing device pre-trains the initial log anomaly detection model using m word sequences, which can be an MLM method or a MASS method, etc. No specific limitations are imposed here.

[0141] Taking the MLM method as an example of the pre-training method used, the process of pre-training the initial log anomaly detection model by the computing device to obtain the pre-trained log anomaly detection model can specifically include the following steps:

[0142] S4031, The computing device performs masking processing on words in the m word sequence at a preset proportion, and obtains the masked m word sequence.

[0143] The preset ratio can be 10%, 15%, 20%, etc.

[0144] The operation of the computing device to mask words in a predetermined proportion of m word sequences can be found in the description of the MLM method above.

[0145] S4032. The computing device pre-trains the initial log anomaly detection model using the masked m-word sequence to obtain the pre-trained log anomaly detection model.

[0146] In a specific embodiment of this application, the computing device pre-trains the initial log anomaly detection model using a masked sequence of m words. The specific process of obtaining the pre-trained log anomaly detection model may include the following steps:

[0147] S1. The computing device obtains the word embedding vector and position embedding vector corresponding to each word in the m-word sequence after masking.

[0148] The word embedding vector for each word is a multi-dimensional vector representing that word. Word embedding vectors are a collective term for a set of language modeling and feature learning techniques in natural language processing. They are a way to mathematically represent words in a language; as the name suggests, their task is to transform a word into a multi-dimensional vector. In practice, the word embedding vector for each word can be obtained through one-hot encoding, a Word2Vec model, or a Glove model. The dimension of the word embedding vector for each word can be 256 or 512 dimensions, or more or fewer dimensions; no specific limitation is made here.

[0149] Using the i-th word sequence after masking as <1> <mask><6> <7> <4> <9> <9> <10> <0> <0> For example, Figure 6 As shown, assuming the method for obtaining the word embedding vector corresponding to each word in the i-th word sequence after masking is the Word2Vec model, and the word embedding vector has a dimension of 5, then the word "1" in the i-th word sequence after masking can be a 5-dimensional vector V. i,1 (-0.065, -0.035, 0.019, -0.026, 0.085), the word "MASK" can be represented by a 5-dimensional vector V. i,2 (0.000, 0.000, 0.000, 0.000, 0.000), ..., the word "0" at the end of the sentence can be represented by a 5-dimensional vector V. i,10 (-0.027, -0.013, 0.006, 0.023, 0.014), such as Figure 7 As shown. Among them, the 5-dimensional vector V i,t The subscript t in the sequence represents the position of each word in the i-th word sequence.

[0150] It should be noted that the numerical values ​​in the word embedding vectors exemplified above, including three decimal places, are merely examples. In specific implementations, they may include fewer or more decimal places, and no specific restrictions are imposed here.

[0151] The position embedding vector corresponding to each word is used to represent the position of each word in the word sequence, and its dimension is the same as that of the word embedding vector.

[0152] In one possible implementation, a word's positional embedding vector can be obtained using the following formula:

[0153]

[0154]

[0155] Where PE() represents the position embedding vector, pos represents the position of the word in the word sequence, and its value ranges from [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 1, 1, 1, 2 ... model Let d represent the dimension of the position embedding vector, 2j represent the even-numbered dimension index of the position embedding vector, and 2j+1 represent the odd-numbered dimension index of the position embedding vector. model Taking 5 as an example, the dimension d of the embedding vector at the position is... model When j is 5, j takes the values ​​0, 1, and 2 respectively. When j is 0, the calculated value of PE(pos,2j) (i.e., PE(pos,0)) is the value of the zeroth dimension of the embedding vector at that position, and the value of PE(pos,2j+1) (i.e., PE(pos,1)) is the value of the first dimension of the embedding vector at that position. When j is 1, the calculated value of PE(pos,2j) (i.e., PE(pos,2)) is the value of the second dimension of the embedding vector at that position, and the calculated value of PE(pos,2j+1) (i.e., PE(pos,3)) is the value of the third dimension of the embedding vector at that position. When j is 2, the calculated value of PE(pos,2j) (i.e., PE(pos,4)) is the value of the fourth dimension of the embedding vector at that position.

[0156] Continue with Figure 6 The sequence of the i-th word after masking is shown below. <1> <mask><6> <7> <4> <9> <9> <10> <0> <0> The dimension d of the position embedding vector to be obtained model Taking 5 as an example, the embedding vector V corresponding to the word "1" in the i-th word sequence after masking is obtained by the above formula. i,1 'for:

[0157] ((PE(0,2*0)=sin(0 / 10000 (2*0 / 5) ), (PE(0,2*0+1)=cos(0 / 10000 (2*0 / 5) ), (PE(0,2*1)=sin(0 / 10000) (2*1 / 5) ), (PE(0,2*1+1)=cos(0 / 10000 (2*1 / 5) ), (PE(0,2*2)=sin(0 / 10000) (2*2 / 5) That is, (0.000, 1.000, 0.000, 1.000, 0.000), such as Figure 7 As shown;

[0158] The embedding vector V corresponding to the word "MASK" in the i-th word sequence after masking, obtained through the above formula. i,2 'for:

[0159] ((PE(1,2*0)=sin(1 / 10000 (2*0 / 5) ), (PE(1,2*0+1)=cos(1 / 10000 (2*0 / 5) ), (PE(1,2*1)=sin(1 / 10000) (2*1 / 5) ), (PE(1,2*1+1)=cos(1 / 10000 (2*1 / 5) ), (PE(1,2*2)=sin(1 / 10000) (2*2 / 5) That is, (0.842, 0.540, 0.025, 1.000, 0.001), such as Figure 7 As shown; ...;

[0160] The embedding vector V corresponds to the position of the "0" at the end of the sentence. i,10 'for:

[0161] ((PE(9,2*0)=sin(9 / 10000 (2*0 / 5) ), (PE(9,2*0+1)=cos(9 / 10000 (2*0 / 5) ), (PE(9,2*1)=sin(9 / 10000) (2*1 / 5) ), (PE(9,2*1+1)=cos(9 / 10000 (2*1 / 5) ), (PE(9,2*2)=sin(9 / 10000) (2*2 / 5) That is, (0.412, -0.911, 0.224, 0.975, 0.006), such as Figure 7 As shown.

[0162] It should be noted that the numerical values ​​in the position embedding vectors mentioned above, which include three decimal places, are merely examples. In specific implementations, they may include fewer or more decimal places, and no specific restrictions are imposed here.

[0163] In another possible implementation, a word's positional embedding vector can be obtained using the following formula:

[0164]

[0165]

[0166] S2. The computing device obtains the m first row vectors corresponding to the m word sequence after masking, based on the word embedding vector and position embedding vector corresponding to each word in the m word sequence after masking.

[0167] Specifically, the computing device can obtain the word vector corresponding to each word in each masked word sequence by superimposing the word embedding vector and the position embedding vector corresponding to each word in each masked word sequence, thereby obtaining the first row vector corresponding to each masked word sequence. Other methods of obtaining the first row vector through the word embedding vector and the position embedding vector are also within the protection scope of this application and are not specifically limited here.

[0168] Continue with Figure 6 Taking the word embedding vector and position embedding vector corresponding to each word in the i-th word sequence after masking as an example, the word vector V corresponding to the word "1" is... i,1 "for:

[0169] like Figure 8 As shown;

[0170] The word vector V corresponding to the word "MASK" i,2 "for:

[0171] like Figure 8 As shown; ...;

[0172] The word vector V corresponding to the word "0" at the end of the sentence i,10 "for:

[0173] like Figure 8 As shown.

[0174] After obtaining the word vector corresponding to each word in the masked word sequence, the combination of all the word vectors corresponding to the words in the masked word sequence is the first row vector V corresponding to the masked word sequence. i '.

[0175] In this embodiment, the process of obtaining the first row vector corresponding to each word sequence in the masked m-word sequence is the same as the process of obtaining the first row vector V corresponding to the i-th word sequence after masking. i The process is similar to that of ', and you can refer to the relevant description above for details, which will not be elaborated here.

[0176] S3. The computing device uses m first row vectors to train the initial log anomaly detection model, thus obtaining the pre-trained log anomaly detection model.

[0177] In a specific embodiment of this application, the computing device uses m first row vectors to train an initial log anomaly detection model. The specific process of obtaining the pre-trained log anomaly detection model is as follows: Figure 9 As shown, it may include the following steps:

[0178] A1. Input the m first row vectors into the initial log anomaly detection model for training to obtain the m second row vectors corresponding to the m word sequences after masking.

[0179] In this context, the second row vector corresponding to the masked word sequence represents a vector containing the semantic information of the masked word sequence, and the second row vector corresponding to each word sequence is the output of the initial log anomaly detection model corresponding to the CLS tag of each word sequence.

[0180] Continue with Figure 6 The i-th word sequence after masking shown <1> <mask><6> <7> <4> <9> <9> <10> <0> <0> The first row vector V in i For example, if the first row vector V is... i 'By training the initial log anomaly detection model, we can obtain the second row vector V.' i ,like Figure 6 As shown, the second row vector V i The semantic information of the i-th word sequence after masking is "Supplier information check complete".

[0181] A2. Randomly select any one of the m second row vectors and use it as the initial cluster center c of the normal log class.

[0182] A3. Calculate the loss from each of the m second row vectors to the initial cluster center c.

[0183] In a specific embodiment of this application, the i-th second row vector V can be obtained using the following loss function. i Loss to the initial cluster center c:

[0184]

[0185] in, This indicates that for the second row vector V i Take the square root of the value in each dimension.

[0186] A4. Based on the loss from the i-th second row vector to the initial cluster center c, determine whether the i-th second row vector can be assigned to the normal log class. If it is determined that the i-th second row vector can be assigned to the normal log class, assign the i-th second row vector to the normal log class.

[0187] Specifically, the loss from the i-th second row vector to the initial cluster center c can be calculated as loss(c,V). i If the value is less than the first classification threshold, the i-th second row vector is determined to belong to the normal log class; otherwise, the i-th second row vector is determined not to belong to the normal log class. The first classification threshold can be set by the user according to the actual situation.

[0188] A5. After assigning the second row vectors that can be assigned to the normal log class from among the m second row vectors to the normal log class, recalculate the centroid of the normal log class and determine the calculated centroid as the new cluster center c1 of the normal log class.

[0189] A6. Iterate through steps A3 to A5 until the termination condition is met to obtain the pre-trained log anomaly detection model.

[0190] The termination condition can be the maximum number of iterations, minimizing the squared error, the rate of change of the cluster center point, etc., and no specific limitation is made here.

[0191] When the termination condition is met, the normal log class and its centroid no longer change. Here, C represents the centroid of the normal log class that no longer changes, which is the target cluster center as described below.

[0192] like Figure 10 As shown, m first row vectors are input into the initial log anomaly detection model for training, resulting in m second row vectors. These are ultimately divided into second row vectors belonging to the normal log class (referring to...). Figure 10 The vector within the circle shown) and the second row vector that does not belong to the normal log class (referring to...) Figure 10 The vector outside the circle shown is C, and the centroid of the normal log class is C.

[0193] Although the pre-trained log anomaly detection model already has high-quality model parameters, it is trained using a first log sample set from the target object. If this model is directly used to detect anomalies in logs generated by the target sub-object, the model parameters will not be accurate enough, resulting in low accuracy of the detection results. Therefore, the pre-trained log anomaly detection model can be fine-tuned using a training sample set from the target sub-object (i.e., the second log sample set including n second log samples as described below) to obtain more accurate model parameters. With more accurate model parameters, the detection results for anomalies in logs generated by the target sub-object will also be more accurate.

[0194] It is understandable that since the pre-trained log anomaly detection model already has high-quality model parameters, when fine-tuning the pre-trained log anomaly detection model using a second training sample set from the target sub-object, the second training sample set only needs to include a small number of second training samples to obtain more accurate model parameters. The process of fine-tuning to obtain more accurate model parameters only requires a small amount of manpower and time cost, and the model training efficiency is high.

[0195] S404. The computing device obtains a second log sample set including n second log samples. The second log sample set is obtained by processing the log data of the target sub-object.

[0196] Among them, a large amount of log data for the target sub-objects can be historical logs generated by the target sub-objects.

[0197] It's understandable that the larger the value of n, the more accurate the parameters of the trained log anomaly detection model will be. Therefore, in practical implementations, the computing device can acquire as many second log samples as possible and add them to the second log sample set. Here, n is a natural number greater than 1, and n is typically less than m.

[0198] S405 and the computing device respectively perform word segmentation on n second log samples to obtain n word sequences of the second log.

[0199] S406. The computing device fine-tunes the pre-trained log anomaly detection model using n word sequences to obtain a trained log anomaly detection model.

[0200] It is understandable that the model parameters of the fine-tuned log anomaly detection model are more accurate than those of the pre-trained log anomaly detection model. Therefore, the detection results obtained by using the trained log anomaly detection model to perform anomaly detection on the logs generated by the target sub-object will also be more accurate.

[0201] In this embodiment, the process by which the computing device acquires a second log sample set including n second log samples is similar to the process in S401 where the computing device acquires a first log sample set including m first log samples; for details, please refer to the relevant description in S401. The process by which the computing device segments the n second log samples to obtain n word sequences corresponding to the n second log samples is similar to the process in S402 where the computing device segments the m first log samples to obtain m word sequences corresponding to the m first log samples; for details, please refer to the relevant description in S402. The process by which the computing device fine-tunes the pre-trained log anomaly detection model using the n word sequences to obtain a trained log anomaly detection model is similar to the process in S403 where the computing device pre-trains the initial log anomaly detection model using the m word sequences to obtain a pre-trained log anomaly detection model; for details, please refer to the relevant description in S403.

[0202] In a specific embodiment of this application, after obtaining the target cluster center C, the computing device can also calculate the second classification threshold for the trained log anomaly detection model to perform anomaly detection on the log to be detected based on the target cluster center C and m second row vectors.

[0203] Furthermore, the process by which the computing device calculates the second classification threshold based on the target cluster center C and m second row vectors may include:

[0204] B1. Calculate the loss from the m second row vectors to the target cluster center C using the computing device.

[0205] The process by which the computing device obtains the loss from the m second row vectors to the target cluster center C is similar to the process described above for obtaining the loss from the m second row vectors to the initial cluster center C. For details, please refer to the relevant description above.

[0206] B2. The computing device obtains the percentiles corresponding to the loss from the m second row vectors to the target cluster center C.

[0207] Percentile is a statistical term. If a set of data is sorted from smallest to largest and the corresponding cumulative percentiles are calculated, the value of the data corresponding to a certain percentile is called the percentile of that percentile. For example, the value at the 80th position is called the 80th percentile.

[0208] Therefore, the computing device obtains the percentiles corresponding to the losses from the m second row vectors to the target cluster center C. That is, the computing device sorts the losses from the m second row vectors to the target cluster center C from smallest to largest and calculates the corresponding cumulative percentiles.

[0209] B3. The computing device determines the target percentile based on the percentiles corresponding to the losses from the m second row vectors to the target cluster center C.

[0210] B4. The computing device determines the second classification threshold based on the target percentile.

[0211] In a specific embodiment of this application, the second classification threshold T can be determined using the following formula:

[0212] T = P·β

[0213] Where P represents the value at the target percentile, and β is used to expand the distance around the target cluster center C. P and β can be determined based on the ratio of normal to abnormal samples in the m first log samples. For example, if the number of normal samples is much greater than the number of abnormal samples in the m first log samples (e.g., a ratio of 10000:1 or 5000:1), the target percentile can be as large as possible, such as 90% or 95%, and β can be 1.8, 2.0, or 2.5. If the number of normal samples is close to the number of abnormal samples in the m first log samples (e.g., a ratio of 500:1 or 100:1), the target percentile can be close to 50%, such as 45% or 51%, and β can be 1.2 or 1.5. The second classification threshold T is used by the trained log anomaly detection model to perform anomaly detection on the logs to be detected. (See also: [link to documentation]). Figure 11 The relevant description in the document.

[0214] As can be seen, this application calculates the second classification threshold T based on the target cluster center C and m second row vectors. Furthermore, the calculation of the second classification threshold T takes into account the number of normal samples and the number of abnormal samples in the m first log samples, using the target percentile value and the value of β. This differs from existing technologies where the classification threshold is set manually based on experience. Manually set thresholds that are too large or too small can significantly impact the accuracy of the trained log anomaly detection model. For example, a manually set threshold that is too large may cause the trained log anomaly detection model to misclassify abnormal logs as normal logs, while a manually set threshold that is too small may cause the trained log anomaly detection model to misclassify normal logs as abnormal logs. Therefore, the method provided in this application for determining the second classification threshold T can improve the accuracy of the trained log anomaly detection model in anomaly detection.

[0215] In a specific implementation, after training the log anomaly detection model using the log anomaly detection model training method provided in this application, the trained log anomaly detection model can be deployed to the target sub-object, and the trained log anomaly detection model can be used to perform anomaly detection on the logs generated by the target sub-object.

[0216] Please see Figure 11 , Figure 11 This application provides an exemplary flowchart for detecting anomalies in the logs of a target sub-object using a trained log anomaly detection model, as shown below. Figure 11 As shown, the detection process includes:

[0217] S111. Obtain the log entries to be detected generated by the target sub-object.

[0218] It is understandable that the log entries to be detected here are the same as the logs to be detected mentioned above.

[0219] S112. Obtain the content of the event to be detected from the log entries to be detected.

[0220] S113. Segment the content of the event to be detected into words to obtain the sequence of words to be detected corresponding to the content of the event to be detected.

[0221] S114. Obtain the first row vector corresponding to the word sequence to be detected.

[0222] S115. Input the first row vector corresponding to the word sequence to be detected into the trained log anomaly detection model to perform anomaly detection and obtain the detection result.

[0223] In a specific embodiment of this application, the process of inputting the first row vector corresponding to the word sequence to be detected into the trained log anomaly detection model for anomaly detection and obtaining the detection result may include the following steps:

[0224] S1151. Input the first row vector corresponding to the word sequence to be detected into the trained log anomaly detection model to obtain the second row vector corresponding to the word sequence to be detected.

[0225] S1152. Obtain the loss from the second row vector corresponding to the word sequence to be detected to the cluster center C'.

[0226] Wherein, the cluster center C' represents the centroid of the normal log class that no longer changes when the n second row vectors corresponding to the n second log samples are clustered together after the log anomaly detection model is trained during the fine-tuning stage.

[0227] The loss(C',X) from the second row vector corresponding to the word sequence to be detected to the cluster center C' can be obtained by the following formula:

[0228]

[0229] Where X represents the second row vector corresponding to the word sequence to be detected.

[0230] S1153. Determine whether the loss(C',X) is less than the second classification threshold T. If the loss(C',X) is less than the second classification threshold T, execute S1154. If the loss(C',X) is greater than or equal to the second classification threshold T, execute S1155.

[0231] In a specific implementation, it can also be determined whether the loss(C',X) is less than or equal to the second classification threshold T. If it is determined that the loss(C',X) is less than or equal to the second classification threshold T, S1154 is executed. If it is determined that the loss(C',X) is greater than the second classification threshold T, S1155 is executed.

[0232] S1154. Determine that there is no information about device abnormality in the log entry to be detected. Here, "device" refers to the device that generated the log entry to be detected.

[0233] S1155. Confirm that there is information about device malfunction in the log entries to be tested.

[0234] like Figure 12 As shown, the first row vector x corresponding to the word sequence to be detected is input into the trained log anomaly detection model for anomaly detection. The anomaly detection result includes the loss(C',X) of the second row vector X corresponding to the word sequence to be detected relative to the cluster center C'. Assuming that the loss(C',X) is 5 and the second classification threshold T is 8, if the loss(C',X) is less than the second classification threshold T, the trained log anomaly detection model will classify the second row vector X into the normal log class and output the detection result that there is no device anomaly information in the log entry to be detected.

[0235] It should be noted that in the examples above, the output of the trained log anomaly detection model showing no device anomaly information in the log entries to be detected is merely an example. In a specific implementation, the output detection result can also be "device normal" or similar, and no specific restrictions are imposed here.

[0236] In this embodiment, the definitions of the word sequence to be detected, the first row vector corresponding to the word sequence to be detected, etc., are the same as those in this embodiment. Figure 4 The definitions of word sequences, first row vectors, etc., in the embodiments are the same; please refer to [link / reference needed] for details. Figure 4 The relevant content in the illustrated embodiment will not be described in detail here. In this embodiment, the process of segmenting the content of the event to be detected to obtain the sequence of words to be detected corresponding to the content of the event to be detected is similar to the process in S402 where the computing device segments m first log samples to obtain m word sequences corresponding to the m first log samples. For details, please refer to the relevant description in S402. The process of obtaining the first row vector corresponding to the sequence of words to be detected is similar to the process in S403 where the computing device obtains the m first row vectors corresponding to the m word sequences after masking. For details, please refer to the relevant description in S403.

[0237] It should be noted that when obtaining the first row vector corresponding to the word sequence to be detected, there is no need to mask the word sequence to be detected. The word embedding vector and position embedding vector corresponding to each word in the word sequence to be detected can be obtained directly, thereby obtaining the first row vector corresponding to the word sequence to be detected.

[0238] It should be noted that although the above description of the log anomaly detection model training method provided in this application uses computing devices as the execution subject, in specific implementations, the execution subject of the log anomaly detection model training method provided in this application can also be a computing device cluster including at least two computing devices. The at least two computing devices in the computing device cluster can collaboratively implement the log anomaly detection model training method provided in this application. For example, if the computing device cluster includes computing device A and computing device B, step S401 is executed by computing device A, and steps S402 to S406 are executed by computing device B, or steps S401 to S403 are executed by computing device A, and steps S404 and S406 are jointly executed by computing device A and computing device B.

[0239] The above section details a method for training a log anomaly detection model provided in this application. Based on the same inventive concept, the following section introduces a training device for a log anomaly detection model provided in this application.

[0240] See Figure 13 , Figure 13 This is a schematic diagram of a log anomaly detection model training device 100 provided in this application. The device 100 includes: an acquisition module 110 and a training module 120, wherein...

[0241] The acquisition module 110 is used to acquire a first log sample set, wherein the first log sample set is obtained by processing the log data of the target object;

[0242] The training module 120 pre-trains the initial log anomaly detection model using the first log sample set to obtain a pre-trained log anomaly detection model.

[0243] The acquisition module 110 is further configured to acquire a second log sample set, wherein the second log sample set is obtained by processing the log data of the target sub-object, and the target sub-object belongs to the target object;

[0244] The training module 120 is further configured to fine-tune the pre-trained log anomaly detection model using the second log sample set to obtain a trained log anomaly detection model.

[0245] In one possible implementation, the target object includes at least one of the following sub-objects: hard disk, memory, flash memory, network device, and processor, wherein the target sub-object is any type of sub-object among the target objects.

[0246] In one possible implementation, the first log sample set includes m log samples, where m is a natural number greater than 1, and the training module 120 is specifically used for:

[0247] Each of the m log samples is segmented into words to obtain a sequence of m words corresponding to the m log samples;

[0248] The initial log anomaly detection model is pre-trained using the m word sequences to obtain a pre-trained log anomaly detection model.

[0249] In one possible implementation, the training module 120 is specifically used for:

[0250] Each word in the m word sequence is masked according to a predetermined proportion to obtain a masked m word sequence;

[0251] The initial log anomaly detection model is pre-trained using the m-word sequence after masking to obtain the pre-trained log anomaly detection model.

[0252] In one possible implementation, the training module 120 is specifically used for:

[0253] Obtain the word embedding vector and position embedding vector corresponding to each word in the m word sequences after masking, wherein the word embedding vector corresponding to each word is a multi-dimensional vector used to represent each word, and the position embedding vector corresponding to each word represents the position of each word in its word sequence;

[0254] Based on the word embedding vector and position embedding vector corresponding to each word in the masked m-word sequence, obtain the m first row vectors corresponding to the masked m-word sequence;

[0255] Using the m first row vectors, the initial log anomaly detection model is pre-trained to obtain the pre-trained log anomaly detection model.

[0256] In one possible implementation, the training module 120 is specifically used for:

[0257] The m first row vectors are respectively input into the initial log anomaly detection model for training to obtain m second row vectors. The m second row vectors have a one-to-one correspondence with the m word sequences after masking. Each of the m second row vectors includes the semantic information of the word sequence after masking.

[0258] Obtain the loss from the m second row vectors to the initial cluster centers;

[0259] Based on the loss from the m second row vectors to the initial cluster center, train the initial log anomaly detection model to obtain the pre-trained log anomaly detection model and the target cluster center.

[0260] In one possible implementation, the training module 120 is further configured to:

[0261] Obtain the percentiles of the losses from the m second row vectors to the target cluster center;

[0262] The classification threshold is determined based on the percentiles corresponding to the losses from the m second row vectors to the target cluster center. The classification threshold is used by the trained log anomaly detection model to perform anomaly detection on the log to be detected and obtain the detection result.

[0263] In one possible implementation, the formula for obtaining the loss from the m second row vectors to the initial cluster centers is:

[0264]

[0265] Among them, V i Let represent the i-th second row vector among the m second row vectors, and let c represent the initial cluster center. loss(c, V) i ) represents the loss from the i-th second row vector to the initial cluster center, where i is a natural number.

[0266] Specifically, the specific implementation of various operations performed by the above-mentioned log anomaly detection model training device 100 can be referred to the description in the relevant content of the above-mentioned log anomaly detection model training method embodiment. For the sake of brevity, it will not be repeated here.

[0267] It should be understood that the log anomaly detection model training device 100 is merely one example provided in this application embodiment, and the log anomaly detection model training device 100 may have more than Figure 13 The components shown may have more or fewer components, two or more components may be combined, or different configurations of components may be implemented.

[0268] The log anomaly detection model training device 100 provided in this application can be applied to various computing devices such as cloud servers, personal computers, and terminal devices, and can also be applied to a computing device cluster including at least two computing devices. The following description takes its application to one computing device as an example.

[0269] See Figure 14 , Figure 14 This is a schematic diagram of the structure of a computing device 200 provided in this application. The computing device 200 includes a processor 210, a memory 220, and a communication interface 230, wherein the processor 210, the memory 220, and the communication interface 230 are interconnected via a bus 240.

[0270] The processor 210 can read the program code (including instructions) stored in the memory 220, execute the program code stored in the memory 220, so that the computing device 200 executes the steps in the log anomaly detection model training method provided in the above method embodiments, or so that the computing device 200 deploys the log anomaly detection model training device 100.

[0271] The processor 210 can be implemented in various ways. For example, it can be a central processing unit (CPU), a graphics processing unit (GPU), or a single-core or multi-core processor. The processor 210 can be a combination of a CPU and hardware chips. These hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. The processor 210 can also be implemented using logic devices with built-in processing logic, such as FPGAs or DSPs.

[0272] The memory 220 can store program code and program data. The program code includes the code of the acquisition module 110 and the code of the training module 120, etc., and the program data includes the first log sample set, the second log sample set, the word sequence before masking, and the word sequence after masking, etc.

[0273] In practical applications, memory 220 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 220 can also be volatile memory, which can be random access memory (RAM) used as an external cache.

[0274] The communication interface 230 can be a wired interface (e.g., an Ethernet interface) or a wireless interface (e.g., a cellular network interface or a wireless LAN interface) for communicating with other computing nodes or devices. When the communication interface 230 is a wired interface, it can employ a protocol family based on Transmission Control Protocol / Internet Protocol (TCP / IP), such as Remote Function Call (RFC), Simple Object Access Protocol (SOAP), Simple Network Management Protocol (SNMP), Common Object Request Broker Architecture (CORBA), and distributed protocols, etc.

[0275] Bus 240 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 240 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0276] The computing device 200 described above is used to execute the method described in the above-described log anomaly detection model training method embodiment. It belongs to the same concept as the above-described method embodiment, and its specific implementation process is detailed in the above-described method embodiment, which will not be repeated here.

[0277] The functional modules of the computing device 200 deployment log anomaly detection model training device 100 are described in the following reference: Figure 13 The illustrated device embodiment.

[0278] It should be understood that the computing device 200 is merely one example provided in the embodiments of this application, and the computing device 200 may have more than Figure 14 The components shown may have more or fewer components, two or more components may be combined, or different configurations of components may be implemented.

[0279] This application also provides a non-transient computer-readable storage medium storing instructions that, when executed, can implement some or all of the steps of the log anomaly detection model training method described in the above embodiments.

[0280] This application also provides a computer program product that, when read and executed by a computer, can implement some or all of the steps of the log anomaly detection model training method described in the above method embodiments.

[0281] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0282] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium, etc.

[0283] The steps in the method of this application embodiment can be adjusted, merged, or deleted in order according to actual needs; the units in the device of this application embodiment can be divided, merged, or deleted according to actual needs.

[0284] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / mask> < / mask> < / mask> < / pad> < / pad> < / flag> < / number> < / done> < / check> < / cls> < / pad> < / pad> < / flag> < / number> < / done> < / check> < / cls> < / flag> < / number> < / done> < / check> < / cls> < / flag> < / number> < / done> < / check> < / cls> < / flag> < / number> < / done> < / check> < / cls> < / flag> < / number> < / done> < / check> < / flag> < / number> < / done> < / check> < / flag> < / done> < / check> < / flag> < / done> < / check> < / flag> < / done> < / check> < / cls> < / mask> < / mask> < / mask> < / cls> < / cls> < / cls> < / cls>

Claims

1. A method for training a log anomaly detection model, characterized in that, The method includes: Obtain a first log sample set, wherein the first log sample set is obtained by processing the log data of the target object; the first log sample set includes m log samples, where m is a natural number greater than 1; Each of the m log samples is segmented into words to obtain a sequence of m words corresponding to the m log samples; Each word in the m word sequence is masked according to a predetermined proportion to obtain a masked m word sequence; Obtain the word embedding vector and position embedding vector corresponding to each word in the m word sequences after masking, wherein the word embedding vector corresponding to each word is a multi-dimensional vector used to represent each word, and the position embedding vector corresponding to each word represents the position of each word in its word sequence; Based on the word embedding vector and position embedding vector corresponding to each word in the masked m-word sequence, obtain the m first row vectors corresponding to the masked m-word sequence; The m first row vectors are respectively input into the initial log anomaly detection model for training to obtain m second row vectors. The m second row vectors have a one-to-one correspondence with the m word sequences after masking. Each of the m second row vectors includes the semantic information of the word sequence after masking. The loss from the m second row vectors to the initial cluster centers is obtained; the formula for obtaining the loss from the m second row vectors to the initial cluster centers is: ； in, Let c represent the i-th second row vector among the m second row vectors, and let c represent the initial cluster center. This represents the loss from the i-th second row vector to the initial cluster center, where i is a natural number; Based on the loss from the m second row vectors to the initial cluster center, train the initial log anomaly detection model to obtain the pre-trained log anomaly detection model and the target cluster center; Obtain a second log sample set, wherein the second log sample set is obtained by processing the log data of the target sub-object, and the target sub-object belongs to the target object; The pre-trained log anomaly detection model is fine-tuned using the second log sample set to obtain a trained log anomaly detection model.

2. The method according to claim 1, characterized in that, The target object includes at least one of the following sub-objects: hard disk, memory, flash memory, network device, and processor, wherein the target sub-object is any type of sub-object among the target objects.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the percentiles of the losses from the m second row vectors to the target cluster center; The classification threshold is determined based on the percentiles corresponding to the losses from the m second row vectors to the target cluster center. The classification threshold is used by the trained log anomaly detection model to perform anomaly detection on the log to be detected and obtain the detection result.

4. A log anomaly detection model training device, characterized in that, The device includes: The acquisition module is used to acquire a first log sample set, wherein the first log sample set is obtained by processing the log data of the target object; the first log sample set includes m log samples, where m is a natural number greater than 1; The training module is used to segment the m log samples into words to obtain m word sequences corresponding to the m log samples; to mask a predetermined proportion of words in the m word sequences to obtain masked m word sequences; to obtain the word embedding vector and position embedding vector corresponding to each word in the masked m word sequences, wherein the word embedding vector is a multi-dimensional vector representing each word, and the position embedding vector represents the position of each word in its word sequence; to obtain m first row vectors corresponding to the masked m word sequences based on the word embedding vector and position embedding vector corresponding to each word in the masked m word sequences; and to input the m first row vectors into the initial log anomaly detection model for training to obtain m second row vectors, wherein the m second row vectors have a one-to-one correspondence with the masked m word sequences, and each of the m second row vectors includes the semantic information of the masked word sequence it corresponds to. The loss from the m second row vectors to the initial cluster centers is obtained; the formula for obtaining the loss from the m second row vectors to the initial cluster centers is: ； in, Let c represent the i-th second row vector among the m second row vectors, and let c represent the initial cluster center. This represents the loss from the i-th second row vector to the initial cluster center, where i is a natural number; Based on the loss from the m second row vectors to the initial cluster center, train the initial log anomaly detection model to obtain the pre-trained log anomaly detection model and the target cluster center; The acquisition module is further configured to acquire a second log sample set, wherein the second log sample set is obtained by processing the log data of the target sub-object, and the target sub-object belongs to the target object; The training module is further configured to fine-tune the pre-trained log anomaly detection model using the second log sample set to obtain a trained log anomaly detection model.

5. The apparatus according to claim 4, characterized in that, The target object includes at least one of the following sub-objects: hard disk, memory, flash memory, network device, and processor, wherein the target sub-object is any type of sub-object among the target objects.

6. The apparatus according to claim 4, characterized in that, The training module is also used for: Obtain the percentiles of the losses from the m second row vectors to the target cluster center; The classification threshold is determined based on the percentiles corresponding to the losses from the m second row vectors to the target cluster center. The classification threshold is used by the trained log anomaly detection model to perform anomaly detection on the log to be detected and obtain the detection result.

7. A non-transient computer-readable storage medium, characterized in that, The non-transient computer-readable medium stores instructions for implementing the method according to any one of claims 1 to 3.

8. A computing device, characterized in that, The computing device includes a processor and a memory; the processor is configured to execute instructions stored in the memory, causing the computing device to implement the method according to any one of claims 1 to 3.

Images