Log anomaly detection model training method, log anomaly detection method and system

Abstract

The present invention provides a log anomaly detection model training method, a log anomaly detection method and a system, which belong to the field of big data platform anomaly detection technology. The log anomaly detection model is trained, and the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch; the input of the self-knowledge distillation branch is the feature map of each independent convolution layer, and a transverse convolution layer is used to perform convolution on the convolution feature map. The transverse layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation; two layers of fully connected layers with activation functions are used to classify log events. The present invention fully learns log information representation through a dual-branch structure of self-knowledge distillation, extracts the spatiotemporal logic between logs, enhances the generalization ability of the model, and makes the model more robust to complex and unstable data sets in the real world, while ensuring the log anomaly detection ability of the model and the accuracy of anomaly detection.

Description

Technical Field

[0001] The present invention relates to the technical field of big data platform anomaly detection, and in particular to a big data platform log anomaly detection model training method based on self-knowledge distillation, a log anomaly detection method and a system. Background Art

[0002] At present, many domestic and foreign researchers have conducted in-depth research on log anomaly detection tasks. From the overall perspective of log anomaly detection tasks, it can be divided into three types of methods: statistical analysis-based, machine learning-based, and deep learning-based. Among them, statistical-based log anomaly detection methods include statistical analysis methods such as log error keyword retrieval and log statement template matching. The ability to identify log anomalies is weak, and it is difficult to effectively identify potential log abnormal behaviors such as system deadlocks and process sequence abnormalities, and the application effect in actual production is poor.

[0003] Log anomaly detection methods based on machine learning usually adopt the mode of supervised learning and offline detection. In the training stage, relevant features in the log sequence are manually extracted, such as log key quantity features and order features, and combined to construct an input vector; the model output is set as a two-dimensional vector, and the two dimensions represent the classification flags of normal and abnormal logs respectively. Machine learning methods mainly include decision trees, support vector machines, and multi-layer perceptrons. Compared with machine learning, log anomaly detection methods based on deep learning can significantly improve the detection and identification capabilities of abnormal log behaviors, and have more flexible application scenarios and richer anomaly detection strategies. It is the mainstream research trend in the current field of log anomaly detection.

[0004] The existing big data platform log anomaly detection methods are mainly divided into statistical anomaly detection methods, machine learning anomaly detection methods and deep learning anomaly detection methods. Although statistical log anomaly detection methods do not require a large amount of data drive, training and learning, the generalization ability of this type of method is not strong, and there are problems such as unclear function meaning. Traditional machine learning methods have better performance than statistical methods, but there is still the problem of inaccurate feature extraction. Some machine learning methods, such as support vector machines (SVM) and hidden Markov models (HMM), have demonstrated effective feature extraction and anomaly detection capabilities, but their anomaly detection accuracy still has a lot of room for improvement.

[0005] Existing technologies use more deep learning methods, which solve the problems existing in traditional statistical learning methods and machine learning methods. The log anomaly detection method based on deep learning can automatically extract features and efficiently characterize log features. Among them, Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN) and Convolutional Neural Networks (CNN) are widely used in log anomaly detection. These methods show high accuracy in log anomaly detection.

[0006] Although existing log-based anomaly detection methods have achieved significant improvements in accuracy, these methods still have certain limitations in terms of robustness of anomaly detection involving complex log data in the real world. First, log statements are constantly changing. If we only use previous log events as the standard to judge whether an unknown log event is abnormal, this will lead to incorrect judgments of anomalies. At the same time, there is a lot of noise in real-world logs. For example, during the log collection process, the recorded logs often have duplications, missing, or other records that do not match the actual log events. These two major problems with log data lead to the instability of the log data analysis model, thereby reducing the effectiveness and robustness of existing log-based anomaly detection methods.

[0007] In summary, the current deep learning-based log anomaly detection methods focus more on improving accuracy, but generally have problems such as the extracted features are not general enough, the generalization ability is not strong, the robustness is poor, and it is difficult to deal with unstable log data. Summary of the invention

[0008] The purpose of the present invention is to provide a big data platform log anomaly detection model training method, log anomaly detection method and system based on self-knowledge distillation. Through the dual-branch structure of self-knowledge distillation, the log information representation is fully learned, the spatiotemporal logic between logs is extracted, and the generalization ability of the model is enhanced, so that the model has strong robustness for complex and unstable data sets in the real world. At the same time, the log anomaly detection capability of the model is guaranteed, and the accuracy of anomaly detection is guaranteed, so as to solve at least one technical problem existing in the above-mentioned background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] On the one hand, the present invention provides a big data platform log anomaly detection model training method, comprising:

[0011] Acquire training data; the training data includes a plurality of log events and log features that label the log events as abnormal;

[0012] The plurality of log events are converted into structured digital data as input of a convolutional neural network, and a log anomaly detection model is trained based on log features of the label log events as output of the convolutional neural network; the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch;

[0013] The anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer;

[0014] The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolutional layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events.

[0015] Furthermore, in the attention layer, the three outputs of the maximum pooling layer are used for attention operation; the Hadamard product is used for element-wise multiplication between the convolution output and the attention weight; after the product, features are extracted according to the importance between the three output feature maps; finally, the weighted attention output is obtained; wherein, the dropout operation is used to regularize the model; finally, after the two fully connected layers, the softmax layer is used as the output layer for binary classification.

[0016] Furthermore, each independent convolutional layer C 1 , ..., C n The feature map represents n layers of convolution operations; the horizontal convolution layer is used to perform convolution on the convolution feature map:

[0017] F i =Convolution(C i ;c i , o i ),

[0018] Among them, Convolution (·) is based on C i For input, F i It is the horizontal convolution operation of the output; according to the feature map channel c i and the output size o i Set the input size, i.e. o i =c i × i , where w i is the channel width parameter.

[0019] Furthermore, self-knowledge distillation includes feature knowledge distillation and soft label knowledge distillation; feature distillation uses attention transfer to calculate the difference between the feature map of the anomaly detection branch and the feature map of the self-knowledge distillation branch;

[0020] Characteristic distillation loss L F for:

[0021]

[0022] in, Implemented channel pooling function and L2 normalization;

[0023] The soft label distillation loss is calculated as follows:

[0024]

[0025] Among them, the anomaly detection branch soft label p d And the soft labels ps of the self-knowledge distillation branch are:

[0026]

[0027] where D(·) and S(·) represent the anomaly detection branch and the self-knowledge distillation branch, respectively, and T represents the temperature coefficient.

[0028] Furthermore, both the anomaly detection branch and the self-knowledge distillation branch use binary cross entropy loss as the loss function to measure the difference between the actual situation and the prediction;

[0029] The total loss function L of the log anomaly detection model is:

[0030]

[0031] Among them, α and β are the hyperparameters of knowledge distillation, x represents the true value; y d and s denote the predicted values ​​of the anomaly detection branch and the self-knowledge distillation branch respectively; L CE (x,y d ) represents the loss function of the anomaly detection branch, L CE (x,y s ) represents the loss function of the self-knowledge distillation branch.

[0032] In a second aspect, the present invention provides a big data platform log anomaly detection model training system, comprising:

[0033] An acquisition module is used to acquire training data; the training data includes a plurality of log events and log features that label log events as abnormal;

[0034] A training module, used to convert the multiple log events into structured digital data as input of a convolutional neural network, and train a log anomaly detection model based on the log features of the label log events as output of the convolutional neural network; the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch;

[0035] The anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer;

[0036] The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolutional layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events.

[0037] In a third aspect, the present invention provides a method for detecting anomalies in logs of a big data platform, comprising:

[0038] Get the log events to be detected;

[0039] The log event to be detected is preprocessed to obtain structured digital data, which is input into the log anomaly detection model for processing to obtain a result of whether the log event is abnormal; wherein the log anomaly detection model is obtained by the above-mentioned model training method.

[0040] In a fourth aspect, the present invention provides a big data platform log anomaly detection system, comprising:

[0041] The acquisition module is used to obtain the log events to be detected;

[0042] The detection module is used to pre-process the log event to be detected to obtain structured digital data and input it into the log anomaly detection model for processing to obtain the result of whether the log event is abnormal; wherein the log anomaly detection model is obtained by the model training method described above.

[0043] In a fifth aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the big data platform log anomaly detection method based on self-knowledge distillation as described above is implemented.

[0044] In a sixth aspect, the present invention provides a computer program product, including a computer program, which, when running on one or more processors, is used to implement the big data platform log anomaly detection method based on self-knowledge distillation as described above.

[0045] In the seventh aspect, the present invention provides an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the big data platform log anomaly detection method based on self-knowledge distillation as described above.

[0046] Terminology explanation:

[0047] Knowledge distillation: A deep learning model structure that improves the performance of a student model without changing its structure by guiding a lightweight student model to learn a teacher model with better performance and more complex structure. Typically, the complex model is called the "teacher model" and the simplified model is called the "student model."

[0048] Self-knowledge distillation: The teacher model and the student model are one model, and the model learns its own knowledge and further improves its performance.

[0049] Big data platform: Big data platform refers to a technical foundation platform for storing, processing and analyzing large-scale data. It is an integrated system that can process data of various types, sources and sizes, and provide powerful computing and analysis capabilities to support the needs of enterprises or organizations in data-driven decision-making, business optimization and innovation. Popular open source big data platforms include Apache Hadoop, Spark, etc.

[0050] Neural Network: A neural network is a computational model inspired by the human nervous system for simulating and solving complex problems. It consists of a large number of artificial neurons (or nodes) that are connected to each other through connection weights to form a network. Each neuron receives input from other neurons, processes the input through an activation function, and then passes the output to other neurons.

[0051] Log constant: The fixed part of each log line in the log file, namely the log key (Logkey).

[0052] Log sequence: refers to a line of log after preprocessing.

[0053] The beneficial effects of the present invention are as follows: for complex data sets, the self-knowledge distillation mechanism is combined with the convolutional neural network for anomaly detection of log sequences, and the feature distillation and soft label distillation are used to enable the anomaly detection model to fully obtain the potential information in the log, so as to obtain a log anomaly detection model that takes both robustness and accuracy into account; the big data log anomaly detection model based on self-knowledge distillation is an offline detection model that utilizes the redundant logs generated by the big data platform. Compared with traditional methods, it combines the advantage of convolutional neural networks that can capture more information; it combines the self-knowledge distillation mechanism, utilizes feature distillation and soft label knowledge distillation to cope with complex real-world data sets, and takes into account robustness, accuracy and stability.

[0054] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0056] Figure 1 This is a flow chart of the big data platform log anomaly detection method based on self-knowledge distillation described in an embodiment of the present invention.

[0057] Figure 2 Schematic diagram of the dual-branch anomaly detection model architecture described in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0059] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0060] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0061] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0062] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0063] Example 1

[0064] In this embodiment 1, a big data platform log anomaly detection model training system is first provided, including: an acquisition module, used to acquire training data; the training data includes multiple log events and log features of the label log events as abnormalities; a training module, used to convert the multiple log events into structured digital data as input of a convolutional neural network, and train a log anomaly detection model based on the log features of the label log events as output of the convolutional neural network; the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch;

[0065] The anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer;

[0066] The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolutional layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events.

[0067] In this embodiment 1, the above-mentioned system is used to implement a big data platform log anomaly detection model training method to obtain a big data platform log anomaly detection model, the method comprising: using an acquisition module to acquire training data; the training data comprises multiple log events and log features of label log events as abnormalities; using a detection module to convert the multiple log events into structured digital data as input of a convolutional neural network, and based on the log features of the label log events as outputs of the convolutional neural network, training a log anomaly detection model; the log anomaly detection model comprises an anomaly detection branch and a self-knowledge distillation branch.

[0068] Among them, the anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer.

[0069] The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolutional layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events.

[0070] In the attention layer, the three outputs of the maximum pooling layer are used for attention operation; the Hadamard product is used to perform element-wise multiplication between the convolution output and the attention weight; after the product, features are extracted according to the importance between the three output feature maps; finally, the weighted attention output is obtained; among them, the dropout operation is used to regularize the model; finally, after the two fully connected layers, the softmax layer is used as the output layer for binary classification.

[0071] Each independent convolutional layer C 1 , ..., C n The feature map represents n layers of convolution operations; the horizontal convolution layer is used to perform convolution on the convolution feature map:

[0072] F i =Convolution(C i ;c i , o i ),

[0073] Among them, Convolution (·) is based on C i For input, F i It is the horizontal convolution operation of the output; according to the feature map channel c i and the output size o i Set the input size, i.e. o i=c i × i , where w i is the channel width parameter.

[0074] Self-knowledge distillation includes feature knowledge distillation and soft label knowledge distillation; feature distillation uses attention transfer to calculate the difference between the feature map of the anomaly detection branch and the feature map of the self-knowledge distillation branch;

[0075] Characteristic distillation loss L F for:

[0076]

[0077] in, Implemented channel pooling function and L2 normalization;

[0078] The soft label distillation loss is calculated as follows:

[0079]

[0080] Among them, the anomaly detection branch soft label p d And the soft labels ps of the self-knowledge distillation branch are:

[0081]

[0082] where D(·) and S(·) represent the anomaly detection branch and the self-knowledge distillation branch, respectively, and T represents the temperature coefficient.

[0083] Both the anomaly detection branch and the self-knowledge distillation branch use binary cross entropy loss as the loss function to measure the difference between the actual situation and the prediction;

[0084] The total loss function L of the log anomaly detection model is:

[0085]

[0086] Among them, α and β are the hyperparameters of knowledge distillation, x represents the true value; y d and s denote the predicted values ​​of the anomaly detection branch and the self-knowledge distillation branch respectively; L CE (x,y d ) represents the loss function of the anomaly detection branch, L CE (x,y s ) represents the loss function of the self-knowledge distillation branch.

[0087] Example 2

[0088] In this embodiment 2, firstly, a big data platform log anomaly detection system is provided, including: an acquisition module for acquiring log events to be detected; a detection module for preprocessing the log events to be detected to obtain structured digital data, inputting the structured digital data into the log anomaly detection model for processing, and obtaining a result of whether the log event is abnormal. A big data platform log anomaly detection method is implemented by using the system, including: using the acquisition module to acquire the log events to be detected; using the detection module to preprocess the log events to be detected to obtain structured digital data, inputting the structured digital data into the log anomaly detection model for processing, and obtaining a result of whether the log event is abnormal.

[0089] In this embodiment, the log anomaly detection model training method includes:

[0090] Acquire training data; the training data includes a plurality of log events and log features that label the log events as abnormal;

[0091] The plurality of log events are converted into structured digital data as input of a convolutional neural network, and a log anomaly detection model is trained based on log features of the label log events as output of the convolutional neural network; the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch;

[0092] The anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer;

[0093] The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolutional layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events.

[0094] In the attention layer, the three outputs of the maximum pooling layer are used for attention operation; the Hadamard product is used to perform element-wise multiplication between the convolution output and the attention weight; after the product, features are extracted according to the importance between the three output feature maps; finally, the weighted attention output is obtained; among them, the dropout operation is used to regularize the model; finally, after the two fully connected layers, the softmax layer is used as the output layer for binary classification.

[0095] Each independent convolutional layer C 1 , ..., C nThe feature map represents n layers of convolution operations; the horizontal convolution layer is used to perform convolution on the convolution feature map:

[0096] F i =Convolution(C i ;c i , o i ),

[0097] Among them, Convolution (·) is based on C i For input, F i It is the horizontal convolution operation of the output; according to the feature map channel c i and the output size o i Set the input size, i.e. o i =c i × i , where w i is the channel width parameter.

[0098] Self-knowledge distillation includes feature knowledge distillation and soft label knowledge distillation; feature distillation uses attention transfer to calculate the difference between the feature map of the anomaly detection branch and the feature map of the self-knowledge distillation branch;

[0099] Characteristic distillation loss L F for:

[0100]

[0101] in, Implemented channel pooling function and L2 normalization;

[0102] The soft label distillation loss is calculated as follows:

[0103]

[0104] Among them, the anomaly detection branch soft label p d And the soft labels ps of the self-knowledge distillation branch are:

[0105]

[0106] where D(·) and S(·) represent the anomaly detection branch and the self-knowledge distillation branch, respectively, and T represents the temperature coefficient.

[0107] Both the anomaly detection branch and the self-knowledge distillation branch use binary cross entropy loss as the loss function to measure the difference between the actual situation and the prediction;

[0108] The total loss function L of the log anomaly detection model is:

[0109]

[0110] Among them, α and β are the hyperparameters of knowledge distillation, x represents the true value; y d and s denote the predicted values ​​of the anomaly detection branch and the self-knowledge distillation branch respectively; L CE (x,y d ) represents the loss function of the anomaly detection branch, L CE (x,y s ) represents the loss function of the self-knowledge distillation branch.

[0111] Example 3

[0112] In view of the fact that existing log anomaly detection methods are difficult to cope with the ever-changing complex logs in the real world, and the collected big data platform logs often contain a lot of noise, in this Example 3, a log anomaly detection method based on self-knowledge distillation is proposed to explore key issues such as how to improve the model's feature extraction capability for real-world noisy data sets, and how to improve the model's ability to learn the potential representation of unstable logs through the self-distillation mechanism, and ensure the model's generalization ability, so that the model can still effectively extract potential features in complex real-world unstable data sets, so that the model can achieve high accuracy and high robustness.

[0113] The log anomaly detection method based on self-knowledge distillation for big data platform logs proposed in this embodiment combines convolutional neural network with self-knowledge distillation to achieve high-precision and high-robustness anomaly detection for big data platform logs. Figure 1 As shown in the figure, the whole process can be divided into three parts: log preprocessing, anomaly detection and detection results.

[0114] In the data preprocessing module, after obtaining the original log data set through the big data platform, the text-type log data needs to be preprocessed and converted into structured digital data so that it can be correctly recognized and input by the neural network. For the defined data structure, we divide each log entry into variable and constant parts. Among them, the log variable part, also known as the log value, contains information such as IP address, timestamp, file ID, etc. The log constant part, also known as the log template, is a series of fixed frequent text strings. Log preprocessing is carried out in the following steps. First, log parsing is performed on the log sequence, and the template parser is used to parse the log file. The log template parser separates the log variable value while retaining the log constant. Next, a unique number is designed for each log template to encode each log template. This unique number is called logkey. For example, there are 299 log keys in the Hadoop application log, and we map them to 299 unique numbers. All unique log keys are calculated based on the entire data set, and each unique log template corresponds to a unique number. Then, these log key sequences are divided into log sessions using the log block id corresponding to each log sequence. After log preprocessing, a series of log sequences are obtained. In each log session, there is a series of related log sequences. Therefore, the length of each session may be different. To solve this configuration problem, longer vectors are pruned and shorter vectors are filled with zeros to ensure the same length of log vectors.

[0115] In the anomaly detection module, a dual-branch anomaly detection model is used to perform log anomaly detection.

[0116] One of the branches is: using a convolutional neural network model combined with an attention mechanism as an anomaly detection model, which mainly contains three main types of layers, word embedding layer, convolution layer and attention layer. In the embedding layer, after preprocessing the original log, the log vector is converted into a learnable codebook matrix. The logkey2vec word embedding method is used to convert the original log sequence into a learnable matrix. Logkey2vec is trained and optimized by stochastic gradient descent during the training phase. The codebook of logkey2vec is used to change the dimension of the input vector. Then, the matrix is ​​input into the convolution layer as an input data set to learn the potential relationship in the log. In the convolution layer, the features of the log vector embedding matrix are extracted through three independent convolutions. In addition, the maximum pooling layer is added after the convolution layer, which can effectively alleviate overfitting. It can filter out irrelevant features and retain relevant features. In the attention layer, the three outputs of the maximum pooling layer are used for attention operation. The Hadamard product is used to perform element-by-element multiplication between the convolution output and the attention weight. After the product, the features are extracted more effectively according to the importance between the three output feature maps. Finally, the weighted attention output is obtained. Next, a dropout operation is used to regularize the model to prevent overfitting. Finally, after the two fully connected layers, a softmax layer is used as the output layer for binary classification.

[0117] The second branch is: an additional branch is added to the improved convolutional neural network model as a self-teacher network. The anomaly detection model learns the knowledge in the self-teacher model to improve accuracy and generalization ability. The self-teacher model uses self-knowledge distillation, which is mainly reflected in the self-teacher model in the dual-branch structure. Its structure is as follows Figure 2 As shown in Figure 2. The input of the self-teacher model is the feature map of each independent convolutional layer C1, ..., Cn, representing n layers of convolution operations. The self-teacher model uses a horizontal convolution layer to perform convolution on the convolution feature map. Unlike the common convolution layer, the horizontal layer does not change the dimension of the feature map to preserve the mined log information as much as possible. The horizontal convolution operation is expressed as:

[0118] F i =Convolution(C i ;c i , o i ),

[0119] Among them, Convolution(·) is a lateral convolution operation with Ci as input and Fi as output. In particular, in order to handle feature maps of different sizes, the lateral convolution operation does not limit the dimension of the input feature map. Instead, it sets the input size according to the feature map channel ci and the output size oi, that is, oi = ci × wi, where wi is the channel width parameter. Afterwards, like the anomaly detection model, the self-teacher model uses leaky ReLU as the activation function. Next, the optimized feature maps are connected through the maximum pooling operation. Finally, two layers of fully connected layers with softmax activation function are used to classify log events. In particular, the soft labels ps generated by the self-teacher model are retained to prepare for the subsequent calculation of the loss.

[0120] The self-knowledge distillation used in this embodiment includes two parts: feature knowledge distillation and soft label knowledge distillation. On the one hand, for feature distillation, the training model uses the features from the detection model and the self-teacher model. Figure 1 Feature distillation uses attention transfer to calculate the difference between the detection model feature map and the self-taught feature map. Feature distillation loss L F Given by:

[0121]

[0122] in, The channel pooling function and L2 normalization are implemented. The loss of feature distillation enables the detection network to obtain the latent log information of the feature map from the self-teacher model.

[0123] On the other hand, similar to other self-knowledge distillation methods, the model also uses soft label distillation, and uses KL (Kullback-Leibler) divergence to measure the difference between the two. The soft label distillation loss is calculated as follows:

[0124]

[0125] Among them, the detection model soft label pd and the self-teaching model soft label ps are expressed as:

[0126]

[0127] Where D(·) and S(·) represent the detection model and the self-teaching model, respectively, and T represents the temperature coefficient. In addition, both the detection model and the self-teaching model use Binary Cross Entropy (BCE) as the loss function to measure the difference between the actual situation and the prediction. Finally, the above four loss functions are combined into the loss function L, which can be expressed as:

[0128]

[0129] Where α and β are the hyperparameters of knowledge distillation, x represents the true value, yd and ys represent the predicted values ​​of the detection model and the self-teaching model respectively; L CE (x,y d ) represents the loss function of the anomaly detection branch, L CE (x,y s ) represents the loss function of the self-knowledge distillation branch. It should be noted that the self-teacher model is only used to train the detection model, and it is only used in the training phase to pass more potential information to the detection model. Although the anomaly detection model is extended with an additional branch, its impact on the anomaly detection efficiency is negligible.

[0130] Example 4

[0131] This embodiment 4 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned big data platform log anomaly detection method based on self-knowledge distillation is implemented. The method includes:

[0132] Get the log events to be detected;

[0133] The log event to be detected is preprocessed to obtain structured digital data, which is input into the log anomaly detection model for processing to obtain a result of whether the log event is abnormal; wherein the log anomaly detection model is trained by the model training method described in the above embodiment.

[0134] Example 5

[0135] This embodiment 5 provides a computer program product, including a computer program. When the computer program is executed on one or more processors, it is used to implement the big data platform log anomaly detection method based on self-knowledge distillation as described above. The method includes:

[0136] Get the log events to be detected;

[0137] The log event to be detected is preprocessed to obtain structured digital data, which is input into the log anomaly detection model for processing to obtain a result of whether the log event is abnormal; wherein the log anomaly detection model is trained by the model training method described in the above embodiment.

[0138] Example 6

[0139] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the above-mentioned big data platform log anomaly detection method based on self-knowledge distillation, the method comprising:

[0140] Get the log events to be detected;

[0141] The log event to be detected is preprocessed to obtain structured digital data, which is input into the log anomaly detection model for processing to obtain a result of whether the log event is abnormal; wherein the log anomaly detection model is trained by the model training method described in the above embodiment.

[0142] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. A big data platform log anomaly detection model training method, characterized in that: include: Acquire training data; the training data includes a plurality of log events and log features that label the log events as abnormal; The plurality of log events are converted into structured digital data as input of a convolutional neural network, and a log anomaly detection model is trained based on log features of the label log events as output of the convolutional neural network; the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch; The anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer; The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolution layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events. Self-knowledge distillation includes feature knowledge distillation and soft label knowledge distillation. Feature distillation uses attention transfer to calculate the difference between the feature map of the anomaly detection branch and the feature map of the self-knowledge distillation branch. Characteristic distillation loss L F for: in, Channel pooling function and L2 normalization are implemented; each independent convolution layer C1,…,C n The feature map represents n layers of convolution operations; the horizontal convolution layer is used to perform convolution on the convolution feature map: F i =Convolution(C i ;c i , o i ), where Convolution(·) is based on C i For input, F i It is the horizontal convolution operation of the output; according to the feature map channel c i and the output size o i Set the input size, i.e. o i =c i × i , where w i is the channel width parameter; The soft label distillation loss is calculated as follows: Among them, the anomaly detection branch soft label p d And the soft labels ps of the self-knowledge distillation branch are: where D(·) and S(·) represent the anomaly detection branch and the self-knowledge distillation branch, respectively, and T represents the temperature coefficient.

2. The big data platform log anomaly detection model training method according to claim 1 is characterized in that: In the attention layer, the three outputs of the maximum pooling layer are used for attention operation; the Hadamard product is used to perform element-wise multiplication between the convolution output and the attention weight; after the product, features are extracted based on the importance between the three output feature maps; Finally, the weighted attention output is obtained; the dropout operation is used to regularize the model; finally, after the two fully connected layers, the softmax layer is used as the output layer for binary classification.

3. The big data platform log anomaly detection model training method according to claim 1 is characterized in that: Both the anomaly detection branch and the self-knowledge distillation branch use binary cross entropy loss as the loss function to measure the difference between the actual situation and the prediction; The total loss function L of the log anomaly detection model is: Among them, α and β are the hyperparameters of knowledge distillation, x represents the true value; y d and s denote the predicted values ​​of the anomaly detection branch and the self-knowledge distillation branch respectively; L CE (x,y d ) represents the loss function of the anomaly detection branch, L CE (x,y s ) represents the loss function of the self-knowledge distillation branch.

4. A big data platform log anomaly detection model training system, characterized in that: include: An acquisition module is used to acquire training data; The training data includes a plurality of log events and log features that label the log events as abnormal; A training module, used to convert the multiple log events into structured digital data as input of a convolutional neural network, and train a log anomaly detection model based on the log features of the label log events as output of the convolutional neural network; the log anomaly detection model includes an anomaly detection branch and a self-knowledge distillation branch; The anomaly detection branch includes a word embedding layer, a convolution layer and an attention layer; the embedding layer is used to convert the log vector obtained after the original log is transformed into a learnable log vector embedding matrix; the convolution layer is used to extract the features of the log vector embedding matrix; the attention layer is used to perform attention operations using the three outputs of the maximum pooling layer; The input of the self-knowledge distillation branch is the feature map of each independent convolutional layer. A horizontal convolution layer is used to perform convolution on the convolutional feature map. The horizontal layer does not change the dimension of the feature map, and the optimized feature map is connected through a maximum pooling operation. Two layers of fully connected layers with activation functions are used to classify log events. Self-knowledge distillation includes feature knowledge distillation and soft label knowledge distillation. Feature distillation uses attention transfer to calculate the difference between the feature map of the anomaly detection branch and the feature map of the self-knowledge distillation branch. Characteristic distillation loss L F for: in, Channel pooling function and L2 normalization are implemented; each independent convolution layer C1,…,C n The feature map represents n layers of convolution operations; the horizontal convolution layer is used to perform convolution on the convolution feature map: F i =Convolution(C i ;c i ,o i ), Among them, Convolution (·) is based on C i For input, F i It is the horizontal convolution operation of the output; according to the feature map channel c i and the output size o i Set the input size, i.e. o i =c i × i , where w i is the channel width parameter; The soft label distillation loss is calculated as follows: Among them, the anomaly detection branch soft label p d And the soft labels ps of the self-knowledge distillation branch are: where D(·) and S(·) represent the anomaly detection branch and the self-knowledge distillation branch, respectively, and T represents the temperature coefficient.

5. A method for detecting anomaly in logs of a big data platform, characterized in that: include: Get the log events to be detected; Preprocess the log event to be detected to obtain structured digital data, and input it into the log anomaly detection model for processing to obtain the result of whether the log event is abnormal; The log anomaly detection model is trained by the model training method described in any one of claims 1 to 3.

6. A big data platform log anomaly detection system, characterized in that: include: The acquisition module is used to obtain the log events to be detected; The detection module is used to pre-process the log event to be detected to obtain structured digital data and input it into the log anomaly detection model for processing to obtain the result of whether the log event is abnormal; The log anomaly detection model is trained by the model training method described in any one of claims 1 to 3.

7. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the big data platform log anomaly detection method as described in claim 5 is implemented.

8. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the big data platform log anomaly detection method as described in claim 5.

Images