Log anomaly detection method and device, equipment, storage medium and program product

Abstract

The invention relates to a log anomaly detection method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a target semantic vector corresponding to a log sequence, determining a time vector corresponding to the log sequence according to a time difference value of timestamps of two adjacent logs in the log sequence, determining a target log vector of the log sequence according to the target semantic vector and the time vector, and sending the target log vector to the log sequence. The method comprises the following steps: performing anomaly detection on a next log of a log sequence based on a target log vector and a log anomaly detection model to obtain a log anomaly detection result, and performing anomaly detection on the next log of the log sequence based on the target log vector and the log anomaly detection model to obtain the log anomaly detection result. Therefore, the accuracy of the log anomaly detection result can be improved.

Description

Technical Field

[0001] This application relates to the technical field of log detection, and particularly to a log anomaly detection method, device, equipment, storage medium, and program product. Background Art

[0002] In today's digital age, cloud-native technology has been booming, bringing efficient, flexible, and scalable IT solutions to enterprises. Based on concepts such as containerization and microservices, cloud-native architecture enables applications to be rapidly iterated and deployed, greatly enhancing the competitiveness of enterprises. Against this background, the IT systems of enterprises have become more complex and changeable. Among them, logs, as an important carrier for recording key data such as system operation status and fault information, have become increasingly prominent in importance. Log data is like the "black box" of system operation, recording various operations, events, and abnormal information, and plays an irreplaceable role in aspects such as system operation and maintenance, fault troubleshooting, and performance optimization. Therefore, it is necessary to analyze log data.

[0003] Traditional log analysis methods are for humans to analyze logs based on set rules to determine log anomaly detection results. However, due to the large number and wide distribution of system components in the cloud-native environment, the log data generated is huge in volume and diverse in format, and with the continuous expansion of business and the continuous growth of system scale, log data shows an explosive growth trend. Using traditional log analysis methods to analyze log data in the cloud-native environment has the problem of low accuracy of log anomaly detection results. Summary of the Invention

[0004] Based on this, it is necessary to provide a log anomaly detection method, device, equipment, storage medium, and program product that can improve the accuracy of log anomaly detection results for the above technical problems.

[0005] In a first aspect, this application provides a log anomaly detection method, including:

[0006] Obtain a target semantic vector corresponding to a log sequence;

[0007] Determine a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence;

[0008] Determine a target log vector of the log sequence according to the target semantic vector and the time vector;

[0009] Based on the target log vector and a log anomaly detection model, perform anomaly detection on the next log in the log sequence to obtain a log anomaly detection result.

[0010] In one of the embodiments, obtaining a target semantic vector corresponding to a log sequence includes:

[0011] Semantically encode the log sequence to obtain semantic vectors;

[0012] Obtain the weights corresponding to the word vectors of each log in the log sequence;

[0013] Based on the semantic vectors and the weights corresponding to the word vectors of each log in the log sequence, obtain the target semantic vectors.

[0014] In one embodiment, based on the target log vector and the log anomaly detection model, perform anomaly detection on the next log of the log sequence to obtain the log anomaly detection result, including:

[0015] Based on the target log vector and the log anomaly detection model, determine the probability distribution of the next log of the log sequence; the probability distribution is determined based on the probabilities that the next log belongs to each preset log phrase in the preset log phrase set;

[0016] Determine the log anomaly detection result according to the probability distribution and the probability threshold.

[0017] In one embodiment, determine the target log vector of the log sequence according to the target semantic vector and the time vector, including:

[0018] Concatenate the target semantic vector and the time vector to obtain the target log vector of the log sequence.

[0019] In one embodiment, the method further includes:

[0020] Obtain the target log vector sample corresponding to the log sequence sample;

[0021] Train the initial log anomaly detection model based on the target log vector sample to obtain the log anomaly detection model.

[0022] In one embodiment, training the initial log anomaly detection model based on the target log vector sample to obtain the log anomaly detection model includes:

[0023] Input the target log vector sample into the bidirectional gated recurrent unit in the initial log anomaly detection model to obtain the output feature sequence of the bidirectional gated recurrent unit;

[0024] According to the output feature sequence and the multi-head attention model in the initial log anomaly detection model, obtain the predicted probability distribution; the predicted probability distribution includes the probabilities that the next log of the log sequence sample belongs to each preset log phrase in the preset log phrase set;

[0025] Train the initial log anomaly detection model according to the predicted probability distribution and the actual next log of the log sequence sample to obtain the log anomaly detection model.

[0026] In a second aspect, the present application further provides a log anomaly detection device, which includes:

[0027] A first acquisition module, configured to acquire a target semantic vector corresponding to a log sequence;

[0028] A first determination module, configured to determine a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence;

[0029] A second determination module, configured to determine a target log vector of the log sequence according to the target semantic vector and the time vector;

[0030] A detection module, configured to perform anomaly detection on the next log of the log sequence based on the target log vector and a log anomaly detection model to obtain a log anomaly detection result.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] Acquire a target semantic vector corresponding to a log sequence;

[0033] Determine a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence;

[0034] Determine a target log vector of the log sequence according to the target semantic vector and the time vector;

[0035] Perform anomaly detection on the next log of the log sequence based on the target log vector and a log anomaly detection model to obtain a log anomaly detection result.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0037] Acquire a target semantic vector corresponding to a log sequence;

[0038] Determine a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence;

[0039] Determine a target log vector of the log sequence according to the target semantic vector and the time vector;

[0040] Perform anomaly detection on the next log of the log sequence based on the target log vector and a log anomaly detection model to obtain a log anomaly detection result.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0042] Obtain the target semantic vector corresponding to the log sequence;

[0043] Determine the time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence;

[0044] Determine the target log vector of the log sequence according to the target semantic vector and the time vector;

[0045] Based on the target log vector and the log anomaly detection model, perform anomaly detection on the next log in the log sequence to obtain the log anomaly detection result.

[0046] The above log anomaly detection method, device, equipment, storage medium and program product, by obtaining the target semantic vector corresponding to the log sequence, determining the time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence, determining the target log vector of the log sequence according to the target semantic vector and the time vector, and based on the target log vector and the log anomaly detection model, performing anomaly detection on the next log in the log sequence to obtain the log anomaly detection result. Since the next log in the log sequence is subjected to anomaly detection based on the target log vector and the log anomaly detection model to obtain the log anomaly detection result, the accuracy of the log anomaly detection result can be improved. Description of the Drawings

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is an internal structure diagram of a computer device provided by an embodiment of the present application;

[0049] Figure 2 It is a flowchart of a log anomaly detection method provided by an embodiment of the present application;

[0050] Figure 3 It is a flowchart of a method for obtaining a target semantic vector provided by an embodiment of the present application;

[0051] Figure 4 It is a flowchart of a method for determining a log anomaly detection result provided by an embodiment of the present application;

[0052] Figure 5 It is a process of splicing the target semantic vector and the time vector to obtain the target log vector provided by an embodiment of the present application;

[0053] Figure 6 is a schematic flowchart of a method for obtaining a log anomaly detection model provided by an embodiment of the present application;

[0054] Figure 7 is a schematic diagram of an initial log anomaly detection model provided by an embodiment of the present application;

[0055] Figure 8 is a schematic flowchart of a method for training a log anomaly detection model provided by an embodiment of the present application;

[0056] Figure 9 is a schematic flowchart of the overall process of the log anomaly detection method provided by an embodiment of the present application;

[0057] Figure 10 is a structural block diagram of a log anomaly detection device provided by an embodiment of the present application. Detailed implementation manners

[0058] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] In today's digital age, cloud-native technologies are booming, bringing efficient, flexible and scalable IT solutions to enterprises. The cloud-native architecture is based on concepts such as containerization and microservices, enabling application programs to be rapidly iterated and deployed, greatly enhancing the competitiveness of enterprises. Against this background, the IT systems of enterprises have become more complex and changeable. Among them, logs, as an important carrier for recording key data such as system operation status and fault information, have become increasingly important. Log data is like the "black box" of system operation, recording various operations, events and anomaly information, and plays an irreplaceable role in aspects such as system operation and maintenance, fault troubleshooting and performance optimization. Therefore, it is necessary to analyze log data.

[0060] Traditional log analysis methods are to manually analyze logs based on set rules to determine log anomaly detection results. However, due to the large number of system components and wide distribution in the cloud-native environment, the generated log data is huge in volume and diverse in format, and with the continuous expansion of business and the continuous increase in system scale, log data shows an explosive growth trend. Using traditional log analysis methods to analyze log data in the cloud-native environment has the problem of low accuracy of log anomaly detection results.

[0061] To solve the above technical problems, an embodiment of the present application provides an abnormal log analysis method, which can be applied to Figure 1 the computer device shown inFigure 1 is an internal structure diagram of a computer device provided by an embodiment of the present application. The computer device may be a terminal, and its internal structure diagram may be as follows Figure 1 shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it realizes a method for detecting log anomalies. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0062] Those skilled in the art can understand that Figure 1 the structure shown in

[0063] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. Figure 2 shown, Figure 2 is a schematic flowchart of a method for detecting log anomalies provided by an embodiment of the present application. The method includes the following steps S201 to S204:

[0064] S201, obtain a target semantic vector corresponding to the log sequence.

[0065] The collected logs can be preprocessed to obtain a log sequence. The preprocessing operations can include screening and filtering the logs with non-standard formats to obtain valid logs, so as to ensure the quality of log data. Subsequently, perform word segmentation on the valid logs, and normalize special words to obtain target logs. For example, convert uppercase letters to lowercase to ensure text consistency, and split compound words to improve the accuracy of log parsing. Use Drain to construct a fixed-depth parsing tree to parse the target logs to obtain the parsed logs, and use the multiple parsed logs as a log sequence. Multiple logs in a log sequence are logs with log order relevance. Exemplarily, a log sequence includes 3 logs. The first log in the log sequence is registration, the second log is login, and the third log is login success.

[0066] The pre-trained language representation model (Bidirectional Encoder Representations from Transformers, BERT) can be used to perform semantic encoding on the log sequence to obtain semantic vectors. BERT is a bidirectional encoder representation based on Transformer. It aims to capture the deep features of language through pre-training on a large corpus of text, and then provide powerful language representations for various natural language processing tasks. Assume the log sequence is , first the logs in the log sequence are first passed through BERT to generate corresponding semantic vectors as , where is the semantic vector of the th log extracted by BERT, the dimension is d, and n is a positive integer. The generated semantic vectors can be used as target semantic vectors. It is also possible to perform weighted processing on the generated semantic vectors to obtain weighted semantic vectors, and use the weighted semantic vectors as target semantic vectors.

[0067] S202. Determine the time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence.

[0068] In a possible implementation, an initial time vector corresponding to the log sequence is obtained based on the time differences corresponding to each log in the log sequence, the initial time vector is normalized to obtain an intermediate time vector corresponding to the log sequence, and the intermediate time vector is mapped to a high-dimensional space to obtain the time vector corresponding to the log sequence.

[0069] In another possible implementation, an initial time vector corresponding to the log sequence is obtained based on the time differences corresponding to each log in the log sequence, and the initial time vector is mapped to a high-dimensional space to obtain the time vector corresponding to the log sequence.

[0070] Micro-services may have problems with abnormal log order. This method enhances the ability to detect the above-mentioned log anomalies by introducing a time vector to capture the time dependencies between logs. Assume that the log sequence is arranged in chronological order as follows: , where represents the content of the th log, and represents the timestamp of the i-th log. The log time difference is calculated as: , where is set to 0, and the time differences of other logs are obtained by subtracting the timestamp of the previous log from the timestamp of the current log. The corresponding time difference is . Based on the time differences corresponding to each log in the log sequence, the initial time vector corresponding to the log sequence is obtained. The initial time vector is defined as . The initial time vector T reflects the time dependencies between logs. Then, the initial time vector is normalized to obtain the intermediate time vector corresponding to the log sequence. After that, the intermediate time vector is mapped to a high-dimensional space through a 1x1 convolution to obtain the time vector corresponding to the log sequence, capturing the non-linear features and potential patterns in the time differences, enabling time information to participate in subsequent model learning in a more complex and detailed manner. Through the time vector, the model can learn the time difference distribution of normal logs and identify the time deviations of abnormal logs.

[0071] S203. Determine the target log vector of the log sequence according to the target semantic vector and the time vector.

[0072] The target semantic vector and the time vector can be concatenated to obtain the target log vector of the log sequence.

[0073] S204. Based on the target log vector and the log anomaly detection model, perform anomaly detection on the next log of the log sequence to obtain the log anomaly detection result.

[0074] The log anomaly detection model can include, but is not limited to, a Convolutional Neural Networks (CNN) model, a Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM) model, etc. It can also be a model combining a Bidirectional Gated Recurrent Unit (Bi-GRU) and a Multi-Head Attention (MHA) mechanism.

[0075] The method provided in this embodiment determines the target time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence by obtaining the target semantic vector corresponding to the log sequence, determines the target log vector of the log sequence according to the target semantic vector and the time vector, and performs anomaly detection on the next log in the log sequence based on the target log vector and the log anomaly detection model to obtain the log anomaly detection result. Since the anomaly detection is performed on the next log in the log sequence based on the target log vector and the log anomaly detection model, the accuracy of the log anomaly detection result can be improved.

[0076] In an exemplary embodiment, as Figure 3 shown, Figure 3 is a schematic flowchart of a method for obtaining a target semantic vector provided by an embodiment of the present application. On the basis of the above embodiment, the above S201 may include S301 to S303:

[0077] S301, perform semantic encoding on the log sequence to obtain a semantic vector.

[0078] S302, obtain the weights corresponding to the word vectors of each log in the log sequence.

[0079] Obtaining the weights corresponding to the word vectors of each log in the log sequence includes the following steps: for each log in the log sequence, obtain the word frequency of the words and sentences in the log in the log sequence; determine the inverse document frequency according to the total number of logs in the log sequence and the number of logs including the words and sentences in the log; use the product of the word frequency and the inverse document frequency as the weight corresponding to the word vector of the log.

[0080] Among them, the word frequency refers to the frequency of a certain word in the log sequence. The higher the word frequency, the more important the word is in the log sequence. The word frequency is equal to the number of times the words and sentences in a log appear divided by the total number of times the words and sentences appear in the log sequence.

[0081] It is possible to determine the sum result of the number of logs including the words and sentences in the log and 1, and determine the ratio of the total number of logs to the sum result, and determine the logarithm of the ratio, and use the logarithm as the inverse document frequency.

[0082] S303, obtain the target semantic vector according to the semantic vector and the weights corresponding to the word vectors of each log in the log sequence.

[0083] For the log , the weight matrix of its word vector can be calculated using the Term Frequency-Inverse Document Frequency (TF-IDF) method , finally, the semantic vector generated by BERT Perform element-wise multiplication with the weight matrix W generated by TF-IDF to obtain the final fused vector: , which expands to , where ⊙ represents element-wise multiplication, that is , and then perform weighted feature fusion, . This fusion method enables the semantic vector of BERT to combine the importance weights of the text calculated by TF-IDF, so that important log words will obtain higher weights, while redundant information will be suppressed, improving the sensitivity of the log anomaly detection model to abnormal logs.

[0084] The method provided in this embodiment effectively highlights the key information in the log, suppresses irrelevant noise, and enhances the discrimination ability of the log anomaly detection model for abnormal logs by performing semantic encoding on the log sequence to obtain a semantic vector, and obtaining the weights corresponding to the word vectors of each log in the log sequence, and obtaining the target semantic vector based on the semantic vector and the weights corresponding to the word vectors of each log in the log sequence, thereby further improving the accuracy of the log anomaly detection result.

[0085] In an exemplary embodiment, as Figure 4 shown, Figure 4 is a schematic flowchart of a method for determining a log anomaly detection result provided by an embodiment of the present application. Based on the above embodiment, the above S204 may include S401 to S402:

[0086] S401, based on the target log vector and the log anomaly detection model, determine the probability distribution of the next log in the log sequence; the probability distribution is determined based on the probability that the next log belongs to each preset log sentence in the preset log sentence set.

[0087] The log sentence set covers all possible log sentences that may appear in the system. For example, if the log sentence set includes log sentences A1 to A20, then the probability that the next log in the log sequence belongs to log sentence A1, the probability that it belongs to log sentence A2,..., the probability that it belongs to log sentence A20 can be obtained, and these 20 probabilities form the probability distribution of the next log in the log sequence.

[0088] The following is an exemplary introduction in combination with the log anomaly detection model of Bi-GRU combined with MHA. After the target log vector is input into Bi-GRU, an output feature sequence is obtained after being processed by Bi-GRU, and then the output feature sequence passes through the first linear layer, scaled dot attention, concatenation, second linear layer and fully connected layer in the MHA model in sequence and then outputs the target feature map, and the Softmax function obtains the probability distribution of the next log in the log sequence based on the target feature map.

[0089] S402, determine the log anomaly detection result according to the probability distribution and the probability threshold.

[0090] If the probability that the next log in the log sequence belongs to the preset log phrase A1 is the highest, it can be determined whether the highest probability is greater than the probability threshold. If the highest probability is greater than the probability threshold, it can be determined that the next log in the log sequence is a normal log; if the highest probability is not greater than the probability threshold, it can be determined that the next log in the log sequence is an abnormal log. Exemplarily, if the log sequence includes the 1st to 10th logs, the 10th log is the last log, and these 10 logs are related logs, then it can be determined whether the 11th log is abnormal according to the probability distribution of the 11th log determined based on S401 and the probability threshold.

[0091] The method provided in this embodiment determines the probability distribution of the next log in the log sequence based on the target log vector and the log anomaly detection model, and determines the log anomaly detection result according to the probability distribution and the probability threshold, thereby improving the accuracy of the log anomaly detection result.

[0092] In an exemplary embodiment, the above S203, determining the target log vector of the log sequence according to the target semantic vector and the time vector, can be implemented in the following manner:

[0093] Concatenate the target semantic vector and the time vector to obtain the target log vector of the log sequence.

[0094] As Figure 5 shown, Figure 5 is a process of obtaining the target log vector by concatenating the target semantic vector and the time vector provided by an embodiment of the present application.

[0095] The method provided in this embodiment forms the target log vector of the log sequence by concatenating the target semantic vector and the time vector, forming a target log vector that simultaneously contains text information and time information. This method not only retains the semantic features of the log, but also enhances the perception ability of time anomalies, improves the detection accuracy of log sequence anomalies and runtime anomalies, thereby improving the accuracy and comprehensiveness of the log anomaly detection result and reducing the probability of missed detection of abnormal logs.

[0096] In an exemplary embodiment, as Figure 6 shown, Figure 6 is a schematic flowchart of a method for obtaining a log anomaly detection model provided by an embodiment of the present application. The method includes the following steps S601 to S602:

[0097] S601, obtain the target log vector sample corresponding to the log sequence sample.

[0098] The method for obtaining the target log vector sample is similar to the method for obtaining the target log vector introduced in the above embodiments, that is, the target log vector sample can be a vector sample obtained by concatenating the semantic vector sample and the time vector sample, which will not be elaborated here.

[0099] S602. Train the initial log anomaly detection model based on the target log vector sample to obtain the log anomaly detection model.

[0100] As Figure 7 shown, Figure 7 is a schematic diagram of an initial log anomaly detection model provided by an embodiment of the present application. If represents the target log vector sample, then an element in the target log vector sample is represented by .

[0101] In the log anomaly detection task, logs are usually recorded in the form of a time series and contain complex context dependencies. Therefore, this method uses a Bidirectional Gated Recurrent Unit (Bi-GRU) to model the log sequence to fully capture the bidirectional dependencies between the front and back logs, and combines a Multi-Head Attention (MHA) mechanism to further improve the model's focusing ability on key log information. The overall structure is as Figure 7 shown. Bi-GRU uses two GRU layers, forward and backward, to simultaneously capture the forward and backward information of the log sequence, thereby improving the ability to model log patterns. The forward GRU can be expressed as , and the backward GRU can be expressed as . The final output of Bi-GRU is , where is the input log vector at time step t, represents the hidden state of the forward GRU unit, respectively represent the hidden states of the backward GRU units, is the final hidden state of Bi-GRU, which is obtained by concatenating the forward and backward hidden states, as Figure 7 shown It forms the output feature sequence of the Bi-GRU. After that, the output feature sequence passes through the first linear layer, scaled dot attention, concatenation, the second linear layer, and the fully connected layer in sequence to output the target feature map. The Softmax function obtains the predicted probability distribution based on the target feature map. According to the predicted probability distribution and the actual next log of the log sequence sample, the initial log anomaly detection model is trained to obtain the log anomaly detection model. The attention mechanism can capture the global features of the log sequence and dynamically adjust the importance of different log events, enabling the model to pay more attention to anomaly events and reducing noise interference. The core of attention calculation is:

[0102]

[0103] Among them, , , , are trainable parameter matrices, A is the output feature sequence of the Bi-GRU, , is the scaling factor of the feature dimension. Combining the Bi-GRU with the multi-head attention mechanism enables the model to have both the ability to model temporal dependencies and the ability to extract global information, and can more accurately identify anomaly patterns in a complex microservice log environment.

[0104] In an exemplary embodiment, as Figure 8 shown, Figure 8 is a schematic flowchart of a method for training a log anomaly detection model provided by an embodiment of the present application. The method includes the following steps S801 to S802:

[0105] S801, Input the target log vector sample into the bidirectional gated recurrent unit in the initial log anomaly detection model to obtain the output feature sequence of the bidirectional gated recurrent unit.

[0106] As Figure 7 shown, input the target log vector sample into the bidirectional gated recurrent unit in the initial log anomaly detection model to obtain the output feature sequence .

[0107] S802, According to the output feature sequence and the multi-head attention model in the initial log anomaly detection model, obtain the predicted probability distribution; the predicted probability distribution includes the probabilities that the next log of the log sequence sample belongs to each log phrase in the log phrase set.

[0108] The log phrase set covers all possible log phrases that may appear in the system. For example, if the log phrase set includes log phrases A1 to A20, then the probabilities that the next log of the log sequence sample belongs to log phrase A1, belongs to log phrase A2,..., belongs to log phrase A20 can be obtained, and these 20 probabilities form the predicted probability distribution.

[0109] S803. Train the initial log anomaly detection model based on the predicted probability distribution and the actual next log in the log sequence sample to obtain the log anomaly detection model.

[0110] For a certain log sample, if the probability that the next log in the log sequence sample belongs to the preset log phrase A1 is the highest, then the preset log phrase A1 can be used as the prediction result of the next log in the log sequence sample, that is, the predicted next log of the log sequence sample includes the preset log phrase A1. Then, based on this prediction result and the actual next log of the log sequence sample, train the initial log anomaly detection model to obtain the log anomaly detection model. Here, the actual next log refers to the next log of the log sequence sample under normal circumstances. Exemplarily, for example, if the log sequence sample includes 10 logs, these 10 logs are context-related logs, and the last log is the 10th log, then the actual next log refers to the 11th log recorded under normal circumstances. If the 10th log is a login, under normal circumstances, the 11th log is a login success or a login failure.

[0111] The method provided in this embodiment, by inputting the target log vector sample into the bidirectional gated recurrent unit in the initial log anomaly detection model, obtains the output feature sequence of the bidirectional gated recurrent unit, inputs the output feature sequence into the multi-head attention model in the initial log anomaly detection model, obtains the log anomaly prediction result, and trains the initial log anomaly detection model based on the log anomaly prediction result and the labels of each log sample in the log sequence sample to obtain the log anomaly detection model, thus laying a foundation for log anomaly detection based on the log anomaly detection model.

[0112] Next, in combination with Figure 9 introduce the overall solution of the log anomaly detection method provided in the embodiments of the present application. Figure 9 is the overall flow schematic diagram of the log anomaly detection method provided in the embodiments of the present application. The method includes the following steps S901 to S909:

[0113] S901. Perform semantic encoding on the log sequence sample to obtain the semantic vector sample.

[0114] S902. Obtain the weights corresponding to the word vectors of each log in the log sequence sample.

[0115] S903. Obtain the target semantic vector sample according to the semantic vector sample and the weights corresponding to the word vectors of each log in the log sequence.

[0116] S904. Determine the time vector sample corresponding to the log sequence sample according to the time difference between the timestamps of two adjacent logs in the log sequence.

[0117] S905. Concatenate the target semantic vector sample and the time vector sample to obtain a target vector sample.

[0118] S906. Bi - GRU is used to extract the features of the target vector sample bidirectionally to obtain an output feature sequence.

[0119] S907. Based on the output feature sequence, a prediction probability distribution is obtained through a multi - head attention model.

[0120] S908. According to the prediction probability distribution and the actual next log of the log sequence sample, train the initial log anomaly detection model until the cross - entropy loss converges or reaches the maximum number of iterations to obtain the log anomaly detection model.

[0121] If it does not converge or does not reach the maximum number of iterations, return to execute S906.

[0122] S909. Perform log anomaly detection based on the log anomaly detection model.

[0123] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0124] Based on the same inventive concept, the embodiments of the present application also provide a log anomaly detection device for implementing the log anomaly detection method involved above. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the log anomaly detection device provided below can refer to the limitations on the log anomaly detection method in the above text and will not be repeated here.

[0125] In an exemplary embodiment, as Figure 10 shown, Figure 10 is a structural block diagram of a log anomaly detection device provided by an embodiment of the present application. The device 1000 includes:

[0126] A first acquisition module 1001, configured to acquire a target semantic vector corresponding to a log sequence;

[0127] The first determination module 1002 is configured to determine a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence;

[0128] The second determination module 1003 is configured to determine a target log vector of the log sequence according to the target semantic vector and the time vector;

[0129] The detection module 1004 is configured to perform anomaly detection on the next log of the log sequence based on the target log vector and the log anomaly detection model to obtain a log anomaly detection result.

[0130] In an exemplary embodiment, the first acquisition module 1001 is specifically configured to perform semantic encoding on the log sequence to obtain a semantic vector; acquire the weights corresponding to the word vectors of each log in the log sequence; and obtain a target semantic vector according to the semantic vector and the weights corresponding to the word vectors of each log in the log sequence.

[0131] In an exemplary embodiment, the detection module 1004 is specifically configured to determine a probability distribution of the next log of the log sequence based on the target log vector and the log anomaly detection model; the probability distribution is determined based on the probabilities of the next log belonging to each preset log phrase in a preset log phrase set; and determine the log anomaly detection result according to the probability distribution and a probability threshold.

[0132] In an exemplary embodiment, the second determination module 1003 is specifically configured to splice the target semantic vector and the time vector to obtain a target log vector of the log sequence.

[0133] In an exemplary embodiment, the apparatus 1000 may further include:

[0134] A second acquisition module, configured to acquire a target log vector sample corresponding to a log sequence sample;

[0135] A training module, configured to train an initial log anomaly detection model based on the target log vector sample to obtain a log anomaly detection model.

[0136] In an exemplary embodiment, the training module is specifically configured to input the target log vector sample into a bidirectional gated recurrent unit in the initial log anomaly detection model to obtain an output feature sequence of the bidirectional gated recurrent unit; obtain a predicted probability distribution according to the output feature sequence and a multi-head attention model in the initial log anomaly detection model; the predicted probability distribution includes the probabilities of the next log of the log sequence sample belonging to each preset log phrase in a preset log phrase set; and train the initial log anomaly detection model based on the predicted probability distribution and the actual next log of the log sequence sample to obtain a log anomaly detection model.

[0137] Each module in the above-mentioned log anomaly detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0138] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the technical solution of the log anomaly detection method provided in the above embodiment is implemented. The implementation principle and technical effect are similar and will not be elaborated here.

[0139] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the technical solution of the log anomaly detection method provided in the above embodiment is implemented. The implementation principle and technical effect are similar and will not be elaborated here.

[0140] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the technical solution of the log anomaly detection method provided in the above embodiment is implemented. The implementation principle and technical effect are similar and will not be elaborated here.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant regulations.

[0142] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0143] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0144] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for detecting log anomalies, characterized in that, The method includes: Obtaining a target semantic vector corresponding to a log sequence; Determining a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence; Determining a target log vector of the log sequence according to the target semantic vector and the time vector; Based on the target log vector and a log anomaly detection model, performing anomaly detection on the next log of the log sequence to obtain a log anomaly detection result.

2. The method according to claim 1, wherein The obtaining a target semantic vector corresponding to a log sequence includes: Performing semantic encoding on the log sequence to obtain a semantic vector; Obtaining weights corresponding to word vectors of each log in the log sequence; Obtaining the target semantic vector according to the semantic vector and the weights corresponding to the word vectors of each log in the log sequence.

3. The method according to claim 1, wherein The performing anomaly detection on the next log of the log sequence based on the target log vector and a log anomaly detection model to obtain a log anomaly detection result includes: Based on the target log vector and a log anomaly detection model, determining a probability distribution of the next log of the log sequence; the probability distribution is determined based on the probability that the next log belongs to each preset log sentence in a preset log sentence set; Determining the log anomaly detection result according to the probability distribution and a probability threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The determining a target log vector of the log sequence according to the target semantic vector and the time vector includes: Concatenating the target semantic vector and the time vector to obtain a target log vector of the log sequence.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtaining a target log vector sample corresponding to a log sequence sample; Training an initial log anomaly detection model based on the target log vector sample to obtain the log anomaly detection model.

6. The method according to claim 5, characterized in that, The training an initial log anomaly detection model based on the target log vector sample to obtain the log anomaly detection model includes: Inputting the target log vector sample into a bidirectional gated recurrent unit in the initial log anomaly detection model to obtain an output feature sequence of the bidirectional gated recurrent unit; Obtaining a predicted probability distribution according to the output feature sequence and a multi-head attention model in the initial log anomaly detection model; the predicted probability distribution includes the probability that the next log of the log sequence sample belongs to each preset log sentence in a preset log sentence set; Training the initial log anomaly detection model according to the predicted probability distribution and the actual next log of the log sequence sample to obtain the log anomaly detection model.

7. A log anomaly detection device, characterized in that, The apparatus includes: A first obtaining module, configured to obtain a target semantic vector corresponding to a log sequence; A first determining module, configured to determine a time vector corresponding to the log sequence according to the time difference between the timestamps of two adjacent logs in the log sequence; A second determining module, configured to determine a target log vector of the log sequence according to the target semantic vector and the time vector; A detection module, configured to perform anomaly detection on the next log of the log sequence based on the target log vector and a log anomaly detection model to obtain a log anomaly detection result.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images