Log anomaly detection method and device and storage medium

Abstract

The invention provides a log anomaly detection method. The log anomaly detection method comprises the following steps: acquiring log data; preprocessing the log data, including log analysis, log template extraction and log labeling, to obtain structured log data, a log template and an exception judgment tag; and vectorizing the log template, and representing the log template as a sentence vector with a fixed dimension based on the structured log data and the sentence vector model. Storing the log data and the sentence vectors thereof in a vector database; training a GRU model by adopting the labeled log data; and carrying out anomaly detection on the latest collected log. And performing real-time anomaly detection on newly collected logs based on the trained model and the constructed vector database data. GRU prediction and KNN detection based on a vector database are fused, double verification of time sequence and semantics is achieved, false alarm and missing alarm are reduced, and the accuracy of log anomaly detection is improved. The invention further provides a log anomaly detection device.

Description

Technical Field

[0001] This invention relates to the field of log detection technology, specifically to a log anomaly detection method and apparatus. Background Technology

[0002] With the increasing complexity of information technology infrastructure and systems, various devices and systems generate massive amounts of log data to record critical information such as system operating status, security events, and user behavior. Log data plays a vital role in system operation and maintenance, security auditing, and troubleshooting. However, due to the large volume, complex format, and diverse semantic expressions of log data, traditional log anomaly detection methods based on rules or static feature matching cannot effectively identify complex anomaly patterns and suffer from low detection accuracy, high maintenance costs, and poor adaptability.

[0003] In recent years, deep learning-based log anomaly detection methods have been gradually proposed, identifying anomalous events by mining contextual dependencies and temporal patterns in log sequences. Among them, unsupervised learning methods based on models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and Gated Recurrent Units (GRUs) can effectively capture temporal dependencies in log sequences and determine whether log events match normal sequence patterns, thus discovering potential anomalies. However, when the log stream data is large, simple model inference suffers from low inference efficiency and the inability to correlate historical similar anomalies in real time.

[0004] To improve the efficiency and accuracy of log anomaly detection, log processing methods based on vectorized semantic representation have emerged in recent years. These methods transform log text into fixed-length or dynamically lengthed sentence vectors using pre-trained word vectors, Word2Vec, BERT, and other techniques. This allows machine learning or deep learning models to model the log text and capture its deeper semantic information. However, log sentence vectorization only solves the log representation problem; it still needs to be combined with temporal modeling for anomaly pattern learning and detection.

[0005] Vector databases, as a type of database specifically designed for high-dimensional vector similarity retrieval and management, can support efficient similarity searches of vectors. They are commonly used in recommendation systems, image retrieval, and semantic retrieval. Combining vector databases with log anomaly detection can quickly retrieve historical similar log vectors when anomalies are detected or reasoning is performed, thereby helping to determine the context and type of the current anomaly and improving the efficiency of anomaly detection and localization. Summary of the Invention

[0006] The purpose of this invention is to provide a log anomaly detection method, apparatus, and storage medium to solve the problems existing in the prior art, comprising the following steps:

[0007] S1, retrieve log data;

[0008] S2, preprocess the log data, including log parsing, log template extraction and log annotation, to obtain structured log data, log templates and anomaly judgment tags;

[0009] S3, Log Template Vectorization, represents log templates as fixed-dimensional sentence vectors based on structured log data and sentence vector models;

[0010] S4, store the log data and its sentence vectors into the vector database;

[0011] S5 uses labeled log data to train the GRU model;

[0012] S6 performs anomaly detection on the latest collected logs.

[0013] Furthermore, the acquisition of log data mentioned in S1 refers to the collection of one type of logs within a certain period of time.

[0014] Furthermore, the preprocessing procedure described in S2 is as follows:

[0015] S21, Log parsing, parses the raw log into a structured log, obtaining multiple structured parts including log occurrence time, device name, and log message;

[0016] S22, Log template extraction: For the log message part, regular expressions are used to extract and replace variables in the log message, and the Drain algorithm is used to generate a log template.

[0017] S23, Log annotation, refers to marking the abnormal attributes of the original log.

[0018] Furthermore, the sentence vector model described in S3 is Sentence-Bert or Simcse, used to transform log templates into sentence vectors of fixed dimensions.

[0019] Furthermore, the training of the GRU model described in S5 includes:

[0020] S51, Sliding window time sequence construction, refers to constructing a sliding window sequence based on the generation time of log data, that is, constructing input-output pairs for subsequent training of the GRU model;

[0021] S52, data partitioning, is the process of dividing labeled log data into training and testing sets for model training and evaluation.

[0022] S53, data normalization, refers to mapping the numerical range of different features to a unified interval, eliminating differences in units, and improving model stability and training efficiency.

[0023] S54, GRU model construction includes a given model structure and model compilation method, wherein:

[0024] The model structure includes a GRU layer, hidden layers, and an output layer. The model is compiled by specifying the optimizer, loss function, and evaluation metric.

[0025] The GRU layer is a time-series neural network model built using the GRU structure, which is used to learn the patterns in time series data to complete prediction, classification or anomaly detection tasks.

[0026] The hidden layer uses the ReLU activation function to perform nonlinear transformation and feature extraction on the temporal features extracted by GRU;

[0027] The output layer uses the Sigmoid activation function, with an output range of [0,1], representing the anomaly probability of the current window;

[0028] S55, model training, refers to learning the temporal patterns in the data by minimizing the loss function through an optimizer;

[0029] S56, Model Evaluation, is based on the trained model and evaluates its prediction accuracy on the test set to measure the model's generalization ability.

[0030] S57, Model Saving, refers to saving the parameters of a trained model or the complete model for deployment and prediction.

[0031] Furthermore, S6 describes anomaly detection for the latest collected logs:

[0032] S61, Log parsing, parses the log to be predicted, and obtains the log header and log message part. The log header includes the log occurrence time and device information, and the log message part refers to the content in the log that expresses the core event;

[0033] S62, Log template extraction, using the Drain algorithm to generate templates for the log message portion;

[0034] S63, Log template vectorization, adopts sentence vector model to transform log template into sentence vector;

[0035] S64 uses a trained GRU model, inputs the sentence vector of the log to be predicted into the model, and outputs the anomaly probability of the log to be predicted.

[0036] S65, perform KNN prediction based on a vector database to obtain anomaly probability;

[0037] S66, Calculate the combined anomaly probability of the log to be predicted based on the anomaly probability inferred from the GRU model and the anomaly probability predicted by KNN.

[0038] S67. Based on the calculated anomaly probability compared with the anomaly probability threshold, determine the label of the log to be predicted.

[0039] Furthermore, the anomaly probability of the output log to be predicted specifically includes:

[0040] Load the trained GRU model;

[0041] Load the most recent log vector window, that is, load the sentence vector sequence corresponding to the most recent time log according to the defined sequence length;

[0042] Concatenate log vectors, which means discarding the first log vector in the log vector window and then concatenating it with the sentence vector of the latest collected log.

[0043] Normalize the concatenated log vector sequence;

[0044] The normalized log vector is input into the GRU model for inference, and the anomaly probability is calculated.

[0045] Furthermore, the stored log data in S4 includes metadata and corresponding sentence vectors; the metadata includes log ID, original log, log occurrence time, template ID, log template, and tag.

[0046] The present invention also provides a log anomaly detection device, comprising:

[0047] The acquisition module is used to acquire log data;

[0048] The preprocessing module is used to preprocess log data, including log parsing, log template extraction, and log annotation, to obtain structured log data, log templates, and anomaly detection tags;

[0049] The vectorization module is used for log template vectorization, which represents log templates as fixed-dimensional sentence vectors based on structured log data and sentence vector models;

[0050] The training module uses labeled log data to train the GRU model;

[0051] The storage module is used to store log data and its sentence vectors into a vector database;

[0052] The real-time detection module is used to detect anomalies in the latest collected logs.

[0053] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, is a log anomaly detection method as described above.

[0054] This invention provides a log anomaly detection method, apparatus, and storage medium. The above-mentioned technical solution has the following beneficial effects:

[0055] 1. The GRU model can extract the time-series dependencies of logs and identify anomalies caused by context changes and time-series feature abnormalities, thus making up for the shortcomings of static detection of a single log.

[0056] 2. Based on KNN retrieval in a vector database, similarity with historical logs is obtained. Anomalies in the logs to be predicted are then determined based on the anomalies in the historical logs, demonstrating good detection performance for non-time-dependent structural anomalies. Furthermore, the vector database supports incremental writing and deletion of vectors, enabling continuous updates to log data, rapid adaptation to new anomaly patterns, and ensuring real-time anomaly detection.

[0057] 3. By integrating GRU prediction with KNN detection based on a vector database, dual verification of temporal and semantic data is achieved, reducing false positives and false negatives and improving the accuracy of log anomaly detection. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 A flowchart of the log anomaly detection method provided by the present invention;

[0060] Figure 2 The log preprocessing flowchart provided by this invention;

[0061] Figure 3 The GRU model training flowchart provided by this invention;

[0062] Figure 4 A schematic diagram of the GRU model structure provided by this invention;

[0063] Figure 5 The real-time log detection flowchart provided by this invention;

[0064] Figure 6 This is a schematic diagram of the log anomaly detection device provided by the present invention. Detailed Implementation

[0065] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Combination Figure 1 As shown, a log anomaly detection method includes the following steps:

[0067] S1, retrieve log data;

[0068] S2, preprocess the log data, including log parsing, log template extraction and log annotation, to obtain structured log data, log templates and anomaly judgment tags;

[0069] S3, Log Template Vectorization, represents log templates as fixed-dimensional sentence vectors based on structured log data and sentence vector models;

[0070] S4, store the log data and its sentence vectors into the vector database;

[0071] S5 uses labeled log data to train the GRU model;

[0072] S6 performs anomaly detection on the latest collected logs.

[0073] Based on the trained model and the constructed vector database, real-time anomaly detection is performed on newly collected logs.

[0074] Specifically, the acquisition of log data mentioned in S1 refers to the collection of one type of log within a certain period of time.

[0075] One of these types refers to the log type used for anomaly detection, such as the operating system's security log.

[0076] The log type determines the scope of logs for anomaly detection. For example, to detect anomalies in Windows security logs, it is necessary to collect Windows security logs over a certain period of time. This certain period of time is a continuous period.

[0077] In one embodiment, see Figure 2 As shown, the S2 preprocessing flow is as follows:

[0078] S21, Log parsing, parses the raw log into a structured log, obtaining multiple structured parts including log occurrence time, device name, and log message;

[0079] For example, given the original log entry "Dec 10 06:55:46LabSZ sshd

[24200] :Invalid userwebmaster from 173.234.31.186", the parsed result is shown below:

[0080] Log entry time: Dec 10 06:55:46

[0081] Equipment Name: LabSZ

[0082] Process name: sshd

[0083] Process ID: 24200

[0084] Log message: Invalid user webmaster from 173.234.31.186

[0085] S22, Log Template Extraction, involves using regular expressions to extract and replace variables in the log message section, and then using the Drain algorithm to generate a log template.

[0086] The original log and its log template are shown below:

[0087] The original log is Dec 10 06:55:46LabSZ sshd

[24200] :Invalid user webmasterfrom 173.234.31.186. After parsing, the log message is Invalid user webmasterfrom173.234.31.186. Regular expressions are used to extract and replace variables in the log message, including the username (webmaster) and IP (173.234.31.186). Finally, the log template is generated: Invalid user*from****.

[0088] For a set of n original log entries {l1, l2, ..., l n The Drain algorithm extracts m (m≤n) templates {t1,t2,...,t}. m}

[0089] S23, Log Annotation, refers to marking abnormal attributes in the original log.

[0090] The log template does not contain variable information such as IP address or username. If the log volume is small, manual annotation can be used directly. If the log volume is large, large model and other technologies can be used to assist in data annotation.

[0091] The log dataset obtained through log preprocessing is shown below:

[0092] Log ID Original Log Log template Label 1 <![CDATA[l1]]> <![CDATA[t1]]> normal 2 <![CDATA[l2]]> <![CDATA[t2]]> normal 3 <![CDATA[l3]]> <![CDATA[t3]]> abnormal 4 <![CDATA[l4]]> <![CDATA[t1]]> normal 5 <![CDATA[l5]]> <![CDATA[t4]]> abnormal ... ... ... ... n <![CDATA[l n ]]> <![CDATA[t m ]]> abnormal

[0093] In one embodiment, the sentence vector model described in S3 is Sentence-Bert or Simcse, used to convert log templates into sentence vectors of fixed dimensions.

[0094] For a set of n original log entries, there are m log templates {t1, t2, ..., t...} mThe generated sentence vector sequence is} The sentence vector sequence corresponding to the original log is {v1,v2,v3,...,v...} n}

[0095] In one embodiment, see Figure 3 As shown in S5, the training of the GRU model includes:

[0096] S51, Sliding window time sequence construction, refers to constructing a sliding window sequence based on the generation time of log data, that is, constructing input-output pairs for subsequent training of the GRU model;

[0097] S52, data partitioning, is the process of dividing labeled log data into training and testing sets for model training and evaluation.

[0098] S53, data normalization, refers to mapping the numerical range of different features to a unified interval, eliminating differences in units, and improving model stability and training efficiency.

[0099] Using the Min-Max normalization method to normalize the log vector sequence can effectively preserve the time-series information of the log.

[0100] The Min-Max normalization formula is as follows:

[0101]

[0102] Where, x scaled The data is normalized, and x is the original data. min x is the minimum value of the data. max This represents the maximum value of the data.

[0103] The normalization formula is applied to the log vector feature normalization.

[0104] Suppose the existing log vector sequence is {v1,v2,v3}:

[0105] v1=[1,2,3], v2=[4,5,6], v3=[7,8,9]

[0106] It can be concluded that the minimum value of each feature is [1,2,3], and the maximum value of each feature is [7,8,9].

[0107] Normalization of the first feature of v1:

[0108] Normalization of the first feature for v2:

[0109] Normalization of the first feature for v3:

[0110] Normalization of the second feature of v1:

[0111] Normalization of the second feature for v2:

[0112] Normalization of the second feature for v3:

[0113] Normalization for the third feature of v1:

[0114] Normalization for the third feature of v2:

[0115] Normalization for the third feature of v3:

[0116] After the above process, the normalized log vector sequence is obtained as {v 1s ,v 2s ,v 3s}:

[0117] v 1s =[0,0,0],v 2s =[0.5,0.5,0.5],v 3s =[1,1,1]

[0118] S54, GRU model construction includes a given model structure and model compilation method, wherein:

[0119] The model structure includes a GRU layer, hidden layers, and an output layer. The model is compiled by specifying the optimizer, loss function, and evaluation metric.

[0120] The GRU layer is a time-series neural network model built using the GRU structure, which is used to learn the patterns in time series data to complete prediction, classification or anomaly detection tasks.

[0121] The hidden layer uses the ReLU activation function to perform nonlinear transformation and feature extraction on the temporal features extracted by GRU;

[0122] The output layer uses the Sigmoid activation function, with an output range of [0,1], representing the anomaly probability of the current window;

[0123] The GRU structure is described below. Figure 4 It controls the flow and retention of information through two gating units: the reset gate and the update gate.

[0124] The GRU structure formula is expressed as follows:

[0125]

[0126] Where, x t This represents the input information at the current moment; h t-1 Indicates the hidden state at the previous time step; h t This indicates the hidden state that will be passed on to the next time step; Represents the candidate hidden state; r t Indicates resetting the door; z t This represents the update gate; σ is the sigmoid function, which transforms the data into values ​​in the range [0,1]; tanh is the hyperbolic tangent function, which transforms the data into values ​​in the range [-1,1].

[0127] Suppose the input sequence x = {x1, x2, ..., xn} T After processing by the GRU structure, the hidden states are updated sequentially to obtain the sequence H = {h1, h2, ... h}. T}, GRU will hide the last time state h T As a temporal feature vector, it is then fed into the hidden layer.

[0128] The hidden layer uses the ReLU activation function to perform nonlinear transformation and feature extraction on the temporal features extracted by GRU.

[0129] First, regarding the vector h output by the GRU layer T After transformation, we get:

[0130] x = W h ·h T +b h

[0131] Where x represents the transformed vector, W h Let b be the weight matrix of the hidden layer. h Let x be the bias vector. Further, the ReLU activation function is applied to x, assuming x = {x1, x2, ..., x...}. m},

[0132] y i =ReLU(x i ) = max(0, x i )

[0133] The above formula indicates that if x i If x > 0, then output x; if x > 0, then output x. i If ≤0, then output 0.

[0134] This leads to y = ReLU(x), where y is the output vector of the hidden layer.

[0135] y = {y1, y2, ..., y m}

[0136] The output layer uses the Sigmoid activation function, with an output range of [0,1], representing the anomaly probability of the current window.

[0137] In one embodiment, the anomaly probability P is calculated using the Sigmoid function based on the output vector y of the hidden layer. gru .

[0138] The calculation formula is as follows:

[0139]

[0140] Among them, P gru Let W represent the anomaly probability, y be the output vector of the hidden layer, and W be the output vector of the hidden layer. o b represents the output layer weights. o This is the bias vector for the output layer.

[0141] S55, model training, refers to learning the temporal patterns in the data by minimizing the loss function through an optimizer;

[0142] The parameters of the constructed GRU model are updated based on the training set data, thereby minimizing the loss function and learning the mapping relationship between input and output.

[0143] S56, Model Evaluation, is based on the trained model and evaluates its prediction accuracy on the test set to measure the model's generalization ability.

[0144] Metrics of measurement can include accuracy, F1 score, etc.

[0145] S57, Model Saving, refers to saving the parameters of the trained model or the complete model for deployment and prediction.

[0146] The saved model can be model weights or a complete model.

[0147] See Figure 5 As shown, in one embodiment, step S6 involves anomaly detection for the latest collected log:

[0148] S61, Log parsing, parses the log to be predicted, and obtains the log header and log message part. The log header includes the log occurrence time and device information, and the log message part refers to the content in the log that expresses the core event;

[0149] S62, Log template extraction, using the Drain algorithm to generate templates for the log message portion;

[0150] S63, Log template vectorization, adopts sentence vector model to transform log template into sentence vector;

[0151] S64 uses a trained GRU model, inputs the sentence vector of the log to be predicted into the model, and outputs the anomaly probability of the log to be predicted.

[0152] The probability of an anomaly in the predicted log is defined as P. gru .

[0153] Load the complete GRU model or load the weights of the GRU model;

[0154] Load the log vector sequence of the most recent time window. If the current time is t and the time window W = 5, then the loaded log vector sequence is {v t-5 ,v t-4 ,v t-3 ,v t-2 ,v t-1};

[0155] The log vector sequence is concatenated by removing the first log vector from the most recent time window and then concatenating the log vectors to receive the predicted logs. The log vector at the current time is v. t The concatenated log vector sequence is {v t-4 ,v t-3 ,v t-2 ,v t-1 ,v t};

[0156] Normalize the concatenated log vector sequence;

[0157] The normalized log vector sequence is input into the GRU model for inference, and the anomaly probability P is output. gru

[0158] S65, perform KNN prediction based on a vector database to obtain anomaly probability;

[0159] Calculate the anomaly probability P vd P vd The calculation formula is as follows:

[0160]

[0161] Assuming k=5, and the neighbor labels obtained through KNN retrieval are [0,0,1,0,1], where 0 represents normal and 1 represents abnormal, then the probability of abnormality is...

[0162] S66. Calculate the combined anomaly probability of the log to be predicted based on the anomaly probability inferred from the GRU model and the anomaly probability predicted by KNN.

[0163] The comprehensive anomaly probability is defined as follows:

[0164] P t =α·Pgru +β·P vd

[0165] Where α and β are weight parameters, P gru P represents the anomaly probability output by the GRU model. vd The anomaly probability is calculated after KNN retrieval.

[0166] The optimal combination of weight parameters α and β can be found through methods such as grid search.

[0167] S67. Based on the calculated anomaly probability compared with the anomaly probability threshold, determine the label of the log to be predicted.

[0168] In one embodiment, the output of the anomaly probability of the log to be predicted specifically includes:

[0169] Load the trained GRU model;

[0170] Load the most recent log vector window, that is, load the sentence vector sequence corresponding to the most recent time log according to the defined sequence length;

[0171] Concatenate log vectors, which means discarding the first log vector in the log vector window and then concatenating it with the sentence vector of the latest collected log.

[0172] Normalize the concatenated log vector sequence;

[0173] The normalized log vector is input into the GRU model for inference, and the anomaly probability is calculated.

[0174] In one embodiment, the stored log data in S4 includes metadata and corresponding sentence vectors; the metadata includes log ID, original log, log occurrence time, template ID, log template, and tag.

[0175] See Figure 6 As shown, in another embodiment, a log anomaly detection device includes:

[0176] The acquisition module is used to acquire log data;

[0177] The preprocessing module is used to preprocess log data, including log parsing, log template extraction, and log annotation, to obtain structured log data, log templates, and anomaly detection tags;

[0178] The vectorization module is used for log template vectorization, which represents log templates as fixed-dimensional sentence vectors based on structured log data and sentence vector models;

[0179] The storage module is used to store log data and its sentence vectors into a vector database;

[0180] The training module uses labeled log data to train the GRU model;

[0181] The real-time detection module is used to detect anomalies in the latest collected logs.

[0182] In yet another embodiment, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the log anomaly detection method as described above.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting log anomalies, characterized in that, Includes the following steps: S1, retrieve log data; S2, preprocess the log data, including log parsing, log template extraction and log annotation, to obtain structured log data, log templates and anomaly judgment tags; S3, Log Template Vectorization, represents log templates as fixed-dimensional sentence vectors based on structured log data and sentence vector models; S4, store the log data and its sentence vectors into the vector database; S5 uses labeled log data to train the GRU model; S6 performs anomaly detection on the latest collected logs.

2. The method according to claim 1, characterized in that, The acquisition of log data mentioned in S1 refers to the collection of one type of log within a certain period of time.

3. The method according to claim 1, characterized in that, The preprocessing procedure described in S2 is as follows: S21, Log parsing, parses the raw log into a structured log, obtaining multiple structured parts including log occurrence time, device name, and log message; S22, Log template extraction: For the log message part, regular expressions are used to extract and replace variables in the log message, and the Drain algorithm is used to generate a log template. S23, Log annotation, refers to marking the abnormal attributes of the original log.

4. The method according to claim 3, characterized in that, The sentence vector model described in S3 is either Sentence-Bert or Simcse, used to convert log templates into sentence vectors of fixed dimensions.

5. The method according to claim 4, characterized in that, The training of the GRU model described in S5 includes: S51, Sliding window time sequence construction, refers to constructing a sliding window sequence based on the generation time of log data, that is, constructing input-output pairs for subsequent training of the GRU model; S52, data partitioning, is the process of dividing labeled log data into training and testing sets for model training and evaluation. S53, data normalization, refers to mapping the numerical range of different features to a unified interval, eliminating differences in units, and improving model stability and training efficiency. S54, GRU model construction includes a given model structure and model compilation method, wherein: The model structure includes a GRU layer, hidden layers, and an output layer. The model is compiled by specifying the optimizer, loss function, and evaluation metric. The GRU layer is a time-series neural network model built using the GRU structure, which is used to learn the patterns in time series data to complete prediction, classification or anomaly detection tasks. The hidden layer uses the ReLU activation function to perform nonlinear transformation and feature extraction on the temporal features extracted by GRU; The output layer uses the Sigmoid activation function, with an output range of [0,1], representing the anomaly probability of the current window; S55, model training, refers to learning the temporal patterns in the data by minimizing the loss function through an optimizer; S56, Model Evaluation, is based on the trained model and evaluates its prediction accuracy on the test set to measure the model's generalization ability. S57, Model Saving, refers to saving the parameters of a trained model or the complete model for deployment and prediction.

6. The method according to claim 5, characterized in that, S6 describes anomaly detection for the latest collected logs: S61, Log parsing, parses the log to be predicted, and obtains the log header and log message part. The log header includes the log occurrence time and device information, and the log message part refers to the content in the log that expresses the core event; S62, Log template extraction, using the Drain algorithm to generate templates for the log message portion; S63, Log template vectorization, adopts sentence vector model to transform log template into sentence vector; S64 uses a trained GRU model, inputs the sentence vector of the log to be predicted into the model, and outputs the anomaly probability of the log to be predicted. S65, perform KNN prediction based on a vector database to obtain anomaly probability; S66. Calculate the combined anomaly probability of the log to be predicted based on the anomaly probability inferred from the GRU model and the anomaly probability predicted by KNN. S67. Based on the calculated anomaly probability compared with the anomaly probability threshold, determine the label of the log to be predicted.

7. The method according to claim 6, characterized in that, The anomaly probability of the output log to be predicted specifically includes: Load the trained GRU model; Load the most recent log vector window, that is, load the sentence vector sequence corresponding to the most recent time log according to the defined sequence length; Concatenate log vectors, which means discarding the first log vector in the log vector window and then concatenating it with the sentence vector of the latest collected log. Normalize the concatenated log vector sequence; The normalized log vector is input into the GRU model for inference, and the anomaly probability is calculated.

8. The method according to claim 1, characterized in that, The stored log data in S4 includes metadata and corresponding sentence vectors; the metadata includes log ID, original log, log occurrence time, template ID, log template, and tag.

9. A log anomaly detection device, characterized in that, include: The acquisition module is used to acquire log data; The preprocessing module is used to preprocess log data, including log parsing, log template extraction, and log annotation, to obtain structured log data, log templates, and anomaly detection tags; The vectorization module is used for log template vectorization, which represents log templates as fixed-dimensional sentence vectors based on structured log data and sentence vector models; The storage module is used to store log data and its sentence vectors into a vector database; The training module uses labeled log data to train the GRU model; The real-time detection module is used to detect anomalies in the latest collected logs.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the log anomaly detection method according to any one of claims 1-8.

Images