A method and apparatus for detecting log anomalies

Abstract

This application discloses a log anomaly detection method and apparatus, relating to the field of artificial intelligence technology. It addresses the problem that existing log anomaly detection methods ignore the semantic information contained in the self-assigned parameters of log entries, leading to reduced sensitivity in anomaly detection. The method includes: extracting fixed template information and dynamically assigned parameters, or vice versa, from each log entry using a template parsing method; obtaining a corresponding log representation vector with deep semantic association features using a pre-trained language model; inputting the log representation vector with deep semantic association features corresponding to each log entry into a pre-trained log anomaly detection model to obtain the corresponding log anomaly probability; and determining the current log as an anomaly log when the log anomaly probability of any log entry among multiple log entries exceeds a probability threshold.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for detecting log anomalies. Background Technology

[0002] With the rapid development of information technology, various systems (such as industrial control systems, server clusters, and IoT devices) generate massive amounts of log data during operation. Log data records key information such as system status, operational behaviors, and abnormal events, serving as crucial evidence for system fault diagnosis, status monitoring, and security auditing. Log anomaly detection, as a core component of ensuring stable system operation, aims to promptly identify log entries with abnormal states from massive amounts of log data, providing decision support for system operation and maintenance and risk warning. Existing log anomaly detection methods typically only focus on system-level issues indicated by log anomalies, such as service crashes and connection failures—macro-level faults—ignoring the semantic information contained in self-assigned parameters within log entries. For example, the semantic information contained in dynamic variables such as device ID, temperature value, timestamp, and error code. Since the specific values ​​and semantics of self-assigned parameters are key to determining the nature of the anomaly, ignoring parameter semantics—for example, judging solely by the macro-level label of temperature anomaly—fails to distinguish these subtle differences, leading to reduced sensitivity in anomaly detection. Summary of the Invention

[0003] The embodiments of this application provide a log anomaly detection method and apparatus, the main purpose of which is to solve the problem that existing log anomaly detection methods ignore the semantic information contained in the self-assigned parameters in log entries, resulting in reduced sensitivity of anomaly detection.

[0004] According to a first aspect of this application, a log anomaly detection method is provided, comprising:

[0005] Get the current log containing multiple log entries, and use the template parsing method to extract fixed template information, dynamically assigned parameters, or dynamically assigned parameters from each log entry;

[0006] Based on the fixed template information and dynamically assigned parameters corresponding to each log entry, or the dynamically assigned parameters, a log representation vector with deep semantic association features corresponding to each log entry is obtained using a pre-trained language model.

[0007] The log representation vector with deep semantic association features corresponding to each log entry is input into the pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model.

[0008] A probability threshold is obtained, and when the probability of log anomaly corresponding to any log entry among the plurality of log entries exceeds the probability threshold, the current log is determined to be an abnormal log.

[0009] According to a second aspect of this application, a log anomaly detection device is provided, comprising:

[0010] The extraction module is used to obtain the current log, which includes multiple log entries, and uses the template parsing method to extract fixed template information, dynamically assigned parameters, or dynamically assigned parameters from each log entry.

[0011] The transformation module is used to obtain a log representation vector with deep semantic association features for each log entry based on the fixed template information, dynamic self-assigned parameters, or dynamic self-assigned parameters corresponding to each log entry, using a pre-trained language model.

[0012] The input module is used to input the log representation vector with deep semantic association features corresponding to each log entry into the pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model.

[0013] The determination module is used to obtain a probability threshold, and when the probability of log anomaly corresponding to any log entry among the plurality of log entries exceeds the probability threshold, the current log is determined to be an abnormal log.

[0014] By employing the above technical solution, this application provides a log anomaly detection method and apparatus. This application can extract fixed template information, dynamically assigned parameters, or dynamically assigned parameters from each log entry using a template parsing method; then, based on the fixed template information and dynamically assigned parameters corresponding to each log entry, a pre-trained language model is used to obtain a log representation vector with deep semantic association features corresponding to each log entry; subsequently, the log representation vector with deep semantic association features corresponding to each log entry is input into a pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model; finally, when the log anomaly probability corresponding to any log entry among multiple log entries exceeds a probability threshold, the current log is determined to be an abnormal log. For log entries with preset templates, this application extracts fixed template information and dynamically assigned parameters from each log entry using a template parsing method. Then, a pre-trained language model is used to not only deeply mine the fixed template information, preserving the detection efficiency of structured logs, but also to semantically activate the dynamically assigned parameters, releasing the hidden value of parameters in unstructured logs. For log entries without preset templates, the template parsing method extracts dynamically assigned parameters from each log entry, and the pre-trained language model deeply binds the dynamically assigned parameters to the semantic description of the log entry without a preset template. Furthermore, the pre-trained log anomaly detection model in this application is pre-trained and can accurately understand the inherent relationship patterns in log representation vectors with deep semantic association features, especially its ability to capture hidden anomalies, accurately outputting the log anomaly probability corresponding to each log entry, thus improving the sensitivity and accuracy of log anomaly detection. Attached Figure Description

[0015] Figure 1 A flowchart of a log anomaly detection method provided in an embodiment of this application is shown;

[0016] Figure 2 A flowchart of another log anomaly detection method provided in an embodiment of this application is shown;

[0017] Figure 3 This paper shows a schematic diagram of the structure of a log anomaly detection device provided in an embodiment of this application;

[0018] Figure 4 This paper shows a schematic diagram of the structure of a log anomaly detection device provided in an embodiment of this application;

[0019] Figure 5 A schematic diagram of the structure of a log anomaly detection device provided in an embodiment of this application is shown. Detailed Implementation

[0020] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0021] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0022] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0023] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0024] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0025] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0026] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0027] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0028] This application provides a log anomaly detection method, such as... Figure 1 As shown, it includes:

[0029] 101. Obtain the current log containing multiple log entries, and use the template parsing method to extract fixed template information, dynamically assigned parameters, or dynamically assigned parameters from each log entry.

[0030] In this embodiment, the current log can be understood as the current original log. For subsequent abnormal log detection, the original log entries need to be parsed, i.e., the unstructured text is converted into quantifiable features. This application uses a template parsing method to parse the original log entries, distinguishing between the fixed structural parts and the variable parts that change with time / scene, resulting in a structured representation of "template + parameters". Then, for some log entries, due to their chaotic format, variable structure, or semantic ambiguity, the template parsing method cannot accurately extract the fixed target information. Therefore, the template parsing method can be used to extract dynamically assigned parameters from the log entries. These log entries with chaotic format, variable structure, or semantic ambiguity can be called log entries without a preset template, while the log entries with clear structured features can be called log entries with a preset template. It should be noted that in this embodiment, the preferred template parsing method is the log template mining tool Drain3, which is existing technology and easily accessible to those skilled in the art; therefore, it will not be specifically described here.

[0031] 102. Based on the fixed template information, dynamic self-assigned parameters, or dynamic self-assigned parameters corresponding to each log entry, a log representation vector with deep semantic association features corresponding to each log entry is obtained using a pre-trained language model.

[0032] In this embodiment, for log entries with preset templates, fixed template information and dynamically assigned parameters are extracted from the log entries using a template parsing method. The structured parsing results are then transformed into high-dimensional semantic features using a pre-trained language model. This means that the pre-trained language model not only preserves the structural information and specific parameter values ​​of the log entries but also captures the hidden semantic relationships within them, providing semantically rich input for subsequent log anomaly detection. For log entries where fixed templates cannot be extracted (log entries without preset templates), for example, "Oh dear, the device seems to have a problem. The TC-890 sensor suddenly jumped to 99 degrees. I wonder if the cable is loose," "[Monitoring module] timeout; Device log: TC-890=overheat; Third-party plugin: 2023-10-01 14:30" Error code "x7f3" indicates that dynamic self-assigned parameters were extracted from the log entries in step 1. In the log entry "Oops, the device seems to have a problem. The TC-890 sensor suddenly jumped to 99 degrees. I wonder if the cable is loose," TC-890 and 99 are dynamic self-assigned parameters. Since log entries without a preset template usually contain rich natural language semantics (such as "the device seems to have a problem" and "suddenly jumped to 99 degrees"), this semantic information is an important supplement to the features of dynamic self-assigned parameters. Using a pre-trained language model, the dynamic self-assigned parameters are first converted into a semantic representation vector. Then, the log entries without a preset template after extracting the dynamic self-assigned parameters are converted into target semantic vectors to capture deep semantics such as "abnormal tendencies" and "sudden changes." Finally, the pre-trained language model is used to concatenate the semantic representation vectors of the self-assigned parameters and the target semantic vectors to form a fused feature of "parameters + semantics." Therefore, it can be seen that a log representation vector with deep semantic association features corresponding to each log entry can be obtained through a pre-trained language model.

[0033] 103. Input the log representation vector with deep semantic association features corresponding to each log entry into the pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model.

[0034] In this embodiment, the pre-trained log anomaly detection model integrates a semantic understanding module and an anomaly judgment module. Through pre-training with massive amounts of log data, the model is able to further mine the hidden anomaly features in the log representation vector and accurately output the anomaly probability.

[0035] 104. Obtain the probability threshold. When the probability of any log entry among multiple log entries exceeds the probability threshold, the current log is determined to be an abnormal log.

[0036] For example, the probability threshold can be set to 0.7. If the log anomaly probability corresponding to a certain log entry is 0.92, since 0.92 > 0.7, the current log can be determined to be an abnormal log, and the abnormal log entry can be marked and a warning can be triggered.

[0037] The method provided in this application first extracts fixed template information and dynamically assigned parameters, or dynamically assigned parameters, from each log entry using a template parsing method. Then, based on the fixed template information and dynamically assigned parameters corresponding to each log entry, a pre-trained language model is used to obtain a log representation vector with deep semantic association features for each log entry. Subsequently, the log representation vector with deep semantic association features corresponding to each log entry is input into a pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model. Finally, when the log anomaly probability corresponding to any log entry among multiple log entries exceeds a probability threshold, the current log is determined to be an abnormal log. For log entries with preset templates, this application extracts fixed template information and dynamically assigned parameters from each log entry using a template parsing method. Then, a pre-trained language model is used to not only deeply mine the fixed template information, preserving the detection efficiency of structured logs, but also to semantically activate the dynamically assigned parameters, releasing the hidden value of parameters in unstructured logs. For log entries without preset templates, the template parsing method extracts dynamically assigned parameters from each log entry, and the pre-trained language model deeply binds the dynamically assigned parameters to the semantic description of the log entry without a preset template. Furthermore, the pre-trained log anomaly detection model in this application is pre-trained and can accurately understand the inherent relationship patterns in log representation vectors with deep semantic association features, especially its ability to capture hidden anomalies, accurately outputting the log anomaly probability corresponding to each log entry, thus improving the sensitivity and accuracy of log anomaly detection.

[0038] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation methods of this embodiment, this application provides another log anomaly detection method, such as... Figure 2 As shown, it includes:

[0039] 201. Obtain the current log containing multiple log entries, and use the template parsing method to extract fixed template information, dynamically assigned parameters, or dynamically assigned parameters from each log entry.

[0040] In this embodiment, the executing entity is a detection system capable of performing log anomaly detection methods. This system first obtains the current logs of multiple log entries. Then, it preprocesses these log entries, specifically by: first, performing format unification and cleaning operations, such as removing irrelevant symbols (e.g., redundant spaces, special characters) and unifying the timestamp format (e.g., unifying "14:30:00" and "14-30-00" into "14:30:00"); then performing noise filtering operations, such as removing invalid logs (e.g., blank lines, duplicate and redundant heartbeat logs) and retaining entries containing key information (e.g., status changes, error messages). Next, it extracts dynamically assigned parameters based on regular expressions and replaces them with unified placeholders. The purpose is to transform unstructured logs into semi-structured intermediate log entries (template prototypes). This process preserves the fixed text framework of log entries and unifies the representation of dynamic parameters through placeholders, laying the foundation for subsequent processing. Finally, it determines the log category of each intermediate log entry using a log template, i.e., with or without a preset template, providing a basis for subsequent differentiated processing. Finally, for log entries with preset templates, it has been determined that they have preset templates, for example, such as " <time>equipment <id>Temperature = <num>"Status = Abnormal", placeholder (e.g., <id> <num>The template parsing method marks the location and type of dynamic parameters, and based on the correspondence between placeholders and actual dynamic parameters, precise location and extraction can be achieved. That is, the template parsing method can accurately extract fixed template information and dynamically assigned parameters from log entries with preset templates. For log entries without preset templates, the template parsing method extracts dynamically assigned parameters from these log entries for further processing.

[0041] It should be noted that regular expressions are an efficient text matching tool, and since they are existing technology, readily available to anyone skilled in the art, they will not be discussed in detail here. In addition to using the log template mining tool Drain3, template parsing methods can also be chosen based on clustering, such as the LogCluster algorithm (log clusterer): which generates clusters by progressively merging logs with edit distances less than a threshold, comparing each log in each cluster character by character, and extracting the invariant parts as templates; and the LenMa algorithm (Leuwenberg-Marquardt algorithm): which first groups logs by length, then calculates the string similarity of logs within the same group to optimize the clustering. Token-based clustering methods can also be chosen, such as the Spell algorithm (an online stream processing log parsing method based on the longest common subsequence): which replaces dynamic tokens such as values ​​and IDs with... <num> <id>After labeling and standardizing the format, clustering can be performed. Alternatively, deep learning-based template parsing methods can be used, such as: LSTM-CRF (Long Short-Term Memory Network-Conditional Random Field): LSTM captures contextual dependencies, CRF optimizes the rationality of the label sequence, and outputs a template / parameter label for each character; BERT+BiLSTM (Bidirectional Language Understanding and Encoding Representation + Bidirectional Long Short-Term Memory Network): BERT extracts deep semantic features, and bidirectional LSTM improves annotation accuracy. In this embodiment, at least one of the above algorithms can be flexibly selected as the template parsing method according to the actual situation.

[0042] Specifically, the process of obtaining a log template set including multiple log templates based on the template parsing method, and determining the log category of each intermediate log entry using the log templates, includes the following steps: First, based on the template parsing method, the character length of the fixed text portion in each intermediate log entry is calculated to obtain the character length values ​​of multiple intermediate log entries. At least one intermediate log entry with the same character length value is grouped into one category, resulting in multiple first intermediate log entry subsets, where each character length corresponds to one first intermediate log entry subset. Then, for each first intermediate log entry subset, the following operations are performed: using the template parsing method, at least one intermediate log entry in the first intermediate log entry subset is categorized based on the log start word, resulting in at least one second intermediate log entry subset, where each log start word corresponds to one second intermediate log entry subset. Finally, for each intermediate log entry in each subset of second intermediate log entries, the following operations are performed: A log template set including multiple log templates is obtained based on the template parsing method. An intermediate log entry is compared word-by-word with each log template in the log template set to determine the number of words matching each log template. The total number of words in each log template is counted, and the ratio of the number of words to the total number of words is calculated to obtain multiple similarity scores. If one of the multiple similarity scores exceeds a score threshold, the log category of the intermediate log entry is determined to be a log entry with a preset template. If none of the multiple similarity scores exceeds the score threshold, further log type verification is performed on the intermediate log entry to obtain a type verification result. The type verification result indicates that the log type of the intermediate log entry is either a log entry with a preset template or a log entry without a preset template.

[0043] Furthermore, the intermediate log entries undergo further log type verification to obtain type verification results, including: first, creating a new log template for the intermediate log entry based on the template parsing method; then, calculating the structural similarity between the new log template and each log template in the log template set based on the text matching algorithm to obtain multiple structural similarity values; next, obtaining a similarity threshold based on the template parsing method; when a structural similarity value exceeds the similarity threshold, the log type of the intermediate log entry is determined to be a log entry with a preset template; when no structural similarity value exceeds the similarity threshold, the log type of the intermediate log entry is determined to be a log entry without a preset template.

[0044] For example, the fixed structure can be directly extracted from intermediate log entries using the Drain3 tool; that is, the intermediate log entries are divided into token sequences based on placeholders and fixed text, for example: intermediate log entry " <time>sensor <id>The temperature suddenly rose to <num>"To the token sequence: [" <time>", " ", "sensor", " <id>", "suddenly rose to", " <num>"]; then filter the placeholders in the token sequence ( <time> <id> <num>), the fixed text tokens are reserved and spliced in the original order to form a new log template. The text matching algorithm can select the Levenshtein Distance algorithm or the Jaccard coefficient algorithm, both of which are prior art and are readily available to those skilled in the art, and will not be described in detail here. The above achieves the automatic classification of intermediate log entries (with preset templates / without preset templates).

[0045] 202、Based on the fixed template information corresponding to each log entry, the dynamic self-assignment parameter, or the dynamic self-assignment parameter, a pre-trained language model is used to obtain a log representation vector with deep semantic association features corresponding to each log entry.

[0046] In the embodiments of the present application, by classifying the log entries, it can be determined that the log type of the log entries is a preset template log entry or a non-preset template log entry. For each preset template log entry, the following operations are performed: based on the fixed template information corresponding to the preset template log entry, the dynamic self-assignment parameter, and the pre-trained language model, a log representation vector with deep semantic association features is obtained. For each non-preset template log entry, the following operations are performed: based on the dynamic self-assignment parameter corresponding to the non-preset template log entry, a pre-trained language model is used to obtain a log representation vector with deep semantic association features.

[0047] Specifically, based on the preset template log entry corresponding to the fixed template information, the dynamic self-assignment parameter, the pre-trained language model is used to obtain a log representation vector with deep semantic correlation characteristics, including: first, based on the CodeBERT model, the preset template log entry corresponding to the fixed template information, the dynamic self-assignment parameter is converted into a fixed template information semantic representation vector and a self-assignment parameter semantic representation vector; then, based on the part-of-speech analysis algorithm, the syntax category of each first word in the fixed template information semantic representation vector is labeled, and the syntax category of the second word in the self-assignment parameter semantic representation vector is labeled; the syntax category of each first word in the fixed template information semantic representation vector is matched with the syntax category of each second word in the self-assignment parameter semantic representation vector to determine the mapping relationship between the first word and the second word; based on the mapping relationship between the first word and the second word, the fixed template information semantic representation vector and the self-assignment parameter semantic representation vector are spliced to obtain a first spliced log representation vector; based on the term frequency-inverse document frequency algorithm, the static weight of each word in the first spliced log representation vector is calculated; based on the cosine similarity algorithm, the cosine similarity between any two words in the first spliced log representation vector is calculated, and the dynamic weight of each word in the first spliced log representation vector is adjusted using the cosine similarity; based on the static weight and the dynamic weight of each word in the first spliced log representation vector, a fusion weight matrix is obtained using a weight normalization algorithm, and the first spliced log representation vector and the fusion weight matrix are weighted and scaled by matrix multiplication to obtain a log representation vector with deep semantic correlation characteristics. For the preset template log entry (i.e. structured log entry), for the structured log whose "template + self-assignment parameter" has been parsed, the CodeBERT model is used to generate a semantic representation, for example, the fixed template information is "Process [] crash with signal []", and the dynamic self-assignment parameter is "pid=1234" "SIGSEGV". The two are input into the CodeBERT model as text data. The CodeBERT model divides the text data into "token" (such as "Process", "crash", "pid", "1234", etc. The smallest semantic unit), and then the multi-head attention coding layer in the CodeBERT model determines the importance of each token by calculating the attention weight between tokens, and finally outputs the semantic representation of each token through the CodeBERT. Finally, the entire log vector is aggregated, i.e. the fixed template information semantic representation vector and the self-assignment parameter semantic representation vector. The CodeBERT model, the part-of-speech analysis algorithm, the term frequency-inverse document frequency algorithm, and the cosine similarity algorithm are all prior art and can be easily obtained by those skilled in the art, and will not be described in detail here.The first and second in the first and second words above have no explicit physical meaning, only as a distinction.

[0048] It should be noted that the present application can mine the fixed template information from the log entries, and determine the fixed structure and core scene of the log entries. Meanwhile, after extracting the dynamic self-assignment parameters from the log entries, the meanings of the parameters in specific scenes are activated through syntax category matching and semantic mapping, so that the log representation vectors with deep semantic association features can be obtained.

[0049] Further, based on the dynamic self-assignment parameter corresponding to the no-preset template log entry, a pre-trained language model is used to obtain a log representation vector with deep semantic correlation features, including: first, based on a part-of-speech analysis algorithm, the grammatical category of each third word in the no-preset template log entry is labeled, and the grammatical category of the fourth word in the dynamic self-assignment parameter corresponding to the no-preset template log entry is labeled; then, based on the CodeBERT model, the no-preset template log entry with the labeled grammar type is converted into a no-preset template log vector, and the dynamic self-assignment parameter with the labeled grammar type is converted into an intermediate self-assignment parameter semantic representation vector; subsequently, the grammatical category of each third word in the no-preset template log vector is matched with the grammatical category of each fourth word in the intermediate self-assignment parameter semantic representation vector to determine the mapping relationship between the third word and the fourth word; based on the mapping relationship between the third word and the fourth word, the no-preset template log vector and the intermediate self-assignment parameter semantic representation vector are then spliced to obtain a second spliced log representation vector; further, the static weight of each word in the second spliced log representation vector is calculated based on the term frequency-inverse document frequency algorithm; then, the cosine similarity between any two words in the second spliced log representation vector is calculated based on the cosine similarity algorithm, and the dynamic weight of each word in the second spliced log representation vector is adjusted using the cosine similarity; finally, based on the static weight and dynamic weight of each word in the second spliced log representation vector, a fusion weight matrix is obtained using a weight normalization algorithm, and the second spliced log representation vector is weighted and scaled with the fusion weight matrix through matrix multiplication to obtain a log representation vector with deep semantic correlation features. Through the above regular expression, the dynamic self-assignment parameter has been separated from the no-preset template log entry, for example, the no-preset template log is: "2025-08-09 14:30: Device ID=789 suddenly offline, sensor data jump detected, possibly related to voltage fluctuation", and the dynamic self-assignment parameter is: Device ID=789, sensor data jump (specific change value), voltage fluctuation (reason details); then, the part-of-speech analysis algorithm is used to perform word segmentation on the third words in the no-preset template log entry with the extracted dynamic self-assignment parameter and label the part-of-speech, for example, verbs (offline), nouns (devices, sensors), and adverbs (suddenly), etc. At the same time, the part-of-speech analysis algorithm is also used to perform word segmentation on the dynamic self-assignment parameter corresponding to the no-preset template log entry and label the part-of-speech, for example, numerals (789). The CodeBERT model is used to convert the no-preset template log entry with the labeled grammar type into a no-preset template log vector, and convert the dynamic self-assignment parameter with the labeled grammar type into an intermediate self-assignment parameter semantic representation vector; then, the CodeBERT model is used to encode the no-preset template log entry with the labeled grammar type into a semantic vector, i.e., a no-preset template log vector. The above process can ignore format differences and focus on deep semantics.Here, the BERT model (BERT model) and the RoBERTa model (RoBERTa model) can also be used to achieve the above purposes. At the same time, the dynamic self-assignment parameter of the annotated syntax type is converted into an intermediate self-assignment parameter semantic representation vector. Then, by comparing the syntax categories of the specified position words in the two vectors, the corresponding relationship between the two words in terms of function is found. For example, the log segmentation corresponding to the preset template-free log vector is: ["Oh", ",", "TC-890", "suddenly", "warmed up", "to", "99", "degrees"], among which, the "TC-890" syntax category is annotated as "device ID"; and the parameter segmentation corresponding to the intermediate self-assignment parameter semantic representation vector is: ["TC-890", "at", "14:30", "warmed up"], among which, the "warmed up" syntax category is annotated as "action (temperature change)"; "device ID" and "action (temperature change)" have a "device-to-device action" association, and it is determined that they are associated and matched. The "device ID" to "device action" in the parameter in the log entry reflects the belonging relationship between "TC-890" and "warmed up". According to the mapping relationship between the third word and the fourth word, mark the associated anchor points in the two vectors, count the vector dimensions of the preset template-free log vector and the intermediate self-assignment parameter semantic representation vector, and unify the lengths of the two vectors by padding or truncation. Then, according to the type of the mapping relationship (for example, one-to-one, functional association, etc.), the elements of the two vectors are weighted and fused. After weighted fusion and sequential splicing, the final second spliced log representation vector is obtained, and the dimension of the vector is the sum of the dimensions of the two vectors. Then, the term frequency-inverse document frequency algorithm (TF-IDF algorithm) is used to calculate the static weight, and the static weight is used to evaluate the individual importance of each word in the second spliced log representation vector. The cosine similarity algorithm is used to determine the dynamic weight, which is used to adjust the importance of the word and consider the semantic association of the word with other words in the vector (the closer the association, the higher the dynamic weight), making up for the deficiency of the static weight which only focuses on individual importance. The dynamic weight of each word is the aggregation result (usually taking the average or weighted sum) of the cosine similarity of the word with all other words in the vector, reflecting the semantic association strength of the word in the entire vector.Finally, the difference in magnitude between static and dynamic weights is eliminated by weight normalization, and then the vector is weighted and scaled by matrix multiplication. This ultimately strengthens the important and closely related semantic features in the vector, resulting in a log representation vector with deeper semantic connections. The preferred normalization method here is the Min-Max normalization method, which is existing technology and will not be specifically described in this application. Then, the normalized static and dynamic weights are weighted and combined to obtain a fusion weight matrix. Next, the second concatenated log representation vector is multiplied by the fusion weight matrix by matrix multiplication to achieve weighted scaling of each word vector. In other words, the original concatenated vector is optimized into a representation that better reflects deep semantic connections. This vector retains key information while strengthening the semantic connections between words, which can significantly improve the ability of the subsequent log anomaly detection model to capture hidden association patterns.

[0050] It should be noted that for entries without a preset template, due to the intertwining of dynamic parameters and text, they are difficult to separate. Therefore, dynamic self-assigned parameters are precisely extracted and replaced with uniform placeholders to generate intermediate log entries, while a temporary template framework is constructed for them. Since the meaning of dynamic parameters in log entries without a preset template depends on the contextual semantics, a mapping relationship between log text words and dynamic parameters is established through grammatical category matching. Then, the log vector and parameter semantic vector are concatenated to form a feature vector that integrates parameter values ​​and semantic environment. Simultaneously, this application innovatively uses TF-IDF to calculate static word weights and cosine similarity to calculate dynamic weights. After fusion, a weighted vector is generated to strengthen key features and suppress noise, thereby obtaining a deep feature vector that integrates semantics and parameters.

[0051] 203. Obtain the total training set.

[0052] In this embodiment, the total training set is first obtained, i.e., a normal sample set is first obtained, which includes multiple normal logs, and each normal log includes multiple normal log entry representation vectors, with each normal log entry representation vector marked as 1; an unlabeled log set is also obtained, which includes multiple unlabeled logs, and each unlabeled log includes multiple unlabeled log entry representation vectors; then, unlabeled log samples with the same number of samples as the normal sample set are randomly selected from the unlabeled log set. These unlabeled log samples with the same number of samples as the normal sample set are then combined with the normal sample set to form a balanced log sample set. In a balanced log sample set, all normal log entries are represented by vectors labeled with 1, and all unlabeled log entries are represented by vectors labeled with 0. The initial random forest classifier is trained using the balanced log sample set, and its parameters are updated to obtain a target random forest classifier. The unlabeled log set is then input into the target random forest classifier to obtain the labeled log set output by the target random forest classifier. In this labeled log set, each log entry's vector is labeled with an anomaly probability label. The normal sample set and the labeled log set are combined to form the total training set, where all log entry's vectors in each total training set are labeled with anomaly probability labels. For simplicity, the above description does not include iterative training of the model. In practical applications, iterative methods are typically used to train the random forest classifier. In each iteration, a different subset is randomly sampled; that is, each time, unlabeled log samples with the same number of samples as the normal sample set are randomly selected from the unlabeled log set. This ensures that the training set used each time is different. Through multiple iterations, the model parameters are gradually adjusted from a randomly initialized state to a better value, and the error continuously decreases until the target state is reached, ultimately resulting in a target random forest classifier with strong generalization ability. By inputting the unlabeled log set into the target random forest classifier, a log set with anomaly probability labels can be obtained. Since the model has been trained, these anomaly probability labels can accurately reflect the state of the log entries.

[0053] 204. Use the total training set to train the initial log anomaly detection model to obtain the pre-trained log anomaly detection model.

[0054] In this embodiment, the initial log anomaly detection model comprises a first-layer bidirectional gated recurrent unit, a second-layer bidirectional gated recurrent unit, and a nonlinear anomaly classifier connected in sequence. The first-layer bidirectional gated recurrent unit extracts local temporal features of the log sequence (such as parameter changes between adjacent logs and state fluctuations within a short period of time). The second-layer bidirectional gated recurrent unit is used to capture global temporal correlations (such as trend changes over a long period of time and anomaly accumulation across logs) based on the features of the first layer. The nonlinear anomaly classifier is used to map high-level features to anomaly probabilities (0~1, 1 represents anomaly), and it includes a fully connected layer and an activation function (such as sigmoid). The initial log anomaly detection model is trained using the total training set. The first-layer bidirectional gated recurrent unit and the second-layer bidirectional gated recurrent unit gradually master the temporal correlation features of the logs, and the nonlinear classifier can also accurately map the anomaly probabilities, ultimately forming a pre-trained log anomaly detection model with strong generalization ability, providing core support for log anomaly detection in subsequent real-world scenarios.

[0055] 205. Input the log representation vector with deep semantic association features corresponding to each log entry into the pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model.

[0056] For example, the log representation vector is X = [v1, v2, v3, v4, v5] (v1-v5 are the semantic vectors of the logs, and v3 is the abnormal log of a sudden temperature rise). The normal fluctuations from v1 to v2 and the sudden changes from v2 to v3 are captured by the first layer of bidirectional gated recurrent units. The abnormal continuous trend from v3 to v4 to v5 is associated by the second layer of bidirectional gated recurrent units. The nonlinear layer anomaly classifier outputs anomaly probability = 0.92.

[0057] 206. Obtain the probability threshold. When the probability of any log entry among multiple log entries exceeds the probability threshold, the current log is determined to be an abnormal log.

[0058] In this embodiment of the application, after determining that the current log is an abnormal log, a corresponding alarm (such as SMS, email, or audible and visual alarm) is triggered according to the severity of the abnormality (e.g., high temperature is high risk, and log format error is low risk), so as to facilitate relevant technical personnel to handle the problem in a timely manner and prevent the abnormality from spreading.

[0059] Furthermore, as Figure 1 To specifically implement the method, this application provides a log anomaly detection device, such as... Figure 3 As shown, it includes: extraction module 301, conversion module 302, input module 303, and determination module 304.

[0060] The extraction module 301 is used to obtain the current log including multiple log entries, and to extract fixed template information, dynamically assigned parameters, or dynamically assigned parameters from each log entry using a template parsing method.

[0061] The transformation module 302 is used to obtain a log representation vector with deep semantic association features for each log entry based on the fixed template information, dynamic self-assigned parameters, or dynamic self-assigned parameters corresponding to each log entry using a pre-trained language model.

[0062] The input module 303 is used to input the log representation vector with deep semantic association features corresponding to each log entry into the pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model.

[0063] The determining module 304 is used to obtain a probability threshold, and when the probability of log anomaly corresponding to any log entry among the plurality of log entries exceeds the probability threshold, the current log is determined to be an abnormal log.

[0064] In specific application scenarios, the extraction module 301 is also used to extract dynamically self-assigned parameters from each log entry based on regular expressions, and replace the dynamically self-assigned parameters in each log entry with uniform placeholders to obtain intermediate log entries that retain fixed text portions; obtain a log template set including multiple log templates based on the template parsing method, and determine the log category of each intermediate log entry using the log templates, wherein the log category of the intermediate log entry is either a log entry with a preset template or a log entry without a preset template; extract fixed template information and dynamically self-assigned parameters from each log entry with a preset template based on the template parsing method; or, extract dynamically self-assigned parameters from each log entry without a preset template based on the template parsing method.

[0065] In specific application scenarios, the extraction module 301 is further used to calculate the character length of the fixed text portion in each intermediate log entry based on the template parsing method, obtain the character length values ​​of multiple intermediate log entries, and group at least one intermediate log entry with the same character length value into one category to obtain multiple first intermediate log entry subsets, wherein each character length corresponds to one first intermediate log entry subset; for each first intermediate log entry subset, the following operations are performed: using the template parsing method, at least one intermediate log entry in the first intermediate log entry subset is classified according to the log start word to obtain at least one second intermediate log entry subset, wherein each log start word corresponds to one second intermediate log entry subset; for each intermediate log entry in each second intermediate log entry subset, the following operations are performed: based on the template parsing... The method obtains a log template set including multiple log templates, compares an intermediate log entry with each log template in the set word by word, determines the number of words that match each log template in the intermediate log entry, counts the total number of words in each log template, calculates the ratio of the number of words to the total number of words in the set, and obtains multiple similarity scores. If one of the multiple similarity scores exceeds a score threshold, the log category of the intermediate log entry is determined to be a log entry with a preset template. If none of the multiple similarity scores exceeds the score threshold, the intermediate log entry is further validated for log type, and a type validation result is obtained. The type validation result is that the log type of the intermediate log entry is a log entry with a preset template or the log type of the intermediate log entry is a log entry without a preset template.

[0066] In a specific application scenario, the extraction module 301 is further configured to create a new log template for the intermediate log entry based on the template parsing method; calculate the structural similarity between the new log template and each log template in the log template set based on a text matching algorithm to obtain multiple structural similarity values; obtain a similarity threshold based on the template parsing method; when a structural similarity value among the multiple structural similarity values ​​is detected to exceed the similarity threshold, determine that the log type of the intermediate log entry is a log entry with a preset template; when no structural similarity value among the multiple structural similarity values ​​is detected to exceed the similarity threshold, determine that the log type of the intermediate log entry is a log entry without a preset template.

[0067] In specific application scenarios, the conversion module 302 is also used to determine whether the log entry is a log entry with a preset template or a log entry without a preset template; for each log entry with a preset template, the following operations are performed: based on the fixed template information and dynamic self-assigned parameters corresponding to the log entry with the preset template, the log representation vector with deep semantic association features is obtained using the pre-trained language model;

[0068] For each log entry without a preset template, perform the following operations: based on the dynamically assigned parameters corresponding to the log entry without a preset template, use the pre-trained language model to obtain a log representation vector with deep semantic association features.

[0069] In specific application scenarios, the conversion module 302 is further used to convert the fixed template information and dynamically self-assigned parameters corresponding to the preset template log entries into a fixed template information semantic representation vector and a self-assigned parameter semantic representation vector based on the CodeBERT model; to label the grammatical category of each first word in the fixed template information semantic representation vector and the grammatical category of each second word in the self-assigned parameter semantic representation vector based on a part-of-speech tagging algorithm; to match the grammatical category of each first word in the fixed template information semantic representation vector with the grammatical category of each second word in the self-assigned parameter semantic representation vector to determine the mapping relationship between the first word and the second word; and to convert the fixed template information semantic representation vector into a fixed template information semantic representation vector based on the mapping relationship between the first word and the second word. The log representation vector and the self-assigned parameter semantic representation vector are concatenated to obtain a first concatenated log representation vector. The static weight of each word in the first concatenated log representation vector is calculated based on the term frequency-inverse document frequency algorithm. The cosine similarity between any two words in the first concatenated log representation vector is calculated based on the cosine similarity algorithm, and the dynamic weight of each word in the first concatenated log representation vector is adjusted using the cosine similarity. Based on the static and dynamic weights of each word in the first concatenated log representation vector, a fusion weight matrix is ​​obtained using a weight normalization algorithm. Finally, the first concatenated log representation vector and the fusion weight matrix are weighted and scaled using matrix multiplication to obtain the log representation vector with deep semantic association features.

[0070] In specific application scenarios, the conversion module 302 is further used to annotate the grammar category of each third word in the log entries without a preset template based on a part-of-speech analysis algorithm, and to annotate the grammar category of the fourth word in the dynamically self-assigned parameters corresponding to the log entries without a preset template; to convert the log entries without preset templates with annotated grammar types into log vectors without preset templates based on the CodeBERT model, and to convert the dynamically self-assigned parameters with annotated grammar types into semantic representation vectors of intermediate self-assigned parameters; to match the grammar category of each third word in the log vectors without preset templates with the grammar category of each fourth word in the semantic representation vectors of intermediate self-assigned parameters, and to determine the mapping relationship between the third and fourth words; based on the mapping relationship between the third and fourth words, to convert the log entries without preset templates into log vectors without preset templates. The template log vector and the intermediate self-assigned parameter semantic representation vector are concatenated to obtain a second concatenated log representation vector. The static weight of each word in the second concatenated log representation vector is calculated based on the term frequency-inverse document frequency algorithm. The cosine similarity between any two words in the second concatenated log representation vector is calculated based on the cosine similarity algorithm, and the dynamic weight of each word in the second concatenated log representation vector is adjusted using the cosine similarity. Based on the static and dynamic weights of each word in the second concatenated log representation vector, a fusion weight matrix is ​​obtained using a weight normalization algorithm. Finally, the second concatenated log representation vector and the fusion weight matrix are weighted and scaled using matrix multiplication to obtain the log representation vector with deep semantic association features.

[0071] In specific application scenarios, such as Figure 4 As shown, the device also includes an acquisition module 305.

[0072] The acquisition module 305 is used to acquire a normal sample set, which includes multiple normal logs, each of which includes multiple normal log entry representation vectors, and each normal log entry representation vector is marked as 1; acquire an unlabeled log set, which includes multiple unlabeled logs, each of which includes multiple unlabeled log entry representation vectors; randomly select unlabeled log samples from the unlabeled log set with the same number of samples as the normal sample set, and combine the unlabeled log samples with the normal sample set to form a balanced log sample set, wherein all... Normal log entries are represented by a vector labeled as 1, and all unlabeled log entries in the balanced log sample set are represented by vectors labeled as 0. The initial random forest classifier is trained using the balanced log sample set, and its parameters are updated to obtain a target random forest classifier. The unlabeled log set is then input into the target random forest classifier to obtain a labeled log set output by the target random forest classifier, wherein each log entry representation vector in the labeled log set is labeled with an anomaly probability label. The normal sample set and the labeled log set are combined to form a total training set, wherein all log entry representation vectors in each total training set are labeled with anomaly probability labels.

[0073] In specific application scenarios, such as Figure 5 As shown, the device also includes a training module 306.

[0074] The training module 306 is used to obtain an initial log anomaly detection model, which includes a first-layer bidirectional gated recurrent unit, a second-layer bidirectional gated recurrent unit, and a nonlinear anomaly classifier connected in sequence. The first-layer bidirectional gated recurrent unit, the second-layer bidirectional gated recurrent unit, and the nonlinear anomaly classifier in the initial log anomaly detection model are trained using the total training set, and the model parameters are updated to obtain the pre-trained log anomaly detection model.

[0075] The apparatus provided in this application extracts fixed template information and dynamically assigned parameters, or dynamically assigned parameters, from each log entry using a template parsing method via an extraction module. Then, a conversion module uses a pre-trained language model to obtain a log representation vector with deep semantic association features for each log entry based on the fixed template information and dynamically assigned parameters corresponding to each log entry. Subsequently, an input module inputs the log representation vector with deep semantic association features for each log entry into a pre-trained log anomaly detection model to obtain the log anomaly probability for each log entry output by the pre-trained log anomaly detection model. Finally, a determination module determines the current log as an anomalous log when the log anomaly probability for any log entry among multiple log entries exceeds a probability threshold. For log entries with preset templates, this application extracts fixed template information and dynamically assigned parameters from each log entry using a template parsing method. Then, a pre-trained language model is used to not only deeply mine the fixed template information, preserving the detection efficiency of structured logs, but also to semantically activate the dynamically assigned parameters, releasing the hidden value of parameters in unstructured logs. For log entries without preset templates, the template parsing method extracts dynamically assigned parameters from each log entry, and the pre-trained language model deeply binds the dynamically assigned parameters to the semantic description of the log entry without a preset template. Furthermore, the pre-trained log anomaly detection model in this application is pre-trained and can accurately understand the inherent relationship patterns in log representation vectors with deep semantic association features, especially its ability to capture hidden anomalies, accurately outputting the log anomaly probability corresponding to each log entry, thus improving the sensitivity and accuracy of log anomaly detection.

[0076] It should be noted that other corresponding descriptions of the functional units involved in the log anomaly detection method provided in this embodiment of the invention can be found in [reference needed]. Figure 1 and Figures 3 to 5 The corresponding description in [the document] will not be repeated here.

[0077] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0078] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.< / num> < / id> < / time> < / num> < / id> < / time> < / num> < / id> < / time> < / id> < / num> < / num> < / id> < / num> < / id> < / time>

Claims

1. A log anomaly detection method, characterized by, The method comprises the following steps: obtaining a current log comprising a plurality of log entries, extracting fixed template information, dynamic self-assignment parameters or dynamic self-assignment parameters from each log entry by using a template parsing method; based on the fixed template information, dynamic self-assignment parameters or dynamic self-assignment parameters corresponding to each log entry, using a pre-trained language model to obtain a log representation vector with deep semantic association features corresponding to each log entry; inputting the log representation vector with deep semantic association features corresponding to each log entry into a pre-trained log anomaly detection model to obtain a log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model; obtaining a probability threshold, and determining that the current log is an abnormal log when it is detected that the log anomaly probability corresponding to any log entry in the plurality of log entries exceeds the probability threshold; the method comprises the following steps: determining whether the log type of the log entry is a preset template log entry or a non-preset template log entry; for each preset template log entry, the following operations are performed: based on the fixed template information and dynamic self-assignment parameters corresponding to the preset template log entry, a log representation vector with deep semantic association features is obtained by using the pre-trained language model; for each non-preset template log entry, the following operations are performed: based on the dynamic self-assignment parameters corresponding to the non-preset template log entry, a log representation vector with deep semantic association features is obtained by using the pre-trained language model; the method comprises the following steps: based on the CodeBERT model, the fixed template information and the dynamic self-assignment parameters corresponding to the preset template log entry are converted into a fixed template information semantic representation vector and a self-assignment parameter semantic representation vector; based on a part-of-speech analysis algorithm, each first word in the fixed template information semantic representation vector is labeled with a grammatical category, and each second word in the self-assignment parameter semantic representation vector is labeled with a grammatical category; matching the grammatical category of each first word in the fixed template information semantic representation vector with the grammatical category of each second word in the self-assignment parameter semantic representation vector to determine the mapping relationship between the first word and the second word; based on the mapping relationship between the first word and the second word, the fixed template information semantic representation vector and the self-assignment parameter semantic representation vector are spliced to obtain a first spliced log representation vector; based on the term frequency-inverse document frequency algorithm, the static weight of each word in the first spliced log representation vector is calculated; based on the cosine similarity algorithm, the cosine similarity between any two words in the first spliced log representation vector is calculated, and the dynamic weight of each word in the first spliced log representation vector is adjusted by using the cosine similarity; The first spliced log representation vector is weighted and scaled with a fusion weight matrix obtained by using a weight normalization algorithm based on the static weight and the dynamic weight of each word in the first spliced log representation vector, and the first spliced log representation vector is weighted and scaled with the fusion weight matrix through matrix multiplication, to obtain the log representation vector with deep semantic correlation features.

2. The log anomaly detection method of claim 1, wherein, The extracting fixed template information, dynamic self-assignment parameters, or dynamic self-assignment parameters from each log entry by using the template analysis method includes: The dynamic self-assignment parameters are extracted from each log entry based on a regular expression, and the dynamic self-assignment parameters in each log entry are replaced with a uniform placeholder to obtain an intermediate log entry that retains a fixed text part. The log template set including a plurality of log templates is obtained based on the template analysis method, and the log category of each intermediate log entry is determined by using the log templates, the log category of the intermediate log entry being a preset template log entry or a non-preset template log entry. The fixed template information and the dynamic self-assignment parameters are extracted from each preset template log entry based on the template analysis method; or The dynamic self-assignment parameters are extracted from each non-preset template log entry based on the template analysis method.

3. The log anomaly detection method of claim 2, wherein, The log template set including a plurality of log templates is obtained based on the template analysis method, and the log category of each intermediate log entry is determined by using the log templates, the log category of the intermediate log entry being a preset template log entry or a non-preset template log entry. The character length values of the plurality of intermediate log entries are obtained by counting the character length of the fixed text part in each intermediate log entry based on the template analysis method, at least one intermediate log entry with the same character length value in the plurality of intermediate log entries is classified into a category, and a plurality of first intermediate log entry subsets are obtained, wherein each character length corresponds to a first intermediate log entry subset. For each first intermediate log entry subset, the following operations are performed: at least one intermediate log entry in the first intermediate log entry subset is classified according to a log starting vocabulary by using the template analysis method to obtain at least one second intermediate log entry subset, wherein each log starting vocabulary corresponds to a second intermediate log entry subset. For each intermediate log entry in each second intermediate log entry subset, the following operations are performed: based on the template parsing method, a log template set including a plurality of log templates is obtained, an intermediate log entry is compared with each log template in the log template set bit by bit, the number of words matching the intermediate log entry and each log template in the plurality of log templates is determined, the total number of words of each log template is counted, the ratio of the number of words to the corresponding total number of words is calculated, a plurality of similarity scores are obtained, if a similarity score in the plurality of similarity scores exceeds a score threshold, it is determined that the log category of the intermediate log entry is a preset template log entry, if each similarity score in the plurality of similarity scores does not exceed the score threshold, further log type verification is performed on the intermediate log entry, and a type verification result is obtained, the type verification result being that the log type of the intermediate log entry is a preset template log entry or the log type of the intermediate log entry is a non-pre-set template log entry.

4. The log anomaly detection method of claim 3, wherein, The further log type verification on the intermediate log entry to obtain the type verification result comprises: creating a new log template for the intermediate log entry based on the template parsing method; calculating the structural similarity of the new log template and each log template in the log template set based on a text matching algorithm to obtain a plurality of structural similarity values; obtaining a similarity threshold based on the template parsing method, when it is detected that a structural similarity value in the plurality of structural similarity values exceeds the similarity threshold, determining that the log type of the intermediate log entry is a preset template log entry, and when it is detected that each structural similarity value in the plurality of structural similarity values does not exceed the similarity threshold, determining that the log type of the intermediate log entry is a non-pre-set template log entry.

5. The log anomaly detection method of claim 1, wherein, The obtaining of the log representation vector with deep semantic association characteristics based on the pre-trained language model comprises: annotating each third word in the non-pre-set template log entry and a fourth word in the dynamic self-assignment parameter corresponding to the non-pre-set template log entry with a grammatical category based on a part-of-speech analysis algorithm; converting the non-pre-set template log entry with the annotated grammatical type into a non-pre-set template log vector and converting the dynamic self-assignment parameter with the annotated grammatical type into an intermediate self-assignment parameter semantic representation vector based on a CodeBERT model; matching the grammatical category of each third word in the non-pre-set template log vector with the grammatical category of each fourth word in the intermediate self-assignment parameter semantic representation vector to determine the mapping relationship between the third word and the fourth word; based on the mapping relationship between the third word and the fourth word, splicing the non-pre-set template log vector and the intermediate self-assignment parameter semantic representation vector to obtain a second spliced log representation vector; calculating the static weight of each word in the second spliced log representation vector based on a term frequency-inverse document frequency algorithm; calculating cosine similarity between any two words in the second spliced log representation vector based on a cosine similarity algorithm, and adjusting the dynamic weight of each word in the second spliced log representation vector based on the cosine similarity; based on the static weight and the dynamic weight of each word in the second spliced log representation vector, obtaining a fusion weight matrix by using a weight normalization algorithm, and performing weighted scaling on the second spliced log representation vector and the fusion weight matrix by matrix multiplication, to obtain the log representation vector with deep semantic correlation characteristics.

6. The log anomaly detection method of claim 1, wherein, Before the log representation vector with deep semantic correlation characteristics corresponding to each log entry is input into the pre-trained log anomaly detection model to obtain the log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model, the method further includes: obtaining a normal sample set, wherein the normal sample set includes a plurality of normal logs, and each normal log includes a plurality of normal log entry representation vectors, and each normal log entry representation vector is marked as 1; obtaining an unlabeled log set, wherein the unlabeled log set includes a plurality of unlabeled logs, and each unlabeled log includes a plurality of unlabeled log entry representation vectors; randomly selecting a number of unlabeled log samples consistent with the number of samples in the normal sample set from the unlabeled log set, and combining the number of unlabeled log samples consistent with the number of samples in the normal sample set with the normal sample set to form a balanced log sample set, wherein all normal log entry representation vectors in the balanced log sample set are marked as 1, and all unlabeled log entry representation vectors in the balanced log sample set are marked as 0; training an initial random forest classifier using the balanced log sample set, updating parameters, and obtaining a target random forest classifier; inputting the unlabeled log set into the target random forest classifier to obtain a labeled log set output by the target random forest classifier, wherein each log entry representation vector in the labeled log set is marked with an anomaly probability label; combining the normal sample set and the labeled log set to form a total training set, wherein all log entry representation vectors in each total training set are marked with an anomaly probability label.

7. The log anomaly detection method of claim 6, wherein, After the normal sample set and the labeled log set are combined to form the total training set, the method further includes: obtaining an initial log anomaly detection model, wherein the initial log anomaly detection model includes a first layer of bidirectional gated recurrent units, a second layer of bidirectional gated recurrent units, and a nonlinear layer anomaly classifier connected in sequence; training the first layer of bidirectional gated recurrent units, the second layer of bidirectional gated recurrent units, and the nonlinear layer anomaly classifier in the initial log anomaly detection model using the total training set, updating model parameters, and obtaining the pre-trained log anomaly detection model.

8. A log anomaly detection apparatus characterized by comprising: The method includes: an extraction module configured to obtain a current log including a plurality of log entries, and extract fixed template information, dynamic self-assignment parameters, or dynamic self-assignment parameters from each log entry by using a template parsing method; The conversion module is configured to obtain, based on the fixed template information, the dynamic self-assignment parameter, or the dynamic self-assignment parameter corresponding to each log entry, a log representation vector corresponding to each log entry by using a pre-trained language model, the log representation vector having deep semantic correlation features; The input module is configured to input the log representation vector corresponding to each log entry into a pre-trained log anomaly detection model, to obtain a log anomaly probability corresponding to each log entry output by the pre-trained log anomaly detection model; The determination module is configured to obtain a probability threshold, and determine that the current log is an abnormal log when it is detected that the log anomaly probability corresponding to any log entry in the plurality of log entries exceeds the probability threshold; The conversion module is further configured to determine whether a log type of a log entry is a preset template log entry or a non-preset template log entry; The conversion module is further configured to, for each preset template log entry, obtain the log representation vector having deep semantic correlation features based on the fixed template information and the dynamic self-assignment parameter corresponding to the preset template log entry by using the pre-trained language model; The conversion module is further configured to, for each non-preset template log entry, obtain the log representation vector having deep semantic correlation features based on the dynamic self-assignment parameter corresponding to the non-preset template log entry by using the pre-trained language model; The conversion module is further configured to convert the fixed template information and the dynamic self-assignment parameter corresponding to the preset template log entry into a fixed template information semantic representation vector and a self-assignment parameter semantic representation vector based on a CodeBERT model; The conversion module is further configured to label a grammatical category of each first word in the fixed template information semantic representation vector and a grammatical category of each second word in the self-assignment parameter semantic representation vector based on a part-of-speech analysis algorithm; The conversion module is further configured to match the grammatical category of each first word in the fixed template information semantic representation vector with the grammatical category of each second word in the self-assignment parameter semantic representation vector, to determine a mapping relationship between the first word and the second word; The conversion module is further configured to splice the fixed template information semantic representation vector and the self-assignment parameter semantic representation vector based on the mapping relationship between the first word and the second word, to obtain a first spliced log representation vector; The conversion module is further configured to calculate a static weight of each word in the first spliced log representation vector based on a term frequency-inverse document frequency algorithm; The conversion module is further configured to calculate a cosine similarity between any two words in the first spliced log representation vector based on a cosine similarity algorithm, and adjust a dynamic weight of each word in the first spliced log representation vector by using the cosine similarity; The transformation module is further configured to obtain a fusion weight matrix by using a weight normalization algorithm based on the static weight and the dynamic weight of each word in the first spliced log representation vector, and obtain the log representation vector with deep semantic correlation features by weighting and scaling the first spliced log representation vector and the fusion weight matrix through matrix multiplication.

Images