Abnormal log detection method and device, and electronic equipment

Abstract

The application discloses a kind of detection methods, device and electronic equipment of abnormal log. Among them, the method comprises: obtaining log file to be detected;Using abnormal log identification model to predict the target prediction log template to which target log in log file to be detected belongs, wherein the target log is any one log in log file to be detected, and abnormal log identification model is used to predict the probability of the log template to which target log belongs according to the context of target log;According to the frequency of occurrence of target log in training set, the first position of target log in training set is determined, and the second position of target prediction log template in prediction log template set is determined;According to the first position and the second position, it is determined whether target log is abnormal or not.The application solves the technical problem that the identification ability of risk operation log in the log abnormal detection technology based on keyword in the related art is poor.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and electronic device for detecting abnormal logs. Background Technology

[0002] During nighttime operations, such as equipment cutovers, risky processes are prone to anomalies, which are reflected in the real-time log sequences generated by the equipment. For equipment safety, it is necessary to detect anomalies in the equipment logs as quickly as possible and take timely remedial measures. However, since the logs generated during high-risk periods are often massive, manual screening of daily logs is not feasible. Therefore, it is necessary to research automated algorithms to identify daily logs.

[0003] Traditional log anomaly detection algorithms are based on keyword matching technology, which means that the algorithm pre-sets some suspicious keywords and considers logs containing such keywords as anomalies. However, this keyword-based log anomaly detection technology has poor ability to identify risky operation logs.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for detecting abnormal logs, in order to at least solve the technical problem that keyword-based log anomaly detection technology has poor identification ability for risky operation logs in related technologies.

[0006] According to one aspect of the embodiments of this application, an abnormal log detection method is provided, comprising: acquiring a log file to be detected; using an abnormal log recognition model to predict the target predicted log template to which a target log in the log file to be detected belongs, wherein the target log is any log in the log file to be detected, and the abnormal log recognition model is used to predict the probability of the target log belonging to the log template based on the context of the target log; determining a first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and determining a second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model; and determining whether the target log is abnormal based on the first position and the second position.

[0007] Optionally, after obtaining the log file to be detected, the method further includes: identifying the variable parameters in each log entry in the log file to be detected; converting the variable parameters into wildcards according to the log template set to obtain the template log file corresponding to the log file to be detected; and sorting the template log file according to the time series to obtain the log sequence to be detected.

[0008] Optionally, after obtaining the template log file corresponding to the log file to be detected, the method further includes: determining the string contained in each template log in the template log file; converting the string into a tree structure, wherein the root node of the tree structure is the first character of the string; and storing the tree structure in the database.

[0009] Optionally, after obtaining the template log file corresponding to the log file to be detected, the method further includes: obtaining a first template log and a second template log from the template log file, wherein the first template log and the second template log are any two template logs in the template log file; determining the similarity between the first template log and the second template log; if the similarity is greater than the similarity threshold, determining that the first template log and the second template log are of the same type of template; merging the first template log and the second template log to obtain the updated template log file.

[0010] Optionally, determining the similarity between the first template log and the second template log includes: obtaining a first string from the first template log and a second string from the second template log; simultaneously comparing the first and second strings sequentially starting from the first character to determine the substrings in the first string that are identical to the characters in the second string from the first position to the second position, where the first position is the position of the first character in the first or second string, and the second position is the position of the last character in the shortest string between the first and second strings; determining the length of each substring and identifying the longest substring as the target substring; and determining the similarity between the first template log and the second template log based on the target substring.

[0011] Optionally, the abnormal log recognition model is used to predict the target predicted log template to which the target log in the log file to be detected belongs, including: sorting the log file to be detected according to the time series to obtain the log sequence to be detected; inputting the log sequence to be detected into the embedding layer of the abnormal log recognition model to obtain a first vector corresponding to the log sequence to be detected; masking the target vector corresponding to the target log from the first vector to obtain a second vector; inputting the second vector into the encoder of the abnormal log recognition model to obtain an output vector, wherein the encoder is trained based on the context information in the historical log file where the target log is located, and the output vector is used to represent the probability that the target log belongs to different log templates; and determining the log template corresponding to the maximum value in the output vector as the target predicted log template.

[0012] Optionally, determining whether the target log is abnormal based on the first position and the second position includes: determining the difference between the second position and the first position; if the difference is greater than a preset threshold, determining the target log as an abnormal log; if the difference is less than or equal to the preset threshold, determining the target log as a normal log.

[0013] According to another aspect of the embodiments of this application, an abnormal log detection device is also provided, comprising: an acquisition module for acquiring a log file to be detected; a prediction module for using an abnormal log recognition model to predict the target predicted log template to which a target log in the log file to be detected belongs, wherein the target log is any log in the log file to be detected, and the abnormal log recognition model is used to predict the probability of the log template to which the target log belongs based on the context of the target log; a first determination module for determining a first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and determining a second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model; and a second determination module for determining whether the target log is abnormal based on the first position and the second position.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: acquiring a log file to be detected; using an abnormal log recognition model to predict the target predicted log template to which a target log in the log file to be detected belongs, wherein the target log is any log in the log file to be detected, and the abnormal log recognition model is used to predict the probability of the log template to which the target log belongs based on the context of the target log; determining a first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and determining a second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model; and determining whether the target log is abnormal based on the first position and the second position.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned abnormal log detection method by running the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described method for detecting abnormal logs.

[0017] In this embodiment, the following steps are taken: A log file to be detected is acquired; an anomaly log identification model is used to predict the target predicted log template to which the target log in the log file belongs. The target log is any log entry in the log file to be detected. The anomaly log identification model predicts the probability of the target log belonging to a log template based on the context of the target log. The first position of the target log in the training set is determined based on its frequency of occurrence in the training set, and the second position of the target predicted log template in the predicted log template set is determined. The training set is the set of logs used to train the anomaly log identification model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the anomaly log identification model. The presence of anomalies in the target log is determined based on the first and second positions, thus achieving the goal of automatically identifying anomaly logs. This improves the efficiency and accuracy of log analysis and solves the technical problem that keyword-based log anomaly detection technology has poor identification capabilities for risk operation logs in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an abnormal log detection method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of an abnormal log detection method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of template extraction according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram illustrating a log template classification according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of a prefix tree structure for a Spell algorithm according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of a BERT model according to an embodiment of this application;

[0025] Figure 7This is a schematic diagram illustrating how a BERT model learns log context patterns according to an embodiment of this application;

[0026] Figure 8 This is a structural diagram of an abnormal log detection device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0030] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0031] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language model that differs from traditional models that use unidirectional language models or shallow concatenation of two unidirectional models for pre-training. Instead, it employs a novel training task to generate deep bidirectional language representations. BERT takes long sentences as input, and after processing by intermediate layers, it generates a series of vectors representing the meaning of each word in the sentence and the overall meaning of the sentence. These vectors can then be fed into other output layers for various downstream tasks, such as named entity recognition, sentence classification, and keyword extraction.

[0032] The use of keyword-based log anomaly detection techniques in related technologies has the following drawbacks:

[0033] (1) Whether different keywords are abnormal or not will change over time and with changes in business scenarios. They are not universal and need to be updated in real time.

[0034] (2) Keyword technology lacks the ability to identify abnormal logs that have never been seen before;

[0035] (3) Keyword technology can only make judgments on a single log entry, ignoring the context in which the log entry is located. In fact, the same log entry appearing in different contexts will affect the judgment of whether it is abnormal or not to a certain extent. Therefore, an excellent log anomaly detection model must have the ability to judge whether a log entry is abnormal based on the context.

[0036] To address the problems existing in related technologies, such as the inability of keyword-based log anomaly detection techniques to handle semi-structured data, lack of ability to identify unseen risk operation logs, and neglect of the context information of the logs, embodiments of this application provide an anomaly log detection method. This method can be run on... Figure 1 The computer terminal shown is described below.

[0037] The abnormal log detection method provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an anomaly log detection method is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the abnormal log detection method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned abnormal log detection method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0042] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0043] In the above operating environment, this application provides an embodiment of an abnormal log detection method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] Figure 2 This is a flowchart of an abnormal log detection method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S202: Obtain the log file to be detected.

[0046] In step S202 above, the format of the log file to be detected can be, for example, a semi-structured log file. A semi-structured log file is a record file or collection of files that records system operation events, containing both structured and unstructured data. The structured data typically includes fixed information (or fixed fields) such as timestamps, IP addresses, and levels, which can be easily identified by their positions. The unstructured part is mainly the main text, the content of which is variable and needs to be parsed according to the specific application scenario.

[0047] Step S204: Use an abnormal log recognition model to predict the target predicted log template to which the target log in the log file to be detected belongs. The target log is any log in the log file to be detected. The abnormal log recognition model is used to predict the probability of the log template to which the target log belongs based on the context of the target log.

[0048] In step S204 above, the abnormal log identification model is a pre-trained machine learning model designed to identify anomalies in log files. This model can be built based on deep learning or other machine learning algorithms, learning from large amounts of historical log data how to distinguish between normal and abnormal logs. For a log file to be detected, the model attempts to predict which log template each log entry (referred to here as the "target log") most likely belongs to. Here, a "log template" typically refers to a formatted structure or pattern of logs, representing a specific type or level of log information. By predicting the template to which the target log belongs, the nature and possible meaning of the log can be indirectly understood. Furthermore, the model considers not only the content of the target log itself but also its contextual information (such as the preceding and following log content, timestamps, etc.) to more accurately predict which template the target log most likely belongs to. Simultaneously, the model outputs a probability value, representing the confidence level that the target log belongs to a specific template.

[0049] It should be noted that the target log is a log randomly selected or processed sequentially from the log file to be detected. This process is iterative, and the model will perform the same processing on each log in the file.

[0050] Step S206: Determine the first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and determine the second position of the target predicted log template in the predicted log template set. The training set is the set of logs used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model.

[0051] Step S208: Determine whether there is an anomaly in the target log based on the first position and the second position.

[0052] In steps S206 to S208 above, by taking into account both the first and second positions, the abnormal situation of the target log can be assessed.

[0053] Through steps S202 to S208 above, the goal of automatically identifying abnormal logs is achieved, thereby improving the efficiency and accuracy of log analysis. This also solves the technical problem that keyword-based log anomaly detection technology has poor identification capability for risky operation logs. The following is a detailed explanation.

[0054] In step S202 of the above-mentioned abnormal log detection method, after obtaining the log file to be detected, the method further includes the following steps: identifying the variable parameters in each log in the log file to be detected; converting the variable parameters into wildcards according to the log template set to obtain the template log file corresponding to the log file to be detected; and sorting the template log file according to the time series to obtain the log sequence to be detected.

[0055] In this embodiment, the semi-structured log data (i.e., the file format corresponding to the log file to be detected) is first converted into a structured template (i.e., the template log file). For example, a template extraction algorithm based on the Spell algorithm can be used. The goal of the template extraction algorithm is to automatically identify the variable parameter parts (such as IP address, username, node ID, etc.) in the original semi-structured log and convert them into a structured template containing wildcards. Figure 3 This demonstrates an example of a template extraction algorithm automatically identifying template types from semi-structured logs. The left side shows the semi-structured logs, and the right side shows the converted structured templates. It should be noted that the template extraction process described above can occur during training or model inference.

[0056] After template recognition is completed, semi-structured logs can be converted into structured template IDs based on the templates, which can reduce abnormal noise and make it easier for subsequent models to capture the internal correlations of log data. Figure 4 This diagram illustrates how semi-structured log data is converted into a uniquely corresponding structured log template ID. The left side of the arrow represents the semi-structured log data, and the right side represents the structured log template ID. This structured log template ID is... Figure 4 It includes Mode 1 and Mode 2.

[0057] In the above steps, after obtaining the template log file corresponding to the log file to be detected, the method further includes: determining the string contained in each template log in the template log file; converting the string into a tree structure, wherein the root node of the tree structure is the first character of the string; and storing the tree structure in the database.

[0058] In this embodiment of the application, the Spell template extraction algorithm uses a prefix tree structure to store existing structured log templates, such as... Figure 5As shown, the path from the root node to the leaf node represents a corresponding log template. If two logs share a common path, it means they have the same prefix. Since the inference process of the risk operation log anomaly detection model often involves processing hundreds of thousands or even millions of logs, the prefix tree structure can improve the time efficiency of searching and matching within existing templates when matching log templates.

[0059] In the above steps, after obtaining the template log file corresponding to the log file to be detected, the method further includes: obtaining a first template log and a second template log from the template log file, wherein the first template log and the second template log are any two template logs in the template log file; determining the similarity between the first template log and the second template log; if the similarity is greater than the similarity threshold, determining that the first template log and the second template log are of the same type of template; merging the first template log and the second template log to obtain the updated template log file.

[0060] In the above steps, determining the similarity between the first template log and the second template log specifically includes the following steps: obtaining the first string from the first template log and the second string from the second template log; simultaneously comparing the first and second strings sequentially starting from the first character to determine the substrings in the first and second strings where the characters in the first string are completely identical from the first position to the second position, where the first position is the position of the first character in the first or second string, and the second position is the position of the last character in the shortest string between the first and second strings; determining the length of each substring and identifying the longest substring as the target substring; and determining the similarity between the first and second template logs based on the target substring.

[0061] In this embodiment, considering that risk operation logs change over time and with evolving business scenarios, an algorithm for fuzzy matching of log similarity needs to be designed to prevent an excessive number of log template types and reduce the false positive rate of the model. This embodiment uses an algorithm based on LCS (Longest Common Substring), which calculates the length of the longest common substring between two strings and uses it as a similarity index. In the abnormal log identification model, if the similarity between two log templates exceeds a certain threshold, they are considered the same template.

[0062] simSeq = LCS(seq1, seq2)

[0063] Here, simSeq represents the similarity between the first string seq1 and the second string seq2.

[0064] Similarly, since the abnormal log identification model for risky operations often needs to process hundreds of thousands or even millions of log sequences at once during the inference stage, it places very high demands on the algorithm's time performance. To improve the time efficiency of the Spell algorithm in calculating log sequence similarity, this application's embodiment adopts a dynamic programming-based algorithm for LCS calculation. The description of the LCS algorithm based on the dynamic programming idea is as follows:

[0065] 1. Define the state: Let the two sequences be X[0,..,m-1] and Y[0,..,n-1], and let dp[i][j] represent the length of the longest common subsequence of X[0,..,i-1] and Y[0,..,j-1].

[0066] 2. Initialization: When i = 0 or j = 0, that is, when one of the sequences is empty, the length of the longest common subsequence is 0. Therefore, dp[i][0] = 0 (for all i) and dp[0][j] = 0 (for all j).

[0067] 3. State transition equation: For dp[i][j], there are two cases:

[0068] (1) If X[i-1] = Y[j-1], that is, the two characters are the same, then the length of the longest common subsequence is increased by one, that is, dp[i][j] = dp[i-1][j-1] + 1.

[0069] (2) If X[i-1] != Y[j-1], then the longest common subsequence is either in the first i-1 characters of X or in the first j-1 characters of Y. Take the larger of the two, i.e., dp[i][j] = max(dp[i-1][j], dp[i][j-1]).

[0070] 4. Calculate the longest common subsequence: Using the state transition equation above, starting from dp[1][1] and continuing until dp[m][n], the entire dp table can be filled.

[0071] 5. Backtracking to construct LCS: After obtaining the value of dp[m][n], we can backtrack the dp table and construct the specific longest common subsequence based on the value of each position and the relationship between adjacent positions.

[0072] In step S204 of the above-mentioned abnormal log detection method, the abnormal log recognition model is used to predict the target predicted log template to which the target log in the log file to be detected belongs. This includes: sorting the log file to be detected according to the time series to obtain the log sequence to be detected; inputting the log sequence to be detected into the embedding layer of the abnormal log recognition model to obtain a first vector corresponding to the log sequence to be detected; masking the target vector corresponding to the target log from the first vector to obtain a second vector; inputting the second vector into the encoder of the abnormal log recognition model to obtain an output vector, wherein the encoder is trained based on the context information in the historical log file where the target log is located, and the output vector is used to represent the probability that the target log belongs to different log templates; and determining the log template corresponding to the maximum value in the output vector as the target predicted log template.

[0073] This application embodiment uses an abnormal log identification model to predict the target prediction log template to which the target log belongs. During the training process, the model extracts the template from the risk operation logs provided by the user, generates a log sequence in chronological order, and then trains the model. During the inference process, the user provides unknown new risk operation logs. The solution comprehensively judges whether each log is normal or abnormal based on the template (including template content and template frequency) and language model obtained during the training process.

[0074] The following tasks need to be completed in advance during the training of the abnormal log recognition model:

[0075] (1) Users need to prepare a large number of risk operation logs under the same network device and the same professional level. The number should preferably reach more than 100,000. The logs must clearly contain the time information of the log generation.

[0076] (2) Users need to provide the macro format of the logs generated by network devices at this professional level;

[0077] (3) Users need to provide hardware devices that can support deep learning for model training. Ideally, the machine should support GPU-based deep learning model training. If the amount of data is small or the training time requirement is not high, training can also be performed on CPU.

[0078] The specific implementation of the training process is as follows:

[0079] Step 1: The user imports the risk operation log data used for training into the model and informs the model of the professional level of equipment and log format to which this type of data belongs;

[0080] Step 2: Wait for the model to perform structured processing on the semi-structured training logs, extract templates and count frequencies. The statistical results will be saved in the local path, and users can view and fine-tune them themselves.

[0081] Step 3: Train the BERT language model, wait for the model iteration to complete, and save the corresponding weight file in the local path. The loss function descent process during the iteration will be automatically presented to the user in the form of a graph.

[0082] The specific implementation of the reasoning process is as follows:

[0083] Step 1: The user imports the risk operation logs used for inference into the model. The inference logs and training logs must be from the same professional level device and have the same format.

[0084] Step 2: The model and template files are used to convert the risk operation logs into corresponding templates and perform inference to obtain the anomaly label for each risk operation log.

[0085] Step 3: Experts provide follow-up feedback. If a misjudgment occurs, the misjudgment log is returned to the model for online updates.

[0086] In this embodiment, an encoder structure based on the BERT language model is used to identify abnormal logs in risky operations. Furthermore, the traditional BERT language model is optimized by changing the calculation formula at the output end, making it more suitable for risky operation business scenarios. The BERT language model structure is as follows: Figure 6 As shown. At the end of the step preceding the inference stage, the semi-structured log text is transformed into structured log template numbers. These structured log templates serve as input parameters to the BERT model, undergoing transformation in the embedding layer to generate the corresponding vector E (i.e., the first vector). Vector E undergoes a multi-layer encoder structure, involving a series of weight matrix operations and normalization, ultimately generating the output vector T. The output vector T corresponds one-to-one with the input structured log text, with each position of vector T representing the input log text combined with contextual semantic information.

[0087] The original BERT model consists of a default weight matrix (including the embedding and encoder layers) and lacks sufficient understanding of the log domain. Therefore, to enable the BERT model to better understand the contextual patterns in logs, this embodiment requires users to provide a large number of logs from normal risky operation periods (typically exceeding 100,000). The BERT model reads these logs, automatically learns the patterns within them, and fine-tunes the matrix weights to continuously optimize its performance. After a long period of learning, the model's weights are adjusted to understand the contextual patterns of risky operation logs. The weight layer of the BERT model is essentially an M×N matrix multiplication from input to output, where each straight line connecting the input and output nodes represents a value in one dimension of the matrix.

[0088] The BERT model learns log context patterns in the following ways: Figure 7As shown. The model first randomly masks a portion of the input log (e.g., Figure 7 In log 5), this part is invisible to the model. The model needs to use contextual information, combined with its own weight matrix, to predict what template the masked log is. For example, in Figure 7 In the process, the BERT model, after calculating the current weight matrix, considers the obscured logs to have the highest probability of belonging to template 8, reaching 10%.

[0089] The BERT model continuously optimizes its performance by calculating a loss function, the formula for which is as follows:

[0090]

[0091] in, This indicates that after the i-th log in the j-th log sequence is masked, the BERT model uses its own weight matrix W... C and bias matrix b C The calculated prediction y_pred for the obscured position, where h represents the hidden layer vector corresponding to the i-th log in the j-th log sequence after obscuring. To determine the magnitude of the difference between the prediction and the actual result, and thus adjust the weights, the model calculates the loss function as follows:

[0092]

[0093] in, Let M represent the actual template to which the i-th log in the j-th log sequence belongs, M represent the total number of logs, and N represent the total number of log sequences.

[0094] By calculating the loss function and the backpropagation gradient, the BERT model continuously adjusts its weight matrix, enabling it to predict an unknown log template in a sequence of logs better and better.

[0095] In the inference phase, the user inputs a log sequence with unknown labels (the sequence must be tightly continuous in time, and the format and hardware must be the same as the logs used by the model in the inference phase), i.e., the log sequence to be detected mentioned above. Similar to the training phase, each log entry in the log sequence is masked sequentially. Then, the model obtained from the training phase performs calculations and predictions on the entire sequence, inferring the template probability distribution of the masked logs (i.e., the target logs mentioned above). By comparing the actual template of the masked logs with the model's predicted template probability distribution, it can be determined whether each log entry is abnormal.

[0096] In step S208 of the above-mentioned abnormal log detection method, determining whether the target log is abnormal based on the first position and the second position includes: determining the difference between the second position and the first position; if the difference is greater than a preset threshold, determining the target log as an abnormal log; if the difference is less than or equal to the preset threshold, determining the target log as a normal log.

[0097] In this embodiment, when judging logs, in addition to the probability results inferred by the BERT model, the frequency of the log itself appearing in the training data is also considered. Assuming that, according to the BERT language model's inference, the predicted probability of the target log appearing in its true location ranks m-th among all logs (i.e., the second position mentioned above), and in the training set, the frequency of this log itself ranks n-th among all logs (i.e., the first position mentioned above), then the algorithm will ultimately determine whether the log is abnormal using the following formula:

[0098] score = mn

[0099]

[0100] Here, threshold represents the aforementioned preset threshold. The larger the score value, the more likely the target log will appear normally, but its frequency has been reduced after the BERT language model infers its context. If the reduction exceeds a certain threshold, it means that the log should not appear in such a context and should be judged as abnormal.

[0101] After the anomaly log recognition model is trained, experts can manually label logs that the model has misjudged as anomalies based on their own experience and actual business scenarios. The model will provide feedback on the expert's labeling, thereby improving the accuracy of the next judgment.

[0102] The abnormal log detection method provided in this application has the following advantages:

[0103] 1. Semi-structured logs are uniformly converted into structured templates for easier subsequent processing. Specifically, semi-structured log text contains various parameters, which may include technical terms and sensitive information (such as usernames and passwords). Directly using the original text for anomaly detection would lead to numerous inconveniences. This application uses a log template matching algorithm during both training and inference to uniformly convert semi-structured logs into structured templates, thereby enhancing the model's efficiency and confidentiality.

[0104] 2. The training approach based on the BERT model addresses the problem of imbalanced samples in log anomaly detection models. Specifically, in log anomaly detection samples, the ratio of normal samples to anomaly samples often exceeds 10,000:1. Traditional binary classification machine learning algorithms face the problem of imbalanced samples during training. This application uses the BERT language model for training, which can learn the inherent patterns in log distribution even if all training samples are normal, thus reducing the problem of imbalanced training sample ratios.

[0105] 3. Utilizing the contextual information of log entries within the sequence for anomaly detection. Specifically, existing log anomaly detection techniques are generally based on keyword detection, classifying log entries containing certain keywords as anomalies. In reality, even the same log entry can have completely different meanings depending on the context in which it appears. This application, based on the BERT language model, incorporates the log's contextual information as a factor in log anomaly detection, resulting in stronger detection capabilities.

[0106] 4. Anomaly detection is performed by combining the contextual and statistical information of the logs. Specifically, when detecting anomalies, this application combines the contextual information of the logs with the statistical frequency of the logs appearing in the training data, integrates the prior and posterior probabilities, and finally calculates whether the logs are abnormal. This approach has stronger fault tolerance and reduces the frequency of false alarms.

[0107] Figure 8 This is a structural diagram of an anomaly log detection device according to an embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0108] Module 42 is used to acquire the log file to be detected;

[0109] The prediction module 44 is used to predict the target predicted log template to which the target log in the log file to be detected belongs using an abnormal log recognition model. The target log is any log in the log file to be detected, and the abnormal log recognition model is used to predict the probability of the log template to which the target log belongs based on the context of the target log.

[0110] The first determining module 46 is used to determine the first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and to determine the second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model.

[0111] The second determining module 48 is used to determine whether there is an anomaly in the target log based on the first position and the second position.

[0112] By using the acquisition module 42, prediction module 44, first determination module 46, and second determination module 48 in the above-mentioned abnormal log detection device, the purpose of automatically identifying abnormal logs is achieved, thereby realizing the technical effect of improving the efficiency and accuracy of log analysis, and thus solving the technical problem that the keyword-based log anomaly detection technology in related technologies has poor identification ability for risk operation logs.

[0113] In the acquisition module of the above-mentioned abnormal log detection device, the acquisition module is also used to identify the variable parameters in each log in the log file to be detected; convert the variable parameters into wildcards according to the log template set to obtain the template log file corresponding to the log file to be detected; and sort the template log file according to the time sequence to obtain the log sequence to be detected.

[0114] In the acquisition module of the aforementioned abnormal log detection device, the acquisition module is also used to determine the string contained in each template log in the template log file; convert the string into a tree structure, wherein the root node of the tree structure is the first character of the string; and store the tree structure in the database.

[0115] In the acquisition module of the aforementioned abnormal log detection device, the acquisition module is further configured to acquire a first template log and a second template log from the template log file, wherein the first template log and the second template log are any two template logs in the template log file; determine the similarity between the first template log and the second template log; if the similarity is greater than the similarity threshold, determine that the first template log and the second template log are of the same type of template; merge the first template log and the second template log to obtain an updated template log file.

[0116] In the acquisition module of the aforementioned abnormal log detection device, the acquisition module is further used to acquire a first string from a first template log and a second string from a second template log; simultaneously, starting from the first character of the first string and the second string, it compares them sequentially to determine a substring from a first position to a second position where the characters in the first string and the characters in the second string are completely identical, wherein the first position is the position of the first character in the first string or the second string, and the second position is the position of the last character in the shortest string between the first string and the second string; it determines the length of each substring and identifies the longest substring as the target substring; and it determines the similarity between the first template log and the second template log based on the target substring.

[0117] In the prediction module of the aforementioned abnormal log detection device, the prediction module is further used to sort the log files to be detected according to the time series to obtain the log sequence to be detected; input the log sequence to be detected into the embedding layer of the abnormal log recognition model to obtain a first vector corresponding to the log sequence to be detected; mask the target vector corresponding to the target log from the first vector to obtain a second vector; input the second vector into the encoder of the abnormal log recognition model to obtain an output vector, wherein the encoder is trained based on the context information in the historical log file where the target log is located, and the output vector is used to represent the probability that the target log belongs to different log templates; and determine the log template corresponding to the maximum value in the output vector as the target predicted log template.

[0118] In the second determining module of the above-mentioned abnormal log detection device, the second determining module is further used to determine the difference between the second position and the first position; if the difference is greater than a preset threshold, the target log is determined to be an abnormal log; if the difference is less than or equal to the preset threshold, the target log is determined to be a normal log.

[0119] It should be noted that, Figure 8 The abnormal log detection device shown is used to perform... Figure 2 The method for detecting abnormal logs shown above is also applicable to the detection device for abnormal logs, and will not be repeated here.

[0120] This application also provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor is connected to the memory and executes program instructions to perform the following functions: acquiring a log file to be detected; using an anomaly log recognition model to predict the target predicted log template to which a target log in the log file to be detected belongs, wherein the target log is any log in the log file to be detected, and the anomaly log recognition model is used to predict the probability of the target log belonging to the log template based on the context of the target log; determining a first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and determining a second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the anomaly log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the anomaly log recognition model; and determining whether the target log is abnormal based on the first position and the second position.

[0121] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The method for detecting abnormal logs shown above also applies to this electronic device, and will not be repeated here.

[0122] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following method for detecting abnormal logs by running the computer program: acquiring a log file to be detected; using an abnormal log recognition model to predict the target predicted log template to which a target log in the log file belongs, wherein the target log is any log in the log file to be detected, and the abnormal log recognition model is used to predict the probability of the target log belonging to a log template based on the context of the target log; determining a first position of the target log in the training set based on the frequency of the target log in the training set, and determining a second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model; and determining whether the target log is abnormal based on the first and second positions.

[0123] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The method for detecting abnormal logs shown above also applies to this non-volatile storage medium, and will not be repeated here.

[0124] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the abnormal log detection method in various embodiments of this application.

[0125] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the exception log detection method in various embodiments of this application.

[0126] This application also provides a computer program that, when executed by a processor, implements the steps of the abnormal log detection method in various embodiments of this application.

[0127] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0133] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting abnormal logs, characterized in that, include: Obtain the log file to be tested; An abnormal log identification model is used to predict the target predicted log template to which the target log in the log file to be detected belongs. The target log is any log in the log file to be detected. The abnormal log identification model is used to predict the probability of the target log belonging to the log template based on the context of the target log. The first position of the target log in the training set is determined based on the frequency of occurrence of the target log in the training set, and the second position of the target predicted log template in the predicted log template set is determined based on the frequency of occurrence of the target log in the training set. The training set is the set of logs used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model. Determining whether the target log is abnormal based on the first position and the second position includes: determining the difference between the second position and the first position; if the difference is greater than a preset threshold, determining the target log as an abnormal log; if the difference is less than or equal to the preset threshold, determining the target log as a normal log.

2. The method according to claim 1, characterized in that, After obtaining the log file to be detected, the method further includes: Identify the variable parameters in each log entry of the log file to be detected; The variable parameters are converted into wildcards based on the log template set to obtain the template log file corresponding to the log file to be detected; The template log files are sorted according to time series to obtain the log sequence to be detected.

3. The method according to claim 2, characterized in that, After obtaining the template log file corresponding to the log file to be detected, the method further includes: Determine the string contained in each template log in the template log file; The string is converted into a tree structure, wherein the root node of the tree structure is the first character of the string; The tree structure is stored in a database.

4. The method according to claim 2, characterized in that, After obtaining the template log file corresponding to the log file to be detected, the method further includes: Obtain a first template log and a second template log from the template log file, wherein the first template log and the second template log are any two template logs in the template log file; Determine the similarity between the first template log and the second template log; If the similarity is greater than the similarity threshold, the first template log and the second template log are determined to be of the same type of template. The first template log and the second template log are merged to obtain the updated template log file.

5. The method according to claim 4, characterized in that, Determining the similarity between the first template log and the second template log includes: Retrieve the first string from the first template log and the second string from the second template log; Simultaneously, starting from the first character of the first string and the second string, compare them sequentially to determine the substring from the first position to the second position where the characters in the first string and the characters in the second string are completely identical. Here, the first position is the position of the first character in the first string or the second string, and the second position is the position of the last character in the shortest string between the first string and the second string. Determine the length of each substring, and identify the longest substring as the target substring; The similarity between the first template log and the second template log is determined based on the target substring.

6. The method according to claim 1, characterized in that, The method employs an anomaly log identification model to predict the target predicted log template to which the target log in the log file to be detected belongs, including: The log files to be detected are sorted according to time sequence to obtain the log sequence to be detected; The log sequence to be detected is input into the embedding layer of the abnormal log recognition model to obtain a first vector corresponding to the log sequence to be detected. The target vector corresponding to the target log is masked from the first vector to obtain the second vector; The second vector is input into the encoder in the abnormal log recognition model to obtain an output vector, wherein the encoder is trained based on the context information in the historical log file where the target log is located, and the output vector is used to represent the probability that the target log belongs to different log templates; The log template corresponding to the maximum value in the output vector is determined as the target prediction log template.

7. An anomaly log detection device, characterized in that, include: The acquisition module is used to acquire the log file to be detected; The prediction module is used to predict the target predicted log template to which the target log in the log file to be detected belongs using an abnormal log identification model, wherein the target log is any log in the log file to be detected, and the abnormal log identification model is used to predict the probability of the log template to which the target log belongs based on the context of the target log. The first determining module is used to determine the first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and to determine the second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the abnormal log recognition model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the abnormal log recognition model; The second determining module is used to determine whether the target log is abnormal based on the first position and the second position, including: determining the difference between the second position and the first position; if the difference is greater than a preset threshold, determining the target log as an abnormal log; if the difference is less than or equal to the preset threshold, determining the target log as a normal log.

8. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions to perform the following functions: acquiring a log file to be detected; using an anomaly log identification model to predict the target predicted log template to which a target log in the log file to be detected belongs, wherein the target log is any log in the log file to be detected, and the anomaly log identification model is used to predict the probability of the target log belonging to the log template based on the context of the target log; determining a first position of the target log in the training set based on the frequency of occurrence of the target log in the training set, and determining a second position of the target predicted log template in the predicted log template set, wherein the training set is the log set used to train the anomaly log identification model, and the predicted log template set is the set of predicted log templates corresponding to all logs in the log file to be detected determined by the anomaly log identification model; determining whether the target log is abnormal based on the first position and the second position includes: determining the difference between the second position and the first position; if the difference is greater than a preset threshold, determining the target log as an abnormal log; if the difference is less than or equal to the preset threshold, determining the target log as a normal log.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the abnormal log detection method according to any one of claims 1 to 6 by running the computer program.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the abnormal log detection method according to any one of claims 1 to 6.

Images