A log anomaly detection method, system, device, equipment and storage medium

The log sequence to be detected is processed through the autoencoder model, and the abnormality is identified by reconstructing mean square error, which solves the problem of low log abnormality detection efficiency in the prior art, and realizes automated log abnormality detection.

CN118567901BActive Publication Date: 2024-11-12国家超级计算天津中心
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411046292.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-12
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

In the prior art, log abnormality detection relies on manual labor, is low in efficiency, and is difficult to effectively improve.

Method used

The autoencoder model is used to process the detection log sequence, and the log sequence is determined by reconstructing mean square error with the autoencoder model trained based on the normal log sample set, and an alarm is issued.

Benefits of technology

It improves the efficiency of log exception detection, can automatically identify abnormal log sequences, and reduces manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118567901B_ABST
    Figure CN118567901B_ABST
Patent Text Reader

Abstract

The present disclosure provides a log anomaly detection method, system, device, equipment and storage medium. The method includes: obtaining a log sequence to be detected, inputting the log sequence to be detected into an autoencoder model, and outputting a reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model, wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal. It can be seen that the present disclosure uses the autoencoder model to process the log sequence to be detected to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a log anomaly detection method, system, device, equipment and storage medium. Background Art

[0002] High-performance computers are composed of multiple tightly coupled subsystems with different functions, such as computing, management, storage, input / output, and interconnection and communication. The complexity and large scale of their structure make ensuring system reliability an extremely challenging task.

[0003] System logs usually record important events that occur in the system at every moment, such as abnormalities in neighboring nodes, plugging and unplugging of devices, reading and writing files, and the status of processes. Reasonable and effective analysis of system logs can provide insights into the system status.

[0004] At present, the detection of log anomalies relies on manual work, resulting in low efficiency of log anomaly detection. Therefore, how to improve the efficiency of log detection is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present disclosure provide a log anomaly detection method, system, device, equipment and storage medium.

[0006] In a first aspect, the present disclosure provides a log anomaly detection method, the method comprising:

[0007] Get the log sequence to be detected;

[0008] The log sequence to be detected is input into the autoencoder model, and after being processed by the autoencoder model, the reconstructed mean square error of the log sequence to be detected is output; wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal.

[0009] In an optional implementation, the inputting the log sequence to be detected into the autoencoder model, and before outputting the reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model, further includes:

[0010] Obtaining a log sample set, and performing sentence embedding on the log sample set based on a target language model to obtain an embedded log sample set;

[0011] Perform abnormal log screening on the embedded log sample set based on an abnormal log screening algorithm to determine a normal log sample set;

[0012] Model training is performed based on the normal log sample set to obtain an autoencoder model.

[0013] In an optional implementation manner, the acquiring the log sample set and performing sentence embedding on the log sample set based on the target language model to obtain the embedded log sample set further includes:

[0014] Obtaining a log data set of a preset length, and preprocessing the log data set to obtain a preprocessed log data set;

[0015] Based on the preprocessed log data set, construct a comparative learning data set;

[0016] Based on the contrastive learning data set, a contrastive learning method is used to fine-tune the language model, the fine-tuned language model is output, and the fine-tuned language model is used as the target language model.

[0017] In an optional implementation, after inputting the log sequence to be detected into the autoencoder model and outputting the reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model, the method further includes:

[0018] If it is determined that the reconstruction mean square error is greater than a preset threshold, a log abnormality alarm is issued.

[0019] In a second aspect, the present disclosure provides a log anomaly detection system, the system comprising a global management center unit and a regional management center unit, the regional management center unit being deployed in a first cabinet, the first cabinet being a cabinet in a first subsystem of a high-performance computer system;

[0020] The regional management center unit is used to obtain a log sequence to be detected, input the log sequence to be detected into an autoencoder model, and output a reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model; wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal;

[0021] The global management center unit is used to obtain a log sample set, perform sentence embedding on the log sample set based on a target language model to obtain an embedded log sample set, perform abnormal log screening on the embedded log sample set based on an abnormal log screening algorithm, determine a normal log sample set, perform model training based on the normal log sample set to obtain an autoencoder model, and push the autoencoder model to the regional management center unit.

[0022] In an optional implementation, the global management center unit is further used to obtain a log data set of a preset length, preprocess the log data set to obtain a preprocessed log data set, construct a comparative learning data set based on the preprocessed log data set, fine-tune the language model based on the comparative learning data set using a comparative learning method, output the fine-tuned language model, and use the fine-tuned language model as the target language model.

[0023] In an optional implementation manner, the regional management center unit is further configured to generate an abnormal alarm when determining that the reconstructed mean square error is greater than a preset threshold, and send the abnormal alarm to the global management center unit;

[0024] The global management center unit is also used to issue the abnormal alarm.

[0025] In a third aspect, the present disclosure provides a log anomaly detection device, the device comprising:

[0026] The first acquisition module is used to acquire the log sequence to be detected;

[0027] A processing module is used to input the log sequence to be detected into the autoencoder model, and after being processed by the autoencoder model, output the reconstructed mean square error of the log sequence to be detected; wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal.

[0028] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the above method.

[0029] In a fifth aspect, the present disclosure provides a log anomaly detection device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0030] In a sixth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.

[0031] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0032] The disclosed embodiment provides a log anomaly detection method, which obtains a log sequence to be detected, inputs the log sequence to be detected into an autoencoder model, and outputs a reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model, wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal. It can be seen that the disclosed embodiment uses the autoencoder model to process the log sequence to be detected to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0035] Figure 1 A flow chart of a log anomaly detection method provided by an embodiment of the present disclosure;

[0036] Figure 2 A flowchart of another log anomaly detection method provided by an embodiment of the present disclosure;

[0037] Figure 3 A structural diagram of a log anomaly detection system provided by an embodiment of the present disclosure;

[0038] Figure 4 A schematic diagram of another log detection system provided by an embodiment of the present disclosure;

[0039] Figure 5 A schematic diagram of the structure of a log anomaly detection device provided by an embodiment of the present disclosure;

[0040] Figure 6 A schematic diagram of the structure of a log anomaly detection device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0043] High-performance computers have complex structures, large scales, and tightly coupled components, making it a challenging task to ensure their reliability. High-performance computers in production environments are often threatened by failures, and software failures are not only more common than hardware failures, but are also usually more difficult to recover from than hardware failures. As high-performance computers are known to be difficult to develop, have a long development cycle, and are expensive, it is difficult for high-performance computer manufacturers to conduct complete testing before production, and it is also difficult for users to obtain a large amount of testing time before production. Many problems in the system require a long period of accumulation before they can be discovered in the later production environment. However, at this time, the possibility of error avoidance, error transfer, and fault tolerance has actually been missed. Reducing the time to discover faults through fault prediction methods and anomaly detection methods is an effective means to improve the reliability of supercomputing systems.

[0044] System logs usually record important events that occur in the system at every moment, such as abnormalities in neighboring nodes, plugging and unplugging of devices, reading and writing files, and the status of processes. Reasonable and effective analysis of system logs can provide insights into the system status. However, the growing scale and complex software architecture of modern distributed systems have led to a surge in the number of logs, making traditional analysis methods difficult to implement. The use of artificial intelligence algorithms to analyze logs is gradually replacing traditional methods. Anomaly detection is an important application direction of logs. Early use of log anomaly detection usually involves the following steps: log collection, log parsing, feature extraction, and anomaly detection.

[0045] Feature extraction is a traditional method that relies on manual design or domain knowledge to obtain useful information from log text. Related technologies only consider factors such as word position and frequency of occurrence, or only consider the synonym and antonym characteristics in domain knowledge. However, for text, text representation learning can usually fully learn the representation form of text to better provide representation and perform better in some tasks.

[0046] The current log anomaly detection methods mainly include anomaly detection for log volume, anomaly detection for log sequence, and anomaly detection methods based on log semantics and simultaneous anomaly detection of log volume changes and log sequences. As a language description of the system status, the method of simply detecting log volume changes obviously cannot fully utilize the semantic information contained in the log. The anomaly detection method of log sequence uses the log template sequence number to learn the pattern of abnormal and normal logs, ignoring the semantics of the log. In fact, different templates have different sequence numbers but may have the same semantics. In addition, the above methods are all supervised or semi-supervised learning methods, that is, at least part of the data needs to be labeled. In modern information technology systems such as large-scale supercomputers, even labeling part of the data is impractical.

[0047] In order to improve the efficiency of log anomaly detection, an embodiment of the present disclosure provides a log anomaly detection method.

[0048] Specifically, a log sequence is measured, and the log sequence to be detected is input into the autoencoder model. After being processed by the autoencoder model, the reconstructed mean square error of the log sequence to be detected is output, wherein the autoencoder model is trained based on the constructed normal log sample set, and the reconstructed mean square error is the difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model. The reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal. It can be seen that the present disclosure uses the method of processing the log sequence to be detected by the autoencoder model to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection.

[0049] Embodiment 1:

[0050] Based on this, the present disclosure provides a log anomaly detection method, referring to Figure 1 , is a flow chart of a log anomaly detection method provided by an embodiment of the present disclosure, the method comprising:

[0051] S101: Obtain a log sequence to be detected.

[0052] In the embodiment of the present disclosure, the log sequence to be detected refers to a series of log records that need to be detected. Specifically, the log sequence to be detected may include at least one type of log among system logs, security logs, and software logs.

[0053] In the disclosed embodiment, the length of the log sequence to be detected may be the same as the sample length required for training the autoencoder model.

[0054] After the log sequence to be detected is input into the autoencoder model and processed by the autoencoder model, before the reconstructed mean square error of the log sequence to be detected is output, the autoencoder model needs to be constructed. Specifically, the method for constructing the autoencoder model is as follows:

[0055] In an optional implementation, a log sample set is obtained, sentences are embedded in the log sample set based on a target language model to obtain an embedded log sample set, abnormal logs are screened for the embedded log sample set based on an abnormal log screening algorithm, a normal log sample set is determined, and model training is performed based on the normal log sample set to obtain an autoencoder model.

[0056] In the embodiment of the present disclosure, the target language model may be any language model for performing sentence embedding on the log sample set, and the embodiment of the present disclosure does not impose any limitation thereto.

[0057] In the disclosed embodiment, the abnormal log screening algorithm may be any algorithm for screening abnormal logs for the embedded log sample set, such as a LOF (Local Outlier Factor) algorithm.

[0058] Among them, the LOF algorithm is a density-based outlier detection method, which evaluates the outlier degree of a given data point by comparing the local density estimate of the point with its neighboring points. For example, if the density of a log entry in the feature space is much lower than the density of its surrounding neighbors, the log entry is considered to be an outlier.

[0059] In the disclosed embodiment, the embedded log sample set may be input into the LOF algorithm to identify outlier logs, the outlier logs may be constructed as an abnormal log sample set, and other logs may be constructed as a normal log sample set.

[0060] It can be seen that the embodiments of the present disclosure can perform abnormal log screening based on the LOF algorithm, thereby improving the efficiency of screening.

[0061] In the disclosed embodiment, a log sample set is first obtained, the log sample set is input into a target language model, sentences are embedded in the log sample set based on the target language model to obtain an embedded log sample set, and then an abnormal log screening algorithm is used to screen abnormal logs in the embedded log sample set to obtain a normal log sample set, thereby training a model based on the normal log sample set to obtain an autoencoder model.

[0062] In the disclosed embodiment, the autoencoding model may be updated based on a preset periodic frequency.

[0063] Specifically, the training process for the autoencoder model is as follows:

[0064] During the autoencoder model training phase, the constructed data set is input into the model training to learn the normal sequence pattern of the logs, wherein the length of the data set can be determined based on the requirements of the autoencoder model.

[0065] Exemplarily, a log sample sequence of a first length may be selected from a normal log sample set as the data set for training, and it is assumed that the log sample sequence of the first length is a first log sequence.

[0066] The goal of the training phase is to enable the autoencoder model to capture the intrinsic structure and patterns of the log data so that it can accurately reconstruct the input log sequence.

[0067] In the embodiment of the present disclosure, the structure of the autoencoder includes an encoder and a decoder, specifically:

[0068] Encoder: The role of the encoder is to convert high-dimensional input data (such as the first log sequence mentioned above) into a low-dimensional hidden representation (i.e., feature vector). This step is an information compression process that aims to capture and retain the most important information in the input data.

[0069] Decoder: The decoder is responsible for remapping the low-dimensional hidden representation back to the original data space (that is, the data space corresponding to the first log sequence mentioned above) and reconstructing the input log sequence. This step is used to test whether the encoder can effectively capture the key features of the input data.

[0070] During the training process, the performance of the model is evaluated by calculating the difference between the reconstructed log sequence and the original log sequence (i.e., the first log sequence mentioned above) (such as MSE (Mean Squared Error) as the loss function). The goal of training is to minimize the reconstruction error. Specifically, the autoencoder model parameters can be continuously adjusted through the back-propagation algorithm so that the autoencoder model can reconstruct the normal log sequence more accurately.

[0071] After sufficient training, the output autoencoder model can effectively encode and decode the log sequence to be detected, and can accurately reconstruct the log sequence that reflects the normal behavior of the system. For abnormal log sequences, since there are significant differences between the abnormal log sequence and the training data of the autoencoder model (such as the first log sequence), the autoencoder model exhibits a large reconstruction error when reconstructing the abnormal log sequence. This feature allows the trained autoencoder model to be used for real-time anomaly detection, and possible abnormal events can be identified by monitoring the reconstruction error.

[0072] In addition, in order to improve the effect of sentence embedding for the log sample set, in an optional implementation, a log data set of a preset length is obtained, the log data set is preprocessed to obtain a preprocessed log data set, a contrastive learning data set is constructed based on the preprocessed log data set, and based on the contrastive learning data set, a contrastive learning method is used to fine-tune the language model, the fine-tuned language model is output, and the fine-tuned language model is used as the target language model.

[0073] In the embodiments of the present disclosure, preprocessing refers to a series of cleaning and conversion operations performed on raw data to improve data quality and make it more suitable for subsequent analysis.

[0074] Specifically, the preprocessing steps include:

[0075] 1. Text replacement: Replace specific text strings, such as IP (Internet Protocol Address), MAC (Media Access Control Address), UUID (Universally Unique Identifier), URL (Uniform Resource Locator), and internally defined addresses, with abstract identifiers, such as IP, MAC, UUID, URL, and address, to reduce data complexity.

[0076] 2. Remove punctuation and extra spaces: Eliminate punctuation, independent characters, and extra spaces to reduce noise in the data.

[0077] 3. Number replacement: Replace the remaining numbers with a unified identifier num(number, number).

[0078] After obtaining the preprocessed log data set, a comparative learning data set is constructed based on the preprocessed log data set.

[0079] Taking the application of the log detection method provided by the embodiment of the present disclosure in a high-performance computer system scenario as an example, when the high-performance computer system is started for the first time, a comparative learning data set can be constructed according to the order in which the logs appear. For example, the order in which the logs appear is loga, logb, logc, logd, then the positive sample pairs can be (loga, logb), (logc, logd), and the negative samples can be constructed together with the log sequences loge, logf, logg, logh that are far away from the log sequence, for example, 1 day, such as (loga, loge), (logc, logf).

[0080] It can be seen that the embodiments of the present disclosure can utilize the characteristics of the log itself to form an unsupervised comparative learning data set without the need for manual labeling, thereby greatly reducing the overhead of labeling data.

[0081] When training the autoencoder model for the second time and thereafter, the abnormal log sample set and the normal log sample set screened out by the above-mentioned abnormal log screening algorithm (such as the LOF algorithm) can be used as comparative learning data sets. Specifically, assuming that the abnormal log sample set screened out by the above-mentioned abnormal log screening algorithm (such as the LOF algorithm) is flog={floga, flogb, flogc, flogd}, and the normal log sample set is log={loga, logb, logc, logd}, accordingly, the positive sample shapes are (loga, logb), (logc, logd), and the negative sample shapes are (loga, flogb), (logc, floga).

[0082] After the contrastive learning dataset is constructed, the contrastive learning method can be used to fine-tune the language model. Specifically, the language model can be an open-world language model, and the fine-tuned language model is output and used as the target language model.

[0083] In the embodiment of the present disclosure, an open world language model is used as the language model of the embodiment of the present disclosure, which can make use of external knowledge and the characteristics of logs, and is conducive to better learning log text representation.

[0084] S102: Input the log sequence to be detected into the autoencoder model, and after being processed by the autoencoder model, output the reconstructed mean square error of the log sequence to be detected.

[0085] Among them, the autoencoder model is trained based on the constructed normal log sample set, the reconstruction mean square error is the difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstruction mean square error is used to determine whether the log sequence to be detected is abnormal.

[0086] In the disclosed embodiment, the log sequence to be detected is input into the autoencoder model. After being processed by the autoencoder model, the reconstructed mean square error of the log sequence to be detected can be output, so that whether the log sequence to be detected is abnormal can be determined based on the reconstructed mean square error.

[0087] In an optional implementation, if it is determined that the reconstructed mean square error is greater than a preset threshold, a log abnormality alarm is issued.

[0088] The preset threshold refers to a preset numerical limit for determining whether the log sequence to be detected is abnormal.

[0089] In the disclosed embodiment, if it is determined that the reconstructed mean square error is greater than a preset threshold, it indicates that an abnormality exists in the log sequence to be detected, and a log abnormality alarm may be issued.

[0090] The embodiments of the present disclosure do not impose any limitation on the form of abnormal alarm.

[0091] In the log anomaly detection method provided by the present invention, a log sequence to be detected is obtained, and the log sequence to be detected is input into an autoencoder model. After being processed by the autoencoder model, a reconstructed mean square error of the log sequence to be detected is output, wherein the autoencoder model is trained based on a constructed normal log sample set, and the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model. The reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal. It can be seen that the encoder model of the embodiment of the present invention processes the log sequence to be detected to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection.

[0092] Embodiment 2:

[0093] To facilitate understanding of the above method embodiments, the present disclosure also provides a log anomaly detection method, referring to Figure 2 , is a flow chart of another log anomaly detection method provided by an embodiment of the present disclosure, specifically, as shown below:

[0094] Firstly, a log data set of preset length is obtained and preprocessed to obtain a preprocessed log data set. Then, a contrastive learning data set is constructed using the preprocessed log data set. Based on the contrastive learning data set, a contrastive learning method is used to fine-tune the language model. The fine-tuned language model is output and used as the target language model.

[0095] Get a log sample set and embed the log sample set into sentences based on the target language model (i.e. Figure 2 The embedded log sample set is obtained by using the log embedding shown in the figure. The abnormal log screening algorithm is then used to screen the abnormal logs on the embedded log sample set to obtain the normal log sample set. The normal log sample set is then used for model training to obtain the autoencoder model.

[0096] A log sequence to be detected is obtained, and the log sequence to be detected is input into the autoencoder model. After being processed by the autoencoder model, a reconstructed mean square error of the log sequence to be detected is output, and it is determined whether the reconstructed mean square error is greater than a preset threshold. If it is determined that the reconstruction error is greater than the preset threshold, a log anomaly alarm is issued. It can be seen that the present invention uses the method of processing the log sequence to be detected by the autoencoder model to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection.

[0097] Embodiment three:

[0098] Based on the above method embodiment, the present disclosure also provides a log anomaly detection system, which includes a global management center unit and a regional management center unit. The regional management center unit is deployed in a first cabinet, which is a cabinet in a first subsystem in a high-performance computer system.

[0099] In the disclosed embodiments, a high-performance computer system is generally composed of a computing subsystem, a management subsystem, a storage subsystem, an I / O (input / output) subsystem, and an interconnection communication subsystem, wherein each subsystem is responsible for a different functional role, wherein the computing subsystem carries the computing load of high-performance computing applications, and is composed of a plurality of computer cabinets; the storage subsystem is responsible for storing supercomputer application data, and is composed of a plurality of storage cabinets; the I / O subsystem is responsible for data forwarding from the computing subsystem to the storage subsystem, and is composed of a plurality of I / O cabinets; the management subsystem is responsible for user login operations and the operation of system software such as resource scheduling software, and is composed of a plurality of management cabinets. Since the usage of the equipment needs to be set according to actual needs, some cabinets will be deployed with equipment of a plurality of different functional roles, thereby generating a hybrid cabinet, which does not affect actual data collection.

[0100] In the embodiment of the present disclosure, the first cabinet may be any cabinet included in the first subsystem in the high-performance computer system, and the first subsystem may be any subsystem in the high-performance computer system, such as a computing subsystem.

[0101] refer to Figure 3 , is a structural diagram of a log anomaly detection system provided by an embodiment of the present disclosure, taking a computer cabinet as a first cabinet as an example, such as Figure 3 As shown, a regional management center unit is deployed in the first cabinet. The regional management center unit can be two active and standby AMCs (Area Management Center) for collecting logs of the first cabinet. The global management center unit can be a GMC (Area Management Center) for aggregating logs collected in all AMCs.

[0102] In the disclosed embodiment, since the regional management center unit can be composed of two AMCs, namely, a primary AMC and a backup AMC, when an abnormality occurs in the primary AMC, the backup AMC can take over the primary AMC and continue to work, thereby improving the reliability of the system.

[0103] It should be noted that the actual deployment of the server also needs to consider environmental factors such as cabinet power consumption and cooling capacity. Figure 3 Just as an example.

[0104] The regional management center unit is used to obtain the log sequence to be detected, input the log sequence to be detected into the autoencoder model, and output the reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model; wherein the autoencoder model is trained based on the constructed normal log sample set, the reconstructed mean square error is the difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal.

[0105] The global management center unit is used to obtain a log sample set, perform sentence embedding on the log sample set based on a target language model to obtain an embedded log sample set, perform abnormal log screening on the embedded log sample set based on an abnormal log screening algorithm, determine a normal log sample set, perform model training based on the normal log sample set to obtain an autoencoder model, and push the autoencoder model to the regional management center unit.

[0106] In the disclosed embodiment, the global management center unit, after obtaining the autoencoder model, pushes the character encoder model to the regional management center unit, so that the regional management center unit can process the log sequence to be detected based on the autoencoder model, thereby outputting the reconstructed mean square error of the log sequence to be detected.

[0107] The global management center unit is also used to obtain a log data set of a preset length, preprocess the log data set to obtain a preprocessed log data set, construct a contrastive learning data set based on the preprocessed log data set, fine-tune the language model based on the contrastive learning data set using a contrastive learning method, output the fine-tuned language model, and use the fine-tuned language model as the target language model.

[0108] The regional management center unit is also used to generate an abnormal alarm when it is determined that the reconstruction mean square error is greater than a preset threshold, and send the abnormal alarm to the global management center unit.

[0109] The global management center unit is also used to issue abnormal alarms.

[0110] In the disclosed embodiment, when the regional management unit determines that the reconstructed mean square error is greater than a preset threshold, it generates an abnormal alarm and sends the abnormal alarm to the global management center unit, so that the global management center unit issues the abnormal alarm.

[0111] The embodiments of the present disclosure do not impose any limitation on the form of abnormal alarm.

[0112] It should be noted that for the specific description of the log anomaly detection system provided in the embodiment of the present disclosure, reference may be made to the specific description of the log anomaly detection method described above.

[0113] In the log anomaly detection system provided by the embodiment of the present disclosure, the system includes a global management center unit and a regional management center unit. The regional management center unit is deployed in a first cabinet, which is a cabinet in a first subsystem of a high-performance computer system. The regional management center unit is used to obtain a log sequence to be detected, input the log sequence to be detected into an autoencoder model, and output a reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model, wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal. The global management center unit is used to obtain a log sample set, perform sentence embedding on the log sample set based on a target language model to obtain an embedded log sample set, perform abnormal log screening on the embedded log sample set based on an abnormal log screening algorithm, determine a normal log sample set, perform model training based on the normal log sample set to obtain an autoencoder model, and push the autoencoder model to the regional management center unit. It can be seen that the embodiment of the present disclosure can utilize the autoencoder model to process the log sequence to be detected to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection.

[0114] Embodiment 4:

[0115] To facilitate understanding of the log detection system provided by the embodiment of the present disclosure, see Figure 4 , is a schematic diagram of another log detection system provided by an embodiment of the present disclosure, specifically:

[0116] Each cabinet is used as a unit area, and a regional management center unit is deployed for each cabinet. The regional management center unit can be two active and standby AMCs for collecting logs in the cabinet. In addition, a global management center unit is also set up, and the global management center unit can be a GMC to aggregate the logs collected in all AMCs.

[0117] Specifically, although the functions of each subsystem in a high-performance computer system are different, the logs collected in devices with different roles can be divided into system logs (mainly used to record system behavior), security logs (mainly used to record user access and security behavior audits), and software logs (mainly used to record key software behaviors such as file system logs, resource scheduling system logs, etc.). The specific collection log details are shown in Table 1.

[0118] Table 1: Log collection details

[0119]

[0120] It should be noted that Table 1 is only provided as an example and should not be considered as a limitation.

[0121] The overall process of the log anomaly detection system is as follows:

[0122] At each node (i.e. Figure 4 The computing blade servers, input / output nodes, storage servers, management nodes, and login nodes shown in the figure deploy log collection tools to push logs of different nodes to the corresponding AMCs in real time. The GMC periodically collects logs of all AMCs. The period for GMC to collect logs is set based on demand. For example, the set period length can refer to the training period of the autoencoder model and be dynamically adjusted according to the accuracy of the autoencoder model.

[0123] In GMC, a persistent log storage module and an anomaly detection model training module are deployed. The anomaly detection model training module is used to train the autoencoder model at regular intervals. The anomaly detection training module also includes three sub-functional modules, namely, a data preprocessing module, an anomaly log screening module, and an autoencoder training module. Before the autoencoder model is trained, the data preprocessing module selects a log data set of a preset length and first preprocesses the log data set, including text cleaning, construction of a comparative learning data set, etc. Then, the comparative learning data set is used to fine-tune the open-world language model (i.e., the above-mentioned language model) (the open-world language model is used for the first fine-tuning, and the fine-tuned open-world language model can be used for fine-tuning or based on the open-world model when the open-world language model is updated later). The fine-tuned open-world language model is used to embed the log sample set at the sentence level to obtain the embedded log sample set, which is input into the log screening module. The log screening module can use the LOF algorithm to screen out the outlier logs and construct a normal log sample set. The normal log sample set is input into the anomaly detection model training module to learn the normal sequence pattern of the log, and output an autoencoder model that can reconstruct the normal log sequence. After the autoencoder model training is completed, GMC pushes the autoencoder model to all AMCs. The anomaly detection module is deployed in the AMC. The anomaly detection module includes the autoencoder model and the anomaly judgment function. For the newly collected logs in the AMC, based on the anomaly detection module in the corresponding AMC, according to a fixed window (such as a length window or a time window, which is consistent with the sequence length of the normal log sample set when training the autoencoder model), the newly collected logs in the AMC are input into the autoencoder model for sequence reconstruction and the reconstruction error is calculated. When the error is greater than the set threshold, an abnormal alarm is generated and pushed back to the GMC, and the GMC issues an external alarm.

[0124] Exemplarily, the length window may be 100 consecutive newly collected log records, and the time window may be log data from 1 hour ago.

[0125] Specifically, the overall process of the above log anomaly detection system includes the following steps:

[0126] S201: Log collection.

[0127] At each node (i.e. Figure 4 The log collection tool is deployed on the computing blade servers, input / output nodes, storage servers, management nodes, and login nodes shown in the figure. The log files that can be collected and the command outputs that need to be monitored are specified in the configuration file. The log collection tool reads the configuration file content when it is started. Once a new log is generated after startup, it will be pushed to the AMC in real time. The GMC regularly collects logs of all AMCs, where the cycle for GMC to collect logs is set based on demand. For example, the set cycle length can be the training cycle of the autoencoder model, and can also be dynamically adjusted according to the accuracy of the autoencoder model. The GMC is equipped with persistent log storage and anomaly detection model training modules, where the persistent log storage module is used to persistently store system logs, and the log storage cycle can be set, usually stored in cold, warm, and hot layers.

[0128] S202: Autoencoder model training.

[0129] The anomaly detection model training module includes a data preprocessing module, an anomaly log screening module, and an autoencoder training module.

[0130] Specifically, for the autoencoder model training, the input is a log data set of preset length collected from the GMC persistent log storage module, and the output is the autoencoder model.

[0131] The autoencoder model training includes data preprocessing, building a comparative learning data set, determining the target language model, abnormal log screening, autoencoder training and other steps. For details, please refer to the description of the log anomaly detection method.

[0132] S203: Based on the anomaly detection module deployed in the AMC, determine whether the log sequence to be detected is abnormal.

[0133] After the autoencoder model training is completed, GMC pushes the autoencoder model to all AMCs, and deploys anomaly detection modules in AMC. The anomaly detection module includes an autoencoder model and anomaly judgment function. For the newly collected logs in AMC (i.e., the above-mentioned log sequence to be detected), based on the anomaly detection module in the corresponding AMC, according to a fixed window (such as a length window or a time window, which is consistent with the sequence length of the normal log sample set when training the autoencoder model), the newly collected logs in AMC are input into the autoencoder model for sequence reconstruction, and the reconstruction mean square error is calculated. The anomaly judgment function determines whether to generate an abnormal alarm based on the relationship between the reconstruction mean square error and the preset threshold. Specifically, when the reconstruction mean square error is greater than the preset threshold, it indicates that the log sequence to be detected is abnormal, an abnormal alarm is generated, and it is pushed back to GMC, and GMC issues an external alarm.

[0134] It can be seen that the log anomaly detection system provided by the embodiment of the present disclosure designs a log anomaly detection framework, which can serve high-performance computing clusters, distributed anomaly judgment, reduce the centralized judgment overhead of core servers, centrally train autoencoder models, and reduce the resource consumption required for daily regional services.

[0135] Embodiment five:

[0136] Based on the above method embodiment, the present disclosure also provides a log anomaly detection device, referring to Figure 5 , is a structural diagram of a log anomaly detection device provided by an embodiment of the present disclosure, the device comprising:

[0137] The first acquisition module 501 is used to acquire a log sequence to be detected;

[0138] The processing module 502 is used to input the log sequence to be detected into the autoencoder model, and after being processed by the autoencoder model, output the reconstructed mean square error of the log sequence to be detected; wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal.

[0139] In an optional implementation, the device further includes:

[0140] A second acquisition module is used to acquire a log sample set, and perform sentence embedding on the log sample set based on a target language model to obtain an embedded log sample set;

[0141] A screening module, used to screen the embedded log sample set for abnormal logs based on an abnormal log screening algorithm to determine a normal log sample set;

[0142] The training module is used to perform model training based on the normal log sample set to obtain an autoencoder model.

[0143] In an optional implementation, the device further includes:

[0144] A third acquisition module is used to acquire a log data set of a preset length, and preprocess the log data set to obtain a preprocessed log data set;

[0145] A construction module, used to construct a comparative learning dataset based on the preprocessed log dataset;

[0146] A fine-tuning module is used to fine-tune the language model based on the contrastive learning data set by using a contrastive learning method, output the fine-tuned language model, and use the fine-tuned language model as the target language model.

[0147] In an optional implementation, the device further includes:

[0148] The abnormal alarm module is used to issue a log abnormality alarm when it is determined that the reconstruction mean square error is greater than a preset threshold.

[0149] In the log anomaly detection device provided by the present invention, a log sequence to be detected is obtained, and the log sequence to be detected is input into the autoencoder model. After being processed by the autoencoder model, a reconstructed mean square error of the log sequence to be detected is output, wherein the autoencoder model is trained based on a constructed normal log sample set, and the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model. The reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal. It can be seen that the encoder model of the embodiment of the present invention processes the log sequence to be detected to determine whether the log sequence to be detected is abnormal, thereby improving the efficiency of log anomaly detection.

[0150] Embodiment six:

[0151] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device implements the log anomaly detection method described in the embodiments of the present disclosure.

[0152] Embodiment seven:

[0153] The embodiment of the present disclosure further provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the log anomaly detection method described in the embodiment of the present disclosure is implemented.

[0154] Embodiment eight:

[0155] In addition, the present disclosure also provides a log anomaly detection device, see Figure 6 As shown, it may include:

[0156] Processor 601, memory 602, input device 603 and output device 604. The number of processors 601 in the log anomaly detection device can be one or more. Figure 6 In some embodiments of the present disclosure, the processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means, wherein: Figure 6 The example of connecting through bus is taken in the following.

[0157] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing of the log anomaly detection device by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The input device 603 may be used to receive input digital or character information, and to generate signal input related to user settings and function control of the log anomaly detection device.

[0158] Specifically in this embodiment, the processor 601 will load the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 will run the applications stored in the memory 602, thereby realizing the various functions of the above-mentioned log anomaly detection device.

[0159] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0160] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log anomaly detection method, characterized in that: The method comprises: Get the log sequence to be detected; Obtaining a log data set of a preset length, and preprocessing the log data set to obtain a preprocessed log data set; Based on the preprocessed log data set, construct a comparative learning data set; Based on the contrastive learning data set, fine-tune the language model using a contrastive learning method, output the fine-tuned language model, and use the fine-tuned language model as the target language model; Acquire a log sample set, and perform sentence embedding on the log sample set based on the target language model to obtain an embedded log sample set; Perform abnormal log screening on the embedded log sample set based on an abnormal log screening algorithm to determine a normal log sample set; Performing model training based on the normal log sample set to obtain an autoencoder model; The log sequence to be detected is input into the autoencoder model, and after being processed by the autoencoder model, the reconstructed mean square error of the log sequence to be detected is output; wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal.

2. The method according to claim 1, characterized in that: After inputting the to-be-detected log sequence into the autoencoder model and outputting the reconstructed mean square error of the to-be-detected log sequence after being processed by the autoencoder model, the method further includes: If it is determined that the reconstruction mean square error is greater than a preset threshold, a log abnormality alarm is issued.

3. A log anomaly monitoring system, the system comprising a global management center unit and a regional management center unit, the regional management center unit being deployed in a first cabinet, the first cabinet being a cabinet in a first subsystem of a high-performance computer system; The regional management center unit is used to obtain a log sequence to be detected, input the log sequence to be detected into the autoencoder model, and output the reconstructed mean square error of the log sequence to be detected after being processed by the autoencoder model; wherein, The autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal; The global management center unit is used to obtain a log sample set, perform sentence embedding on the log sample set based on a target language model to obtain an embedded log sample set, perform abnormal log screening on the embedded log sample set based on an abnormal log screening algorithm, determine a normal log sample set, perform model training based on the normal log sample set to obtain an autoencoder model, and push the autoencoder model to the regional management center unit; The global management center unit is also used to obtain a log data set of a preset length, preprocess the log data set to obtain a preprocessed log data set, construct a contrastive learning data set based on the preprocessed log data set, fine-tune the language model based on the contrastive learning data set using a contrastive learning method, output the fine-tuned language model, and use the fine-tuned language model as the target language model.

4. The system according to claim 3, characterized in that The regional management center unit is also used to generate an abnormal alarm when it is determined that the reconstruction mean square error is greater than a preset threshold, and send the abnormal alarm to the global management center unit; The global management center unit is also used to issue the abnormal alarm.

5. A log anomaly detection device, characterized in that: The device comprises: The first acquisition module is used to acquire the log sequence to be detected; A third acquisition module is used to acquire a log data set of a preset length, and preprocess the log data set to obtain a preprocessed log data set; A construction module, used to construct a comparative learning dataset based on the preprocessed log dataset; A fine-tuning module, configured to fine-tune the language model using a contrastive learning method based on the contrastive learning dataset, output the fine-tuned language model, and use the fine-tuned language model as a target language model; A second acquisition module is used to acquire a log sample set, and perform sentence embedding on the log sample set based on the target language model to obtain an embedded log sample set; A screening module, used to screen the embedded log sample set for abnormal logs based on an abnormal log screening algorithm to determine a normal log sample set; A training module, used for performing model training based on the normal log sample set to obtain an autoencoder model; A processing module is used to input the log sequence to be detected into the autoencoder model, and after being processed by the autoencoder model, output the reconstructed mean square error of the log sequence to be detected; wherein the autoencoder model is trained based on a constructed normal log sample set, the reconstructed mean square error is the degree of difference between the log sequence to be detected and the log sequence reconstructed by the autoencoder model, and the reconstructed mean square error is used to determine whether the log sequence to be detected is abnormal.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 2.

7. A log anomaly detection device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.