A log detection method and device, electronic equipment and storage medium

Abstract

The application provides a log detection method and device, electronic equipment and computer storage medium, the method comprises: obtaining the trained absolute encoding model and relative encoding model based on the obtained original log set; the trained absolute encoding model is used to represent the specific meaning and time sequence change rule of each log in the original log set; the trained absolute encoding model is used to represent the time sequence change rule of each log in the original log set; obtain the log set to be detected, input the log set to be detected into the trained absolute encoding model and relative encoding model respectively, and obtain the respective output results; when it is determined that there is an abnormal log in the log set to be detected according to the output result of the trained absolute encoding model, the corresponding content of the abnormal log in the output result of the trained relative encoding model is used to determine whether the abnormal log is abnormal again.

Description

Technical Field

[0001] This application relates to the field of log analysis technology, and in particular to a log detection method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] Currently, system logs are used to record system execution workflows, helping operations and maintenance personnel to discover and locate problems. Typically, workflows without anomalies usually follow certain patterns, so the corresponding log data is recorded. The model learns from the log data under normal working conditions, and when log data deviates from the normal working pattern, an anomaly is considered to have occurred. However, in actual log data, in addition to the original log events, new logs will appear. If these new logs are not processed, they will all be identified as abnormal logs by the model, leading to error alerts and increasing the time and effort required for operations and maintenance personnel to troubleshoot anomalies. Summary of the Invention

[0003] This application provides a log detection method, apparatus, electronic device, and computer storage medium; it can solve the problem in related technologies where new logs appearing during the log detection process are identified as abnormal logs, leading to increased operation and maintenance costs.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a log detection method, the method comprising:

[0006] Based on the obtained original log set, a trained absolute encoding model and a relative encoding model are obtained; the trained absolute encoding model is used to characterize the specific meaning and temporal variation pattern of each log in the original log set; the trained absolute encoding model is used to characterize the temporal variation pattern of each log in the original log set.

[0007] Obtain the log set to be detected, and input the log set to be detected into the trained absolute encoding model and relative encoding model respectively to obtain their respective output results; the output result represents the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected.

[0008] When it is determined that there are abnormal logs in the log set to be detected based on the output of the trained absolute encoding model, the corresponding content of the abnormal logs in the output of the trained relative encoding model is used to determine whether the abnormal logs are abnormal.

[0009] In some embodiments, inputting the log set to be detected into the trained absolute encoding model and relative encoding model respectively includes:

[0010] Determine the log template for each log in the log set to be detected;

[0011] Map the log template of each log in the log set to be detected to an index;

[0012] The index corresponding to each log in the log set to be detected is determined by a sliding window method, and the index in each sliding window is input into the trained absolute encoding model; and when determining the index in each sliding window, the index in each sliding window is encoded according to the index size, and the encoded index in each sliding window is input into the trained relative encoding model.

[0013] In some embodiments, obtaining the trained absolute coding model and relative coding model based on the acquired original log set includes:

[0014] Determine the log template for each log in the original log set;

[0015] Map the log template of each log in the original log set to an index;

[0016] The index corresponding to each log in the original log set is determined using a sliding window method, and the index within each sliding window and the index of the corresponding next actual log are determined. The index within each sliding window and the index of the corresponding next actual log are then input into an absolute encoding model for training to obtain a trained absolute encoding model. In addition, when determining the index within each sliding window, the index within each sliding window and the index of the corresponding next actual log are encoded according to the index size. The encoded index within each sliding window and the index of the corresponding next actual log are then input into a relative encoding model for training to obtain a trained relative encoding model.

[0017] In some embodiments, determining the presence of anomalous logs in the log set to be detected based on the output of the trained absolute coding model includes:

[0018] Based on the output of the trained absolute encoding model, if the log to be predicted is not among the logs with the highest classification probability (k), then the log to be predicted is determined to be an abnormal log.

[0019] In some embodiments, each log in the original log set is a normal log.

[0020] This application provides a log detection device, the device comprising a first obtaining module, a second obtaining module, and a determining module, wherein...

[0021] The first acquisition module is used to obtain a trained absolute encoding model and a relative encoding model based on the acquired original log set; the trained absolute encoding model is used to characterize the specific meaning and temporal variation pattern of each log in the original log set; the trained absolute encoding model is used to characterize the temporal variation pattern of each log in the original log set.

[0022] The second module is used to obtain the log set to be detected, and input the log set to be detected into the trained absolute encoding model and relative encoding model respectively to obtain their respective output results; the output result represents the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected.

[0023] The determination module is used to, when determining that there are abnormal logs in the log set to be detected based on the output results of the trained absolute encoding model, use the corresponding content of the abnormal logs in the output results of the trained relative encoding model to determine whether the abnormal logs are abnormal again.

[0024] This application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the log detection method provided by one or more of the foregoing technical solutions.

[0025] This application provides a computer storage medium storing a computer program; when the computer program is executed, it can implement the log detection method provided by one or more of the aforementioned technical solutions.

[0026] This application provides a log detection method, apparatus, electronic device, and computer storage medium. The method includes: obtaining a trained absolute encoding model and a relative encoding model based on an acquired original log set; the trained absolute encoding model is used to characterize the specific meaning and temporal variation pattern of each log in the original log set; the trained absolute encoding model is used to characterize the temporal variation pattern of each log in the original log set; obtaining a log set to be detected, inputting the log set to be detected into the trained absolute encoding model and the relative encoding model respectively, and obtaining their respective output results; the output results represent the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected; when it is determined that there is an abnormal log in the log set to be detected based on the output result of the trained absolute encoding model, the corresponding content of the abnormal log in the output result of the trained relative encoding model is used to further determine whether the abnormal log is abnormal.

[0027] As can be seen, in this embodiment, after obtaining the trained absolute encoding model and relative encoding model from the original log set, these two models are used to perform anomaly detection on each log in the log set to be detected. Because the trained absolute encoding model is only trained on a limited number of logs in the original log set, it is very likely that newly appearing logs will be misidentified as abnormal logs when using this model for log prediction. In this case, the prediction results of the relative encoding model are used again for log anomaly detection. This is because the relative encoding model is only used to characterize the temporal change pattern of each log and ignores the specific meaning of the log. It will not misidentify new logs as abnormal logs because the model has not seen them before, thus reducing the occurrence of false alarms. In this way, when using these two models to detect logs in the log set to be detected, this embodiment can solve the problem of increased operation and maintenance costs caused by all new logs appearing during the log detection process being identified as abnormal logs by fusing the prediction results of the two models, thereby saving operation and maintenance costs. Attached Figure Description

[0028] Figure 1 This is a flowchart of a log detection method according to an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of another log detection process according to an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of the composition structure of the log detection device according to an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions in this application will now be clearly and completely described with reference to the accompanying drawings.

[0033] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the present application can be implemented in any combination.

[0034] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of a processor, program, or software, etc.) in the method or apparatus that includes that element.

[0035] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0036] For example, the log detection method provided in this application includes a series of steps, but the log detection method provided in this application is not limited to the steps described therein. Similarly, the log detection device provided in this application includes a series of modules, but the log detection device provided in this application is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on the information.

[0037] This application can be implemented based on electronic devices, which may be thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, etc.

[0038] Electronic devices can perform corresponding functions through the execution of program modules. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc. They perform specific tasks or implement specific abstract data types. Computer systems can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0039] In related technologies, methods for log anomaly detection are mainly divided into the following three types:

[0040] 1) Supervised machine learning methods, such as Support Vector Machines (SVM), decision trees, and Convolutional Neural Networks (CNN), require a large amount of labeled data. Specifically, they use labeled data to classify logs into abnormal or normal logs.

[0041] 2) Unsupervised machine learning methods. The representative model of this type of method is deeplog. This model is based on Long Short Term Memory (LSTM) network. It treats the log workflow as a time series and trains the model with the log workflow of normal working mode. When training and making predictions, if the log flow is detected to be inconsistent with the log workflow of normal working mode, it is considered that an anomaly has occurred.

[0042] 3) Semantic-based methods, such as Template2Vec, first construct a set of synonyms and antonyms, then find the vector of each word, and finally weight these word vectors to obtain Templates2Vec. This method can extract semantic information hidden in log templates to detect abnormal log sequence patterns. When a new log appears, it uses semantic similarity to compare with existing logs and then infers whether it is abnormal.

[0043] However, the above methods have the following shortcomings. First, supervised machine learning methods require labeled training data. Since millions of logs are generated daily in the cloud or other systems, labeling data is labor-intensive, making it difficult for supervised machine learning methods to obtain a large amount of effective training data. Second, existing unsupervised machine learning methods can only learn based on existing log templates. New log templates frequently appear in the system, and these methods will identify logs that have not appeared in the training dataset as abnormal. This will lead to many incorrectly predicted abnormal logs. These abnormal logs predicted by the model will need to be manually reviewed to confirm whether they are abnormal. It can be seen that the appearance of new logs will increase the cost of manual detection. Furthermore, the Template2Vec method, which is based on synonyms and antonyms, cannot accurately capture the semantic information of many words in log templates in specific domains. For example, words with opposite meanings have similar contexts and are actually similar log events. Template2Vec is difficult to handle such log events.

[0044] To address the above problems, the following embodiments are proposed.

[0045] In some embodiments of this application, the log detection method can be implemented using a processor in the log detection device. The processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor.

[0046] Figure 1 This is a flowchart of a log detection method according to an embodiment of this application, such as... Figure 1 As shown, the process may include:

[0047] Step 100: Based on the obtained original log set, obtain the trained absolute encoding model and relative encoding model.

[0048] In some embodiments, each log in the original log set is a normal log; a normal log refers to a log generated under normal working conditions. Here, the original log set can be represented as a log workflow composed of multiple normal logs, that is, there is a time dependency between the logs in the original log set; wherein, a log is a time-series text data, which can consist of a series of instructions that record the running status of related services in real time, such as obtaining a Hypertext Transfer Protocol (HTTP) address, obtaining a parameter, etc.

[0049] For example, there is no limitation on the method of obtaining the original log set. For instance, it can be obtained through a log collection system or through other methods.

[0050] In this embodiment of the application, after obtaining the original log set, the absolute coding model and the relative coding model are trained respectively to obtain the trained absolute coding model and the relative coding model. Then, the trained absolute coding model and the relative coding model can be used to discover or predict the faults that have occurred or may occur in the business system.

[0051] Here, the trained absolute encoding model is used to characterize the specific meaning and temporal variation patterns of each log in the original log set; the trained absolute encoding model is used to characterize the temporal variation patterns of each log in the original log set. The specific training process is explained below.

[0052] In some embodiments, obtaining a trained absolute encoding model and a relative encoding model based on the acquired original log set may include: determining a log template for each log in the original log set; mapping the log template for each log in the original log set to an index; determining the index within each sliding window and the index of the corresponding next actual log in a sliding window manner; inputting the index within each sliding window and the index of the corresponding next actual log into the absolute encoding model for training to obtain a trained absolute encoding model; and, when determining the index within each sliding window, encoding the index within each sliding window and the index of the corresponding next actual log in accordance with the index size, inputting the encoded index within each sliding window and the index of the corresponding next actual log into the relative encoding model for training to obtain a trained relative encoding model.

[0053] For example, since each log in the original log set can be viewed as a template plus parameters, after obtaining the original log set, a log template can be extracted from each log in the original log set to obtain the log template of each log in the original log set; this can be represented by a log template set, where each log template in the log template set corresponds one-to-one with each log in the original log set.

[0054] Here, there are no restrictions on the method for extracting log templates. For example, you can use regular expression matching to remove log parameters to extract the log template, or you can extract it in other ways.

[0055] For example, suppose there is a specific log in the original log set that is Send 120 Bytes toblk4612; it can be seen that the parameters of this log are size=120 and block=blk4612. If the log template is extracted from this log, the log template Send Bytes to can be obtained.

[0056] In this embodiment of the application, after determining the log template (log template set) of each log in the original log set based on the original log set, the log template of each log in the original log set can be mapped to an index to obtain the index corresponding to each log in the original log set; it can be represented by a log index set, where each index included in the log index set corresponds one-to-one with each log template included in the log template set; at the same time, each index included in the log index set corresponds one-to-one with each log included in the original log set.

[0057] In one implementation, after determining the log template set, each log template in the log template set can be directly mapped to obtain the index corresponding to each log template, thereby obtaining the log index set.

[0058] In another implementation, after determining the log template set, the log templates in the log template set can be deduplicated first, and then each deduplicated log template can be mapped to obtain the index of each deduplicated log template; for the index of these log templates that have not been mapped, it can be determined based on the index of the log templates that are the same as these log templates and have been mapped; in this way, the index of each log template in the log template set can be obtained, and thus, the log index set can be obtained.

[0059] Understandably, by converting the original log set into a log index set, the original log set is transformed into a format suitable for input to both absolute and relative encoding models, facilitating subsequent model training. In addition, when there are many duplicate log templates in the log template set, using the second method described above to determine the log index set can reduce the amount of data that needs to be mapped and speed up the determination of the log index set.

[0060] In this embodiment of the application, after obtaining the log index set, the index in each sliding window and the corresponding index of the next actual log can be determined using a sliding window method.

[0061] For example, for each log in the original log set, the index within the sliding window at each time step can be regarded as an input sequence; and each input sequence has a corresponding index of the next actual log.

[0062] For example, the size setting of the sliding window is not limited in this embodiment of the application; for example, it can be set to 50, 60, etc.

[0063] For example, if the original log set includes 100 logs and the sliding window is set to 50, the index of the first 50 logs will be used as the index within the first sliding window. In this case, the index of the 51st log will be the index of the next actual log corresponding to the first sliding window.

[0064] Furthermore, after determining the index within each sliding window and the corresponding index of the next actual log using the sliding window method, the index within each sliding window and the corresponding index of the next actual log can be used as input to the absolute encoding model to train the absolute encoding model.

[0065] For example, when determining the index within each sliding window, the index within each sliding window and the corresponding index of the next actual log can be encoded according to the index size, and the encoded index within each sliding window and the corresponding index of the next actual log can be used as input to the absolute encoding model to train the absolute encoding model.

[0066] For example, in the process of encoding the index within each sliding window and the corresponding index of the next actual log, the indexes are encoded in ascending order; for example, the smallest index is encoded starting from 1, and other indexes are encoded by adding 1 to the encoding, until the encoding of the index within each sliding window and the corresponding index of the next actual log is completed.

[0067] For example, during model training, the parameters of the absolute encoding model and the relative encoding model are adjusted according to the deviation between the prediction results of the absolute encoding model and the index of the next actual log for each sliding window, until the prediction results of the absolute encoding model and the relative encoding model for each sliding window meet the set requirements. At this point, the trained absolute encoding model and relative encoding model are obtained.

[0068] Here, the prediction results of the absolute encoding model and the relative encoding model represent the classification probability of each index in the log index set for the next actual log. Since the log index set corresponds to the original log set, the classification probability of each log in the original log set for the next actual log can be determined based on the prediction results of these two models.

[0069] Step 101: Obtain the log set to be detected, and input the log set to be detected into the trained absolute encoding model and relative encoding model respectively to obtain their respective output results.

[0070] Here, the log set to be detected may include the original log set and at least one new log; the new log represents a log that has not appeared in the original log set; for example, the new log may be a log that first appears at the next moment or the next time interval of the original log set; the value of the time interval can be set according to the actual situation, and this application embodiment does not limit it.

[0071] For example, the method of obtaining the log set to be detected is the same as the method of obtaining the original log set. For example, it can be obtained through a log collection system or through other methods.

[0072] In some embodiments, inputting the log set to be detected into the trained absolute encoding model and relative encoding model respectively may include: determining the log template of each log in the log set to be detected; mapping the log template of each log in the log set to be detected to an index; determining the index within each sliding window according to the sliding window method for the index corresponding to each log in the log set to be detected, and inputting the index within each sliding window into the trained absolute encoding model; and, when determining the index within each sliding window, encoding the index within each sliding window according to the index size, and inputting the encoded index within each sliding window into the trained relative encoding model.

[0073] For example, after obtaining the log set to be detected, the log template is first extracted from each log in the log set to be detected, and then the log template is mapped to the corresponding index. Here, since the log set to be detected includes the original log set, and the original log set has already undergone the log template extraction and mapping process during the above model training process, in the process of determining the index of each log in the log set to be detected, the above processing can be performed only on the logs in the log set to be detected other than the original log set, and finally the index corresponding to each log in the log set to be detected is obtained.

[0074] Furthermore, after obtaining the index corresponding to each log in the log set to be detected, the index within each sliding window can be determined in a sliding window manner. The index within each sliding window is then sequentially input into the trained absolute encoding model to obtain the output result corresponding to the index within each sliding window. The output result of this model represents the classification probability of the log to be predicted as each log in the log set to be detected. Here, the log to be predicted represents the next actual log corresponding to the log within each sliding window.

[0075] In one implementation, when determining the index within each sliding window according to the sliding window method, the index within each sliding window can be directly encoded according to the index size, and then the encoded index within each sliding window can be input into the trained relative encoding model.

[0076] In another implementation, when determining the index within each sliding window according to the sliding window method, the indexes within each sliding window can be deduplicated first, and then each deduplicated index can be encoded according to its size to obtain the encoding of each deduplicated index. For the unencoded indexes within each sliding window, they can be determined based on the indexes of the same size that have already been encoded. In this way, the encoding of each index within each sliding window can be obtained, and then the encoded indexes within each sliding window can be input into the trained relative encoding model.

[0077] Understandably, relative encoding allows the relative encoding model to ignore the specific meaning of the encoding representation and only use it to learn the temporal change patterns of logs. When encountering new logs, due to relative encoding, the relative encoding model infers the result of the relative encoding based on the learned patterns. It will not misidentify the logs as abnormal logs because the model has not seen the new logs before, thus reducing the occurrence of erroneous alarms. In addition, when there are many repeated indexes in each sliding window, if the second method mentioned above is used to determine the encoding of the index, the amount of data can be reduced and the encoding speed can be accelerated.

[0078] For example, after inputting the index within each coded sliding window into the trained relative encoding model, the output of the model represents the classification probability of the log to be predicted as each log in the log set to be detected; wherein the log to be predicted represents the next actual log corresponding to the log in each sliding window.

[0079] Step 102: When it is determined that there are abnormal logs in the log set to be detected based on the output of the trained absolute coding model, the corresponding content of the abnormal logs in the output of the trained relative coding model is used to determine whether the abnormal logs are abnormal again.

[0080] In some embodiments, determining that there are anomalous logs in the log set to be detected based on the output of the trained absolute coding model may include: determining that the log to be predicted is not among the logs with the highest classification probabilities based on the output of the trained absolute coding model, and thus determining that the log to be predicted is an anomalous log.

[0081] For example, the output of the trained absolute encoding model can be sorted in descending order of classification probability, and the logs with the top k classification probabilities can be selected as the prediction results.

[0082] Furthermore, if it is determined that the log to be predicted (the next actual log corresponding to the log in the sliding window at the current time) is among the logs with the highest classification probabilities (k), then the log to be predicted is determined to be a normal log; otherwise, if it is determined that the log to be predicted is not among the logs with the highest classification probabilities (k), then the log to be predicted is determined to be an abnormal log.

[0083] Here, the value of k can be set according to the actual scenario. This application embodiment does not limit this. For example, it can be 20, 15, etc.

[0084] For example, assuming the log set to be detected includes 100 logs and the value of k is 20, if it is determined that the log to be predicted is among the top 20 logs in the output of the absolute encoding model, it means that the appearance of the log to be predicted conforms to the normal working mode and no abnormality has occurred; otherwise, it is considered that an abnormality has occurred.

[0085] In this embodiment of the application, when it is determined that the log to be predicted is an abnormal log based on the output result of the trained absolute coding model, the corresponding content of the abnormal log in the output result of the trained relative coding model is used to determine whether the abnormal log is abnormal again.

[0086] For example, when the log to be predicted is determined to be an anomalous log based on the output of the trained absolute encoding model, the output of the relative encoding model is used to determine whether the log to be predicted is among the logs with the highest classification probabilities (k). If so, it means that the appearance of the log to be predicted conforms to the normal working mode and no anomalous event has occurred; otherwise, it is considered that an anomalous event has occurred.

[0087] Understandably, absolute coding models are better at learning because they are used to characterize the specific meaning and temporal change patterns of each log in the original log set. However, this model cannot predict new log templates. In this embodiment, the abnormal parts predicted in the output of the absolute coding model are replaced with the output of the relative coding model. In this way, the normal working mode of the log workflow can be learned, and new log templates can be discovered, preventing error alarms caused by the appearance of new log templates and effectively reducing operation and maintenance costs.

[0088] This application provides a log detection method, apparatus, electronic device, and computer storage medium. The method includes: obtaining a trained absolute encoding model and a relative encoding model based on an acquired original log set; the trained absolute encoding model is used to characterize the specific meaning and temporal variation pattern of each log in the original log set; the trained absolute encoding model is used to characterize the temporal variation pattern of each log in the original log set; obtaining a log set to be detected, inputting the log set to be detected into the trained absolute encoding model and the relative encoding model respectively, and obtaining their respective output results; the output results represent the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected; when it is determined that there is an abnormal log in the log set to be detected based on the output result of the trained absolute encoding model, the corresponding content of the abnormal log in the output result of the trained relative encoding model is used to further determine whether the abnormal log is abnormal.

[0089] As can be seen, in this embodiment, after obtaining the trained absolute encoding model and relative encoding model from the original log set, these two models are used to perform anomaly detection on each log in the log set to be detected. Because the trained absolute encoding model is only trained on a limited number of logs in the original log set, it is very likely that newly appearing logs will be misidentified as abnormal logs when using this model for log prediction. In this case, the prediction results of the relative encoding model are used again for log anomaly detection. This is because the relative encoding model is only used to characterize the temporal change pattern of each log and ignores the specific meaning of the log. It will not misidentify new logs as abnormal logs because the model has not seen them before, thus reducing the occurrence of false alarms. In this way, when using these two models to detect logs in the log set to be detected, this embodiment can solve the problem of increased operation and maintenance costs caused by all new logs appearing during the log detection process being identified as abnormal logs by fusing the prediction results of the two models, thereby saving operation and maintenance costs.

[0090] To better illustrate the purpose of this application, further explanation will be provided based on the above embodiments.

[0091] Figure 2 This is a schematic diagram of another log detection process according to an embodiment of this application, such as... Figure 2 As shown, the process may include:

[0092] Step A1: Obtain the log index set.

[0093] For example, the logs in the original log set are composed of a series of instructions. The logs can be abstracted as a combination of templates and parameters. After removing the parameters through regular expression matching and other methods, the log templates are extracted from the logs. After deduplication, the log templates are mapped to indexes. In this way, the index corresponding to each log in the original log set can be obtained, that is, the log index set.

[0094] Here, the log workflow under normal working mode is regarded as a time series. When training the model, log workflows that have not experienced any anomalies are used to enable the model to learn the normal log workflow pattern. After the model is trained, it is used to detect whether the online logs are abnormal. When the online logs contradict the normal working mode, it is considered that an anomaly has occurred.

[0095] Step A2: Absolute encoding model training.

[0096] For example, the training process of the absolute encoding model includes three parts: input, embedding features, and output. First, after obtaining the log index set in step A1, the time series corresponding to the log index set can be learned through a two-layer attention model. The time series is processed in a sliding window manner. Through experiments, the sliding window size is selected as 50, that is, the next log is predicted using 50 logs in each sliding window. This embodiment of the application adopts a self-supervised method, so no labeled data is used. In the figure, [MASK] marks the position of the index corresponding to the log to be predicted, and the remaining log events (i.e., logs in each sliding window) are used as features to predict the log to be predicted at the [MASK] position. Taking the data 240, 81, 85, 88, 241, [MASK] 241 shown in the figure as an example, 240, 81, 85, 88, 241 represent the index in the first sliding window, and [MASK] 241 represents the position of the log to be predicted corresponding to index 241. The index in each sliding window and the index at the [MASK] position are input into the training of the absolute encoding model.

[0097] Furthermore, after the index corresponding to each log in the original log set is input into the absolute encoding model via a sliding window, it is first mapped into three embeddings: token embedding, segment embedding, and position embedding. Token embedding is the vector representation of each token, segment embedding is used to distinguish different sentences, and position embedding is used to learn the positional information of each token within a sentence. These three embeddings are summed to obtain the final embedding, which is then passed through an encoder. The encoder's final layer outputs the vector representation of the entire time series. To improve the performance and speed of time series learning, a two-layer encoder was chosen experimentally; therefore, the output of the second layer is the vector representation of the time series.

[0098] Finally, after the time series is mapped into a vector by the encoder, it passes through a fully connected neural network and outputs the classification result at the [MASK] position. If there are 1000 duplicate log templates after deduplication, there are 1000 deduplicated indices. At this point, the model will output the classification probability on these 1000 indices (corresponding to 1000 log templates). Subsequently, the classification probabilities of the absolute encoding model and the relative encoding model will be fused to output the final result.

[0099] Step A3: Relative encoding model training.

[0100] For example, the training process of the relative encoding model also includes three parts: input, embedding features, and output. Relative encoding involves sliding a window of 50 through the time series corresponding to the log index set. Indices within the same sliding window are represented using relative encoding. This means that after deduplicating the 50 log templates within the same window, they are encoded starting from 1, and the encoding index is incremented by 1 when encountering different logs. Taking the data 240, 81, 85, 88, 241, [MASK]241 shown in the figure as an example, since index 81 is the smallest, the encoding for index 81 is 1, and so on. Similarly, index 85 is encoded as 2, index 88 as 3, index 240 as 4, and index 241 as 5. That is, after the above relative encoding, the input to the relative encoding model is 4, 1, 2, 3, 5, [MASK]5. This encoding method allows the model to ignore the specific meaning of the encoding representation and only use it to learn the temporal variation patterns of the logs. When encountering a new log template index, due to the relative encoding, the model infers the result of the relative encoding based on the learned patterns. Subsequently, the true result of the absolute encoding is recovered through the mapping between the relative and absolute encodings. Since the embedding features and output classification results are the same as the feature representation process of the absolute encoding model, they will not be elaborated further here.

[0101] Step A4: Fuse the outputs of the absolute coding model and the relative coding model.

[0102] For example, after both the absolute encoding model and the relative encoding model have been trained, the trained absolute encoding model and relative encoding model are used to output the prediction results of the log to be predicted on the log set to be detected.

[0103] For example, after obtaining the index corresponding to each log in the log set to be detected, the index within each sliding window can be determined using a sliding window approach. The index within each sliding window is then sequentially input into the trained absolute encoding model to obtain the corresponding output result. Alternatively, when determining the index within each sliding window using the sliding window approach, the index within each sliding window can be directly encoded according to the index size. The encoded index within each sliding window is then input into the trained relative encoding model to obtain the corresponding output result. It should be noted that after encoding the index within each sliding window, the mapping relationship between the index and the encoding is recorded.

[0104] Since the absolute encoding model is trained on the original log set, its output represents the classification probability of each log in the original log set. The dimensionality of the absolute encoding model's output is the same as the number of logs in the original log set. For the relative encoding model, after obtaining its initial output (directly output by the trained model), the index within each sliding window before encoding is determined based on the pre-recorded mapping between indices and encodings. Then, based on the initial output of the relative encoding model, the indices within each sliding window before encoding, and the correspondence between indices and logs, the classification probability of each log in the log set to be detected (the next actual log corresponding to each log in each sliding window) is obtained, which is the output of the relative encoding model. Finally, the outputs of the absolute and relative encoding models can be fused.

[0105] In one embodiment, since the logs in the original log set are all normal logs, there are fewer log templates than the log set to be detected after deduplication. The number of logs in the log set to be detected is greater than that in the original log set. In order to fuse with the output of the relative coding model, the output dimension of the absolute coding model can be expanded to be the same as that of the output of the relative coding model. Thus, for a new log in the log set to be detected relative to the original log set, when the output of the relative coding model includes the classification probability of the log to be predicted as a new log, since the original log set does not include the new log, the output dimension of the original log set is smaller than that of the output of the relative coding model. Therefore, the output dimension of the absolute coding model can be expanded to be the same as that of the output of the relative coding model, and then the outputs of the absolute coding model and the relative coding model are fused.

[0106] Because absolute coding models use real data and achieve better learning results, they cannot predict newly emerging log templates. Therefore, the parts predicted as anomalous in the output of the absolute coding model are replaced with the output of the relative coding model. The logs with the highest classification probabilities in the final fusion result are then identified. If the log to be predicted (the next actual log corresponding to the log in the sliding window at the current moment) is among the top k logs, it indicates that the appearance of the log to be predicted conforms to the normal working pattern and no anomaly has occurred; otherwise, it is considered an anomaly. Thus, the final result can both learn the normal working pattern of the log workflow and discover newly emerging log templates, preventing error alerts caused by newly emerging log templates and reducing operational costs.

[0107] Understandably, since a large number of new logs that the model has not seen will be generated during system operation, the relative encoding model proposed in this application learns the generation rules of time series. This method can predict log events that have not appeared in the training set and solve the problem of false alarms caused by new logs. In addition, the model is trained in a self-supervised manner, and the position of the predicted log is marked with [MASK], which is consistent with the training form of the upstream task of the model. This can make use of the large number of logs generated in the system and reduce the dependence on labeled data.

[0108] Figure 3 This is a schematic diagram of the composition structure of the log detection device according to an embodiment of this application, as shown below. Figure 3 As shown, the device includes: a first obtaining module 300, a second obtaining module 301, and a determining module 302, wherein:

[0109] The first acquisition module 300 is used to obtain a trained absolute encoding model and a relative encoding model based on the acquired original log set; the trained absolute encoding model is used to characterize the specific meaning and temporal change pattern of each log in the original log set; the trained absolute encoding model is used to characterize the temporal change pattern of each log in the original log set.

[0110] The second module 301 is used to obtain the log set to be detected, and input the log set to be detected into the trained absolute encoding model and relative encoding model respectively to obtain their respective output results; the output result represents the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected.

[0111] The determination module 302 is used to determine whether the abnormal log is abnormal again by using the corresponding content of the abnormal log in the output result of the trained relative encoding model when it is determined that there is an abnormal log in the log set to be detected based on the output result of the trained absolute encoding model.

[0112] In some embodiments, the second obtaining module 301 is used to input the log set to be detected into the trained absolute encoding model and relative encoding model, respectively, including:

[0113] Determine the log template for each log in the log set to be detected;

[0114] Map the log template of each log in the log set to be detected to an index;

[0115] The index corresponding to each log in the log set to be detected is determined by a sliding window method, and the index in each sliding window is input into the trained absolute encoding model; and when determining the index in each sliding window, the index in each sliding window is encoded according to the index size, and the encoded index in each sliding window is input into the trained relative encoding model.

[0116] In some embodiments, the first obtaining module 300 is used to obtain the trained absolute coding model and relative coding model based on the acquired original log set, including:

[0117] Determine the log template for each log in the original log set;

[0118] Map the log template of each log in the original log set to an index;

[0119] The index corresponding to each log in the original log set is determined using a sliding window method, and the index within each sliding window and the index of the corresponding next actual log are determined. The index within each sliding window and the index of the corresponding next actual log are then input into an absolute encoding model for training to obtain a trained absolute encoding model. In addition, when determining the index within each sliding window, the index within each sliding window and the index of the corresponding next actual log are encoded according to the index size. The encoded index within each sliding window and the index of the corresponding next actual log are then input into a relative encoding model for training to obtain a trained relative encoding model.

[0120] In some embodiments, the determining module 302 is configured to determine, based on the output of the trained absolute coding model, that there are abnormal logs in the log set to be detected, including:

[0121] Based on the output of the trained absolute encoding model, if the log to be predicted is not among the logs with the highest classification probability (k), then the log to be predicted is determined to be an abnormal log.

[0122] In some embodiments, each log in the original log set is a normal log.

[0123] In practical applications, the first obtaining module 300, the second obtaining module 301, and the determining module 302 can all be implemented by a processor located in an electronic device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0124] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0125] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] Specifically, the computer program instructions corresponding to a log detection method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to a log detection method in the storage media are read or executed by an electronic device, any of the log detection methods in the aforementioned embodiments can be implemented.

[0127] Based on the same technical concept as the foregoing embodiments, see Figure 4 It illustrates an electronic device 400 provided in an embodiment of this application, which may include: a memory 401 and a processor 402; wherein,

[0128] Memory 401 is used to store computer programs and data;

[0129] Processor 402 is configured to execute a computer program stored in memory to implement any of the log detection methods described in the foregoing embodiments.

[0130] In practical applications, the memory 401 mentioned above can be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 402.

[0131] The processor 402 described above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that for different model training devices, the electronic device used to implement the above processor function can also be other types, and this application embodiment does not specifically limit the specific types.

[0132] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0133] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0134] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0135] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0136] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0138] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A log detection method, characterized in that, The method includes: Based on the obtained original log set, a trained absolute encoding model and a trained relative encoding model are obtained; the trained absolute encoding model is used to characterize the specific meaning and temporal variation pattern of each log in the original log set; the trained relative encoding model is used to characterize the temporal variation pattern of each log in the original log set. Obtain the log set to be detected, and input the log set to be detected into the trained absolute encoding model and relative encoding model respectively to obtain their respective output results; the output result represents the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected. When it is determined that there are abnormal logs in the log set to be detected based on the output of the trained absolute encoding model, the corresponding content of the abnormal logs in the output of the trained relative encoding model is used to determine whether the abnormal logs are abnormal.

2. The method according to claim 1, characterized in that, The step of inputting the log set to be detected into the trained absolute encoding model and relative encoding model respectively includes: Determine the log template for each log in the log set to be detected; Map the log template of each log in the log set to be detected to an index; The index corresponding to each log in the log set to be detected is determined by a sliding window method, and the index in each sliding window is input into the trained absolute encoding model; and when determining the index in each sliding window, the index in each sliding window is encoded according to the index size, and the encoded index in each sliding window is input into the trained relative encoding model.

3. The method according to claim 1, characterized in that, The process of obtaining the trained absolute encoding model and relative encoding model based on the acquired original log set includes: Determine the log template for each log in the original log set; Map the log template of each log in the original log set to an index; The index corresponding to each log in the original log set is determined using a sliding window method, and the index within each sliding window and the index of the corresponding next actual log are determined. The index within each sliding window and the index of the corresponding next actual log are then input into an absolute encoding model for training to obtain a trained absolute encoding model. In addition, when determining the index within each sliding window, the index within each sliding window and the index of the corresponding next actual log are encoded according to the index size. The encoded index within each sliding window and the index of the corresponding next actual log are then input into a relative encoding model for training to obtain a trained relative encoding model.

4. The method according to claim 1, characterized in that, The step of determining the presence of abnormal logs in the log set to be detected based on the output of the trained absolute coding model includes: Based on the output of the trained absolute encoding model, if the log to be predicted is not among the logs with the highest classification probability (k), then the log to be predicted is determined to be an abnormal log.

5. The method according to claim 1, characterized in that, Each log in the original log set is a normal log.

6. A log detection device, characterized in that, The device includes: The first acquisition module is used to obtain a trained absolute encoding model and a relative encoding model based on the acquired original log set; the trained absolute encoding model is used to characterize the specific meaning and temporal variation pattern of each log in the original log set; the trained relative encoding model is used to characterize the temporal variation pattern of each log in the original log set. The second module is used to obtain the log set to be detected, and input the log set to be detected into the trained absolute encoding model and relative encoding model respectively to obtain their respective output results; the output result represents the classification probability of the log to be predicted as each log in the log set to be detected; the log to be predicted is a log in the log set to be detected. The determination module is used to, when determining that there are abnormal logs in the log set to be detected based on the output results of the trained absolute encoding model, use the corresponding content of the abnormal logs in the output results of the trained relative encoding model to determine whether the abnormal logs are abnormal again.

7. The apparatus according to claim 6, characterized in that, The second obtaining module is used to input the log set to be detected into the trained absolute encoding model and relative encoding model, respectively, including: Determine the log template for each log in the log set to be detected; Map the log template of each log in the log set to be detected to an index; The index corresponding to each log in the log set to be detected is determined by a sliding window method, and the index in each sliding window is input into the trained absolute encoding model; and when determining the index in each sliding window, the index in each sliding window is encoded according to the index size, and the encoded index in each sliding window is input into the trained relative encoding model.

8. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1 to 5.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 5.

Images