A multi-source heterogeneous alarm log anomaly detection method and device, electronic equipment and storage medium

Abstract

The application discloses a multi-source heterogeneous alarm log anomaly detection method and device, electronic equipment and storage medium, and the method comprises the steps of acquiring the multi-source heterogeneous alarm log of a to-be-detected period and time interval data between adjacent logs; analyzing the log to generate a log template; inputting the log template into a semantic model to generate a log semantic feature vector; performing linear transformation on the time interval data to generate a log time feature vector; converting each word in the log template into a numerical vector to generate a log index feature vector; splicing the log semantic feature vector, the time feature vector and the index feature vector to generate a log fusion feature vector; inputting the log fusion feature vector into an anomaly detection model, randomly masking a plurality of features and predicting the results; and if the proportion of the prediction results of the masked features and the actual results is inconsistent exceeds a preset threshold, it is determined that there is an abnormal alarm in the log. Through the implementation of the application, the accuracy of log anomaly detection can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a multi-source heterogeneous alarm log anomaly detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of industrial internet, more and more intelligent devices and systems are connected to the network, resulting in a sharp increase in the number of alarm logs. Alarm logs play an important role in helping to identify potential security threats, system failures and abnormal behaviors. However, with the explosion of information, how to quickly and accurately find anomalies from massive, multi-source, heterogeneous alarm logs has become a key to ensuring the safe and stable operation of industrial internet systems. Especially in the case of diverse sources of security alarm data and complex structure, efficient anomaly detection can not only reduce false positives and improve response efficiency, but also help security analysts quickly identify high-risk events and effectively support security protection and incident traceability analysis. Therefore, it is particularly important to develop anomaly detection technology that can handle multi-source heterogeneous alarm logs, which can improve network security protection capabilities, reduce security risks and ensure reliable operation of industrial internet.

[0003] Although there are many technologies for processing multi-source heterogeneous alarm logs, existing solutions still face many challenges. Many existing technologies rely too much on template structures, ignoring the semantic information and time information of logs, making it difficult to deeply mine the underlying features of logs, which leads to missed detection and misjudgment of many potential threats. SUMMARY

[0004] The present application provides a multi-source heterogeneous alarm log anomaly detection method, device, electronic equipment and storage medium. By implementing the present application, the expression ability of log features can be enhanced, and the accuracy of log anomaly detection can be improved.

[0005] An embodiment of the present application provides a multi-source heterogeneous alarm log anomaly detection method, comprising:

[0006] Obtaining multi-source heterogeneous alarm logs of a to-be-detected period and time interval data between adjacent logs of the multi-source heterogeneous alarm logs;

[0007] Parsing the multi-source heterogeneous alarm logs to generate log templates;

[0008] inputting the log template into a semantic model, so that the semantic model generates a log semantic feature vector according to the log template; performing linear transformation on the time interval data to generate a log time feature vector; converting each word in the log template into a numerical vector to generate a log index feature vector; splicing the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector;

[0009] inputting the log fusion feature vector into an anomaly detection model, so that the anomaly detection model randomly masks some features in the log fusion feature vector and generates a prediction result of the masked features; if a proportion of the prediction result of the masked features that is inconsistent with the actual situation exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the to-be-detected period.

[0010] Further, the parsing of the multi-source heterogeneous alarm log to generate a log template comprises:

[0011] based on a regular expression, desensitizing and cleaning the multi-source heterogeneous alarm log to generate a preprocessed multi-source heterogeneous alarm log;

[0012] performing word segmentation on the preprocessed multi-source heterogeneous alarm log to generate a plurality of word lists;

[0013] constructing a parse tree according to the word list to generate a multi-source heterogeneous alarm log parse tree;

[0014] generating a log template according to a leaf node in the multi-source heterogeneous alarm log parse tree.

[0015] Further, the training of the semantic model comprises:

[0016] obtaining a log data set, parsing the log data set to generate a first training log template;

[0017] dividing the first training log template into a plurality of batches of first training sample sets according to a preset number;

[0018] inputting each batch of first training sample sets into the semantic model in turn to train the semantic model until a preset training number is reached; wherein the semantic model outputs a log semantic feature vector of each first training log template in the first training sample set when receiving each batch of first training sample sets; calculates the similarity between the log semantic feature vectors of each first training log template to generate a plurality of similarity results; calculates a first loss function value through a first loss function according to the similarity results; and updates the semantic model according to the first loss function value by using an optimizer.

[0019] Furthermore, the step of performing a linear transformation on the time interval data to generate a log time feature vector includes:

[0020] The time interval data is linearly transformed according to the following formula to generate a time vector:

[0021] k = softmax(Δ t W+b)

[0022] Where k is the time vector; Δ t The data represents time intervals; W is the weight matrix; b is the random bias vector.

[0023] Calculate the product of the time vector and the preset weighted identity matrix to generate the log time feature vector.

[0024] Furthermore, the step of converting each word in the log template into a numerical vector to generate a log index feature vector includes:

[0025] Construct a trie based on the words in the log template; where each node in the trie represents a character in a word, and the path represents the sequence of characters in the word.

[0026] By using word embedding technology, each word in the trie is converted into a numerical vector, generating the corresponding numerical vector result;

[0027] The numerical vector result is stored in the corresponding node of the trie;

[0028] Based on the word order in the log template, the numerical vector corresponding to each word is found in the trie and arranged according to the word order in the log template to generate a log index feature vector.

[0029] Furthermore, the training of the anomaly detection model includes:

[0030] Obtain the log fusion feature vector dataset, and randomly divide the log fusion feature training vectors in the log fusion feature vector dataset into several batches of second training sample sets according to a preset number;

[0031] Each batch of the second training sample set is sequentially input into the anomaly detection model to train the model until a preset number of training iterations are reached. When the anomaly detection model receives each batch of the second training sample set, it randomly masks several features of each log fusion feature training vector in the second training sample set and generates prediction results for the masked features of each log fusion feature training vector. Based on the prediction results and the corresponding masked features, a second loss function value is calculated using a second loss function. The anomaly detection model is then updated using an optimizer based on the second loss function value.

[0032] Furthermore, the acquisition of the log fusion feature vector dataset includes:

[0033] The log dataset is parsed to generate a second training log template;

[0034] The second training log template is input into the semantic model so that the semantic model generates a training log semantic feature vector based on the second training log template.

[0035] Extract the time interval data between adjacent abnormal logs in the log dataset;

[0036] The abnormal log time interval data is linearly transformed to generate a training log time feature vector.

[0037] Each word in the second training log template is converted into a numerical vector to generate a training log index feature vector.

[0038] The training log semantic feature vector, training log time feature vector, and training log index feature vector are concatenated to generate a log fusion feature vector dataset.

[0039] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0040] An embodiment of the present invention provides a multi-source heterogeneous alarm log anomaly detection device, comprising: an alarm log data acquisition module, a log template generation module, a log fusion feature vector generation module, and an anomaly detection module;

[0041] The alarm log data acquisition module is used to acquire multi-source heterogeneous alarm logs for the period to be detected and the time interval data between adjacent logs of the multi-source heterogeneous alarm logs;

[0042] The log template generation module is used to parse the multi-source heterogeneous alarm logs and generate log templates;

[0043] The log fusion feature vector generation module is used to input the log template into the semantic model so that the semantic model generates a log semantic feature vector based on the log template; to perform a linear transformation on the time interval data to generate a log time feature vector; to convert each word in the log template into a numerical vector to generate a log index feature vector; and to concatenate the log semantic feature vector, the log time feature vector, and the log index feature vector to generate a log fusion feature vector.

[0044] The anomaly detection module is used to input the log fusion feature vector into the anomaly detection model, so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results for the masked features. The anomaly detection model is trained unsupervised using a masking prediction method on the log fusion feature vector dataset. The log fusion feature vector dataset is composed of log semantic feature vectors, log time feature vectors, and log index feature vectors extracted from the log dataset. If the proportion of the predicted results of the masked features that are inconsistent with the actual results exceeds a preset threshold, it is determined that there are abnormal alarms in the multi-source heterogeneous alarm logs of the detection period.

[0045] Based on the above method embodiments, the present invention provides corresponding electronic device embodiments.

[0046] An embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the multi-source heterogeneous alarm log anomaly detection methods described in the above-described method embodiments.

[0047] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments.

[0048] One embodiment of the present invention provides a storage medium storing a computer program thereon, which, when executed by a processor, can implement the multi-source heterogeneous alarm log anomaly detection method described in any of the above-described method embodiments.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] This invention provides a method, apparatus, electronic device, and storage medium for detecting anomalies in multi-source heterogeneous alarm logs. The method acquires alarm logs and adjacent log time interval data for the period to be detected; parses the log data to extract common structures and generates a log template; inputs the log template into a semantic model to generate a log semantic feature vector; performs a linear transformation on the time interval data to generate a log time feature vector; converts each word in the log template into a numerical vector to generate a log index feature vector; subsequently, concatenates the semantic features, time features, and index feature vectors to generate a log fusion feature vector; inputs the log fusion feature vector into an anomaly detection model, causing the anomaly detection model to randomly mask some features and predict the masked elements for anomaly detection; if the proportion of prediction results that do not match the actual values ​​exceeds a preset threshold, an abnormal alarm is determined to exist.

[0051] This invention parses multi-source heterogeneous alarm logs and, leveraging the powerful semantic expression capabilities of semantic models, transforms the parsed log templates into semantic feature vectors. Furthermore, it fuses the log's temporal and index feature vectors with the semantic feature vectors. This effectively addresses the problem of existing technologies over-relying on template structures and neglecting log semantic and temporal information, thereby deeply mining the underlying features of the logs, significantly enhancing the expressive power of log features, and improving the accuracy of log anomaly detection. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a method for detecting anomalies in multi-source heterogeneous alarm logs according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of a multi-source heterogeneous alarm log anomaly detection device provided in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting anomalies in multi-source heterogeneous alarm logs, which includes at least the following steps:

[0056] Step S1: Obtain the multi-source heterogeneous alarm logs for the period to be detected and the time interval data between adjacent logs of the multi-source heterogeneous alarm logs.

[0057] Specifically, acquiring multi-source heterogeneous alarm logs for the period to be detected, along with the time intervals between adjacent logs, firstly requires collecting alarm information from log systems of different sources, including but not limited to device status, fault alarms, and operation records. Then, the timestamp information of adjacent alarm logs is extracted, and the time intervals between adjacent log entries are calculated as temporal features, providing necessary time-series data for subsequent feature extraction and analysis. This log and time interval data will provide fundamental support for subsequent steps such as log parsing, feature extraction, and anomaly detection.

[0058] Step S2: Parse the multi-source heterogeneous alarm logs and generate a log template.

[0059] In a preferred embodiment, the step of parsing the multi-source heterogeneous alarm logs and generating a log template includes:

[0060] Based on regular expressions, the multi-source heterogeneous alarm logs are de-identified and cleaned to generate pre-processed multi-source heterogeneous alarm logs.

[0061] The preprocessed multi-source heterogeneous alarm logs are segmented to generate a list of words.

[0062] Based on the word list, construct a parse tree to generate a multi-source heterogeneous alarm log parse tree;

[0063] Log templates are generated based on the leaf nodes in the multi-source heterogeneous alarm log parsing tree.

[0064] It should be noted that the multi-source heterogeneous alarm logs are anonymized and cleaned using regularization expressions. First, sensitive information (such as IP addresses, user information, and device IDs) is anonymized using regularization expressions to ensure data privacy and security. The cleaned data is then converted to a standardized format, removing redundant or irrelevant information to generate pre-processed multi-source heterogeneous alarm logs. This lays the foundation for subsequent data analysis and feature extraction.

[0065] Next, the preprocessed multi-source heterogeneous alarm logs are segmented. Each log entry is split into words, symbols, or numbers using a word segmentation tool, generating a list of words.

[0066] Based on the generated list of words, a parse tree is constructed. The parse tree construction process analyzes the dependencies and syntactic structure between words, gradually building a hierarchical tree structure that reflects the hierarchy and relationships of different elements in the log, thus forming a multi-source heterogeneous alarm log parse tree.

[0067] Finally, key information is extracted from the leaf nodes of the multi-source heterogeneous alarm log parsing tree, and log templates are generated based on these leaf nodes. The log templates reflect the structured pattern of the log data, unifying different alarm logs into a standardized format for easier subsequent analysis and processing.

[0068] Step S3: Input the log template into the semantic model so that the semantic model generates a log semantic feature vector based on the log template; perform a linear transformation on the time interval data to generate a log time feature vector; convert each word in the log template into a numerical vector to generate a log index feature vector; and concatenate the log semantic feature vector, the log time feature vector, and the log index feature vector to generate a log fusion feature vector.

[0069] In a preferred embodiment, the step of linearly transforming the time interval data to generate a log time feature vector includes:

[0070] Perform a linear transformation on the time interval data according to the following formula to generate a time vector:

[0071] k = softmax(Δ t W + b)

[0072] where k is the time vector; Δ t is the time interval data; W is the weight matrix; b is the random bias vector;

[0073] Calculate the product of the time vector and a preset weighted identity matrix to generate a log time feature vector, that is:

[0074] T = kE

[0075] where T is the log time feature vector; E is the weighted identity matrix.

[0076] In one embodiment, the converting each word in the log template into a numerical vector to generate a log index feature vector includes:

[0077] Construct a trie according to the words in the log template; wherein, the nodes of the trie represent one character in the word, and the path represents the character sequence of the word;

[0078] Convert each word in the trie into a numerical vector through word embedding technology to generate a corresponding numerical vector result;

[0079] Store the numerical vector result in the corresponding node of the trie;

[0080] According to the word order in the log template, find the numerical vector corresponding to each word in the trie and arrange them in the order of the words in the log template to generate a log index feature vector.

[0081] Specifically, construct a trie according to the words in the log template. A trie is a tree - like data structure where each node represents one character in the word and the path represents the character sequence of the word. For example, for the word "fault", the node of the trie is the character "故" and its next - level node is the character "障". In this way, the trie effectively organizes the character sequences of each word in the log template.

[0082] Next, word embedding techniques are used to convert each word in the trie into a corresponding numerical vector. Word embedding maps words to vectors in a high-dimensional space, and by training the model, words with similar semantics are grouped closer together in the vector space. Through this technique, each word in the trie is converted into a numerical vector, generating corresponding numerical vector results. These vectors can capture the semantic features of the words, providing valuable information for subsequent log analysis. Word embedding techniques used here can include Word2Vec, GloVe, and ELMo.

[0083] The numerical vector result of each word is stored in the corresponding node of the trie, so that the trie records not only the information of the character sequence, but also the semantic features of each word. This step ensures that each node in the trie contains both character-level and semantic-level information.

[0084] Based on the word order in the log template, the numerical vector corresponding to each word is retrieved from the trie. During the search, the corresponding numerical vectors are extracted according to the word order in the log template, and these vectors are arranged in order to generate the final log index feature vector. This feature vector combines the word order in the log template with the semantic information of each word, providing an efficient representation for subsequent log analysis tasks.

[0085] It should be noted here that the training of the semantic model includes:

[0086] Obtain the log dataset, parse the log dataset, and generate the first training log template;

[0087] The first training log template is randomly divided into several batches of the first training sample set according to a preset number;

[0088] The first training sample set of each batch is sequentially input into the semantic model to train the semantic model until the preset number of training iterations is reached. Specifically, when the semantic model receives a new batch of the first training sample set, it outputs the log semantic feature vectors of each first training log template in the first training sample set. The similarity between the log semantic feature vectors of each first training log template is calculated, generating several similarity results. Based on the similarity results, a first loss function value is calculated using a first loss function. Finally, the semantic model is updated using an optimizer based on the first loss function value.

[0089] In practice, the log dataset may include any one or a combination of the following: HDFS, BGL, and Tbird. The HDFS dataset comes from over 200 Amazon EC2 nodes running Hadoop-based jobs and contains 11,175,629 raw log messages, of which 284,818 are marked as anomalous. This log data is primarily used for log analysis and anomaly detection in big data environments, helping to identify potential problems in distributed computing environments. The BGL dataset is collected from the BlueGene / L supercomputer system and contains 4,747,963 log messages, of which 348,469 are anomalous logs. It is mainly used for research on log analysis and fault diagnosis in high-performance computing environments. The Tbird dataset comes from the Thunderbird supercomputer system at Sandia National Laboratories and contains a large amount of log information related to system operating status and anomaly records. It is used for anomaly detection and performance optimization of supercomputer systems. In this invention, we selected the first 20,000,000 log messages to form the dataset, of which 758,562 are anomalous. These three datasets provide valuable data support for log analysis, anomaly detection, fault diagnosis, and system performance optimization, and are widely used in the field of high-performance computing.

[0090] Furthermore, the first loss function can be the cross-entropy function; the optimizer can be the Admn optimizer.

[0091] Step S4: Input the log fusion feature vector into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results of the masked features; if the proportion of the prediction results of the masked features that are inconsistent with the actual values ​​exceeds a preset threshold, it is determined that there are abnormal alarms in the multi-source heterogeneous alarm logs of the period to be detected.

[0092] Specifically, the log fusion feature vector is input into the anomaly detection model, enabling the model to perform anomaly detection based on the fusion feature vector. During this process, the anomaly detection model randomly masks several features in the log fusion feature vector to simulate missing data, and generates predictions for the masked features based on the remaining features. By comparing these predictions with the actual original feature values, if the proportion of the predicted values ​​for the masked features that do not match the actual values ​​exceeds a preset threshold, it is determined that there are abnormal alarms in the multi-source heterogeneous alarm logs for that period. This method, through masking prediction and error threshold determination mechanisms, can effectively capture abnormal patterns, helping to identify potential system failures or security threats, thereby improving the accuracy and reliability of anomaly detection. The preset threshold here can be flexibly adjusted according to actual conditions; a recommended value of 30% is suggested here.

[0093] In a preferred embodiment, the training of the anomaly detection model includes:

[0094] Obtain the log fusion feature vector dataset, and randomly divide the log fusion feature training vectors in the log fusion feature vector dataset into several batches of second training sample sets according to a preset number;

[0095] Each batch of the second training sample set is sequentially input into the anomaly detection model to train the model until a preset number of training iterations are reached. When the anomaly detection model receives each batch of the second training sample set, it randomly masks several features of each log fusion feature training vector in the second training sample set and generates prediction results for the masked features of each log fusion feature training vector. Based on the prediction results and the corresponding masked features, a second loss function value is calculated using a second loss function. The anomaly detection model is then updated using an optimizer based on the second loss function value.

[0096] The second loss function here is specifically:

[0097] L loss =w T L T +w F L F +w I L I

[0098] Among them, L loss The second loss function; w T The weight coefficients for training the log time feature vector; w F The weight coefficients for training the log semantic feature vector; w I The weight coefficients for the training log index feature vector; L T The loss for training log time feature vectors; L F The loss for training log semantic feature vectors; L I The loss for training log index feature vectors;

[0099] The losses for the training log time feature vector, the training log semantic feature vector, and the training log index feature vector are calculated using the following formulas:

[0100]

[0101] Where N is the number of samples in the second training sample set in the first batch; M T To identify the masked features in the training log time feature vector; M F To identify the masked features in the training log semantic feature vector; M I To identify the masked features in the training log index feature vector; This represents the actual value of the i-th masked feature in the j-th training log time feature vector; This represents the actual value of the ith masked feature in the j-th training log semantic feature vector. This represents the actual value of the ith masked feature in the j-th training log index feature vector; The predicted value of the i-th masked feature in the j-th training log time feature vector; For the first j In the training log semantic feature vector, the th... i Predicted values ​​for each masked feature; For the first j The predicted value of the i-th masked feature in the training log index feature vector.

[0102] In an optional embodiment, the masking ratios of the training log time feature vector, the training log index feature vector, and the training log semantic feature vector can be freely set. Here, the masking ratio of the training log time feature vector can be set to 20%, the masking ratio of the training log index feature vector can be set to 10%, and the masking ratio of the training log semantic feature vector can be set to 40%.

[0103] It should be noted that the acquisition of the log fusion feature vector dataset includes:

[0104] The log dataset is parsed to generate a second training log template;

[0105] The second training log template is input into the semantic model so that the semantic model generates a training log semantic feature vector based on the second training log template.

[0106] Extract the time interval data between adjacent abnormal logs in the log dataset;

[0107] The abnormal log time interval data is linearly transformed to generate a training log time feature vector.

[0108] Each word in the second training log template is converted into a numerical vector to generate a training log index feature vector.

[0109] The training log semantic feature vector, training log time feature vector, and training log index feature vector are concatenated to generate a log fusion feature vector dataset.

[0110] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0111] like Figure 2As shown, an embodiment of the present invention provides a multi-source heterogeneous alarm log anomaly detection device, including: an alarm log data acquisition module, a log template generation module, a log fusion feature vector generation module, and an anomaly detection module;

[0112] The alarm log data acquisition module is used to acquire multi-source heterogeneous alarm logs for the period to be detected and the time interval data between adjacent logs of the multi-source heterogeneous alarm logs;

[0113] The log template generation module is used to parse the multi-source heterogeneous alarm logs and generate log templates;

[0114] The log fusion feature vector generation module is used to input the log template into the semantic model so that the semantic model generates a log semantic feature vector based on the log template; to perform a linear transformation on the time interval data to generate a log time feature vector; to convert each word in the log template into a numerical vector to generate a log index feature vector; and to concatenate the log semantic feature vector, the log time feature vector, and the log index feature vector to generate a log fusion feature vector.

[0115] The anomaly detection module is used to input the log fusion feature vector into the anomaly detection model, so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results for the masked features. The anomaly detection model is trained unsupervised using a masking prediction method on the log fusion feature vector dataset. The log fusion feature vector dataset is composed of log semantic feature vectors, log time feature vectors, and log index feature vectors extracted from the log dataset. If the proportion of the predicted results of the masked features that are inconsistent with the actual results exceeds a preset threshold, it is determined that there are abnormal alarms in the multi-source heterogeneous alarm logs of the detection period.

[0116] It should be noted that the embodiments of the device described above correspond to the embodiments of the present invention described above, and can realize the multi-source heterogeneous alarm log anomaly detection method described in any one of the above embodiments of the present invention. Furthermore, the embodiments of the device described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without creative effort.

[0117] Based on the above-described method embodiments of the present invention, a corresponding embodiment of an electronic device is provided.

[0118] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the multi-source heterogeneous alarm log anomaly detection method according to any one of the present invention, or the processor executes the computer program to implement the functions of each module in the above-described device embodiments.

[0119] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device.

[0120] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0121] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0122] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0123] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments;

[0124] Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the storage medium is located to execute any one of the above-described multi-source heterogeneous alarm log anomaly detection methods of the present invention.

[0125] The aforementioned storage medium is a computer-readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0126] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0127] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in multi-source heterogeneous alarm logs, characterized in that, include: Obtain multi-source heterogeneous alarm logs for the period to be detected, as well as the time interval data between adjacent logs of the multi-source heterogeneous alarm logs; The multi-source heterogeneous alarm logs are parsed to generate log templates; The log template is input into the semantic model so that the semantic model generates a log semantic feature vector based on the log template; the time interval data is linearly transformed to generate a log time feature vector. Construct a trie based on the words in the log template; where each node in the trie represents a character in a word, and the path represents the sequence of characters in the word. By using word embedding technology, each word in the trie is converted into a numerical vector, generating the corresponding numerical vector result; The numerical vector result is stored in the corresponding node of the trie; Based on the word order in the log template, the numerical vector corresponding to each word is found in the trie and arranged according to the word order in the log template to generate a log index feature vector; The log semantic feature vector, log time feature vector, and log index feature vector are concatenated to generate a log fusion feature vector; The log fusion feature vector is input into the anomaly detection model so that the anomaly detection model randomly masks some features in the log fusion feature vector and generates prediction results of the masked features; if the proportion of the prediction results of the masked features that are inconsistent with the actual values ​​exceeds a preset threshold, it is determined that there are abnormal alarms in the multi-source heterogeneous alarm logs of the period to be detected. The training of the anomaly detection model includes: Obtain the log fusion feature vector dataset, and randomly divide the log fusion feature training vectors in the log fusion feature vector dataset into several batches of second training sample sets according to a preset number; Each batch of the second training sample set is sequentially input into the anomaly detection model to train the model until a preset number of training iterations are reached. When the anomaly detection model receives each batch of the second training sample set, it randomly masks several features of each log fusion feature training vector in the second training sample set and generates prediction results for the masked features of each log fusion feature training vector. Based on the prediction results and the corresponding masked features, a second loss function value is calculated using a second loss function. The anomaly detection model is then updated using an optimizer based on the second loss function value.

2. The method for detecting anomalies in multi-source heterogeneous alarm logs as described in claim 1, characterized in that, The step of parsing the multi-source heterogeneous alarm logs and generating a log template includes: Based on regular expressions, the multi-source heterogeneous alarm logs are de-identified and cleaned to generate pre-processed multi-source heterogeneous alarm logs. The preprocessed multi-source heterogeneous alarm logs are segmented to generate a list of words. Based on the word list, construct a parse tree to generate a multi-source heterogeneous alarm log parse tree; Log templates are generated based on the leaf nodes in the multi-source heterogeneous alarm log parsing tree.

3. The method for detecting anomalies in multi-source heterogeneous alarm logs as described in claim 2, characterized in that, The training of the semantic model includes: Obtain the log dataset, parse the log dataset, and generate the first training log template; The first training log template is randomly divided into several batches of the first training sample set according to a preset number; The first training sample set of each batch is sequentially input into the semantic model to train the semantic model until the preset number of training iterations is reached. Specifically, when the semantic model receives a new batch of the first training sample set, it outputs the log semantic feature vectors of each first training log template in the first training sample set. The similarity between the log semantic feature vectors of each first training log template is calculated, generating several similarity results. Based on the similarity results, a first loss function value is calculated using a first loss function. Finally, the semantic model is updated using an optimizer based on the first loss function value.

4. The method for detecting anomalies in multi-source heterogeneous alarm logs as described in claim 3, characterized in that, The step of performing a linear transformation on the time interval data to generate a log time feature vector includes: The time interval data is linearly transformed according to the following formula to generate a time vector: in, It is a time vector; Data is for time intervals; This is the weight matrix; It is a random deviation vector; Calculate the product of the time vector and the preset weighted identity matrix to generate the log time feature vector.

5. The method for detecting anomalies in multi-source heterogeneous alarm logs as described in claim 4, characterized in that, The acquisition of the log fusion feature vector dataset includes: The log dataset is parsed to generate a second training log template; The second training log template is input into the semantic model so that the semantic model generates a training log semantic feature vector based on the second training log template. Extract the time interval data between adjacent abnormal logs in the log dataset; The abnormal log time interval data is linearly transformed to generate a training log time feature vector. Each word in the second training log template is converted into a numerical vector to generate a training log index feature vector. The training log semantic feature vector, training log time feature vector, and training log index feature vector are concatenated to generate a log fusion feature vector dataset.

6. A multi-source heterogeneous alarm log anomaly detection device, characterized in that, include: The system includes an alarm log data acquisition module, a log template generation module, a log fusion feature vector generation module, and an anomaly detection module. The alarm log data acquisition module is used to acquire multi-source heterogeneous alarm logs for the period to be detected and the time interval data between adjacent logs of the multi-source heterogeneous alarm logs; The log template generation module is used to parse the multi-source heterogeneous alarm logs and generate log templates; The log fusion feature vector generation module is used to input the log template into the semantic model so that the semantic model generates a log semantic feature vector based on the log template; and to perform a linear transformation on the time interval data to generate a log time feature vector. Based on the words in the log template, a trie is constructed; where each node in the trie represents a character in a word, and the path represents a character sequence of the word; using word embedding technology, each word in the trie is converted into a numerical vector, generating a corresponding numerical vector result; the numerical vector result is stored in the corresponding node of the trie; according to the word order in the log template, the numerical vector corresponding to each word is found in the trie, and arranged according to the word order in the log template to generate a log index feature vector; the log semantic feature vector, log time feature vector, and log index feature vector are concatenated to generate a log fusion feature vector; The anomaly detection module is used to input the log fusion feature vector into the anomaly detection model, so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results for the masked features. The anomaly detection model is trained unsupervised using a masking prediction method on the log fusion feature vector dataset. The log fusion feature vector dataset is composed of log semantic feature vectors, log time feature vectors, and log index feature vectors extracted from the log dataset. If the proportion of the predicted results of the masked features that are inconsistent with the actual results exceeds a preset threshold, it is determined that there are abnormal alarms in the multi-source heterogeneous alarm logs of the detection period. The training of the anomaly detection model includes: Obtain the log fusion feature vector dataset, and randomly divide the log fusion feature training vectors in the log fusion feature vector dataset into several batches of second training sample sets according to a preset number; Each batch of the second training sample set is sequentially input into the anomaly detection model to train the model until a preset number of training iterations are reached. When the anomaly detection model receives each batch of the second training sample set, it randomly masks several features of each log fusion feature training vector in the second training sample set and generates prediction results for the masked features of each log fusion feature training vector. Based on the prediction results and the corresponding masked features, a second loss function value is calculated using a second loss function. The anomaly detection model is then updated using an optimizer based on the second loss function value.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can implement the multi-source heterogeneous alarm log anomaly detection method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program can implement the multi-source heterogeneous alarm log anomaly detection method according to any one of claims 1 to 5.

Images