Multi-source heterogeneous alarm log anomaly detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-source heterogeneous alarm log anomaly detection method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining multi-source heterogeneous alarm logs in a to-be-detected time period and time interval data between adjacent logs; analyzing the log to generate a log template; inputting the log template into a semantic model to generate a log semantic feature vector; performing linear transformation on the time interval data to generate a log time feature vector; converting each word in the log template into a numerical vector, and generating a log index feature vector; splicing the log semantic feature vector, the time feature vector and the index feature vector to generate a log fusion feature vector; inputting the log fusion feature vector into an anomaly detection model, randomly shielding a plurality of features and predicting a result; and if the proportion of the prediction result of the shielding feature to the actual inconsistency exceeds a preset threshold value, determining that an abnormal alarm exists in the log. By implementing the method, the accuracy of log anomaly detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a multi-source heterogeneous alarm log anomaly detection method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of the industrial Internet, more and more smart devices and systems are connected to the network, resulting in a sharp increase in the number of alarm logs. Alarm logs play an important role in helping to identify potential security threats, system failures, and abnormal behaviors. However, with the surge in the amount of information, how to quickly and accurately detect anomalies from massive, multi-source, and heterogeneous alarm logs has become the key to ensuring the security and stable operation of industrial Internet systems. Especially when the sources of security alarm data are diverse and the structure is complex, efficient anomaly detection can not only reduce the false alarm rate and improve response efficiency, but also help security analysts quickly lock high-risk events and effectively support security protection and accident tracing analysis. Therefore, it is particularly important to develop anomaly detection technology that can process multi-source heterogeneous alarm logs, which can improve network security protection capabilities, reduce security risks, and ensure the reliable operation of the industrial Internet.

[0003] Although there are many technologies for processing multi-source heterogeneous alarm logs, existing solutions still face multiple challenges. Many existing technologies rely too much on template structures, ignore the semantic information and time information of logs, and are difficult to deeply explore the deep-level features behind logs, which leads to the omission and misjudgment of many potential threats. Summary of the invention

[0004] The embodiment of the present invention provides a method, device, electronic device and storage medium for detecting anomalies of multi-source heterogeneous alarm logs. By implementing the present invention, the expressiveness of log features can be enhanced and the accuracy of log anomaly detection can be improved.

[0005] An embodiment of the present invention provides a method for detecting anomalies of multi-source heterogeneous alarm logs, including:

[0006] Acquire multi-source heterogeneous alarm logs of a time period to be detected and time interval data between adjacent logs of the multi-source heterogeneous alarm logs;

[0007] Parsing the multi-source heterogeneous alarm logs to generate a log template;

[0008] Input the log template into the semantic model so that the semantic model generates a log semantic feature vector according to the log template; perform linear transformation on the time interval data to generate a log time feature vector; convert each word in the log template into a numerical vector to generate a log index feature vector; concatenate the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector;

[0009] The log fusion feature vector is input into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates a prediction result of the masked feature; if the proportion of inconsistency between the prediction result of the masked feature and the actual result exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the detection period.

[0010] Further, the parsing of the multi-source heterogeneous alarm logs to generate a log template includes:

[0011] Based on the regularized expression, the multi-source heterogeneous alarm log is desensitized and cleaned to generate a pre-processed multi-source heterogeneous alarm log;

[0012] Segment the preprocessed multi-source heterogeneous alarm logs to generate several word lists;

[0013] According to the word list, a parse tree is constructed to generate a parse tree of multi-source heterogeneous alarm logs;

[0014] Generate a log template based on the leaf nodes in the multi-source heterogeneous alarm log parsing tree.

[0015] Furthermore, the training of the semantic model includes:

[0016] Acquire a log data set, parse the log data set, and generate a first training log template;

[0017] Randomly dividing the first training log template into a plurality of batches of first training sample sets according to a preset number;

[0018] The first training sample sets of each batch are sequentially input into the semantic model, and the semantic model is trained until a preset number of training times is reached; wherein, when the semantic model receives each batch of the first training sample sets, it outputs the log semantic feature vector of each first training log template in the first training sample set; the similarity between the log semantic feature vectors of each first training log template is calculated to generate a number of similarity results; based on the similarity results, a first loss function value is calculated by a first loss function; and the semantic model is updated by an optimizer according to the first loss function value.

[0019] Furthermore, the linear transformation of the time interval data to generate a log time feature vector includes:

[0020] The time interval data is linearly transformed according to the following formula to generate a time vector:

[0021] k=softmax(Δ t W+b)

[0022] Where k is the time vector; Δ t is the time interval data; W is the weight matrix; b is the random deviation vector;

[0023] Calculate the product of the time vector and the preset weighted identity matrix to generate the log time feature vector.

[0024] Furthermore, converting each word in the log template into a numerical vector to generate a log index feature vector includes:

[0025] According to the words in the log template, a dictionary tree is constructed; wherein a node of the dictionary tree represents a character in the word, and a path represents a character sequence of the word;

[0026] Through word embedding technology, each word in the dictionary tree is converted into a numerical vector to generate the corresponding numerical vector result;

[0027] Storing the numerical vector result in a corresponding node of the dictionary tree;

[0028] According to the order of words in the log template, the numerical vector corresponding to each word is found from the dictionary tree, and the words are arranged in the order of the words in the log template to generate a log index feature vector.

[0029] Furthermore, the training of the anomaly detection model includes:

[0030] Obtaining a log fusion feature vector data set, and randomly dividing the log fusion feature training vectors in the log fusion feature vector data set into a plurality of batches of second training sample sets according to a preset number;

[0031] The second training sample sets of each batch are sequentially input into the anomaly detection model, and the anomaly detection model is trained until a preset number of training times is reached; wherein, when the anomaly detection model receives each batch of the second training sample sets, it randomly masks several features of each log fusion feature training vector in the second training sample set, and generates a prediction result of the masked features of each log fusion feature training vector; according to the prediction results and the corresponding masked features, a second loss function value is calculated by a second loss function; and the anomaly detection model is updated by an optimizer according to the second loss function value.

[0032] Furthermore, the obtaining of the log fusion feature vector data set includes:

[0033] Parsing the log data set to generate a second training log template;

[0034] Inputting the second training log template into a semantic model so that the semantic model generates a training log semantic feature vector according to the second training log template;

[0035] Extracting abnormal log time interval data between adjacent abnormal logs in the log data set;

[0036] Performing a linear transformation on the abnormal log time interval data to generate a training log time feature vector;

[0037] Convert each word in the second training log template into a numerical vector to generate a training log index feature vector;

[0038] The training log semantic feature vector, the training log time feature vector and the training log index feature vector are concatenated to generate a log fusion feature vector data set.

[0039] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0040] An embodiment of the present invention provides a multi-source heterogeneous alarm log anomaly detection device, comprising: an alarm log data acquisition module, a log template generation module, a log fusion feature vector generation module and an anomaly detection module;

[0041] The alarm log data acquisition module is used to acquire multi-source heterogeneous alarm logs of a time period to be detected and time interval data between adjacent logs of the multi-source heterogeneous alarm logs;

[0042] The log template generation module is used to parse the multi-source heterogeneous alarm logs and generate log templates;

[0043] The log fusion feature vector generation module is used to input the log template into the semantic model so that the semantic model generates a log semantic feature vector according to the log template; perform linear transformation on the time interval data to generate a log time feature vector; convert each word in the log template into a numerical vector to generate a log index feature vector; splice the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector;

[0044] The anomaly detection module is used to input the log fusion feature vector into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results of the masked features; wherein the anomaly detection model performs unsupervised training by using a masked prediction method through a log fusion feature vector data set; the log fusion feature vector data set is made of a log semantic feature vector, a log time feature vector and a log index feature vector extracted from the log data set; if the proportion of inconsistency between the prediction result of the masked feature and the actual one exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the detection period.

[0045] Based on the above method item embodiments, the present invention provides corresponding electronic device item embodiments.

[0046] An embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multi-source heterogeneous alarm log anomaly detection method described in any one of the above method embodiments can be implemented.

[0047] Based on the above method item embodiments, the present invention provides a corresponding storage medium item embodiment.

[0048] An embodiment of the present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the multi-source heterogeneous alarm log anomaly detection method described in any one of the above method embodiments can be implemented.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The embodiment of the present invention provides a multi-source heterogeneous alarm log anomaly detection method, device, electronic device and storage medium. The method obtains the alarm log and adjacent log time interval data of the time period to be detected; parses the log data to extract the common structure and generates a log template; inputs the log template into a semantic model to generate a log semantic feature vector; performs a linear transformation on the time interval data to generate a log time feature vector; converts each word in the log template into a numerical vector to generate a log index feature vector; then, concatenates the semantic feature, time feature and index feature vector to generate a log fusion feature vector; inputs the log fusion feature vector into an anomaly detection model so that the anomaly detection model randomly masks some features and predicts the masked elements to perform anomaly detection; if the proportion of inconsistency between the predicted result and the actual result exceeds a preset threshold, it is determined that an abnormal alarm exists.

[0051] The present invention parses multi-source heterogeneous alarm logs and converts the parsed log template into a semantic feature vector by using the powerful semantic expression ability of the semantic model. In addition, we also merge the time feature and index feature vector of the log with the semantic feature vector. This effectively solves the problem of over-reliance on template structure and neglect of log semantic information and time information in the prior art, thereby deeply mining the deep-level features behind the log, significantly enhancing the expressiveness of log features, and improving the accuracy of log anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a multi-source heterogeneous alarm log anomaly detection method provided by an embodiment of the present invention.

[0053] Figure 2 It is a structural diagram of a multi-source heterogeneous alarm log anomaly detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting anomalies of multi-source heterogeneous alarm logs, which at least includes the following steps:

[0056] Step S1: Acquire multi-source heterogeneous alarm logs in a time period to be detected and time interval data between adjacent logs of the multi-source heterogeneous alarm logs.

[0057] Specifically, to obtain the multi-source heterogeneous alarm logs and the time interval data between adjacent logs in the period to be detected, it is first necessary to collect alarm information from log systems of different sources, including but not limited to equipment status, fault alarms, operation records, etc. Subsequently, the timestamp information of adjacent alarm logs is extracted, and the time interval between adjacent log entries is calculated as a time feature to provide the necessary time series data for subsequent feature extraction and analysis. These log and time interval data will provide basic support for subsequent log parsing, feature extraction, anomaly detection and other steps.

[0058] Step S2: parse the multi-source heterogeneous alarm logs to generate a log template.

[0059] In a preferred embodiment, the parsing of the multi-source heterogeneous alarm logs to generate a log template includes:

[0060] Based on the regularized expression, the multi-source heterogeneous alarm log is desensitized and cleaned to generate a pre-processed multi-source heterogeneous alarm log;

[0061] Segment the preprocessed multi-source heterogeneous alarm logs to generate several word lists;

[0062] According to the word list, a parse tree is constructed to generate a parse tree of multi-source heterogeneous alarm logs;

[0063] Generate a log template based on the leaf nodes in the multi-source heterogeneous alarm log parsing tree.

[0064] It should be noted here that the multi-source heterogeneous alarm logs are desensitized and cleaned based on regular expressions. Sensitive information (such as IP addresses, user information, device IDs, etc.) is first anonymized through regular expressions to ensure data privacy and security. The cleaned data is converted into a standardized format, redundant or irrelevant information is removed, and a pre-processed multi-source heterogeneous alarm log is generated, which lays the foundation for subsequent data analysis and feature extraction.

[0065] Next, the pre-processed multi-source heterogeneous alarm logs are segmented. Each log is split into words, symbols or numbers through the word segmentation tool to generate several word lists.

[0066] According to the generated word list, a parse tree is constructed. The parse tree construction process analyzes the dependency and grammatical structure between words, and gradually constructs a hierarchical tree structure to reflect the hierarchy and relationship of different elements in the log, thereby forming a multi-source heterogeneous alarm log parse tree.

[0067] Finally, according to the leaf nodes in the multi-source heterogeneous alarm log parsing tree, key information is extracted and log templates are generated based on these leaf nodes. The log template reflects the structured pattern of the log data and summarizes different alarm logs into a standardized format to facilitate subsequent analysis and processing.

[0068] Step S3, inputting the log template into the semantic model so that the semantic model generates a log semantic feature vector according to the log template; performing a linear transformation on the time interval data to generate a log time feature vector; converting each word in the log template into a numerical vector to generate a log index feature vector; concatenating the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector.

[0069] In a preferred embodiment, the linear transformation of the time interval data to generate a log time feature vector includes:

[0070] Perform a linear transformation on the time interval data according to the following formula to generate a time vector:

[0071] k = softmax(Δ t W + b)

[0072] where k is the time vector; Δ t is the time interval data; W is the weight matrix; b is the random bias vector;

[0073] Calculate the product of the time vector and a preset weighted identity matrix to generate a log time feature vector, that is:

[0074] T = kE

[0075] where T is the log time feature vector; E is the weighted identity matrix.

[0076] In one embodiment, the converting each word in the log template into a numerical vector to generate a log index feature vector includes:

[0077] Construct a trie tree according to the words in the log template; wherein, the nodes of the trie tree represent one character in the word, and the path represents the character sequence of the word;

[0078] Convert each word in the trie tree into a numerical vector through word embedding technology to generate a corresponding numerical vector result;

[0079] Store the numerical vector result in the corresponding node of the trie tree;

[0080] Find the numerical vector corresponding to each word from the trie tree according to the word order in the log template, and arrange them in the order of the words in the log template to generate a log index feature vector.

[0081] Specifically, construct a trie tree according to the words in the log template. A trie tree is a tree - like data structure, where each node represents one character in the word, and the path represents the character sequence of the word. For example, for the word "fault", the node of the trie tree is the character "故", and its next - level node is the character "障". In this way, the trie tree effectively organizes the character sequences of each word in the log template.

[0082] Next, use word embedding technology to convert each word in the dictionary tree into a corresponding numerical vector. Word embedding technology can map words to vectors in a high-dimensional space, and train the model so that words with similar semantics are close in the vector space. Through this technology, each word in the dictionary tree is converted into a numerical vector and a corresponding numerical vector result is generated. These vectors can capture the semantic features of words and provide valuable information for subsequent log analysis. The word embedding technology here can be Word2Vec, GloVe, ELMo, etc.

[0083] The numerical vector result of each word is stored in the corresponding node of the dictionary tree, so that the dictionary tree not only records the information of the character sequence, but also saves the semantic features of each word. This step ensures that each node in the dictionary tree contains information at the character level and the semantic level.

[0084] According to the order of words in the log template, the numerical vector corresponding to each word is searched from the dictionary tree. During the search process, the corresponding numerical vectors are extracted according to the order of words in the log template, and these vectors are arranged in order to generate the final log index feature vector. This feature vector combines the word order in the log template and the semantic information of each word, providing an efficient representation for subsequent log analysis tasks.

[0085] It should be noted here that the training of the semantic model includes:

[0086] Acquire a log data set, parse the log data set, and generate a first training log template;

[0087] Randomly dividing the first training log template into a plurality of batches of first training sample sets according to a preset number;

[0088] The first training sample sets of each batch are sequentially input into the semantic model, and the semantic model is trained until a preset number of training times is reached; wherein, when the semantic model receives each batch of the first training sample sets, it outputs the log semantic feature vector of each first training log template in the first training sample set; the similarity between the log semantic feature vectors of each first training log template is calculated to generate a number of similarity results; based on the similarity results, a first loss function value is calculated by a first loss function; and the semantic model is updated by an optimizer according to the first loss function value.

[0089] In actual operation, the log data set may include any one of the following or a combination thereof: HDFS, BGL, and Tbird. Among them, the HDFS data set comes from more than 200 Amazon EC2 nodes running Hadoop-based jobs, containing 11,175,629 original log messages, of which 284,818 are marked as abnormal. These log data are mainly used for log analysis and anomaly detection in big data environments, helping to identify potential problems in distributed computing environments. The BGL data set is collected from the BlueGene / L supercomputer system, containing 4,747,963 log messages, of which 348,469 are abnormal logs, mainly used to study log analysis and fault diagnosis in high-performance computing environments. The Tbird data set comes from the Thunderbird supercomputer system of Sandia National Laboratory, containing a large amount of log information, involving system operation status and abnormal records, and is used for anomaly detection and performance optimization of supercomputer systems. In the present invention, we selected the first 20,000,000 log messages to form a data set, of which 758,562 are abnormal. These three datasets provide valuable data support for log analysis, anomaly detection, fault diagnosis, and system performance optimization, and are widely used in the field of high-performance computing.

[0090] In addition, the first loss function may be a cross entropy function; and the optimizer may be an Admn optimizer.

[0091] Step S4: input the log fusion feature vector into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates a prediction result of the masked feature; if the proportion of inconsistency between the prediction result of the masked feature and the actual result exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the detection period.

[0092] Specifically, the log fusion feature vector is input into the anomaly detection model so that the model performs anomaly detection based on the fusion feature vector. In this process, the anomaly detection model will randomly mask several features in the log fusion feature vector, simulate the situation of missing data, and generate prediction results for the masked features based on the remaining features. By comparing with the actual original feature value, if the proportion of inconsistency between the predicted result of the masked feature and the actual value exceeds the preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the period. This method can effectively capture abnormal patterns and help identify potential system failures or security threats through the masking prediction and error threshold judgment mechanism, thereby improving the accuracy and reliability of anomaly detection. The preset threshold here can be flexibly adjusted according to the actual situation, and the recommended value of 30% can be selected here.

[0093] In a preferred embodiment, the training of the anomaly detection model includes:

[0094] Obtaining a log fusion feature vector data set, and randomly dividing the log fusion feature training vectors in the log fusion feature vector data set into a plurality of batches of second training sample sets according to a preset number;

[0095] The second training sample sets of each batch are sequentially input into the anomaly detection model, and the anomaly detection model is trained until a preset number of training times is reached; wherein, when the anomaly detection model receives each batch of the second training sample sets, it randomly masks several features of each log fusion feature training vector in the second training sample set, and generates a prediction result of the masked features of each log fusion feature training vector; according to the prediction results and the corresponding masked features, a second loss function value is calculated by a second loss function; and the anomaly detection model is updated by an optimizer according to the second loss function value.

[0096] The second loss function here is specifically:

[0097] L loss =w T L T +w F L F +w I L I

[0098] Among them, L loss is the second loss function; w T is the weight coefficient of the training log time feature vector; w F is the weight coefficient of the semantic feature vector of the training log; w I is the weight coefficient of the training log index feature vector; L T is the loss of training log-time feature vector; L F is the loss of training log semantic feature vector; L I The loss of the feature vector for training log indices;

[0099] The loss of the training log time feature vector, the loss of the training log semantic feature vector, and the loss of the training log index feature vector are calculated by the following formula:

[0100]

[0101] Where N is the number of samples in a batch of the second training sample set; M T is the identifier of the masked feature in the training log time feature vector; M F is the identifier of the masked feature in the semantic feature vector of the training log; M I The identifier of the masked feature in the training log index feature vector; is the actual value of the i-th masked feature in the j-th training log-time feature vector; is the actual value of the i-th masked feature in the j-th training log semantic feature vector; The actual value of the i-th masked feature in the j-th training log index feature vector; is the predicted value of the i-th masked feature in the j-th training log time feature vector; For the j The first i The predicted value of the masked feature; For the j The predicted value of the i-th masked feature in the training log index feature vector.

[0102] In an optional embodiment, the masking ratios of the training log time feature vector, the training log index feature vector, and the training log semantic feature vector can be freely set. Here, the masking ratio of the training log time feature vector can be set to 20%, the masking ratio of the training log index feature vector can be set to 10%, and the masking ratio of the training log semantic feature vector can be set to 40%.

[0103] It should be noted that the step of obtaining the log fusion feature vector data set includes:

[0104] Parsing the log data set to generate a second training log template;

[0105] Inputting the second training log template into a semantic model so that the semantic model generates a training log semantic feature vector according to the second training log template;

[0106] Extracting abnormal log time interval data between adjacent abnormal logs in the log data set;

[0107] Performing a linear transformation on the abnormal log time interval data to generate a training log time feature vector;

[0108] Convert each word in the second training log template into a numerical vector to generate a training log index feature vector;

[0109] The training log semantic feature vector, the training log time feature vector and the training log index feature vector are concatenated to generate a log fusion feature vector data set.

[0110] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0111] like Figure 2As shown, an embodiment of the present invention provides a multi-source heterogeneous alarm log anomaly detection device, including: an alarm log data acquisition module, a log template generation module, a log fusion feature vector generation module and an anomaly detection module;

[0112] The alarm log data acquisition module is used to acquire multi-source heterogeneous alarm logs of a time period to be detected and time interval data between adjacent logs of the multi-source heterogeneous alarm logs;

[0113] The log template generation module is used to parse the multi-source heterogeneous alarm logs and generate log templates;

[0114] The log fusion feature vector generation module is used to input the log template into the semantic model so that the semantic model generates a log semantic feature vector according to the log template; perform linear transformation on the time interval data to generate a log time feature vector; convert each word in the log template into a numerical vector to generate a log index feature vector; splice the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector;

[0115] The anomaly detection module is used to input the log fusion feature vector into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results of the masked features; wherein the anomaly detection model performs unsupervised training by using a masked prediction method through a log fusion feature vector data set; the log fusion feature vector data set is made of a log semantic feature vector, a log time feature vector and a log index feature vector extracted from the log data set; if the proportion of inconsistency between the prediction result of the masked feature and the actual one exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the detection period.

[0116] It should be noted that the embodiment of the device described above corresponds to the above-mentioned embodiment of the present invention, and it can implement any of the above-mentioned multi-source heterogeneous alarm log anomaly detection methods of the present invention. In addition, the above-mentioned embodiment of the device is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0117] Based on the above method embodiment of the present invention, a corresponding electronic device embodiment is provided.

[0118] An embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the multi-source heterogeneous alarm log anomaly detection method described in any one of the present invention is implemented, or, when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are implemented.

[0119] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0120] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0121] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0122] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0123] Based on the above method embodiment, the present invention provides a storage medium embodiment;

[0124] Another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute any of the above-mentioned multi-source heterogeneous alarm log anomaly detection methods of the present invention.

[0125] The storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0126] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0127] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in multi-source heterogeneous alarm logs, characterized in that: include: Acquire multi-source heterogeneous alarm logs of a time period to be detected and time interval data between adjacent logs of the multi-source heterogeneous alarm logs; Parsing the multi-source heterogeneous alarm logs to generate a log template; Input the log template into the semantic model so that the semantic model generates a log semantic feature vector according to the log template; perform linear transformation on the time interval data to generate a log time feature vector; convert each word in the log template into a numerical vector to generate a log index feature vector; concatenate the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector; The log fusion feature vector is input into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates a prediction result of the masked feature; if the proportion of inconsistency between the prediction result of the masked feature and the actual result exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the detection period.

2. The multi-source heterogeneous alarm log anomaly detection method according to claim 1, characterized in that: The step of parsing the multi-source heterogeneous alarm logs and generating a log template includes: Based on the regularized expression, the multi-source heterogeneous alarm log is desensitized and cleaned to generate a pre-processed multi-source heterogeneous alarm log; Segment the preprocessed multi-source heterogeneous alarm logs to generate several word lists; According to the word list, a parse tree is constructed to generate a parse tree of multi-source heterogeneous alarm logs; Generate a log template based on the leaf nodes in the multi-source heterogeneous alarm log parsing tree.

3. The multi-source heterogeneous alarm log anomaly detection method according to claim 2, characterized in that: The training of the semantic model includes: Obtaining a log data set, parsing the log data set, and generating a first training log template; Randomly dividing the first training log template into a plurality of batches of first training sample sets according to a preset number; The first training sample sets of each batch are sequentially input into the semantic model, and the semantic model is trained until a preset number of training times is reached; wherein, when the semantic model receives each batch of the first training sample sets, it outputs the log semantic feature vector of each first training log template in the first training sample set; the similarity between the log semantic feature vectors of each first training log template is calculated to generate a number of similarity results; based on the similarity results, a first loss function value is calculated by a first loss function; and the semantic model is updated by an optimizer according to the first loss function value.

4. The multi-source heterogeneous alarm log anomaly detection method according to claim 3, characterized in that: The linear transformation of the time interval data to generate a log time feature vector includes: The time interval data is linearly transformed according to the following formula to generate a time vector: k=softmax(Δ t W+b) Where k is the time vector; Δ t is the time interval data; W is the weight matrix; b is the random deviation vector; Calculate the product of the time vector and the preset weighted identity matrix to generate the log time feature vector.

5. The method for detecting anomalies of multi-source heterogeneous alarm logs according to claim 4, characterized in that: The step of converting each word in the log template into a numerical vector to generate a log index feature vector includes: According to the words in the log template, a dictionary tree is constructed; wherein a node of the dictionary tree represents a character in the word, and a path represents a character sequence of the word; Through word embedding technology, each word in the dictionary tree is converted into a numerical vector to generate the corresponding numerical vector result; Storing the numerical vector result in a corresponding node of the dictionary tree; According to the order of words in the log template, the numerical vector corresponding to each word is found from the dictionary tree, and the words are arranged in the order of the words in the log template to generate a log index feature vector.

6. The method for detecting anomalies of multi-source heterogeneous alarm logs according to claim 5, characterized in that: The training of the anomaly detection model includes: Obtaining a log fusion feature vector data set, and randomly dividing the log fusion feature training vectors in the log fusion feature vector data set into a plurality of batches of second training sample sets according to a preset number; The second training sample sets of each batch are sequentially input into the anomaly detection model, and the anomaly detection model is trained until a preset number of training times is reached; wherein, when the anomaly detection model receives each batch of the second training sample sets, it randomly masks several features of each log fusion feature training vector in the second training sample set, and generates a prediction result of the masked features of each log fusion feature training vector; according to the prediction results and the corresponding masked features, a second loss function value is calculated by a second loss function; and the anomaly detection model is updated by an optimizer according to the second loss function value.

7. The method for detecting anomalies of multi-source heterogeneous alarm logs according to claim 6, characterized in that: The step of obtaining a log fusion feature vector dataset includes: Parsing the log data set to generate a second training log template; Inputting the second training log template into a semantic model so that the semantic model generates a training log semantic feature vector according to the second training log template; Extracting abnormal log time interval data between adjacent abnormal logs in the log data set; Performing a linear transformation on the abnormal log time interval data to generate a training log time feature vector; Convert each word in the second training log template into a numerical vector to generate a training log index feature vector; The training log semantic feature vector, the training log time feature vector and the training log index feature vector are concatenated to generate a log fusion feature vector data set.

8. A multi-source heterogeneous alarm log anomaly detection device, characterized in that: include: Alarm log data acquisition module, log template generation module, log fusion feature vector generation module and anomaly detection module; The alarm log data acquisition module is used to acquire multi-source heterogeneous alarm logs of a time period to be detected and time interval data between adjacent logs of the multi-source heterogeneous alarm logs; The log template generation module is used to parse the multi-source heterogeneous alarm logs and generate log templates; The log fusion feature vector generation module is used to input the log template into the semantic model so that the semantic model generates a log semantic feature vector according to the log template; perform linear transformation on the time interval data to generate a log time feature vector; convert each word in the log template into a numerical vector to generate a log index feature vector; and concatenate the log semantic feature vector, the log time feature vector and the log index feature vector to generate a log fusion feature vector; The anomaly detection module is used to input the log fusion feature vector into the anomaly detection model so that the anomaly detection model randomly masks several features in the log fusion feature vector and generates prediction results of the masked features; wherein the anomaly detection model performs unsupervised training by using a masked prediction method through a log fusion feature vector data set; the log fusion feature vector data set is made of a log semantic feature vector, a log time feature vector and a log index feature vector extracted from the log data set; if the proportion of inconsistency between the prediction result of the masked feature and the actual one exceeds a preset threshold, it is determined that there is an abnormal alarm in the multi-source heterogeneous alarm log of the detection period.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multi-source heterogeneous alarm log anomaly detection method described in any one of claims 1 to 7 can be implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the multi-source heterogeneous alarm log anomaly detection method as described in any one of claims 1 to 7.

Images