Operation and maintenance log anomaly detection method and device

Abstract

The invention provides an operation and maintenance log anomaly detection method and device. The method comprises the steps of obtaining a to-be-detected log; inputting the to-be-detected log into a pre-trained bidirectional LSTM model for semantic analysis to obtain target log semantic information; matching the target log semantic information with each pre-constructed security rule; and if a security rule corresponding to the target log semantic information is matched, determining that the to-be-detected log is an abnormal log. In the scheme, deep understanding of log semantics is realized while the bidirectional LSTM model adapts to the change of the unstructured log in advance, and the security rule is constructed based on the obtained log semantic information to perform log anomaly detection by using the security rule, so that the anomaly detection accuracy and generalization ability are improved by fully utilizing the context information, and the log anomaly detection efficiency is improved. And the maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for detecting anomalies in operation and maintenance logs. Background Art

[0002] As modern software systems become increasingly complex, the importance of operational logging becomes increasingly prominent. By monitoring logs, developers can understand software status, identify anomalies, locate errors, and resolve system issues. However, as log volumes increase in large systems, developers struggle to read every line of log files. Therefore, efficient and accurate automated analysis of operational logs has become an urgent need.

[0003] To improve the effectiveness of operation and maintenance log analysis, many researchers have used statistical methods to automate log analysis. For example, SLCT and LFA are used to automate the analysis of structured logs. Both methods analyze logs by counting word and phrase frequencies, making them suitable for scenarios with relatively fixed log formats and simple anomaly patterns. However, these methods have the following drawbacks: 1. Insufficient understanding of log semantics: They can only identify frequently occurring patterns, but fail to deeply understand the log's semantic information. For example, words like "status" or "500" may not appear anomalous when appearing alone, but "status 500" as a whole phrase has an anomalous meaning. Simple word frequency statistics cannot distinguish these contextual semantic differences. 2. Poor adaptability to unstructured logs: Logs are typically unstructured or semi-structured, and their formats vary widely. Statistical methods are poorly adaptable to these changes. Once the log format changes, the existing statistical rules may become invalid, requiring manual pattern re-definition and training, resulting in high maintenance costs.

[0004] In summary, the existing technology has insufficient understanding of log semantics and poor adaptability to unstructured logs, resulting in poor anomaly detection accuracy and generalization ability, and high maintenance costs. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a method and apparatus for detecting anomalies in operation and maintenance logs, so as to fully utilize context information to improve anomaly detection accuracy and generalization capability, and reduce maintenance costs.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of an embodiment of the present invention discloses a method for detecting anomalies in an operation and maintenance log, the method comprising:

[0008] Get the log to be tested;

[0009] Input the log to be detected into a pre-trained bidirectional LSTM model for semantic parsing to obtain the semantic information of the target log;

[0010] Matching the target log semantic information with each pre-built security rule; the security rule is constructed based on the abnormal log semantic information of the abnormal log sample; the abnormal log semantic information is obtained by pre-inputting the abnormal log sample into the bidirectional LSTM model for semantic parsing; the abnormal log sample includes at least: an unstructured abnormal log;

[0011] If the security rule corresponding to the target log semantic information is matched, it is determined that the log to be detected is an abnormal log.

[0012] Preferably, the process of constructing each of the security rules includes:

[0013] Input the abnormal log samples into the pre-trained bidirectional LSTM model for semantic analysis to obtain the abnormal log semantic information;

[0014] If the exception log semantic information contains concepts, a security rule is constructed with the concepts as matching conditions; if the exception log semantic information contains instances, a security rule is constructed with the instances as matching conditions; if the exception log semantic information contains concept-instance pairs consisting of concepts and instances, a security rule is constructed with the concept-instance pairs as matching conditions; the concepts include readable words; and the instances include character string groups.

[0015] Preferably, matching the target log semantic information with various pre-built security rules includes:

[0016] If the target log semantic information includes a target concept, and the target concept is consistent with a concept in a security rule with a concept as a matching condition, then it is determined that the target log semantic information matches the security rule with a concept as a matching condition;

[0017] If the target log semantic information includes a target instance, and the target instance is consistent with an instance in a security rule with an instance as a matching condition, then it is determined that the target log semantic information matches the security rule with an instance as a matching condition;

[0018] If the target log semantic information contains a target concept instance pair, and the concept instance pair is consistent with the concept instance pair in the security rule with the concept instance pair as the matching condition, it is determined that the target log semantic information matches the security rule with the concept instance pair as the matching condition.

[0019] Preferably, the training process of the bidirectional LSTM model includes:

[0020] Build a bidirectional LSTM model to be trained;

[0021] Get multiple unstructured logs;

[0022] For each of the unstructured logs, a word segmenter is used to segment the logs to obtain log words of multiple word types;

[0023] Calculating the ratio of the number of the log words corresponding to each word type to the total number of the log words in the unstructured log to obtain log word-level attributes;

[0024] Eliminating unstructured logs that do not meet preset requirements from each of the unstructured logs based on the log word-level attributes, and constructing a training data set based on the remaining unstructured logs;

[0025] For each log word in each unstructured log in the training data set, construct a word-level feature matrix corresponding to the log word;

[0026] For each readable sentence in each of the unstructured logs in the training data set, the word-level feature matrices corresponding to the log words in the readable sentence are sequentially arranged to form a forward sentence-level feature matrix, and the word-level feature matrices corresponding to the log words in the readable sentence are reversely arranged to form a reverse sentence-level feature matrix;

[0027] The forward sentence-level feature matrix is used as the forward input of the bidirectional LSTM model to be trained, and the reverse sentence-level feature matrix is used as the reverse input of the bidirectional LSTM model to be trained. The bidirectional LSTM model to be trained is trained to obtain a trained bidirectional LSTM model.

[0028] Preferably, for each log word in each unstructured log in the training data set, constructing a word-level feature matrix corresponding to the log word includes:

[0029] For each log word in each unstructured log in the training data set, identifying the word type of the log word and calculating the hash value of the log word; the word type includes: a readable word type and a meaningless string type;

[0030] Based on the word type and the hash value, a word-level feature matrix corresponding to the log word is constructed.

[0031] A second aspect of an embodiment of the present invention discloses a device for detecting anomalies in an operation and maintenance log, the device comprising:

[0032] An acquisition unit, used to acquire the log to be detected;

[0033] A semantic analysis unit is used to input the log to be detected into a pre-trained bidirectional LSTM model for semantic analysis to obtain semantic information of the target log;

[0034] A matching unit is configured to match the target log semantic information with each pre-constructed security rule; the security rule is constructed based on the abnormal log semantic information of the abnormal log sample; the abnormal log semantic information is obtained by pre-inputting the abnormal log sample into the bidirectional LSTM model for semantic parsing; the abnormal log sample includes at least: an unstructured abnormal log;

[0035] The determining unit is configured to determine that the log to be detected is an abnormal log if the security rule corresponding to the target log semantic information is matched.

[0036] Preferably, the device further comprises:

[0037] A rule construction unit is used to input an exception log sample into a pre-trained bidirectional LSTM model for semantic analysis to obtain exception log semantic information; if the exception log semantic information contains a concept, a security rule is constructed with the concept as a matching condition; if the exception log semantic information contains an instance, a security rule is constructed with the instance as a matching condition; if the exception log semantic information contains a concept-instance pair consisting of a concept and an instance, a security rule is constructed with the concept-instance pair as a matching condition; the concept includes a readable vocabulary; the instance includes a string group.

[0038] Preferably, the matching unit is specifically used to:

[0039] If the target log semantic information includes a target concept, and the target concept is consistent with a concept in a security rule with a concept as a matching condition, then it is determined that the target log semantic information matches the security rule with a concept as a matching condition;

[0040] If the target log semantic information includes a target instance, and the target instance is consistent with an instance in a security rule with an instance as a matching condition, then it is determined that the target log semantic information matches the security rule with an instance as a matching condition;

[0041] If the target log semantic information contains a target concept instance pair, and the concept instance pair is consistent with the concept instance pair in the security rule with the concept instance pair as the matching condition, it is determined that the target log semantic information matches the security rule with the concept instance pair as the matching condition.

[0042] Preferably, the device further comprises:

[0043] A training unit is used to build a bidirectional LSTM model to be trained; obtain multiple unstructured logs; for each of the unstructured logs, use a word segmenter to perform word segmentation to obtain log words of multiple word types; calculate the ratio of the number of the log words corresponding to each word type to the total number of the log words in the unstructured log to obtain log word-level attributes; based on the log word-level attributes, remove the unstructured logs that do not meet the preset requirements from each of the unstructured logs, and build a training data set based on the remaining unstructured logs; for each log word in each of the unstructured logs in the training data set, build the log word The method comprises the following steps: for each readable sentence in each unstructured log in the training data set, arranging the word-level feature matrices corresponding to the log words in the readable sentence in sequence to form a forward sentence-level feature matrix, and arranging the word-level feature matrices corresponding to the log words in the readable sentence in reverse order to form a reverse sentence-level feature matrix; using the forward sentence-level feature matrix as the forward input of the bidirectional LSTM model to be trained, and using the reverse sentence-level feature matrix as the reverse input of the bidirectional LSTM model to be trained, and training the bidirectional LSTM model to obtain a trained bidirectional LSTM model.

[0044] Preferably, the training unit for constructing a word-level feature matrix corresponding to each log word in each unstructured log in the training data set is specifically configured to:

[0045] For each log word in each unstructured log in the training data set, identifying the word type of the log word and calculating the hash value of the log word; the word type includes: a readable word type and a meaningless string type;

[0046] Based on the word type and the hash value, a word-level feature matrix corresponding to the log word is constructed.

[0047] Based on the above-mentioned embodiment of the present invention, a method and device for detecting anomalies in operation and maintenance logs are provided. The method comprises obtaining a log to be detected; inputting the log to be detected into a pre-trained bidirectional LSTM model for semantic parsing to obtain target log semantic information; matching the target log semantic information with various pre-constructed security rules; the security rules are constructed based on the abnormal log semantic information of an abnormal log sample; the abnormal log semantic information is obtained by pre-inputting the abnormal log sample into the bidirectional LSTM model for semantic parsing; the abnormal log sample includes at least an unstructured abnormal log; if the security rule corresponding to the target log semantic information is matched, the log to be detected is determined to be an abnormal log. In this solution, a bidirectional LSTM model is used to adapt to changes in unstructured logs in advance while achieving a deep understanding of log semantics. Security rules are constructed based on the obtained log semantic information to use the security rules for log anomaly detection, thereby fully utilizing contextual information to improve anomaly detection accuracy and generalization capabilities, and reduce maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 This is a flow chart of a method for detecting anomalies in operation and maintenance logs disclosed in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of constructing a word-level feature matrix disclosed in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a bidirectional LSTM model training disclosed in an embodiment of the present invention;

[0052] Figure 4 This is a structural diagram of an anomaly detection device for operation and maintenance logs disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0055] As can be seen from the background technology, the existing technology has insufficient understanding of log semantics and poor adaptability to unstructured logs, resulting in poor anomaly detection accuracy and generalization ability, and high maintenance costs.

[0056] Therefore, an embodiment of the present invention discloses a method and device for anomaly detection in operation and maintenance logs. In this solution, a bidirectional LSTM model is used to adapt to changes in unstructured logs in advance while achieving a deep understanding of log semantics. Security rules are constructed based on the obtained log semantic information to use the security rules for log anomaly detection, thereby achieving the purpose of fully utilizing contextual information to improve anomaly detection accuracy and generalization ability, and reducing maintenance costs.

[0057] like Figure 1 FIG. 1 is a flowchart of a method for detecting anomalies in operation and maintenance logs disclosed in an embodiment of the present invention, comprising the following steps:

[0058] Step S101: Obtain the log to be detected.

[0059] In step S101 , the log to be detected may be an unstructured operation and maintenance log generated during the operation of a modern software system.

[0060] Step S102: Input the log to be detected into the pre-trained bidirectional LSTM model for semantic analysis to obtain the semantic information of the target log.

[0061] In step S102, the log to be detected needs to be processed accordingly to obtain the forward input and reverse input of the bidirectional LSTM model. The specific processing process is consistent with the process of processing the training abnormal log samples in the bidirectional LSTM model training process described below.

[0062] Specifically, the bidirectional LSTM model training process includes the following steps:

[0063] Step S201: Construct a bidirectional LSTM model to be trained.

[0064] Step S202: Acquire multiple unstructured logs.

[0065] Step S203: For each unstructured log, perform word segmentation using a word segmenter to obtain log words of multiple word types.

[0066] Exemplarily, word types include: system words, user words, irrelevant words, and the like.

[0067] Among them, system words refer to words related to system instructions, such as lock, stop, etc.; user words refer to words related to user returns, such as return, etc.; words that are neither system words nor user words are irrelevant words.

[0068] Step S204: Calculate the ratio of the number of log words corresponding to each word type to the total number of log words in the unstructured log to obtain log word-level attributes.

[0069] Specifically, the proportion of system words, user words, and irrelevant words in the log to the total number of all log words is calculated as the log word-level attribute.

[0070] The log word-level attributes can be expressed as [p1, p2, p3], where p1 represents the ratio of the number of system words to the total number of log words, p2 represents the ratio of the number of user words to the total number of log words, and p3 represents the ratio of the number of irrelevant words to the total number of log words.

[0071] Step S205: based on the log word-level attributes, unstructured logs that do not meet the preset requirements are eliminated from the unstructured logs, and a training data set is constructed based on the remaining unstructured logs.

[0072] In step S205 , the unstructured logs that do not meet the preset requirements refer to low-quality unstructured logs that cannot meet the normal training requirements of the bidirectional LSTM model and may be noise or special case logs.

[0073] In the specific implementation process of step S205 , the log word-level attributes of the log that meets the preset requirements are [a1, a2, a3], where a1=0.3; a2=0.2; and a3=0.5.

[0074] Calculation formula based on log word-level attributes [p1, p2, p3] and floating values:

[0075] [P]n=(p1-a1) 2 + (p2-a2) 2 + (p3-a3) 2[ / P]

[0076] The floating value n is calculated. If n*100%>e, it means that the corresponding unstructured log does not meet the preset requirements and needs to be eliminated.

[0077] Step S206: for each log word in each unstructured log in the training data set, construct a word-level feature matrix corresponding to the log word.

[0078] In the specific implementation process of step S206, for each log word in each unstructured log in the training data set, the word type of the log word is identified and the hash value of the log word is calculated.

[0079] The word types include: readable word type and meaningless string type.

[0080] Then, based on the word type and hash value, a word-level feature matrix corresponding to the log words is constructed.

[0081] like Figure 2 , which is a schematic diagram of constructing a word-level feature matrix disclosed in an embodiment of the present invention.

[0082] It should be noted that the word-level feature matrix for log words is [type, hash]. Type represents the word type. If the log word is a readable word, type is 0; if the log word is a meaningless string, type is 1. Hash represents the hash value of the log word.

[0083] Figure 2 The leftmost part of the image contains log word examples, such as "Listing" and "949e1227." The middle part uses feature extraction to divide log words into readable words (type=0) and meaningless strings (type=1). The corresponding hash values are calculated: 0x2e31fd and 0xf3a511, respectively. The final word-level feature matrices [0, 0x2e31fd] and [1, 0xf3a511] are constructed on the far right.

[0084] Step S207: For each readable sentence in each unstructured log in the training data set, the word-level feature matrices corresponding to the log words in the readable sentence are arranged in sequence to form a forward sentence-level feature matrix, and the word-level feature matrices corresponding to the log words in the readable sentence are arranged in reverse order to form a reverse sentence-level feature matrix.

[0085] In step S207 , a readable sentence refers to a natural language sentence or phrase fragment in the log that is directly understandable by development and operation and maintenance personnel and has clear semantics.

[0086] For example, “Listing instance in cell 949e1227” and “Returning 500 to user” are readable sentences.

[0087] In contrast, unreadable sentences are usually random strings, hash codes, memory addresses, process numbers, or other identifiers with no clear semantic meaning. For example: "0x7f3ad4cb" and "17893231".

[0088] In the specific implementation process of step S207, each readable sentence is taken as the input unit, the forward input is the forward sentence-level feature matrix formed by all the log words of the readable sentence in sequence, and the reverse input is the reverse sentence-level feature matrix formed by all the log words of the readable sentence in reverse sequence.

[0089] Taking the log statement "Listing instance in cell 949e1227" as an example, the forward and reverse inputs of the bidirectional LSTM model corresponding to this log statement are:

[0090] {matrix[Listing,instance in cell 949e1227],matrix[949e1227,cell,in,instance, Listing]}.

[0091] Here, matrix represents the method for generating the word-level feature matrix described in step S206.

[0092] Step S208: Using the forward sentence-level feature matrix as the forward input of the bidirectional LSTM model to be trained, and using the reverse sentence-level feature matrix as the reverse input of the bidirectional LSTM model to be trained, the bidirectional LSTM model to be trained is trained to obtain a trained bidirectional LSTM model.

[0093] like Figure 3 , which is a schematic diagram of a bidirectional LSTM model training disclosed in an embodiment of the present invention.

[0094] Figure 3 The left side shows multiple log statements used for training, such as: Listing instance in cell949e1227; Lock 949e1227 acquired by nova.context.get_cell; Returning 500 touser.

[0095] The input of the bidirectional LSTM model is a sentence-level feature matrix generated based on the order and reverse order of the log words in the log sentence:

[0096] Forward input: Log words arranged in order form a forward sentence-level feature matrix. For example: matrix[Listing,instance,in,cell,949e1227].

[0097] Reverse input: Log words arranged in reverse order form a reverse sentence-level feature matrix. For example: matrix[949e1227,cell,in,instance,Listing].

[0098] These sentence-level feature matrices are the input of the bidirectional LSTM model to capture the structural and semantic information of the logs.

[0099] The right side shows the internal logic of the bidirectional LSTM model during training. The bidirectional LSTM model is a common machine learning model and is not described here.

[0100] When the final model correctly outputs the semantic information of the unstructured log, the bidirectional LSTM model training is determined to be complete.

[0101] It is understandable that the target log semantic information obtained by using the log to be detected and the trained bidirectional LSTM model is used for downstream log analysis (Downstream Log Analysis), that is, steps S103 to S104.

[0102] Step S103: Match the target log semantic information with each pre-built security rule.

[0103] Among them, the security rules are constructed based on the abnormal log semantic information of the abnormal log samples; the abnormal log semantic information is obtained by pre-inputting the abnormal log samples into the bidirectional LSTM model for semantic analysis; the abnormal log samples at least include: unstructured abnormal logs.

[0104] Specifically, the construction process of each security rule includes:

[0105] Step S301: Input the abnormal log sample into the pre-trained bidirectional LSTM model for semantic analysis to obtain abnormal log semantic information.

[0106] Among them, the abnormal log sample is a collection of unstructured abnormal logs that are clearly marked as abnormal, and the abnormality of the unstructured abnormal log collection can be confirmed by manual marking or other anomaly detection methods.

[0107] In step S301, the abnormal log samples need to be processed accordingly to obtain the forward input and reverse input of the bidirectional LSTM model. The specific processing process is consistent with the process of processing the training abnormal log samples in the above-mentioned bidirectional LSTM model training process.

[0108] It should be noted that log semantic information is divided into three categories: concept class, instance class and concept-instance pair class.

[0109] Among them, the concept class is a set of readable words whose context is not connected to any string, such as Listing, instance; the instance class is a set of strings whose context is not connected to any readable words, such as 949e1227, 17893231; the concept-instance pair class is a group of readable words and strings that are connected to each other in the context, such as {status, 500}, {project, 5250C}.

[0110] Step S302: If the exception log semantic information contains concepts, a security rule is constructed with the concepts as matching conditions; if the exception log semantic information contains instances, a security rule is constructed with the instances as matching conditions; if the exception log semantic information contains concept-instance pairs consisting of concepts and instances, a security rule is constructed with the concept-instance pairs as matching conditions.

[0111] The concept-instance pair consists of a concept and an instance; the concept includes readable words; and the instance includes a string group.

[0112] Exemplarily, for the obtained concepts, instances and concept-instance pairs, security rules <concept, *>, <*, instance> and <concept, instance> are formed for subsequent security rule matching.

[0113] Common exception log semantic information<status,500> For example, based on the concept instance, security rules are formed for status and 500<status,500> .

[0114] It should be noted that, for example, for the concept instance pair status and 500, "status" or "500" appearing alone may not be abnormal, but "status 500" as a whole has abnormal meaning. Therefore, it is necessary to form a safety rule for status and 500 based on the concept instance.<status,500> .

[0115] Based on the above security rule construction process, the process of matching the target log semantic information with each security rule is as follows:

[0116] If the target log semantic information contains the target concept, and the target concept is consistent with the concept in the security rule with the concept as the matching condition, then it is determined that the target log semantic information matches the security rule with the concept as the matching condition;

[0117] If the target log semantic information includes the target instance, and the target instance is consistent with the instance in the security rule with the instance as the matching condition, then it is determined that the target log semantic information matches the security rule with the instance as the matching condition;

[0118] If the target log semantic information contains a target concept instance pair, and the concept instance pair is consistent with the concept instance pair in the security rule with the concept instance pair as the matching condition, then it is determined that the target log semantic information matches the security rule with the concept instance pair as the matching condition.

[0119] For example, when the log to be detected contains the log information "Returning 500 to user", the target log semantic information obtained can match the security rule <*, 500> and is determined to be an abnormal log; when the log information ""GET / v2.1 / 5250c / flavors""status:500" exists, the target log semantic information obtained can match the security rule<status,500> , which is determined to be abnormal log information.

[0120] It can be understood that if the semantic information of the target log to be detected can match any of the established security rules, it means that the log content is consistent with the known abnormal pattern and is determined to be an abnormal log;

[0121] If the target log semantic information of the log to be detected cannot match any existing security rules, it means that the log does not belong to a known abnormal pattern. However, this does not absolutely mean that it is a normal log. It only means that it does not match the currently known abnormality.

[0122] Step S104: If the security rule corresponding to the semantic information of the target log is matched, the log to be detected is determined to be an abnormal log.

[0123] In one embodiment, if no security rule corresponding to the target log semantic information is matched, it is determined that the log to be detected does not belong to a known abnormal situation.

[0124] Based on the above-mentioned embodiment of the present invention, a method for detecting anomalies in operation and maintenance logs is disclosed. In this solution, a bidirectional LSTM model is used to adapt to changes in unstructured logs while achieving a deep understanding of log semantics. Security rules are constructed based on the obtained log semantic information to detect log anomalies using the security rules, thereby fully utilizing contextual information to improve anomaly detection accuracy and generalization ability, and reduce maintenance costs.

[0125] Corresponding to the anomaly detection method of an operation and maintenance log disclosed in the above embodiment of the present invention, as Figure 4 , which is a structural diagram of an anomaly detection device for operation and maintenance logs disclosed in an embodiment of the present invention, includes: an acquisition unit 401 , a semantic analysis unit 402 , a matching unit 403 and a determination unit 404 .

[0126] The acquisition unit 401 is used to acquire the log to be detected.

[0127] The semantic analysis unit 402 is used to input the log to be detected into a pre-trained bidirectional LSTM model for semantic analysis to obtain the semantic information of the target log.

[0128] In one embodiment, the apparatus further comprises:

[0129] The training unit is used to build a bidirectional LSTM model to be trained; obtain multiple unstructured logs; for each unstructured log, use the word segmenter to segment the words to obtain log words of multiple word types; calculate the ratio of the number of log words corresponding to each word type to the total number of log words in the unstructured log to obtain log word-level attributes; based on the log word-level attributes, remove the unstructured logs that do not meet the preset requirements from each unstructured log, and build a training data set based on the remaining unstructured logs; for each log word in each unstructured log in the training data set, build a log word-level attribute. The word-level feature matrix corresponding to the words; for each readable sentence in each unstructured log in the training data set, the word-level feature matrices corresponding to the log words in the readable sentence are arranged in sequence to form a forward sentence-level feature matrix, and the word-level feature matrices corresponding to the log words in the readable sentence are arranged in reverse order to form a reverse sentence-level feature matrix; the forward sentence-level feature matrix is used as the forward input of the bidirectional LSTM model to be trained, and the reverse sentence-level feature matrix is used as the reverse input of the bidirectional LSTM model to be trained, and the bidirectional LSTM model to be trained is trained to obtain a trained bidirectional LSTM model.

[0130] In one embodiment, a training unit for constructing a word-level feature matrix corresponding to each log word in each unstructured log in a training dataset is specifically configured to:

[0131] For each log word in each unstructured log in the training dataset, identify the word type of the log word and calculate the hash value of the log word; the word type includes: readable word type and meaningless string type;

[0132] Based on the word type and hash value, a word-level feature matrix corresponding to the log words is constructed.

[0133] Matching unit 403 is used to match the target log semantic information with each pre-built security rule; the security rule is constructed based on the abnormal log semantic information of the abnormal log sample; the abnormal log semantic information is obtained by pre-inputting the abnormal log sample into a bidirectional LSTM model for semantic parsing; the abnormal log sample includes at least: unstructured abnormal log.

[0134] In one embodiment, the apparatus further comprises:

[0135] The rule construction unit is used to input the abnormal log samples into the pre-trained bidirectional LSTM model for semantic analysis to obtain the abnormal log semantic information; if the abnormal log semantic information contains concepts, the security rules are constructed with the concepts as matching conditions; if the abnormal log semantic information contains instances, the security rules are constructed with the instances as matching conditions; if the abnormal log semantic information contains concept-instance pairs consisting of concepts and instances, the security rules are constructed with the concept-instance pairs as matching conditions; concepts include readable words; instances include string groups.

[0136] In one embodiment, the matching unit 403 is specifically configured to:

[0137] If the target log semantic information contains the target concept, and the target concept is consistent with the concept in the security rule with the concept as the matching condition, then it is determined that the target log semantic information matches the security rule with the concept as the matching condition;

[0138] If the target log semantic information includes the target instance, and the target instance is consistent with the instance in the security rule with the instance as the matching condition, then it is determined that the target log semantic information matches the security rule with the instance as the matching condition;

[0139] If the target log semantic information contains a target concept instance pair, and the concept instance pair is consistent with the concept instance pair in the security rule with the concept instance pair as the matching condition, then it is determined that the target log semantic information matches the security rule with the concept instance pair as the matching condition.

[0140] The determining unit 404 is configured to determine that the log to be detected is an abnormal log if a security rule corresponding to the semantic information of the target log is matched.

[0141] Based on the above-mentioned embodiment of the present invention, an anomaly detection device for operation and maintenance logs is disclosed. In this solution, a bidirectional LSTM model is used to adapt to changes in unstructured logs while achieving a deep understanding of log semantics. Security rules are constructed based on the obtained log semantic information to use the security rules for log anomaly detection, thereby fully utilizing contextual information to improve anomaly detection accuracy and generalization ability, and reduce maintenance costs.

[0142] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and system embodiment described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple model units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0143] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0144] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting anomalies in operation and maintenance logs, characterized in that: The method comprises: Get the log to be tested; Input the log to be detected into a pre-trained bidirectional LSTM model for semantic parsing to obtain the semantic information of the target log; Matching the target log semantic information with each pre-built security rule; the security rule is constructed based on the abnormal log semantic information of the abnormal log sample; the abnormal log semantic information is obtained by pre-inputting the abnormal log sample into the bidirectional LSTM model for semantic parsing; the abnormal log sample includes at least: an unstructured abnormal log; If the security rule corresponding to the target log semantic information is matched, it is determined that the log to be detected is an abnormal log.

2. The method according to claim 1, characterized in that The construction process of each of the security rules includes: Input the abnormal log samples into the pre-trained bidirectional LSTM model for semantic analysis to obtain the abnormal log semantic information; If the exception log semantic information contains concepts, a security rule is constructed with the concepts as matching conditions; if the exception log semantic information contains instances, a security rule is constructed with the instances as matching conditions; if the exception log semantic information contains concept-instance pairs consisting of concepts and instances, a security rule is constructed with the concept-instance pairs as matching conditions; the concepts include readable words; and the instances include character string groups.

3. The method according to claim 2, characterized in that Matching the target log semantic information with various pre-built security rules includes: If the target log semantic information includes a target concept, and the target concept is consistent with a concept in a security rule with a concept as a matching condition, then it is determined that the target log semantic information matches the security rule with a concept as a matching condition; If the target log semantic information includes a target instance, and the target instance is consistent with an instance in a security rule with an instance as a matching condition, then it is determined that the target log semantic information matches the security rule with an instance as a matching condition; If the target log semantic information contains a target concept instance pair, and the concept instance pair is consistent with the concept instance pair in the security rule with the concept instance pair as the matching condition, it is determined that the target log semantic information matches the security rule with the concept instance pair as the matching condition.

4. The method according to claim 1, wherein The training process of the bidirectional LSTM model includes: Build a bidirectional LSTM model to be trained; Get multiple unstructured logs; For each of the unstructured logs, a word segmenter is used to segment the logs to obtain log words of multiple word types; Calculating the ratio of the number of the log words corresponding to each word type to the total number of the log words in the unstructured log to obtain log word-level attributes; Eliminating unstructured logs that do not meet preset requirements from each of the unstructured logs based on the log word-level attributes, and constructing a training data set based on the remaining unstructured logs; For each log word in each unstructured log in the training data set, construct a word-level feature matrix corresponding to the log word; For each readable sentence in each of the unstructured logs in the training data set, the word-level feature matrices corresponding to the log words in the readable sentence are sequentially arranged to form a forward sentence-level feature matrix, and the word-level feature matrices corresponding to the log words in the readable sentence are reversely arranged to form a reverse sentence-level feature matrix; The forward sentence-level feature matrix is used as the forward input of the bidirectional LSTM model to be trained, and the reverse sentence-level feature matrix is used as the reverse input of the bidirectional LSTM model to be trained. The bidirectional LSTM model to be trained is trained to obtain a trained bidirectional LSTM model.

5. The method according to claim 4, characterized in that The step of constructing a word-level feature matrix corresponding to each log word in each unstructured log in the training data set includes: For each log word in each unstructured log in the training data set, identifying the word type of the log word and calculating the hash value of the log word; the word type includes: a readable word type and a meaningless string type; Based on the word type and the hash value, a word-level feature matrix corresponding to the log word is constructed.

6. A device for detecting anomalies in operation and maintenance logs, characterized in that: The device comprises: An acquisition unit, used to acquire the log to be detected; A semantic analysis unit is used to input the log to be detected into a pre-trained bidirectional LSTM model for semantic analysis to obtain semantic information of the target log; A matching unit is configured to match the target log semantic information with each pre-constructed security rule; the security rule is constructed based on the abnormal log semantic information of the abnormal log sample; the abnormal log semantic information is obtained by pre-inputting the abnormal log sample into the bidirectional LSTM model for semantic parsing; the abnormal log sample includes at least: an unstructured abnormal log; The determining unit is configured to determine that the log to be detected is an abnormal log if the security rule corresponding to the target log semantic information is matched.

7. The device according to claim 6, characterized in that The device further comprises: A rule construction unit is used to input an exception log sample into a pre-trained bidirectional LSTM model for semantic analysis to obtain exception log semantic information; if the exception log semantic information contains a concept, a security rule is constructed with the concept as a matching condition; if the exception log semantic information contains an instance, a security rule is constructed with the instance as a matching condition; if the exception log semantic information contains a concept-instance pair consisting of a concept and an instance, a security rule is constructed with the concept-instance pair as a matching condition; the concept includes a readable vocabulary; the instance includes a string group.

8. The device according to claim 7, characterized in that The matching unit is specifically used to: If the target log semantic information includes a target concept, and the target concept is consistent with a concept in a security rule with a concept as a matching condition, then it is determined that the target log semantic information matches the security rule with a concept as a matching condition; If the target log semantic information includes a target instance, and the target instance is consistent with an instance in a security rule with an instance as a matching condition, then it is determined that the target log semantic information matches the security rule with an instance as a matching condition; If the target log semantic information contains a target concept instance pair, and the concept instance pair is consistent with the concept instance pair in the security rule with the concept instance pair as the matching condition, it is determined that the target log semantic information matches the security rule with the concept instance pair as the matching condition.

9. The device according to claim 6, characterized in that The device further comprises: A training unit is used to build a bidirectional LSTM model to be trained; obtain multiple unstructured logs; for each of the unstructured logs, use a word segmenter to perform word segmentation to obtain log words of multiple word types; calculate the ratio of the number of the log words corresponding to each word type to the total number of the log words in the unstructured log to obtain log word-level attributes; based on the log word-level attributes, remove the unstructured logs that do not meet the preset requirements from each of the unstructured logs, and build a training data set based on the remaining unstructured logs; for each log word in each of the unstructured logs in the training data set, build the log word The method comprises the following steps: for each readable sentence in each unstructured log in the training data set, arranging the word-level feature matrices corresponding to the log words in the readable sentence in sequence to form a forward sentence-level feature matrix, and arranging the word-level feature matrices corresponding to the log words in the readable sentence in reverse order to form a reverse sentence-level feature matrix; using the forward sentence-level feature matrix as the forward input of the bidirectional LSTM model to be trained, and using the reverse sentence-level feature matrix as the reverse input of the bidirectional LSTM model to be trained, and training the bidirectional LSTM model to obtain a trained bidirectional LSTM model.

10. The device according to claim 9, characterized in that The training unit for constructing a word-level feature matrix corresponding to each log word in each unstructured log in the training data set is specifically configured to: For each log word in each unstructured log in the training data set, identifying the word type of the log word and calculating a hash value of the log word; The word types include: readable word type and meaningless character string type; Based on the word type and the hash value, a word-level feature matrix corresponding to the log word is constructed.

Images