Log analysis method and device, log anomaly detection method and device, equipment and medium

Abstract

The invention provides a log analysis method and device, a log anomaly detection method and device, equipment and a medium. Relates to the technical field of data analysis. The log analysis method comprises the following steps: receiving original log data, wherein the log data comprises a plurality of pieces of log information; obtaining length information and first word information of the log information, and replacing the first word information with preset information when the first word information is a parameter; determining a target log group to which the log information belongs in an analytic tree according to the length information and the first word information of the log information; and merging the log information into the target log group to which the log information belongs. According to the invention, the log analysis speed and accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data analysis, and in particular to a log parsing method and device, a log anomaly detection method and device, an electronic device, and a computer-readable medium. Background Art

[0002] Log parsing is one of the key parts of log analysis. Its purpose is to convert unstructured or semi-structured log data into structured log data. The parsed log data can be used for log anomaly detection or other functions.

[0003] Some existing log parsing methods often require manual writing of regular expressions, and a fixed number of original logs are needed to train templates for parsing. The accuracy and speed of log parsing still have room for improvement.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a log parsing method and device, a log anomaly detection method and device, an electronic device, and a computer-readable medium, thereby improving the accuracy and speed of log parsing at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a log parsing method is provided, the method comprising:

[0007] Receiving original log data, wherein the log data includes a plurality of log information;

[0008] Acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter;

[0009] Determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0010] The log information is merged into the target log group to which it belongs.

[0011] According to a second aspect of the present disclosure, a log anomaly detection method is provided, the method comprising:

[0012] Receiving original log data, wherein the log data includes a plurality of log information;

[0013] Acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter;

[0014] Determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0015] Merging the log information into the target log group to which it belongs to obtain target log data;

[0016] Perform log anomaly detection on the target log data.

[0017] According to a third aspect of the present disclosure, a log parsing device is provided, the device comprising:

[0018] A receiving module, used for receiving original log data, wherein the log data includes a plurality of log information;

[0019] An information acquisition module, used to acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter;

[0020] A matching module, used to determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0021] The merging module is used to merge the log information into the target log group to which it belongs.

[0022] According to a fourth aspect of the present disclosure, a log anomaly detection device is provided, the device comprising:

[0023] A receiving module, used for receiving original log data, wherein the log data includes a plurality of log information;

[0024] An information acquisition module, used to acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter;

[0025] A matching module, used to determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0026] A merging module, used for merging the log information into the target log group to which it belongs, to obtain target log data;

[0027] The detection module is used to perform log anomaly detection on the target log data.

[0028] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0029] According to a sixth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the methods described above is implemented.

[0030] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0031] In the log parsing method provided by the disclosed exemplary embodiment, when the first word information is a parameter, the first word information can be replaced with preset information, and the target log group to which the log information belongs can be automatically determined in the parsing tree based on the length information and the first word information of the log information, and then the log information can be merged into the target log group. On the one hand, not only the length information is used to match the target log group, but also the first word information is used to match the target log group, thereby improving the accuracy of log parsing; on the other hand, when the first word information is a parameter, the parameter is replaced with preset information, which can avoid branch explosion, and can also reduce the number of nodes to increase the traversal speed, thereby increasing the speed of log parsing.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0034] Figure 1 A schematic diagram showing an exemplary system architecture of a log parsing method and device to which the embodiments of the present disclosure can be applied;

[0035] Figure 2 A flow chart of a log parsing method according to an embodiment of the present disclosure is schematically shown;

[0036] Figure 3 A flowchart of obtaining the length information and the first word information of log information according to an embodiment of the present disclosure is schematically shown;

[0037] Figure 4 The structure diagram of the parse tree according to one embodiment of the present disclosure is schematically shown;

[0038] Figure 5 A flowchart of determining a target log group in a parse tree according to an embodiment of the present disclosure is schematically shown;

[0039] Figure 6A flowchart of determining a target log group in a leaf node according to an embodiment of the present disclosure is schematically shown;

[0040] Figure 7 A flowchart of updating a parse tree after failing to match a target log group according to an embodiment of the present disclosure is schematically shown;

[0041] Figure 8 Schematically shows an example diagram of updating a parse tree according to an embodiment of the present disclosure;

[0042] Fig. 9 The effect diagram of log parsing according to an embodiment of the present disclosure is schematically shown;

[0043] Fig.10 The flowchart of the log anomaly detection method according to an embodiment of the present disclosure is schematically shown;

[0044] Fig.11 A block diagram of a log parsing device according to an embodiment of the present disclosure is schematically shown;

[0045] Fig.12 A block diagram of a log anomaly detection device according to an embodiment of the present disclosure is schematically shown;

[0046] Fig.13 The structure diagram of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0048] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0049] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a log parsing method and apparatus according to an embodiment of the present disclosure can be applied is shown.

[0050] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0051] The log parsing method provided by the disclosed embodiment can be executed on the server 105, and the log parsing device is generally arranged in the server 105. For example, in an exemplary embodiment, the user can run the software on the terminal devices 101, 102, and 103, and the terminal devices 101, 102, and 103 upload the original log data including multiple log information generated during the software operation to the server 105; after receiving the log information, the server 105 can first obtain the length information and the first word information of the log information, and replace the first word information with the preset information when the first word information is a parameter, and then determine the target log group to which the log information belongs in the parsing tree according to the length information and the first word information of the log information, and finally merge the log information into the target log group to obtain the target log data. However, it is easy for those skilled in the art to understand that the above application scenario is only used for example, and is not limited to this in this exemplary embodiment.

[0052] The technical solution of the embodiment of the present disclosure is described in detail below:

[0053] This exemplary embodiment provides a log parsing method. Figure 2 As shown, the log parsing method may include the following steps:

[0054] Step S210, receiving original log data, the log data including a plurality of log information;

[0055] Step S220, obtaining the length information and the first word information of the log information, and replacing the first word information with preset information when the first word information is a parameter;

[0056] Step S230, determining the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0057] Step S240: merge the log information into the target log group to which it belongs.

[0058] In the log parsing method provided in this exemplary embodiment, on the one hand, not only length information is used to match the target log group, but also first word information is used to match the target log group, thereby improving the accuracy of log parsing; on the other hand, when the first word information is a parameter, the parameter is replaced with preset information, which can avoid branch explosion and reduce the number of nodes to increase the traversal speed, thereby improving the speed of log parsing.

[0059] In another exemplary embodiment, the above steps are described in more detail below.

[0060] In step S210 , original log data is received, where the log data includes a plurality of log information.

[0061] In this exemplary embodiment, the original log data can be obtained through log collection tools, for example, using log collection tools such as Chukwa, Apache Flume, Scribe and Kafka to collect logs; the original log data can also be a publicly available log data set, such as a BLG data set, an HPC data set, an HDFS data set and other log data sets; in addition, in this exemplary embodiment, the original log data can also be the log data generated in real time by the user during the running of the software.

[0062] In this exemplary embodiment, the original log data can be text-type log data or system-type log data. In addition, it can also be SNMP (Simple Network Management Protocol)-type log data. Of course, the original log data can also be database-type log data. This exemplary embodiment does not specifically limit the type of original log data.

[0063] In this exemplary embodiment, log information refers to a log record in log data, which records relevant activity information of a specific event in the system. The log information records relevant operation information such as date, time, user and action.

[0064] In addition, in other exemplary embodiments of the present disclosure, after receiving the log information, the log information can also be preprocessed to remove information that is useless for log parsing. For example, specific parameters in the original log data, such as IP address, time, etc., can be identified through regular matching, and then replaced with wildcards or deleted. Of course, preprocessing can also be to remove or replace the timestamp, level or component information of the log, and this exemplary embodiment does not specifically limit the preprocessing method.

[0065] In step S220, the length information and the first word information of the log information are obtained, and when the first word information is a parameter, the first word information is replaced with the preset information.

[0066] In this exemplary embodiment, the length information is the number of segmented words in the log information, and the first word information is the first segmented word in the log information. Figure 3 As shown, in an exemplary embodiment of the present disclosure, after obtaining the log information, the length information and the first word information of the log information can be obtained through the following steps S310 to S330, wherein:

[0067] In step S310, the log information is segmented to obtain multiple segmented words. Taking the natural language toolkit (NLTK) segmentation as an example, segmented word groups can be obtained according to separators such as spaces and symbols, and then segmented words are filtered, and finally stems are extracted to obtain multiple segmented words. Of course, it is easy for those skilled in the art to understand that the above segmentation processing can also be implemented in any possible segmentation method, such as Keras segmentation, Spacy segmentation, or Gensim segmentation.

[0068] In step S320, the length information is determined according to the number of segmented words. The length information indicates the number of segmented words in a log message. For example, assuming that the log message is "Receive from node23", the result of segmentation processing is "Receive", "from", "node", "23", then the length information of the log message is 4.

[0069] In step S330, the first word information is determined according to the first word in the word segmentation. The first word information represents the first word of a log message. For example, assuming that the log message is "Verification succeeded for blk_145", the result of word segmentation processing is "Verification", "succeeded", "for", "blk_145", then the first word information of the log message is "Verification".

[0070] In this exemplary embodiment, the parameter is replaced with preset information when the first word information is a parameter. This is because for log information whose first word is a parameter, there are many parameters. If each log information whose first word is a parameter is matched with a child node, each different parameter will be traversed to a child node, which will cause the branches of the parse tree to explode when parsing through the parse tree, thereby reducing the parsing efficiency.

[0071] In an exemplary embodiment of the present disclosure, when the first word information is a parameter, the parameter can be replaced with preset information by the following steps: when the first word information is a parameter, the first word information is replaced with a wildcard, and the parameter is stored in the log information.

[0072] The wildcard can be "*" or "?". Of course, the wildcard can also be other types of characters, and the present exemplary embodiment is not limited thereto. The purpose of storing the parameters in the log information is to retain the original log data and ensure the integrity of the parsed log data. For example, assuming that the log information is "120 bytes received", in order to avoid branch explosion, the wildcard "*" can be used to replace the first word information "120", and "120" can be stored in the log information.

[0073] In step S230, the target log group to which the log information belongs is determined in the parse tree according to the length information and the first word information of the log information.

[0074] In this exemplary embodiment, the parsing tree may be a tree structure, which is used to parse the log information. Figure 4As shown, the parse tree can be composed of a root node 401, first-layer child nodes 402, 403, 404, second-layer child nodes 405, 406, 407, 408, and leaf nodes 409, 410, 411, wherein the second-layer child nodes are child nodes of the first-layer child nodes. Root node 401 can be used to receive original log data. First-layer child nodes 402, first-layer child nodes 403, and first-layer child nodes 404 store length information for matching log information whose length is consistent with the length information stored in the child nodes. Second-layer child nodes 405, second-layer child nodes 406, second-layer child nodes 407, and second-layer child nodes 407 are connected to their corresponding first-layer child nodes, and word information is stored thereon, which is used to match log information whose first word is consistent with the word information stored in the node. Among them, the root node, first-layer child nodes, and second-layer child nodes are used to guide the search process, and they do not store log information. Leaf nodes 409, 410, and 411 are used to store log groups. Referring to log group 412, each log group can be divided into two parts: a log template and log parameters. The log template is a template that can best describe the log information stored by the log group, and is composed of the constant part in the log information. The log parameters are composed of the parameters in the log information stored by the log group.

[0075] refer to Figure 5 As shown, in other exemplary embodiments of the present disclosure, the target log group to which the log information belongs can be determined through the following steps S510 to S530. Among them:

[0076] In step S510, according to the length information, the first-layer child nodes are traversed to determine the target first-layer child node. Figure 4 As shown, for the log information “Receive from node 23” with a length of 4, the matched target first-layer child node is the first-layer child node 402 .

[0077] In step S520, according to the first word information, the second-layer child nodes under the target first-layer child node are traversed to determine the target second-layer child node. Figure 4 As shown, for the log information “Receive from node23”, the matched target second-layer child node is the second-layer child node 406 .

[0078] In step S530, the leaf nodes under the target second-level child node are traversed to determine the target log group. Each leaf node stores a log group, which is composed of a log template and log parameters. Figure 6As shown, in an exemplary embodiment of the present disclosure, the target log group can be determined by traversing the leaf nodes under the target second-layer child node through the following steps S610 to S630, wherein:

[0079] In step S610, the similarity between each log group and the log information in the leaf node under the target second-layer child node is calculated. Specifically, the similarity between each log group and the log information in the leaf node is calculated, that is, the similarity between the log template and the log information of each log group is calculated, and the similarity can be determined by the ratio of the number of segmentations with the same position and content in the log template and the log information to the length of the log information.

[0080] For example, assuming that the log information is "Node 001is unconnected" and the log template is "Node*isunconnected", the three words "Node", "is" and "unconnected" are the same and in the same position, and the similarity between the log information and the log template is 0.75.

[0081] In addition, in other exemplary embodiments of the present disclosure, the similarity between the log information and each log group can also be calculated by other methods, for example, by LCS (Longest Common Subsequence), simSeq = LCS (seq1, seq2) to represent the similarity, where seq1 represents the log information and seq2 represents the log template; the similarity can also be represented by calculating the number of identical participles in the log information and the log template, etc., which also fall within the protection scope of the present disclosure.

[0082] In step S620, if the maximum similarity is not less than the similarity threshold, the log group with the maximum similarity is used as the target log group. The similarity threshold is preset, and assuming that the similarity threshold is 0.7, the log group with the log template "Node*is unconnected" can be used as the target log group with the log information "Node 001is unconnected".

[0083] In step S630, if the maximum similarity is less than the similarity threshold, an identifier is generated to indicate that the target log group is not matched. The identifier may be a letter, a number, or an underscore, but this exemplary embodiment is not limited thereto.

[0084] In an exemplary embodiment of the present disclosure, in the process of determining the target log group to which the log information belongs, it can also be determined whether the number of first-layer child nodes, second-layer child nodes and leaf nodes reaches a threshold; if the number of nodes reaches the threshold, the unmatched log information is matched to the specified child node.

[0085] Specifically, a maxChild parameter can be defined, which represents the threshold value of the number of child nodes, and is used to limit the maximum number of nodes of the first-layer child nodes, the second-layer child nodes, and the leaf nodes. If the number of nodes reaches maxChild, all unmatched log information will be matched to the specified child nodes. Among them, the specified child node is a preset child node, which stores information that can be distinguished from other child nodes. For example, the information stored in the specified child node can be preset to "×", "~", "—", etc. The maxChild parameter can be set according to the performance of the running device. The better the performance of the device, the larger the value that can be set for the maxChild parameter.

[0086] In other exemplary embodiments of the present disclosure, for log information that has no matching log group in the parse tree, the parse tree can update the parse tree based on the log information, add corresponding child nodes, and refer to Figure 7 As shown, after generating an identifier to indicate that the target log group is not matched, the following steps S710 to S720 may also be included, wherein:

[0087] In step S710, a new log group is generated based on the log information. Figure 8 As shown in the figure, it indicates the update process of the parse tree after the target log group is not matched. The length information of the log information "Receive 88bytes" is 3, and the first word information is "Receive". There is no second-level child node with the first word information "Receive" under the first-level child node with the length information of "3" in the parse tree. Therefore, the log information cannot match the target log group, so a new log group can be generated, and its log template is "Receive 88bytes".

[0088] In step S720, the parse tree is updated according to the new log group, wherein updating the parse tree means adding the missing child nodes and leaf nodes in the parse tree according to the new log group.

[0089] In an exemplary embodiment of the present disclosure, updating the parse tree according to the new log group may be traversing from the root node, and adding missing first-level child nodes, second-level child nodes, and leaf nodes along the path. Figure 8 As shown, for the log message "Receive 88bytes", after traversing the parse tree, it is found that the second-level child nodes and leaf nodes are missing, so the missing second-level child nodes 801 and leaf nodes 802 are added based on the new log group, the first word information in the second-level child node 801 is "Receive", and the new log group is stored in the leaf node 802.

[0090] In step S240, the log information is merged into the target log group. Specifically, the log information and the log template of the target log group can be scanned for the same word segmentation. If the two word segments are the same, the word segmentation in the log template remains unchanged. If the two word segments are different, the different word segments in the log template are replaced with wildcards, and then the parameters in the log message are added to the log parameters in the target log group. The original log data is parsed into two parts: the log template and the log parameters. Fig. 9 As shown, it represents the effect of log parsing, where 901 is the original log data, which includes multiple log information, 902 is the parsed log template, and 903 is the parsed log parameters. It can be seen that the original log data is parsed into structured log data and the log parameters are retained.

[0091] In another exemplary embodiment of the present disclosure, a log anomaly detection method is also provided. Fig.10 As shown, the log anomaly detection method may include the following steps S1010 to S1050:

[0092] Step S1010, receiving original log data, wherein the log data includes a plurality of log information;

[0093] Step S1020, obtaining the length information and the first word information of the log information, and replacing the first word information with the preset information when the first word information is a parameter;

[0094] Step S1030, determining the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0095] Step S1040, merging the log information into the target log group to which it belongs, to obtain target log data;

[0096] Step S1050: perform log anomaly detection on the target log data.

[0097] Specifically, log anomaly detection aims to discover abnormal patterns in log data. The anomaly types can be divided into hardware anomalies, operating system anomalies, database anomalies, etc. Log parsing is the previous step of log anomaly detection. The more accurate the classification of the parsed log data and the better the integrity of the information, the better the effect of log anomaly detection. Among them, log anomaly detection on the target log data can be to determine whether the log parameters in the target log data exceed the threshold. For example, set the parameter threshold to 30, input the log information "Create instance 6 milliseconds" and the log information "Create instance 600 milliseconds" into the parsing tree for parsing, and the returned log template is "Create instance * milliseconds", and the log parameter is "6,600", where "600" in the log parameter is greater than the parameter threshold 30, then the log information "Create instance 600 milliseconds" can be determined as an abnormal log.

[0098] In addition, in other exemplary embodiments of the present disclosure, the average value of the log parameters in the target log data can be calculated, and the log message whose difference with the average value exceeds a threshold value can be regarded as an abnormal log; the log parameters in the target log data can also be clustered, and the log message farthest from the cluster center can be regarded as an abnormal log. Of course, other types of methods can also be used to perform log anomaly detection, etc., which also fall within the protection scope of the present disclosure.

[0099] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0100] Furthermore, in this exemplary embodiment, a log parsing device is also provided. The log parsing device can be applied to a server or a terminal device. Fig.11 As shown, the log parsing device 1100 may include a receiving module 1110, an information acquisition module 1120, a matching module 1130 and a merging module 1140, wherein:

[0101] The receiving module 1110 may be used to receive original log data, where the log data includes a plurality of log information;

[0102] The information acquisition module 1120 may be used to acquire the length information and the first word information of the log information, and replace the first word information with the preset information when the first word information is a parameter;

[0103] The matching module 1130 may be used to determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0104] The merging module 1140 may be used to merge the log information into the target log group to which it belongs.

[0105] Furthermore, in an exemplary embodiment of the present disclosure, a log anomaly detection device is also provided. The log anomaly detection device can be applied to a server or a terminal device. Fig.12 As shown, the log parsing device 1200 may include a receiving module 1210, an information acquisition module 1220, a matching module 1230, a merging module 1240 and a detection module 1250, wherein:

[0106] The receiving module 1210 may be used to receive original log data, where the log data includes multiple log information;

[0107] The information acquisition module 1220 may be used to acquire the length information and the first word information of the log information, and replace the first word information with the preset information when the first word information is a parameter;

[0108] The matching module 1230 may be used to determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information;

[0109] The merging module 1240 may be used to merge the log information into the target log group to which it belongs, and obtain target log data;

[0110] The detection module 1250 may be used to perform log anomaly detection on target log data.

[0111] The specific details of each module or unit in the above device have been described in detail in its corresponding method, so they will not be repeated here.

[0112] As another aspect, the present disclosure further provides an electronic device, Fig.13 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0113] It should be noted that Fig.13 The computer system 1300 of the electronic device shown is only an example and should not bring any limitation to the functions and application scope of the embodiments of the present disclosure.

[0114] like Fig.13 As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for system operation are also stored. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0115] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0116] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the central processing unit (CPU) 1301, various functions defined in the method and apparatus of the present disclosure are executed. In some embodiments, the computer system 1300 may also include a data processor for processing computing operations related to log data.

[0117] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device may implement the following Figure 2 to Figure 10 The various steps shown, etc.

[0118] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0119] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A log parsing method, It is characterized in that The method comprises: Receiving original log data, wherein the log data includes a plurality of log information; Acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter; Determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information; The log information is merged into the target log group to which it belongs.

2. The method according to claim 1, It is characterized in that The parse tree includes: a root node, a first-layer child node, a second-layer child node and a leaf node, wherein the second-layer child node is a child node of the first-layer child node; The step of determining the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information includes: According to the length information, traverse the first-layer child nodes to determine a target first-layer child node; According to the first word information, traverse the second-layer child nodes under the target first-layer child node to determine the target second-layer child node; The target log group is determined by traversing the leaf nodes under the target second-level child node.

3. The method according to claim 2, It is characterized in that The method further comprises: Determine whether the number of the first-layer child nodes, the second-layer child nodes, and the leaf nodes reaches a threshold; If the number of nodes reaches a threshold, the unmatched log information is matched to a designated child node.

4. The method according to claim 2, It is characterized in that The traversing the leaf nodes under the target second-layer child node to determine the target log group includes: Calculate the similarity between each log group in the leaf node under the target second-layer child node and the log information; If the maximum value of the similarity is not less than the similarity threshold, the log group with the greatest similarity is used as the target log group; If the maximum value of the similarity is less than the similarity threshold, an identifier is generated to indicate that the target log group is not matched.

5. The method according to claim 4, It is characterized in that If the maximum value of the similarity is less than the similarity threshold, an identifier is generated to indicate that the target log group is not matched, and further includes: Generate a new log group based on the log information; Updating the parse tree according to the new log group includes: Traverse from the root node and add the missing first-layer child nodes, the second-layer child nodes and the leaf nodes along the path.

6. A log anomaly detection method, It is characterized in that The method comprises: Receiving original log data, wherein the log data includes a plurality of log information; Acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter; Determine the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information; Merging the log information into the target log group to which it belongs to obtain target log data; Perform log anomaly detection on the target log data.

7. A log parsing device, It is characterized in that The device comprises: A receiving module, used for receiving original log data, wherein the log data includes a plurality of log information; An information acquisition module, used to acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter; A matching module, used for determining the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information; The merging module is used to merge the log information into the target log group to which it belongs.

8. A log anomaly detection device, It is characterized in that The device comprises: A receiving module, used for receiving original log data, wherein the log data includes a plurality of log information; An information acquisition module, used to acquire the length information and the first word information of the log information, and replace the first word information with preset information when the first word information is a parameter; A matching module, used for determining the target log group to which the log information belongs in the parse tree according to the length information and the first word information of the log information; A merging module, used to merge the log information into the target log group to which it belongs, to obtain target log data; The detection module is used to perform log anomaly detection on the target log data.

9. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the log parsing method according to any one of claims 1 to 5, or implement the log anomaly detection method according to claim 6, by executing the executable instructions.

10. A computer readable medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the log parsing method according to any one of claims 1 to 5 is implemented, or each step of the log anomaly detection method according to claim 6 is implemented.

Images