Mass log classification and analysis method and device, computer equipment and storage medium
By segmenting and matching massive logs and generating feature chains and feature names, the problem of massive log analysis in the existing technology is solved, which is time-consuming, costly and low hit rate, and efficient and fast log analysis and classification is achieved.
Patent Information
- Application Number
- CN202411677381.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-04-11
AI Technical Summary
When processing massive security logs, the analysis process takes a long time, is costly and has a low hit rate, making it difficult to meet the current monitoring and analysis requirements.
By segmenting massive logs, log atomic units are formed, and matched with the pre-built log element feature library, feature chains are generated, similarity calculations and grouping are performed, and log feature names are automatically generated to increase log readability.
It improves the efficiency of log parsing, can quickly classify and format logs, reduces the difficulty and cost of manual resolution, and quickly judges security logs with analysis value.
Smart Images
Figure CN119807757A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud security technology, and in particular to a method, device, computer equipment and storage medium for classifying and parsing massive logs. Background Art
[0002] In the prior art, faced with massive security logs, it is necessary to manually analyze the log format and the similarity of the logs, discover valuable and extractable log content, and form regular expressions, specific parsing code segments, etc. When parsing the logs, it is usually necessary to use regular expression code segments to test the logs to be parsed in sequence until one of the parsing rules can be matched, otherwise the log will not be parsed.
[0003] In the related technology, when the number of logs is very large, a large number of regular expressions or code snippets need to be written, which is a heavy workload and has extremely poor maintainability. Moreover, the higher the complexity of the logs, the more complex the regular expressions or code snippets, which means more R&D manpower is required and longer investment time, resulting in a time-consuming, costly and low hit rate process for parsing massive log information, which makes it difficult to meet the current monitoring and analysis requirements. Summary of the invention
[0004] In view of this, the present invention provides a method, device, computer equipment and storage medium for classifying and parsing massive logs to solve the problem that the process of parsing massive log information is time-consuming, costly, and has a low hit rate, which is difficult to meet the current monitoring and analysis requirements.
[0005] In a first aspect, the present invention provides a method for classifying and parsing massive logs, the method comprising:
[0006] Get the logs to be analyzed;
[0007] Split the log to be analyzed into atomic units;
[0008] Match the log atomic unit with the pre-built log meta-feature library, and store the successfully matched log atomic unit into the feature chain until all the log atomic units of the log to be analyzed are matched, and the feature chain corresponding to the log to be analyzed is obtained;
[0009] Calculate the similarity of feature chains, and group the logs to be analyzed of the same type into feature chains based on the similarity calculation results;
[0010] Generate a feature name corresponding to each group feature chain, calculate the feature name hash value of the feature name corresponding to each group feature chain, and add the feature chain to the pre-built log meta feature library based on the feature name hash value.
[0011] In the present invention, by segmenting massive logs, parsing the logs, forming feature chains, and then forming feature chain groups, the log feature names are automatically generated. When a feature chain automatically generates a feature name, the hash values of all feature names of this feature chain are calculated and matched against the feature chain library. If it already exists, other user-defined attributes of this feature chain in the feature chain library are called to increase the readability of the logs. This improves the efficiency of log parsing when processing massive logs, and can quickly classify the logs and form a formatted log format to increase the readability of the logs. Security analysts only need to mark the logs to complete the parsing work, which greatly reduces the difficulty and cost of manual parsing and quickly determines the security logs with analysis value.
[0012] In an optional implementation, the meta-feature library is constructed by the following steps:
[0013] Log features with a specific format and an occurrence frequency greater than a preset threshold are used as meta-features, and a meta-feature library is constructed based on the meta-features.
[0014] In this method, log features with a fixed format and high frequency of occurrence are collected in advance and extracted as meta-features to form a rich meta-feature library, which is convenient for matching during subsequent log analysis, thereby facilitating the analysis of the logs.
[0015] In an optional implementation, the log to be analyzed is segmented to obtain log atomic units, including:
[0016] Each log to be analyzed is segmented by using a preset log separator to obtain log atomic units, and each log atomic unit is stored in the form of a dynamic array so that the log to be analyzed corresponds to the log atomic unit array.
[0017] In this method, each input log is separated by a log separator to form a log atomic unit. Each log atomic unit is stored in a dynamic array. Each log corresponds to a log atomic unit array, which facilitates log parsing and feature name generation using the log atomic unit array.
[0018] In an optional implementation, matching the log atomic unit with a pre-built log meta-feature library and storing the successfully matched log atomic unit into a feature chain includes:
[0019] Each element in the log atomic cell array is matched with the pre-built log meta-feature library, a meta-feature object is constructed from the successfully matched log atomic cell array, and the feature value of the meta-feature object is updated.
[0020] In this method, by constructing a meta-feature object from the successfully matched log atomic cell array and updating the feature value of the meta-feature object, the successfully matched log atomic cell array is named, which facilitates the formation of a feature chain and the parsing of the feature chain composed of a large number of logs.
[0021] In an optional implementation, when the log atomic cell array fails to match, the log atomic cell array that fails to match is used to construct a log feature object, the feature value of the log feature object is updated, and the feature name of the log feature object is left blank.
[0022] In this method, the log atomic units that cannot match the meta-features are formed into a new log feature object, the feature value is updated, the feature name is left blank, and the log atomic units that cannot match the meta-features are manually named and combined with the successfully matched log atomic units to form a feature chain to facilitate the subsequent automatic generation of feature names for logs.
[0023] In an optional implementation, generating a feature name corresponding to each group feature chain includes:
[0024] The preset feature name is used as the feature name of the meta-feature of the feature chain of the same group;
[0025] All log features on each feature chain are compared and compared to determine the log features with the same position and the same feature value, and the feature values of the log features with the same position and the same feature value are used as the feature names of the log features.
[0026] In this method, for the same set of feature chains, the feature name of the meta-feature uses the predefined feature name and does not need to be changed. A difference comparison is performed on each log feature on the feature chain. The feature value of the log feature with the same position and the same feature value will be used as the feature name of the log feature to form a new feature chain. After the above process, a feature chain model of this type of log is formed and stored in the log feature chain library, which is convenient for calling other user-defined attributes of this feature chain in the feature chain library to increase log readability.
[0027] In a second aspect, the present invention provides a massive log classification and parsing device, the device comprising:
[0028] Log acquisition module, used to obtain logs to be analyzed;
[0029] The log segmentation module is used to segment the logs to be analyzed and obtain log atomic units;
[0030] The log matching module is used to match the log atomic unit with the pre-built log meta-feature library, and store the successfully matched log atomic unit into the feature chain until all the log atomic units of the log to be analyzed are matched, and the feature chain corresponding to the log to be analyzed is obtained;
[0031] The log grouping module is used to calculate the similarity of feature chains and group the logs to be analyzed of the same type into feature chains based on the similarity calculation results;
[0032] The log name generation module is used to generate a feature name corresponding to each group feature chain, calculate the feature name hash value of the feature name corresponding to each group feature chain, and add the feature chain to the pre-built log meta-feature library based on the feature name hash value.
[0033] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the massive log classification and parsing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for classifying and parsing massive logs according to the first aspect or any corresponding embodiment thereof.
[0035] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the method for classifying and parsing massive logs according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0037] Figure 1 The figure is a flow chart of a method for classifying and parsing massive logs according to an embodiment of the present invention.
[0038] Figure 2 The present invention is a flowchart of a method for automatically classifying and parsing massive security logs according to an embodiment of the present invention.
[0039] Figure 3 It is a schematic diagram of a feature chain after feature names are automatically generated according to an embodiment of the present invention.
[0040] Figure 4 It is a schematic diagram of a record in a feature chain library according to an embodiment of the present invention.
[0041] Figure 5The figure is a flowchart of another method for classifying and parsing massive logs according to an embodiment of the present invention.
[0042] Figure 6 The figure is a flowchart of another method for classifying and parsing massive logs according to an embodiment of the present invention.
[0043] Figure 7 It is a structural block diagram of a device for classifying and analyzing massive logs according to an embodiment of the present invention.
[0044] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0046] In the related technology, when the number of logs is very large, a large number of regular expressions or code snippets need to be written, which is a heavy workload and has extremely poor maintainability. Moreover, the higher the complexity of the logs, the more complex the regular expressions or code snippets, which means more R&D manpower is required and longer investment time, resulting in a time-consuming, costly and low hit rate process for parsing massive log information, which makes it difficult to meet the current monitoring and analysis requirements.
[0047] In order to solve the above problems, a method for classifying and parsing massive logs is provided in an embodiment of the present invention, which is used in a computer device. It should be noted that its execution subject can be a massive log classification and parsing device, which can be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware. The computer device can be a terminal, a client, or a server. The server can be a single server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer, or other intelligent hardware devices. In the following method embodiments, the execution subject is a computer device as an example for explanation.
[0048] The computer device in this embodiment is suitable for use scenarios in which massive security logs are parsed. The present invention provides a method for classifying and parsing massive logs, which divides massive logs, parses logs, forms feature chains, and then forms feature chain groups, automatically generates log feature names, and when a feature chain automatically generates a feature name, calculates the hash values of all feature names of this feature chain, matches the feature chain library, and if it already exists, calls other user-defined attributes of this feature chain in the feature chain library to increase log readability, thereby improving the efficiency of log parsing when processing massive logs, and can quickly classify logs and form formatted log formats to increase log readability. Security analysts only need to mark logs to complete parsing work, greatly reducing the difficulty and cost of manual parsing, and quickly judging security logs with analysis value.
[0049] According to an embodiment of the present invention, an embodiment of a method for classifying and parsing massive logs is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0050] In this embodiment, a method for classifying and parsing a large amount of logs is provided, which can be used for the above-mentioned computer device. Figure 1 FIG. 1 is a flow chart of a method for classifying and parsing massive logs according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0051] Step S101, obtaining the log to be analyzed.
[0052] In one example, the massive amount of logs to be analyzed are input through input interfaces such as syslog, http, kafka, snmp, etc., and the input interface is not limited in the present invention. Security products and network devices generally have the ability to send logs and status outward. The analysis system can cover most security products and network products log input requirements by implementing log collection interfaces such as syslog, http, kafka and snmp query capabilities.
[0053] Step S102, splitting the log to be analyzed to obtain log atomic units.
[0054] In one example, each input log is separated by a log separator to form a log atomic unit. Each log atomic unit is stored in a dynamic array, and each log corresponds to a log atomic unit array.
[0055] Step S103, matching the log atomic unit with the pre-built log meta-feature library, storing the successfully matched log atomic unit into the feature chain, until all the log atomic units of the log to be analyzed are matched, and obtaining the feature chain corresponding to the log to be analyzed.
[0056] In one example, each element of the log atomic cell array is first matched with the meta-feature library, and if the match is successful, a new meta-feature object is formed, and the feature value of this object is updated and stored in the feature chain.
[0057] Step S104, performing similarity calculation on the feature chains, and grouping the same type of logs to be analyzed into feature chains based on the similarity calculation results.
[0058] In one example, the cosine similarity judgment method is used to take the characteristic value of each element of the log feature chain as the calculation object. When the similarity is greater than 9.5, it represents the same type of log. In the present invention, the threshold of the similarity is not limited to form a feature chain grouping and enter the process of automatically generating feature names for logs.
[0059] Step S105, generating a feature name corresponding to each group feature chain, calculating a feature name hash value of the feature name corresponding to each group feature chain, and adding the feature chain to a pre-built log meta feature library based on the feature name hash value.
[0060] In one example, for the same set of feature chains, the feature name of the meta-feature uses a predefined feature name and does not need to be changed. A difference comparison is performed on each log feature on the feature chain. The feature value of the log feature with the same position and the same feature value will be used as the feature name of the log feature to form a new target feature chain. In the subsequent log parsing process, when a feature chain automatically generates a feature name, the hash value of all feature names of this feature chain will be calculated and matched with the feature chain library. If it already exists, other user-defined attributes of this feature chain in the feature chain library will be called to increase log readability.
[0061] In one implementation scenario, Figure 2 FIG. 1 is a flow chart of a method for automatically classifying and parsing massive security logs according to an embodiment of the present invention. Figure 2 As shown, the method for automatically classifying and parsing massive security logs includes: massive logs are formed into log atomic units through log delimiters; log atomic units are used to parse logs, and log atomic units that successfully match meta-features are merged into feature chains, and log atomic units that fail to match are extracted and merged into feature chains; feature chains are grouped; feature names are automatically generated; and feature chains are added to feature chain libraries.
[0062] The log parsing process may include: In a computer system, the operating system kernel, various application servers, and most applications may output logs. The content, size, and purpose of the logs are also different. This information often contains a lot of useful information for users. Users can analyze different types of log information to achieve different functional requirements. For example, an SSH login log: 14:39:29ubuntu login
[24290] :ROOT LOGIN on' / dev / tty1' can display the content to obtain the visitor's IP, access time, login username, etc.
[0063] Original log:
[0064] First entry: 14:39:29ubuntu login
[24290] : ROOT LOGIN on' / dev / tty1'
[0065] Second entry: 15:22:35ubuntu login
[24324] : admin LOGIN on ' / dev / tty1'
[0066] The third item: 18:14:40ubuntu login
[26245] : user2 LOGIN on' / dev / tty1' ......
[0068] Meta-feature: Time
[0069] The results after log separation and parsing:
[0070] First log:
[0071] 0: {ID=1, KEY="TIME", PATTERN="^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]$", VALUE="14:39:29"};
[0072] 1: {KEY="UNKNOWN", VALUE="ubuntu"};
[0073] 2: {KEY="UNKNOWN", VALUE="login"};
[0074] 3: {KEY="UNKNOWN", VALUE="24324"};
[0075] 4: {KEY="UNKNOWN", VALUE="ROOT"};
[0076] 5: {KEY = "UNKNOWN", VALUE = "LOGIN"};
[0077] 6: {KEY = "UNKNOWN", VALUE = "on"};
[0078] 7: {KEY = "UNKNOWN", VALUE = " / dev / tty1"}.
[0079] The second log:
[0080] 0: {ID = 1, KEY = "TIME", PATTERN = "^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]$", VALUE = "15:22:35"};
[0081] 1: {KEY = "UNKNOWN", VALUE = "ubuntu"};
[0082] 2: {KEY = "UNKNOWN", VALUE = "login"};
[0083] 3: {KEY = "UNKNOWN", VALUE = "24324"};
[0084] 4: {KEY = "UNKNOWN", VALUE = "admin"};
[0085] 5: {KEY = "UNKNOWN", VALUE = "LOGIN"};
[0086] 6: {KEY = "UNKNOWN", VALUE = "on"};
[0087] 7: {KEY = "UNKNOWN", VALUE = " / dev / tty1"}.
[0088] The third log:
[0089] 0: {ID = 1, KEY = "TIME", PATTERN = "^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]$", VALUE = "18:14:40"};
[0090] 1: {KEY = "UNKNOWN", VALUE = "ubuntu"};
[0091] 2: {KEY = "UNKNOWN", VALUE = "login"};
[0092] 3: {KEY="UNKNOWN", VALUE="26245"};
[0093] 4: {KEY="UNKNOWN", VALUE="user2"};
[0094] 5: {KEY="UNKNOWN", VALUE="LOGIN"};
[0095] 6: {KEY="UNKNOWN", VALUE="on"};
[0096] 7: {KEY="UNKNOWN", VALUE=" / dev / tty1"}.
[0097] Figure 3 is a schematic diagram of a feature chain after a feature name is automatically generated according to an embodiment of the present invention. Figure 3 shown. Figure 4 is a schematic diagram of a record in a feature chain library according to an embodiment of the present invention. By manually defining tags and merging them into a feature chain, a record in the feature chain library is finally formed. Figure 4 The sha256 hash value of the text "TIME ubuntu loginUNKNOWN UNKNOWN LOGIN on / dev / tty1" is calculated as fe4e376343ddd61b410ba47d13f02de3e6aba56cda36ceef9840050b49c1aaa1. This record will be used as a basic record in the feature chain library to match subsequent security logs.
[0098] The massive log classification and parsing method provided in this embodiment divides the massive logs, parses the logs, forms feature chains, and then forms feature chain groups, automatically generates log feature names, and when a feature chain automatically generates a feature name, calculates the hash values of all feature names of this feature chain, matches the feature chain library, and if it already exists, calls other user-defined attributes of this feature chain in the feature chain library to increase log readability, thereby improving the efficiency of log parsing when processing massive logs, and can quickly classify logs and form formatted log formats to increase log readability. Security analysts only need to mark the logs to complete the parsing work, which greatly reduces the difficulty and cost of manual parsing and quickly determines security logs with analysis value.
[0099] In this embodiment, a method for classifying and parsing a large amount of logs is provided, which can be used for the above-mentioned computer device. Figure 5 FIG. 1 is a flow chart of another method for classifying and parsing massive logs according to an embodiment of the present invention. Figure 5As shown, the process includes the following steps:
[0100] Step S501: Obtain the log to be analyzed. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0101] Step S502: Segment the log to be analyzed to obtain log atomic units.
[0102] Specifically, the above step S502 includes:
[0103] Step S5021 , using a preset log separator, split each log to be analyzed to obtain log atomic units, and store each log atomic unit in the form of a dynamic array, so that the log to be analyzed and the log atomic unit array correspond to each other.
[0104] In one example, each input log is separated by a log separator to form a log atomic unit. Each log atomic unit is stored in a dynamic array, and each log corresponds to a log atomic unit array.
[0105] In this method, each input log is separated by a log separator to form a log atomic unit. Each log atomic unit is stored in a dynamic array. Each log corresponds to a log atomic unit array, which facilitates log parsing and feature name generation using the log atomic unit array.
[0106] Step S503, matching the log atomic unit with the pre-built log meta-feature library, and storing the successfully matched log atomic unit into the feature chain, until all the log atomic units of the log to be analyzed are matched, and the feature chain corresponding to the log to be analyzed is obtained. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.
[0107] In an optional implementation, the meta-feature library is constructed by the following steps: log features having a specific format and an occurrence frequency greater than a preset threshold are used as meta-features, and the meta-feature library is constructed based on the meta-features.
[0108] In one example, the construction of a log meta-feature library may include: security logs are usually composed of some basic features, such as time, IP, port, host name, log string, etc. For meta-features with a fixed format, fixed matching units need to be prepared to form a meta-feature library. The meta-feature library contains meta-feature ID, matching mode, feature name (KEY), etc. The content contained in the meta-feature library is not limited in the present invention. Meta-features have the characteristics of being accumulative and reusable. In daily log analysis work, log features with a fixed format and high frequency of occurrence are extracted as meta-features to form a rich meta-feature library.
[0109] In this method, log features with a fixed format and high frequency of occurrence are collected in advance and extracted as meta-features to form a rich meta-feature library, which is convenient for matching during subsequent log analysis, thereby facilitating the analysis of the logs.
[0110] Step S504: perform similarity calculation on the feature chains, and group the logs to be analyzed of the same type into feature chains based on the similarity calculation results. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.
[0111] Step S505: Generate a feature name corresponding to each group feature chain, calculate the feature name hash value of each group feature chain, and add the feature chain to the pre-built log meta feature library based on the feature name hash value. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.
[0112] The massive log classification and parsing method provided in this embodiment collects log features with fixed formats and high occurrence frequencies in advance and extracts them as meta-features to form a rich meta-feature library, which is convenient for subsequent log parsing to use the meta-feature library for matching, thereby facilitating the parsing of logs. By separating each input log with a log separator to form a log atomic unit, each log atomic unit is stored in a dynamic array, and each log corresponds to a log atomic unit array, which facilitates log parsing and feature name generation using the log atomic unit array.
[0113] In this embodiment, a method for classifying and parsing a large amount of logs is provided, which can be used for the above-mentioned computer device. Figure 6 FIG. 1 is a flowchart of another method for classifying and parsing massive logs according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:
[0114] Step S601: Obtain the log to be analyzed. Figure 5 Step S501 of the illustrated embodiment will not be described in detail here.
[0115] Step S602: Segment the log to be analyzed to obtain log atomic units. Figure 5 Step S502 of the illustrated embodiment will not be described in detail here.
[0116] Step S603, matching the log atomic unit with the pre-built log meta-feature library, storing the successfully matched log atomic unit into the feature chain, until all the log atomic units of the log to be analyzed are matched, and obtaining the feature chain corresponding to the log to be analyzed.
[0117] Specifically, the above step S603 includes:
[0118] Step S6031, matching each element in the log atomic cell array with the pre-built log meta-feature library, constructing a meta-feature object from the successfully matched log atomic cell array, and updating the feature value of the meta-feature object.
[0119] Step S6032, when the log atomic cell array fails to match, construct a log feature object with the log atomic cell array that fails to match, update the feature value of the log feature object, and leave the feature name of the log feature object blank.
[0120] In one example, each element of the log atomic unit array is first matched to the meta-feature library. If the match is successful, a new meta-feature object is formed, and the feature value of this object is updated and stored in the feature chain. For the log atomic unit that cannot match the meta-feature, a new log feature object is formed, the feature value is updated, and the feature name is left blank for manual naming of this feature. The log atomic unit array is traversed, and the above matching process is repeated to finally form the feature chain of this log. Every time 500 feature chains are formed, the feature chain analysis process is entered. The number of feature chains entering the feature chain analysis process is not limited in the present invention.
[0121] In this method, the array of successfully matched log atomic units is used to construct a meta-feature object, and the feature value of the meta-feature object is updated, so that the array of successfully matched log atomic units can be named, which is convenient for forming a feature chain and parsing the feature chain composed of a large number of logs. The log atomic units that cannot match the meta-feature are formed into a new log feature object, the feature value is updated, and the feature name is left blank. The log atomic units that cannot match the meta-feature are manually named later, and combined with the successfully matched log atomic units, a feature chain is formed to facilitate the subsequent automatic generation of feature names for logs.
[0122] Step S604: perform similarity calculation on the feature chains, and group the logs to be analyzed of the same type into feature chains based on the similarity calculation results. Figure 5 Step S504 of the illustrated embodiment will not be described in detail here.
[0123] Step S605, generating a feature name corresponding to each group feature chain, calculating a feature name hash value of the feature name corresponding to each group feature chain, and adding the feature chain to a pre-built log meta feature library based on the feature name hash value.
[0124] Specifically, the above step S605 includes:
[0125] Step S6051: Use a preset feature name as the feature name of the meta-feature of the feature chain of the same group.
[0126] Step S6052, performing a difference comparison on all log features on each feature chain, determining log features with the same position and the same feature value, and using the feature values of the log features with the same position and the same feature value as feature names of the log features.
[0127] In one example, for the same set of feature chains, the feature name of the meta-feature uses the predefined feature name and does not need to be changed. A difference comparison is performed on each log feature on the feature chain. The feature value of the log feature with the same position and the same feature value will be used as the feature name of the log feature to form a new target feature chain. After the above process, all log features with undefined feature names are marked with "UNKNOWN" and manually marked by the user during the analysis process. At this point, a feature chain model for this type of log is formed and stored in the log feature chain library. Among them, all automatically generated feature names are calculated to generate hash values as the unique ID of the log feature chain in the feature library.
[0128] In the subsequent log parsing process, when a feature chain automatically generates a feature name, the hash values of all feature names of this feature chain will be calculated and matched against the feature chain library. If it already exists, other user-defined attributes of this feature chain in the feature chain library will be called to increase log readability.
[0129] In this method, for the same set of feature chains, the feature name of the meta-feature uses the predefined feature name and does not need to be changed. A difference comparison is performed on each log feature on the feature chain. The feature value of the log feature with the same position and the same feature value will be used as the feature name of the log feature to form a new feature chain. After the above process, a feature chain model of this type of log is formed and stored in the log feature chain library, which is convenient for calling other user-defined attributes of this feature chain in the feature chain library to increase log readability.
[0130] The massive log classification and parsing method provided in this embodiment constructs a meta-feature object from the array of successfully matched log atomic units, updates the feature value of the meta-feature object, and realizes the naming of the array of successfully matched log atomic units, which is convenient for forming a feature chain and parsing the feature chain composed of a large number of logs. By forming a new log feature object from the log atomic unit that cannot match the meta-feature, updating the feature value, leaving the feature name blank, and subsequently manually naming the log atomic unit that cannot match the meta-feature, combined with the successfully matched log atomic unit, a feature chain is formed to facilitate the subsequent automatic generation of feature names for logs. For the same set of feature chains, the feature name of the meta-feature uses a predefined feature name and does not need to be changed. A difference comparison is made for each log feature on the feature chain. The feature value of the log feature with the same position and the same feature value will be used as the feature name of the log feature to form a new feature chain. After the above process, a feature chain model of such logs is formed and stored in the log feature chain library, which is convenient for calling other user-defined attributes of this feature chain in the feature chain library to increase the readability of the log.
[0131] In this embodiment, a massive log classification and parsing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0132] This embodiment provides a massive log classification and analysis device, such as Figure 7 As shown, including:
[0133] The log acquisition module 701 is used to acquire the logs to be analyzed. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0134] The log segmentation module 702 is used to segment the log to be analyzed to obtain log atomic units. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0135] The log matching module 703 is used to match the log atomic unit with the pre-built log meta-feature library, and store the successfully matched log atomic unit into the feature chain until all the log atomic units of the log to be analyzed are matched, and the feature chain corresponding to the log to be analyzed is obtained. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.
[0136] The log grouping module 704 is used to calculate the similarity of the feature chain, and based on the similarity calculation results, group the logs to be analyzed of the same type into feature chains. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.
[0137] The log name generation module 705 is used to generate a feature name corresponding to each group feature chain, calculate the feature name hash value of the feature name corresponding to each group feature chain, and add the feature chain to the pre-built log meta-feature library based on the feature name hash value. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.
[0138] In some optional implementations, the massive log classification and parsing device further includes:
[0139] The log meta-feature library construction unit is used to take log features with a specific format and an occurrence frequency greater than a preset threshold as meta-features, and to construct a meta-feature library based on the meta-features.
[0140] In some optional implementations, the log segmentation module 702 includes:
[0141] The log segmentation unit is used to segment each log to be analyzed using a preset log separator to obtain log atomic units, and store each log atomic unit in the form of a dynamic array so that the log to be analyzed corresponds to the log atomic unit array.
[0142] In some optional implementations, the log matching module 703 includes:
[0143] The log matching success unit is used to match each element in the log atomic unit array with the pre-built log meta-feature library, construct a meta-feature object from the successfully matched log atomic unit array, and update the feature value of the meta-feature object.
[0144] In some optional implementations, the log matching module 703 includes:
[0145] The log matching failure unit is used to construct a log feature object with the log atomic cell array that failed to match, update the feature value of the log feature object, and leave the feature name of the log feature object blank when the log atomic cell array fails to match.
[0146] In some optional implementations, the log name generation module 705 includes:
[0147] The meta-feature name generating unit is used to use the preset feature name as the feature name of the meta-feature of the feature chain of the same group.
[0148] The log feature name generating unit is used to compare the differences of all log features on each feature chain, determine the log features with the same position and the same feature value, and use the feature values of the log features with the same position and the same feature value as the feature names of the log features.
[0149] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0150] The massive log classification and analysis device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0151] The embodiment of the present invention also provides a computer device having the above Figure 7 The massive log classification and parsing device shown.
[0152] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.
[0153] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0154] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0155] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0156] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0157] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.
[0158] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0159] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0160] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0161] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for classifying and parsing massive logs, characterized in that: The method comprises: Get the logs to be analyzed; Segmenting the log to be analyzed to obtain log atomic units; Match the log atomic unit with the pre-built log meta-feature library, and store the successfully matched log atomic unit into the feature chain until all the log atomic units of the log to be analyzed are matched, thereby obtaining the feature chain corresponding to the log to be analyzed; Calculate similarity of the feature chains, and group the logs to be analyzed of the same type into feature chains based on the similarity calculation results; Generate a feature name corresponding to each group feature chain, calculate a feature name hash value of the feature name corresponding to each group feature chain, and add the feature chain to the pre-built log meta feature library based on the feature name hash value.
2. The method according to claim 1, characterized in that The meta-feature library is constructed by the following steps: Log features with a specific format and an occurrence frequency greater than a preset threshold are used as meta-features, and the meta-feature library is constructed based on the meta-features.
3. The method according to claim 1, characterized in that The step of segmenting the log to be analyzed to obtain log atomic units includes: Each of the logs to be analyzed is segmented by using a preset log separator to obtain the log atomic unit, and each of the log atomic unit is stored in the form of a dynamic array, so that the log to be analyzed corresponds to the log atomic unit array.
4. The method according to claim 3, characterized in that: The step of matching the log atomic unit with a pre-built log meta-feature library and storing the successfully matched log atomic unit into a feature chain includes: Each element in the log atomic unit array is matched with the pre-built log meta-feature library, a meta-feature object is constructed from the successfully matched log atomic unit array, and a feature value of the meta-feature object is updated.
5. The method according to claim 4, characterized in that When the log atomic unit array fails to match, a log feature object is constructed using the log atomic unit array that fails to match, a feature value of the log feature object is updated, and a feature name of the log feature object is left blank.
6. The method according to claim 5, characterized in that The generating of the feature name corresponding to each group feature chain includes: The preset feature name is used as the feature name of the meta-feature of the feature chain of the same group; A difference comparison is performed on all log features on each of the feature chains to determine log features with the same position and the same feature value, and the feature values of the log features with the same position and the same feature value are used as feature names of the log features.
7. A massive log classification and analysis device, characterized in that: The device comprises: Log acquisition module, used to obtain logs to be analyzed; A log segmentation module, used to segment the log to be analyzed to obtain log atomic units; A log matching module is used to match the log atomic unit with a pre-built log meta-feature library, and store the successfully matched log atomic unit into a feature chain until all log atomic units of the log to be analyzed are matched, thereby obtaining a feature chain corresponding to the log to be analyzed; A log grouping module is used to perform similarity calculation on the feature chains, and group the same type of logs to be analyzed into feature chains based on the similarity calculation results; The log name generation module is used to generate a feature name corresponding to each group feature chain, calculate the feature name hash value of the feature name corresponding to each group feature chain, and add the feature chain to the pre-built log meta-feature library based on the feature name hash value.
8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for classifying and parsing massive logs according to any one of claims 1 to 6 by executing the computer instructions.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the mass log classification and parsing method according to any one of claims 1 to 6.
10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the mass log classification and parsing method according to any one of claims 1 to 6.