A log unification strategy matching method, device and electronic equipment

Abstract

The application provides a log normalization strategy matching method and device and electronic equipment, and relates to the technical field of network security. In the application, a first original log set corresponding to a first sample attribute label is acquired; from the first original log set, each candidate original log containing the same word group type is selected to obtain a second original log set; an arbitrary candidate original log in the second original log set is selected as a target log, and a first word group contained in the target log is determined, and the first word group is matched with each sample log word group; if the matching result is not empty, a first normalization strategy set corresponding to the first word group is acquired, and when it is determined that the first normalization strategy set only contains one candidate normalization strategy, the candidate normalization strategy is taken as a target normalization strategy of the target log; if the matching result is empty, a new normalization strategy is created for the target log. In the above manner, the matching efficiency of the log normalization strategy can be improved.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a log generalization policy matching method, apparatus and electronic device. Background Technology

[0002] Logs are event records generated by network devices, systems, service programs, etc. during runtime, recording descriptions of related operations such as date, time, user, and actions.

[0003] When a network device is running, it generates specific log records according to a set log format. Because different network devices have different log formats, different types of log records are generated. Before analyzing these various logs, log parsing techniques are usually used to parse them and extract the valuable information contained within.

[0004] Currently, log parsing and normalization still rely on manual processing. In current log auditing systems, a large number of logs are accessed, written and recorded by different network devices according to their respective log writing formats. Typically, a log writing format corresponds to a specific normalization strategy, which refers to the parsing rules for logs. However, the total number of normalization strategies that can be created for parsing various types of logs is finite.

[0005] In existing technologies, to match suitable normalization strategies to logs with different writing formats, normalization strategies are used to parse the logs. The log auditing system selects a higher-priority normalization strategy from its set of normalization strategies and polls it against the logs according to a preset normalization strategy priority. For example, the log auditing system may have normalization strategy 1 and normalization strategy 2, with normalization strategy 1 having a higher priority than normalization strategy 2. Normalization strategy 1 is selected and compared with the logs to determine if the logs meet the log parsing rules set by normalization strategy 1. If so, normalization strategy 1 is used as the target normalization strategy for the logs; otherwise, normalization strategy 2 is compared with the logs.

[0006] However, if the amount of log data is large, using the above polling method to match the normalization strategy for logs with different writing formats will result in low matching efficiency of the log normalization strategy, which will reduce the efficiency of log analysis and processing. Summary of the Invention

[0007] This invention provides a log normalization strategy matching method, apparatus, electronic device, and storage medium to improve the matching efficiency of log normalization strategies. The specific technical solution is as follows:

[0008] Firstly, this application provides a log normalization strategy matching method, including:

[0009] Obtain the first set of original logs corresponding to the first sample attribute label, wherein the first sample attribute label represents the identifier corresponding to a set of generalization strategies;

[0010] From the first set of original logs, select each candidate original log containing the same phrase type to obtain the second set of original logs;

[0011] In the second set of original logs, select any candidate original log as the target log, and determine the first phrase contained in the target log. Match the first phrase with each sample log phrase in the set of sample log phrases, wherein one sample log phrase in the set of sample log phrases corresponds to a generalization strategy.

[0012] If there is a sample log phrase in the sample log phrase set that matches the first phrase, then obtain the first normalization strategy set corresponding to the first phrase. When it is determined that the first normalization strategy set contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy for the target log.

[0013] If there is no sample log phrase in the sample log phrase set that matches the first phrase, then a new normalization strategy is created for the target log.

[0014] Based on the above method, logs containing the same phrase type are grouped into the same cluster, enabling batch matching and generalization strategies for logs. By selecting any candidate original log in the same cluster as the target log, and performing phrase matching based on the phrases contained in the target log with each sample log phrase in the set of sample log phrases, the appropriate generalization strategy can be accurately matched for each candidate original log in the same cluster, improving the accuracy of matching between logs and generalization strategies and increasing the matching efficiency of log generalization strategies.

[0015] In one possible implementation, before obtaining the first raw log set corresponding to the first sample attribute label, the method further includes:

[0016] Retrieve the attribute tags corresponding to the original logs and the set sample attribute tag set;

[0017] Select one candidate sample attribute label from the set of sample attribute labels as the first sample attribute label;

[0018] Extract the first priority value and the first key-value logarithm from the attribute labels of the first sample;

[0019] Determine whether the original priority value and the original key-value logarithm in the original log match the first priority value and the first key-value logarithm.

[0020] If so, the original log is added to the first original log set corresponding to the first sample attribute label, and the next original log is polled.

[0021] If not, the original logs are added to the remaining original log sets corresponding to the remaining sample attribute labels in the sample attribute label set.

[0022] Based on the above method, it is possible to initially select the original logs that meet the priority value and key-value logarithm conditions corresponding to the attribute labels of each candidate sample. Furthermore, for some original logs that are bound with corresponding normalization strategy information, the normalization strategy corresponding to this type of log can be directly selected from the normalization strategy set, which can reduce the number of subsequent accurate matching operations for various types of original logs and reduce the resource overhead of the system.

[0023] In one possible implementation, the step of selecting candidate original logs containing the same phrase type from the first original log set to obtain a second original log set includes:

[0024] Determine the phrase type corresponding to each of the first original logs in the first original log set;

[0025] Cluster the first original logs in the first original log set according to phrase type;

[0026] Obtain a first set of candidate raw logs containing the first phrase type and a second set of candidate raw logs containing the second phrase type;

[0027] From the first candidate original log set and the second candidate original log set, a target log set that meets the set word frequency condition is selected, and the target log set is used as the second original log set.

[0028] Based on the above method, it is possible to classify various original logs containing the same phrase type into the same cluster, thereby realizing a batch matching and generalization strategy for logs.

[0029] In one possible implementation, determining the phrase type corresponding to each of the first original logs in the first original log set includes:

[0030] Select any one of the first original logs from the first set of original logs;

[0031] The text content in the first original log is divided into multiple word groups according to the set word group length M. Each candidate word group in the multiple word groups is associated with the previous candidate word group. M is a positive integer greater than or equal to 2.

[0032] The frequency of each of the multiple word groups in a set of words is counted to determine the most frequent word group among the multiple word groups. The set of words includes labeled word groups extracted from the text of various types of original logs.

[0033] The phrase type of the high-frequency phrases is used as the phrase type of the first original log.

[0034] Based on the above method, the word type of each first original log in the first original log set can be determined. The text content in the first original log is divided according to the set word length M, and M can be set to 2, 3, or 4. First, M is set to 2. Then, based on the actual matching results, it is determined whether to increment M. The precise matching steps are repeated to continue matching the optimal target normalization strategy for each first original log. This method can cluster original logs of the same type as much as possible and match the appropriate normalization strategy for each type of clustered original log, thereby improving the matching efficiency of log normalization strategy.

[0035] Secondly, this application provides a generalized strategy matching apparatus, comprising:

[0036] The data acquisition module is used to acquire the first original log set corresponding to the first sample attribute label, wherein the first sample attribute label represents the identifier corresponding to a set of generalization strategies;

[0037] The data filtering module is used to select each candidate original log containing the same phrase type from the first original log set to obtain the second original log set;

[0038] The generalization strategy matching module is used to select any candidate original log as the target log in the second original log set, determine the first phrase contained in the target log, and match the first phrase with each sample log phrase in the set of sample log phrases, wherein one sample log phrase in the set of sample log phrases corresponds to one generalization strategy.

[0039] If there is a sample log phrase in the sample log phrase set that matches the first phrase, then obtain the first normalization strategy set corresponding to the first phrase. When it is determined that the first normalization strategy set contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy for the target log.

[0040] If there is no sample log phrase in the sample log phrase set that matches the first phrase, then a new normalization strategy is created for the target log.

[0041] In one possible implementation, the data acquisition module is further configured to:

[0042] Retrieve the attribute tags corresponding to the original logs and the set sample attribute tag set;

[0043] Select one candidate sample attribute label from the set of sample attribute labels as the first sample attribute label;

[0044] Extract the first priority value and the first key-value logarithm from the attribute labels of the first sample;

[0045] Determine whether the original priority value and the original key-value logarithm in the original log match the first priority value and the first key-value logarithm.

[0046] If so, the original log is added to the first original log set corresponding to the first sample attribute label, and the next original log is polled.

[0047] If not, the original logs are added to the remaining original log sets corresponding to the remaining sample attribute labels in the sample attribute label set.

[0048] In one possible implementation, the data filtering module is specifically used for:

[0049] Determine the phrase type corresponding to each of the first original logs in the first original log set;

[0050] Cluster the first original logs in the first original log set according to phrase type;

[0051] Obtain a first set of candidate raw logs containing the first phrase type and a second set of candidate raw logs containing the second phrase type;

[0052] From the first candidate original log set and the second candidate original log set, a target log set that meets the set word frequency condition is selected, and the target log set is used as the second original log set.

[0053] In one possible implementation, the data filtering module is specifically used for:

[0054] Select any one of the first original logs from the first set of original logs;

[0055] The text content in the first original log is divided into multiple word groups according to the set word group length M. Each candidate word group in the multiple word groups is associated with the previous candidate word group. M is a positive integer greater than or equal to 2.

[0056] The frequency of each of the multiple word groups in a set of words is counted to determine the most frequent word group among the multiple word groups. The set of words includes labeled word groups extracted from the text of various types of original logs.

[0057] The phrase type of the high-frequency phrases is used as the phrase type of the first original log.

[0058] Thirdly, this application provides an electronic device, comprising:

[0059] Memory, used to store computer programs;

[0060] When a processor executes a computer program stored in the memory, it implements the steps of the log normalization strategy matching method described above.

[0061] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the log normalization strategy matching method described above.

[0062] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0063] Figure 1 A flowchart of a log normalization strategy matching method provided in this application;

[0064] Figure 2 This application provides a schematic diagram of the log normalization strategy matching system architecture.

[0065] Figure 3 The flowcharts for tag matching and exact matching provided for this application;

[0066] Figure 4 A schematic diagram of a log normalization strategy matching device provided in this application;

[0067] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0069] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0070] Logs are event records generated by network devices, systems, service programs, etc. during runtime, recording descriptions of related operations such as date, time, user, and actions.

[0071] When a network device is running, it generates specific log records according to a set log format. Because different network devices have different log formats, different types of log records are generated. Before analyzing these various logs, log parsing techniques are usually used to parse them and extract the valuable information contained within.

[0072] Currently, log parsing and normalization still rely on manual processing. In current log auditing systems, a large number of logs are accessed, written and recorded by different network devices according to their respective log writing formats. Typically, a log writing format corresponds to a specific normalization strategy, which refers to the parsing rules for logs. However, the total number of normalization strategies that can be created for parsing various types of logs is finite.

[0073] In existing technologies, to match suitable normalization strategies to logs with different writing formats, normalization strategies are used to parse the logs. The log auditing system selects a higher-priority normalization strategy from its set of normalization strategies and polls it against the logs according to a preset normalization strategy priority. For example, the log auditing system may have normalization strategy 1 and normalization strategy 2, with normalization strategy 1 having a higher priority than normalization strategy 2. Normalization strategy 1 is selected and compared with the logs to determine if the logs meet the log parsing rules set by normalization strategy 1. If so, normalization strategy 1 is used as the target normalization strategy for the logs; otherwise, normalization strategy 2 is compared with the logs.

[0074] However, if the amount of log data is large, using the above polling method to match the normalization strategy for logs with different writing formats will result in low matching efficiency of the log normalization strategy, which will reduce the efficiency of log analysis and processing.

[0075] In view of this, in order to quickly match suitable normalization strategies for logs of different formats when there are many types and large amounts of data, and to improve the matching efficiency between logs and normalization strategies, this application provides a log normalization strategy matching method, which specifically includes: firstly, obtaining a first set of original logs corresponding to the first sample attribute labels; then, selecting each candidate original log containing the same phrase type from the first set of original logs to obtain a second set of original logs; finally, selecting any candidate original log as the target log from the second set of original logs, and determining the first phrase contained in the target log, and matching the first phrase with each sample log phrase in the set of sample log phrases; if there is a sample log phrase in the set of sample log phrases that matches the first phrase, then obtaining a first set of normalization strategies corresponding to the first phrase; when it is determined that the first set of normalization strategies contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy for the target log; if there is no sample log phrase in the set of sample log phrases that matches the first phrase, then a new normalization strategy is created for the target log.

[0076] The method provided in this application clusters candidate original logs containing the same word type in the first original log set into the same cluster. It prioritizes matching the cluster of original logs that meets the set word frequency condition, ensuring that even with massive log volumes (many log types and large data volume), it prioritizes matching normalization strategies for logs containing high-frequency words. Then, it selects any candidate original log as the target log from the cluster containing high-frequency words. Based on the first word contained in the target log, it matches the first word with each sample log word in the set of sample log words to obtain the normalization strategy corresponding to the target log. This achieves accurate matching of the normalization strategy for any candidate original log in the cluster, avoiding polling and comparing each candidate original log in the original log set with each sample log in the normalization strategy set, thus improving the matching efficiency of log normalization strategies and the performance of log processing.

[0077] Reference Figure 1 The diagram shown is a flowchart of a log normalization strategy matching method provided in an embodiment of this application. The method includes:

[0078] S1, obtain the first set of original logs corresponding to the attribute label of the first sample.

[0079] Firstly, the method provided in this application can be applied to Figure 2 The system architecture shown includes a log generation device and a log auditing system. The method provided in this application can be run in the log auditing system.

[0080] Log generation devices can be security devices, network devices, servers, databases, application systems, hosts, etc. in an enterprise IT system. This application does not impose specific restrictions on the type or number of such log generation devices. The aforementioned devices can generate specific log records according to their respective set log formats. The logs include information such as device operation, alarms, operations, messages, and status, which will not be elaborated here.

[0081] The log auditing system is used to receive various types of logs generated by the above-mentioned devices, and to parse each type of log according to its specific log parsing rules (normalization strategy). After parsing various types of logs, audit analysis is performed to obtain valuable information from the logs. It should be noted that the log parsing rules can be stored in the memory of the log auditing system, and a normalization strategy corresponding to a certain type of log can be added to the memory according to the actual normalization strategy usage requirements.

[0082] In this embodiment of the application, in order to reduce the amount of data required for precise matching of the log normalization strategy, the log auditing system needs to perform preliminary screening of the original logs before obtaining the first set of original logs corresponding to the first sample attribute labels, that is, label matching between the original logs and the sample logs.

[0083] The log auditing system first obtains the attribute tags corresponding to the original logs and the set sample attribute tag set. The attribute tags corresponding to the original logs can be a type of tag that records the importance of the log, the storage location of the log, the type of the log, or any other attribute of the log.

[0084] The defined sample attribute label set refers to the subset of generalization strategies obtained by dividing the generalization strategies in the generalization strategy set according to the same attribute. It consists of attribute identifiers corresponding to each subset of generalization strategies. For example, generalization strategies containing the same priority value and / or number of key-value pairs can be grouped into a single subset, and corresponding sample attribute labels can be added to this subset. Furthermore, each generalization strategy within each subset can be set with a sample log that satisfies the current generalization strategy rules. The sample log can also be set with attribute labels similar to those of the original log, used to record information related to the sample log; this will not be elaborated further here.

[0085] Then, a candidate sample attribute label is selected from the sample attribute label set as the first sample attribute label. For example, candidate sample attribute label 1 from candidate sample attribute label 1 and candidate sample attribute label 2 is selected as the first sample attribute label. It should be noted that when selecting a candidate sample attribute label from the sample attribute label set, a subset of generalization strategies corresponding to that candidate sample attribute label can also be obtained. This application does not impose a specific limit on the number of generalization strategies in each subset of generalization strategies.

[0086] After obtaining the first sample attribute tag, the log auditing system can directly extract or extract the first priority value and the first key-value pair from the first sample attribute tag according to the set extraction rules. The first priority value records the priority of the sample log. In some embodiments, the first priority value Priority (PRI) can include two parts: Facility and Level. Facility is used to determine which program in the device generated the log information; Level is used to determine the severity level of the event. The calculation method for PRI can refer to the following formula:

[0087] PRI = Facility * 8 + Level

[0088] The first key-value log record contains the field information of the sample log, which can include two parts: key and value. The key represents the name of the log field, and the value represents the data of the stored log field.

[0089] It should be noted here that each sample log or original log has a specific priority value and number of key-value pairs, which will not be elaborated here.

[0090] The log auditing system determines whether the original priority value and the original key-value log in the original log match the first priority value and the first key-value log in the first sample log based on the original priority value, the original key-value log in the original log, and the first priority value and the first key-value log in the first sample attribute label (the label of any first sample log in the normalization strategy subset). If they match, the original log is added to the first original log set corresponding to the first sample attribute label, and the next original log is polled. If not, the original log is added to the remaining original log set corresponding to the remaining sample attribute labels in the sample attribute label set.

[0091] It should be noted here that the condition for determining whether the original priority value and the original key-value pair in the original log match the first priority value and the first key-value pair is that the original priority value is the same as the first priority value and / or the original key-value pair is the same as the first key-value pair.

[0092] The following example uses a raw log entry to detail the tag matching process between the raw log entry and the target log entry:

[0093] Suppose that the original priority value (PRI) in an original log is 15 and the number of original key-value pairs is 100; the number of candidate sample attribute tags in the sample attribute tag set is N, and there are candidate sample attribute tags in the sample attribute tag set that match the attribute tags corresponding to the original log, where N is a positive integer greater than or equal to 1.

[0094] If N=2, in the sample attribute label set, the first priority value (PRI) of the first sample attribute label is 15 and the number of first key-value pairs is 100; the second priority value (PRI) of the second sample attribute label is 10 and the number of second key-value pairs is 50. The log auditing system can determine that the original priority value and the number of original key-value pairs match the first priority value and the number of first key-value pairs. Then, the original log is added to the first initial screening log set, and the next original log is polled until all original logs that match the original priority value and the number of original key-value pairs are added to the first initial screening log set. Then, the first initial screening log set is used as the first original log set corresponding to the first sample attribute label.

[0095] If the first priority value (PRI) is 10 and the first key-value pair is 50; and the second priority value (PRI) in the second sample attribute label is 15 and the second key-value pair is 100, the log auditing system can determine that the original priority value and the original key-value pair fail to match the first priority value and the first key-value pair. In this case, the original log is added to the second initial screening log set, and the next original log is polled until all original logs that do not meet the requirement of matching the original priority value and the original key-value pair with the first priority value and the first key-value pair are added to the second initial screening log set. Then, the second initial screening log set is used as the remaining original log set corresponding to the second sample attribute label (remaining sample attribute label).

[0096] In one possible embodiment, if N=2, and there are no candidate sample attribute tags in the sample attribute tag set that match the attribute tags corresponding to the original log, for example, in the sample attribute tag set, the first priority value (PRI) of the first sample attribute tag is 16 and the first key-value pair is 80; the second priority value (PRI) of the second sample attribute tag is 10 and the second key-value pair is 50; the log auditing system can determine that the original priority value and the original key-value pair fail to match the first priority value and the first key-value pair. Before adding the original log to the second initial screening log set, it can also determine whether the original priority value and the original key-value pair match the second priority value and the second key-value pair corresponding to the second sample attribute tag. When it is determined that the original priority value and the original key-value pair fail to match the second priority value and the second key-value pair, the current original log is skipped, the next original log is selected, and the tag matching is repeated according to the above process until the corresponding original log is added to the first initial screening log set and the second initial screening log set, and the first original log set corresponding to the first sample attribute tag and the remaining original log set corresponding to the second sample attribute tag (remaining sample attribute tag) are obtained.

[0097] The original log sets corresponding to the attribute labels of each candidate sample are shown in Table 1 below:

[0098]

[0099] Table 1

[0100] It should be noted that the number of candidate sample attribute labels in the sample attribute label set can be added adaptively according to the business needs of the log auditing system. This application does not impose a specific limit on the number of candidate sample attribute labels, and will not elaborate further here.

[0101] By using the above method, we can initially select raw logs from a large number of raw logs that meet the priority value and key-value logarithm conditions corresponding to the attribute labels of each candidate sample. This reduces the number of subsequent precise matches on various types of raw logs, increases the speed of matching generalization strategies for various types of raw logs, and reduces the overhead of system resources.

[0102] S2, select each candidate original log containing the same phrase type from the first original log set to obtain the second original log set.

[0103] In this embodiment, after obtaining the original log sets corresponding to each candidate sample attribute label, the log auditing system can perform subsequent precise matching on the original log sets corresponding to each candidate sample attribute label according to the priority value of the candidate sample attribute label. For example, the first original log set corresponding to the first sample attribute label can be selected for priority processing.

[0104] The process of performing exact matching on each of the first original logs in the first original log set is as follows:

[0105] The log auditing system first determines the term type corresponding to each of the first original logs in the first original log set. The specific method for determining the term type corresponding to each of the first original logs is as follows:

[0106] The log auditing system selects any one of the first original logs from the first set of original logs and divides the text content of the original log into multiple phrases according to a set phrase length M, where M is a positive integer greater than or equal to 2. For example, if the text content in the original log contains Hostname, Version, and app-name, and M=2 is selected, then phrase 1 is: Hostname->Version; phrase 2 is: Version->app-name; and phrase 3 is: app-name->Hostname. It should be noted that any candidate phrase in the multiple phrases is associated with the previous candidate phrase.

[0107] Log auditing systems can use N-Gram dictionaries to statistically analyze phrases in logs, counting the frequency of each phrase within a defined phrase set, and identifying the most frequent phrase. The defined phrase set includes tagged phrases extracted from the text of various raw logs. This set is similar to the corpus in an N-Gram dictionary and will not be elaborated upon here.

[0108] After identifying the most frequent phrases among multiple phrases, the log auditing system can use the phrase type of the most frequent phrases as the phrase type of the first original log.

[0109] After determining the phrase type of a first original log in the first original log set, the log auditing system can select the next first original log in the first original log set and repeat the above steps until the phrase type corresponding to each first original log in the first original log set is determined.

[0110] In this embodiment of the application, after determining the word type corresponding to each first original log, the log auditing system can also cluster according to the word type of each first original log in the first original log set to obtain a first candidate original log set containing the first word type and a second candidate original log set containing the second word type.

[0111] The resulting first and second candidate raw log sets are shown in Table 2:

[0112]

[0113] Table 2

[0114] As shown in Table 2, by clustering each of the first original logs in the first original log set, candidate original logs containing the same word group type can be classified into the same cluster, realizing a batch matching and generalization strategy for logs. It should be noted that this application does not impose a specific limit on the number of the same clusters obtained after clustering each of the first original logs. The invention content of this application is explained only using the first and second candidate original log sets as examples.

[0115] After clustering the various original logs in the first original log set, the log auditing system can select a target log set that meets the set of frequency conditions for phrase occurrence from the first and second candidate original log sets, and use this target log set as the second original log set. For example, if the first phrase type (hostname and version) occurs 100 times and the second phrase type (app-name and msgid) occurs 50 times, then the first candidate original log set that meets the set of frequency conditions for phrase occurrence (phrase occurrence frequency greater than 50 times) can be selected as the target log set, and precise matching will be prioritized for the target log set.

[0116] By selecting a target log set that meets the frequency criteria for a given phrase (i.e., the target log set is relatively common in the sample log set), and prioritizing precise matching of this target log set, the system can quickly select the generalization strategy corresponding to the phrase from the set of generalization strategies, thereby improving the log generalization strategy matching speed.

[0117] S3, select any candidate original log as the target log in the second original log set, determine the first phrase contained in the target log, and match the first phrase with each sample log phrase in the set of sample log phrases;

[0118] In this embodiment of the application, after determining that the first set of candidate original logs containing the first phrase type will be used as the second set of original logs, the log auditing system can select any one of the candidate original logs from the second set of original logs as the target log and determine the first phrase contained in the target log. For example, the first phrase of the target log is hostname and version, and then the first phrase is matched with each sample log phrase in the set of sample log phrases.

[0119] The set of sample log phrases can contain phrases consisting of any N words from priority, version, timestamp, hostname, app-name, procid, msgid, and structured-data, i.e., each sample log phrase. Each sample log phrase corresponds to a specific normalization strategy, where N is a positive integer greater than or equal to 2.

[0120] If there is a sample log phrase in the sample log phrase set that matches the first phrase, then proceed to step S4; if there is no sample log phrase in the sample log phrase set that matches the first phrase, then proceed to step S5.

[0121] S4. Obtain the first normalization strategy set corresponding to the first phrase. When it is determined that the first normalization strategy set contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy of the target log.

[0122] In this embodiment of the application, if the sample log phrase set includes sample log phrase 1: hostname and version; sample log phrase 2: version and priority; sample log phrase 3: hostname and priority.

[0123] When the first phrase of the target log is determined to be "hostname" and "version," meaning there exists a sample log phrase "hostname" and "version" in the sample log set that matches the first phrase, the set of generalization strategies corresponding to the first phrase can be obtained based on the mapping relationship between the sample log phrases and the generalization strategies set for the corresponding sample log phrases. The mapping relationship between the sample log phrases and the generalization strategies set for the corresponding sample log phrases is shown in Table 3 below:

[0124] hostname and version Generalization Strategy 1 version and priority Generalization Strategy 2 hostname and priority Generalization Strategy 3 … …

[0125] Table 3

[0126] As shown in Table 3, the first normalization strategy set corresponding to the first phrase includes normalization strategy 1, and it can be determined that the first normalization strategy set contains only one candidate normalization strategy. The log auditing system will then use normalization strategy 1 as the target normalization strategy for the target log.

[0127] S5, create a new normalization strategy for the target log.

[0128] In this embodiment of the application, if the sample log phrase set includes sample log phrase 1: hostname and app-name; sample log phrase 2: version and priority; sample log phrase 3: hostname and priority.

[0129] The mapping relationship between sample log phrases and the corresponding normalization strategies set for the sample log phrases is as follows:

[0130] As shown in Table 4:

[0131]

[0132]

[0133] Table 4

[0134] As shown in Table 4, if there is no sample log phrase in the sample log phrase set that matches the first phrase "hostname" and "version" of the target log, then the log auditing system will create a new normalization strategy for the target log that contains the first phrase type.

[0135] In one possible implementation, after determining that a sample log phrase matching the first phrase exists in the sample log phrase set and obtaining the first normalization strategy set corresponding to the first phrase, if the number of candidate normalization strategies in the first normalization strategy set is greater than 1, i.e., the first phrase of the target log is hostname and version; the sample log phrase set includes sample log phrase 1: hostname and version; sample log phrase 2: hostname, version, and priority; sample log phrase 3: hostname, version, and procid. Then, the text content in the target log of this type is divided according to the set phrase length M+1, and the above steps of accurately matching the first original log are repeated to match a normalization strategy for the target log of this type. The specific method is as follows:

[0136] Select any first target log from the target log set, and divide the text content in the first target log into multiple word groups according to the set word group length M+1. Each candidate word group is associated with the previous candidate word group. Count the number of times each word group appears in the set word group set, determine the most frequent word group, and use the word group type of the most frequent word group as the word group type of the first target log.

[0137] Next, the word type corresponding to each first target log in the first target log set is determined, and each first target log in the first target log set is clustered according to word type to obtain a third candidate original log set containing the third word type and a fourth candidate original log set containing the fourth word type; from the third candidate original log set and the fourth candidate original log set, the target log set that meets the set word frequency condition is selected, and the target log set is used as the second original log set.

[0138] Finally, select any candidate original log from the second set of original logs as the target log, and determine the third phrase contained in the target log. Match the third phrase with each sample log phrase in the set of sample log phrases. If there is a sample log phrase in the set of sample log phrases that matches the third phrase, obtain the first normalization strategy set corresponding to the third phrase. If the first normalization strategy set contains only one candidate normalization strategy, use the candidate normalization strategy as the target normalization strategy for the target log. If there is no sample log phrase in the set of sample log phrases that matches the third phrase, create a new normalization strategy for the target log.

[0139] By increasing the length of the phrases used to divide the text content in the target log, when multiple sample log phrases in the sample log phrase set match the target log phrases, the optimal sample log phrase that matches the target log phrases can be further filtered out, thus accurately matching the corresponding normalization strategy for each target log.

[0140] It should be noted that if the word length for dividing the text content in the target log is increased once, and after iterating through the above steps for precise matching of the first original log, there are still multiple sample log word groups in the sample log word group set that match the word group of the target log, then the text content in the target log of this type is divided according to the set word length M+2, and the above steps for precise matching of the first original log are iterated through again, until it is determined that the normalization strategy set corresponding to the target log contains only one candidate normalization strategy, then the candidate normalization strategy is used as the target normalization strategy of the target log.

[0141] The following is an appendix Figure 3 The method of the present invention will be described in detail using an example:

[0142] (1) Label Matching: Obtain a large amount of raw logs. Perform initial screening on the raw logs based on their priority value (PRI) and key-value pairs to obtain a set of raw logs corresponding to each candidate sample attribute label and a subset of generalization strategies corresponding to each candidate sample attribute label. It should be noted that during label matching, some raw logs may contain information about generalization strategies corresponding to the raw logs. The generalization strategies corresponding to each bound raw log can be directly selected from the set of generalization strategies.

[0143] (2) Precise matching: From the original log sets corresponding to the attribute labels of each candidate sample, select the first original log set corresponding to the first sample attribute label with the higher priority value (and determine a subset of generalization strategies corresponding to the first sample attribute label, and select a generalization strategy from the subset of generalization strategies to perform precise matching with the first original log in the first original log set), and prioritize matching the first original log set; determine the word type of each first original log in the first original log set (for any first original log in the first original log set, divide the text content in the first original log according to the set word length M, divide it into multiple word groups, determine the most frequent word group among the multiple word groups, and take the word type of the high-frequency word group as the word type of the first original log); extract the high-frequency word groups of each first original log in the first original log set.

[0144] Clustering is performed on the word types of each original log in the first original log set to obtain each candidate original log set containing the same word type, namely, the first candidate original log set containing the first word type and the second candidate original log set containing the second word type.

[0145] From each candidate original log set, select the target log set that meets the set word frequency condition, and use the target log set as the second original log set; select any candidate original log as the target log from the second original log set, determine the first word contained in the target log, and match the first word with each sample log word in the set of sample log word sets.

[0146] If a sample log phrase set contains a sample log phrase that matches the first phrase, then the first normalization strategy set corresponding to the first phrase is obtained. If it is determined that the first normalization strategy set contains only one candidate normalization strategy, then the candidate normalization strategy is used as the target normalization strategy for the target log. If it is determined that the number of candidate normalization strategies in the first normalization strategy set is greater than 1, then the text content in the target log of this type is divided according to the set phrase length M+1, and the above steps of accurately matching the first original log are repeated to match a normalization strategy for the target log of this type. This application does not impose a specific limit on the number of times the text content in the target log is divided according to the set phrase length, and normalization strategies can be matched for the target log in an incremental manner according to the phrase length.

[0147] If there is no sample log phrase in the sample log phrase set that matches the first phrase, then a new normalization strategy is created for the target log.

[0148] In summary, the log normalization strategy matching method provided in this application first performs label matching on a large number of raw logs, which can initially select raw logs that meet the priority value and key-value logarithm conditions corresponding to the attribute labels of each candidate sample. Furthermore, for some raw logs that are bound with corresponding normalization strategy information, the normalization strategy corresponding to this type of log can be directly selected from the normalization strategy set, which can reduce the number of subsequent accurate matches for various types of raw logs and reduce the resource overhead of the system.

[0149] In precise matching, the text content of the first original log is first divided according to the phrase length M (here, M is 2), and the high-frequency phrases of each first original log are determined. Then, the first original logs containing the same phrase type are grouped into the same cluster, which enables batch matching and generalization strategies for logs. Finally, any candidate original log in the same cluster (the second set of original logs) is selected as the target log. Based on the phrases contained in the target log, phrase matching is performed with each sample log phrase in the set of sample log phrases. This can accurately match each candidate original log in the same cluster with its appropriate generalization strategy. Furthermore, when matching generalization strategies for target logs with high similarity in phrase type, the phrase length M can be increased sequentially, and the above precise matching steps can be repeated to match the optimal generalization strategy for each target log. This improves the accuracy of matching between logs and generalization strategies and increases the matching efficiency of log generalization strategies.

[0150] Based on the methods provided in the above embodiments, this application also provides a log normalization strategy matching device, such as... Figure 4 The diagram shown is a structural schematic of a log normalization strategy matching device according to an embodiment of this application. The device includes:

[0151] Data acquisition module 401 is used to acquire the first original log set corresponding to the first sample attribute label, wherein the first sample attribute label represents the identifier corresponding to a set of generalization strategies;

[0152] The data filtering module 402 is used to select each candidate original log containing the same word group type from the first original log set to obtain a second original log set;

[0153] The generalization strategy matching module 403 is used to select any candidate original log as the target log in the second original log set, and determine the first phrase contained in the target log, and match the first phrase with each sample log phrase in the set of sample log phrases, wherein one sample log phrase in the set of sample log phrases corresponds to one generalization strategy.

[0154] If there is a sample log phrase in the sample log phrase set that matches the first phrase, then obtain the first normalization strategy set corresponding to the first phrase. When it is determined that the first normalization strategy set contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy for the target log.

[0155] If there is no sample log phrase in the sample log phrase set that matches the first phrase, then a new normalization strategy is created for the target log.

[0156] In one possible implementation, the data acquisition module 401 is further configured to:

[0157] Retrieve the attribute tags corresponding to the original logs and the set sample attribute tag set;

[0158] Select one candidate sample attribute label from the set of sample attribute labels as the first sample attribute label;

[0159] Extract the first priority value and the first key-value logarithm from the attribute labels of the first sample;

[0160] Determine whether the original priority value and the original key-value logarithm in the original log match the first priority value and the first key-value logarithm.

[0161] If so, the original log is added to the first original log set corresponding to the first sample attribute label, and the next original log is polled.

[0162] If not, the original logs are added to the remaining original log sets corresponding to the remaining sample attribute labels in the sample attribute label set.

[0163] In one possible implementation, the data filtering module 402 is specifically used for:

[0164] Determine the phrase type corresponding to each of the first original logs in the first original log set;

[0165] Cluster the first original logs in the first original log set according to phrase type;

[0166] Obtain a first set of candidate raw logs containing the first phrase type and a second set of candidate raw logs containing the second phrase type;

[0167] From the first candidate original log set and the second candidate original log set, a target log set that meets the set word frequency condition is selected, and the target log set is used as the second original log set.

[0168] In one possible implementation, the data filtering module 402 is specifically used for:

[0169] Select any one of the first original logs from the first set of original logs;

[0170] The text content in the first original log is divided into multiple word groups according to the set word group length M. Each candidate word group in the multiple word groups is associated with the previous candidate word group. M is a positive integer greater than or equal to 2.

[0171] The frequency of each of the multiple word groups in a set of words is counted to determine the most frequent word group among the multiple word groups. The set of words includes labeled word groups extracted from the text of various types of original logs.

[0172] The phrase type of the high-frequency phrases is used as the phrase type of the first original log.

[0173] Based on the same inventive concept, this application also provides an electronic device that can realize the function of the aforementioned log normalization strategy matching device. (Refer to...) Figure 5 The electronic device includes:

[0174] At least one processor 501 and a memory 502 connected to at least one processor 501. In this embodiment, the specific connection medium between the processor 501 and the memory 502 is not limited. Figure 5 The example shown is the connection between processor 501 and memory 502 via bus 500. Bus 500 is... Figure 5The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The Bus 500 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 5 The term 501 is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 501 can also be called a controller; there is no restriction on the name.

[0175] In this embodiment, memory 502 stores instructions executable by at least one processor 501. By executing the instructions stored in memory 502, at least one processor 501 can execute the log normalization strategy matching method discussed above. Processor 501 can implement... Figure 4 The functions of each module in the device shown.

[0176] The processor 501 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 502 and calling data stored in memory 502, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0177] In one possible design, processor 501 may include one or more processing units. Processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 501. In some embodiments, processor 501 and memory 502 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0178] Processor 501 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the log normalization strategy matching method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor.

[0179] Memory 502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 502 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 502 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 502 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0180] By designing and programming the processor 501, the code corresponding to the log normalization strategy matching method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute it during runtime. Figure 1 The steps of the log normalization strategy matching method in the illustrated embodiment are described below. How to design and program the processor 501 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0181] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the log normalization strategy matching method described above.

[0182] In some possible implementations, various aspects of the log normalization strategy matching method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the log normalization strategy matching method according to various exemplary embodiments of this application described above.

[0183] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0187] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A log normalization strategy matching method, characterized in that, The method includes: Obtain the first set of original logs corresponding to the first sample attribute label, wherein the first sample attribute label represents the identifier corresponding to a set of generalization strategies; From the first set of original logs, select each candidate original log containing the same phrase type to obtain the second set of original logs; In the second set of original logs, select any candidate original log as the target log, and determine the first phrase contained in the target log. Match the first phrase with each sample log phrase in the set of sample log phrases, wherein one sample log phrase in the set of sample log phrases corresponds to a generalization strategy. If there is a sample log phrase in the sample log phrase set that matches the first phrase, then obtain the first normalization strategy set corresponding to the first phrase. When it is determined that the first normalization strategy set contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy for the target log. If there is no sample log phrase in the sample log phrase set that matches the first phrase, then a new normalization strategy is created for the target log.

2. The method as described in claim 1, characterized in that, Before obtaining the first set of original logs corresponding to the first sample attribute label, the method further includes: Retrieve the attribute tags corresponding to the original logs and the set sample attribute tag set; Select one candidate sample attribute label from the set of sample attribute labels as the first sample attribute label; Extract the first priority value and the first key-value logarithm from the attribute labels of the first sample; Determine whether the original priority value and the original key-value logarithm in the original log match the first priority value and the first key-value logarithm. If so, the original log is added to the first original log set corresponding to the first sample attribute label, and the next original log is polled. If not, the original logs are added to the remaining original log sets corresponding to the remaining sample attribute labels in the sample attribute label set.

3. The method as described in claim 1, characterized in that, The step of selecting candidate original logs containing the same phrase type from the first original log set to obtain the second original log set includes: Determine the phrase type corresponding to each of the first original logs in the first original log set; Cluster the first original logs in the first original log set according to phrase type; Obtain a first set of candidate raw logs containing the first phrase type and a second set of candidate raw logs containing the second phrase type; From the first candidate original log set and the second candidate original log set, a target log set that meets the set word frequency condition is selected, and the target log set is used as the second original log set.

4. The method as described in claim 3, characterized in that, Determining the phrase type corresponding to each of the first original logs in the first original log set includes: Select any one of the first original logs from the first set of original logs; The text content in the first original log is divided into multiple word groups according to the set word group length M. Each candidate word group in the multiple word groups is associated with the previous candidate word group. M is a positive integer greater than or equal to 2. The frequency of each of the multiple word groups in a set of words is counted to determine the most frequent word group among the multiple word groups. The set of words includes labeled word groups extracted from the text of various types of original logs. The phrase type of the high-frequency phrases is used as the phrase type of the first original log.

5. A log normalization strategy matching device, characterized in that, The device includes: The data acquisition module is used to acquire the first original log set corresponding to the first sample attribute label, wherein the first sample attribute label represents the identifier corresponding to a set of generalization strategies; The data filtering module is used to select each candidate original log containing the same phrase type from the first original log set to obtain the second original log set; The generalization strategy matching module is used to select any candidate original log as the target log in the second original log set, determine the first phrase contained in the target log, and match the first phrase with each sample log phrase in the set of sample log phrases, wherein one sample log phrase in the set of sample log phrases corresponds to one generalization strategy. If there is a sample log phrase in the sample log phrase set that matches the first phrase, then obtain the first normalization strategy set corresponding to the first phrase. When it is determined that the first normalization strategy set contains only one candidate normalization strategy, the candidate normalization strategy is used as the target normalization strategy for the target log. If there is no sample log phrase in the sample log phrase set that matches the first phrase, then a new normalization strategy is created for the target log.

6. The apparatus as claimed in claim 5, characterized in that, The data acquisition module is also used for: Retrieve the attribute tags corresponding to the original logs and the set sample attribute tag set; Select one candidate sample attribute label from the set of sample attribute labels as the first sample attribute label; Extract the first priority value and the first key-value logarithm from the attribute labels of the first sample; Determine whether the original priority value and the original key-value logarithm in the original log match the first priority value and the first key-value logarithm. If so, the original log is added to the first original log set corresponding to the first sample attribute label, and the next original log is polled. If not, the original logs are added to the remaining original log sets corresponding to the remaining sample attribute labels in the sample attribute label set.

7. The apparatus as claimed in claim 5, characterized in that, The data filtering module is specifically used for: Determine the phrase type corresponding to each of the first original logs in the first original log set; Cluster the first original logs in the first original log set according to phrase type; Obtain a first set of candidate raw logs containing the first phrase type and a second set of candidate raw logs containing the second phrase type; From the first candidate original log set and the second candidate original log set, a target log set that meets the set word frequency condition is selected, and the target log set is used as the second original log set.

8. The apparatus as claimed in claim 5, characterized in that, The data filtering module is specifically used for: Select any one of the first original logs from the first set of original logs; The text content in the first original log is divided into multiple word groups according to the set word group length M. Each candidate word group in the multiple word groups is associated with the previous candidate word group. M is a positive integer greater than or equal to 2. The frequency of each of the multiple word groups in a set of words is counted to determine the most frequent word group among the multiple word groups. The set of words includes labeled word groups extracted from the text of various types of original logs. The phrase type of the high-frequency phrases is used as the phrase type of the first original log.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method steps of any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-4.

Images