Log grouping method, device and electronic device

Abstract

The present application relates to a method, device and electronic device for log grouping, which is used to solve the problem that existing log grouping is not suitable for real log analysis scenarios, resulting in inaccurate log analysis results. The method includes determining a first target field sequence and a second target sequence corresponding to the log to be grouped, the first target field sequence includes at least one numerical target field, and the second target sequence includes at least one text target field, and then calculating the first distance between the first target field sequence and the first reference field sequence corresponding to the first log grouping, and the second distance between the second target field sequence and the second reference field sequence corresponding to the first log grouping. If the sum of the first distance and the second distance is less than or equal to a preset threshold, the log to be grouped is added to the first log grouping. Based on the above method, it can help to improve the accuracy of log analysis results.

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a log grouping method, device, and electronic device. Background Art

[0002] In network security operations and maintenance, operations and maintenance personnel generally discover security risks such as network attacks and vulnerability risks by analyzing log operations, and then adopt corresponding security strategies to protect the discovered security risks.

[0003] However, faced with massive log volumes, operations and maintenance personnel struggle to complete manual analysis in a short period of time. Existing techniques typically employ log text clustering, grouping logs with similar text data into the same group based on the similarity of the text data within the logs. This grouping is then analyzed to improve the efficiency of manual analysis.

[0004] However, logs are typically heterogeneous data. In other words, they can contain both textual and other types of data. Therefore, existing techniques for grouping logs based on textual data only consider a single type of data. This results in inaccurate groupings that are not suitable for real-world log analysis scenarios. Summary of the Invention

[0005] The present application provides a log grouping method, device, and electronic device for performing portrait clustering on massive portrait data.

[0006] In a first aspect, the present application provides a method for log grouping, the method comprising:

[0007] Determine a first target field sequence and a second target sequence corresponding to the logs to be grouped, wherein the first target field sequence includes at least one numerical target field, and the second target sequence includes at least one text target field;

[0008] Calculating a first distance between the first target field sequence and a first reference field sequence corresponding to the first log group, where the first reference field sequence includes at least one numerical reference field;

[0009] Calculating a second distance between the second target field sequence and a second reference field sequence corresponding to the first log group, where the second reference field sequence includes at least one text reference field;

[0010] If the sum of the first distance and the second distance is less than or equal to a preset threshold, the log to be grouped is added to the first log group.

[0011] By using the above method, grouping logs based on the different types of fields in the logs can effectively improve the accuracy of the grouping results, thereby helping to improve the efficiency of subsequent log analysis and the accuracy of the analysis results. On the one hand, calculating the distances between numerical fields and text fields separately can be more in line with actual application scenarios. It is further proposed to dynamically adjust the weights in the distance calculation, which helps to obtain more accurate log groupings and has a better interpretation of the calculated distances. On the other hand, the first reference field sequence and the second reference field sequence are used to represent the numerical fields and text fields of the logs in the first log group, respectively. In this way, the processing performance in the online processing scenario is effectively improved, saving computing time and computing resources.

[0012] In one possible design, determining the first target field sequence and the second target field sequence corresponding to the logs to be grouped includes: determining the first numerical field corresponding to each classification field in the logs to be grouped based on the mapping relationship between the classification field and the numerical field; using each numerical field in the logs to be grouped and each determined first numerical field as the numerical target field constituting the first target field sequence; and using each text field in the logs to be grouped as the text target field constituting the second target field sequence.

[0013] Through the above method, considering the type of categorical fields in logs, by mapping categorical fields to numerical fields, the types of fields that need to calculate distances in logs can be effectively reduced. Mapping them to numerical fields also helps improve the computing time and efficiency consumed in subsequent distance calculations.

[0014] In one possible design, the calculation of the first distance between the first target field sequence and the first reference field sequence corresponding to the first log group includes: determining the attributes corresponding to each numerical target field in the first target field sequence, and the attributes corresponding to each numerical reference field in the first reference field sequence corresponding to the first log group; calculating the field distance between the numerical target field and the numerical reference field corresponding to the same attribute, and obtaining multiple field distances between each numerical target field and each numerical reference field; and using the sum of the multiple field distances as the first distance between the first target field sequence and the second reference field sequence.

[0015] Through the above method, the first distance is obtained by calculating the field distance between the numerical target field and the numerical reference field corresponding to the same attribute one by one. The first distance calculated in this way is not only more in line with the actual application scenario, but also has better explanatory significance.

[0016] In one possible design, the calculation of the second distance between the second target field sequence and the second reference field sequence corresponding to the first log group includes: determining the attributes corresponding to each text-class target field in the second target field sequence, and the attributes corresponding to each text-class reference field in the second reference field sequence corresponding to the first log group; calculating the minimum edit distance between the text-class target field and the text-class reference field corresponding to the same attribute, and obtaining multiple minimum edit distances between the text-class target field and the text-class reference field; and performing weighted summation on the multiple minimum edit distances to obtain the second distance between the second target field sequence and the second reference field sequence.

[0017] Through the above method, in some scenarios, there may be multiple text-type reference fields corresponding to the same attribute. Multiple edit distances between the text-type target field and the text-type reference field corresponding to the same attribute can be calculated. The minimum edit distance is selected from these multiple edit distances. In this way, the minimum edit distances corresponding to different attributes are calculated one by one, and these minimum edit distances are weighted and summed to obtain the second distance, which can be more in line with actual application scenarios.

[0018] In one possible design, after adding the log to be grouped to the first log group if the sum of the first distance and the second distance is less than or equal to a preset threshold, the method further includes: if the sum of the first distance and the second distance is greater than the preset threshold and there are no other log groups, creating a second log group and adding the log to be grouped to the second log group.

[0019] The above method can be applied to online scenarios, that is, adaptive dynamic grouping can be achieved, and further, logs can be added to log groups that meet the conditions.

[0020] In one possible design, adding the log to be grouped to the first log group includes: determining a first tag corresponding to the first log group, and adding the first tag to the log to be grouped.

[0021] The above method can be applied to online scenarios, realize adaptive dynamic grouping, add logs to log groups that meet the conditions, and add the corresponding tags of these groups to the logs, thereby establishing the correlation between the log groups, making it more suitable for actual application scenarios, especially some complex log analysis scenarios.

[0022] In one possible design, after adding the first tag to the log to be grouped, it also includes: in response to removing the first tag of the log to be grouped in the first log group, respectively calculating the sum of the first distance and the second distance between each log in the first log group and the first reference field sequence and the second reference sequence; in other logs in the first log group, determining the associated log corresponding to the calculated sum value greater than the calculated sum value of the log to be grouped, and removing the first tag of the associated log, the other log being the log other than the log to be grouped in the first log group.

[0023] When deleting a log tag using the above method, the other logs in the log group corresponding to the tag are also identified. By calculating the distance between each field of the other logs and the reference field corresponding to the log group, the tags corresponding to the logs whose calculated distance is greater than the distance between the log and the reference field are also deleted. This method for deleting log tags, based on the tags added to the logs, can achieve the association of logs in different groups, as well as the association of logs in the same group. That is, when deleting a log, its corresponding tag can be deleted, and other similar logs based on this tag can also be deleted accordingly. This will help improve the efficiency of log analysis and the accuracy of analysis results.

[0024] In a second aspect, the present application provides a log grouping device, the device comprising:

[0025] A determination module determines a first target field sequence and a second target sequence corresponding to the logs to be grouped, wherein the first target field sequence includes at least one numerical target field, and the second target sequence includes at least one text target field;

[0026] A first calculation module is configured to calculate a first distance between the first target field sequence and a first reference field sequence corresponding to the first log group, where the first reference field sequence includes at least one numerical reference field;

[0027] a second calculation module, calculating a second distance between the second target field sequence and a second reference field sequence corresponding to the first log group, where the second reference field sequence includes at least one text reference field;

[0028] The grouping module adds the log to be grouped into the first log group if the sum of the first distance and the second distance is less than or equal to a preset threshold.

[0029] In one possible design, the determination module is specifically used to determine the first numerical class field corresponding to each classification field in the log to be grouped based on the mapping relationship between the classification field and the numerical class field; use each numerical class field in the log to be grouped and each determined first numerical class field as the numerical class target field constituting the first target field sequence; use each text class field in the log to be grouped as the text class target field constituting the second target field sequence.

[0030] In one possible design, the first calculation module is specifically used to determine the attributes corresponding to each numerical target field in the first target field sequence, and the attributes corresponding to each numerical reference field in the first reference field sequence corresponding to the first log group; calculate the field distances between the numerical target fields and the numerical reference fields corresponding to the same attributes, and obtain multiple field distances between the each numerical target field and the each numerical reference field; and use the sum of the multiple field distances as the first distance between the first target field sequence and the second reference field sequence.

[0031] In one possible design, the second calculation module is specifically used to determine the attributes corresponding to each text-class target field in the second target field sequence, and the attributes corresponding to each text-class reference field in the second reference field sequence corresponding to the first log group; calculate the minimum edit distance between the text-class target field and the text-class reference field corresponding to the same attribute, and obtain multiple minimum edit distances between the each text-class target field and the each text-class reference field; perform weighted summation on the multiple minimum edit distances to obtain the second distance between the second target field sequence and the second reference field sequence.

[0032] In one possible design, the grouping module further includes: if the sum of the first distance and the second distance is greater than the preset threshold and there are no other log groups, creating a second log group and adding the log to be grouped to the second log group.

[0033] In one possible design, the grouping module is specifically configured to determine a first tag corresponding to the first log group and add the first tag to the log to be grouped.

[0034] In one possible design, the grouping module further includes: in response to removing the first label of the log to be grouped in the first log group, respectively calculating the sum of the first distance and the second distance between each log in the first log group and the first reference field sequence and the second reference sequence; among other logs in the first log group, determining the associated log corresponding to the calculated sum value greater than the calculated sum value of the log to be grouped, and removing the first label of the associated log, the other logs being logs other than the log to be grouped in the first log group.

[0035] In a third aspect, the present application provides an electronic device, comprising:

[0036] Memory for storing computer programs;

[0037] The processor is configured to implement the above-mentioned log grouping method steps when executing the computer program stored in the memory.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned method steps for log grouping.

[0039] For each of the above-mentioned aspects from the second to the fourth aspects and the technical effects that may be achieved by each of the aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible solutions in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of a log grouping method provided by this application;

[0041] Figure 2 A schematic diagram of an exemplary log provided for this application;

[0042] Figure 3 A flowchart of online processing of log data provided by this application;

[0043] Figure 4 A schematic diagram of a label removal method provided for this application;

[0044] Figure 5 A flowchart for removing labels provided for this application;

[0045] Figure 6 A schematic diagram of a log grouping device provided by this application;

[0046] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the application will be further described in detail below with reference to the accompanying drawings. The specific operation methods in the method embodiments can also be applied to the device embodiments or system embodiments.

[0048] In the description of this application, "multiple" is understood to mean "at least two." "And / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A and B are connected, which can mean: A and B are directly connected, and A and B are connected through C. In addition, in the description of this application, words such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be understood as indicating or implying relative importance or order.

[0049] The embodiments of the present application provide a log grouping method, apparatus, and electronic device to improve the accuracy of log grouping.

[0050] According to the method provided in the embodiment of the present application, first determine the first target field sequence and the second target sequence corresponding to the log to be grouped, where the first target field sequence includes at least one numerical target field, and the second target sequence includes at least one text target field. Then calculate the first distance between the first target field sequence and the first reference field sequence corresponding to the first log group, and the second distance between the second target field sequence and the second reference field sequence corresponding to the first log group. If the sum of the first distance and the second distance is less than or equal to the preset threshold, the log to be grouped is added to the first log group. Through the above method, the logs are grouped in combination with the numerical fields and text fields of the logs, and the logs in the obtained log group have a higher correlation, which helps to improve the accuracy of the log analysis results.

[0051] Furthermore, after adding the logs to be grouped to the first log group, the first tag corresponding to the first log group can also be added to the logs to be grouped. In this way, all logs in the same log group can be added with the same tag, that is, the correlation of the logs in the same log group is achieved. In addition, a log can belong to multiple groups. In this case, the log will have multiple tags corresponding to it, that is, this tagging method can also achieve the correlation of logs in different log groups.

[0052] Based on the addition of the above tags, you can also delete a tag of the log. That is, when deleting a log, you can delete its corresponding tag, and based on this tag, other similar logs can also be deleted accordingly. This will help improve the efficiency of log analysis and the accuracy of analysis results.

[0053] It is worth noting that the technical features included in the embodiments of the present application can be used in any combination. Those skilled in the art should understand that, based on actual application conditions, the technical solutions obtained by reasonably combining the technical features in the embodiments of the present application can also solve the same technical problems or achieve the same technical effects.

[0054] The method provided in the embodiments of the present application is further described in detail below with reference to the accompanying drawings.

[0055] See Figure 1 As shown, the embodiment of the present application provides a method for log grouping, the specific process is as follows:

[0056] Step 101: Determine a first target field sequence and a second target sequence corresponding to the logs to be grouped;

[0057] The first target field sequence includes at least one numerical target field, and the second target sequence includes at least one text target field.

[0058] Before executing step 101 , the first numerical field corresponding to each classification field in the logs to be grouped may be determined according to a mapping relationship between the classification field and the numerical field.

[0059] Then, each numerical field in the log to be grouped and each determined first numerical field are used as numerical target fields constituting the first target field sequence, and each text field in the log to be grouped is used as text target fields constituting the second target field sequence.

[0060] Specifically, the log to be grouped is a security log, which can contain multiple types of fields, such as numerical fields, classification fields, and text fields. Specifically, numerical fields include source IP (Internet Protocol), event time, status code, and request data byte count; classification fields include rule ID (Identity document), attack type, action, and threat level; and text fields include request URI (Uniform Resource Identifier) and User-Agent (UA).

[0061] For example, Figure 2This is an exemplary log, which may be a log in the default configuration of the open source WAF (Web Application Firewall) project OpenWAF. Here, OpenWAF may be an open source WAF project.

[0062] It is worth mentioning that Figure 2 The log shown is an exemplary log, and the specific content involved is also used in the following exemplary description.

[0063] In such Figure 2 In the log shown, the source IP of the log is "192.168.0.1", the event type is "2021-10-1116:12:42", the attack type is "SQLi", the action is "Deny", the threat level is "High", the rule ID is "10001", the request URI is " / index.php?p=1%20or%201=1", and the User-Agent is "curl / 7.29.0", see Table 1 below for details.

[0064] Source IP 192.168.0.1 Event Time 2021-10-11 16:12:42 Attack Type SQLi action Deny Threat Level High Rule ID 10001 URI / index.php?p=1%20or%201=1 User-Agent curl / 7.29.0

[0065] Table 1

[0066] The left column represents each attribute in the log, and the right column represents each field in the log, and there is a one-to-one correspondence between each field and each attribute.

[0067] As shown in Table 1, the fields corresponding to the attack type, action, threat level, and rule ID are the classification fields of the log. Here, the numerical fields corresponding to each classification field can be obtained based on the mapping relationship between the classification field and the numerical field. Specifically, the mapping can be performed in the form of label encoding, that is, mapping the classification fields of N categories one by one to integers in the range of 0 to N-1, where N is an integer greater than or equal to 1. Taking the threat level attribute as an example, if the preset threat level corresponds to four fields, these four fields are mapped to the four numerical fields of 0, 1, 2, and 3 respectively, and when the threat level is "High", it corresponds to the numerical field of "3", then Figure 2 In the log shown, the "High" field is mapped to the numeric field "3".

[0068] After the above mapping process, rounding processing can be performed on the fields corresponding to the two attributes of source IP and event time. The fields after processing are shown in Table 2 below.

[0069] Source IP 3232235521 Event Time 1633939962 Attack Type 0 action 1 Threat Level 3 Rule ID 0 URI / index.php?p=1%20or%201=1 User-Agent curl / 7.29.0

[0070] Table 2

[0071] Among them, the source IP is rounded from the original "192.168.0.1" to "3232235521", the event type is changed from the original "2021-10-11 16:12:42" to "1633939962", the attack type is mapped from the original "SQLi" to "0", the action is mapped from the original "Deny" to "1", the threat level is mapped from the original "High" to "3", and the rule ID is mapped from the original "10001" to "0".

[0072] Through the above processing, the categorical fields originally existing in the log are converted into numerical fields, which simplifies the field types of the log without losing the content of the log itself, thereby reducing the computing time and computing resources required for subsequent log analysis.

[0073] In other words, through the above processing, the logs to be grouped can obtain d target fields corresponding to the logs to be grouped, where d is an integer greater than or equal to 2, and the target fields include numerical target fields and text target fields. Here, because the attributes corresponding to each target field are different, if a single attribute is used as a dimension of the target field, then the logs to be grouped correspond to d target fields with different dimensions. In the embodiment of this application, the attributes of each target field are represented by dimensions, and will not be repeated below.

[0074] Furthermore, the numerical target fields of the logs to be grouped constitute the first target field sequence, and the text target fields of the logs to be grouped constitute the second target field sequence. Here, numerical target fields and text target fields are collectively referred to as target fields. The target fields in the first and second target field sequences are arranged in a specific order based on the size of the dimension.

[0075] For example, the log shown in Table 2 may contain eight dimensions of numerical fields. If the first dimension is the source IP address and the eighth dimension is the User-Agent, the first target field sequence of the log may be {3232235521, 1633939962, 0, 1, 3, 0}, and the second target field sequence of the log may be { / index.php?p=1%20or%201=1, curl / 7.29.0}.

[0076] Based on the above method, the first target field sequence and the second target field sequence corresponding to the logs to be grouped can be determined.

[0077] Step 102: Calculate a first distance between the first target field sequence and a first reference field sequence corresponding to the first log group; the first reference field sequence includes at least one numerical reference field;

[0078] In an embodiment of the present application, the sequence position of each numerical target field in the first target field sequence and the sequence position of each numerical reference field in the first reference field sequence corresponding to the first log group are first determined, and then the field distances between the numerical target fields and the numerical reference fields corresponding to the same sequence position are calculated to obtain multiple field distances between each numerical target field and each numerical reference field, and finally the sum of the multiple field distances is used as the first distance between the first target field sequence and the first reference field sequence.

[0079] It is worth noting that although the first log grouping is described in the embodiments of the present application, the first log grouping here should not constitute a limitation on the number of log groups. It is understandable that there can also be multiple log groups. The processing method of one log grouping is only used as an example for specific description, and other log groups are not repeated.

[0080] In detail, the first log group contains at least one log. If the first log group contains only one log, the numerical reference field in the first reference field sequence corresponding to the first log group is the numerical field of this log; if the first log group contains multiple logs, the numerical reference field in the first reference field sequence corresponding to the first log group is the average value of the numerical fields of these multiple logs. The following is a detailed explanation with reference to the formula.

[0081] The first log group Data containing n logs = {L1, L2, ..., L n}, where n≥2, L1, L2, L n Each log entry in the first log group has a corresponding field in dimension d, where d is an integer greater than or equal to 2. The fields in dimension d include the numerical field in dimension a and the text field in dimension b. Here, for dimensions i = 1 to a, the average value of the numerical field in dimension i corresponding to the n log entries in the first log group is calculated, and the calculated average value is used as the numerical reference field in dimension i corresponding to the first log group. For details, see the following formula:

[0082]

[0083] in, The numerical reference field for the first log group Data corresponding to the i-th dimension, is the numerical field of the i-th dimension corresponding to the j-th log in the first log group Data, and n is the total number of logs in the first log group Data.

[0084] Through the above method, a first reference field sequence corresponding to the first log group can be obtained, and the first reference sequence includes a numerical reference field of dimension a.

[0085] It is understandable that the first target field sequence corresponding to the logs to be grouped also includes a numerical target field in dimension a. Here, for dimensions i = 1 to a, the weighted squared difference between the numerical reference field in dimension i in the first reference field sequence and the numerical target field in dimension i in the first target field sequence is calculated. The a weighted squared difference values calculated in dimension a are summed and squared, and the calculated result is used as the first distance between the first target field sequence and the first reference field sequence. For details, see the following formula:

[0086]

[0087] Wherein, L1 is the first distance between the first target field sequence and the first reference field sequence, L i is the numerical target field of the i-th dimension in the first target field sequence, C i is the numerical reference field of the i-th dimension in the first reference field sequence, w i is the weight of the i-th dimension. The weight is set according to the actual situation. For example, if the source IP dimension is more important, then the weight of this dimension is increased.

[0088] It is worth noting that the first distance here represents the distance between the first target field sequence and the first reference field sequence. Therefore, the smaller the first distance, the more similar the first target field sequence is to the first reference field sequence, and the larger the first distance, the less similar the first target field sequence is to the first reference field sequence.

[0089] Through the above method, the first distance between the first target field sequence and the first reference field sequence is obtained.

[0090] Step 103: Calculate a second distance between the second target field sequence and a second reference field sequence corresponding to the first log group; the second reference field sequence includes at least one text reference field;

[0091] In an embodiment of the present application, the sequence position of each text-class target field in the second target field sequence is first determined, as is the sequence position of each text-class reference field in the second reference field sequence corresponding to the first log group. Then, the minimum edit distance between the text-class target field and the text-class reference field corresponding to the same sequence position is calculated to obtain multiple minimum edit distances between each text-class target field and each text-class reference field. Finally, a weighted sum is performed on these multiple minimum edit distances to obtain a second distance between the second target field sequence and the second reference field sequence.

[0092] In detail, the first log group contains at least one log. If the first log group contains only one log, the text reference field in the second reference field sequence corresponding to the first log group is the text field of this log; if the first log group contains multiple logs, the text reference field in the second reference field sequence corresponding to the first log group is the average value of the text fields of these multiple logs. The following is a detailed explanation with reference to the formula.

[0093] The first log group Data containing n logs = {L1, L2, ..., L n}, where n≥2, L1, L2, L n Each log entry in the first log group corresponds to a field in dimension d, where d is an integer greater than or equal to 2. The fields in dimension d include the numeric field in dimension a and the text field in dimension b. Here, for dimensions k = 1 to b, the text fields corresponding to each of the n log entries in the first log group are calculated to obtain the n text fields corresponding to the first log group in dimension k. T text fields are randomly selected from these n text fields and collectively used as the text reference fields corresponding to dimension k for the first log group. For details, see the following formula:

[0094]

[0095] in, The text class reference field corresponding to the k-th dimension of the first log group Data, The text class field of the kth dimension corresponding to any t logs in the first log group Data.

[0096] Through the above method, the second reference field sequence corresponding to the first log grouping can be obtained, and the second reference sequence includes the text class reference field of the b dimension. It is understandable that the second target field sequence corresponding to the log to be grouped also includes the text class target field of the b dimension. Here, for the dimensions k=1~b, the minimum edit distance between each text class reference field of the kth dimension in the second reference field sequence and the text class target field of the kth dimension in the second target field sequence is calculated, the b minimum edit distances calculated in the b dimension are weighted and summed, and the calculated result is used as the second distance between the second target field sequence and the second reference field sequence, as shown in the following formula:

[0097]

[0098] Wherein, L2 is the second distance between the second target field sequence and the second reference field sequence, L kis the text class target field of the kth dimension in the second target field sequence, C k is a single or multiple text-type reference fields of the kth dimension in the second reference field sequence, v is a single text-type reference field of the kth dimension in the second reference field sequence, E(L k , v) is L k The edit distance between v and w k is the weight of the kth dimension. The weight is set according to the actual situation. For example, if the request URI dimension is more important, then the weight of this dimension is increased.

[0099] It is worth noting that the second distance here represents the distance between the second target field sequence and the second reference field sequence. Therefore, the smaller the second distance, the more similar the second target field sequence is to the second reference field sequence, and the larger the second distance, the less similar the second target field sequence is to the second reference field sequence.

[0100] Through the above method, the second distance between the second target field sequence and the second reference field sequence is obtained.

[0101] Step 104: If the sum of the first distance and the second distance is less than or equal to a preset threshold, the log to be grouped is added to the first log group.

[0102] The first distance in step 102 and the second distance in step 103 are summed to obtain the target distance between the logs to be grouped and the first log group. The specific calculation is shown in the following formula:

[0103] L=L1+L2

[0104] Wherein, L1 is the first distance, L2 is the second distance, and L is the target distance.

[0105] In an embodiment of the present application, it is determined whether the target distance is less than or equal to a preset threshold: if the target distance is less than or equal to the preset threshold, the log to be grouped is added to the first log group; if the target distance is greater than the preset threshold and there are no other log groups, a second log group is created and the log to be grouped is added to the second log group; if the target distance is greater than the preset threshold and there are other log groups, the target distances of the log to be grouped and the other log groups are calculated according to the method of steps 101 to 104, and the log to be grouped is added to the log group corresponding to the calculated target distance greater than or equal to the preset threshold.

[0106] Furthermore, if the log to be grouped is added to the first log group, the first tag corresponding to the first log group will also be determined and then added to the log to be grouped. In addition, because a new log to be grouped is added to the first log group, the first reference field sequence and the second reference field sequence corresponding to the first log group will also be updated.

[0107] The concept of updating the first reference field sequence is the same as step 102. Specifically, for dimensions i=1 to (a+1), the average value of the numerical field corresponding to the i-th dimension of the n+1 logs in the first log group is calculated. For details, see the following formula:

[0108]

[0109] Among them, the left side of the equation is the numerical reference field of the first log group corresponding to the i-th dimension after the update. The right side of the equation is the numerical reference field of the first log group before the update corresponding to the i-th dimension, n is the number of logs in the first log group before the update, (n+1) is the number of logs in the first log group after the update, L i The numeric target field corresponding to the i-th dimension of the logs to be grouped.

[0110] Let's take the jth dimension as an example. In the original second reference field sequence Add the text target field L corresponding to the j-th dimension of the log to be grouped j , forming a new field sequence S = [text1, text2, ..., text t , L j Here, text1 is taken as the first element of the field sequence S, text2 is taken as the second element of the field sequence S, and so on. j As the t+1th element of the field sequence S. For each element S in the field sequence S i , i=0~(t+1), calculate S i The edit distance between the elements in the field sequence S is obtained by taking the set D of t edit distances. i =[E(S i , S1),…,E(S i , S t+1 )]. Then, calculate the variance of each element corresponding to the set Among them, E ij is the edit distance between the i-th element and the j-th element in the set, is the average of the t edit distances in the set. After all variances are calculated, the element with the largest variance is removed from the field sequence S, and the field sequence after removing one element is used as the updated second reference field sequence.

[0111] It can be seen from the update of the second reference field sequence that the updated second reference field sequence retains text reference fields with relatively small variance, thereby ensuring the stability of the text features of the text reference fields in the second reference field sequence.

[0112] In this way, real-time updates of log groups are achieved, which can adapt to the scenarios of online iterative processing, ensure the timeliness of log analysis, and improve the efficiency and accuracy of log analysis.

[0113] In addition, adding the same tag to logs in the same log group can establish correlation between similar logs in the same group. Furthermore, a log can be added to multiple log groups. In this case, the tags corresponding to each of the multiple log groups will be added to the log, thus establishing correlation between the same log in different groups.

[0114] In order to facilitate those skilled in the art to better understand the solution provided in the embodiments of the present application, the application of the above process in actual application scenarios is further explained below with reference to the accompanying drawings.

[0115] like Figure 3 As shown in the figure, it is a flowchart for online processing of log data. It is worth noting that the online processing here is applicable to a scenario, and the following method is also applicable to offline processing scenarios.

[0116] S301: Get the first log;

[0117] Among them, the first log is the log to be grouped.

[0118] S302: Calculate the distance between the first log and the data center of the log group;

[0119] The data center is the collective name of the first reference field sequence and the second reference field sequence corresponding to the log group, and the distance is the target distance.

[0120] S303: Determine whether the distance is greater than or equal to a preset threshold;

[0121] If the distance is greater than or equal to the preset threshold, step S305 is executed; if the distance is less than the preset threshold, step 304 is executed.

[0122] S304: Add the first log to the log group and update the data center and label corresponding to the log group;

[0123] S305: Create a new log group, add the first log to the new log group, and update the data center and label corresponding to the log group.

[0124] In one possible design, if a specific tag is needed for a log and its similar logs, the log group to which the log belongs is first determined, and then the specific tag is added to all logs in the same group. Here, the specific tag can be a tag that ignores the log or a tag that focuses on the log.

[0125] By using the above method, the self-adaptation of log analysis is achieved, which can effectively improve the efficiency of log analysis.

[0126] In one possible design, if a log L needs to be removed k The tag lb of the log and the tag lb of the same log as this log also need to be removed. This can be achieved by the following method: Determine the log L that has the tag lb and contains k Multiple log groups, calculate log L k The target distances of each of these multiple log groups are taken as the final target distance. To facilitate better understanding by those skilled in the art, here we take a log group as an example to calculate the log L k The final target distance from this log group is calculated, and then the target distance between each of the other logs in this log group and this log group is calculated to obtain multiple target distances. The logs corresponding to the target distances greater than the final target distance are selected as the logs with the labels to be removed, and the labels lb of these logs with the removed labels are removed.

[0127] For example, see Figure 4 The figure shows a schematic diagram of removing labels, where the center of the circle is the reference field for log grouping, the distance between log A and the center of the circle is the final target distance, and the distances between logs B, C, D, E, and F and the center of the circle are the target distances. When removing the label of log A, because the target distances of logs D, E, and F are greater than the final target distance, the labels of logs D, E, and F are also removed synchronously.

[0128] Furthermore, in order to facilitate those skilled in the art to better understand the solution provided by the embodiments of the present application, the application of the above process in actual application scenarios is further explained below with reference to the accompanying drawings.

[0129] like Figure 5 The flowchart of removing labels is shown in FIG.

[0130] S501: Remove log L k The label lb;

[0131] S502: Calculate log L k With log L k The distance between the data centers of each log group;

[0132] The data center is the collective name of the first reference field sequence and the second reference field sequence corresponding to the log group, and the distance is the target distance.

[0133] S503: Contain log L k Log grouping, remove all data centers whose distance to the log group is greater than log L k Logs of distance to data centers;

[0134] S504: Update the data center of the log group of the removed log;

[0135] S505: updating the preset threshold of the log group for removing logs;

[0136] Among them, the preset threshold is a distance threshold.

[0137] S506: Determine whether to adjust the weight;

[0138] If it is determined that the weight needs to be adjusted, then S507 is executed; if it is determined that the weight does not need to be adjusted, then S510 is executed.

[0139] S507: Determine whether to increase the weight;

[0140] If it is determined that the weight is to be increased, then S509 is executed; if it is determined that the weight is not to be increased, then S508 is executed;

[0141] S508: reducing the learning rate for the corresponding weight;

[0142] S509: Increase the learning rate for the corresponding weight;

[0143] S510: Complete the process of removing the label lb.

[0144] Generally speaking, removing a tag indicates an error. Therefore, the above method, upon discovering an error in a particular tag, can adaptively detect and remove similar log tags. This effectively adapts to unexpected situations, meeting the needs of various businesses and security operations, and effectively improving the efficiency of log analysis. Furthermore, by enabling real-time log interaction based on tags and dynamically adjusting the parameters corresponding to log groupings, the updated log groupings can quickly adapt to the needs of various businesses and security operations.

[0145] The technical effects that can be achieved by the methods provided in the embodiments of this application include:

[0146] 1. Separate distance calculations for numeric and text fields better align with actual application scenarios. Dynamically adjusting the weights in distance calculations also helps achieve more accurate log grouping and better interpret the calculated distances.

[0147] 2. We propose reference fields for log grouping and use them to characterize the characteristics of each log in the group. This effectively improves online processing performance, saves computing time and resources, and, to some extent, improves the accuracy of log grouping.

[0148] 3. The method provided in the embodiment of the present application is applicable to online scenarios, that is, through the dynamic grouping method, the same log can be added to multiple different log groups, and the same log will also add the labels corresponding to the different groups to establish the correlation between the various log groups, which can be more in line with the actual scenario and is further applicable to complex log analysis scenarios.

[0149] Based on the same inventive concept, the present application also provides a log grouping device to improve the accuracy of log grouping, solve the problem that the existing log grouping is not suitable for real log analysis scenarios, resulting in inaccurate log analysis results, and help improve the accuracy of log analysis results. Figure 6 , the device comprises:

[0150] Determination module 601, determining a first target field sequence and a second target field sequence corresponding to the logs to be grouped, wherein the first target field sequence includes at least one numerical target field, and the second target field sequence includes at least one text target field;

[0151] A first calculation module 602 calculates a first distance between the first target field sequence and a first reference field sequence corresponding to the first log group, where the first reference field sequence includes at least one numerical reference field.

[0152] A second calculation module 603 calculates a second distance between the second target field sequence and a second reference field sequence corresponding to the first log group, where the second reference field sequence includes at least one text reference field.

[0153] The grouping module 604 adds the log to be grouped into the first log group if the sum of the first distance and the second distance is less than or equal to a preset threshold.

[0154] In one possible design, the determination module 601 is specifically used to determine the first numerical class field corresponding to each classification field in the log to be grouped based on the mapping relationship between the classification field and the numerical class field; use each numerical class field in the log to be grouped and each determined first numerical class field as the numerical class target field constituting the first target field sequence; use each text class field in the log to be grouped as the text class target field constituting the second target field sequence.

[0155] In one possible design, the first calculation module 602 is specifically used to determine the attributes corresponding to each numerical target field in the first target field sequence, and the attributes corresponding to each numerical reference field in the first reference field sequence corresponding to the first log group; calculate the field distances between the numerical target fields and the numerical reference fields corresponding to the same attributes, and obtain multiple field distances between the each numerical target field and the each numerical reference field; and use the sum of the multiple field distances as the first distance between the first target field sequence and the second reference field sequence.

[0156] In one possible design, the second calculation module 603 is specifically used to determine the attributes corresponding to each text-class target field in the second target field sequence, and the attributes corresponding to each text-class reference field in the second reference field sequence corresponding to the first log group; calculate the minimum edit distance between the text-class target field and the text-class reference field corresponding to the same attribute, and obtain multiple minimum edit distances between the each text-class target field and the each text-class reference field; perform weighted summation on the multiple minimum edit distances to obtain the second distance between the second target field sequence and the second reference field sequence.

[0157] In one possible design, the grouping module 604 further includes: if the sum of the first distance and the second distance is greater than the preset threshold and there is no other log grouping, creating a second log grouping and adding the log to be grouped to the second log grouping.

[0158] In a possible design, the grouping module 604 is specifically configured to determine a first tag corresponding to the first log group, and add the first tag to the log to be grouped.

[0159] In one possible design, the grouping module 604 further includes: in response to removing the first label of the log to be grouped in the first log group, respectively calculating the sum of the first distance and the second distance between each log in the first log group and the first reference field sequence and the second reference sequence; in other logs in the first log group, determining the associated log corresponding to the calculated sum value greater than the calculated sum value of the log to be grouped, and removing the first label of the associated log, the other logs being logs other than the log to be grouped in the first log group.

[0160] Based on the above device, the distances between numerical fields and text fields are calculated separately, which can be more in line with actual application scenarios. It is further proposed to dynamically adjust the weights in the distance calculation, which helps to obtain more accurate log grouping and has a better interpretation of the calculated distance.

[0161] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The electronic device can realize the function of the aforementioned log grouping device. Figure 7 , the electronic device includes:

[0162] At least one processor 701, and a memory 702 connected to the at least one processor 701. The specific connection medium between the processor 701 and the memory 702 is not limited in the embodiment of the present application. Figure 7 In the example, the processor 701 and the memory 702 are connected via a bus 700. Figure 7 The bus 700 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 7 The diagram is represented by only one thick line, but this does not mean that there is only one bus or one type of bus. Alternatively, the processor 701 may also be referred to as a controller, without limitation to the name.

[0163] In the embodiment of the present application, the memory 702 stores instructions that can be executed by at least one processor 701. The at least one processor 701 can execute the log grouping method discussed above by executing the instructions stored in the memory 702. The processor 701 can implement Figure 6 The functions of each module in the device shown.

[0164] Among them, the processor 701 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 702 and calling data stored in the memory 702, the various functions of the device and processing data.

[0165] In one possible design, processor 701 may include one or more processing units. Processor 701 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0166] Processor 701 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the log grouping method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0167] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 702 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 702 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 702 in the embodiment of the present application can also be a circuit or any other device / system that can realize a storage function, for storing program instructions and / or data.

[0168] By designing and programming the processor 701, the code corresponding to the log grouping method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1 The steps of the log grouping method in the embodiment shown are as follows: How to design and program the processor 701 is a technique well known to those skilled in the art and will not be described in detail here.

[0169] Based on the same inventive concept, an embodiment of the present application further provides a storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes the log grouping method discussed above.

[0170] In some possible implementations, various aspects of the log grouping method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on the device, the program code is used to enable the control device to execute the steps of the log grouping method according to various exemplary embodiments of the present application described above in this specification.

[0171] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices / systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A log grouping method, characterized in that: The method comprises: Determine a first target field sequence and a second target field sequence corresponding to the logs to be grouped, wherein the first target field sequence includes at least one numerical target field, and the second target field sequence includes at least one text target field; Determining an attribute corresponding to each numerical target field in the first target field sequence, and an attribute corresponding to each numerical reference field in the first reference field sequence corresponding to the first log group, calculating field distances between the numerical target fields and the numerical reference fields corresponding to the same attribute, obtaining multiple field distances between the numerical target fields and the numerical reference fields, and using the sum of the multiple field distances as the first distance between the first target field sequence and the first reference field sequence; Determining an attribute corresponding to each text-class target field in the second target field sequence, and an attribute corresponding to each text-class reference field in the second reference field sequence corresponding to the first log group, calculating a minimum edit distance between the text-class target field and the text-class reference field corresponding to the same attribute, obtaining multiple minimum edit distances between the text-class target field and the text-class reference field, and performing a weighted summation on the multiple minimum edit distances to obtain a second distance between the second target field sequence and the second reference field sequence; If the sum of the first distance and the second distance is less than or equal to a preset threshold, the log to be grouped is added to the first log group.

2. The method according to claim 1, wherein The step of determining the first target field sequence and the second target field sequence corresponding to the logs to be grouped includes: Determine the first numerical field corresponding to each of the classification fields in the logs to be grouped according to the mapping relationship between the classification fields and the numerical fields; Using each numerical field in the log to be grouped and each determined first numerical field as a numerical target field constituting a first target field sequence; Each text field in the log to be grouped is used as a text target field constituting a second target field sequence.

3. The method according to claim 1, wherein After adding the log to be grouped to the first log group if the sum of the first distance and the second distance is less than or equal to a preset threshold, the method further includes: If the sum of the first distance and the second distance is greater than the preset threshold and no other log group exists, a second log group is created, and the log to be grouped is added to the second log group.

4. The method according to any one of claims 1 to 3, wherein The adding the log to be grouped into the first log group includes: Determine a first tag corresponding to the first log group, and add the first tag to the log to be grouped.

5. The method according to claim 4, wherein After adding the first tag to the log to be grouped, the method further includes: In response to removing the first label of the log to be grouped in the first log group, respectively calculating the sum of the first distance and the second distance between each log in the first log group and the first reference field sequence and the second reference field sequence; Among other logs in the first log group, determine an associated log whose calculated sum value is greater than the calculated sum value of the log to be grouped, and remove the first tag of the associated log, where the other logs are logs in the first log group other than the log to be grouped.

6. A device for grouping logs, characterized in that: The device comprises: A determination module determines a first target field sequence and a second target field sequence corresponding to the logs to be grouped, wherein the first target field sequence includes at least one numerical target field, and the second target field sequence includes at least one text target field; a first calculation module, configured to determine an attribute corresponding to each numerical target field in the first target field sequence, and an attribute corresponding to each numerical reference field in the first reference field sequence corresponding to the first log group, calculate field distances between the numerical target fields and the numerical reference fields corresponding to the same attribute, obtain multiple field distances between the numerical target fields and the numerical reference fields, and use the sum of the multiple field distances as a first distance between the first target field sequence and the first reference field sequence; a second calculation module, configured to determine an attribute corresponding to each text-type target field in the second target field sequence, and an attribute corresponding to each text-type reference field in the second reference field sequence corresponding to the first log group, calculate a minimum edit distance between the text-type target fields and the text-type reference fields corresponding to the same attribute, obtain multiple minimum edit distances between the text-type target fields and the text-type reference fields, and perform a weighted summation of the multiple minimum edit distances to obtain a second distance between the second target field sequence and the second reference field sequence; An adding module is configured to add the log to be grouped to the first log group if the sum of the first distance and the second distance is less than or equal to a preset threshold.

7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method steps of any one of claims 1 to 5 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 5 are implemented.

Images