Log analysis method and device, equipment and storage medium

Abstract

The invention provides a log analysis method and device, equipment and a storage medium, which are used for solving the problem of low analysis accuracy and efficiency in a log analysis process. The method at least comprises the following steps: extracting a plurality of universal fields contained in each collected log to be analyzed; aiming at a plurality of preset time intervals, respectively executing the following steps: respectively counting respective field values of a plurality of universal fields and respective occurrence times of each field value in each to-be-analyzed log in the time intervals to obtain a corresponding to-be-processed item set; based on the respective field values and the occurrence times of the plurality of universal fields, respectively determining respective value uniformity of the plurality of universal fields; when it is determined that a target field with the value uniformity smaller than a uniformity threshold exists in the multiple universal fields, data filtering is conducted on the to-be-processed item set corresponding to the target field, and a target item set is obtained; and on the basis of each obtained target item set, performing association behavior analysis on each to-be-analyzed log to obtain a log analysis result.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log analysis method, device, equipment and storage medium. Background Art

[0002] As technology continues to develop, more and more devices can analyze logs to identify potential security threats or abnormal behaviors.

[0003] In related technologies, the log analysis method usually involves scanning massive logs hundreds of times to determine the hidden correlation behaviors between logs from different sources, thereby obtaining log analysis results.

[0004] However, due to the large number of logs, scanning the massive logs hundreds of times will consume a lot of computing resources, making the log analysis inefficient. In addition, hundreds of scans will generate a large number of item sets, and the item sets are unevenly distributed, making the log analysis more affected by the item set distribution, resulting in unstable log analysis accuracy.

[0005] It can be seen that the analysis accuracy and efficiency in the log analysis process under related technologies are low. Summary of the invention

[0006] The embodiments of the present application provide a log analysis method, apparatus, device and storage medium for solving the problem of low analysis accuracy and efficiency in the log analysis process.

[0007] In a first aspect, a log analysis method is provided, comprising:

[0008] Extract multiple common fields contained in each collected log to be analyzed;

[0009] For multiple preset time intervals, execute respectively:

[0010] Respectively counting the field values ​​of the plurality of common fields and the number of occurrences of the field values ​​in the respective logs to be analyzed within a time interval to obtain corresponding item sets to be processed;

[0011] Based on the field values ​​and occurrence counts of the respective general fields, respectively determine the value uniformity of the respective general fields;

[0012] When it is determined that there is a target field among the plurality of general fields whose value uniformity is less than a uniform threshold, data is filtered on a to-be-processed item set corresponding to the target field to obtain a target item set;

[0013] Based on the obtained target item sets, the associated behavior analysis is performed on the logs to be analyzed to obtain log analysis results.

[0014] Optionally, the multiple common fields contained in the collected logs to be analyzed include:

[0015] Collect various logs to be analyzed from multiple log sources;

[0016] Extracting multiple log fields contained in each of the logs to be analyzed respectively to obtain corresponding log field sets;

[0017] Count the number of log field sets that contain the same log field in each log field set obtained;

[0018] Based on the statistical quantities of each set, the log fields whose corresponding set quantities are greater than a preset quantity threshold are taken as general fields to obtain multiple general fields.

[0019] Optionally, after extracting and collecting the plurality of common fields contained in each of the logs to be analyzed, the method further includes:

[0020] Get the preset time window;

[0021] The time window is divided into a plurality of time intervals according to a preset number of intervals.

[0022] Optionally, after extracting and collecting the plurality of common fields contained in each of the logs to be analyzed, the method further includes:

[0023] Determine the asset group to which each of the logs to be analyzed belongs;

[0024] The logs to be analyzed belonging to the same asset group are divided into one log group to obtain multiple log groups;

[0025] Based on the pre-stored mapping relationship between each asset group and each time interval set, the time interval sets corresponding to each of the multiple log groups are respectively determined; wherein the time interval set includes multiple time intervals.

[0026] Optionally, the determining the value uniformity of each of the plurality of general fields based on the field values ​​and the number of occurrences of each of the plurality of general fields includes:

[0027] For the multiple common fields, perform the following operations respectively:

[0028] Determine the ratio of the number of occurrences of each field value to the combined number of occurrences in the plurality of field values ​​of the general field; wherein the combined number of occurrences is the sum of the number of occurrences of each of the plurality of field values;

[0029] The uniformity of the value of the common field is determined based on the sum of squares of the frequency ratios corresponding to the multiple field values.

[0030] Optionally, filtering the to-be-processed item set corresponding to the target field to obtain the target item set includes:

[0031] Based on the item set similarity between the to-be-processed item set corresponding to the target field and at least one other to-be-processed item set, under-sampling processing is performed on the to-be-processed item set corresponding to the target field to obtain a target item set; wherein the other to-be-processed item set is a to-be-processed item set among the obtained to-be-processed item sets except the to-be-processed item set corresponding to the target field.

[0032] In a second aspect, a log analysis device is provided, comprising:

[0033] Processing module: used to extract multiple common fields contained in each collected log to be analyzed;

[0034] The processing module is further used to: for a plurality of preset time intervals, respectively execute:

[0035] Respectively counting the field values ​​of the plurality of common fields and the number of occurrences of the field values ​​in the respective logs to be analyzed within a time interval to obtain corresponding item sets to be processed;

[0036] Based on the field values ​​and occurrence counts of the respective general fields, respectively determine the value uniformity of the respective general fields;

[0037] When it is determined that there is a target field among the plurality of general fields whose value uniformity is less than a uniform threshold, data is filtered on a to-be-processed item set corresponding to the target field to obtain a target item set;

[0038] The processing module is further used to: perform association behavior analysis on each log to be analyzed based on each obtained target item set to obtain a log analysis result.

[0039] Optionally, the device further comprises an acquisition module, wherein the acquisition module is used to: collect logs to be analyzed from multiple log sources;

[0040] The processing module is specifically used for:

[0041] Extracting multiple log fields contained in each of the logs to be analyzed respectively to obtain corresponding log field sets;

[0042] Count the number of log field sets that contain the same log field in each log field set obtained;

[0043] Based on the statistical quantities of each set, the log fields whose corresponding set quantities are greater than a preset quantity threshold are taken as general fields to obtain multiple general fields.

[0044] Optionally, the processing module is further used for:

[0045] After extracting and collecting the plurality of common fields contained in each of the logs to be analyzed, obtaining a preset time window;

[0046] The time window is divided into a plurality of time intervals according to a preset number of intervals.

[0047] Optionally, the processing module is further used for:

[0048] After extracting and collecting the plurality of common fields contained in each of the logs to be analyzed, determining the asset group to which each of the logs to be analyzed belongs;

[0049] The logs to be analyzed belonging to the same asset group are divided into one log group to obtain multiple log groups;

[0050] Based on the pre-stored mapping relationship between each asset group and each time interval set, the time interval sets corresponding to each of the multiple log groups are respectively determined; wherein the time interval set includes multiple time intervals.

[0051] Optionally, the processing module is specifically used for:

[0052] For the multiple common fields, perform the following operations respectively:

[0053] Determine the ratio of the number of occurrences of each field value to the combined number of occurrences in the plurality of field values ​​of the general field; wherein the combined number of occurrences is the sum of the number of occurrences of each of the plurality of field values;

[0054] The uniformity of the value of the common field is determined based on the sum of squares of the frequency ratios corresponding to the multiple field values.

[0055] Optionally, the processing module is specifically used for:

[0056] Based on the item set similarity between the to-be-processed item set corresponding to the target field and at least one other to-be-processed item set, under-sampling processing is performed on the to-be-processed item set corresponding to the target field to obtain a target item set; wherein the other to-be-processed item set is a to-be-processed item set among the obtained to-be-processed item sets except the to-be-processed item set corresponding to the target field.

[0057] According to a third aspect, a computer program product is provided, comprising a computer program, which implements the method according to the first aspect when executed by a processor.

[0058] According to a fourth aspect, a computer device is provided, comprising:

[0059] A memory for storing program instructions;

[0060] The processor is used to call the program instructions stored in the memory and execute the method as described in the first aspect according to the obtained program instructions.

[0061] According to a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in the first aspect.

[0062] In the embodiment of the present application, data filtering is performed for multiple time intervals so that the obtained target item set data is uniform in each time interval, thereby avoiding the problem of unstable log analysis accuracy caused by uneven item set data.

[0063] Furthermore, in each time interval, data filtering is performed only on the item sets to be processed corresponding to the target fields whose value uniformity is less than the uniform threshold. By dividing the time intervals and calculating the value uniformity, the item sets to be processed with uneven data in each item set to be processed can be effectively identified, thereby improving the accuracy of data filtering and thus the accuracy of log analysis.

[0064] Furthermore, by counting the field values ​​of multiple common fields in each log to be analyzed, as well as the number of occurrences of each field value, each set of items to be processed is obtained. This eliminates the need to scan the log hundreds of times, reduces the amount of calculation, avoids unnecessary occupation of computing resources, and improves log analysis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 An application scenario of the log analysis method provided in the embodiment of the present application;

[0066] Figure 2 A flow chart of a log analysis method provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of the log analysis method provided in the embodiment of the present application Figure 1 ;

[0068] Figure 4 A schematic diagram of the log analysis method provided in the embodiment of the present application Figure 2 ;

[0069] Figure 5 A schematic diagram of the log analysis method provided in the embodiment of the present application Figure 3 ;

[0070] Figure 6 A structural diagram of a log analysis device provided in an embodiment of the present application Figure 1 ;

[0071] Figure 7A structural diagram of a log analysis device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0073] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0074] (1) Frequent Pattern Growth (FP-Growth) algorithm:

[0075] FP-Growth algorithm is an efficient data mining algorithm, which is specially used to discover frequent itemsets in data sets. By constructing a special tree structure - FP tree (Frequent Pattern Tree), the efficiency of mining frequent itemsets is greatly improved.

[0076] (2) Undersampling:

[0077] Undersampling is a technique for dealing with imbalanced datasets. It balances the distribution of classes in a dataset by reducing the number of samples in the majority class, with the goal of preventing the model from being overly biased towards the majority class. The majority class samples are selectively deleted by calculating the distance between samples, usually retaining those majority class samples that are closer to the minority class samples.

[0078] It should be noted that the embodiments of the present application involve operations to obtain data such as logs to be analyzed. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0079] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0080] The following is a brief introduction to the application fields of the log analysis method provided in the embodiments of the present application.

[0081] As technology continues to develop, more and more devices can analyze logs to identify potential security threats or abnormal behaviors.

[0082] For example, a security device log aggregation system may need to analyze a large number of logs from multiple sources to identify potential security threats or abnormal behaviors. The hidden correlation behaviors between logs from different sources are particularly important for identifying security threats. Usually, log correlation analysis is required to achieve log aggregation and security pattern extraction, or further used to portray asset group portraits.

[0083] In the related art, the log analysis method is usually to scan hundreds of times in front of massive logs to determine the hidden correlation behavior between logs from different sources, so as to obtain log analysis results. For example, the Apriori algorithm is used to scan hundreds of times to achieve data mining.

[0084] However, due to the large number of logs, scanning the massive logs hundreds of times will consume a lot of computing resources, making the log analysis inefficient. In addition, hundreds of scans will generate a large number of item sets, and the item sets are unevenly distributed, making the log analysis more affected by the item set distribution, resulting in unstable log analysis accuracy.

[0085] It can be seen that the analysis accuracy and efficiency in the log analysis process under related technologies are low.

[0086] In order to solve the problem of low analysis accuracy and efficiency in the process of log analysis, the present application proposes a log analysis method. In this method, multiple common fields contained in each collected log to be analyzed are extracted. For multiple preset time intervals, respectively perform: respectively count the field values ​​of multiple common fields in each log to be analyzed within the time interval, and the number of occurrences of each field value, to obtain the corresponding set of items to be processed. Based on the field values ​​and the number of occurrences of each of the multiple common fields, respectively determine the uniformity of the values ​​of the multiple common fields. When it is determined that there is a target field whose value uniformity is less than the uniform threshold among the multiple common fields, the set of items to be processed corresponding to the target field is filtered to obtain the target item set. Based on the target item sets obtained for the preset multiple time intervals, the associated behavior analysis is performed on each log to be analyzed to obtain the log analysis results.

[0087] In the embodiment of the present application, data filtering is performed for multiple time intervals so that the obtained target item set data is uniform in each time interval, thereby avoiding the problem of unstable log analysis accuracy caused by uneven item set data.

[0088] Furthermore, in each time interval, data filtering is performed only on the item sets to be processed corresponding to the target fields whose value uniformity is less than the uniform threshold. By dividing the time intervals and calculating the value uniformity, the item sets to be processed with uneven data in each item set to be processed can be effectively identified, thereby improving the accuracy of data filtering and thus the accuracy of log analysis.

[0089] Furthermore, by counting the field values ​​of multiple common fields in each log to be analyzed, as well as the number of occurrences of each field value, each set of items to be processed is obtained. This eliminates the need to scan the log hundreds of times, reduces the amount of calculation, avoids unnecessary occupation of computing resources, and improves log analysis efficiency.

[0090] The following describes the application scenarios of the log analysis method provided by this application.

[0091] Please refer to Figure 1 , which is a schematic diagram of an application scenario of the log analysis method provided in this application. The application scenario includes a server 101 and a client 102, and the server 101 and the client 102 can communicate with each other, and the communication method can be to use wired communication technology for communication, for example, by connecting a network cable or a serial port line for communication; or to use wireless communication technology for communication, for example, by using Bluetooth or wireless fidelity (wireless fidelity, WIFI) and other technologies for communication, without specific limitation.

[0092] The client 102 generally refers to a device that can provide logs to be analyzed, such as a terminal device, a third-party application program accessible to the terminal device, or a web page accessible to the terminal device, etc. The server 101 generally refers to a device that can perform log analysis, such as a terminal device or a server, etc.

[0093] Terminal devices include but are not limited to mobile phones, computers, smart medical devices, smart home appliances, vehicle-mounted terminals or aircraft, etc. Servers include but are not limited to cloud servers, local servers or associated third-party servers, such as independent physical servers, server clusters or distributed systems composed of multiple physical servers, and cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Both the client 101 and the server 102 can use cloud computing to reduce the occupation of local computing resources; cloud storage can also be used to reduce the occupation of local storage resources.

[0094] As an embodiment, the server 101 and the client 102 may be the same device, or may be different devices, or may be different devices with some modules in common, etc., without specific limitation.

[0095] The following is based on Figure 1 , the log analysis method provided in the embodiment of this application is specifically introduced. Please refer to Figure 2 , which is a flow chart of the log analysis method provided in an embodiment of the present application.

[0096] S201, extracting a plurality of common fields contained in each of the collected logs to be analyzed.

[0097] The common field may be a field that appears in all logs to be analyzed, a field that appears in most of the logs to be analyzed, or a preset field, etc., without any specific limitation.

[0098] As an embodiment, before extracting the multiple common fields contained in each collected log to be analyzed, each log source data can be collected first, and log parsing can be performed on each log source data respectively, and the information to be analyzed can be extracted from the log source data to obtain each log to be analyzed. Log parsing can convert the log text into a structured data format for further analysis, storage and processing. Structured data formats are, for example, JSON, CSV, etc.

[0099] Before performing log parsing, you can first define log fields that represent the information to be analyzed, such as timestamp, which is used to record the time when the event occurred; host name or Internet Protocol (IP) address, which is used to record the host name or IP address where the event occurred; user number (ID), which is used to record the user who performs the operation; request type, which is used to record the type of request (such as GET, POST); response code, which is used to record the response status code of the request; error information, which is used to record error information or exception information, etc., without specific restrictions.

[0100] A regular expression is written using the defined log fields, and log parsing is performed on each log source data respectively, so that information to be analyzed can be extracted, and the extracted information to be analyzed is converted into a structured data format to obtain the corresponding log to be analyzed.

[0101] As an embodiment, after the extracted information to be analyzed is converted into a structured data format, normalization processing can also be performed to convert log source data from different log sources into a unified, standard format to eliminate differences between different log sources so that they can be processed and analyzed consistently.

[0102] As an embodiment, when extracting multiple common fields, various logs to be analyzed from multiple log sources can be collected, so that hidden association behaviors between logs from different log sources can be determined through log analysis to improve the accuracy of analysis. The log source can be a server, application, network device, etc., without specific limitation.

[0103] Extract multiple log fields contained in each log to be analyzed respectively to obtain corresponding log field sets. Count the number of log field sets containing the same log field in each log field set obtained. Based on the counted number of each set, the log field whose corresponding set number is greater than a preset number threshold is taken as a general field to obtain multiple general fields. Thus, the log fields that appear in the logs to be analyzed whose number is greater than the preset number threshold in each log to be analyzed can be determined as general fields.

[0104] As an embodiment, when performing log analysis based on common fields, at least one time window can be pre-set, for example, a time window can be set for each log source, or a time window can be set for all logs to be analyzed, or a time window can be set for each asset group, etc., without specific limitation. By setting a time window, log analysis can be performed on the logs to be analyzed generated within the time window in a targeted manner, avoiding the negative impact of misleading log data in other unnecessary time periods on log analysis.

[0105] After obtaining the preset time window, the time window can be divided into multiple time intervals according to the preset number of intervals. In this way, data filtering can be performed for each time to ensure that the item set data in each time interval is uniform, and avoid the situation where the item set data cannot be uniform when the amount of data in the log to be analyzed is large. By improving the uniformity of the item set data, the accuracy of log analysis can be improved.

[0106] For example, if the time window is from 15:00 to 16:00, the time interval may be every five minutes within the time window.

[0107] As an embodiment, each log source can be divided into multiple asset groups, each asset group contains multiple devices or systems with associated relationships. For example, a server cluster can be an asset group, and each server can perform the same or similar tasks; for example, a network device cluster can be an asset group, and each network device can include connected routers, switches or other network devices; an application instance can be an asset group, and each application instance can be multiple application instances deployed in the same environment; a database cluster can be an asset group, and each database can work together, such as a master-slave relationship, a master-slave relationship, or a distributed relationship, etc., without specific limitations.

[0108] Then, a time window can be set for each asset group, and the corresponding number of intervals can be set. Thus, the asset group to which each log to be analyzed belongs can be determined, and the logs to be analyzed belonging to the same asset group can be divided into one log group to obtain multiple log groups. Based on the pre-stored mapping relationship between each asset group and each time interval set, the time interval sets corresponding to each of the multiple log groups are determined respectively, and the time interval set contains multiple time intervals.

[0109] S202 to S204 are respectively executed for the preset multiple time intervals; after obtaining multiple time intervals for each log group, S202 to S204 can be respectively executed for each log to be analyzed contained in the corresponding log group for the preset multiple time intervals, which are not listed one by one here.

[0110] S202 , respectively counting the field values ​​of a plurality of common fields and the number of occurrences of each field value in each log to be analyzed within a time interval to obtain a corresponding item set to be processed.

[0111] In the time interval t i In the log to be analyzed, the common field X is counted. k The number of occurrences of each field value is X j,i,k ,j∈[1,N i,k ], where the number of possible field values ​​is N i,k When the log groups are divided, statistics can be collected in the time interval t i In each log to be analyzed contained in the log group, the common field X k The number of occurrences of each field value is X j,i,k ,j∈[1,N i,k ], where the number of possible field values ​​is N i,k .

[0112] For example, in the first five minutes, in each log to be analyzed, the field value of the general field "request type" is "login", which appears 10 times; the field value of the general field "request type" is "logout", which appears 5 times; the field value of the general field "request type" is "view", which appears 3 times. Then, the set of items to be processed can be obtained: {"login": 10, "logout": 5, "view": 3}.

[0113] For another example, within the first five minutes, a log to be analyzed contains the general field "request type" and the general field "response code". In each log to be analyzed, the field value and number of occurrences of the general field "request type" can refer to the previous example; in each log to be analyzed, the field value of the general field "response code" is "200", which appears 30 times; the field value of the general field "response code" is "404", which appears 10 times; the field value of the general field "response code" is "500", which appears 2 times. Then the set of items to be processed can be obtained {request type ["login": 10, "logout": 5, "view": 3], response code ["200": 30, "404": 1, "500": 2]}.

[0114] S203, based on the field values ​​and occurrence times of the respective general fields, respectively determine the value uniformity of the respective general fields.

[0115] After obtaining the field values ​​and occurrence times of the respective general fields, the value uniformity of the respective general fields can be determined based on the field values ​​and occurrence times of the respective general fields. For example, when the occurrence times of the respective field values ​​of each general field differ greatly, it means that the values ​​of the general field are uneven, and the value uniformity is lower; when the occurrence times of the respective field values ​​of each general field differ less, it means that the values ​​of the general field are uniform, and the value uniformity is higher.

[0116] There are many ways to determine the uniformity of values. For example, determine the difference in the number of occurrences of each two field values. When it is determined that there is at least one field value with a difference greater than a difference threshold, determine the value uniformity of the corresponding common field to be a smaller value. The more the number of at least one difference, the smaller the value of the value uniformity.

[0117] For another example, for multiple common fields, perform the following operations respectively:

[0118] Determine the ratio of the number of occurrences of each field value to the total number of occurrences among multiple field values ​​of the common field; wherein the total number of occurrences is the sum of the number of occurrences of each of the multiple field values;

[0119] Based on the sum of squares of the frequency ratios of multiple field values, determine the uniformity of the values ​​of the common field. Please refer to formula (1).

[0120]

[0121] S204: When it is determined that there is a target field whose value uniformity is less than a uniform threshold value among the multiple common fields, data is filtered on the to-be-processed item set corresponding to the target field to obtain a target item set.

[0122] When it is determined that among multiple common fields, there is a target field whose value uniformity is less than the uniform threshold, data is filtered on the to-be-processed item set corresponding to the target field to obtain the target item set; when it is determined that among multiple common fields, there is no target field whose value uniformity is less than the uniform threshold, each to-be-processed item set is used as the corresponding target item set.

[0123] There are many methods for data filtering, for example, using the Near-Miss algorithm, etc., to perform undersampling on the item set to be processed corresponding to the target field to obtain the target item set. The common field with the most field values ​​in the time interval is taken as a positive example, and the average minimum distance or maximum distance is calculated according to the Near-Miss algorithm, and the field values ​​in the positive examples that meet the conditions are selected for selective elimination.

[0124] For another example, based on the item set similarity between the item set to be processed corresponding to the target field and at least one other item set to be processed, undersampling processing may be performed on the item set to be processed corresponding to the target field to obtain the target item set. The other item sets to be processed are the item sets to be processed other than the item set to be processed corresponding to the target field among the obtained item sets to be processed.

[0125] Based on the item set similarity between the to-be-processed item set corresponding to the target field and at least one other to-be-processed item set, a similar to-be-processed item set having the greatest item set similarity with the to-be-processed item set corresponding to the target field is selected from the other to-be-processed item sets.

[0126] Based on the similar data between the similar to-be-processed item sets and the to-be-processed item sets corresponding to the target field, the to-be-processed item sets corresponding to the target field are filtered to obtain the target item set, so that the target item set only contains the similar data, effectively filtering out the niche data in the to-be-processed item sets corresponding to the target field, which is helpful for accurate log analysis.

[0127] S205 , based on the obtained target item sets, performing correlation behavior analysis on each log to be analyzed to obtain a log analysis result.

[0128] After obtaining target item sets for multiple time intervals, correlation behavior analysis can be performed on each log to be analyzed based on the obtained target item sets to obtain log analysis results.

[0129] For example, the analysis of associated behaviors is as follows: after obtaining each target item set, data integration is performed on each target item set to establish at least one item set to be analyzed; an FP tree is generated based on the at least one item set to be analyzed, and at least one frequent item set is determined based on the generated FP tree using the FP-Growth algorithm; and a log analysis result is generated based on the at least one frequent item set obtained. The at least one frequent item set can be directly used as the log analysis result, or at least one associated behavior can be determined based on the association relationship between the pre-stored frequent item set and the associated behavior, and the at least one associated behavior can be used as the log analysis result.

[0130] The following is an example introduction to the log analysis method provided in the embodiment of the present application.

[0131] Please refer to Figure 3 , including multiple log sources, namely log source A, log source B, ... and log source N; also including an item set optimization module, which is used to divide multiple time intervals according to the set time window, and for each time interval, determine the value uniformity of each common field, and then perform undersampling processing based on the value uniformity to obtain a target item set; also including a frequent item set generation module, which is used to build an FP tree based on the obtained target item set, and use the FP-Grownth algorithm to generate frequent item sets according to the FP tree to obtain log analysis results; also including an acquisition module, which is used to collect each log to be analyzed from multiple log sources, and then extract multiple common fields from each log to be analyzed, denoted as X k .

[0132] Please refer to Figure 4 After obtaining each common field, the field value and the number of occurrences of each common field in each time interval within the time window are counted. The number of occurrences of each field value is recorded as X j,i,k ,j∈[1,N i,k ] According to the above formula (1), the value uniformity A of each general field is determined i，k .

[0133] Determine the existence of value uniformity A i，k When the target field is less than the uniform threshold a, the to-be-processed item set corresponding to the target field is undersampled to achieve the purpose of compressing the uneven data in the to-be-processed item set corresponding to the target field in the corresponding time interval, and obtain the target item set. Figure 5 , which is a comparison chart before and after data compression. It can be seen intuitively that the amount of uneven data is greatly reduced. i，k For a common field that is not less than the uniform threshold a, the item set to be processed corresponding to the common field is taken as the target item set.

[0134] By accumulating the target item sets obtained in each time interval, all target item sets obtained in the time window can be aggregated, so that an FP tree can be established based on all target item sets, and the FP-Grownth algorithm is used to generate frequent item sets according to the FP tree to obtain the log analysis results.

[0135] In an embodiment of the present application, the item set extraction time is divided to adapt to the streaming processing of multi-source logs, common fields are extracted from multi-source logs, the uniformity calculation of asset groups is introduced within the time interval, and partial under-sampling processing is performed based on the threshold. This can reduce data noise and redundancy, reduce the sensitivity of FP-Growth to the support threshold parameter, determine whether to perform under-sampling processing based on the uniformity of a single asset group, and aggregate multiple asset groups for item set construction, thereby reducing the memory consumption of the FP tree.

[0136] Based on the same inventive concept, the present application embodiment provides a log analysis device that can implement the functions corresponding to the aforementioned log analysis method. Figure 6 , the device includes a processing module 601 and an acquisition module 602, wherein:

[0137] Processing module 601: used to extract multiple common fields contained in each collected log to be analyzed;

[0138] The processing module 601 is further used to: for a plurality of preset time intervals, respectively execute:

[0139] The field values ​​of multiple common fields and the number of occurrences of each field value in each log to be analyzed within the time interval are counted to obtain the corresponding item set to be processed;

[0140] Based on the field values ​​and occurrence times of the respective general fields, respectively determine the value uniformity of the respective general fields;

[0141] When it is determined that among multiple common fields, there is a target field whose value uniformity is less than the uniformity threshold,

[0142] Filter the data of the item set to be processed corresponding to the target field to obtain the target item set;

[0143] The processing module 601 is further used to: perform correlation behavior analysis on each log to be analyzed based on each obtained target item set, and obtain a log analysis result.

[0144] Optionally, the device further includes an acquisition module 602, and the acquisition module 602 is used to: collect various logs to be analyzed from multiple log sources;

[0145] The processing module 601 is specifically used for:

[0146] Extract multiple log fields contained in each log to be analyzed respectively to obtain a corresponding log field set;

[0147] Count the number of log field sets that contain the same log field in each log field set obtained;

[0148] Based on the statistical quantities of each set, the log fields whose corresponding set quantities are greater than a preset quantity threshold are taken as general fields to obtain multiple general fields.

[0149] Optionally, the processing module 601 is further used for:

[0150] After extracting multiple common fields contained in each of the collected logs to be analyzed, a preset time window is obtained;

[0151] The time window is divided into multiple time intervals according to the preset number of intervals.

[0152] Optionally, the processing module 601 is further used for:

[0153] After extracting a plurality of common fields contained in each of the collected logs to be analyzed, determining the asset group to which each of the logs to be analyzed belongs;

[0154] The logs to be analyzed belonging to the same asset group are divided into one log group to obtain multiple log groups;

[0155] Based on the pre-stored mapping relationship between each asset group and each time interval set, the time interval sets corresponding to each of the multiple log groups are determined respectively; wherein the time interval set includes multiple time intervals.

[0156] Optionally, the processing module 601 is specifically used for:

[0157] For multiple common fields, perform the following operations:

[0158] Determine the ratio of the number of occurrences of each field value to the total number of occurrences among multiple field values ​​of the common field; wherein the total number of occurrences is the sum of the number of occurrences of each of the multiple field values;

[0159] The uniformity of the values ​​of the common field is determined based on the sum of the squares of the frequency ratios corresponding to the values ​​of multiple fields.

[0160] Optionally, the processing module 601 is specifically used for:

[0161] Based on the item set similarity between the to-be-processed item set corresponding to the target field and at least one other to-be-processed item set, under-sampling processing is performed on the to-be-processed item set corresponding to the target field to obtain the target item set; wherein the other to-be-processed item sets are the to-be-processed item sets among the obtained to-be-processed item sets except the to-be-processed item set corresponding to the target field.

[0162] Please refer to Figure 7 , is a computer device 700 provided in an embodiment of the present application, and the computer device 700 can be, for example, Figure 1 The client 101 or the server 102 in the data storage program. The current version and the historical version of the data storage program and the application software corresponding to the data storage program can be installed on the computer device 700, and the computer device 700 includes a processor 780 and a memory 720. In some embodiments, the computer device 700 may include a display unit 740, and the display unit 740 includes a display panel 741 for displaying a user interactive operation interface, etc.

[0163] In a possible embodiment, the display panel 741 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0164] The processor 780 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 780 reads the data storage program or file, so as to run the data storage program on the computer device 700 and display the corresponding interface on the display unit 740. The processor 780 may include one or more general-purpose processors and may also include one or more DSPs (Digital Signal Processors) to perform related operations to implement the technical solutions provided in the embodiments of the present application.

[0165] The memory 720 generally includes internal memory and external memory, and the internal memory may be a random access memory (RAM), a read-only memory (ROM), and a cache (CACHE), etc. The external memory may be a hard disk, an optical disk, a USB disk, a floppy disk, or a tape drive, etc. The memory 720 is used to store computer programs and other data, and the computer program includes applications corresponding to each client, etc. Other data may include data generated after the operating system or application is run, and the data includes system data (such as configuration parameters of the operating system) and user data. In the embodiment of the present application, the computer program is stored in the memory 720, and the processor 780 executes the computer program in the memory 720 to implement any of the methods discussed in the previous figure.

[0166] The display unit 740 is used to receive input digital information, character information or contact touch operation / contactless gesture, and generate signal input related to user settings and function control of the computer device 700. Specifically, in the embodiment of the present application, the display unit 740 may include a display panel 741. The display panel 741 is, for example, a touch screen, which can collect user touch operations on or near it (such as operations performed by the user using fingers, stylus, or any other suitable object or accessory on or on the display panel 741), and drive corresponding connection devices according to a pre-set program.

[0167] In a possible embodiment, the display panel 741 may include two parts: a touch detection device and a touch controller. The touch detection device detects the touch position of the player, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 780, and can receive and execute the command sent by the processor 780.

[0168] The display panel 741 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 740, in some embodiments, the computer device 700 may further include an input unit 730, which may include an image input device 731 and other input devices 732, wherein the other input devices may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0169] In addition to the above, the computer device 700 may also include a power supply 790 for supplying power to other modules, an audio circuit 760, a near field communication module 770, and an RF circuit 710. The computer device 700 may also include one or more sensors 750, such as an acceleration sensor, a light sensor, a pressure sensor, etc. The audio circuit 760 specifically includes a speaker 761 and a microphone 762, etc. For example, the computer device 700 can collect the user's voice through the microphone 762 to perform corresponding operations, etc.

[0170] As an embodiment, the number of processors 780 may be one or more, and the processor 780 and the memory 720 may be coupled or relatively independently configured.

[0171] As an example, Figure 7 The processor 780 in the embodiment may be used to implement the following Figure 6 The functions of the processing module 601 and the acquisition module 602 in FIG.

[0172] As an example, Figure 7The processor 780 in can be used to implement the corresponding functions of the server or terminal device discussed above.

[0173] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by a computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0174] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product, for example, through a computer program product, which is stored in a storage medium, including a computer program for a computer device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0175] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A log analysis method, characterized in that: include: Extract multiple common fields contained in each collected log to be analyzed; For multiple preset time intervals, execute respectively: Respectively counting the field values ​​of the plurality of common fields and the number of occurrences of the field values ​​in the respective logs to be analyzed within a time interval to obtain corresponding item sets to be processed; Determining the value uniformity of each of the plurality of general fields based on the field values ​​and the number of occurrences of each of the plurality of general fields; When it is determined that there is a target field whose value uniformity is less than a uniform threshold value among the plurality of general fields, data is filtered on a to-be-processed item set corresponding to the target field to obtain a target item set; Based on the obtained target item sets, the associated behavior analysis is performed on the logs to be analyzed to obtain log analysis results.

2. The method according to claim 1, characterized in that The multiple common fields contained in the extracted and collected logs to be analyzed include: Collect various logs to be analyzed from multiple log sources; Extracting multiple log fields contained in each of the logs to be analyzed respectively to obtain corresponding log field sets; Count the number of log field sets that contain the same log field in each log field set obtained; Based on the statistical quantities of each set, the log fields whose corresponding set quantities are greater than a preset quantity threshold are taken as general fields to obtain multiple general fields.

3. The method according to claim 1, characterized in that After extracting and collecting the plurality of common fields contained in each log to be analyzed, the method further includes: Get the preset time window; The time window is divided into a plurality of time intervals according to a preset number of intervals.

4. The method according to claim 1, characterized in that: After extracting and collecting the plurality of common fields contained in each log to be analyzed, the method further includes: Determine the asset group to which each of the logs to be analyzed belongs; The logs to be analyzed belonging to the same asset group are divided into one log group to obtain multiple log groups; Based on the pre-stored mapping relationship between each asset group and each time interval set, the time interval sets corresponding to each of the multiple log groups are respectively determined; wherein the time interval set includes multiple time intervals.

5. The method according to any one of claims 1 to 4, characterized in that: The uniformity of the values ​​of the plurality of general fields based on the field values ​​and the number of occurrences of the plurality of general fields respectively comprises: For the multiple common fields, perform the following operations respectively: Determine the ratio of the number of occurrences of each field value to the combined number of occurrences in the plurality of field values ​​of the general field; wherein the combined number of occurrences is the sum of the number of occurrences of each of the plurality of field values; The uniformity of the value of the common field is determined based on the sum of squares of the frequency ratios corresponding to the multiple field values.

6. The method according to any one of claims 1 to 4, characterized in that: The filtering of the to-be-processed item set corresponding to the target field to obtain the target item set includes: Based on the item set similarity between the to-be-processed item set corresponding to the target field and at least one other to-be-processed item set, under-sampling processing is performed on the to-be-processed item set corresponding to the target field to obtain a target item set; wherein the other to-be-processed item set is a to-be-processed item set among the obtained to-be-processed item sets except the to-be-processed item set corresponding to the target field.

7. A log analysis device, characterized in that: include: Processing module: used to extract multiple common fields contained in each collected log to be analyzed; The processing module is further used to: for a plurality of preset time intervals, respectively execute: Respectively counting the field values ​​of the plurality of common fields and the number of occurrences of the field values ​​in the respective logs to be analyzed within a time interval to obtain corresponding item sets to be processed; Determining the value uniformity of each of the plurality of general fields based on the field values ​​and the number of occurrences of each of the plurality of general fields; When it is determined that there is a target field whose value uniformity is less than a uniform threshold value among the plurality of general fields, data is filtered on a to-be-processed item set corresponding to the target field to obtain a target item set; The processing module is further used to: perform association behavior analysis on each log to be analyzed based on each obtained target item set to obtain a log analysis result.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: include: A memory for storing program instructions; The processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 6 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images