Big Data-Based Network Log Analysis Methods and Systems

Abstract

This invention relates to the field of web log analysis, and more particularly to a method and system for web log analysis based on big data. The method includes: calculating the weight of each decision tree in a pre-defined optimal random forest model; inputting real-time acquired web logs into the optimal random forest model; and using the weights of each decision tree to perform a weighted summation of the prediction results to obtain the final prediction result for the web logs. This invention fully considers the phenomena of polysemy and polyphy when analyzing web logs using keywords, facilitating the further integration of log prediction results based on keyword importance to obtain the final prediction result, thus improving the accuracy of the prediction results.

Description

Technical Field

[0001] This invention relates to the field of network log analysis, and more particularly to network log analysis methods and systems based on big data. Background Technology

[0002] Network logs are files that record the operational status, error messages, and event logs of network devices, services, or applications. They contain a wealth of information about system operations, network attacks, and malicious intrusions. Analyzing network log data allows for the timely detection of abnormal traffic, intrusions, or potential security threats, enabling the implementation of appropriate defensive measures and reducing the risk of network attacks. Deep learning and big data analysis of massive amounts of network logs enable more intelligent operation and maintenance management. Automated log analysis can quickly locate problems, predict potential faults, and even proactively identify abnormal behavior in devices or systems, thereby improving operational efficiency and reducing the cost of manual intervention.

[0003] Chinese patent document CN111130877B discloses a network log processing system based on NLP, including a natural language processing component and a database. The database contains a classification lexicon, a preset semantic lexicon, and a language processing model. The classification lexicon is set as keywords based on multiple direct translations corresponding to specific device types or high-frequency words derived from word segmentation. The classification lexicon is mapped to and associated with the preset semantic lexicon, which is also associated with the language processing model. The natural language processing component summarizes, classifies, analyzes, and determines the meaning of natural language statements from the device's syslog source data and log files.

[0004] As can be seen from the above scheme, when analyzing network logs, keywords can be extracted from the network logs, and then the keywords can be used to analyze whether the network logs are abnormal. Existing network log analysis methods only consider the word frequency of keywords and do not consider the phenomenon of polysemy. Different word meanings will lead to different analysis results, resulting in low accuracy of the final analysis results. Summary of the Invention

[0005] To address the issue of low accuracy in keyword-based web log analysis in existing technologies, this invention provides a web log analysis method and system based on big data.

[0006] Firstly, the present invention provides a network log analysis method based on big data, employing the following technical solution:

[0007] Calculate the weight of each decision tree in the preset optimal random forest model, input the real-time acquired network logs into the optimal random forest model, and use the weight of each decision tree to perform a weighted sum of the prediction results to obtain the final prediction result of the network logs.

[0008] The weight of each decision tree is calculated as follows:

[0009] Retrieve multiple web logs and assign tags to each log, including normal and abnormal tags; extract keywords from each log and further obtain various keyword combinations;

[0010] Calculate the match degree between each keyword combination and the tag; calculate the relevance between keywords in the keyword combination and the tag; relevance is positively correlated with match degree; calculate the importance of each keyword; importance is positively correlated with relevance.

[0011] The sum of the importance of all keywords in each decision tree is taken as the importance of the corresponding decision tree, and the ratio of the importance of each decision tree to the sum of the importance of all decision trees is taken as the weight of the corresponding decision tree.

[0012] The correlation between each keyword and tag is calculated, and the importance of the keywords is calculated based on the correlation. This ensures that when using keywords to analyze web logs, the phenomenon of polysemy and polysemy is fully considered. It is also convenient to integrate the log prediction results based on the importance of keywords to obtain the final prediction result, thereby improving the accuracy of the prediction result.

[0013] Preferably, the expression for the match degree between keyword combinations and tags is:

[0014]

[0015] In the formula, Let represent the matching degree between the i-th keyword combination and the tag, and exp represent an exponential function with base e. This represents the entropy value of the tag corresponding to the i-th keyword combination.

[0016] By using the entropy value of the tag to reflect the matching degree between the keyword combination and the tag, the matching degree is quantified and the accuracy of the calculation results is improved.

[0017] Preferably, the expression for the match degree between keyword combinations and tags is:

[0018]

[0019] In the formula, Let Ln represent the matching degree between the i-th keyword combination and the tag, and let Ln represent the logarithmic function with base e. α represents the entropy value of the tag corresponding to the i-th keyword combination, and α is a preset hyperparameter.

[0020] Preferably, the expression for the relevance between keywords and tags is:

[0021]

[0022] in, This represents the relevance between the j-th keyword and the tag. This represents the number of keyword combinations containing the j-th keyword. This represents the matching degree between the i-th keyword combination containing the j-th keyword and the tag.

[0023] By calculating the relevance between keywords and tags, a theoretical basis is provided for calculating the importance of keywords, which facilitates the analysis of web logs using keywords to determine whether web logs are abnormal.

[0024] The preferred method for calculating keyword importance is as follows:

[0025] The TF-IDF value of a keyword is calculated using the following expression to determine its importance:

[0026]

[0027] in, This indicates the importance of the j-th keyword. This represents the TF-IDF value of the j-th keyword. This represents the relevance between the j-th keyword and the tag. This represents the normalization function.

[0028] When calculating the importance of keywords, the TF-IDF value of the keywords is considered, as well as the correlation between keywords and tags. The more stable the relationship between keywords and tags, the greater the correlation, and the greater the importance of the keyword. By considering the correlation, the phenomenon of polysemy or polysemy is solved to some extent, thus improving the accuracy of web log analysis results.

[0029] The preferred method for calculating keyword importance is as follows:

[0030] The TensorFlow (TF) score for keywords is calculated using the following expression to indicate their importance:

[0031]

[0032] in, This indicates the importance of the j-th keyword. This represents the mean of the TF values ​​for the j-th keyword. This represents the relevance between the j-th keyword and the tag. This represents the normalization function.

[0033] Preferably, the expression for the network log prediction result is:

[0034]

[0035] in, This represents the final prediction result of the optimal random forest model for network logs. Indicate the importance of the k-th decision tree. This represents the prediction result of the k-th decision tree.

[0036] By using the weights of the decision tree to weight and integrate the prediction results, the final prediction result is obtained, making the prediction result more accurate.

[0037] Secondly, this invention provides a network log analysis system based on big data, employing the following technical solution:

[0038] A network log analysis system based on big data includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the network log analysis method based on big data described above.

[0039] The beneficial effect is that the above-mentioned network log analysis method based on big data generates a computer program, which is stored in the memory so that it can be loaded and executed by the processor. Thus, the system can be made according to the memory and processor, making it convenient to use.

[0040] The present invention has the following technical effects:

[0041] The correlation between each keyword and tag is calculated, and the importance of the keyword is calculated based on the correlation. The weight of the decision tree is then obtained based on the importance of the keyword. Finally, the decision results of the decision tree are weighted and summed. This method fully considers the phenomenon of polysemy and polysemy when using keywords to analyze web logs. It is also convenient to integrate the prediction results of the logs based on the importance of the keywords to obtain the final prediction result, thereby improving the accuracy of the prediction results. Attached Figure Description

[0042] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts.

[0043] Figure 1 This is a flowchart of the network log analysis method based on big data according to the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] It should be understood that when the terms "first," "second," etc., are used in the claims, specification, and drawings of this invention, they are only used to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0046] This invention discloses a network log analysis method based on big data, referring to... Figure 1 This includes the following steps:

[0047] S1: Obtain multiple web logs and set tags for each web log, including normal and abnormal tags. Extract keywords from each web log and further obtain various keyword combinations.

[0048] Multiple web logs were collected and preprocessed to obtain processed web logs. Preprocessing included filtering, screening, classification, and completion of missing data. Then, tags were manually assigned to the processed web logs based on their characteristics, including normal and abnormal tags. Keywords were extracted from the web logs using the jieba word segmentation algorithm, and these keywords were combined to obtain various keyword combinations. It is understood that within the same web log post, the keywords and their corresponding tags are identical.

[0049] For example, for the first blog post, its corresponding tag is "normal". Using the jieba word segmentation algorithm, the keywords are extracted to obtain keywords a, b, c, and keyword combinations ab, ac, and bc. The tags corresponding to keywords a, b, c and keyword combinations ab, ac, and bc are all "normal". For the second blog post, its corresponding tag is "abnormal". Using the jieba word segmentation algorithm, the keywords are extracted to obtain keywords a, c, e, and keyword combinations ac, ae, and ce. The tags corresponding to keywords a, c, e and keyword combinations ac, ae, and ce are all "abnormal".

[0050] S2: Calculate the match degree between each keyword combination and the tag.

[0051] In one embodiment, the expression for the match degree between keyword combinations and tags is:

[0052]

[0053] In the formula, Let represent the matching degree between the i-th keyword combination and the tag, and exp represent an exponential function with base e. This represents the entropy value of the tag corresponding to the i-th keyword combination. A larger entropy value indicates a more chaotic distribution of tags corresponding to the keyword combination, and thus a lower degree of matching between the keyword combination and the tag. The matching degree reflects the degree of matching between the keyword combination and the tag; it can also be understood as the tag reflecting the correctness of the keyword combination. For example, for the keyword combination 'ac' and its corresponding normal tag, the calculated matching degree is 0.8, indicating that the tag for the keyword combination 'ac' is likely normal.

[0054] In one embodiment, the expression for the match degree between keyword combinations and tags is:

[0055]

[0056] In the formula, Let Ln represent the matching degree between the i-th keyword combination and the tag, and let Ln represent the logarithmic function with base e. α represents the entropy value of the tag corresponding to the i-th keyword combination, and α is a preset hyperparameter.

[0057] S3: Calculate the relevance between keywords and tags in the keyword combination. The relevance is positively correlated with the matching degree.

[0058] The expression for the relevance between keywords and tags is:

[0059]

[0060] in, This represents the relevance between the j-th keyword and the tag. This represents the number of keyword combinations containing the j-th keyword. This represents the match degree between the i-th keyword combination containing the j-th keyword and the tag. Relevance reflects the connection between keywords and tags; a higher relevance indicates a stronger connection between the keywords and tags.

[0061] For example, to calculate the relevance between keyword 'a' and tag 'normal', there are three keyword combinations containing keyword 'a': ab, ac, and ae. Among them, the matching degree between ab and normal is 0.8, the matching degree between ac and normal is 0.7, and the matching degree between ae and normal is 0.6. Therefore, the relevance between keyword 'a' and tag 'normal' is (0.8+0.7+0.6) / 3=0.7, which further illustrates that keyword 'a' and tag are likely to be normal.

[0062] S4: Calculate the importance of each keyword. Importance is positively correlated with relevance.

[0063] In one embodiment, the TF-IDF value of the keyword is calculated. The TF-IDF value is used to assess the importance of the keyword to the web logs. The calculation method is existing technology, and the calculation process will not be described in detail here. The expression for importance is:

[0064]

[0065] in, This indicates the importance of the j-th keyword. This represents the TF-IDF value of the j-th keyword. This represents the relevance between the j-th keyword and the tag. This represents the normalization function. In calculating keyword importance, the TF-IDF value of the keyword and the correlation between the keyword and tags are considered. The more stable the relationship between the keyword and tags, and the greater the correlation, the greater the keyword's importance. Correlation, to some extent, addresses the issue of TF-IDF values ​​not considering polysemy or polysemy, thus improving the accuracy of keyword importance calculations. Greater keyword importance indicates a greater impact of the corresponding keyword in analyzing web logs.

[0066] In one embodiment, the TF value of the keyword is calculated using existing techniques, and the calculation process will not be detailed here. The expression for importance is:

[0067]

[0068] in, This indicates the importance of the j-th keyword. This represents the mean of the TF values ​​for the j-th keyword. This represents the relevance between the j-th keyword and the tag. This represents the normalization function.

[0069] S5: Calculate the weight of each decision tree in the preset optimal random forest model.

[0070] An initial random forest model is constructed, which includes multiple decision trees. For the multiple web logs obtained, keywords are extracted from each web log, and labels are set for each web log. The keywords and corresponding labels of each web log are input into the initial random forest model to train the initial random forest model and obtain the optimal random forest model.

[0071] In the decision trees of the optimal random forest model, the sum of the importance of all keywords in each decision tree is taken as the importance of the corresponding decision tree, and the ratio of the importance of each decision tree to the sum of the importance of all decision trees is taken as the weight of the corresponding decision tree.

[0072] For example, the optimal random forest model has three decision trees. For the first decision tree... There are 3 keywords in total, and their importance is as follows: , , The importance of the first decision tree For the second decision tree There are 4 keywords in total, and their importance is as follows: , , , The importance of the first decision tree Similarly, the third decision tree is obtained. Importance .

[0073] The weights of the first decision tree are: The weights of the second decision tree are: The weights of the third decision tree are: .

[0074] S6: Input the real-time acquired network logs into the optimal random forest model, and use the weights of each decision tree to perform a weighted summation of the prediction results to obtain the final prediction result of the network logs.

[0075] The expression for the network log prediction result is:

[0076]

[0077] in, This represents the final prediction result of the optimal random forest model for network logs. Indicate the importance of the k-th decision tree. This represents the prediction result of the k-th decision tree.

[0078] The prediction result of the k-th decision tree is the probability of network log anomalies. The final prediction result of the optimal random forest model for network logs represents the final probability of network log anomalies. When the final prediction result is greater than the preset anomaly threshold, it indicates that the corresponding network log is in an abnormal state and the network device needs to be detected in time.

[0079] This invention also discloses a network log analysis system based on big data, including a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the network log analysis method based on big data according to this invention.

[0080] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0081] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device.

[0082] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

[0083] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for network log analysis based on big data, characterized in that, The method comprises the steps of: calculating the weight of each decision tree in the preset optimal random forest model, inputting the real-time acquired network log into the optimal random forest model, and performing weighted summation on the prediction results by using the weight of each decision tree to obtain the final prediction result of the network log; wherein the calculation method of the weight of each decision tree is: acquiring a plurality of network logs, and setting a label for each network log, the label including normal and abnormal; extracting keywords in each network log to further obtain a plurality of keyword combinations; calculating the matching degree between each keyword combination and the label, calculating the association degree between the keywords in the keyword combination and the label, and the association degree is positively correlated with the matching degree; calculating the importance of each keyword, and the importance is positively correlated with the association degree; summing the importance of all keywords on each decision tree as the importance of the corresponding decision tree, and taking the ratio of the importance of each decision tree to the sum of the importance of all decision trees as the weight of the corresponding decision tree. 2.The big data based network log analysis method according to claim 1, wherein, The expression of the matching degree between the keyword combination and the label is: In the formula, Indicates the first The degree of matching between keyword combinations and tags, where exp represents an exponential function with base e. Indicates the first The entropy value of the tag corresponding to a combination of keywords. 3.The big data based network log analysis method of claim 1, wherein, The expression of the matching degree between the keyword combination and the label is: In the formula, Indicates the first The degree of matching between keyword combinations and tags, where Ln represents the logarithmic function with base e. Indicates the first The entropy value of the tag corresponding to each keyword combination, where α is a preset hyperparameter. 4.The big data based network log analysis method of claim 1, wherein, The expression of the association degree between the keyword and the label is: wherein, represents the relevance of the key word and the tag, represents the number of key word combinations containing the key word, represents the matching degree between the key word combination and the tag containing the key word. 5.The big data based network log analysis method of claim 1, wherein, The calculation method of the importance of the keyword is: calculating the tf-idf value of the keyword, and the expression of the importance is: wherein, represents the importance of the th keyword, represents the tf-idf value of the th keyword, represents the relevance of the th keyword and the label, represents a normalization function. 6.The big data based network log analysis method of claim 1, wherein, The calculation method of the importance of the keyword is: calculating the tf value of the keyword, and the expression of the importance is: wherein, represents the importance of the th keyword, represents the mean of the tf value of the th keyword, represents the degree of association of the th keyword and the label, represents a normalization function. 7.The big data based network log analysis method of claim 1, wherein, The expression of the network log prediction result is: wherein, represents the final prediction result of the network log by the best random forest model, represents the importance of the th decision tree, represents the prediction result of the th decision tree, represents the total number of decision trees in the best random forest model.

8. A network log analysis system based on big data, characterized by, comprising: a processor and a memory, the memory storing computer program instructions, when the computer program instructions are executed by the processor, realizing the big data based network log analysis method according to any one of claims 1-7.

Images