Network security equipment log association analysis method based on deep learning algorithm

Abstract

The invention relates to the technical field of equipment log association analysis, and discloses a network security equipment log association analysis method based on a deep learning algorithm, which comprises the following steps: firstly, acquiring log data from network security equipment in real time through a log data acquisition tool supporting Syslog and SNMP protocols; cleaning, formatting and standardizing the collected original log data, and converting the log data in different formats into a uniform data structure; the method specifically comprises the following steps: standardizing field names and data types; data analysis is carried out, and feature analysis is carried out on the collected timestamps, IP addresses, port numbers, request types and response code data; the BERT model comprises an application layer, and the application layer outputs the results of the associated log entries recognized by the output layer and the association analysis. The mass data processing efficiency is improved, the correlation analysis accuracy is enhanced, the space-time correlation analysis capability is enhanced, and the adaptive learning capability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of device log correlation analysis, and in particular to a network security device log correlation analysis method based on a deep learning algorithm. Background Art

[0002] With the rapid development of information technology, the Internet has become an indispensable part of modern society. However, the popularity and widespread use of the Internet have also brought many security risks. In order to ensure the security of internal networks, enterprises and institutions usually install a variety of network security devices at the entrances and exits of the network, such as firewalls, intrusion detection systems (IDS), intrusion prevention systems (IPS), Web application firewalls (WAF), etc. These devices will generate a large number of logs to record network events during operation. By studying these multi-source heterogeneous log data and extracting network security events from them, the network security situation can be presented, so that managers can accurately grasp the network operation status and make correct decisions.

[0003] Network security device log correlation analysis is an important technology used to extract valuable information from large amounts of log data. In the early stages of network security device log correlation analysis, it mainly relied on manual operations and simple scripting tools. Administrators need to manually collect log files from different devices and then use text editors or simple scripts to filter, sort, and search. Although this method is intuitive, it is inefficient and error-prone, especially when faced with a large amount of log data. In addition, due to the lack of a unified log format standard, the log formats generated by different manufacturers and devices vary, which increases the difficulty of manual analysis. As the importance of network security becomes increasingly prominent, tools and platforms dedicated to log correlation analysis have begun to appear on the market. These tools and platforms have greatly improved the efficiency and accuracy of log correlation analysis and reduced the need for manual intervention. However, they still have certain limitations, such as the high cost of updating and maintaining the rule base and limited ability to identify new attack methods.

[0004] Traditional data association and fusion methods usually rely on manually coded rules and simple matching algorithms, such as keyword-based matching or parsing based on specific formats. These methods perform well when processing structured data, but are inefficient and error-prone when facing unstructured or semi-structured data (such as log files). Modern data integration technologies include ETL (Extract, Transform, Load) processes, data warehouse technology, and data lake architecture. These technologies are designed to extract data from multiple data sources, transform and load them to support data analysis and decision-making. For example, ETL tools such as Apache NiFi and Talend can help automate the data extraction and transformation process, but these methods require professional knowledge to configure and maintain the ETL process, increase the threshold for use, and commercial ETL tools and data warehouse solutions are usually expensive. Despite their shortcomings, these technologies provide valuable inspiration for the present invention, that is, to simplify the data integration process through automation and intelligent means. The present invention draws on the automation ideas of these technologies, but adopts intelligent association algorithms and dynamic fusion engines that are more suitable for the characteristics of log data.

[0005] The accuracy of traditional correlation analysis is not high: traditional log correlation analysis methods based on rules or statistical models are difficult to accurately identify complex attack patterns and cross-device coordinated attacks. These methods usually rely on predefined rules or thresholds and have limited ability to identify new or variant attack methods.

[0006] Limited log coverage: Many enterprises’ network devices and services generate only a limited amount of log data. These logs may not be written and managed according to unified specifications, making it difficult to effectively retrieve and analyze them to identify potential security threats or abnormal behaviors.

[0007] Low-quality or outdated information: Logs for some devices and services may contain incomplete or erroneous data, and some devices may not provide valuable information for security teams to refer to, affecting the accuracy and timeliness of log analysis.

[0008] Real-time issues: The rapid evolution of modern threats requires security analysts to be able to detect and respond to suspicious activities in a short period of time, which places higher demands on the real-time processing and analysis of log data.

[0009] Therefore, to address the above problems, a network security device log correlation analysis method based on deep learning algorithm is needed. Summary of the invention

[0010] The purpose of the present invention is to provide a network security device log correlation analysis method based on deep learning algorithm. The present invention improves the efficiency of mass data processing, enhances the accuracy of correlation analysis, strengthens the ability of spatiotemporal correlation analysis, enhances adaptive learning ability, and improves visualization and interactive analysis tools. By introducing advanced machine learning technology and innovative analysis methods, the needs of enterprises and institutions to fully understand the network and the network usage of personnel are solved.

[0011] The present invention is achieved in that:

[0012] The present invention provides a network security device log correlation analysis method based on a deep learning algorithm, which is specifically performed in the following steps:

[0013] S1: Collect log data from network security devices in real time through log data collection tools that support Syslog and SNMP protocols; for example, log collectors or open source tools such as Fluentd and Logstash are responsible for extracting log data from security devices;

[0014] S2: Clean, format and standardize the collected raw log data, and convert log data of different formats into a unified data structure; specifically, standardize field names and data types;

[0015] Specifically, use BERT's tokenizer to tokenize the log text, convert the text into a format that the model can understand, convert the tokenization result into the ID sequence required for model input, convert the log message into the input format required by the BERT model, including adding special [CLS] and [SEP] tags, and finally convert the original log data into a format that can be processed by the BERT model.

[0016] S3: Perform data analysis, perform feature analysis on the collected timestamp, IP address, port number, request type, and response code data; load the pre-trained BERT model from Hugging Face's model library or other sources, and specifically select BERT-BASE-UNCASED as the training model; during training, specifically include masking the language model, masking the words in the input text, and predicting these masked words through the BERT-BASE-UNCASED model to enable the model to understand the context and grammar of the text;

[0017] The BERT-BASE-UNCASED model is then used to predict the next sentence. Specifically, the model is used to determine whether two sentences are adjacent, and the model is used to understand the relationship and logic between sentences to predict the next sentence.

[0018] S4: Output the association result. The BERT model includes an application layer, and the application layer will output the associated log entries identified by the output layer, and the result of the association analysis will be output.

[0019] Further, the BERT model includes an input layer, wherein the input layer explicitly represents a single sentence and a pair of sentences in a token sequence, and all inputs are transformed into a format of [CLS]+sentence A+[SEP]+sentence B+[SEP];

[0020] In the input layer, only sentence A is input. At the same time, token_embedding, position_embedding, and segment_embedding are performed through the BERT model input layer, and they are added together to finally get embed_sum.

[0021] Furthermore, the BERT model also includes an intermediate layer, specifically composed of a self-attention layer+ADD&Normalize layer+FFN;

[0022] Among them, in the self-attention layer, for the input word vector x, three learnable matrix parameters are given, x is right-multiplied by the above matrices to obtain Q, K, and V, and the output matrix is ​​calculated as follows;

[0023]

[0024] Then in the ADD&Normalize layer, add x and then perform the Layer Norm operation;

[0025] In the FFN layer, two linear changes and a ReLU activation function are specifically performed, as shown in the following formula:

[0026] FFN(x)=max(0,xW1+b1)W2+b2.

[0027] Furthermore, the BERT model also includes an output layer; the output layer includes outputting the classification of sentences, and then fine-tuning the model, and applying the model to a specific task. During the fine-tuning process, the parameters of the model are updated according to the training data of the specific task; the model is fine-tuned according to the prepared data to enable it to classify text data. The output layer also performs association fusion, extracts features, inputs the encoded log data into the BERT model, obtains the embedded representation of each token, and uses the output of [CLS] token as the representation of the entire sequence, or performs a pooling operation on the embedding of all tokens, and uses the extracted features to perform similarity comparison and cluster analysis between logs, calculate the distance between feature vectors by cosine similarity, and identify related log entries.

[0028] Furthermore, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a network security device log correlation analysis method based on a deep learning algorithm as described in any one of the above items are implemented.

[0029] Furthermore, the present invention provides a computer-storable medium, wherein the computer-readable storage medium includes an embedded processing system and a stored program, which controls the network security device log correlation analysis method based on a deep learning algorithm to execute any one of the steps of the network security device log correlation analysis method based on a deep learning algorithm when the embedded system control program is running.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. Improved the efficiency of massive data processing, enhanced the accuracy of correlation analysis, strengthened the ability of spatiotemporal correlation analysis, improved adaptive learning capabilities, and improved visualization and interactive analysis tools. By introducing advanced machine learning technology and innovative analysis methods, it solves the needs of enterprises and institutions to fully understand the network and the network usage of personnel.

[0032] 2. By introducing advanced algorithms and data structures, the present invention can significantly improve the speed and efficiency of data processing, especially when processing large-scale data sets, and can quickly complete data cleaning, formatting and standardization. Utilizing the powerful feature extraction capabilities of deep learning models, the present invention can more accurately identify key information in log data, thereby improving the accuracy of association analysis and reducing false positives and negatives. Through the continuous dynamic parameter adjustment and model update mechanism of the adaptive learning module, the present invention can continuously adapt to new attack methods and network environments, effectively reduce the false alarm rate, and improve the reliability of the system. Through the automated feature extraction and association analysis process, it reduces dependence on manual intervention, reduces operating costs, and improves work efficiency.

[0033] The technical solution of the present invention brings significant beneficial effects to users by improving data processing efficiency, enhancing data analysis accuracy, reducing false alarm rate, improving system flexibility, saving costs, enhancing security and promoting technological innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It is understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 It is a device log data code diagram of the present invention;

[0037] Figure 3 It is a code diagram of BERT of the present invention for analyzing the correlation between texts;

[0038] Figure 4 It is a diagram of the BERT model architecture of the present invention;

[0039] Figure 5 This is the middle layer architecture diagram of the BERT model of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but is only for selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0041] See also Figure 1-Figure 5 The present invention provides a network security device log correlation analysis method based on a deep learning algorithm; the method is specifically performed in the following steps:

[0042] S1: Collect log data from network security devices in real time through log data collection tools that support Syslog and SNMP protocols; for example, log collectors or open source tools such as Fluentd and Logstash are responsible for extracting log data from security devices;

[0043] S2: Clean, format and standardize the collected raw log data, and convert log data of different formats into a unified data structure; specifically, standardize field names and data types;

[0044] Specifically, use BERT's tokenizer to tokenize the log text, convert the text into a format that the model can understand, convert the tokenization result into the ID sequence required for model input, convert the log message into the input format required by the BERT model, including adding special [CLS] and [SEP] tags, and finally convert the original log data into a format that can be processed by the BERT model.

[0045] S3: Perform data analysis, perform feature analysis on the collected timestamp, IP address, port number, request type, and response code data; load the pre-trained BERT model from Hugging Face's model library or other sources, and specifically select BERT-BASE-UNCASED as the training model; during training, specifically include masking the language model, masking the words in the input text, and predicting these masked words through the BERT-BASE-UNCASED model to enable the model to understand the context and grammar of the text;

[0046] The BERT-BASE-UNCASED model is then used to predict the next sentence. Specifically, the model is used to determine whether two sentences are adjacent, and the model is used to understand the relationship and logic between sentences to predict the next sentence.

[0047] S4: Output the association result. The BERT model includes an application layer, and the application layer will output the associated log entries identified by the output layer, and the result of the association analysis will be output.

[0048] In this embodiment, the BERT model includes an input layer, which explicitly represents a single sentence and a pair of sentences in a token sequence, and transforms all inputs into the format of [CLS]+sentence A+[SEP]+sentence B+[SEP];

[0049] In the input layer, only sentence A is input. At the same time, token_embedding, position_embedding, and segment_embedding are performed through the BERT model input layer, and they are added together to finally get embed_sum.

[0050] In this embodiment, the BERT model also includes an intermediate layer, specifically composed of a self-attention layer+ADD&Normalize layer+FFN;

[0051] Among them, in the self-attention layer, for the input word vector x, three learnable matrix parameters are given, x is right-multiplied by the above matrices to obtain Q, K, and V, and the output matrix is ​​calculated as follows;

[0052]

[0053] Then in the ADD&Normalize layer, add x and then perform the Layer Norm operation;

[0054] In the FFN layer, two linear changes and a ReLU activation function are specifically performed, as shown in the following formula:

[0055] FFN(x)=max(0,xW1+b1)W2+b2.

[0056] In this embodiment, the BERT model further includes an output layer; the output layer includes outputting classification of sentences, and the classification parameters are specifically shown in Table 1;

[0057] Table 1 Classification table of text data

[0058]

[0059]

[0060] Then the model is fine-tuned, and the model is applied to a specific task. During the fine-tuning process, the model parameters are updated according to the training data of the specific task; the model is fine-tuned according to the prepared data to enable it to classify text data. In the output layer, association fusion is also performed to extract features, and the encoded log data is input into the BERT model to obtain the embedded representation of each token and use the output of [CLS] token as the representation of the entire sequence, or the embedding of all tokens is pooled. The extracted features are used to compare the similarity between logs and perform cluster analysis. The distance between feature vectors is calculated by cosine similarity to identify related log entries.

[0061] The cluster analysis is as follows:

[0062]

[0063] Among them, C i is the i-th cluster, x j is the data point belonging to the cluster, μ i

[0064] is the center point of the cluster;

[0065] The cosine similarity is calculated as follows:

[0066]

[0067] Among them, A·B represents the dot product of A and B, and ||A|| and ||B|| represent the Euclidean norm (i.e., vector length) of A and B, respectively.

[0068] In this embodiment, Figure 2-Figure 3 ;Suppose there is a set of device log data; 2023-10-01 12:15:00-User'admin' logged out from IP'192.168.1.100' We preprocess the log data, including removing irrelevant information, standardizing timestamps, etc. Then, we use the BERT model to extract features from the log text. Through the features extracted by the BERT model, we can calculate the similarity between logs and identify related log entries. We can find the following associations:

[0069] Association 1: User 'admin' logged in -> executed dangerous command -> attempted to access restricted file ' / etc / passwd'

[0070] Association 2: User login (User'admin'logged in) -> User logout (User'admin'logged out) Based on the above association analysis, the following conclusions can be drawn:

[0071] During a specific time period, user 'admin' successfully logged in from IP address '192.168.1.100' and executed the dangerous deletion command sudo rm -rf / and subsequently tried to access the restricted file / etc / passwd.

[0072] User 'admin' then successfully logged out of the system. This information can help security administrators quickly locate potential security threats and take appropriate measures.

[0073] like Figure 3 , and here is a code example using BERT to analyze the correlation between texts:

[0074] In this embodiment, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, the steps of a network security device log correlation analysis method based on a deep learning algorithm as described in any one of the above items are implemented.

[0075] In this embodiment, the present invention provides a computer-storable medium, wherein the computer-readable storage medium includes an embedded processing system and a stored program, which controls the network security device log correlation analysis method based on a deep learning algorithm to execute any one of the steps of the network security device log correlation analysis method based on a deep learning algorithm when the embedded system control program is running.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention has various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A network security device log correlation analysis method based on deep learning algorithm, characterized by: Please follow the steps below: S1: Collect log data from network security devices in real time through log data collection tools that support Syslog and SNMP protocols; S2: Clean, format and standardize the collected raw log data, and convert log data of different formats into a unified data structure; specifically, standardize field names and data types; S3: Perform data analysis and feature analysis on the collected timestamp, IP address, port number, request type, and response code data; S4: Output the association result. The BERT model includes an application layer, and the application layer will output the associated log entries identified by the output layer, and the result of the association analysis will be output.

2. According to claim 1, a network security device log correlation analysis method based on deep learning algorithm is characterized in that: In step S2, BERT's tokenizer is used to tokenize the log text, and the tokenization result is converted into the ID sequence required for the model input. The log message is converted into the input format required by the BERT model, including adding special [CLS] and [SEP] tags. Finally, the original log data is converted into the format processed by the BERT model.

3. According to claim 1, a network security device log correlation analysis method based on deep learning algorithm is characterized in that: In step S3, a pre-trained BERT model is loaded from the model library of Hugging Face or other sources, and BERT-BASE-UNCASED is selected as the training model. During the training, a masked language model is used to mask words in the input text, and the masked words are predicted by the BERT-BASE-UNCASED model to enable the model to understand the context and grammar of the text. The BERT-BASE-UNCASED model is then used to predict the next sentence. Specifically, the model is used to determine whether two sentences are adjacent, and the model is used to understand the relationship and logic between sentences to predict the next sentence.

4. According to claim 3, a network security device log correlation analysis method based on a deep learning algorithm is characterized in that: The BERT model includes an input layer, which explicitly represents a single sentence and a pair of sentences in a token sequence, and transforms all inputs into the format of [CLS]+sentence A+[SEP]+sentence B+[SEP]; In the input layer, only sentence A is input. At the same time, token_embedding, position_embedding, and segment_embedding are performed through the BERT model input layer, and they are added together to finally get embed_sum.

5. According to claim 4, a network security device log correlation analysis method based on deep learning algorithm is characterized in that: The BERT model also includes an intermediate layer, specifically composed of self-attention layer + ADD & Normalize layer + FFN; Among them, in the self-attention layer layer, for the input word vector x, three learnable matrix parameters are given, x is right-multiplied by the above matrices to obtain Q, K, and V, and the output matrix is ​​calculated as follows; Then in the ADD&Normalizelayer layer, add x and then perform the Layer Norm operation; In the FFN layer, two linear changes and a ReLU activation function are specifically performed, as shown in the following formula: FFN(x)=max(0,xW1+b1)W2+b2.

6. According to claim 3, a network security device log correlation analysis method based on a deep learning algorithm is characterized in that: The BERT model also includes an output layer; the output layer includes outputting a classification of a sentence, and then fine-tuning the model to apply the model to a specific task. During the fine-tuning process, the parameters of the model are updated according to the training data of the specific task.

7. A network security device log correlation analysis method based on deep learning algorithm according to claim 6, characterized in that: In the output layer, association fusion is also performed to extract features. The encoded log data is input into the BERT model to obtain the embedding representation of each token and use the output of [CLS] token as the representation of the entire sequence, or the embedding of all tokens is pooled. The extracted features are used to compare the similarity and cluster analysis between logs, and the distance between feature vectors is calculated by cosine similarity to identify related log entries.

8. The method for network security device log correlation analysis based on deep learning algorithm according to claim 3 is characterized in that: The BERT model also includes an application layer, which outputs the associated log entries identified by the output layer and outputs the results of the association analysis.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a network security device log correlation analysis method based on a deep learning algorithm as described in any one of claims 1 to 8 are implemented.

10. A computer storable medium, characterized in that: The computer-readable storage medium includes an embedded processing system and a stored program, and when the embedded system control program is running, it controls the network security device log correlation analysis method based on deep learning algorithm to execute the steps of any one of claims 1-8.

Images