Method and system for automatically generating network threat traffic detection rules based on LLM

Abstract

The present invention relates to the field of network security technology, and discloses a method and system for automatically generating network threat traffic detection rules based on LLM. The system constructs a rule automatic generation framework based on a large language model for network threat traffic, including four modules: a prompt word module, a large language model module, a rule engine module, and a rule base management module; the four modules are executed in sequence, forming a complete link from traffic information input to automatic rule generation and verification application; the method integrates domain knowledge into the pre-training model through LoRA fine-tuning, small sample prompt word engineering, and a correction mechanism based on engine feedback, and verifies the validity of the rules automatically generated by the model through actual detection engines. The present invention can fully and effectively analyze network threat traffic samples, while reducing the requirements for training data and generating a rule set that can be actually applied, thereby improving network defense capabilities.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method and system for automatically generating network threat traffic detection rules based on an LLM (Large Language Model). Background Art

[0002] With the rapid development and widespread adoption of network technology, the number of attacks on global networks is also increasing. Detection rules are a crucial component of network intrusion detection engines, defining anomalous activity based on specific characteristics and patterns to identify potential threats. Detection rules offer the advantage of being easier to understand than intelligent models. However, current detection rules rely primarily on experience sharing from experts and the security community, as well as traditional automated generation methods, to drive updates. These methods are unable to adapt to the ever-changing network environment and the rapid evolution of attack techniques. Therefore, automatically generating high-quality detection rules has become a challenging task in ensuring network security.

[0003] Traditional automated detection rule generation methods, such as the longest common string, Apriori sequence mining algorithm, and genetic and genetic algorithms, have made progress in mining features from network traffic information. However, they suffer from significant shortcomings in processing high-dimensional data, ensuring real-time performance, reducing computational costs, and improving generalization capabilities. Furthermore, these methods are often limited to generating specific character sets or regular expressions and cannot directly generate complete detection rules, making them ineffective and inflexible in practical network security applications. To address these limitations, recent research has attempted to employ the GPT-2 language model to automatically generate complete Snort rules. By learning and imitating the latest rule base, the GPT-2 model understands the logic of rule writing, rather than relying solely on data-driven content mining. However, while this approach represents a significant step forward in knowledge incorporation, it does not fully consider the information contained in real-time network traffic samples, instead relying primarily on a database of existing rules. This approach limits the model's understanding of network traffic attack knowledge, thereby reducing the accuracy and usability of the generated rules and their generalization to new attack traffic.

[0004] The existing automatic rule generation has two main shortcomings, including:

[0005] (1) Incomplete rules: Mainstream methods lack the ability to automatically generate end-to-end rules. They can often only generate specific types of character sets or regular expressions, and fail to fully utilize all options for rule configuration. Rules generated by models like GPT-2 may contain grammatical errors, which makes it impossible for the detection engine to directly apply these rules.

[0006] (2) Inaccurate rules: Due to the rapid changes and evolution of network attack samples, existing methods fail to deeply understand network traffic and do not consider the information in real-time network traffic samples. This limits the model's understanding of knowledge related to network traffic attacks and lacks the ability to quickly and accurately capture attack features, thus affecting the accuracy and generalization of the rules. Summary of the Invention

[0007] To address the above issues, the present invention aims to provide a method and system for automatically generating network threat traffic detection rules based on LLM. This method can fully and effectively analyze traffic samples, while reducing the requirements for training data and generating a practical rule set, thereby improving network defense capabilities. The technical solution is as follows:

[0008] A method for automatically generating network threat traffic detection rules based on LLM includes the following steps:

[0009] Step 1: Represent the information of network threat traffic. For HTTP and DNS protocols, directly extract the payload information from the traffic as traffic text. For application layer protocols other than HTTP and DNS, directly extract the payload information of TCP and UDP protocols, encode it, and use it as input traffic text.

[0010] Step 2: Based on the input traffic text, the prompt word module selects a sample rule R according to the rule metadata retrieval and related rule matching algorithm, and adds the sample rule R to the input traffic text to form the complete model input information T+;

[0011] Step 3: Based on the model input information T+, the large language model is trained using a fine-tuning method: First, a specific large language model is selected, and then a corpus dataset is constructed around the network threat traffic detection rules. Then, LoRA fine-tuning is performed, and a low-rank matrix is ​​introduced into the weight matrix of the large language model. The pre-trained large language model is fine-tuned to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rules of the input text traffic.

[0012] Step 4: Clean and match the text information output by the large language model, extract the detection rules it generates, input them into the rule engine for verification, and use feedback correction technology to supplement the correction information and control the correction threshold.

[0013] Furthermore, step 2 specifically includes:

[0014] Step 2.1: Rule metadata retrieval

[0015] Metadata mapping and comparison is performed between the rule dictionary and the dictionary of network traffic header information, and key fields are filtered from the dictionary to achieve preliminary retrieval. For Snort rules, the threat type, network protocol, port number, threat level, and application service fields are matched with the input network threat traffic header information to filter out matching rules.

[0016] Step 2.2: Match relevant rules

[0017] Step 2.2.1: Preprocess the input traffic text and the content field and regular expression pcre field of the Snort rule configuration, including text cleaning and text segmentation; then convert the preprocessed text into a bigram sequence; for the input traffic text T, its bigram sequence set B0 is expressed as:

[0018]

[0019] Where w i is the i-th word in the input traffic text T; |T| represents the total number of words;

[0020] Step 2.2.2: For each bigram sequence b in B0, calculate its TF-IDF value to assess its importance in a specific document relative to the entire document set;

[0021] The TF value of the bigram sequence b in the document d is defined as:

[0022]

[0023] Where, f b,d is the number of times the bigram sequence b appears in document d, and the denominator is the sum of the number of times it appears in all bigrams sequences in document d;

[0024] For a bigram sequence b, its IDF value is calculated as follows:

[0025]

[0026] Where N is the total number of documents in the document set D, and the denominator is the number of documents containing the bigram sequence b;

[0027] Combining the above TF and IDF values, the TF-IDF value of the bigram sequence b in document d is:

[0028] TF-IDF(b,d,D)=TF(b,d)*IDF(b,D)

[0029] Step 2.2.3: Use the TF-IDF value calculated above to match the input traffic text with the content field and the regular expression pcre field in the Snort rule configuration;

[0030] The matching degree between the two is evaluated by calculating the cosine similarity between the TF-IDF vectors of the input traffic text and each Snort rule text:

[0031]

[0032] Where, and are the TF-IDF vectors of T1 and T2 respectively, · represents the vector dot product, and are vectors and The norm of; T1 and T2 are the input traffic text and Snort rule text respectively;

[0033] By comparing the similarity between the input traffic text and each Snort rule text, the candidate Snort rule most relevant to the input traffic is identified;

[0034] Step 2.2.4: Set the hyperparameter K based on the small sample prompt example, retain the top K rules, that is, the sample rules R, and construct the model input information T+ in the order of task description, sample example, and task input, which is used as the input of the large language model module.

[0035] Furthermore, the corpus construction in step 3 specifically includes:

[0036] Step 3.1.1: Construct D1 dataset

[0037] First, we collect all detection rules currently available in the community. Then, we use prompt words to preset model roles and tasks. Then, we input these detection rules into a large-scale language model to generate traffic feature descriptions corresponding to the rules.

[0038] The model input information T+ generated by the prompt word module is used as the input of the initial untrained general model, and the corresponding rules are used as the output. The prompt words of additional downstream tasks are added to form the D1 dataset containing the "traffic feature description-detection rule" relationship;

[0039] Step 3.1.2: Construct the D2 dataset

[0040] First, the latest network attack traffic samples are collected. Then, the detection engine is actually executed to collect the triggered rules. After processing, the data relationship between "traffic text representation-detection rules" is obtained.

[0041] The input network traffic samples are then read at the network protocol layer, classified and extracted from payload information, and processed into natural language descriptions to adapt to the text input mode of the language model.

[0042] All traffic includes UDP and TCP, and its application layer protocols include DNS and HTTP. For the DNS protocol, DNS operation and query information are manually extracted and spliced ​​into natural language descriptions. For the HTTP protocol, its header fields are parsed and key fields are spliced. For requests with binary payloads, the hexadecimal payload text of the set length is retained and the content after the limited length is truncated.

[0043] By analyzing the log files output by the detection engine after detection, the alarm information and corresponding rule entries are extracted, and the corresponding five-tuple traffic and detection rules are used. Finally, the textual traffic and matched rules are organized into input and output to form the D2 data set.

[0044] Furthermore, the model fine-tuning in step 3 specifically includes:

[0045] Step 3.2.1: Formalize the large language model fine-tuning process as:

[0046] h=W0χ+ΔWχ;

[0047] Where ΔW is the parameter corresponding to the fine-tuning update, h is the output feature vector, W0 is the pre-training model weight, and χ is the input text sequence;

[0048] Step 3.2.2: Use two low-rank matrices A and B to approximate the parameters ΔW corresponding to the fine-tuning update:

[0049] Set the approximate parameters Its parameter scale is a d*d matrix. The rank of the low-rank matrices A and B is r, so the size of the low-rank matrix A is adapted to The size of the low-rank matrix B is adapted to The number of fine-tuning parameters is reduced from d*d to 2*r*d without affecting the model structure. The update fine-tuning is expressed as:

[0050]

[0051] Where h′ is the feature vector output after fine-tuning; α is a hyperparameter that represents the scale of low-rank matrix scaling;

[0052] Step 3.2.3: During the LoRA fine-tuning process, the low-rank matrices A and B are first initialized. Then, under the setting of the hyperparameter rank r, the large language model learns the parameter values ​​of the low-rank matrices A and B and extracts the matrix rows and columns with the richest information, ignoring redundant information to achieve the purpose of reducing parameters.

[0053] To determine the appropriate r value, the α hyperparameter is introduced. After determining the optimal approximation effect, the α value is fixed, and then the r value is continuously reduced to obtain a large language model with a closer approximation effect, more concise information, and a smaller rank.

[0054] Furthermore, the step 4 is specifically as follows:

[0055] First, regular expressions are used to match specific characters and formats, and then the generated rules are located to obtain a complete rule. The extracted rules and network threat traffic are then simultaneously input into the rule engine for verification. The validity of the rules is determined by parsing the output results.

[0056] Among them, for feedback on grammatical errors, the edit distance is used to check its format changes, and automatic correction tools in specific formats are used to assist in improving the rules; for feedback on detection effectiveness, the effectiveness of the feedback is judged based on the similarity between rules; if the generated rules do not have enough changes in content, the current feedback information is considered useless to the model, and manual judgment by experts or humans is introduced.

[0057] Furthermore, cosine similarity is used to measure the similarity of rules:

[0058] Convert the Snort rule text into a vector using TF-IDF, where each dimension represents the frequency and weight of a phrase in a bigram sequence b; the cosine similarity calculation formula is:

[0059]

[0060] Among them, X and Y are the two rule texts to be compared, X i and Y i are the bigram sequence phrases corresponding to the rule text, and n is the number of sequences.

[0061] Furthermore, the step 4 further includes a step 5:

[0062] All detection rules are managed through the rule base module, and rule update capabilities are provided to the rule engine, and rule retrieval capabilities are provided to the prompt word module;

[0063] When the rule engine successfully generates a detection rule through the large language model, it updates the detection rule to the rule base;

[0064] Use the new detection rules in the rule base as prompt word examples to facilitate the next automatic rule generation.

[0065] An LLM-based network threat traffic detection rule automatic generation system includes:

[0066] Prompt word module: selects sample rules R based on rule metadata retrieval and related rule matching algorithms, and adds sample rules R to the input traffic text to form complete model input information T+;

[0067] Large language model module: Based on the model input information T+, the large language model is trained using a fine-tuning method: first, a specific large language model is selected, and then a corpus dataset is constructed around the network threat traffic detection rules. Then, LoRA fine-tuning is performed, introducing a low-rank matrix into the weight matrix of the large language model, and fine-tuning the pre-trained large language model to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rules of the input text traffic;

[0068] Rule engine module: This module cleans and matches the text information output by the large language model, extracts the generated detection rules, inputs them into the rule engine for verification, and uses feedback correction technology to supplement correction information and control correction thresholds.

[0069] Rule base module: manages all detection rules, provides rule update capabilities for the rule engine, and provides rule retrieval capabilities for the prompt word module. When the rule engine successfully generates a detection rule through the large language model, it updates the detection rule to the rule base. The new detection rule in the rule base is used as a prompt word example to promote the automatic generation of the next rule.

[0070] The beneficial effects of the present invention are:

[0071] 1) This paper constructs a framework for automatic rule generation based on a large language model, divided into four modules: a prompt word module, a large language model module, a rule engine module, and a rule base management module. These four modules are executed sequentially, forming a complete chain from traffic information input to automatic rule generation and application. Through LoRa fine-tuning, small sample prompt word engineering, and a correction mechanism based on engine feedback, domain knowledge is integrated into the pre-trained model. The validity of the rules generated by the model is verified through actual detection engines.

[0072] 2) Experimental results show that the automatic generation of network threat traffic detection rules based on a large language model proposed in this invention can fully and effectively analyze traffic samples, while reducing the requirements for training data and generating a rule set that can be actually applied, thereby improving network defense capabilities.

[0073] 3) The overall process of this invention includes three key technical points: model fine-tuning, rule correlation analysis, and feedback correction technology (including correction information supplementation and correction threshold control). These three technologies act in different modules and work links, promoting the adaptation of large language models in the field of automatic traffic rule generation from different levels, improving the performance of the model, and improving the quality of the generated rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The figure is a schematic diagram of the overall process of the automatic rule generation framework based on the large language model in the present invention.

[0075] FIG2( a ) is a schematic diagram of the process of constructing the training corpus and data set (D1 data set) in the present invention.

[0076] FIG2( b ) is a schematic diagram of the process of constructing the training corpus and data set (D2 data set) in the present invention.

[0077] FIG3( a ) is a schematic diagram of the LoRA-based fine-tuning method in the present invention.

[0078] FIG3( b ) is a schematic diagram of the LoRA-based fine-tuning method in the present invention.

[0079] Figure 4 This is a graph showing the performance analysis results of fine-tuning the large language model in the present invention.

[0080] Figure 5 This is a visualization analysis chart of the fine-tuning performance analysis results of the large language model in this invention.

[0081] Figure 6 This is an example diagram showing the automatic generation of rules before and after fine-tuning the large language model in the present invention.

[0082] Figure 7 This is a visualization analysis diagram of the performance analysis results of the large language model prompt word engineering in the present invention.

[0083] Figure 8 This is a visualization analysis diagram of the performance analysis results of the large language model feedback mechanism in the present invention. DETAILED DESCRIPTION

[0084] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] The present invention provides a method for automatically generating network threat traffic detection rules based on a large language model.

[0086] This method builds a rule automatic generation framework based on a large language model for network threat traffic, including technologies such as LoRA fine-tuning, small sample prompt word engineering, and a correction mechanism based on engine feedback. It integrates domain knowledge into the pre-trained model, uses the large language model to automatically generate rules end-to-end, and verifies the effectiveness of the rules automatically generated by the model through actual detection engines.

[0087] like Figure 1 The figure shows the overall process flow of the automatic rule generation framework based on a large language model. The overall process of the method of the present invention is divided into four modules: a prompt word module, a large language model module, a rule engine module, and a rule base management module. These four modules are executed sequentially, forming a complete chain from network threat traffic information input to automatic rule generation and application.

[0088] The three technologies include model fine-tuning, rule correlation analysis, and feedback correction technology (including correction information supplementation and correction threshold control). These three technologies act in different modules and work links, promoting the adaptation of large language models in the field of automatic traffic rule generation from different levels, improving the performance of the model, and improving the quality of the generated rules.

[0089] The four modules, combined with input traffic information, complete a complete input process. The first step is to represent network threat traffic information. Network threat traffic has certain textual characteristics, but it is not entirely natural language text. To enable the large language model to understand traffic information and reduce differences in encoding and modality, traffic information in PCAP format must first be preprocessed.

[0090] For common HTTP and DNS protocols, the payload information is directly extracted from the traffic as traffic text. For other application layer protocols, the payload information of the TCP and UDP protocols is directly extracted, encoded in hexadecimal format, and used as the input traffic text T.

[0091] 1. Prompt word module

[0092] The prompt word module, based on input traffic information, supplements domain knowledge to construct the complete input text. Large language models have acquired a wealth of general knowledge through pre-training on large datasets. Prompt words can activate the weights of certain important textual information for the large language model, thereby acting as a domain expert, or allowing the large language model to learn and generalize rapidly from a small number of samples. Therefore, the prompt word module includes several methods for constructing input, including general prompt words and prompt words that provide small sample examples. Depending on the sample selection method, these techniques can be divided into random selection, TF-IDF algorithms that consider rule keyword weights, and embedding algorithms that consider rule semantic similarity. These algorithms retrieve corresponding rule examples from the rule library based on the results of the relevance calculation to construct a complete input.

[0093] Based on the input traffic text T, the prompt word module further supplements the relevant example rules R to form the complete model input information T+. The prompt word module selects the example rules R in three ways: random selection, TF-IDF algorithm that considers the rule keyword weight, and Embedding algorithm that considers the semantic similarity of the rules. Since the prompt word is part of the model input, the length of the input content will be limited by the length of the model sequence. Considering that the text length of traffic information itself is relatively long, the rule examples that can be provided will be limited to about 1-5 examples. Therefore, the algorithm of this method mainly includes two parts: rule metadata retrieval and related rule matching.

[0094] (1) Rule metadata retrieval: Metadata mapping and comparison is performed between the rule dictionary and the dictionary of network traffic header information, and key fields are stably filtered from the dictionary to achieve preliminary retrieval. Taking Snort rules as an example, fields such as threat type (classtype), network protocol (protocol), port number (port), threat level (priority), and application service (service) can be matched with the input network threat traffic header information to filter out matching rules.

[0095] (2) Relevance rule matching: This step focuses on fields such as the rule content text and the matching regular expression (pcre). The present invention selects TF-IDF (Term Frequency-Inverse Document Frequency) as the main relevance matching method. The process is as follows:

[0096] First, the input traffic text and the content field and regular expression pcre field of the Snort rule configuration are preprocessed, including text cleaning and text segmentation. Then, the preprocessed text is converted into a bigram sequence. For text T, its bigram sequence set B0 can be expressed as:

[0097]

[0098] Where w i is the i-th word in the text T. For each bigram sequence, its TF-IDF value is calculated to evaluate its importance in a specific document (input traffic text or Snort rule) relative to the entire document set (all input traffic text and Snort rules). The TF value of a bigram sequence b in a document d is defined as:

[0099]

[0100] Where, f b,d is the number of times the bigram sequence b appears in document d, and the denominator is the sum of the number of occurrences of all bigrams in document d. For the bigram sequence b, its IDF value is calculated as follows:

[0101]

[0102] Where N is the total number of documents in the document set D, and the denominator is the number of documents containing the bigram sequence b. To avoid the denominator being zero, 1 is usually added to the denominator. Combining the above TF and IDF, the TF-IDF value of the bigram sequence b in document d is:

[0103] TF-IDF(b,d,D)=TF(b,d)*IDF(b,D)

[0104] Using the TF-IDF weights calculated above, match the input traffic text with the content field and the regular expression pcre field in the Snort rule configuration. Specifically, the cosine similarity between the TF-IDF vectors of the input traffic text and each Snort rule text can be calculated to evaluate the matching degree between them:

[0105]

[0106] Where, and are the TF-IDF vectors of T1 and T2 respectively, · represents the vector dot product, is a vector The norm of .

[0107] By comparing the similarity between the input traffic text and Snort rules, we can identify the Snort rules most relevant to the input traffic, thus achieving high-precision matching. This method based on bigrams and TF-IDF can effectively capture and utilize the local sequence information of the text, improving the accuracy and relevance of the matching process.

[0108] After completing rule metadata retrieval and matching related rules, we obtain the most relevant candidate rules. We then set the K hyperparameter based on the small sample prompt example, retain the top K rules, and construct the query input in the order of task description (including task output format requirements), sample examples, and task input. This query input is then fed into the large language model.

[0109] 2. Large Language Model Module

[0110] The large language model module is a processing module that generates detection rules based on input text information.

[0111] The main considerations are: First, data security and privacy protection are important requirements in the field of network security. The ability to achieve independent deployment is an important prerequisite. Closed-source or closed-operation platforms are not suitable for direct use. Second, automatic generation of detection rules is a subdivision function in a dedicated field. Smaller-scale models should achieve better results after domain knowledge transfer. Overly large-scale models will lead to excessively high computing and operating costs, and lose practical application benefits.

[0112] The present invention mainly adopts a fine-tuning approach to train the large language model, which mainly includes: basic model selection, corpus construction and LoRA fine-tuning.

[0113] (1) Basic model selection

[0114] For comparison, several open source smaller language models were selected, including LlaMA, BloomZ, DeepSeeker, and ChatGLM.

[0115] (2) Corpus construction

[0116] Two corpus datasets, D1 and D2, were constructed around network threat traffic detection rules, such as Snort rules and Suricata rules. The two corpus datasets are based on the perspectives of rule interpretability and rule applicability, respectively. By combining these rules with actual network traffic data, they reflect the model's understanding of traffic and rules, which helps the model learn how to identify specific patterns in network traffic.

[0117] In the field of network threat traffic detection rules, rules are the conclusions drawn by community members or domain experts after analyzing network attacks. They are shared within the community for discussion and use to address ever-changing network attacks. Common communities include Snort rules and Emerging Threats. However, these communities only publish the rules themselves, lacking corresponding traffic samples. Therefore, there are no directly usable samples that demonstrate the correspondence between traffic and rules to support model fine-tuning.

[0118] The D1 dataset construction process is described in Figure 2(a), including the prompt words used, the rules of the examples, and the generated text. The core knowledge of the D1 dataset is the understanding of the rule text and traffic characteristics, and on this basis, a mapping between the two is formed. In order to achieve the integration of this knowledge, all the detection rules currently open to the community were first collected, and then the model roles and tasks were preset through the prompt words. These detection rules were then input into large-scale language models, such as ChatGPT4 and ChatGLM4, to generate traffic feature descriptions corresponding to the rules. These large-scale language models have stronger understanding and reasoning capabilities, and the text quality generated is also high. They have also been manually verified in practical tests: given valid prompt words, they have the ability to understand Snort detection rules. The main advantages of using a large language model to complete this task include: (1) no need to design labels. Since it is a generation task, only the rule text needs to be provided, which reduces the requirements for data work; (2) it conforms to the model thinking chain method: the large language model can work better under the thinking chain working method. Therefore, in the process of automatic rule generation, the model can also be guided to use the thinking chain method to perform one-step analysis and generation. It can be roughly summarized as: understanding of traffic text - mining of traffic features - mapping of rules. In this thinking chain, traffic features to rules is a crucial step. Using a large-scale large language model to generate traffic feature descriptions can provide other smaller-scale language models with better training corpus in a way similar to distillation, thereby guiding the smaller-scale language models to enhance their rule mapping capabilities. Based on this, after processing the output of the model, the generated text is used as input, the corresponding rules are used as output, and additional prompt words for downstream tasks are added to form the D1 dataset, which contains the "traffic feature description-detection rule" relationship.

[0119] Figure 2(b) illustrates the D2 dataset construction process, including network attack types, detection engines, rule examples, and dataset sample examples. First, the latest network attack traffic samples are collected. The detection engine is then executed, triggering rules are collected, and processing yields the data relationship between "traffic text representation and detection rules." Since the traffic source format (PCAP) is a binary file, it must be correctly parsed to obtain its content. To adapt to the language model's text input mode, the input requires certain processing, including reading the network protocol layer, classifying and extracting payload information, and providing a natural language description. All traffic is primarily categorized into UDP and TCP, with DNS and HTTP being the primary application layer protocols. For the DNS protocol, DNS operation and query information are manually extracted and concatenated into a natural language description. For the HTTP protocol, header fields are parsed, key fields are extracted, and concatenated. For requests with binary payloads, a certain length of hexadecimal payload text is retained, and content after the specified length is truncated. The detection engine can use either Snort or Suricata. Both are open-source engines, easy to deploy, and highly efficient. Suricata is compatible with Snort detection rules. Therefore, this method uses the Snort engine as an example. After detection, the detection engine outputs several log files. By analyzing the log files and extracting the alarm information and corresponding rule entries, we can identify the corresponding five-tuple traffic and detection rules. Finally, the textual traffic and matched rules are organized into input and output, forming the D2 dataset.

[0120] It is worth noting that, in both the D1 and D2 datasets, in order to reduce the input pressure of the model and maintain the generalization ability of the model, all rules need to be cleaned to remove some explanatory text, especially the description under the "meta" field of the rule, which usually includes the reference URL, modification time, creation time, etc.

[0121] (3) LoRA fine-tuning

[0122] Large language models have been trained on a large number of predictions and possess a certain level of semantic understanding and reasoning capabilities. However, many current models tend to focus on natural language text or code corpora, and have not been fully trained for network security, particularly for predictions of Snort rules. To enhance the model's adaptability to the specific task of automatic rule generation, this paper constructs a corresponding dataset and combines it with LoRA fine-tuning technology to perform additional model training.

[0123] The present invention chooses to use LoRA to fine-tune the pre-trained model, as shown in Figures 3(a) and (b). Compared with the fine-tuning of the full parameters, the core idea is to achieve fine-tuning by introducing low-rank matrices A and B in the weight matrix of the model. LoRA introduces a low-rank matrix in the bypass of the original pre-trained matrix, allowing adaptation to any network layer, and only adjusts the weights in a specific model layer (such as the self-attention layer or the feedforward network layer), such as any W_q, W_k, W_v in the Transformer. A common operation is to perform low-rank adaptation on some parameters in the Attention layer. However, LoRA introduces a low-rank matrix, and its effect is not constant on different tasks and data sets. Therefore, it is necessary to perform specific optimization on the rule automatic generation task based on the LoRA method, and select the size and position of the LoRA parameters to find the best model adjustment strategy. The model fine-tuning process can be formally expressed as:

[0124] h=W0χ+ΔWχ;

[0125] Where ΔW is the parameter corresponding to the fine-tuning update. The core algorithm of LoRA uses two low-rank matrices A and B to approximate ΔW. Its parameter scale is a d*d matrix. The rank of the low-rank matrix is ​​r, so the size of the A matrix is ​​adapted to The size of the B matrix is ​​adapted to Then the number of fine-tuning parameters will be reduced from d*d to 2*r*d. Without affecting the model structure, the update fine-tuning is expressed as:

[0126]

[0127] Where α is a hyperparameter, which indicates the scale of low-rank matrix scaling. During the LoRA fine-tuning process, the two matrices A and B will be initialized first, and then the model will learn the parameter values ​​of the low-rank matrices A and B under the setting of the hyperparameter rank r, to achieve the effect similar to SVD decomposition, extract the matrix rows and columns with the richest information, ignore redundant information, and achieve the purpose of reducing parameters. Furthermore, in order to determine the appropriate r value, the α hyperparameter is introduced. After determining the optimal approximation effect, the α value is fixed, and then r is continuously reduced to obtain a model with similar approximation effect, but with more streamlined information and a smaller rank. In this process, The value continues to increase, which plays the role of amplifying the weight of the low-rank matrix and accelerating the gradient descent.

[0128] At this point, the large model module has fine-tuned the model using LoRA to adapt it to the specific task. The model is then saved and used to analyze input text, automatically generate rules, and generate detection rules corresponding to the specific input traffic. However, to further verify the usability and initial effectiveness of the generated rules, the rules are passed to the rule engine module.

[0129] 3. Rule Engine Module

[0130] The rule engine module cleans and matches the text output by the large language model, extracts the generated detection rules, and inputs them into the rule engine for verification. This module primarily utilizes feedback correction technology, including correction information supplementation and correction threshold control.

[0131] First, regular expressions are used to match specific characters and formats, such as the "' syntax in Markdown and the "rule:" prefix required by prompts. The generated rules are then located to obtain a complete rule. The extracted rules and network threat traffic are then simultaneously input into the rule engine for verification. The validity of the rules is determined by parsing the output results (such as alarm logs and execution logs).

[0132] Among them, for grammatical error feedback, the edit distance is used to check its format changes, and a certain format automatic correction tool is used to assist in improving the rules; for detection effectiveness feedback, it is necessary to judge whether the feedback is effective based on the similarity between the rules. If the generated rules do not have enough changes in content, the current feedback information is considered useless for the model, and it is necessary to introduce expert or manual judgment. The present invention mainly uses cosine similarity for similarity measurement: the Snort rule text is converted into a vector, and TF-IDF is also used for implementation. Each dimension represents the frequency and weight of a bigram sequence b phrase. The cosine similarity calculation formula is:

[0133]

[0134] Among them, X and Y are the two rule texts to be compared, X i and Y i are the bigram sequence phrases corresponding to the rule text, and n is the number of sequences.

[0135] In actual operation, testing is required to optimize three thresholds: the edit distance threshold, the similarity threshold, and the number of feedback thresholds. The first two thresholds are used to determine how well the model accepts feedback; lower thresholds allow for more attempts. However, given the decline in model performance after multiple rounds of conversations and the computational cost of the model, setting the thresholds too low is not appropriate. In practice, the edit distance is set to 5, the similarity threshold is set to -0.7, and the number of feedback threshold is set to 5.

[0136] 4. Rule base module

[0137] The rule base module manages all detection rules. It provides rule update capabilities for the rule engine and rule retrieval capabilities for the prompt word module. When the rule engine successfully passes a rule generated by the large language model, the rule is updated to the rule base. The new rule may be used as a prompt word example to facilitate the next automatic rule generation.

[0138] 5. Experimental Evaluation

[0139] In order to verify the technical effect of the technical solution proposed by the present invention, the following experimental evaluation was carried out.

[0140] The experiment used two datasets, named D1 and D2, to train and test the model. The statistical results are shown in the table. The D1 dataset was classified and counted according to Snort's Classtype, while the D2 dataset was classified and counted according to actual attack methods.

[0141] Table 1 Dataset construction statistics

[0142]

[0143] The D1 dataset is an Explain-Rule dataset. Its construction principle is to first collect Snort and Suricata rule data shared by the open source community, then use mature large-scale language models (such as ChatGPT3, ChatGPT4, and ChatGLM4) to explain the rules. The D1 dataset is then constructed using the explained text as input and the rules as output. This experiment uses ChatGLM4 to complete this explanation and knowledge distillation process, with a total of 50,000 rule samples. The prompt words used are as follows:

[0144] “You are a cybersecurity expert familiar with Suricata rules and traffic detection. Please analyze the key recognition features and corresponding detection options of target detection traffic according to the given rules,limited to 100 words.”

[0145] The D2 dataset is a dataset of traffic samples and corresponding detection rules (Traffic-Rule). Its construction principle is to collect the latest network threat traffic sample files, input them into an open-source detection engine, then collect the triggered rules and corresponding traffic samples. After preprocessing the traffic, a data set corresponding to traffic information and rule information is formed. This experiment mainly collected 15,000 network threat traffic samples, which actually triggered 1,518 alerts, a total of 181 rules, and covered 22 categories.

[0146] This experiment designed three types of evaluation metrics to assess the experimental results: model generation quality, speed, and actual performance, as shown in Table 2. Model generation quality primarily refers to relevant indicators in the field of natural language processing, including BLEU (Bilingual Evaluation Understudy), ROUGE-1, ROUGE-2, and ROUGE-L. These metrics are commonly used for machine translation quality assessment. In this experiment, they were used to evaluate the match between automatically generated rule text and standard rule text.

[0147] Table 2 Evaluation indicators of rule-based automatic generation experiments

[0148]

[0149] (1) Comparison of basic model capabilities and performance after fine-tuning

[0150] Attachment Figure 4This report presents the performance of different large-scale language models (ChatGLM3-6B, DeepSeek-Coder-6B, BloomZ-7B, and LLaMA2-7B) before and after fine-tuning on the automatic rule generation task. The main focus is on changes in the BLEU, ROUGE-1, ROUGE-2, and ROUGE-L metrics, sample processing time (seconds), and performance differences on specific tasks (D1: interpretation-rules; D2: flow-rules). Statistics are also included for content hit rate, system load time, and execution detection time. The results in the figures reveal the following key findings:

[0151] 1) Significant fine-tuning results: Comparing the baseline and fine-tuned results, all models show significant improvements in BLEU, ROUGE-1, ROUGE-2, and ROUGE-L metrics after fine-tuning. This demonstrates that fine-tuning is highly effective in improving model performance on automatic rule generation tasks.

[0152] 2) Improved Model Performance: After fine-tuning, BloomZ-7B demonstrated the most significant performance improvement, achieving the highest BLEU and ROUGE scores among all models on both the D1 and D2 tasks. In particular, on the D2 task, BLEU reached 82.9181 and ROUGE-L reached 86.2796, demonstrating exceptionally high generation quality.

[0153] 3) Sample Processing Time: The fine-tuned models took longer to process each sample, likely due to the increased complexity and computational resources required. The increase in sample processing time for BloomZ-7B and LLaMA2-7B after fine-tuning was relatively small, indicating that these models maintained improved performance while maintaining excellent efficiency.

[0154] 4) Content hit rate, system loading and execution detection: BloomZ-7B and DeepSeek-Coder-6B have significantly improved their content hit rate, system loading time, and execution detection time after fine-tuning, indicating that these models not only improve generation quality but also optimize content relevance and system response time.

[0155] Attachment Figure 5 The basic capabilities of each model and the evaluation indicators after fine-tuning are visualized. It can be seen that the changing trends of different models in different indicators are significantly different. Therefore, the selection of which model to fine-tune needs to consider the specific requirements of the task. For example, if the task focuses on the quality of generation, BloomZ-7B is an excellent choice. It has the most obvious effect gain after fine-tuning and shows stability and consistency in various indicators of text generation quality. Figure 6The following table shows output samples from different models before and after fine-tuning. At the top are standard Snort rules, with green text indicating key content, red text indicating hallucinations (text that should not be generated), and yellow text indicating additional knowledge added by the model. As can be seen, the performance of each model has significantly improved before and after fine-tuning.

[0156] (2) Comparison of the basic model capabilities and the performance of the project with the introduction of prompt words

[0157] Attachment Figure 7 It shows that after introducing the prompt word engineering (such as Few Shot (Random) and Few Shot (TF-IDS)), the performance of different models showed significant changes. The following is an analysis and summary of the performance of each model under different strategies: (1) ChatGLM3-6B: From 75.59% of the key content hit rate of Zero Shot to 78.57% of Few Shot (Random), and then to 80.50% of Few Shot (TF-IDS), it shows a significant improvement. System loading and execution detection: It also shows a continuous improvement from Zero Shot to Few Shot (TF-IDS), indicating that the introduction of the prompt word engineering not only improves the key content hit rate of the model, but also improves the efficiency of system loading and execution detection. (2) DeepSeek-Coder-6B: Compared to the high baseline (Zero Shot), this model also showed further improvements in all three indicators, especially the key content hit rate increased from 92.80% to 96.50%, showing that the introduction of prompt word engineering is also effective for models that already perform well. (3) BloomZ-7B: This model already performed well in Zero Shot. After the introduction of prompt word engineering, the key content hit rate was significantly improved from 94.00% to 98.80%. However, the performance of system loading and execution detection fluctuated slightly, which may indicate that for a specific model, the introduction of prompt word engineering has different effects on different performance indicators. (4) LLaMA2-7B: It also showed an overall improvement trend from Zero Shot to Few Shot (TF-IDS). In particular, the key content hit rate increased from 88.50% to 95.80%. This significant improvement emphasizes the effectiveness of prompt word engineering in improving model accuracy.

[0158] All models showed improved performance after introducing cue word engineering, especially in terms of key content hit rate. This suggests that appropriate cue word design can effectively guide models to more accurately understand and execute tasks. Compared with random cueing, the TF-IDS-based cue word selection strategy (Few Shot (TF-IDS)) generally leads to better performance, likely because it provides more targeted task-related information, thereby improving the model's understanding and execution capabilities. Different models respond differently to cue word engineering, which may be related to model architecture, training data, and internal mechanisms. Therefore, optimizing the cue word strategy for specific models may be key to improving performance. These results indicate that introducing and optimizing cue word engineering is an effective way to improve the performance of language models on specific tasks, especially in terms of improving the model's key content hit rate. At the same time, choosing an appropriate cue word strategy and refining the cue word design are crucial to maximizing performance gains.

[0159] (3) Performance comparison between the basic model capabilities and the feedback mechanism

[0160] Attachment Figure 8 The comparison of the effects of each fine-tuned model before and after the feedback mechanism is shown. Since the experiment verifies that the pre-trained model does not have a significant effect under the simple feedback mechanism, the experiment mainly uses the fine-tuned model as the research object. From the experimental results: (1) All models show performance improvement after feedback iteration, among which ChatGLM3-6B has the most significant improvement, with the hit rate increasing from 75.59% to 85%, which shows that feedback iteration has a significant impact on performance improvement. (2) After feedback, the performance improvement of BloomZ-7B is relatively small, but it still maintains its leading position among all models. After feedback iteration, the performance improvement of DeepSeek-Coder-6B and BloomZ-7B is also relatively obvious, indicating that models with high initial performance still have room for improvement. (3) There is no simple positive correlation between the number of feedbacks and performance improvement. For example, LLaMA2-7B has the highest number of feedbacks (4.12), but its performance improvement is not the most significant. In contrast, ChatGLM3-6B achieves the most significant performance improvement despite having fewer feedbacks (3.21), which may indicate that the quality of feedback and the model’s responsiveness to feedback have a greater impact on performance than simply the number of feedbacks.

[0161] Experimental results indicate that feedback iteration can significantly improve the performance of large models in automatic rule generation applications. While all models demonstrated performance improvements, there was no direct correlation between the magnitude of improvement and the number of iterations. This suggests that when implementing feedback iteration, emphasis should be placed on the quality of feedback and the model's response mechanism, rather than simply the number of iterations. Furthermore, experimental results demonstrate that even models with relatively low zero-shot performance can achieve significant performance improvements through feedback iteration.

[0162] This approach builds on the extensive knowledge gained from pre-training a large language model, further incorporating task-specific knowledge through LoRa fine-tuning. It also proposes an iterative correction mechanism based on small sample examples and engine feedback to further improve the quality of automatic rule generation. Experimental results demonstrate that fine-tuning significantly improves the large language model, increasing detection rates from 0 to 91.8%. Furthermore, a sample selection mechanism based on TF-IDS similarity evaluation effectively incorporates the most relevant example rules for the test sample as prompts, guiding the model to generate specific rules and improving the content hit rate by approximately 3.5% to 7%. A correction mechanism based on engine runtime feedback guides the model to correct rules based on erroneous information, improving grammatical correctness and thereby increasing the loading rate or detection rate by 0.7% to 2.5%, facilitating the ultimate application of rules.

Claims

1. A method for automatically generating network threat traffic detection rules based on LLM, characterized in that: The following steps are involved: Step 1: Represent the information of network threat traffic. For HTTP and DNS protocols, directly extract the payload information from the traffic as traffic text. For application layer protocols other than HTTP and DNS, directly extract the payload information of TCP and UDP protocols, encode it, and use it as input traffic text. Step 2: Based on the input traffic text, the prompt word module selects a sample rule R according to the rule metadata retrieval and related rule matching algorithm, and adds the sample rule R to the input traffic text to form the complete model input information T+; Step 3: Based on the model input information T+, the large language model is trained using a fine-tuning method: First, a specific large language model is selected, and then a corpus dataset is constructed based on the network threat traffic detection rules. Then, LoRA fine-tuning is performed, and a low-rank matrix is ​​introduced into the weight matrix of the large language model. The pre-trained large language model is fine-tuned to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rules of the input traffic text. Step 4: Clean and match the text information output by the large language model, extract the detection rules it generates, input them into the rule engine for verification, and use feedback correction technology to supplement the correction information and control the correction threshold.

2. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1 is characterized in that: Step 2 specifically includes: Step 2.1: Rule metadata retrieval Metadata mapping and comparison is performed between the rule dictionary and the dictionary of network traffic header information, and key fields are filtered from the dictionary to achieve preliminary retrieval. For Snort rules, the threat type, network protocol, port number, threat level, and application service fields are matched with the input network threat traffic header information to filter out matching rules. Step 2.2: Match relevant rules Step 2.2.1: Preprocess the input traffic text and the content field and regular expression pcre field of the Snort rule configuration, including text cleaning and text segmentation; then convert the preprocessed text into a bigram sequence; for the input traffic text T, its bigram sequence set B0 is expressed as: Where w i is the i-th word in the input traffic text T; |T| represents the total number of words; Step 2.2.2: For each bigram sequence b in B0, calculate its TF-IDF value to assess its importance in a specific document relative to the entire document set; The TF value of the bigram sequence b in the document d is defined as: Where, f b,d is the number of times the bigram sequence b appears in document d, and the denominator is the sum of the number of times it appears in all bigrams sequences in document d; For a bigram sequence b, its IDF value is calculated as follows: Where N is the total number of documents in the document set D, and the denominator is the number of documents containing the bigram sequence b; Combining the above TF and IDF values, the TF-IDF value of the bigram sequence b in document d is: TF-IDF(b,d,D)=TF(b,d)*IDF(b,D) Step 2.2.3: Use the TF-IDF value calculated above to match the input traffic text with the content field and the regular expression pcre field in the Snort rule configuration; The matching degree between the two is evaluated by calculating the cosine similarity between the TF-IDF vectors of the input traffic text and each Snort rule text: Where, and are the TF-IDF vectors of T1 and T2 respectively, · represents the vector dot product, and are vectors and The norm of; T1 and T2 are the input traffic text and Snort rule text respectively; By comparing the similarity between the input traffic text and each Snort rule text, the candidate Snort rule most relevant to the input traffic is identified; Step 2.2.4: Set the hyperparameter K based on the small sample prompt example, retain the top K rules, that is, the sample rules R, and construct the model input information T+ in the order of task description, sample example, and task input, which is used as the input of the large language model module.

3. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1, characterized in that: The corpus construction in step 3 specifically includes: Step 3.1.1: Construct D1 dataset First, we collect all detection rules currently available in the community. Then, we use prompt words to preset model roles and tasks. Then, we input these detection rules into a large-scale language model to generate traffic feature descriptions corresponding to the rules. The model input information T+ generated by the prompt word module is used as the input of the initial untrained general model, and the corresponding rules are used as the output. Additional prompt words for downstream tasks are added to form the D1 dataset containing the "traffic feature description-detection rule" relationship; Step 3.1.2: Construct the D2 dataset First, the latest network attack traffic samples are collected. Then, the detection engine is actually executed to collect the triggered rules. After processing, the data relationship between "traffic text representation-detection rules" is obtained. The input network traffic samples are then read at the network protocol layer, classified and extracted from payload information, and processed into natural language descriptions to adapt to the text input mode of the language model. All traffic includes UDP and TCP, and its application layer protocols include DNS and HTTP. For the DNS protocol, DNS operation and query information are manually extracted and spliced ​​into natural language descriptions. For the HTTP protocol, its header fields are parsed and key fields are spliced. For requests with binary payloads, the hexadecimal payload text of the set length is retained and the content after the limited length is truncated. By analyzing the log files output by the detection engine after detection, the alarm information and corresponding rule entries are extracted, and the corresponding five-tuple traffic and detection rules are used. Finally, the textual traffic and matched rules are organized into input and output to form the D2 data set.

4. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1, characterized in that: Model fine-tuning in step 3 specifically includes: Step 3.2.1: Formalize the large language model fine-tuning process as: h=W0χ+ΔWχ; Where ΔW is the parameter corresponding to the fine-tuning update, h is the output feature vector, W0 is the pre-training model weight, and χ is the input text sequence; Step 3.2.2: Use two low-rank matrices A and B to approximate the parameters ΔW corresponding to the fine-tuning update: Set the approximate parameters Its parameter scale is a d*d matrix. The rank of the low-rank matrices A and B is r, so the size of the low-rank matrix A is adapted to The size of the low-rank matrix B is adapted to The number of fine-tuning parameters is reduced from d*d to 2 / r / d without affecting the model structure. The updated fine-tuning is expressed as: Where h′ is the feature vector output after fine-tuning; α is a hyperparameter that represents the scale of low-rank matrix scaling; Step 3.2.3: During the LoRA fine-tuning process, the low-rank matrices A and B are first initialized. Then, under the setting of the hyperparameter rank r, the large language model learns the parameter values ​​of the low-rank matrices A and B and extracts the matrix rows and columns with the richest information, ignoring redundant information to achieve the purpose of reducing parameters. To determine the appropriate r value, the α hyperparameter is introduced. After determining the optimal approximation effect, the α value is fixed, and then the r value is continuously reduced to obtain a large language model with a closer approximation effect, more concise information, and a smaller rank.

5. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1, characterized in that: The step 4 is specifically as follows: First, regular expressions are used to match specific characters and formats, and then the generated rules are located to obtain a complete rule. The extracted rules and network threat traffic are then simultaneously input into the rule engine for verification. The validity of the rules is determined by parsing the output results. Among them, for feedback on grammatical errors, the edit distance is used to check its format changes, and automatic correction tools in specific formats are used to assist in improving the rules; for feedback on detection effectiveness, the effectiveness of the feedback is judged based on the similarity between rules; if the generated rules do not have enough changes in content, the current feedback information is considered useless to the model, and manual judgment by experts or humans is introduced.

6. The method for automatically generating network threat traffic detection rules based on LLM according to claim 5, characterized in that: Cosine similarity is used to measure the similarity of rules: Convert the Snort rule text into a vector using TF-IDF, where each dimension represents the frequency and weight of a phrase in a bigram sequence b; the cosine similarity calculation formula is: Among them, X and Y are the two rule texts to be compared, X i and Y i are the bigram sequence phrases corresponding to the rule text, and n is the number of sequences.

7. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1, characterized in that: The step 4 also includes step 5: All detection rules are managed through the rule base module, and rule update capabilities are provided to the rule engine, and rule retrieval capabilities are provided to the prompt word module; When the rule engine successfully generates a detection rule through the large language model, it updates the detection rule to the rule base; Use the new detection rules in the rule base as prompt word examples to facilitate the next automatic rule generation.

8. An automatic generation system of network threat traffic detection rules based on LLM, characterized by: include: Prompt word module: selects sample rules R based on rule metadata retrieval and related rule matching algorithms, and adds sample rules R to the input traffic text to form complete model input information T+; Large language model module: Based on the model input information T+, the large language model is trained using a fine-tuning method: first, a specific large language model is selected, and then a corpus dataset is constructed based on the network threat traffic detection rules. Then, LoRA fine-tuning is performed, introducing a low-rank matrix into the weight matrix of the large language model, and fine-tuning the pre-trained large language model to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rules of the input traffic text; Rule engine module: This module cleans and matches the text information output by the large language model, extracts the generated detection rules, inputs them into the rule engine for verification, and uses feedback correction technology to supplement correction information and control correction thresholds. Rule Base Module: This module manages all detection rules and provides rule update capabilities for the rule engine and rule retrieval capabilities for the prompt word module. When the rule engine successfully generates a detection rule using the large language model, it updates the detection rule to the rule base. Use the new detection rules in the rule base as prompt word examples to facilitate the next automatic rule generation.

Images