LLM-based network threat flow detection rule automatic generation method and system

Abstract

The invention relates to the technical field of network security, and discloses an LLM-based network threat flow detection rule automatic generation method and system. According to the system, an automatic rule generation framework based on a large language model is constructed for network threat traffic, and the system comprises a cue word module, a large language model module, a rule engine module and a rule base management module; the four modules are executed in sequence to form a complete link from traffic information input to rule automatic generation and application verification; according to the method, domain knowledge is fused into a pre-training model through LoRA fine tuning, small sample cue word engineering and a correction mechanism based on engine feedback, verification is carried out through an actual detection engine, and the effectiveness of a rule automatically generated by the model is verified. According to the method, the network threat traffic sample can be fully and effectively analyzed, meanwhile, the requirement for training data is reduced, the rule set capable of being practically applied is generated, and the network defense capability is improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an LLM (Large Language Model)-based network threat traffic detection rule automatic generation method and system. Background Art

[0002] With the rapid development and popularization of network technology, the number of attacks suffered by global networks is also increasing. Detection rules are an important part of network intrusion detection engines. They define abnormal activities based on specific features and patterns to identify potential threats. Detection rules have the advantage of being easier to understand than intelligent models. However, current detection rules mainly rely on the experience sharing of experts and security communities and traditional automated generation methods to promote updates. They cannot adapt to the ever-changing network environment and the rapid evolution of attack technologies. Therefore, automatically generating high-quality detection rules has become a challenging task to ensure network security.

[0003] Traditional automatic detection rule generation methods, such as the longest common string, Apriori sequence mining algorithm, and gene and genetic algorithms, have made progress in mining features from network traffic information, but they have obvious shortcomings in processing high-dimensional data, ensuring real-time performance, reducing computational costs, and improving generalization capabilities. In addition, these methods can often only generate specific types of character sets or regular expressions, and cannot directly generate complete detection rules, which is not effective and flexible enough in actual network security application scenarios. In response to these limitations, recent research work has attempted to use the GPT-2 language model to automatically generate complete Snort rules. By learning and imitating the latest rule base, the GPT-2 model can understand the logic of rule writing, rather than relying solely on data-driven content mining. However, although this method has taken an important step in knowledge integration, it does not fully consider the information in real-time network traffic samples, but mainly relies on the database of existing rules. This approach limits the model's understanding of network traffic attack-related knowledge, thereby reducing the accuracy and usability of the generated rules, as well as the generalization of new attack traffic.

[0004] The existing automatic rule generation has two main deficiencies, including:

[0005] (1) Incomplete rules: Mainstream methods lack the ability to automatically generate end-to-end rules. They can often only generate specific types of character sets or regular expressions and fail to fully utilize all options for rule configuration. Rules generated by models such as GPT-2 may contain grammatical errors, which makes it impossible for detection engines to directly apply these rules.

[0006] (2) Inaccurate rules: Due to the rapid changes and evolution of network attack samples, existing methods fail to deeply understand network traffic and do not consider the information in real-time network traffic samples. This limits the model’s understanding of knowledge related to network traffic attacks and lacks the ability to quickly and accurately capture attack features, thus affecting the accuracy and generalization of the rules. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide a method and system for automatically generating network threat traffic detection rules based on LLM, which can fully and effectively analyze traffic samples, reduce the requirements for training data and generate a rule set that can be actually applied, thereby improving network defense capabilities. The technical solution is as follows:

[0008] A method for automatically generating network threat traffic detection rules based on LLM, comprising the following steps:

[0009] Step 1: Represent the information of network threat traffic. For HTTP and DNS protocols, directly extract the payload information from the traffic as traffic text. For application layer protocols other than HTTP and DNS protocols, directly extract the payload information of TCP and UDP protocols and encode them as input traffic text.

[0010] Step 2: Based on the input traffic text, the prompt word module selects the sample rule R according to the rule metadata retrieval and the relevant rule matching algorithm, and adds the sample rule R to the input traffic text to form the complete model input information T+;

[0011] Step 3: According to the model input information T+, the large language model is trained by fine-tuning: first, a specific large language model is selected, and then a corpus data set is constructed around the network threat traffic detection rules; then LoRA fine-tuning is performed, a low-rank matrix is ​​introduced into the weight matrix of the large language model, and the pre-trained large language model is fine-tuned to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rule of the input text traffic;

[0012] Step 4: Clean and match the text information output by the large language model, extract the detection rules it generates, input them into the rule engine for verification, and use feedback correction technology to supplement the correction information and control the correction threshold.

[0013] Furthermore, step 2 specifically includes:

[0014] Step 2.1: Rule metadata retrieval

[0015] Metadata mapping and comparison is performed through the dictionary of rules and the dictionary of network traffic header information, and key fields are selected from the dictionary to achieve preliminary retrieval; for Snort rules, the threat type, network protocol, port number, threat level and application service fields are matched with the input network threat traffic header information to select matching rules;

[0016] Step 2.2: Match relevant rules

[0017] Step 2.2.1: Preprocess the input traffic text and the content field and regular expression pcre field of the Snort rule configuration, including text cleaning and text segmentation; then convert the preprocessed text into a bigram sequence; for the input traffic text T, its bigram sequence set B0 is expressed as:

[0018]

[0019] In the formula, w i is the i-th word in the input traffic text T; |T| represents the total number of words;

[0020] Step 2.2.2: For each bigram sequence b in B0, calculate its TF-IDF value to assess its importance in a specific document relative to the entire document collection;

[0021] The TF value for a bigram sequence b in a document d is defined as:

[0022]

[0023] In the formula, f b,d is the number of times the bigram sequence b appears in document d, and the denominator is the sum of the number of times it appears in all bigram sequences in document d;

[0024] For the bigrams sequence b, its IDF value is calculated as follows:

[0025]

[0026] Where N is the total number of documents in the document set D, and the denominator is the number of documents containing the bigram sequence b;

[0027] Combining the above TF and IDF values, the TF-IDF value of bigram sequence b in document d is:

[0028] TF-IDF(b,d,D)=TF(b,d)*IDF(b,D)

[0029] Step 2.2.3: Use the TF-IDF value calculated above to match the input traffic text with the content field and the regular expression pcre field in the Snort rule configuration;

[0030] The match between the input traffic text and each Snort rule text is evaluated by calculating the cosine similarity between the TF-IDF vectors:

[0031]

[0032] In the formula, and are the TF-IDF vectors of T1 and T2 respectively, · represents the vector dot product, and Respectively, vector and The norm of; T1 and T2 are the input traffic text and Snort rule text respectively;

[0033] By comparing the similarity between the input traffic text and each Snort rule text, the candidate Snort rules most relevant to the input traffic are identified;

[0034] Step 2.2.4: Set the hyperparameter K according to the small sample prompt example, retain the top K rules, namely the sample rules R, and construct the model input information T+ in the order of task description, sample example and task input, which is used as the input of the large language model module.

[0035] Furthermore, the corpus construction in step 3 specifically includes:

[0036] Step 3.1.1: Construct D1 dataset

[0037] First, we collect all the detection rules currently open to the community, then preset the model roles and tasks through prompt words, and then input these detection rules into a large-scale language model to generate the traffic feature description corresponding to the rules;

[0038] The model input information T+ generated by the prompt word module is used as the input of the initial untrained general model, and the corresponding rules are used as the output. The prompt words of additional downstream tasks are added to form the D1 dataset containing the "traffic feature description-detection rule" relationship;

[0039] Step 3.1.2: Construct D2 dataset

[0040] First, the latest network attack traffic samples are collected, and then the detection engine is actually executed to collect the triggered rules. After processing, the data relationship of "traffic text representation-detection rules" is obtained;

[0041] The input network traffic samples are then read at the network protocol layer, classified and extracted from the load information, and processed by natural language description to adapt the text input mode of the language model.

[0042] All traffic includes two categories: UDP and TCP, and its application layer protocols include DNS and HTTP. For the DNS protocol, DNS operation and query information are manually extracted and spliced ​​into natural language descriptions. For the HTTP protocol, its header fields are parsed and key fields are taken for splicing. For requests with binary payloads, the hexadecimal payload text of the set length is retained and the content after the limited length is truncated.

[0043] By analyzing the log files output by the detection engine after detection, the alarm information and corresponding rule entries are extracted, and the corresponding five-tuple traffic and detection rules are used. Finally, the textual traffic and matched rules are organized into input and output to form the D2 data set.

[0044] Furthermore, the model fine-tuning in step 3 specifically includes:

[0045] Step 3.2.1: Formalize the large language model fine-tuning process as:

[0046] h=W0χ+ΔWχ;

[0047] Where ΔW is the parameter corresponding to the fine-tuning update, h is the output feature vector, W0 is the pre-trained model weight, and χ is the input text sequence;

[0048] Step 3.2.2: Use two low-rank matrices A and B to approximate the parameters ΔW corresponding to the fine-tuning update:

[0049] Set the approximate parameters Its parameter scale is a d*d matrix. The rank of the low-rank matrices A and B is r, so the size of the low-rank matrix A is adapted to The size of the low-rank matrix B is adapted to Then the number of fine-tuning parameters is reduced from d*d to 2*r*d. Without affecting the model structure, the update fine-tuning is expressed as:

[0050]

[0051] Where h′ is the feature vector output after fine-tuning; α is a hyperparameter, which indicates the scale of low-rank matrix scaling;

[0052] Step 3.2.3: During the LoRA fine-tuning process, the low-rank matrices A and B are first initialized, and then the large language model learns the parameter values ​​of the low-rank matrices A and B under the setting of the hyperparameter rank r, and extracts the matrix rows and columns with the richest information, ignoring redundant information, so as to achieve the purpose of reducing parameters;

[0053] In order to determine the appropriate r value, the α hyperparameter is introduced. After determining the optimal approximation effect, the α value is fixed, and then the r value is continuously reduced to obtain a large language model with a closer approximation effect, more concise information, and a smaller rank.

[0054] Furthermore, the step 4 is specifically as follows:

[0055] First, regular expressions are used to match specific characters and formats, and then the generated rules are located to obtain a complete rule. The extracted rules and network threat traffic are then simultaneously input into the rule engine for verification, and the validity of the rules is determined by parsing the output results.

[0056] Among them, for grammatical error feedback, the edit distance is used to check its format changes, and automatic correction tools in specific formats are used to assist in improving the rules; for detection effectiveness feedback, the effectiveness of the feedback is judged based on the similarity between the rules; if the generated rules do not have enough changes in content, the current feedback information is considered useless for the model, and manual judgment by experts or humans is introduced.

[0057] Furthermore, cosine similarity is used to measure the similarity of rules:

[0058] The Snort rule text is converted into a vector using TF-IDF, where each dimension represents the frequency and weight of a phrase in a bigram sequence b; the cosine similarity calculation formula is:

[0059]

[0060] Among them, X and Y are the two rule texts to be compared, X i and Y i are the bigram sequence phrases corresponding to the rule text, and n is the number of sequences.

[0061] Furthermore, the step 4 further includes a step 5:

[0062] All detection rules are managed through the rule base module, and rule update capabilities are provided for the rule engine, and rule retrieval capabilities are provided for the prompt word module;

[0063] When the rule engine successfully generates a detection rule through the large language model, it updates the detection rule to the rule base;

[0064] Use the new detection rules in the rule base as prompt word examples to facilitate the next automatic rule generation.

[0065] A network threat traffic detection rule automatic generation system based on LLM, comprising:

[0066] Prompt word module: selects sample rule R based on rule metadata retrieval and related rule matching algorithm, and adds sample rule R to input traffic text to form complete model input information T+;

[0067] Large language model module: According to the model input information T+, the large language model is trained by fine-tuning: first, a specific large language model is selected, and then a corpus data set is constructed around the network threat traffic detection rules; then LoRA fine-tuning is performed, and a low-rank matrix is ​​introduced into the weight matrix of the large language model, and the pre-trained large language model is fine-tuned to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rule of the input text traffic;

[0068] Rule engine module: cleans and matches the text information output by the large language model, extracts the detection rules generated by it, inputs them into the rule engine for verification, and uses feedback correction technology to supplement correction information and control correction thresholds;

[0069] Rule base module: manages all detection rules, provides rule update capabilities for the rule engine, and provides rule retrieval capabilities for the prompt word module; when the rule engine successfully generates a detection rule through the large language model, the detection rule is updated to the rule base; the new detection rules in the rule base are used as prompt word examples to facilitate the next automatic generation of rules.

[0070] The beneficial effects of the present invention are:

[0071] 1) The present invention constructs a rule automatic generation framework based on a large language model, which is divided into four modules, including a prompt word module, a large language model module, a rule engine module, and a rule base management module. The four modules are executed successively, forming a complete link from traffic information input to automatic rule generation and application; through LoRA fine-tuning, small sample prompt word engineering, and a correction mechanism based on engine feedback, the domain knowledge is integrated into the pre-trained model, and the actual detection engine is used to carry out verification to verify the validity of the rules generated by the model.

[0072] 2) Experimental results show that the automatic generation of network threat traffic detection rules based on a large language model proposed in the present invention can fully and effectively analyze traffic samples, while reducing the requirements for training data and generating a rule set that can be actually applied, thereby improving network defense capabilities.

[0073] 3) The overall process of the present invention includes three technical points: model fine-tuning, rule correlation analysis and feedback correction technology (including correction information supplement and correction threshold control). The three technologies act in different modules and work links, promoting the adaptation of large language models in the field of automatic generation of traffic rules from different levels, improving the performance of the model, and improving the quality of generated rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The figure is a schematic diagram of the overall process of the automatic rule generation framework based on the large language model in the present invention.

[0075] FIG. 2( a ) is a schematic diagram of the process of constructing a training corpus and a data set (D1 data set) in the present invention.

[0076] FIG2( b ) is a schematic diagram of the process of constructing the training corpus and data set (D2 data set) in the present invention.

[0077] FIG3( a ) is a schematic diagram of the LoRA-based fine-tuning method in the present invention.

[0078] FIG3( b ) is a schematic diagram of the LoRA-based fine-tuning method in the present invention.

[0079] Figure 4 This is a graph showing the performance analysis results of fine-tuning the large language model in the present invention.

[0080] Figure 5 This is a visualization analysis diagram of the fine-tuning performance analysis results of the large language model in the present invention.

[0081] Figure 6 This is an example display diagram of the automatic generation of rules before and after fine-tuning the large language model in the present invention.

[0082] Figure 7 This is a visualization analysis diagram of the performance analysis results of the large language model prompt word engineering in the present invention.

[0083] Figure 8 This is a visualization analysis diagram of the performance analysis results of the large language model feedback mechanism in the present invention. DETAILED DESCRIPTION

[0084] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] The present invention provides a method for automatically generating network threat traffic detection rules based on a large language model.

[0086] This method builds an automatic rule generation framework based on a large language model for network threat traffic, including technologies such as LoRA fine-tuning, small sample prompt word engineering, and a correction mechanism based on engine feedback. It integrates domain knowledge into the pre-trained model, uses a large language model to automatically generate rules end-to-end, and verifies the effectiveness of the rules automatically generated by the model through actual detection engines.

[0087] like Figure 1 The overall process diagram of the automatic rule generation framework based on the large language model is shown. The overall process of the method of the present invention is divided into four modules, including a prompt word module, a large language model module, a rule engine module, and a rule base management module. The four modules are executed successively to form a complete link from the input of network threat traffic information to the automatic generation and application of rules.

[0088] The three technologies include model fine-tuning, rule correlation analysis, and feedback correction technology (including correction information supplement and correction threshold control). The three technologies act in different modules and work links, promoting the adaptation of large language models in the field of automatic generation of traffic rules from different levels, improving the performance of the model, and improving the quality of generated rules.

[0089] The four modules combined with the input traffic information can realize a complete input process, the first step of which is to represent the information of network threat traffic. Network threat traffic has certain text characteristics, but it is not completely pure natural language text. In order to enable the large language model to understand the traffic information and reduce the difference between encoding and modality, the traffic information in PCAP format needs to be preprocessed first.

[0090] For common HTTP and DNS protocols, the payload information is directly extracted from the traffic as traffic text. For other application layer protocols, the payload information of TCP and UDP protocols is directly extracted and encoded in hexadecimal format as the input traffic text T.

[0091] 1. Prompt word module

[0092] The prompt word module supplements the domain knowledge based on the input traffic information and constructs the completed input text. The large language model has acquired a wealth of general knowledge through pre-training on large-scale data sets, and the prompt word can activate the weight of some important text information for the large language model, thereby playing the role of a domain expert, or the large language model can learn and generalize quickly from a small number of samples. Therefore, the prompt word module contains several methods for constructing input, including general prompt words and prompt words that provide small sample examples. Depending on the sample selection method, these technologies can be divided into random selection, TF-IDF algorithm considering the rule keyword weight, and Embedding algorithm considering the semantic similarity of rules. These algorithms will obtain the corresponding rule examples from the rule base based on the results of the correlation calculation to construct a complete input.

[0093] Based on the input traffic text T, the prompt word module further supplements the relevant example rules R to form a complete model input information T+. The prompt word module selects the example rules R in two ways: random selection, TF-IDF algorithm considering the rule keyword weight, and Embedding algorithm considering the semantic similarity of the rules. Since the prompt word is part of the model input, the input content length will be limited by the model sequence length. Considering that the text length of the traffic information itself is relatively long, the rule examples that can be provided will be limited to about 1-5 examples. Therefore, the algorithm of this method mainly includes two parts: rule metadata retrieval and related rule matching.

[0094] (1) Rule metadata retrieval: Metadata mapping and comparison is performed through the dictionary of rules and the dictionary of network traffic header information, and key fields are stably screened from the dictionary to achieve preliminary retrieval. Taking Snort rules as an example, fields such as threat type (classtype), network protocol (protocol), port number (port), threat level (priority), and application service (service) can be matched with the input network threat traffic header information to screen out matching rules.

[0095] (2) Relevant rule matching: This step focuses on fields such as rule content text and matching regular expression (pcre). The present invention selects TF-IDF (Term Frequency-Inverse Document Frequency) as the main relevance matching method, and the process is as follows:

[0096] First, the input traffic text and the content field and regular expression pcre field of the Snort rule configuration are preprocessed, including text cleaning and text segmentation. Then, the preprocessed text is converted into a bigram sequence. For text T, its bigram sequence B can be expressed as:

[0097]

[0098] In the formula, w i is the i-th word in the text T. For each bigram, its TF-IDF value is calculated to evaluate its importance in a specific document (input traffic text or Snort rule) relative to the entire document collection (all input traffic text and Snort rules). The TF value of bigram b in document d is defined as:

[0099]

[0100] In the formula, f b,d is the number of times bigrams b appears in document d, and the denominator is the sum of the number of occurrences of all bigrams in document d. For bigrams b, its IDF value is calculated as follows:

[0101]

[0102] In the formula, N is the total number of documents in the document set D, and the denominator is the number of documents containing bigram b. To avoid the denominator being zero, 1 is usually added to the denominator. Combining the above TF and IDF, the TF-IDF value of bigrams b in document d is:

[0103] TF-IDF(b,d,D)=TF(b,d)*IDF(b,D)

[0104] Using the TF-IDF weights calculated above, match the input traffic text with the content field and the regular expression pcre field in the Snort rule configuration. Specifically, the cosine similarity between the TF-IDF vectors of the input traffic text and each Snort rule text can be calculated to evaluate the matching degree between them:

[0105]

[0106] In the formula, and are the TF-IDF vectors of T1 and T2 respectively, · represents the vector dot product, is a vector The norm of .

[0107] By comparing the similarity between the input traffic text and the Snort rules, the Snort rules most relevant to the input traffic can be identified, thus achieving high-precision matching. This method based on bigrams and TF-IDF can effectively capture and utilize the local sequence information of the text, improving the accuracy and relevance of the matching process.

[0108] After completing the rule metadata retrieval and related rule matching, we will obtain the most relevant candidate rules, and then set the K hyperparameters according to the small sample prompt example, retain the TopK rules, and construct the completed query input in the order of task description (including task output format requirements), sample examples, and task input, and input it into the large language model.

[0109] 2. Large Language Model Module

[0110] The large language model module is a processing module that generates detection rules based on input text information.

[0111] The main considerations are: First, data security and privacy protection are important requirements in the field of network security. The ability to achieve independent deployment is an important prerequisite. Closed-source or closed-operation platforms are not suitable for direct use. Second, automatic generation of detection rules is a subdivision function in a dedicated field. Smaller-scale models should achieve better results after domain knowledge migration. Models that are too large will lead to excessively high computing and operating costs and lose actual application benefits.

[0112] The present invention mainly adopts a fine-tuning approach to train the large language model, which mainly includes: basic model selection, corpus construction and LoRA fine-tuning.

[0113] (1) Basic model selection

[0114] For comparison, several open source smaller language models were selected, including LlaMA, BloomZ, DeepSeeker, and ChatGLM.

[0115] (2) Corpus Structure

[0116] Two corpus datasets D1 and D2 are constructed around network threat traffic detection rules, such as Snort rules and Suricata rules. The two corpus datasets are based on the two perspectives of rule interpretability and rule applicability. By combining these rules with actual network traffic data, the model's understanding of traffic and rules is reflected, which helps the model learn how to identify specific patterns in network traffic.

[0117] In the field of network threat traffic detection rules, rules are the conclusions drawn by community members or experts in the field after analyzing network attacks. They will be shared in the community for discussion and use to deal with ever-changing network attacks. Common communities include Snort rules and Emerging Threats. However, in these communities, only the rules themselves are published, and there is a lack of corresponding traffic samples. Therefore, there are no directly available samples with the corresponding relationship between traffic and rules to support model fine-tuning.

[0118] The D1 dataset construction process is described in Figure 2 (a), including the prompt words used, the rules of the examples, and the generated text. The core knowledge of the D1 dataset is the understanding of the rule text and traffic characteristics, and on this basis, a mapping between the two is formed. In order to achieve the integration of this knowledge, all the detection rules currently disclosed by the community are first collected, and then the model roles and tasks are preset through the prompt words, and then these detection rules are input into large-scale large language models, such as ChatGPT4 and ChatGLM4, to generate the traffic feature description corresponding to the rules. These large-scale large language models have stronger understanding and reasoning abilities, and the generated text quality is also high, which has also been manually verified in practical tests: given valid prompt words, they have the ability to understand Snort detection rules. The main advantages of using a large language model to complete this task include: (1) no need to design labels. Since it is a generation task, only the rule text needs to be provided, which reduces the requirements for data work; (2) it conforms to the model thinking chain method: the large language model can work better under the thinking chain working method. Therefore, in the process of automatic rule generation, the model can also be guided to use the thinking chain method to perform one-step analysis and generation, which can be roughly summarized as: understanding of traffic text-mining of traffic features-mapping of rules. In this thinking chain, traffic features to rules is a crucial step. Using a large-scale large language model to generate traffic feature descriptions can provide other smaller-scale language models with better training corpus in a way similar to distillation, thereby guiding the smaller-scale language models to enhance their rule mapping capabilities. Based on this, after processing the output of the model, the generated text is used as input, the corresponding rules are used as output, and additional prompt words for downstream tasks are added to form the D1 data set, which contains the "traffic feature description-detection rule" relationship.

[0119] Figure 2(b) describes the D2 dataset construction process, including network attack types, detection engines, rule examples, and dataset sample examples. First, the latest network attack traffic samples are collected, and then the detection engine is actually executed to collect the triggered rules. After processing, the data relationship of "traffic text representation-detection rules" is obtained. Since the traffic source format (PCAP) is a binary file, it needs to be correctly parsed to obtain its content. In order to adapt to the text input mode of the language model, the input needs to be processed to a certain extent, including: reading the network protocol layer, classifying and extracting payload information, and natural language description. All traffic is mainly divided into two categories: UDP and TCP, and its application layer protocols are mainly DNS and HTTP. For the DNS protocol, manually extract DNS operation and query information and splice them into natural language descriptions. For the HTTP protocol, parse its header fields, take key fields for splicing, and for requests with binary payloads, retain a certain length of hexadecimal payload text and truncate the content after the limited length. The detection engine can use Snort or Suricata engines. Both are open source engines with the characteristics of easy deployment and high efficiency. Among them, Suricata is compatible with Snort detection rules. Therefore, this method uses the Snort engine as an example. After detection, the detection engine will output several log files. By analyzing the log files and extracting the alarm information and corresponding rule entries, the corresponding five-tuple traffic and detection rules can be identified. Finally, the textual traffic and the matched rules are organized into input and output to form the D2 data set.

[0120] It is worth noting that, whether it is the D1 dataset or the D2 dataset, in order to reduce the input pressure of the model and maintain the generalization ability of the model, all rules need to be cleaned to remove some explanatory text, especially the description under the "meta" field of the rule, which usually includes the reference URL, modification time, creation time, etc.

[0121] (3) LoRA fine-tuning

[0122] Large language models have been trained on a large number of predictions and have certain semantic understanding and reasoning capabilities. However, a large number of current models often focus on natural language text or code corpora, and have not been fully studied for predictions in the field of network security, especially Snort rules. In order to enhance the adaptability of the model to the specific task of automatic rule generation, the present invention constructs a corresponding data set and combines the LoRA fine-tuning technology to perform additional training on the model.

[0123] The present invention chooses to use LoRA to fine-tune the pre-trained model, as shown in Figures 3(a) and (b). Compared with the fine-tuning of the full parameters, the core idea is: fine-tuning is achieved by introducing low-rank matrices A and B in the weight matrix of the model. LoRA introduces a low-rank matrix in the bypass of the original pre-trained matrix, allowing adaptation in any network layer, and only adjusts the weights in a specific model layer (such as the self-attention layer or the feedforward network layer), such as any W_q, W_k, W_v in the Transformer. A common operation is to perform low-rank adaptation on some parameters in the Attention layer. However, LoRA introduces a low-rank matrix, and its effect is not constant on different tasks and data sets. Therefore, it is necessary to perform specific optimization on the rule automatic generation task based on the LoRA method, and select the size and position of the LoRA parameters to find the best model adjustment strategy. The model fine-tuning process can be formally expressed as:

[0124] h=W0χ+ΔWχ;

[0125] Where ΔW is the parameter corresponding to the fine-tuning update. The core algorithm of LoRA uses two low-rank matrices A and B to approximate ΔW. Its parameter scale is a d*d matrix. The rank of the low-rank matrix is ​​r, so the size of the A matrix is ​​adapted to The size of the B matrix is ​​adapted to Then the number of fine-tuning parameters will be reduced from d*d to 2*r*d. Without affecting the model structure, the update fine-tuning is expressed as:

[0126]

[0127] In the formula, α is a hyperparameter, which indicates the scale of low-rank matrix scaling. In the process of LoRA fine-tuning, the two matrices A and B are initialized first, and then the model is allowed to learn the parameter values ​​of the low-rank matrices A and B under the setting of the hyperparameter rank r, to achieve the effect of SVD decomposition, extract the matrix rows and columns with the richest information, ignore redundant information, and achieve the purpose of reducing parameters. Furthermore, in order to determine the appropriate r value, the α hyperparameter is introduced. After determining the optimal approximation effect, the α value is fixed, and then r is continuously reduced to obtain a model with similar approximation effect, but more concise information and smaller rank. In this process, The value keeps increasing, which plays a role in amplifying the weight of the low-rank matrix and accelerating the gradient descent.

[0128] So far, the large model module has fine-tuned the model through LoRA to adapt it to specific tasks, and saved the model for analyzing the input text, executing the rule automatic generation task, and generating the corresponding detection rules for specific input traffic. However, in order to further verify whether the generated rules are usable and their preliminary effects, the rules are passed to the rule engine module.

[0129] 3. Rule Engine Module

[0130] The rule engine module cleans and matches the text information output by the large language model, extracts the detection rules it generates, and inputs them into the rule engine for verification. This module mainly uses feedback correction technology, including correction information supplementation and correction threshold control.

[0131] First, regular expressions are used to match specific characters and formats, such as the "' syntax in Markdown and the rule: prefix required by the prompt word, and then the generated rules are located to obtain a complete rule. The extracted rules and network threat traffic are then input into the rule engine for verification. The validity of the rules is determined by parsing the output results (such as alarm logs and execution logs).

[0132] Among them, for grammatical error feedback, the edit distance is used to check its format changes, and a certain format automatic correction tool is used to assist in improving the rules; for detection effect feedback, it is necessary to judge whether the feedback is effective based on the similarity between the rules. If the generated rules do not have enough changes in content, it is considered that the current feedback information is useless for the model, and it is necessary to introduce experts or manual judgment. The present invention mainly uses cosine similarity for similarity measurement: the Snort rule text is converted into a vector, and TF-IDF is also used for implementation. Each dimension represents the frequency and weight of a bigram b phrase. The cosine similarity calculation formula is:

[0133]

[0134] Among them, X and Y are the two rule texts to be compared, X i and Y i are the bigram sequence phrases corresponding to the rule text, and n is the number of sequences.

[0135] In the actual operation environment, three thresholds need to be optimized through testing: edit distance threshold, similarity threshold, and feedback number threshold. The first two thresholds are used to determine the effectiveness of the model in accepting feedback, and lower thresholds allow more attempts. However, considering that the model's effectiveness will decline after multiple rounds of conversations and the model's computational cost, it is not appropriate to set the threshold too low. In practice, the edit distance is set to 5, the similarity threshold is set to -0.7, and the feedback number threshold is 5.

[0136] 4. Rule base module

[0137] The rule base module is responsible for managing all detection rules. It needs to provide rule update capabilities for the rule engine and rule retrieval capabilities for the prompt word module. When the rule engine successfully passes the rule generated by the large language model, the rule will be updated to the rule base. The new rule may be used as a prompt word example to promote the automatic generation of the next rule.

[0138] 5. Experimental evaluation

[0139] In order to verify the technical effect of the technical solution proposed by the present invention, the following experimental evaluation was carried out.

[0140] The experiment uses two sets of data sets to train and test the model, named D1 and D2. The statistical results are shown in the table. The D1 data set is classified and counted according to Snort's Classtype, and the D2 data set is classified and counted according to the actual attack methods.

[0141] Table 1 Dataset construction statistics

[0142]

[0143] The D1 dataset is an Explain-Rule dataset. The construction principle is: first collect Snort and Suricata rule data shared by the open source community, then use mature large language models (such as ChatGPT3, ChatGPT4, ChatGLM4) to explain the rules, and then use the explanation text as input and the rules as output to construct the D1 dataset. This experiment uses ChatGLM4 to complete this explanation and knowledge distillation process, with a total of 50,000 rule samples. The prompt words used are as follows:

[0144] “You are a cybersecurity expert familiar with Suricata rules and traffic detection. Please analyze the key recognition features and corresponding detection options of target detection traffic according to the given rules,limited to 100 words.”

[0145] The D2 dataset is a dataset of traffic samples and corresponding detection rules (Traffic-Rule). Its construction principle is: by collecting the latest network threat traffic sample files, inputting them into the open source detection engine, and then collecting the triggered rules and corresponding traffic samples, and then preprocessing the traffic to form a dataset corresponding to the traffic information-rule information. This experiment mainly collected 15,000 network threat traffic samples, and actually triggered 1,518 alarm records, a total of 181 rules, covering 22 categories.

[0146] This experiment designed three types of evaluation indicators to evaluate the experimental results: model generation quality, speed, and actual effect, as shown in Table 2. The model generation quality mainly refers to the relevant indicators in the field of natural language processing, including BLEU (Bilingual Evaluation Understudy), ROUGE-1, ROUGE-2, and ROUGE-L. These indicators are commonly used for machine translation quality evaluation. In this experiment, they are used to evaluate the matching degree between the automatically generated rule text and the standard rule text.

[0147] Table 2 Evaluation indicators of rule-based automatic generation experiments

[0148]

[0149] (1) Comparison of basic model capabilities and performance after fine-tuning

[0150] Attached Figure 4The performance of different large language models (ChatGLM3-6B, DeepSeek-Coder-6B, BloomZ-7B, LLaMA2-7B) before and after fine-tuning on the rule automatic generation task is shown. The main focus is on the changes in BLEU, ROUGE-1, ROUGE-2, and ROUGE-L indicators, sample processing time (seconds), and performance differences on specific tasks (D1: explanation-rules; D2: traffic-rules). In addition, statistics on content hit rate, system loading time, and execution detection time are also included. From the results in the figure, the main findings are as follows:

[0151] 1) The effect of fine-tuning is significant: Comparing the results of the baseline (Base) and the fine-tuned (Finetuned), all models have significantly improved in BLEU, ROUGE-1, ROUGE-2, ROUGE-L and other indicators after fine-tuning. This shows that fine-tuning is very effective in improving the performance of the model in the task of automatic rule generation.

[0152] 2) Improvement of model performance: After fine-tuning, BloomZ-7B showed the most significant performance improvement, with the BLEU and ROUGE indicators on the D1 and D2 tasks reaching the highest values ​​among all models. In particular, on the D2 task, BLEU reached 82.9181 and ROUGE-L reached 86.2796, showing extremely high generation quality.

[0153] 3) Sample processing time: The time it takes to process each sample after fine-tuning increases, which may be because the fine-tuned model is more complex and requires more computing resources. The increase in sample processing time for BloomZ-7B and LLaMA2-7B after fine-tuning is relatively small, indicating that these models are more efficient while maintaining performance improvements.

[0154] 4) Content hit rate, system loading and execution detection: BloomZ-7B and DeepSeek-Coder-6B have significant improvements in content hit rate, system loading time and execution detection time after fine-tuning, indicating that these models have not only improved in generation quality, but also optimized in content relevance and system response time.

[0155] Attached Figure 5 The basic capabilities of each model and the evaluation indicators after fine-tuning are visualized. It can be seen that the changing trends of different models in different indicators are significantly different. Therefore, the selection of which model to fine-tune needs to consider the specific needs of the task. For example, if the task focuses on the quality of generation, BloomZ-7B is an excellent choice. It has the most obvious effect gain after fine-tuning, and it shows stability and consistency in various indicators of text generation quality. Figure 6The following table shows the output samples of different models before and after fine-tuning. The top is the standard rule of Snort, the green text indicates the key content, the red text indicates the hallucination content (text that should not be generated), and the yellow text indicates the additional knowledge added by the model. It can be seen that the content of different models has been significantly improved before and after fine-tuning.

[0156] (2) Comparison of the basic capabilities of the model and the performance of the introduced prompt word engineering

[0157] Attached Figure 7 It shows that after introducing cue word engineering (such as Few Shot (Random) and Few Shot (TF-IDS)), the effects of different models show significant changes. The following is an analysis and summary of the performance of each model under different strategies: (1) ChatGLM3-6B: From 75.59% of the key content hit rate of Zero Shot to 78.57% of Few Shot (Random), and then to 80.50% of Few Shot (TF-IDS), it shows a significant improvement. System loading and execution detection: It also shows a continuous improvement from Zero Shot to Few Shot (TF-IDS), indicating that the introduction of cue word engineering not only improves the key content hit rate of the model, but also improves the efficiency of system loading and execution detection. (2) DeepSeek-Coder-6B: In the case of a high baseline (Zero Shot), this model also showed further improvements in all three indicators, especially the key content hit rate increased from 92.80% to 96.50%, showing that the introduction of cue word engineering is also effective for models that already perform well. (3) BloomZ-7B: This model already has a high performance in Zero Shot. After the introduction of cue word engineering, the key content hit rate has been significantly improved, from 94.00% to 98.80%. However, the performance of system loading and execution detection fluctuates slightly, which may indicate that for a specific model, the introduction of cue word engineering has different effects on different performance indicators. (4) LLaMA2-7B: It also shows an overall improvement trend from Zero Shot to Few Shot (TF-IDS). In particular, the key content hit rate increased from 88.50% to 95.80%. This significant improvement emphasizes the effectiveness of cue word engineering in improving model accuracy.

[0158] All models showed improved performance after introducing cue word engineering, especially in terms of key content hit rate. This suggests that appropriate cue word design can effectively guide the model to understand and perform tasks more accurately. Compared with random cueing, the TF-IDS-based cue word selection strategy (Few Shot (TF-IDS)) generally leads to better performance improvement, which may be because it can provide more targeted task-related information, thereby improving the model's understanding and execution capabilities. Different models respond differently to cue word engineering, which may be related to the model architecture, training data, and internal mechanisms. Therefore, optimizing the cue word strategy for a specific model may be the key to improving performance. These results show that introducing and optimizing cue word engineering is an effective way to improve the performance of language models on specific tasks, especially in terms of improving the model's key content hit rate. At the same time, choosing an appropriate cue word strategy and refined cue word design are crucial to maximizing performance improvement.

[0159] (3) Comparison of basic model capabilities and performance with the introduction of feedback mechanisms

[0160] Attached Figure 8 The effects of each fine-tuned model before and after the feedback mechanism are compared. Since the experiment verifies that the pre-trained model does not have a significant effect under the simple feedback mechanism, the experiment mainly uses the fine-tuned model as the research object. From the experimental results: (1) All models show performance improvement after feedback iteration, among which ChatGLM3-6B has the most significant improvement, with the hit rate increasing from 75.59% to 85%, which shows that feedback iteration has a significant impact on performance improvement. (2) After feedback, the performance improvement of BloomZ-7B is relatively small, but it still maintains its leading position among all models. After feedback iteration, the performance improvement of DeepSeek-Coder-6B and BloomZ-7B is also relatively obvious, indicating that models with high initial performance still have room for improvement. (3) There is no simple positive correlation between the number of feedbacks and performance improvement. For example, LLaMA2-7B has the highest number of feedbacks (4.12), but its performance improvement is not the most significant. In contrast, ChatGLM3-6B achieved the most significant performance improvement despite having a lower number of feedbacks (3.21), which may indicate that the quality of the feedback and the model's responsiveness to feedback have a greater impact on performance than simply the number of feedbacks.

[0161] The experimental results show that the quality of large models in automatic rule generation applications can be significantly improved through the feedback iteration mechanism. Although all models showed performance improvements, there was no direct relationship between the magnitude of improvement and the number of feedbacks for different models. This may mean that when implementing feedback iterations, attention should be paid to the quality of feedback and the response mechanism of the model, rather than just the number of iterations. In addition, the experimental results also show that even models with lower performance under zero-sample conditions can significantly improve their performance through feedback iterations.

[0162] It can be seen that this method is based on the sufficient knowledge of large language model pre-training, and further integrates task-related knowledge through the LoRa fine-tuning method, and proposes a correction and iteration mechanism based on prompt engineering and engine feedback of small sample examples to further improve the quality of automatic rule generation. Experimental results show that fine-tuning has a great improvement effect on large language models, with the highest detection rate improvement from 0 to 91.8%. At the same time, the sample selection mechanism based on TF-IDS similarity evaluation can effectively integrate the example rules most relevant to the sample to be tested as prompt words, guide the model to generate specific rules, and improve the content hit rate by about 3.5% to 7%; the correction mechanism based on engine operation information feedback can guide the model to correct the rules according to the wrong information, improve the grammatical correctness, thereby increasing the loading rate or detection rate by 0.7% to 2.5%, and promoting the final application of the rules.

Claims

1. A method for automatically generating network threat traffic detection rules based on LLM, characterized in that: The following steps are involved: Step 1: Represent the information of network threat traffic. For HTTP and DNS protocols, directly extract the payload information from the traffic as traffic text. For application layer protocols other than HTTP and DNS protocols, directly extract the payload information of TCP and UDP protocols and encode them as input traffic text. Step 2: Based on the input traffic text, the prompt word module selects the sample rule R according to the rule metadata retrieval and the relevant rule matching algorithm, and adds the sample rule R to the input traffic text to form the complete model input information T+; Step 3: According to the model input information T+, the large language model is trained by fine-tuning: first, a specific large language model is selected, and then a corpus data set is constructed around the network threat traffic detection rules; then LoRA fine-tuning is performed, a low-rank matrix is ​​introduced into the weight matrix of the large language model, and the pre-trained large language model is fine-tuned to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rule of the input text traffic; Step 4: Clean and match the text information output by the large language model, extract the detection rules it generates, input them into the rule engine for verification, and use feedback correction technology to supplement the correction information and control the correction threshold.

2. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1 is characterized in that: Step 2 specifically includes: Step 2.1: Rule metadata retrieval Metadata mapping and comparison is performed through the dictionary of rules and the dictionary of network traffic header information, and key fields are selected from the dictionary to achieve preliminary retrieval; for Snort rules, the threat type, network protocol, port number, threat level and application service fields are matched with the input network threat traffic header information to select matching rules; Step 2.2: Match relevant rules Step 2.2.1: Preprocess the input traffic text and the content field and regular expression pcre field of the Snort rule configuration, including text cleaning and text segmentation; then convert the preprocessed text into a bigram sequence; for the input traffic text T, its bigram sequence set B0 is expressed as: In the formula, w i is the i-th word in the input traffic text T; |T| represents the total number of words; Step 2.2.2: For each bigram sequence b in B0, calculate its TF-IDF value to assess its importance in a specific document relative to the entire document collection; The TF value for a bigram sequence b in a document d is defined as: In the formula, f b,d is the number of times the bigram sequence b appears in document d, and the denominator is the sum of the number of times it appears in all bigram sequences in document d; For the bigrams sequence b, its IDF value is calculated as follows: Where N is the total number of documents in the document set D, and the denominator is the number of documents containing the bigram sequence b; Combining the above TF and IDF values, the TF-IDF value of bigram sequence b in document d is: TF-IDF(b,d,D)=TF(b,d)*IDF(b,D) Step 2.2.3: Use the TF-IDF value calculated above to match the input traffic text with the content field and the regular expression pcre field in the Snort rule configuration; The match between the input traffic text and each Snort rule text is evaluated by calculating the cosine similarity between the TF-IDF vectors: In the formula, and are the TF-IDF vectors of T1 and T2 respectively, · represents the vector dot product, and Respectively, vector and The norm of; T1 and T2 are the input traffic text and Snort rule text respectively; By comparing the similarity between the input traffic text and each Snort rule text, the candidate Snort rules most relevant to the input traffic are identified; Step 2.2.4: Set the hyperparameter K according to the small sample prompt example, retain the top K rules, namely the sample rules R, and construct the model input information T+ in the order of task description, sample example and task input, which is used as the input of the large language model module.

3. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1 is characterized in that: The corpus construction in step 3 specifically includes: Step 3.1.1: Construct D1 dataset First, we collect all the detection rules currently open to the community, then preset the model roles and tasks through prompt words, and then input these detection rules into a large-scale language model to generate the traffic feature description corresponding to the rules; The model input information T+ generated by the prompt word module is used as the input of the initial untrained general model, and the corresponding rules are used as the output. The prompt words of additional downstream tasks are added to form the D1 dataset containing the "traffic feature description-detection rule" relationship; Step 3.1.2: Construct D2 dataset First, the latest network attack traffic samples are collected, and then the detection engine is actually executed to collect the triggered rules. After processing, the data relationship of "traffic text representation-detection rules" is obtained; The input network traffic samples are then read at the network protocol layer, classified and extracted from the load information, and processed by natural language description to adapt the text input mode of the language model. All traffic includes two categories: UDP and TCP, and its application layer protocols include DNS and HTTP. For the DNS protocol, DNS operation and query information are manually extracted and spliced ​​into natural language descriptions. For the HTTP protocol, its header fields are parsed and key fields are taken for splicing. For requests with binary payloads, the hexadecimal payload text of the set length is retained and the content after the limited length is truncated. By analyzing the log files output by the detection engine after detection, the alarm information and corresponding rule entries are extracted, and the corresponding five-tuple traffic and detection rules are used. Finally, the textual traffic and matched rules are organized into input and output to form the D2 data set.

4. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1 is characterized in that: Model fine-tuning in step 3 specifically includes: Step 3.2.1: Formalize the large language model fine-tuning process as: h=W0X+ΔWχ; Where ΔW is the parameter corresponding to the fine-tuning update, h is the output feature vector, W0 is the pre-trained model weight, and χ is the input text sequence; Step 3.2.2: Use two low-rank matrices A and B to approximate the parameters ΔW corresponding to the fine-tuning update: Set the approximate parameters Its parameter scale is a d*d matrix. The rank of the low-rank matrices A and B is r, so the size of the low-rank matrix A is adapted to The size of the low-rank matrix B is adapted to Then the number of fine-tuning parameters is reduced from d*d to 2*r*d. Without affecting the model structure, the update fine-tuning is expressed as: Where h′ is the feature vector output after fine-tuning; α is a hyperparameter, which indicates the scale of low-rank matrix scaling; Step 3.2.3: During the LoRA fine-tuning process, the low-rank matrices A and B are first initialized, and then the large language model learns the parameter values ​​of the low-rank matrices A and B under the setting of the hyperparameter rank r, and extracts the matrix rows and columns with the richest information, ignoring redundant information, so as to achieve the purpose of reducing parameters; In order to determine the appropriate r value, the α hyperparameter is introduced. After determining the optimal approximation effect, the α value is fixed, and then the r value is continuously reduced to obtain a large language model with a closer approximation effect, more concise information, and a smaller rank.

5. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1 is characterized in that: The step 4 is specifically as follows: First, regular expressions are used to match specific characters and formats, and then the generated rules are located to obtain a complete rule. The extracted rules and network threat traffic are then simultaneously input into the rule engine for verification, and the validity of the rules is determined by parsing the output results. Among them, for grammatical error feedback, the edit distance is used to check its format changes, and automatic correction tools in specific formats are used to assist in improving the rules; for detection effectiveness feedback, the effectiveness of the feedback is judged based on the similarity between the rules; if the generated rules do not have enough changes in content, the current feedback information is considered useless for the model, and manual judgment by experts or humans is introduced.

6. The method for automatically generating network threat traffic detection rules based on LLM according to claim 5 is characterized in that: Cosine similarity is used to measure the similarity of rules: The Snort rule text is converted into a vector using TF-IDF, where each dimension represents the frequency and weight of a phrase in a bigram sequence b; the cosine similarity calculation formula is: Among them, X and Y are the two rule texts to be compared, X i and Y i are the bigram sequence phrases corresponding to the rule text, and n is the number of sequences.

7. The method for automatically generating network threat traffic detection rules based on LLM according to claim 1, characterized in that: The step 4 also includes step 5: All detection rules are managed through the rule base module, and rule update capabilities are provided for the rule engine, and rule retrieval capabilities are provided for the prompt word module; When the rule engine successfully generates a detection rule through the large language model, it updates the detection rule to the rule base; Use the new detection rules in the rule base as prompt word examples to facilitate the next automatic rule generation.

8. A network threat traffic detection rule automatic generation system based on LLM, characterized in that: include: Prompt word module: selects sample rule R based on rule metadata retrieval and related rule matching algorithm, and adds sample rule R to input traffic text to form complete model input information T+; Large language model module: According to the model input information T+, the large language model is trained by fine-tuning: first, a specific large language model is selected, and then a corpus data set is constructed around the network threat traffic detection rules; then LoRA fine-tuning is performed, and a low-rank matrix is ​​introduced into the weight matrix of the large language model, and the pre-trained large language model is fine-tuned to analyze the input traffic text, perform the rule automatic generation task, and generate text information corresponding to the detection rule of the input text traffic; Rule engine module: cleans and matches the text information output by the large language model, extracts the detection rules generated by it, inputs them into the rule engine for verification, and uses feedback correction technology to supplement correction information and control correction thresholds; Rule base module: manages all detection rules, provides rule update capabilities for the rule engine, and provides rule retrieval capabilities for the prompt word module; when the rule engine successfully generates a detection rule through the large language model, it updates the detection rule to the rule base; Use the new detection rules in the rule base as prompt word examples to facilitate the next automatic rule generation.

Images