Industrial large model construction method and system for network flow analysis and detection

Abstract

The invention belongs to the field of network traffic, and relates to an industry large model construction method and system oriented to network traffic analysis and detection. The method comprises the following steps: acquiring network flow log data, and constructing an IP intelligence library and a security vulnerability library; generating event details and event discrimination and disposal suggestions according to the network traffic log data, the IP intelligence library and the security vulnerability library, forming a traffic research and judgment report by combining the event details and the event discrimination and disposal suggestions, and forming a pre-training data set by the traffic research and judgment report and the network traffic log data; and utilizing the pre-training data set and the Lora technology to finely tune the language large model, and generating an industry large model for network traffic analysis and detection. The method has the natural language understanding and analysis capability of network traffic, can effectively detect, research and judge abnormal traffic, automatically generates a research and judgment report, provides a reliable basis for supervisors to research and judge network attacks, and effectively improves the network governance level.

Description

Technical Field

[0001] The present invention belongs to the field of network traffic, and specifically relates to a method and system for constructing an industry large model for network traffic analysis and detection. Background Art

[0002] With the rapid development of network applications, network traffic data is numerous, and the content has evolved from plain text to encrypted, making traffic data analysis and detection more difficult. Previously, a large number of researchers have used deep neural network models to extract deep features of raw traffic, which has significantly improved traffic identification capabilities. At present, the main traffic identification technologies are mainly divided into three categories. The first is traffic identification technology based on fingerprint construction. FlowPrint proposes to represent each flow by extracting device, certificate, size and time features, and construct a fingerprint library through clustering and cross-correlation for effective traffic classification. However, these fingerprint information can be easily tampered with in virtual communication networks and lose their correct meaning. The second is a traffic identification method based on feature statistics. AppScanner uses the statistical features of packet size to train random forest classifiers for network traffic identification, and BIND uses the statistical features of traffic time to train traffic identification models. However, this method is difficult to design general statistical features to handle a large number of increasingly complex applications and websites, and its generalization is weak. The third is traffic identification technology based on deep learning. The DF model uses convolutional neural networks (CNNs), FS-Net uses recurrent neural networks (RNNs) to automatically extract traffic representations from the original packet size sequence of encrypted traffic, and the TSCRNN model describes the original payload. However, this approach relies on a large amount of supervised data to capture effective features, thereby learning biased representations in unbalanced data. The above methods have achieved traffic identification in specific categories of web pages and applications and efficient detection of abnormal traffic with low resources. However, technologies such as fingerprint mining, statistical feature construction, and deep feature extraction are affected by the single feature construction and dependence on labeled data, and have problems such as weak generalization and strong dependence on expert experience.

[0003] Existing security personnel or supervisors often find it difficult to effectively deal with complex, changeable and massive network threats due to the lack of highly generalized and automated tools, resulting in insufficient identification and response speed for network attacks. For new security novices, the challenges they face include lack of practical experience, unfamiliarity with complex network environments, and untimely updating of technical knowledge. The present invention can automatically process and analyze large-scale traffic data, significantly improving the efficiency of identifying and responding to network threats. At the same time, it provides an effective learning tool for security novices, helping them quickly familiarize themselves with the network security environment and master the latest security knowledge, thereby improving the work efficiency of the entire security team. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and system for constructing an industry large model for network traffic analysis and detection.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A method for constructing an industry large model for network traffic analysis and detection includes the following steps:

[0007] Obtain network traffic log data and build IP intelligence database and security vulnerability database;

[0008] Generate event details, event identification and disposal suggestions based on network traffic log data, IP intelligence library and security vulnerability library, form a traffic analysis report by combining event details, event identification and disposal suggestions, and combine the traffic analysis report and network traffic log data to form a pre-training data set;

[0009] Use pre-trained datasets and Lora technology to fine-tune the language model and generate industry-leading models for network traffic analysis and detection.

[0010] Furthermore, the obtaining of network traffic log data includes:

[0011] Deploy a traffic detection probe at the front end of the application server. The traffic detection probe is responsible for detecting various attack activities including viruses, Trojans, DDoS attacks, and weak password attacks, and records the attack traffic logs in the traffic log database;

[0012] For the fields in the attack traffic log, key information in the overlong fields is extracted to reduce the field length, the base64 encrypted fields are decoded, and corresponding key fields are screened for specific attack types.

[0013] Furthermore, the IP intelligence library is constructed by the following steps:

[0014] Collect data regularly from public IP intelligence websites, including historical domain names bound to IPs, city-level and district-level locations, affiliated companies, responsible persons, and risk assessment information;

[0015] Use XPath and regular expression technology to extract IP intelligence from the collected data and update it to the IP intelligence database.

[0016] Furthermore, the security vulnerability library is constructed by the following steps:

[0017] Visit the target websites under the security vulnerability intelligence website directory one by one, conduct preliminary manual queries, and verify whether key vulnerability information can be obtained, including AVD number, link, threat rating, affected products, name, type, disclosure date, vulnerability description, handling suggestions, reference link, classification, and vulnerability identification fields;

[0018] For websites that can obtain key vulnerability information, save their HTML files and analyze the DOM structure of the vulnerability information to construct XPath expressions, which are then used for machine batch query to automatically collect vulnerability information;

[0019] After data collection is completed, vulnerability intelligence from different websites is merged according to AVD numbers and updated to the security vulnerability library.

[0020] Furthermore, the generating of event details, event identification and disposal suggestions based on network traffic log data, IP intelligence database and security vulnerability database includes:

[0021] Event details generation: By matching network traffic log data with key fields in the IP intelligence library, identification of discovery time, threat level, alert application, attack location, and malicious domain name is generated as event details;

[0022] Event identification and generation: By matching network traffic log data with the data in the security vulnerability database, the nature of traffic can be determined and the attack type and behavior can be identified;

[0023] Generation of disposal suggestions: Utilizing the analytical capabilities of the open source security big model, event details, event identification, and network traffic log data are integrated as input data, and disposal suggestions with high generalization and accuracy are generated through prompt words.

[0024] Furthermore, the use of the pre-trained dataset and Lora technology to fine-tune the language model to generate an industry model for network traffic analysis and detection includes:

[0025] Using pre-trained data as input and combining it with specific prompt words, Lora-based instruction fine-tuning is performed on the language model. The traffic analysis report generated by the language model is required to be similar to the traffic analysis report in the pre-training dataset.

[0026] Furthermore, the prompt words of the language big model are designed as the role of traffic security analysis expert, whose task is to analyze and generate event details, event identification and disposal suggestions from the attack traffic log. The output format is a dictionary containing relevant descriptions, and the language big model is fine-tuned in an instruction manner, and some parameters of the language big model are updated using Lora. The traffic analysis report generated by the language big model is compared with the traffic analysis report in the training set to calculate the MSE loss for supervised training, and finally an industry big model for network traffic analysis and detection is obtained.

[0027] A large industry model building system for network traffic analysis and detection, comprising:

[0028] Traffic log processing and security intelligence library building module, used to obtain network traffic log data and build IP intelligence library and security vulnerability library;

[0029] Traffic-analysis report fine-tuning training set generation module, which is used to generate event details, event identification and disposal suggestions based on network traffic log data, IP intelligence library and security vulnerability library, and form a traffic analysis report by combining event details, event identification and disposal suggestions, and the traffic analysis report and network traffic log data form a pre-training data set;

[0030] The large model fine-tuning module is used to fine-tune the language large model using the pre-trained dataset and Lora technology to generate an industry large model for network traffic analysis and detection.

[0031] The beneficial effects of the present invention are as follows:

[0032] The present invention is based on network traffic and text data association technology, combined with network security knowledge and large model fine-tuning technology, to achieve the construction of a large model for the network traffic industry and network traffic analysis and detection. The present invention has the ability to understand and analyze network traffic in natural language, can effectively detect and judge abnormal traffic, and automatically generate a judgment report, providing a reliable basis for supervisors to judge network attacks and effectively improving the level of network governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a basic framework diagram of an industry large model construction method for network traffic analysis and detection.

[0034] Figure 2 The present invention is a flowchart of a method for building an industry large model for network traffic analysis and detection. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments.

[0036] The present invention provides a method for constructing an industry large model for network traffic analysis and detection, comprising the following steps:

[0037] S1: Traffic log processing and security intelligence database construction;

[0038] S2: Generation of fine-tuning training set for traffic-analysis report;

[0039] S3: Use Lora (low-rank adaptation) technology to fine-tune the language model to generate an industry-leading model for network traffic analysis and detection.

[0040] The present invention provides a method for constructing a large industry model for network traffic analysis and detection, and its basic framework is as follows: Figure 1 As shown, the overall process is as follows Figure 2 As shown, each step is described in detail below:

[0041] S1-1: Front-end traffic probe deployment and traffic preprocessing. The purpose of this step is to deploy traffic detection probes at the front end of the application server, filter and record traffic data according to predetermined rules, and then preprocess these data and store them in the traffic log database.

[0042] S1-2: IP intelligence database construction. The purpose of this step is to develop scripts to regularly collect data from public IP intelligence websites, use XPath and regular expression technology to extract IP intelligence, and update it to the IP intelligence database.

[0043] S1-3: Security vulnerability library construction. The purpose of this step is to build a resource acquisition script for the public vulnerability intelligence website. Based on XPath and IP regular expression search, IP vulnerability intelligence is regularly collected and the data is updated to the security vulnerability library.

[0044] S2-1: Event details generation. The purpose of this step is to generate identification of discovery time, threat level, alarm application, attack location and malicious domain name by matching key fields in traffic logs and IP intelligence library.

[0045] S2-2: Event identification generation. The purpose of this step is to match the traffic log with the data in the security vulnerability database, determine the nature of the traffic, and identify the attack type and behavior.

[0046] S2-3: Generation of disposal suggestions: This step aims to utilize the analytical capabilities of the open source security big model to integrate event details, event identification, and traffic logs as input data. Through the designed prompt, this system can generate disposal suggestions with high generalization and accuracy.

[0047] S2-4: Generation of pre-training data for traffic analysis report. The purpose of this step is to combine event details, event identification, and disposal suggestions to form a traffic analysis report, which together with the traffic log constitutes a pre-training data set.

[0048] S3: Use Lora technology to fine-tune the language model to generate an industry model for network traffic analysis and detection. Use the traffic log in the pre-training data as input, combined with specific prompts, to perform Lora-based instruction fine-tuning on the language model. The judgment report generated by the large model is required to be similar to the judgment report in the training set.

[0049] Exemplarily, as described in step 1-1, a flow detection probe is deployed at the front end of the application server, which is responsible for detecting various attack activities including viruses, Trojans, DDoS attacks, and weak password attacks, and recording attack flow logs in the database. These logs include key fields such as attack type, hit rule, request body, HTTP header plaintext field, port, IP address, and date. In addition, for the fields in the log, the system will extract key information from overly long fields to reduce the field length, decode the base64 encrypted fields, and filter the corresponding key fields for specific attack types.

[0050] Exemplarily, as described in steps 1-2, the IP intelligence database construction process first visits the target website one by one according to the IP intelligence website directory for preliminary manual query to check whether the historical binding domain name of the IP, city-level and district-level location, affiliated company, person in charge, risk assessment and other information can be obtained. For websites that can provide this information, the system will save its HTML file and analyze the DOM structure of the relevant IP information, based on which XPath expressions are constructed for subsequent machine batch queries to automatically collect IP information. Finally, the system merges the IP intelligence obtained from different websites by IP address and updates it to the IP intelligence database, completing the whole process of data collection and integration.

[0051] Exemplarily, as described in step 1-3, the process of building a security vulnerability library begins with visiting the target website under the security vulnerability intelligence website directory one by one, and performs preliminary manual query. The purpose of this stage is to verify whether key vulnerability information can be obtained, including information such as AVD numbering, link, threat rating, impact product, name, type, disclosure date, vulnerability description, disposal suggestion, reference link, classification, and vulnerability identification field. For the website that can provide this information, the system will save its HTML file and analyze the DOM structure of vulnerability information, construct XPath expression with this, and then be used for machine batch query, automatically collect vulnerability information. After data collection is completed, the system will merge the vulnerability intelligence from different websites by AVD numbering, and update to the security vulnerability intelligence database, thereby complete the whole data collection and integration process.

[0052] Exemplarily, as described in step 2-1, the event details generation process first extracts a record from the traffic log database. Using the IP address and event occurrence time in the record, the IP intelligence library is queried to obtain relevant information, including the most recent domain name bound to this IP, city-level and district-level location, affiliated company, person in charge, and risk assessment. If a high-risk or malicious domain name is identified, the system will generate a tabular event detail containing these details based on the occurrence date of the traffic log, the relevant application, and the matching IP intelligence library information.

[0053] Exemplarily, as described in step 2-2, the event identification generation process first extracts a record from the traffic log database. The attack type in the record is used to query the security vulnerability intelligence library, and the port, protocol, and request body in the traffic record are matched according to the common ports, protocols, and vulnerability identification fields of the attack type recorded in the security vulnerability intelligence library to determine whether this traffic is indeed attack traffic. If so, a threat rating is made, and the attack description includes the port, protocol, vulnerability identification field, affected product, and attack name, forming an event identification description.

[0054] Exemplarily, as described in steps 2-3, the steps for generating disposal suggestions are as follows: First, design a prompt for a large model to clearly instruct the security large model to play the role of a traffic security analysis expert. Next, set the scenario to be based on the description of the security event and generate specific disposal suggestions for the attack event. These suggestions should include at least three items, such as removal or modification measures, security patch updates, and personnel security education, to ensure that each suggestion is practical and accurate. The input data is provided in the form of a dictionary, including the following: event details (event details table converted into text description), event discrimination (event discrimination description), and traffic log request (traffic log request body). After this information is input into the security large model, the corresponding disposal suggestions will be generated. To ensure the quality of the generated suggestions, some disposal suggestions generated in the traffic logs of different attack types are manually evaluated. If it is found that the suggestions are biased, not in line with reality, or answer inaccurate questions, the corresponding prompt will be adjusted. This process is repeated until most of the disposal suggestions meet the requirements. Once this standard is met, the prompt of the attack type can be saved as a template for future large-scale disposal suggestion generation.

[0055] Exemplarily, as described in steps 2-4, the pre-training data generation process of the traffic analysis report first extracts the key fields of the traffic log, including attack type, hit rule, request body, HTTP header plaintext field, port, IP address and date, etc. This information is stored in a dictionary format. Then, according to the previous steps (S2-2, S2-3, S2-4), a detailed description of the event, a discriminant analysis of the event and disposal suggestions are generated. These three parts of information are integrated in sequence to form a complete traffic log analysis report. The integrated analysis report is then stored in a document, and the path of the analysis report document is recorded in the dictionary for easy indexing and access. Finally, the dictionary containing the key fields and the analysis report path is stored as a record in a JSON file. Each record is used as part of the pre-training data set for future model training.

[0056] After this organization, each piece of data includes complete event information and corresponding processing suggestions, which helps to build an efficient and powerful pre-training dataset.

[0057] Exemplarily, as described in step 3, the language big model is fine-tuned using Lora technology to generate an industry big model for network traffic analysis and detection. First, batch data is extracted from the pre-training data set, each batch includes multiple record dictionaries, which contain key log fields such as attack type, hit rule, request body, HTTP header plaintext field, port, IP address and date. Then, the specified log file path content is loaded as ground truth. The prompt of the model is designed as a traffic security analysis expert role, and the task is to analyze and generate event details, event discrimination and disposal suggestions from the attack traffic log, and the output format is a dictionary containing relevant descriptions. In the instruction fine-tuning method, and using Lora to update some parameters of the big model, the judgment report generated by the big model should be supervised training with the judgment report in the training set to find the MSE loss, and finally obtain the traffic analysis industry big model. In actual use, the data including prompt and traffic log dictionary are input to the traffic analysis industry big model, and the model outputs a dictionary result after processing, and these results are processed and merged to finally generate a complete traffic analysis judgment report. The result in dictionary form refers to {event details: event details content; event discrimination: event discrimination content; disposal suggestions: disposal suggestion content}, and merging refers to arranging the dictionary contents from top to bottom according to event details, event discrimination, and disposal suggestions, and generating a word document for saving.

[0058] Another embodiment of the present invention provides an industry large model construction system for network traffic analysis and detection, which includes:

[0059] Traffic log processing and security intelligence library building module, used to obtain network traffic log data and build IP intelligence library and security vulnerability library;

[0060] Traffic-analysis report fine-tuning training set generation module, which is used to generate event details, event identification and disposal suggestions based on network traffic log data, IP intelligence library and security vulnerability library, and form a traffic analysis report by combining event details, event identification and disposal suggestions, and the traffic analysis report and network traffic log data form a pre-training data set;

[0061] The large model fine-tuning module is used to fine-tune the language large model using the pre-trained dataset and Lora technology to generate an industry large model for network traffic analysis and detection.

[0062] The specific implementation process of each module refers to the above description of the method of the present invention.

[0063] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0064] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0065] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A method for constructing an industry large model for network traffic analysis and detection, characterized in that: The following steps are involved: Obtain network traffic log data and build IP intelligence database and security vulnerability database; Generate event details, event identification and disposal suggestions based on network traffic log data, IP intelligence library and security vulnerability library, form a traffic analysis report by combining event details, event identification and disposal suggestions, and combine the traffic analysis report and network traffic log data to form a pre-training data set; Use pre-trained datasets and Lora technology to fine-tune the language model and generate industry-leading models for network traffic analysis and detection.

2. The method according to claim 1, characterized in that The obtaining of network traffic log data includes: Deploy a traffic detection probe at the front end of the application server. The traffic detection probe is responsible for detecting various attack activities including viruses, Trojans, DDoS attacks, and weak password attacks, and records the attack traffic logs in the traffic log database; For the fields in the attack traffic log, key information in the overlong fields is extracted to reduce the field length, the base64 encrypted fields are decoded, and corresponding key fields are screened for specific attack types.

3. The method according to claim 1, characterized in that The IP intelligence database is constructed using the following steps: Collect data regularly from public IP intelligence websites, including historical domain names bound to IPs, city-level and district-level locations, affiliated companies, responsible persons, and risk assessment information; Use XPath and regular expression technology to extract IP intelligence from the collected data and update it to the IP intelligence database.

4. The method according to claim 1, characterized in that: The security vulnerability library is constructed using the following steps: Visit the target websites under the security vulnerability intelligence website directory one by one, conduct preliminary manual queries, and verify whether key vulnerability information can be obtained, including AVD number, link, threat rating, affected products, name, type, disclosure date, vulnerability description, handling suggestions, reference link, classification, and vulnerability identification fields; For websites that can obtain key vulnerability information, save their HTML files and analyze the DOM structure of the vulnerability information to construct XPath expressions, which are then used for machine batch query to automatically collect vulnerability information; After data collection is completed, vulnerability intelligence from different websites is merged according to AVD numbers and updated to the security vulnerability library.

5. The method according to claim 1, characterized in that The generation of event details, event identification and handling suggestions based on network traffic log data, IP intelligence library and security vulnerability library includes: Event details generation: By matching network traffic log data with key fields in the IP intelligence library, identification of discovery time, threat level, alert application, attack location, and malicious domain name is generated as event details; Event identification and generation: By matching network traffic log data with the data in the security vulnerability database, the nature of traffic can be determined and the attack type and behavior can be identified; Generation of disposal suggestions: Utilizing the analytical capabilities of the open source security big model, event details, event identification, and network traffic log data are integrated as input data, and disposal suggestions with high generalization and accuracy are generated through prompt words.

6. The method according to claim 1, characterized in that The method of using the pre-trained dataset and Lora technology to fine-tune the language model and generate an industry model for network traffic analysis and detection includes: Using pre-trained data as input and combining it with specific prompt words, Lora-based instruction fine-tuning is performed on the language model. The traffic analysis report generated by the language model is required to be similar to the traffic analysis report in the pre-training dataset.

7. The method according to claim 6, characterized in that The prompt words of the language big model are designed as the role of traffic security analysis expert, whose task is to analyze and generate event details, event identification and disposal suggestions from the attack traffic log. The output format is a dictionary containing relevant descriptions, and the instruction fine-tuning method is used to update some parameters of the language big model using Lora. The traffic analysis report generated by the language big model and the traffic analysis report in the training set are compared to calculate the MSE loss for supervised training, and finally an industry big model for network traffic analysis and detection is obtained.

8. An industry large model construction system for network traffic analysis and detection, characterized in that: include: Traffic log processing and security intelligence library building module, used to obtain network traffic log data and build IP intelligence library and security vulnerability library; Traffic-analysis report fine-tuning training set generation module, which is used to generate event details, event identification and disposal suggestions based on network traffic log data, IP intelligence library and security vulnerability library, and form a traffic analysis report by combining event details, event identification and disposal suggestions, and the traffic analysis report and network traffic log data form a pre-training data set; The large model fine-tuning module is used to fine-tune the language large model using the pre-trained dataset and Lora technology to generate an industry large model for network traffic analysis and detection.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images