An industry large model construction method and system for network traffic analysis detection

Abstract

The present application belongs to the field of network traffic, and relates to an industry large model construction method and system for network traffic analysis and detection. The method comprises: obtaining network traffic log data, and constructing an IP intelligence library and a security vulnerability library; generating event details, event discrimination and disposal suggestions according to the network traffic log data, the IP intelligence library and the security vulnerability library, forming a traffic research and judgment report by merging the event details, the event discrimination and the disposal suggestions, and constituting a pre-training data set with the traffic research and judgment report and the network traffic log data; and fine-tuning a language large model by using the pre-training data set and Lora technology, and generating an industry large model for network traffic analysis and detection. The present application has natural language understanding and analysis capabilities for network traffic, can effectively detect and research and judge abnormal traffic, and automatically generates a research and judgment report, thereby providing a reliable basis for regulatory personnel to research and judge network attacks and effectively improving network governance level.

Description

Technical Field

[0001] This invention belongs to the field of network traffic, specifically relating to a method and system for constructing large-scale industry models for network traffic analysis and detection. Background Technology

[0002] With the rapid development of network applications and the proliferation of network traffic data, the content has evolved from plaintext to encryption, increasing the difficulty of traffic data analysis and detection. Previously, many researchers used deep neural network models to extract deep features from raw traffic, significantly improving traffic identification capabilities. Currently, the main traffic identification technologies fall into three categories: First, fingerprint-based traffic identification technologies. FlowPrint proposes representing each flow by extracting device, certificate, size, and time features, and constructing a fingerprint database through clustering and cross-correlation for effective traffic classification. However, this fingerprint information is easily tampered with in virtual communication networks, losing its correct meaning. Second, feature-based traffic identification methods. AppScanner uses statistical features of packet size to train a random forest classifier for network traffic identification, while BIND uses statistical features of traffic time for traffic identification model training. However, this method struggles to design general statistical properties to handle the ever-increasing complexity of numerous applications and websites, exhibiting weak generalization. Third, deep learning-based traffic identification technologies. The DF model uses convolutional neural networks (CNNs), FS-Net uses recurrent neural networks (RNNs) to automatically extract traffic representations from the raw packet size sequence of encrypted traffic, while the TSCRNN model describes the raw payload. However, these methods rely on a large amount of supervised data to capture effective features, thus learning biased representations in imbalanced data. The above methods have achieved traffic identification and efficient detection of anomalous traffic with low resources for specific categories of web pages and applications. However, techniques such as fingerprint mining, statistical feature construction, and deep feature extraction suffer from weak generalization and strong reliance on expert experience due to limitations such as the simplification of feature construction and dependence on labeled data.

[0003] Existing security personnel and regulators often struggle to effectively address complex, ever-changing, and massive cyber threats due to a lack of highly generalized and automated tools, resulting in insufficient speed in identifying and responding to cyberattacks. For newly hired security novices, challenges include a lack of practical experience, unfamiliarity with complex network environments, and outdated technical knowledge. This invention automates the processing and analysis of large-scale traffic data, significantly improving the efficiency of cyber threat identification and response. Simultaneously, it provides security novices with an effective learning tool, helping them quickly familiarize themselves with the cybersecurity environment and master the latest security knowledge, thereby improving the overall efficiency of the security team. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a method and system for constructing large-scale industry models for network traffic analysis and detection.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for constructing a large-scale industry model for network traffic analysis and detection includes the following steps:

[0007] Acquire network traffic log data and build an IP intelligence database and a security vulnerability database;

[0008] Based on network traffic log data, IP intelligence database, and security vulnerability database, event details, event identification, and handling suggestions are generated. By merging the event details, event identification, and handling suggestions, a traffic analysis report is formed. The traffic analysis report and network traffic log data are combined to form a pre-trained dataset.

[0009] By fine-tuning a large language model using pre-trained datasets and LoRa technology, industry-wide large models for network traffic analysis and detection are generated.

[0010] Furthermore, the acquisition of network traffic log data includes:

[0011] Deploy a traffic detection probe at the front end of the application server. The traffic detection probe is responsible for detecting various attack activities, including viruses, Trojans, DDoS attacks and weak password attacks, and recording the attack traffic logs to the traffic log database.

[0012] For the fields in the attack traffic log, extract key information from excessively long fields to reduce field length, decode base64-encrypted fields, and filter corresponding key fields for specific attack types.

[0013] Furthermore, the IP intelligence database is constructed using the following steps:

[0014] Data is collected regularly from publicly available IP intelligence websites, including the IP's historical domain names, city-level and district / county-level location, affiliated company, person in charge, and risk assessment information;

[0015] XPath and regular expression techniques are used to extract IP intelligence from the collected data and update it to the IP intelligence database.

[0016] Furthermore, the security vulnerability database is constructed using the following steps:

[0017] Visit each target website in the security vulnerability intelligence website directory one by one, conduct preliminary manual queries, and verify whether key vulnerability information can be obtained, including AVD number, link, threat rating, affected products, name, type, disclosure date, vulnerability description, remediation suggestions, reference links, classification, and vulnerability identification field;

[0018] For websites that can provide key vulnerability information, save their HTML files and analyze the DOM structure of the vulnerability information to construct XPath expressions, which can then be used for batch machine queries to automatically collect vulnerability information.

[0019] After data collection is completed, vulnerability intelligence from different websites is merged according to AVD number and updated to the security vulnerability database.

[0020] Furthermore, the generation of event details, event identification, and handling suggestions based on network traffic log data, IP intelligence database, and security vulnerability database includes:

[0021] Event details generation: By matching network traffic log data with key fields in the IP intelligence database, an identifier is generated for the discovery time, threat level, alert application, attack location, and malicious domain name, which serves as the event details;

[0022] Event identification generation: By matching network traffic log data with data from the security vulnerability database, the nature of the traffic is determined, and the attack type and behavior are identified;

[0023] Action Recommendation Generation: Leveraging the analytical capabilities of open-source security big data models, the system integrates event details, event identification, and network traffic log data as input data, and generates action recommendations with high generalization and accuracy through prompt words.

[0024] Furthermore, the process of fine-tuning the large language model using a pre-trained dataset and LoRa technology to generate an industry-wide large model for network traffic analysis and detection includes:

[0025] Using pre-trained data as input, and combining specific prompt words, LoRa-based instruction fine-tuning is performed on a large language model, requiring that the traffic analysis report generated by the large language model be similar to the traffic analysis report in the pre-trained dataset.

[0026] Furthermore, the prompt words of the language big model are designed as traffic security analysis experts. Their task is to analyze and generate event details, event identification, and handling suggestions from attack traffic logs. The output format is a dictionary containing relevant descriptions. The language big model is fine-tuned using instructions, and some parameters are updated using LoRa. The traffic analysis report generated by the language big model is compared with the traffic analysis reports in the training set to calculate the MSE loss for supervised training, ultimately obtaining an industry big model for network traffic analysis and detection.

[0027] An industry-wide large-scale model construction system for network traffic analysis and detection, comprising:

[0028] The traffic log processing and security intelligence database construction module is used to acquire network traffic log data and build an IP intelligence database and a security vulnerability database.

[0029] The Traffic Analysis Report Fine-tuning Training Set Generation Module is used to generate event details, event identification, and handling suggestions based on network traffic log data, IP intelligence database, and security vulnerability database. It forms a traffic analysis report by merging event details, event identification, and handling suggestions, and combines the traffic analysis report with network traffic log data to form a pre-trained dataset.

[0030] The large model fine-tuning module is used to fine-tune large language models using pre-trained datasets and LoRa technology, generating industry-specific large models for network traffic analysis and detection.

[0031] The beneficial effects of this invention are as follows:

[0032] This invention, based on network traffic and text data correlation technology, combined with network security knowledge and large-scale model fine-tuning technology, enables the construction of large-scale network traffic models and the analysis and detection of network traffic. This invention possesses natural language understanding and analysis capabilities for network traffic, effectively detecting and judging abnormal traffic, and automatically generating analysis reports. This provides reliable evidence for regulators to assess network attacks, effectively improving network governance. Attached Figure Description

[0033] Figure 1 This is a basic framework diagram of an industry-wide large-scale model construction method for network traffic analysis and detection.

[0034] Figure 2 This is a flowchart of an industry-wide large-scale model construction method for network traffic analysis and detection. Detailed Implementation

[0035] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below through specific embodiments.

[0036] This invention provides a method for constructing a large-scale industry model for network traffic analysis and detection, comprising the following steps:

[0037] S1: Traffic log processing and security intelligence database construction;

[0038] S2: Traffic Analysis Report Fine-tuning Training Set Generation;

[0039] S3: Utilizes LoRa (low-rank adaptive) technology to fine-tune large language models to generate industry-wide large models for network traffic analysis and detection.

[0040] This invention provides a method for constructing a large-scale industry model for network traffic analysis and detection, the basic framework of which is as follows: Figure 1 As shown, the overall process is as follows: Figure 2 As shown below, each step will be explained in detail:

[0041] S1-1: Front-end traffic probe deployment and traffic preprocessing. The purpose of this step is to deploy a traffic detection probe at the front end of the application server, filter and record traffic data according to predetermined rules, and then preprocess this data and store it in the traffic log database.

[0042] S1-2: IP Intelligence Database Construction. The purpose of this step is to develop scripts to regularly collect data from publicly available IP intelligence websites, extract IP intelligence using XPath and regular expression techniques, and update the IP intelligence database.

[0043] S1-3: Security vulnerability database construction. The purpose of this step is to build a resource acquisition script for a public vulnerability intelligence website, to regularly collect vulnerability intelligence of IPs based on XPath and IP regular expressions, and to update the data into the security vulnerability database.

[0044] S2-1: Event Details Generation. The purpose of this step is to generate identifiers about the discovery time, threat level, alert application, attack location, and malicious domain by matching key fields in traffic logs and the IP intelligence database.

[0045] S2-2: Event identification generation. The purpose of this step is to determine the nature of the traffic and identify the attack type and behavior by matching the traffic logs with the data in the security vulnerability database.

[0046] S2-3: Action Recommendation Generation: This step aims to leverage the analytical capabilities of the open-source security big data model to integrate event details, event identification, and traffic logs as input data. Through a designed prompt, the system can generate action recommendations with high generalization and accuracy.

[0047] S2-4: Generation of pre-training data for traffic analysis report. The purpose of this step is to merge event details, event identification, and handling suggestions to form a traffic analysis report, which together with traffic logs constitutes a pre-training dataset.

[0048] S3: Fine-tune the large language model using LoRa technology to generate an industry-wide model for network traffic analysis and detection. Utilizing traffic logs from the pre-training data as input, and combining them with specific prompts, perform LoRa-based instruction fine-tuning on the large language model. The analysis reports generated by the large model should be similar to those in the training set.

[0049] For example, as described in step 1-1, a traffic detection probe is deployed at the front end of the application server. This probe is responsible for detecting various attack activities, including viruses, Trojans, DDoS attacks, and weak password attacks, and records the attack traffic logs in the database. These logs include key fields such as attack type, hit rules, request body, plaintext HTTP header fields, port, IP address, and date. Furthermore, for fields in the logs, the system extracts key information from excessively long fields to reduce field length, decodes base64-encrypted fields, and filters corresponding key fields for specific attack types.

[0050] For example, as described in steps 1-2, the IP intelligence database construction process first involves manually querying target websites one by one according to the IP intelligence website directory to check if information such as the IP's historical bound domain name, city-level and district-level location, affiliated company, responsible person, and risk assessment can be obtained. For websites that can provide this information, the system saves their HTML files and analyzes the DOM structure of the relevant IP information. Based on this, an XPath expression is constructed for subsequent batch machine queries, automatically collecting IP information. Finally, the system merges the IP intelligence obtained from different websites according to IP address and updates it to the IP intelligence database, completing the entire process of data collection and integration.

[0051] For example, as described in steps 1-3, the process of building a security vulnerability database begins with manually accessing target websites in the security vulnerability intelligence website directory one by one. The purpose of this stage is to verify whether key vulnerability information can be obtained, including AVD number, link, threat rating, affected products, name, type, disclosure date, vulnerability description, remediation recommendations, referencing links, classification, and vulnerability identification fields. For websites that can provide this information, the system will save their HTML files and analyze the DOM structure of the vulnerability information to construct XPath expressions, which will then be used for batch machine queries to automatically collect vulnerability information. After data collection is completed, the system will merge vulnerability intelligence from different websites by AVD number and update it to the security vulnerability intelligence database, thereby completing the entire data collection and integration process.

[0052] For example, as described in step 2-1, the event details generation process first extracts a record from the traffic log database. Using the IP address and event occurrence time in this record, the system queries the IP intelligence database to obtain relevant information, including the IP's most recently bound domains, city-level and district-level locations, affiliated company, responsible person, and risk assessment. If a high-risk or malicious domain is identified, the system will generate a tabular event details list containing these details based on the traffic log occurrence date, related applications, and matching IP intelligence database information.

[0053] For example, as described in step 2-2, the event discrimination generation process first extracts a record from the traffic log database. Using the attack type in this record, it queries the security vulnerability intelligence database. Based on the commonly used ports, protocols, and vulnerability identifier fields of this attack type recorded in the security vulnerability intelligence database, it matches the port, protocol, and request body in the traffic record to determine whether the traffic is indeed attack traffic. If so, a threat rating is made, and the attack description includes the port, protocol, vulnerability identifier field, affected products, and attack name, forming an event discrimination description.

[0054] For example, as described in steps 2-3, the steps for generating handling recommendations are as follows: First, design a prompt for a large security model, explicitly instructing the large security model to act as a traffic security analysis expert. Next, set the scenario as a description of a security event and generate specific handling recommendations for that attack event. These recommendations should include at least three items, such as cleanup or modification measures, security patch updates, and personnel security training, ensuring that each recommendation is practical and accurate. Input data is provided in dictionary form, containing the following: event details (event details table converted to text description), event identification (event identification description), and traffic log request (traffic log request body). After this information is input into the large security model, corresponding handling recommendations will be generated. To ensure the quality of the generated recommendations, some handling recommendations generated from traffic logs of different attack types are manually evaluated. If the recommendations are found to be biased, unrealistic, or answer inaccurate questions, the corresponding prompt will be adjusted. This process is repeated until most handling recommendations meet the requirements. Once this standard is met, the prompt for that attack type can be saved as a template for future large-scale handling recommendation generation.

[0055] For example, following steps 2-4, the pre-training data generation process for the traffic analysis report first extracts key fields from the traffic logs, including attack type, hit rules, request body, plaintext HTTP header fields, port, IP address, and date. This information is stored in dictionary format. Then, based on previous steps (S2-2, S2-3, S2-4), a detailed description of the event, event discrimination analysis, and handling recommendations are generated. These three parts of information are integrated sequentially to form a complete traffic log analysis report. Next, the integrated analysis report is stored in a document, and the path of the analysis report document is recorded in the dictionary for easy indexing and access. Finally, the dictionary containing the key fields and the analysis report path is stored as a record in a JSON file. Each record serves as part of the pre-training dataset for future model training.

[0056] After this organization, each data point includes complete event information and corresponding processing suggestions, which helps to build an efficient and powerful pre-trained dataset.

[0057] For example, as described in step 3, the large language model is fine-tuned using LoRa technology to generate an industry-wide model for network traffic analysis and detection. First, batch data is extracted from the pre-training dataset. Each batch includes a dictionary of multiple records containing key log fields such as attack type, hit rule, request body, plaintext HTTP header fields, port, IP address, and date. Next, the contents of a specified log file path are loaded as the ground truth. The model's prompt is designed as a traffic security analysis expert role, tasked with analyzing and generating event details, event identification, and handling recommendations from attack traffic logs. The output format is a dictionary containing relevant descriptions. Fine-tuning is performed using instructions, and LoRa is used to update some parameters of the large model. The analysis report generated by the large model is then subjected to supervised training by calculating the MSE loss against the analysis reports in the training set, ultimately yielding the industry-wide traffic analysis model. In practical use, the traffic analysis industry-wide model is input with data including the prompt and the traffic log dictionary. After processing, the model outputs a dictionary-style result. These results are then processed and merged to generate a complete traffic analysis analysis report. The dictionary-style result refers to {Event Details: Event Details Content; Event Judgment: Event Judgment Content; Handling Suggestions: Handling Suggestion Content}. Merging means arranging the dictionary contents from top to bottom according to Event Details, Event Judgment, and Handling Suggestions, and then generating a Word document for saving.

[0058] Another embodiment of the present invention provides an industry-wide large model construction system for network traffic analysis and detection, comprising:

[0059] The traffic log processing and security intelligence database construction module is used to acquire network traffic log data and build an IP intelligence database and a security vulnerability database.

[0060] The Traffic Analysis Report Fine-tuning Training Set Generation Module is used to generate event details, event identification, and handling suggestions based on network traffic log data, IP intelligence database, and security vulnerability database. It forms a traffic analysis report by merging event details, event identification, and handling suggestions, and combines the traffic analysis report with network traffic log data to form a pre-trained dataset.

[0061] The large model fine-tuning module is used to fine-tune large language models using pre-trained datasets and LoRa technology, generating industry-specific large models for network traffic analysis and detection.

[0062] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0063] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0064] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0065] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for constructing a large-scale industry model for network traffic analysis and detection, characterized in that, Includes the following steps: Acquire network traffic log data and build an IP intelligence database and a security vulnerability database; Based on network traffic log data, IP intelligence database, and security vulnerability database, event details, event identification, and handling suggestions are generated. By merging the event details, event identification, and handling suggestions, a traffic analysis report is formed. The traffic analysis report and network traffic log data are combined to form a pre-trained dataset. Fine-tuning of a large language model using pre-trained datasets and LoRa technology generates industry-wide large models for network traffic analysis and detection; The acquisition of network traffic log data includes: Deploy a traffic detection probe at the front end of the application server. The traffic detection probe is responsible for detecting various attack activities, including viruses, Trojans, DDoS attacks and weak password attacks, and recording the attack traffic logs to the traffic log database. For the fields in the attack traffic log, extract key information from excessively long fields to reduce field length, decode base64 encrypted fields, and filter corresponding key fields for specific attack types. The IP intelligence database is constructed using the following steps: Data is collected regularly from publicly available IP intelligence websites, including the IP's historical domain names, city-level and district / county-level location, affiliated company, person in charge, and risk assessment information; XPath and regular expression techniques are used to extract IP intelligence from the collected data and update it to the IP intelligence database. The security vulnerability database is constructed using the following steps: Visit each target website in the security vulnerability intelligence website directory one by one, conduct preliminary manual queries, and verify whether key vulnerability information can be obtained, including AVD number, link, threat rating, affected products, name, type, disclosure date, vulnerability description, remediation suggestions, reference links, classification, and vulnerability identification field; For websites that can provide key vulnerability information, save their HTML files and analyze the DOM structure of the vulnerability information to construct XPath expressions, which can then be used for batch machine queries to automatically collect vulnerability information. After data collection is completed, vulnerability intelligence from different websites is merged according to AVD number and updated to the security vulnerability database; The process of generating event details, event identification, and handling suggestions based on network traffic log data, IP intelligence database, and security vulnerability database includes: Event Details Generation: By matching network traffic log data with key fields in the IP intelligence database, event details are generated, including discovery time, threat level, alert application, attack location, and malicious domain. The event details generation process first extracts a record from the traffic log database. Using the IP address and event occurrence time in the record, the IP intelligence database is queried to obtain relevant information, including the IP's most recently bound domains, city-level and district-level locations, affiliated company, responsible person, and risk assessment. If a high-risk or malicious domain is identified, a table-style event details table containing these details is generated based on the traffic log occurrence date, related applications, and the matched IP intelligence database information. Event identification generation: By matching network traffic log data with data from the security vulnerability database, the nature of the traffic is determined, and the attack type and behavior are identified. The event identification generation process first extracts a record from the traffic log database, uses the attack type in the record to query the security vulnerability database, and matches the port, protocol, and request body in the traffic record with the common ports, protocols, and vulnerability identifier fields of the attack type recorded in the security vulnerability database to determine whether the traffic is indeed attack traffic. If so, a threat rating is made, and the attack description includes the port, protocol, vulnerability identifier field, affected products, and attack name, forming an event identification description. Action Recommendation Generation: Leveraging the analytical capabilities of an open-source security big data model, event details, event identification, and network traffic log data are integrated as input data. Highly generalizable and accurate action recommendations are generated using prompts. The steps for generating action recommendations are as follows: First, a prompt for the big data model is designed, explicitly instructing it to act as a traffic security analysis expert. Next, the scenario is set as a description of a security event, generating specific action recommendations for that attack event. These specific recommendations include at least three items: cleanup or modification measures, security patch updates, and personnel security training, ensuring that each recommendation is practical and accurate. Input data is provided in dictionary form, containing the following: event details, event identification, and traffic log requests. This information is input into the big data model to generate corresponding action recommendations. To ensure the quality of the generated recommendations, some action recommendations generated from traffic logs of different attack types are manually evaluated. If any recommendations are found to be biased, unrealistic, or inaccurate in their answers, the corresponding prompt is adjusted. This process is repeated until most action recommendations meet the requirements. Once this standard is met, the prompt for that attack type is saved as a template for future large-scale action recommendation generation.

2. The method according to claim 1, characterized in that, The process of fine-tuning a large language model using pre-trained datasets and LoRa technology to generate industry-wide large models for network traffic analysis and detection includes: Using pre-trained data as input, and combining specific prompt words, LoRa-based instruction fine-tuning is performed on a large language model, requiring that the traffic analysis report generated by the large language model be similar to the traffic analysis report in the pre-trained dataset.

3. The method according to claim 2, characterized in that, The prompt words of the language big model are designed as traffic security analysis experts. Their task is to analyze and generate event details, event identification, and handling suggestions from attack traffic logs. The output format is a dictionary containing relevant descriptions. The language big model is fine-tuned using instructions, and some parameters are updated using LoRa. The traffic analysis report generated by the language big model is compared with the traffic analysis reports in the training set to calculate the MSE loss for supervised training. Finally, an industry big model for network traffic analysis and detection is obtained.

4. An industry-wide large-scale model construction system for network traffic analysis and detection using the method described in any one of claims 1 to 3, characterized in that, include: The traffic log processing and security intelligence database construction module is used to acquire network traffic log data and build an IP intelligence database and a security vulnerability database. The Traffic Analysis Report Fine-tuning Training Set Generation Module is used to generate event details, event identification, and handling suggestions based on network traffic log data, IP intelligence database, and security vulnerability database. It forms a traffic analysis report by merging event details, event identification, and handling suggestions, and combines the traffic analysis report with network traffic log data to form a pre-trained dataset. The large model fine-tuning module is used to fine-tune large language models using pre-trained datasets and LoRa technology, generating industry-specific large models for network traffic analysis and detection.

5. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 3.

Images