Network flow detection method and device based on AI and electronic equipment

Abstract

The invention provides a network traffic detection method and device based on AI and electronic equipment. The method comprises the following steps: acquiring target network traffic data corresponding to original network traffic; the target network traffic data comprises a classification result carrying traffic feature data; inputting the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; judging whether the original network traffic is abnormal network traffic or not based on the detection result; if the network traffic is the abnormal network traffic, generating and displaying alarm information corresponding to the original network traffic; the detection model is obtained by training based on a pre-obtained target network traffic data set, and the target network traffic data set comprises sample target network traffic data marked with a detection result label. In the scheme, the anomaly detection of the network traffic is carried out through the detection model obtained by training based on the target network traffic data set, the abnormal traffic in the network can be intelligently identified and detected, and the detection accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic detection, and in particular to an AI-based network traffic detection method, device and electronic equipment. Background Art

[0002] At present, there is a surge in enterprise network traffic data, and new types of network threat attacks and data security risk events are emerging in an endless stream. Traditional network traffic detection technologies, including rule-based analysis engines, statistical and behavioral analysis, and threat intelligence data analysis, are difficult to adapt to the complex and changing network environment. Complex network attacks are difficult to identify accurately, and they are highly dependent on static content such as rules, statistical features, and attack signatures, resulting in low detection accuracy and efficiency. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an AI-based network traffic detection method, device and electronic device to intelligently identify abnormal traffic in the detection network and improve detection accuracy and efficiency.

[0004] The present invention provides an AI-based network traffic detection method, the method comprising:

[0005] Obtaining target network traffic data corresponding to the original network traffic; wherein the target network traffic data includes a classification result carrying traffic feature data;

[0006] Input the target network traffic data into the pre-trained detection model to obtain the detection results corresponding to the target network traffic data; the detection model includes: a threat detection model based on machine learning and a behavior detection model based on deep learning;

[0007] Based on the detection results, determine whether the original network traffic is abnormal network traffic;

[0008] If the original network traffic is abnormal network traffic, an alarm message corresponding to the original network traffic is generated and displayed; wherein, the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with a detection result label.

[0009] Furthermore, the detection model includes a behavior detection model and a threat detection model; wherein the threat detection model is obtained by learning a first target network traffic data set pre-acquired based on a machine learning method, and the first target network traffic data set includes a first sample target network traffic data labeled with a first detection result label; the first detection result label includes: a threat label or a no-threat label, and / or a threat type; the behavior detection model is obtained by learning a second target network traffic data set pre-acquired based on a deep learning method, and the second target network traffic data set includes a second sample target network traffic data labeled with a second detection result label; the second detection result label includes: a normal behavior label or an abnormal behavior label, and / or a behavior type.

[0010] Furthermore, the behavior detection model and threat detection model are trained in the following ways:

[0011] Inputting the first sample target network traffic data into the first initial model to extract detection features of the first sample target network traffic data through the first initial model, so as to obtain a first output result based on the detection features;

[0012] Based on the first output result and the first detection result label, the weight parameter of the first initial model is updated; and the step of inputting the first sample target network traffic data into the first initial model is continued until the first initial model converges to obtain a threat detection model;

[0013] Inputting the second sample target network traffic data into the second initial model to extract detection features of the second sample target network traffic data through the second initial model, so as to obtain a second output result based on the detection features;

[0014] Based on the second output result and the second detection result label, the weight parameters of the second initial model are updated; the step of inputting the second sample target network traffic data into the second initial model is continued until the second initial model converges to obtain a behavior detection model.

[0015] Furthermore, the step of obtaining target network traffic data corresponding to the original network traffic includes:

[0016] Monitor and receive original network traffic from the mirror port, splitter or optical splitter through the monitoring port;

[0017] Preprocessing the original network traffic through a preset processing tool to obtain first network traffic data; wherein the first network traffic data includes: parsing data, session statistics data, and data packet data;

[0018] The first network flow data is stored by a preset analysis component, and the stored first network flow data is preprocessed by a preset processing method to obtain target network flow data.

[0019] Further, the preset processing method includes: a first preset processing method and a second preset processing method; the step of preprocessing the stored first network flow data by the preset processing method to obtain the target network flow data includes:

[0020] Performing a first processing on the stored first network traffic data by a first preset processing method to obtain second network traffic data; wherein the first preset processing method includes traffic feature extraction processing and at least one of the following: cleaning processing, denoising processing, and formatting processing;

[0021] The second network traffic data is subjected to a second processing by a second preset processing method to obtain target network traffic data; the second preset processing method includes classification processing.

[0022] Further, the step of performing a second processing on the second network flow data by a second preset processing method to obtain the target network flow data includes:

[0023] The second network traffic data is input into a pre-trained classification model to obtain a classification result corresponding to the second network traffic data and carrying traffic feature data; wherein the second network traffic data includes traffic feature data.

[0024] Furthermore, the classification model includes classification sub-models corresponding to multiple classification dimensions, and the multiple classification dimensions include: plaintext classification, protocol classification, application classification, and behavior classification; the second network traffic data is input into the pre-trained classification model, and the step of obtaining a classification result carrying traffic feature data corresponding to the second network traffic data includes:

[0025] Input the second network traffic data into each pre-trained classification sub-model respectively, and obtain the classification sub-result output by each classification sub-model carrying the corresponding traffic sub-feature data; wherein each classification sub-model is trained based on the corresponding traffic sub-feature data set obtained in advance, and each traffic sub-feature data set includes the corresponding sample traffic sub-feature data marked with the classification sub-result label;

[0026] Each classification sub-result is determined as a classification result carrying flow characteristic data; wherein the flow characteristic data includes flow sub-characteristic data corresponding to each classification sub-model.

[0027] The present invention provides an AI-based network traffic detection device, the device comprising:

[0028] An acquisition module is used to acquire target network traffic data corresponding to the original network traffic; wherein the target network traffic data includes a classification result carrying traffic feature data;

[0029] An input module is used to input the target network traffic data into a pre-trained detection model to obtain the detection results corresponding to the target network traffic data; the detection model includes: a threat detection model based on machine learning and a behavior detection model based on deep learning;

[0030] A judgment module, used to judge whether the original network traffic is abnormal network traffic based on the detection result;

[0031] An alarm module is used to generate and display alarm information corresponding to the original network traffic if the original network traffic is abnormal network traffic; wherein the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with detection result labels.

[0032] An electronic device provided by the present invention includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement any of the above-mentioned AI-based network traffic detection methods.

[0033] The present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement any of the above-mentioned AI-based network traffic detection methods.

[0034] The present invention provides an AI-based network traffic detection method, device and electronic device, the method comprising: obtaining target network traffic data corresponding to the original network traffic; the target network traffic data includes a classification result carrying traffic feature data; inputting the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; based on the detection result, judging whether the original network traffic is abnormal network traffic; if it is abnormal network traffic, generating and displaying an alarm message corresponding to the original network traffic; the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with a detection result label. In this scheme, by performing abnormal network traffic detection based on the detection model trained based on the target network traffic data set, abnormal traffic in the detection network can be intelligently identified and detected, thereby improving detection accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of an AI-based network traffic detection method provided by an embodiment of the present invention;

[0037] Figure 2 A flowchart of another AI-based network traffic detection method provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the structure of an AI-based network traffic detection system provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of an AI-based network traffic detection device provided in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] At present, there is a surge in enterprise network traffic data, and new network threat attacks and data security risk events are emerging one after another. Traditional network traffic detection technologies, including rule-based analysis engines, statistical and behavioral analysis, and threat intelligence data analysis, are difficult to adapt to complex and changing network environments. Complex network attacks are difficult to accurately identify, and they are highly dependent on static content such as rules, statistical features, and attack signatures, resulting in low detection accuracy and efficiency.

[0043] With the update and iteration of IT technology architecture and the acceleration of enterprise digital transformation, enterprise network traffic data has surged, and new network threat attacks and data security risk events have emerged in an endless stream, causing serious impacts on enterprise production and operation activities. Traditional network traffic monitoring technology can only detect some attack behaviors based on known security vulnerabilities and malicious code characteristics. Network security monitoring faces constant challenges from new technologies, new applications, and new businesses, requiring us to further enhance the security threat monitoring of the entire network traffic and multi-scenario audit analysis capabilities.

[0044] The main technical solutions currently available include rule-based analysis engines, statistical and behavioral analysis, and threat intelligence data analysis.

[0045] Rule analysis engine solution: This technical solution matches and analyzes network traffic by setting a series of rules in advance, and issues an alarm or takes corresponding measures when the traffic data meets the rules. The implementation principle is to define a rule set, match and judge the traffic through the rule set, and make corresponding responses based on the matching results. The implementation steps include setting rules, monitoring traffic data, configuring the rule engine, matching rules, and generating threat alerts.

[0046] Statistical and behavioral analysis solution: This technical solution analyzes network traffic based on statistical methods and behavioral analysis technology. The implementation principle is to detect the statistical characteristics of traffic data and network behavior, and audit and identify abnormal behaviors of network traffic. The implementation steps include real-time traffic analysis, feature extraction, and behavioral pattern learning.

[0047] Threat intelligence data analysis solution: This technical solution is based on using threat intelligence data collected in real time to guide and optimize network traffic monitoring and analysis to improve the ability to detect and respond to potential threats. The implementation principle is to obtain real-time threat intelligence data from various security sources, including malicious IP addresses, malicious domain names, malicious file hashes, etc. The collected threat intelligence data is integrated and correlated with the real-time traffic analysis results to identify potential security threat behaviors. The implementation steps include collecting threat intelligence data, threat intelligence driving engine, and integrating threat intelligence with real-time traffic correlation analysis.

[0048] The above prior art mainly has the following problems:

[0049] The rule analysis engine technology has a large number of static rules, a large maintenance workload, and limited ability to identify new threats. It is vulnerable to evasion attacks and has difficulty adapting to complex and changing network environments.

[0050] Statistical and behavioral analysis techniques are highly dependent on the definition of abnormal behavior, making it difficult to accurately identify complex network attacks and are easily interfered by noisy data.

[0051] Threat intelligence analysis technology is built based on information such as known threat behaviors and malicious IP addresses, so it has limited coverage and is highly dependent on third-party data sources. It also has problems such as delayed updates and high false alarm rates.

[0052] Therefore, the above technical solutions may be able to play an effective role in certain scenarios, but they are unable to adapt to complex attacks, have difficulty in accurately identifying new threats, and have a high dependence on static content such as rules, statistical features, and attack signatures.

[0053] To facilitate understanding of this embodiment, firstly, a network traffic detection method based on AI disclosed in an embodiment of the present invention is described in detail. The method is applied to a terminal device, such as Figure 1As shown, the method comprises the following steps:

[0054] Step S102, obtaining target network traffic data corresponding to the original network traffic; wherein the target network traffic data includes a classification result carrying traffic feature data;

[0055] The above-mentioned original network traffic can be understood as the mirror port traffic, splitter traffic or splitter traffic received through the system's monitoring port, wherein the basic data unit transmitted by the original network traffic is a data packet, that is, the original network traffic is actually data packet data.

[0056] The above-mentioned target network traffic data can be understood as being obtained by collecting and analyzing data packets of the original network traffic, then cleaning the analyzed traffic data, and then classifying and processing the cleaned traffic data according to different dimensions.

[0057] In fact, the target network traffic data obtained after classifying the cleaned traffic data according to different dimensions can be understood as the classification result carrying traffic feature data, or it can be understood that the target network traffic data is the traffic feature data with the classification result.

[0058] In the specific implementation process, the system uses the bypass deployment mode to monitor and receive network traffic from the mirror port, splitter or splitter, and implements high-speed data packet collection and analysis through DPDK (Data Plane Development Kit). Next, the data processing module cleans the traffic, and then classifies the cleaned traffic data according to different dimensions to obtain classification results carrying traffic feature data.

[0059] Step S104, input the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; the detection model includes: a threat detection model based on machine learning and a behavior detection model based on deep learning.

[0060] The above detection model can actually be trained based on a pre-acquired target network traffic data set, where the target network traffic data set includes sample target network traffic data annotated with detection result labels.

[0061] In the specific implementation process, sample target network traffic data (i.e., sample classification results carrying sample traffic feature data) can be obtained based on historical detection data collection, and then the sample target network traffic data can be labeled. The result label can be labeled, and other labels can be labeled at the same time to obtain the target network traffic data set training set (i.e., the training sample set of the detection model), and then the target network traffic data set is used to train the initial model to obtain the detection model. In actual implementation, the detection model can extract the detection features (including traffic features and classification features) of the input target network traffic data, and then obtain the output results (i.e., the detection results) based on the detection features.

[0062] In fact, the model training and analysis capabilities provided by the AI ​​engine (including training using machine learning and deep learning algorithms) can be used to train the model. At the same time, the trained model can be used to conduct comprehensive analysis on real-time traffic data to identify network threats and abnormal behaviors in network traffic, trigger corresponding early warning mechanisms and alarm responses, and present them visually in the form of threat alarms.

[0063] In recent years, with the rapid development of machine learning technology, its advantages in classification and prediction have provided effective solutions to problems that are difficult or inapplicable to traditional methods. This ability can significantly improve the accuracy and speed of network traffic detection, so the use of machine learning technology for network traffic detection has become a research hotspot in the field of network security. Machine learning methods are based on statistical principles. In network traffic detection, it is necessary to extract the statistical characteristics of traffic based on expert knowledge / experience, and then use machine learning algorithms to perform classification detection of traffic.

[0064] The main machine learning algorithms include supervised learning and unsupervised learning. Supervised learning algorithms used for network traffic detection mainly include linear regression, logistic regression, K-nearest neighbor (KNN), decision tree (DT), support vector machine (SVM), random forest (RF), etc.; while unsupervised learning algorithms include K-means clustering, PCA principal component analysis, automatic classification algorithm, etc. These algorithms have achieved remarkable results in network traffic detection and analysis. Although machine learning has solved some of the problems of traditional network traffic detection methods, it still has certain limitations in automatic feature extraction due to its over-reliance on expert knowledge / experience. In contrast, deep learning has adaptive and self-learning capabilities, which can effectively solve this problem.

[0065] As deep learning shines in the fields of image recognition, speech recognition, and natural language processing, network security experts continue to use deep learning to solve network security problems. Deep learning can be used to solve the above-mentioned network traffic detection problems. Compared with machine learning, it can automatically extract and select features through training. This feature makes deep learning-based methods a very ideal method for network traffic detection. Another feature of deep learning methods is that compared with traditional machine learning methods, deep learning methods have more powerful learning capabilities. This allows the nonlinear relationship from raw data input to output to be automatically learned when extracting features from network traffic, so highly complex patterns can be learned.

[0066] The methods of using deep learning models for network traffic detection mainly include: deep belief network (DBN), convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), long short-term memory network (LSTM) and hybrid methods.

[0067] In the specific implementation process, after the training of the threat detection model based on machine learning and the behavior detection model based on deep learning is completed, all the currently acquired target network traffic data can be simultaneously input into the above-mentioned threat detection model and behavior detection model. The threat detection model is used to obtain the first detection result (that is, whether there is a threat) and the first type of result (that is, the specific threat type) corresponding to the target network traffic data. The behavior detection model is used to obtain the second detection result (that is, whether there is abnormal behavior) and the second type of result (that is, the specific behavior type) corresponding to the target network traffic data.

[0068] This application proposal uses the machine learning and deep learning technologies in the AI ​​engine to model network traffic (for example, threat detection models can be modeled by using supervised learning algorithms, and behavior detection models can be modeled by using deep learning algorithms, etc.), and security threat detection and abnormal behavior detection of network traffic are performed based on the modeling results, thereby improving detection accuracy and efficiency, and effectively processing large-scale and complex network traffic data, thereby having higher advantages and reliability in full network traffic detection.

[0069] Step S106, based on the detection result, determining whether the original network traffic is abnormal network traffic;

[0070] Step S108, if the original network traffic is abnormal network traffic, generate and display alarm information corresponding to the original network traffic;

[0071] Specifically, the detection result label may include a normal label and an abnormal label, so the detection result may correspond to abnormal or normal. If the detection result is abnormal, the original network traffic is considered to be abnormal network traffic. Generally, if the detection result is normal, that is, the original network traffic is abnormal network traffic, an alarm message may be generated and displayed. In addition, when labeling the sample target network traffic data, the result label and the type label may also be labeled at the same time. Then, the target network traffic data is input into a pre-trained detection model to obtain the detection result and type result corresponding to the target network traffic data. According to the detection result and the type result, it is determined whether to generate and display the alarm message corresponding to the original network traffic. In fact, when the detection result is abnormal, the danger level of some abnormal types may not be very high, so there is no need to generate an alarm message. However, when the danger level of the abnormal type is originally very high, an alarm message needs to be generated. When the detection result is normal, the type result does not need to be considered, and no alarm message needs to be generated.

[0072] This solution uses the model training and analysis capabilities provided by the AI ​​engine to identify abnormal traffic in network traffic (generally including the detection of network threats and abnormal behavior in network traffic). Ultimately, the corresponding early warning mechanism and alarm response are triggered, and visualized in the form of threat alerts. So that relevant personnel can understand the network security situation in a timely manner and take corresponding measures. It aims to improve the accuracy, real-time and adaptability of the network full-flow monitoring and auditing system by introducing AI analysis engine technology, and be able to self-intelligently identify network threats and abnormal behaviors in the network. Improve the efficiency and accuracy of network security event detection, effectively process large-scale and complex network traffic data, and achieve more efficient network security protection, thereby having higher advantages and reliability in network full-flow detection.

[0073] The AI-based network traffic detection method provided in the above embodiment includes: obtaining target network traffic data corresponding to the original network traffic; the target network traffic data includes classification results carrying traffic feature data; inputting the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; based on the detection result, determining whether the original network traffic is abnormal network traffic; if it is abnormal network traffic, generating and displaying alarm information corresponding to the original network traffic; the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with a detection result label. In this solution, by performing abnormal network traffic detection based on the detection model trained based on the target network traffic data set, abnormal traffic in the detection network can be intelligently identified and detected, thereby improving detection accuracy and efficiency.

[0074] The embodiment of the present invention also provides another AI-based network traffic detection method, which is implemented on the basis of the above-mentioned embodiment method; Figure 2 As shown, the method comprises the following steps:

[0075] Step S202, monitoring and receiving original network traffic from the mirror port, splitter or optical splitter through the monitoring port;

[0076] Step S204, preprocessing the original network traffic by a preset processing tool to obtain first network traffic data; wherein the first network traffic data includes: parsing data, session statistics data, and data packet data;

[0077] The above preset processing tool can use DPDK, which is mainly based on the Linux system and is a collection of function libraries and drivers for fast data packet processing. It can greatly improve data processing performance and throughput, and improve the work efficiency of data plane applications. DPDK uses polling instead of interrupts to process data packets. When receiving a data packet, the network card driver overloaded by DPDK will not notify the CPU through an interrupt, but directly store the data packet in the memory and deliver it to the application layer software for direct processing through the interface provided by DPDK, which saves a lot of CPU interrupt time and memory copy time.

[0078] In the specific implementation process, the business working principle is to receive the mirror port, splitter or splitter traffic through the system's monitoring port (the system uses the bypass deployment mode to monitor and receive network traffic from the mirror port, splitter or splitter), and use DPDK to implement high-speed data packet collection and parsing processing (that is, pre-process the original network traffic through preset processing tools) to obtain parsed data, session statistics, and data packet data.

[0079] Specifically, the system uses a new generation of DPDK high-performance full-flow collection and analysis processing engine, based on zero copy, large page memory, parallel analysis, lock-free ring cache, and LUA detection script technology to achieve fast capture of real-time network traffic and data packet analysis. It also supports multi-core parallel computing and efficient processing of a large number of data packets, and supports network traffic up to 10 Gigabit bandwidth, achieving lower latency and higher throughput. Collection and analysis include the identification and restoration analysis of protocols, the identification and restoration analysis of files, and generally supports accurate identification of various protocols at the L2-L7 layer in network traffic, deep packet identification and analysis and splitting of the seventh layer protocol, such as HTTP, FTP, IMAP, POP3, SMTP, SMB, DNS and other dozens of protocols. It also has the ability to restore common file types and conduct retroactive forensic analysis of malicious files.

[0080] The process of traffic collection is mainly that DPDK directly receives and sends network data packets in user space by binding and configuring the network card monitoring port. The received network data packets are stored in the DPDK ring buffer and copied directly from the monitoring port to the user space, avoiding the duplication and copying of data packets between kernel space and user space. DPDK provides an efficient processing interface for network packets, which can realize fast data packet processing and parsing.

[0081] The traffic analysis process of DPDK includes protocol decoding, flow table tracking, session tracking, data reassembly, application identification and application analysis. DPDK extracts the value and identification of each field by parsing the header information of the data packet, including the source IP address, destination IP address, protocol type, port number and other information. According to the information such as protocol type and port number (equivalent to parsing data), the data packet is deeply analyzed, such as HTTP message analysis, DNS message analysis, etc. It can also count the traffic volume (equivalent to session statistics), the number and size of data packets (equivalent to data packet data) in real time for network performance analysis and monitoring.

[0082] Step S206: The first network traffic data is stored by a preset analysis component, and the stored first network traffic data is preprocessed by a preset processing method to obtain target network traffic data.

[0083] The above-mentioned preset analysis components may include databases such as GAUSSDB and ES, and may also include file storage repositories and Kafka software.

[0084] In fact, the data collected and analyzed by DPDK can be classified and stored according to its own business. Specifically, the system can store and cache the first network traffic data obtained after analysis by providing big data analysis components such as Kafka and ES. That is, the data storage capability is reflected in the recording and archiving of network traffic data (also known as parsed data), network session data (also known as session statistics data), Pcap data (also known as data packet data), etc., providing basic raw data for the AI ​​analysis engine for further data processing, modeling, and analysis.

[0085] The network traffic data recorded above can actually be understood as the original data packet data generated by parsing, including all data packet information transmitted in the network, such as source IP address, destination IP address, port number, protocol type, etc. This data can be used for further in-depth parsing and analysis.

[0086] The network session data recorded above can be understood as session statistics about network traffic generated by analysis, including indicators such as traffic volume, traffic rate, packet loss rate, etc. These statistics are very useful for monitoring network performance and detecting anomalies.

[0087] The Pcap data recorded above can be understood as the Pcap packet data generated by parsing, including information such as timestamp, five-tuple, packet length, packet type, etc. These Pcap files can be used to help the system perform threat correlation analysis and source tracing forensic analysis on the data packets.

[0088] In actual implementation, the above preset processing method may include a first preset processing method and a second preset processing method; the step of preprocessing the stored first network flow data by the preset processing method to obtain the target network flow data may be implemented by the following steps 1 to 2:

[0089] Step 1: Performing a first processing on the stored first network traffic data by a first preset processing method to obtain second network traffic data; wherein the first preset processing method includes traffic feature extraction processing and at least one of the following: cleaning processing, denoising processing, and formatting processing;

[0090] Step 2: Perform a second processing on the second network flow data through a second preset processing method to obtain target network flow data.

[0091] The stored first network flow data is preprocessed in a preset processing manner to obtain target network flow data.

[0092] In fact, the data stored in the preset analysis component can be subjected to a first processing and a second processing according to a first preset processing method and a second preset processing method.

[0093] In actual implementation, the system can perform pre-processing operations such as cleaning, denoising, and formatting on the parsed and stored data based on the data processing module (equivalent to the first processing), and extract useful information and features to help users better monitor network operation status, identify potential problems and threats, thereby achieving effective network security management and optimization.

[0094] Specifically, data cleaning refers to preprocessing data to remove incomplete, duplicate, erroneous or invalid data to ensure the accuracy and integrity of the data. The data processing module cleans traffic data through data quality checks, outlier detection, missing value processing and other methods to ensure data quality.

[0095] Data denoising refers to processing the noise and interference in the data to improve the quality and reliability of the data. The data processing module uses denoising algorithms such as filtering and smoothing to remove noise or outliers in the data, making the data more analyzable.

[0096] Data formatting is to organize and convert data into a specified format for subsequent processing and analysis. The data processing module formats the data according to the structure and content of the data, such as converting data types and standardizing data formats, in order to facilitate more advanced data analysis.

[0097] Data feature extraction refers to extracting useful feature information from data for subsequent data analysis and modeling. The data processing module can convert raw data into feature vectors through various existing feature extraction methods, and extract key feature information from traffic data, such as protocol type, port information, packet size, transmission rate, etc., so as to conduct in-depth data analysis and mining in the future to accurately identify security threats and abnormal behaviors in network traffic.

[0098] It should be noted that when cleaning, denoising, formatting, and feature extraction all need to be performed, the parsed and stored data is usually cleaned first, then the cleaned data is denoised, then the denoised data is formatted, and finally the formatted data is feature extracted to obtain the second network traffic data, which includes traffic feature data.

[0099] In actual implementation, the second preset processing method includes classification processing, and the above step 2 can be specifically implemented by the following steps A:

[0100] Step A: Input the second network traffic data into a pre-trained classification model to obtain a classification result corresponding to the second network traffic data and carrying traffic feature data; wherein the second network traffic data includes traffic feature data.

[0101] In actual implementation, the classification model includes classification sub-models corresponding to multiple classification dimensions, and the multiple classification dimensions include: plaintext classification, protocol classification, application classification, and behavior classification; the above step A can be specifically implemented by the following steps a to b:

[0102] In actual implementation, classification processing can be achieved through the data classification module, which is a key component of the system and its main function is to identify and classify traffic in the network. The module has built-in multiple classification technologies, including protocol port-based identification, deep packet inspection (DPI), and statistical feature analysis of data packets (that is, the system can classify the second network traffic data through the classification model of the AI ​​engine, and the AI ​​engine has stronger feature learning ability and classification generalization ability, and the classification accuracy and recall rate are higher), ensuring high accuracy and high recall rate, making the network traffic classification capability more simplified, automated, and intelligent.

[0103] The following will elaborate on the four dimensions of network traffic: plaintext and ciphertext classification, protocol classification, application classification, and behavior classification.

[0104] Plaintext and ciphertext classification: refers to the classification of network traffic at the protocol encryption attribute level, which can distinguish network traffic into plaintext traffic and encrypted traffic. Usually, the traffic is judged as encrypted traffic by detecting specific patterns in the traffic, port usage or statistical characteristics of the traffic. It is found in practice that the degree of confusion of encrypted traffic is slightly greater than that of plaintext traffic, and the degree of confusion can be measured by information. Therefore, after the plaintext and ciphertext classification feature model based on the information entropy vector (that is, the plaintext and ciphertext classification sub-model) is constructed, random forest is used as a classifier to classify plaintext traffic and encrypted traffic. Since random forest is a classic machine learning method, the specific calculation process will not be repeated.

[0105] Protocol classification: refers to the classification of network traffic at the network protocol level. Among them, the application layer protocol classification technology has a wide range of application scenarios, such as based on transport layer port information, application layer message load information, etc. By analyzing the header information of the data packet, such as port number, protocol type field, payload content, etc., the protocol used by the traffic is determined. For example, HTTP traffic usually uses port 80 or 443. In addition, the n-gram elements in the application layer protocol traffic obey a long-tail distribution (Zipf distribution), so the application layer protocol classification method based on the non-parametric model can automatically construct a robust protocol classification model, and can effectively and accurately classify the application layer protocol traffic from the mixed network traffic.

[0106] Application classification: refers to the classification of network traffic at the network application level, that is, identifying which application generates the network traffic. For example, web browsing, instant messaging, email transmission, etc., in addition to using protocol and port information, specific applications can also be identified through deep packet inspection (DPI) or algorithm technology. For example, different applications may have characteristics in data packet size and transmission mode.

[0107] Behavior classification: refers to the classification of network traffic at the network behavior level. Compared with application classification, behavior classification can better distinguish the behaviors of different network users and their habits and patterns, such as normal use, abnormal behavior or potential attack behavior, etc. By building a behavior analysis method, the pattern, frequency and degree of abnormality of the traffic are analyzed. For example, abnormal traffic may appear as a sudden surge in traffic or an abnormal access pattern.

[0108] Step a: input the second network traffic data into each pre-trained classification sub-model respectively, and obtain the classification sub-result carrying the corresponding traffic sub-feature data output by each classification sub-model; wherein each classification sub-model is trained based on the corresponding traffic sub-feature data set acquired in advance, and each traffic sub-feature data set includes corresponding sample traffic sub-feature data marked with the classification sub-result label;.

[0109] The above classification sub-models include plaintext and ciphertext classification sub-model, protocol classification sub-model, application classification sub-model and behavior classification sub-model.

[0110] In fact, the second network traffic data obtained after the first processing can be classified according to different dimensions (that is, the second network traffic data is input into each pre-trained classification sub-model respectively), including plaintext and ciphertext classification, protocol classification, application classification, behavior classification, etc.

[0111] In the specific implementation process, the sample traffic sub-feature data corresponding to each classification sub-model can be obtained based on the historical classification data collection, and then the sample traffic sub-feature data can be labeled, and the classification sub-result label can be labeled to obtain the traffic sub-feature data set (that is, the training sample set corresponding to the classification sub-model), and then the traffic sub-feature data set corresponding to each classification sub-model is used to train the corresponding initial classification sub-model to obtain the corresponding classification sub-model. In actual implementation, each classification sub-model can first select the traffic sub-feature data related to the classification of the classification sub-model from the input second network traffic data, and then extract the data features of the traffic sub-feature data, and obtain the output result (that is, the classification sub-result) based on the data features.

[0112] In fact, the classification sub-result label annotated with the sample traffic sub-feature data corresponding to the plaintext and ciphertext classification model may include a plaintext label and a ciphertext label. Therefore, the classification sub-result output by the plaintext and ciphertext classification model may correspond to plaintext or ciphertext. Regardless of whether the output classification sub-result is plaintext or ciphertext, it carries the traffic sub-feature data selected from the input second network traffic data when the plaintext and ciphertext classification model outputs the classification sub-result.

[0113] The classification sub-result labels annotated with the sample traffic sub-feature data corresponding to the protocol classification sub-model can be various protocol types, the classification sub-result labels annotated with the sample traffic sub-feature data corresponding to the application classification sub-model can be various application types, and the classification sub-result labels annotated with the sample traffic sub-feature data corresponding to the behavior classification sub-model can be various behavior types; the classification sub-results output by the protocol classification sub-model, application classification sub-model and behavior classification sub-model are the same as those of the plaintext and ciphertext classification model, and will not be described in detail.

[0114] Step b: Determine each classification sub-result as a classification result carrying flow characteristic data; wherein the flow characteristic data includes flow sub-characteristic data corresponding to each classification sub-model.

[0115] In the specific implementation process, the classification sub-results output by each classification sub-model can be combined together as the target network traffic data, that is, the classification results carrying traffic feature data.

[0116] Step S208, input the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data.

[0117] In actual implementation, the detection model includes a behavior detection model and a threat detection model; wherein the threat detection model is obtained by learning a first target network traffic data set pre-acquired based on a machine learning method, and the first target network traffic data set includes a first sample target network traffic data labeled with a first detection result label; the first detection result label includes: a threat label or a no-threat label, and / or a threat type; the behavior detection model is obtained by learning a second target network traffic data set pre-acquired based on a deep learning method, and the second target network traffic data set includes a second sample target network traffic data labeled with a second detection result label; the second detection result label includes: a normal behavior label or an abnormal behavior label, and / or a behavior type.

[0118] In the specific implementation process, the above behavior detection model is trained through the following steps 3 to 4:

[0119] Step 3: inputting the first sample target network traffic data into the first initial model to extract the detection features of the first sample target network traffic data through the first initial model, so as to obtain a first output result based on the detection features;

[0120] Step 4: Based on the first output result and the first detection result label, update the weight parameter of the first initial model; continue to perform the step of inputting the first sample target network traffic data into the first initial model until the first initial model converges to obtain a threat detection model;

[0121] In the specific implementation process, the first sample target network traffic data (that is, the first sample classification result carrying the first sample traffic feature data) can be obtained according to the historical detection data collection, and then the first sample target network traffic data can be labeled. The first detection result label can be labeled to obtain the first target network traffic data set training set (that is, the training sample set of the threat detection model), and then the first target network traffic data set is used to train the first initial model to obtain the threat detection model.

[0122] Specifically, the first detection result label may include a threat label (equivalent to an abnormal label) and a non-threat label (equivalent to a normal label), so the first output result may correspond to abnormal or normal. In addition, when labeling the first sample target network traffic data, the first type label (that is, what type of threat) may also be labeled at the same time, so the first output result also includes a specific threat type.

[0123] In fact, the loss function can be used to calculate the loss between the first output result and the first detection result label. If the loss exceeds the preset threshold, it means that the model has not converged. The weight parameters of the first initial model can be backpropagated and updated until the first initial model converges. After that, the model performance can be evaluated (accuracy, precision, recall rate and F1 score are calculated). If the model performance meets the preset requirements, a trained threat detection model is obtained.

[0124] When the currently acquired target network traffic data is input into the above-mentioned trained threat detection model, the first detection result (that is, whether there is a threat) and the first type result (that is, the specific threat type) corresponding to the target network traffic data can be obtained. According to the first detection result and the first type result, that is, according to the first output result, not only can it be confirmed whether there is a threat in the original network traffic, but also the specific threat type when there is a threat can be confirmed, and a judgment can be made whether to generate and display the alarm information and alarm type corresponding to the original network traffic (different threat types can correspond to different alarm types).

[0125] In the specific implementation process, the above threat detection model is trained through the following steps five to six:

[0126] Step 5: inputting the second sample target network traffic data into the second initial model to extract detection features of the second sample target network traffic data through the second initial model, so as to obtain a second output result based on the detection features;

[0127] Step six: Based on the second output result and the second detection result label, update the weight parameters of the second initial model; continue to execute the step of inputting the second sample target network traffic data into the second initial model until the second initial model converges to obtain the behavior detection model.

[0128] In the specific implementation process, the second sample target network traffic data (that is, the second sample classification result carrying the second sample traffic feature data) can be obtained according to the historical detection data collection, and then the second sample target network traffic data can be labeled. The second detection result label can be labeled to obtain the second target network traffic data set training set (that is, the training sample set of the behavior detection model), and then the second initial model is trained using the second target network traffic data set to obtain the behavior detection model.

[0129] Specifically, the second detection result label may include a normal behavior label (equivalent to a normal label) and an abnormal behavior label (equivalent to an abnormal label), so the second output result may correspond to abnormal or normal. In addition, when labeling the second sample target network traffic data, the second type label (that is, what type of behavior) may also be labeled at the same time, so the second output result also includes a specific behavior type.

[0130] In fact, the loss function can be used to calculate the loss between the second output result and the second detection result label. If the loss exceeds the preset threshold, it means that the model has not converged. The weight parameters of the second initial model can be backpropagated and updated until the second initial model converges. After that, the model performance can be evaluated (accuracy, precision, recall rate and F1 score are calculated). If the model performance meets the preset requirements, a trained behavior detection model is obtained.

[0131] In specific implementation, the currently acquired target network traffic data can also be input into the above-mentioned trained behavior detection model to obtain a second detection result (that is, whether there is abnormal behavior) and a second type result (that is, the specific behavior type) corresponding to the target network traffic data. According to the second detection result and the second type result, not only can it be confirmed whether there is abnormal behavior in the original network traffic, but also the specific type of abnormal behavior can be confirmed, and then it can be determined whether to generate and display the alarm information and alarm type corresponding to the original network traffic (different abnormal behavior types can correspond to different alarm types).

[0132] Step S210, based on the detection result, determining whether the original network traffic is abnormal network traffic;

[0133] In actual implementation, when the detection model includes a threat detection model and a behavior detection model, it is possible to determine whether the original network traffic has a security threat based on the first detection result corresponding to the threat detection model, and to determine whether the original network traffic has abnormal behavior based on the second detection result corresponding to the behavior detection model.

[0134] Step S112: if the original network traffic is abnormal network traffic, generate and display alarm information corresponding to the original network traffic.

[0135] In actual implementation, if there is a security threat in the original network traffic, corresponding alarm information is generated according to the threat type; if there is abnormal behavior in the original network traffic, corresponding alarm information is generated according to the abnormal behavior type.

[0136] In order to better understand the above embodiment, the following describes the detection of the threat detection model by taking the application scenario of malicious encrypted traffic monitoring based on machine learning as an example:

[0137] (1) Input HTTP / HTTPS traffic metadata, including various information of HTTP / HTTPS requests and responses, such as request headers, response headers, URLs, cookies, payload content, etc.

[0138] (2) Extract characteristic information of input data, including packet size and distribution, key protocol fields (such as TLS version, client-supported cipher suites, DNS resolution records), and various statistical features.

[0139] (3) The model uses the feature information of the input data to perform identification and prediction. The model calculates the probability that the input feature belongs to malicious traffic and determines the detection result of the traffic (normal or malicious) based on the set threshold.

[0140] A single model is prone to overfitting problems. You can choose an integrated model for training and testing. The algorithms include support vector machine (SVM), random forest, and gradient boosting tree (XGBoost). Support vector machine (SVM) can distinguish between normal and malicious traffic by building a hyperplane. Random forest can make the final classification judgment of normal and malicious traffic by building an integration of multiple decision trees. Gradient boosting tree (XGBoost) can coordinate multiple weak learners through the boosting method to improve prediction accuracy and optimize the prediction ability of the model.

[0141] (4) The output test results are stored in the system and displayed in a visual manner for subsequent audit and analysis.

[0142] This application scenario demonstrates how to use machine learning models to effectively detect and identify malicious HTTP / HTTPS traffic without decrypting the traffic. Through precise data processing and feature extraction, combined with appropriate machine learning algorithm models, the system can achieve efficient traffic monitoring and auditing, effectively reduce the false alarm rate, and ultimately improve network security protection capabilities.

[0143] In order to better understand the above embodiment, the following describes the detection of the behavior model by taking the phishing / fraud email monitoring application scenario based on deep learning as an example:

[0144] (1) Input email traffic metadata, including email body, email subject, attachment information, sender and recipient information, etc.

[0145] (2) Process the email text, including content normalization; segment the email body and extract keywords through natural language processing tools, that is, extract feature information of the input email traffic metadata.

[0146] (3) Identify phishing / fraudulent emails by inputting the features of email traffic metadata. The main algorithm is as follows:

[0147] DOC2VEC: It is used to convert the processed email text into vector form to capture the semantic information of the email content.

[0148] CNN: Extract important features from email text through convolutional layers for pattern recognition.

[0149] LSTM: Used to process sequence data in email text, helping to capture contextual information and long-term dependencies in emails, and enhancing the ability to identify phishing email patterns.

[0150] (4) The identification results (test results) are stored in the system and displayed in a visual manner for subsequent audit and analysis.

[0151] Phishing emails are fraudulent emails that usually disguise themselves as legitimate emails to trick users into clicking on malicious links or attachments, thereby stealing their personal and financial information. In the process of identifying and detecting phishing / fraudulent emails, deep learning algorithms such as DOC2VEC, CNN, and LSTM can effectively improve the ability to detect phishing emails. This complete modeling process not only enhances recognition accuracy, but also enables timely response to potential security threats.

[0152] The AI-based network traffic detection method provided in the above embodiment, the AI ​​engine uses its advanced model training capabilities to deeply analyze and process network traffic data, providing strong technical support for network threat detection and abnormal behavior identification. In the data acquisition phase of the system, the traffic data in the network is captured efficiently and in real time. Subsequently, in the data processing phase, data cleaning and preprocessing techniques are used to ensure the quality and consistency of the data. In the data classification phase, the system uses a refined classification method to divide the traffic data into different categories, and extracts key feature information, which is an important input for model training. Based on these processed traffic data and feature data, the system can select a suitable machine learning or deep learning model according to specific business needs to achieve in-depth mining and precise analysis of specific application scenarios.

[0153] For a better understanding of the above embodiments, please refer to Figure 3 The AI-based network traffic detection system shown in the figure consists of six parts: data acquisition module, data storage module, data processing module, data classification module, AI engine module and data visualization module.

[0154] RSS (Receive Side Scaling) is a network card diversion mechanism to achieve network traffic diversion to improve network performance. The system uses bypass deployment mode to monitor and receive network traffic from mirror ports, diverters or splitters, and implements high-speed packet collection and analysis through DPDK. Next, the data processing module cleans the traffic, and then classifies the cleaned traffic data according to different dimensions, including plaintext classification, protocol classification, application classification, and behavior classification. The model training and analysis capabilities provided by the AI ​​engine are then used to identify network threats and abnormal behaviors in network traffic, and ultimately trigger the corresponding early warning mechanism and alarm response, and visualize them in the form of threat alerts so that relevant personnel can understand the network security situation in a timely manner and take corresponding measures.

[0155] Through the visual analysis module, the system can present the result data of AI engine monitoring and audit analysis in a visual way, including charts, reports, etc., and also provide AI analysis engine configuration management capabilities, which helps enterprise security teams better understand network threats and security status, as well as conveniently operate AI engine-related components to make corresponding reasonable decisions and measures.

[0156] The visual analysis module includes workbench, threat alert, source tracing and forensics, full-text retrieval model management, and system management.

[0157] The workbench supports statistical analysis and display of threat alerts, traffic collection, and AI model applications in multiple dimensions; and has charts and displays of threat monitoring type proportions, alarm level proportions, real-time traffic trend analysis, etc.

[0158] Threat alarm provides threat alarm capability, which can be filtered and queried according to multiple dimensions such as alarm name, alarm level, time, etc.; supports overall threat alarm statistics, threat alarm type ranking, threat alarm trend analysis, alarm list display, etc.; supports threat alarm detail analysis, which can display network quintuple, threat alarm basic information and payload content, etc.; and automatically associates the corresponding original Pcap data packet, with accurate analysis and traceability capabilities;

[0159] Tracing and forensics supports the extraction of massive historical traffic in seconds, supports full-packet tracing and forensics of network traffic data, session trend display, session list display, and supports the download of related original pcap data packets, etc. It helps enterprises quickly trace and locate problems and improve risk management efficiency.

[0160] Full-text search supports rapid retrieval of massive amounts of traffic metadata. It can search and analyze historical data by time, source IP, target IP, protocol type, etc., and quickly search for target information, which can help security teams locate and resolve security issues faster and more accurately.

[0161] Model management supports unified management of machine learning models, deep learning models, etc. As the central storage warehouse for models in the entire system, model management can import models generated by all training tasks or local models and manage them in a unified manner. With visual modeling capabilities and a visual operating environment, users can complete the complete link of building data intelligence applications including data sources, data preprocessing, feature engineering, statistical analysis, model training, prediction, evaluation, and services in one stop, helping developers achieve zero-code machine learning development.

[0162] System management provides maintenance and configuration management functions for system accounts, organizational structures, user roles, and security policies, provides various basic information configuration functions for the system, system menu customization functions, system network port configuration and maintenance functions, and data forwarding and other functional configuration management. It provides system deployment environment status monitoring and generation of monitoring alarm functions. The monitoring content includes: host monitoring, service monitoring, middleware monitoring, database monitoring, etc. It provides real-time collection of various system logs, including operation logs, login logs, etc., and performs centralized storage and query retrieval so that administrators can quickly locate and solve system failure problems. It provides historical data log cleaning of the system, data backup and recovery, component upgrade and update, etc.

[0163] The purpose of this application proposal is to combine artificial intelligence technology to improve the intelligence and adaptability of full-flow network monitoring and auditing, to have better response capabilities when facing complex network security threats, and to improve the accuracy and efficiency of network security incident detection.

[0164] The solution uses an AI analysis engine, including machine learning and deep learning technologies, to provide powerful analysis capabilities and intelligent processing for network traffic monitoring and auditing, and has a complete business process, from traffic data collection to visual presentation, to ensure coverage and support for the entire process of network security monitoring and auditing. At the same time, it uses processing algorithms and classification methods that may include special processing algorithms and classification methods for network traffic data to identify and analyze various types of traffic. In addition, by training AI models and performing real-time analysis, the performance and adaptability of monitoring are continuously optimized. By setting up a user interface and data visualization tool to present the analysis results, users can intuitively understand the network traffic monitoring and auditing results.

[0165] This application proposal uses artificial intelligence technology, machine learning and deep learning methods to achieve dynamic monitoring and audit analysis of the entire network flow, with higher accuracy, real-time and intelligence, and can effectively identify and respond to network security risks. Compared with existing solutions based on feature rules and behavior analysis, the network full-flow monitoring and auditing system based on the AI ​​engine has the following technical advantages:

[0166] Automated learning capabilities: The AI ​​engine can automatically learn and identify patterns and abnormal behaviors in network traffic through machine learning and deep learning technologies, without relying on manually defined rules or features, thereby improving detection accuracy and adaptability.

[0167] Big data processing capabilities: The AI ​​engine is able to process large-scale network traffic data, and through big data analysis and deep learning technology, it can mine the information and patterns hidden in massive data to achieve comprehensive and detailed monitoring and auditing of all traffic.

[0168] Real-time monitoring and analysis: The AI ​​engine has the ability to process and analyze network traffic data in real time, which can quickly respond to network security incidents, promptly detect and respond to potential threats, and reduce the risk of network attacks.

[0169] Intelligent decision support: The AI ​​engine can make intelligent decisions and suggestions based on learned knowledge and patterns, assisting administrators to better manage and maintain network security, and improving the efficiency and accuracy of auditing and monitoring.

[0170] Ability to face unknown threats: Compared with solutions based on known rules and features, AI engines are more capable of discovering unknown threats and new attacks. They can continuously learn and adapt to changes in the network security environment and improve network security protection capabilities.

[0171] To sum up, the network full-flow monitoring and auditing system based on the AI ​​engine has higher automation, intelligence and adaptability, and can monitor and audit network traffic more comprehensively and effectively, thereby improving the level of network security protection.

[0172] In conjunction with this application proposal, the following are some divergent thinking and avoidance solutions:

[0173] Data privacy protection: When processing network traffic data, it is necessary to ensure that the user's privacy is not violated. User data privacy can be protected through technical means such as data desensitization and encryption, while complying with relevant laws, regulations and privacy policies.

[0174] Model explainability: The explainability of AI models is crucial for monitoring and auditing systems, as regulators and users need to understand the decision-making process of the model. Therefore, explainable AI technology can be explored to ensure that the model decision-making process is transparent and explainable.

[0175] Continuous learning and updating: The cybersecurity environment is constantly changing, so it is necessary to ensure that the AI ​​engine system can continue to learn and update, adapt to new cybersecurity threats and attack methods in a timely manner, and maintain the effectiveness and agility of the system.

[0176] Human-machine cooperation mechanism: Although AI engines can automatically process network traffic data, the experience and wisdom of human experts are still indispensable. A human-machine cooperation mechanism can be established to achieve efficient collaboration between AI and human experts and improve the monitoring and auditing effects of the system.

[0177] By comprehensively considering the above divergent thinking and avoidance solutions, the network full-flow monitoring and auditing system based on the AI ​​engine can be further improved to improve the security, reliability and practicality of the system.

[0178] The commercial value of this application proposal lies in providing an intelligent and efficient network security monitoring and auditing solution, which can be widely used in the network security protection field of enterprises, governments and other organizations to provide users with a safer network environment. It has high commercial value and market demand.

[0179] The embodiment of the present invention also provides a network traffic detection device based on AI, such as Figure 4 As shown, the device includes: an acquisition module 40, which is used to obtain target network traffic data corresponding to the original network traffic; wherein the target network traffic data includes a classification result carrying traffic feature data; an input module 41, which is used to input the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; a judgment module 42, which is used to judge whether the original network traffic is abnormal network traffic based on the detection result; an alarm module 43, which is used to generate and display an alarm information corresponding to the original network traffic if the original network traffic is abnormal network traffic; wherein the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with a detection result label.

[0180] The present invention provides an AI-based network traffic detection device, comprising: obtaining target network traffic data corresponding to the original network traffic; the target network traffic data includes a classification result carrying traffic feature data; inputting the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; judging whether the original network traffic is abnormal network traffic based on the detection result; if it is abnormal network traffic, generating and displaying alarm information corresponding to the original network traffic; the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with a detection result label. The device can intelligently identify abnormal traffic in the detection network and improve detection accuracy and efficiency by performing abnormal network traffic detection based on the detection model trained based on the target network traffic data set.

[0181] Furthermore, the detection model includes a behavior detection model and a threat detection model; wherein the threat detection model is obtained by learning a first target network traffic data set pre-acquired based on a machine learning method, and the first target network traffic data set includes first sample target network traffic data labeled with a first detection result label; the behavior detection model is obtained by learning a second target network traffic data set pre-acquired based on a deep learning method, and the second target network traffic data set includes second sample target network traffic data labeled with a second detection result label.

[0182] Furthermore, the behavior detection model and threat detection model are trained in the following ways:

[0183] Inputting the first sample target network traffic data into the first initial model to extract detection features of the first sample target network traffic data through the first initial model, so as to obtain a first output result based on the detection features;

[0184] Based on the first output result and the first detection result label, the weight parameter of the first initial model is updated; and the step of inputting the first sample target network traffic data into the first initial model is continued until the first initial model converges to obtain a threat detection model;

[0185] Inputting the second sample target network traffic data into the second initial model to extract detection features of the second sample target network traffic data through the second initial model, so as to obtain a second output result based on the detection features;

[0186] Based on the second output result and the second detection result label, the weight parameters of the second initial model are updated; the step of inputting the second sample target network traffic data into the second initial model is continued until the second initial model converges to obtain a behavior detection model.

[0187] Furthermore, the acquisition module is also used to:

[0188] Monitor and receive original network traffic from the mirror port, splitter or optical splitter through the monitoring port;

[0189] Preprocessing the original network traffic through a preset processing tool to obtain first network traffic data; wherein the first network traffic data includes: parsing data, session statistics data, and data packet data;

[0190] The first network flow data is stored by a preset analysis component, and the stored first network flow data is preprocessed by a preset processing method to obtain target network flow data.

[0191] Furthermore, the preset processing method includes: a first preset processing method and a second preset processing method; the acquisition module is also used for:

[0192] Performing a first processing on the stored first network traffic data by a first preset processing method to obtain second network traffic data; wherein the first preset processing method includes traffic feature extraction processing and at least one of the following: cleaning processing, denoising processing, and formatting processing;

[0193] The second network traffic data is subjected to a second processing by a second preset processing method to obtain target network traffic data; the second preset processing method includes classification processing.

[0194] Furthermore, the acquisition module is also used to:

[0195] The second network traffic data is input into a pre-trained classification model to obtain a classification result corresponding to the second network traffic data and carrying traffic feature data; wherein the second network traffic data includes traffic feature data.

[0196] Furthermore, the classification model includes classification sub-models corresponding to multiple classification dimensions, and the multiple classification dimensions include: plaintext classification, protocol classification, application classification, and behavior classification; the acquisition module is also used for:

[0197] Input the second network traffic data into each pre-trained classification sub-model respectively, and obtain the classification sub-result output by each classification sub-model carrying the corresponding traffic sub-feature data; wherein each classification sub-model is trained based on the corresponding traffic sub-feature data set obtained in advance, and each traffic sub-feature data set includes the corresponding sample traffic sub-feature data marked with the classification sub-result label;

[0198] Each classification sub-result is determined as a classification result carrying flow characteristic data; wherein the flow characteristic data includes flow sub-characteristic data corresponding to each classification sub-model.

[0199] The AI-based network traffic detection device provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned AI-based network traffic detection method embodiment. For the part of the embodiment of the AI-based network traffic detection device, reference can be made to the corresponding content in the aforementioned AI-based network traffic detection method embodiment.

[0200] The embodiment of the present invention further provides an electronic device, see Figure 5 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned AI-based network traffic detection method.

[0201] Further, Figure 5The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0202] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0203] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0204] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned AI-based network traffic detection method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0205] The AI-based network traffic detection method, device and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0206] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network traffic detection method based on AI, characterized in that: The method comprises: Obtaining target network traffic data corresponding to the original network traffic; wherein the target network traffic data includes a classification result carrying traffic feature data; Input the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; the detection model includes: a threat detection model based on machine learning and a behavior detection model based on deep learning; Based on the detection result, determining whether the original network traffic is abnormal network traffic; If the original network traffic is abnormal network traffic, generate and display alarm information corresponding to the original network traffic; wherein, the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with detection result labels.

2. The method according to claim 1, characterized in that The detection model includes a behavior detection model and a threat detection model; wherein the threat detection model is obtained by learning a first target network traffic data set acquired in advance based on a machine learning method, and the first target network traffic data set includes first sample target network traffic data marked with a first detection result label; the first detection result label includes: a threat label or a no-threat label, and / or a threat type; The behavior detection model is obtained by learning a second target network traffic data set in advance based on a deep learning method, and the second target network traffic data set includes second sample target network traffic data marked with a second detection result label; the second detection result label includes: a normal behavior label or an abnormal behavior label, and / or a behavior type.

3. The method according to claim 2, characterized in that The behavior detection model and threat detection model are trained in the following way: Inputting the first sample target network traffic data into a first initial model to extract detection features of the first sample target network traffic data through the first initial model, so as to obtain a first output result based on the detection features; Based on the first output result and the first detection result label, updating the weight parameters of the first initial model; Continue to execute the step of inputting the first sample target network traffic data into the first initial model until the first initial model converges to obtain the threat detection model; Inputting the second sample target network traffic data into a second initial model to extract detection features of the second sample target network traffic data through the second initial model, so as to obtain a second output result based on the detection features; Based on the second output result and the second detection result label, updating the weight parameters of the second initial model; Continue to execute the step of inputting the second sample target network traffic data into the second initial model until the second initial model converges to obtain the behavior detection model.

4. The method according to claim 1, characterized in that: The steps of obtaining target network traffic data corresponding to the original network traffic include: Monitor and receive original network traffic from the mirror port, splitter or optical splitter through the monitoring port; Preprocess the original network traffic by using a preset processing tool to obtain first network traffic data; wherein the first network traffic data includes: parsing data, session statistics data, and data packet data; The first network traffic data is stored by a preset analysis component, and the stored first network traffic data is preprocessed by a preset processing method to obtain target network traffic data.

5. The method according to claim 4, characterized in that The preset processing method includes: a first preset processing method and a second preset processing method; the step of preprocessing the first stored network flow data by the preset processing method to obtain the target network flow data includes: Performing a first processing on the first stored network traffic data by a first preset processing method to obtain second network traffic data; wherein the first preset processing method includes traffic feature extraction processing and at least one of the following: cleaning processing, denoising processing, and formatting processing; The second network traffic data is subjected to a second processing by a second preset processing method to obtain target network traffic data; the second preset processing method includes classification processing.

6. The method according to claim 5, characterized in that The step of performing a second processing on the second network flow data by a second preset processing method to obtain target network flow data includes: The second network traffic data is input into a pre-trained classification model to obtain a classification result corresponding to the second network traffic data and carrying the traffic feature data; wherein the second network traffic data includes the traffic feature data.

7. The method according to claim 6, characterized in that The classification model includes classification sub-models corresponding to multiple classification dimensions, and the multiple classification dimensions include: plaintext classification, protocol classification, application classification, and behavior classification; the second network traffic data is input into a pre-trained classification model, and the step of obtaining a classification result corresponding to the second network traffic data and carrying the traffic feature data includes: Input the second network traffic data into each pre-trained classification sub-model respectively, and obtain the classification sub-result output by each classification sub-model carrying the corresponding traffic sub-feature data; wherein each classification sub-model is trained based on the corresponding traffic sub-feature data set acquired in advance, and each traffic sub-feature data set includes the corresponding sample traffic sub-feature data marked with the classification sub-result label; Each of the classification sub-results is determined as a classification result carrying the traffic feature data; wherein the traffic feature data includes the traffic sub-feature data corresponding to each of the classification sub-models.

8. An AI-based network traffic detection device, characterized in that: The device comprises: An acquisition module, used to acquire target network traffic data corresponding to the original network traffic; wherein the target network traffic data includes a classification result carrying traffic feature data; An input module is used to input the target network traffic data into a pre-trained detection model to obtain a detection result corresponding to the target network traffic data; the detection model includes: a threat detection model based on machine learning and a behavior detection model based on deep learning; A judgment module, used for judging whether the original network traffic is abnormal network traffic based on the detection result; An alarm module is used to generate and display alarm information corresponding to the original network traffic if the original network traffic is abnormal network traffic; wherein the detection model is trained based on a pre-acquired target network traffic data set, and the target network traffic data set includes sample target network traffic data marked with a detection result label.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the AI-based network traffic detection method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the AI-based network traffic detection method described in any one of claims 1-7.

Images