Real-time malicious traffic detection method and system supporting online algorithm development
Through online algorithm development and real-time data processing, real-time detection and dynamic response to new malicious traffic are achieved, and the problem that existing systems cannot cope with new threats is solved, the accuracy and efficiency of malicious traffic detection is improved, and network security is enhanced. Immediateness.
Patent Information
- Application Number
- CN202411080167.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
The existing malicious traffic detection system cannot effectively deal with new and variant malicious traffic, lacks fast learning and adaptability, and cannot deal with new threats in real time.
A real-time malicious traffic detection system supporting online algorithm development was designed, including data set management module, algorithm online development module and traffic alarm analysis module. Through data preprocessing, feature extraction, algorithm training and model deployment, real-time malicious traffic detection is realized. and dynamic updates.
The new traffic detection algorithm has been developed online customized, which improves the accuracy and efficiency of malicious traffic detection, can detect malicious traffic in a timely manner, enhances the immediacy and cutting-edge nature of network security, simplifies the detection process, and has good scalability and compatibility.
Smart Images

Figure CN119011234A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of network security applications, and in particular relates to a real-time malicious traffic detection method and system supporting online algorithm development. Background Art
[0002] With the rapid development of the Internet, network security issues are becoming increasingly serious. Malicious traffic detection has become an important means to ensure network security. Most existing malicious traffic detection systems rely on predefined rules or features and cannot effectively deal with new and variant malicious traffic. The emergence of new threats requires the detection system to have the ability to quickly learn and adapt. Summary of the invention
[0003] 1. Technical issues to be resolved
[0004] The technical problem to be solved by the present invention is to provide a real-time malicious traffic detection method and system that supports online algorithm development, realizes online customized development of new traffic detection algorithms, and realizes real-time malicious traffic detection at the same time.
[0005] (II) Technical solution
[0006] In order to solve the above technical problems, the present invention provides a real-time malicious traffic detection system supporting online algorithm development, comprising:
[0007] The data set management module is used to implement data set management, data processing, and data set overview functions; the specific functions implemented are as follows:
[0008] 1) Dataset management includes: extracting features from the original traffic PCAP package, uploading the original traffic PCAP package and the standardized CSV data set after feature extraction; the uploaded data is managed uniformly by the system's data warehouse to provide data support for online model training and model performance testing;
[0009] 2) Data processing is to perform data preprocessing operations on the uploaded original traffic PCAP packages, including data standardization, normalization, data dimension reduction, data scaling, data balancing, missing value supplementation and outlier processing; in addition, the feature customization operation of the original traffic PCAP data is also performed to convert the original PCAP file into a standardized feature file. The feature types include traffic statistical features, byte timing features of the PCAP package, and time-frequency features of the PCAP;
[0010] 3) Dataset overview: It realizes the function of accessing and querying the data in the current data warehouse, and analyzes the results of user data sets through data reports and data analysis tools;
[0011] The algorithm online development module is used to implement operator development, algorithm development, model training, and image management. The specific functions implemented are as follows:
[0012] 1) The operator development function is to upload the program code that can independently execute a specific function to the operator warehouse; an operator refers to a program that can independently complete a specific function. The combination of multiple operators that perform different functions can complete more complex tasks;
[0013] 2) The algorithm development function includes sub-functions such as third-party upload of algorithms, multi-operator algorithm arrangement, and online algorithm development. The third-party upload sub-function of algorithms is to upload the algorithm code that has been tested locally directly to the algorithm warehouse. The multi-operator algorithm arrangement sub-function is to select operators with certain specific functions in the operator warehouse, combine and arrange them according to the operator functions, and generate more complex algorithms. The online development sub-function of algorithms is to select packaged machine learning and deep learning algorithms, and select the required data sets in the data warehouse, set the training parameters and the number of iterations, and train the algorithms to obtain trained algorithms.
[0014] 3) The model performance test function includes the calculation and display of the algorithm accuracy, false alarm rate, and missed alarm rate, showing the effect of algorithm training and providing a basis for algorithm tuning;
[0015] 4) The image management function is to manage the docker image containing a specific python environment; both the operator development function and the algorithm development function require the selection of the docker image of the required python environment to implement;
[0016] The traffic alarm analysis module is used to implement traffic collection probe management, traffic analysis engine management, and real-time traffic analysis alarm situation display functions; the specific functions implemented are as follows:
[0017] 1) Traffic collection probe management includes probe operation status management, analysis and sending of probe-captured traffic data; the probe operation status management subfunction is to control the opening and closing of the traffic probe and monitor the operation status of the hardware device in real time; the analysis and sending subfunction of probe-captured traffic data is to analyze the original traffic captured by the traffic probe in the real network according to the corresponding traffic protocol, and then send the analyzed structured data of different protocol traffic through different topics of Kafka;
[0018] 2) Traffic analysis engine management mainly involves docker deployment and operation of algorithms obtained in the system through third-party upload, online development or multi-operator orchestration algorithm mode; the developed traffic algorithm code and the environment required for algorithm operation are put into docker for operation and start, and in the process of docker deployment and operation of the algorithm, the structured data sent through different topics of Kafka are continuously received, the received structured data is analyzed, and the analysis results are stored in the data warehouse;
[0019] 3) Real-time traffic analysis alarm situation display is a real-time alarm information display of the data analysis results obtained in the process of traffic analysis engine management. The displayed information includes analysis time, source IP, source port, destination IP, destination port, application protocol, threat results, and threat level; malicious traffic is identified by the source IP, source port, destination IP, destination port and IP quintuple of the application protocol and the timestamp of the time point when the data packet is transmitted in the network; when malicious traffic is detected, the alarm information is recorded, including analysis time, source IP, destination IP, traffic type, detected threat level, and malicious traffic type information, and stored in the log file and the data warehouse, and notifications are sent at the same time, alarms are sent through emails and dashboards, and the content of the notification includes the alarm information and the urgency of the threat.
[0020] The invention also provides a real-time malicious traffic detection method implemented by utilizing the system.
[0021] Preferably, when the data set management module performs data preprocessing or feature extraction on the uploaded data, it creates an interactive chart through the GoJS component and supports the creation of custom nodes and lines. Each node represents a specific behavior, and the nodes are connected in sequence. The business process composed of multiple complex behaviors is displayed in the form of a chart, and the code corresponding to the corresponding functional behavior is executed in the order of the connection, so that the function of a complex behavior can be executed by dragging the interactive chart.
[0022] Preferably, the feature extraction performed by the data set management module includes statistical feature extraction, timing feature extraction and time-frequency feature extraction of the traffic; the statistical features include traffic features extracted using Cicflowermeter or Nfstream tools, covering basic traffic statistical features, transport layer features, IP layer features, HTTP features, and SSL / TLS features; the timing feature extraction is to extract the byte stream of the PCAP packet and convert it into hexadecimal code, and put the extracted feature hexadecimal code into the CNN-LSTM-Attention model to extract the timing features of the traffic; the time-frequency feature extraction is to convert the original traffic PCAP packet into a time-frequency sequence vector and then perform a wavelet transform, and then perform a second and third wavelet transform on the obtained low-frequency part to obtain a signal mapped in a suitable frequency range as a time-frequency feature vector.
[0023] Preferably, the operator uploaded to the operator warehouse by the algorithm online development module is a zip compressed package, which contains the algorithm source code, Json configuration file, and operator usage instruction document; the multi-operator orchestration algorithm sub-function is implemented using GoJS and K8S, and the front-end interface uses the GoJS component to generate an interactive node connection diagram, where the nodes are operators in the operator warehouse that perform different functions, and the lines between the node connection diagrams represent the order of operator execution. When all connected operators are executed successfully in sequence, it means that the algorithm orchestration is successful and can be stored in the algorithm warehouse.
[0024] Preferably, the Json configuration file includes the operator's input parameters, output parameters, and program startup commands.
[0025] The invention also provides an application of the system in the technical field of network security application.
[0026] The invention also provides an application of the method in the technical field of network security application.
[0027] (III) Beneficial effects
[0028] The system proposed by the present invention integrates functions such as data set management and preprocessing, algorithm development and training, and algorithm model docker engine deployment and real-time alarm detection into one system, providing a one-stop solution and realizing unified management of functions. Users do not need to switch between multiple platforms, which improves work efficiency and simplifies the process of malicious traffic detection. In addition, the algorithm warehouse and operator warehouse provided by the system support the upload and call of third-party codes, and have good scalability and compatibility.
[0029] The system integrates a variety of machine learning and deep learning algorithms. Users can select the optimal algorithm for training and deployment, which improves the accuracy and efficiency of malicious traffic detection. After the model training is completed, the system can deploy it for real-time malicious traffic detection, which can detect malicious traffic in a timely manner and improve the immediacy of network security. In addition, the system supports dynamic updates of algorithms and models, and can respond quickly based on new threat intelligence and data to maintain the cutting-edge detection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 An architecture diagram of a real-time malicious traffic detection system supporting online algorithm development proposed in an embodiment of the present invention;
[0031] Figure 2 A flowchart of data set management proposed in an embodiment of the present invention;
[0032] Figure 3 A flowchart of online development of the algorithm proposed in the embodiment of the present invention;
[0033] Figure 4 This is a flow chart of the traffic alarm analysis proposed in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.
[0035] The present invention aims to provide a real-time malicious traffic detection method and system, which is a one-stop solution. The system of the present invention has the ability to develop new traffic detection algorithms online and also has the ability to deploy a traffic detection engine that integrates big data technology for real-time traffic detection, enriching the categories of traffic detection and simplifying the process of traffic alarm detection. The system supports customized generation of traffic features and online development of traffic detection algorithms. In addition to meeting the specific algorithm development needs of different users, it can also improve the detection effect of the algorithm model through online model training and performance tuning. The system deploys the algorithm to different traffic protocol engines and combines big data technology to detect malicious traffic in real time, forming a multi-level, all-round network security protection system with real-time detection and dynamic update detection capabilities, which can achieve timely response and processing of network threats and reduce the losses and impacts caused by network attacks.
[0036] The present invention provides a real-time malicious traffic detection method and system supporting online algorithm development, the system comprising:
[0037] The data set management module is used to implement data set management, data processing, and data set overview functions. The specific contents are as follows:
[0038] 1) Dataset management includes: uploading the original traffic PCAP package and the standardized CSV data set after PCAP feature extraction. The uploaded data is managed uniformly by the system's data warehouse to provide data support for online model training and model performance testing;
[0039] 2) Data processing is to perform data preprocessing operations on the uploaded raw traffic PCAP packages. The preprocessing operations supported by this system include data standardization, normalization, data dimension reduction, data scaling, data balancing, missing value supplementation, outlier processing, etc. In addition, the system also supports feature customization operations on raw traffic PCAP data, which can convert the original PCAP file into a standardized feature file. The selectable feature types include traffic statistical features, byte timing features of PCAP packages, and time-frequency features of PCAP. The extracted feature files will be stored in the data warehouse in csv format for future model training and testing.
[0040] 3) Dataset overview provides the ability to efficiently access and query data in the current data warehouse, supporting fast and flexible data query operations. Through data reports and data analysis tools, users can view accurate and easy-to-understand data set analysis results.
[0041] The algorithm online development module is used to implement operator development, algorithm development, model training, and image management. The specific contents are as follows:
[0042] 1) The operator development function is to upload the program code that can independently execute a specific function to the operator warehouse of the system. The meaning of an operator refers to a program that can independently complete a specific function, such as preprocessing of structured data, putting structured data into an algorithm model for prediction, and storing algorithm prediction data in a database. This kind of program code with independent input, which can complete a specific function and output the corresponding result, is collectively referred to as an operator in the present invention. The combination of multiple operators that perform different functions can complete more complex tasks. This design improves the ability to automatically process complex tasks, reduces the complexity and error rate of manual operations, simplifies the workload of task execution, and improves the efficiency of task execution.
[0043] 2) The algorithm development function involves sub-functions such as third-party upload of algorithms, multi-operator orchestration algorithms, and online algorithm development. The third-party upload sub-function of the algorithm is to upload the algorithm code that has been tested locally directly to the algorithm warehouse of the system. The multi-operator orchestration algorithm sub-function is to select operators with certain specific functions in the operator warehouse, and combine and arrange them according to the operator functions to achieve the effect of generating more complex algorithms. In this way, algorithms with different functions can be run with low code, which can effectively reduce the frequency of calling repeated code during algorithm debugging. The online development sub-function of the algorithm allows developers to select the encapsulated machine learning and deep learning algorithms (algorithm models) provided by this system, and train the algorithm by selecting the required data sets in the data warehouse, setting basic training parameters such as batch size and number of iterations, and obtaining a trained algorithm.
[0044] 3) Among them, the model performance test function involves the calculation and display of algorithm accuracy, false alarm rate, and missed alarm rate, which can intuitively demonstrate the effect of algorithm training and provide a basis for algorithm tuning.
[0045] 4) The image management function manages the docker image containing a specific python environment. Both the operator development function and the algorithm development function require the selection of the docker image of the required python environment to implement (algorithm operation).
[0046] The traffic alarm analysis module is used to implement traffic collection probe management, traffic analysis engine management, and real-time traffic analysis alarm situation display functions. The specific contents are as follows:
[0047] 1) Traffic collection probe management includes sub-functions such as probe operation status management, analysis and sending of probe-captured traffic data. The probe operation status management sub-function is to control the opening and closing of the traffic probe and monitor the operation status of the hardware device in real time; the analysis and sending sub-function of probe-captured traffic data is to analyze the original traffic captured by the traffic probe in the real network according to the corresponding traffic protocol, and then send the parsed structured data of different protocol traffic through different topics of Kafka.
[0048] 2) Traffic analysis engine management mainly involves docker deployment and operation of (traffic) algorithms obtained in the system through third-party uploads, online development, or multi-operator orchestration algorithms. The management of docker clusters is mainly implemented through K8S. The traffic algorithm code developed in the system and the environment required for the algorithm to run are put into docker for operation and start, and in the process of docker deployment and operation of the algorithm, the structured data sent through different topics of Kafka are continuously received, the received structured data is analyzed, and the analysis results are stored in the data warehouse. The K8S provides an automated system to ensure effective management of Docker, including container creation, start, stop, and deletion, image management, network configuration, performance tuning, and security assurance.
[0049] 3) Real-time traffic analysis alarm situation display is to display the real-time alarm information of the data analysis results obtained in the process of traffic analysis engine management. The displayed information includes analysis time, source IP, source port, destination IP, destination port, application protocol, threat results, and threat level. Malicious traffic is identified by the source IP, source port, destination IP, destination port, and IP quintuple of the application protocol and the timestamp of the time point when the data packet is transmitted in the network. When malicious traffic is detected, the system will record the alarm information, including detailed information such as analysis time, source IP, destination IP, traffic type, detected threat level, malicious traffic type, etc., and store it in the log file and the data warehouse for subsequent analysis and auditing. At the same time, notifications are sent to relevant personnel, and alarms are issued via email and dashboard. The content of the notification contains detailed alarm information and the urgency of the threat.
[0050] The system architecture of the present invention is designed as a hierarchical design mode, which is conducive to modular development and improves the maintainability and scalability of the system. For detailed design, please refer to Figure 1. The main functional layers of the design are data source layer, data collection layer, data storage layer, and algorithm analysis layer. The data source layer is a real network environment. Multiple hardware flow probes are deployed in the data collection layer to monitor and capture flow data. The flow probe captures flow data from the real network environment, performs preliminary cleaning, conversion, and loading into the data storage layer. The data storage layer adopts a combination of HDFS and Kafka. HDFS can process a large amount of data in parallel and support high-concurrency read and write operations, thereby providing high-throughput data access performance. Kafka supports concurrent consumption, that is, multiple consumers can read data from a topic at the same time, and consumers do not affect each other. This enables Kafka to efficiently handle a large number of concurrent requests. The data computing layer uses the Clickhouse database to store the structured data required by the model. Its high-performance column storage and distributed processing capabilities are very suitable for large-scale data analysis and real-time query scenarios. The data application layer is for specific algorithm development, real-time flow detection, and situation alarm display, mainly using docker and K8S technologies. The backend of this system uses Django as the system development framework. Django is an efficient, secure, full-featured and extensible web framework suitable for rapid development and maintenance of complex applications.
[0051] See also Figure 2The data set management module in the present invention provides custom feature extraction of the original PCAP traffic data and performs preprocessing operations on the extracted traffic feature files. When large files such as PCAP, csv, and Json are transmitted on the interface, the system will upload the large files to HDFS and map the file name and HDFS address in the relational database as the unique identifier of the data download address for subsequent model training and testing. The uploaded file will be displayed on the interface. If data preprocessing or feature extraction is required for the uploaded file, the system creates an interactive chart through the GoJS component and supports the creation of custom nodes and lines. Each node represents a specific behavior, and the nodes can be connected and sorted in order. The business process composed of multiple complex behaviors can be displayed in an intuitive chart form, and the code corresponding to the corresponding functional behavior is executed in the order of the connection, so that a complex behavior function can be executed by dragging the interactive chart. Data processing includes data standardization, data standardization, normalization, data dimension reduction, data scaling, data balancing, missing value supplementation, outlier processing and other data preprocessing operations. PCAP feature extraction includes statistical feature extraction, time series feature extraction and time-frequency feature extraction of traffic. Statistical features include traffic features extracted using tools such as Cicflowermeter and Nfstream, covering basic traffic statistical features, transport layer features, IP layer features, HTTP features, SSL / TLS features, etc. Time series feature extraction is to extract the byte stream of the PCAP packet and convert it into hexadecimal code, and put the extracted feature hexadecimal code into the CNN-LSTM-Attention model to extract the time series features of the traffic. Time-frequency feature extraction is to convert the original traffic PCAP packet into a time-frequency sequence vector, first perform a wavelet transform on the obtained low-frequency part, and then perform a third wavelet transform on the obtained low-frequency part to obtain a signal mapped in a suitable frequency range as a time-frequency feature vector. In addition, this module also provides a data set overview function, and uses Python's matplotlib to build line charts, pie charts, bar charts and other charts to intuitively display the number of original PCAP files and feature files in the data warehouse.
[0052] See also Figure 3, the embodiment of the present invention provides the function of algorithm development. Algorithm development involves multi-operator orchestration of the algorithm, third-party upload of the algorithm, and online development of the algorithm. The multi-operator orchestration of the algorithm refers to combining and executing certain operators that can independently perform specific functions to complete more complex algorithm functions. An operator is a piece of program code that receives input, independently runs a specific behavior, and generates output results. Each operator is an independent computing unit that performs different functions such as data preprocessing, model prediction, and result storage. The operator uploaded to the system is a zip compressed package, which contains the algorithm source code, Json configuration file (including the input parameters, output parameters, and program startup commands of the operator), and the instruction document for the operator. The uploaded operator is passed to the operator warehouse. The multi-operator orchestration function is implemented using GoJS and k8s. The front-end interface uses the GoJS component to generate an interactive node connection diagram, where the nodes are operators in the operator warehouse that perform different functions, and the lines between the diagrams represent the order in which the operators are executed. When all connected operators are executed successfully in sequence, it means that the algorithm orchestration is successful and can be stored in the algorithm warehouse. The algorithm orchestration backend is implemented by managing docker through k8s. K8s is an open source container orchestration platform for deploying, scaling and managing containerized applications. This platform uses k8s's yaml file to define various resources, and implements the sequential execution of Docker operator files with different functions, and processes the data input and output between Docker operator files. The algorithm third party uploads the complete algorithm code, which can be directly run in the python docker environment to produce results. The online development of the algorithm encapsulates the currently popular machine learning and deep learning algorithms for developers to call. Developers can select the encapsulated algorithm in the model training interface, set basic training parameters such as batch size, number of iterations and other training parameters, and select the required training data in the data warehouse to conduct systematic online training of the algorithm. The trained model will be directly stored in the algorithm warehouse. The algorithm in the algorithm warehouse is used for real-time alarm analysis of the algorithm later. The function of the model performance test module is to select test data from the data warehouse and perform performance testing on the algorithm. The results of the algorithm test include the calculation and display of the algorithm accuracy, false alarm rate and false negative rate. The image management function includes the Docker image management of the algorithm running environment. The operation of operators and algorithms requires the selection of the Docker image of the corresponding running environment to execute.
[0053] See also Figure 4The present invention provides a function of real-time traffic alarm detection of an algorithm. First, the traffic probe captures the corresponding traffic data and parses the original traffic according to different traffic protocols. The characteristic fields of the parsed traffic data of different protocols are sent to specific IPs through different topics of Kafka. The developed traffic algorithm is deployed in a specific python environment docker image to become a traffic detection engine. All algorithm engines are managed by k8s for unified automatic container scheduling, automatic rolling update and rollback. The started algorithm engine is a consumer of Kafka and receives data from the Kafka producer IP. The received data is transmitted to the algorithm for analysis and detection, and the detected alarm data is stored in the data warehouse. The back end of the situation monitoring platform takes out the data predicted by the algorithm from the data warehouse every minute and sends it to the situation display interface for display. The displayed data includes traffic capture time, source IP, source port, destination IP, destination port, application protocol, threat result, and threat level. Malicious traffic is uniquely identified by IP quintuple and time.
[0054] The above is a detailed introduction to a real-time malicious traffic detection system that supports online algorithm development provided by the present invention. In the current era of rapid development of the Internet, network security issues are becoming increasingly prominent. Designing an integrated, real-time, efficient and flexible traffic intrusion detection system not only has important technical significance, but is also a necessary means to ensure network security. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technicians in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation of the present invention.
Claims
1. A real-time malicious traffic detection system that supports online algorithm development, It is characterized in that include: The data set management module is used to implement data set management, data processing, and data set overview functions; The specific functions implemented are as follows: 1) Dataset management includes: extracting features from the original traffic PCAP package, uploading the original traffic PCAP package and the standardized CSV data set after feature extraction; the uploaded data is managed uniformly by the system's data warehouse to provide data support for online model training and model performance testing; 2) Data processing is to perform data preprocessing operations on the uploaded original traffic PCAP packages, including data standardization, normalization, data dimension reduction, data scaling, data balancing, missing value supplementation and outlier processing; in addition, the feature customization operation of the original traffic PCAP data is also performed to convert the original PCAP file into a standardized feature file. The feature types include traffic statistical features, byte timing features of the PCAP package, and time-frequency features of the PCAP; 3) Dataset overview: It realizes the function of accessing and querying the data in the current data warehouse, and analyzes the results of user data sets through data reports and data analysis tools; The algorithm online development module is used to implement operator development, algorithm development, model training, and image management. The specific functions implemented are as follows: 1) The operator development function is to upload the program code that can independently execute a specific function to the operator warehouse; an operator refers to a program that can independently complete a specific function. The combination of multiple operators that perform different functions can complete more complex tasks; 2) The algorithm development function includes sub-functions such as third-party upload of algorithms, multi-operator algorithm arrangement, and online algorithm development. The third-party upload sub-function of algorithms is to upload the algorithm code that has been tested locally directly to the algorithm warehouse. The multi-operator algorithm arrangement sub-function is to select operators with certain specific functions in the operator warehouse, combine and arrange them according to the operator functions, and generate more complex algorithms. The online development sub-function of algorithms is to select packaged machine learning and deep learning algorithms, and select the required data sets in the data warehouse, set the training parameters and the number of iterations, and train the algorithms to obtain trained algorithms. 3) The model performance test function includes the calculation and display of the algorithm accuracy, false alarm rate, and missed alarm rate, showing the effect of algorithm training and providing a basis for algorithm tuning; 4) The image management function is to manage the docker image containing a specific python environment; both the operator development function and the algorithm development function require the selection of the docker image of the required python environment to implement; The traffic alarm analysis module is used to implement traffic collection probe management, traffic analysis engine management, and real-time traffic analysis alarm situation display functions; the specific functions implemented are as follows: 1) Traffic collection probe management includes probe operation status management, analysis and sending of probe-captured traffic data; the probe operation status management subfunction is to control the opening and closing of the traffic probe and monitor the operation status of the hardware device in real time; the analysis and sending subfunction of probe-captured traffic data is to analyze the original traffic captured by the traffic probe in the real network according to the corresponding traffic protocol, and then send the analyzed structured data of different protocol traffic through different topics of Kafka; 2) Traffic analysis engine management mainly involves docker deployment and operation of algorithms obtained in the system through third-party upload, online development or multi-operator orchestration algorithm mode; the developed traffic algorithm code and the environment required for algorithm operation are put into docker for operation and start, and in the process of docker deployment and operation of the algorithm, the structured data sent through different topics of Kafka are continuously received, the received structured data is analyzed, and the analysis results are stored in the data warehouse; 3) Real-time traffic analysis alarm situation display is a real-time alarm information display of the data analysis results obtained in the process of traffic analysis engine management. The displayed information includes analysis time, source IP, source port, destination IP, destination port, application protocol, threat results, and threat level; malicious traffic is identified by the source IP, source port, destination IP, destination port and IP quintuple of the application protocol and the timestamp of the time point when the data packet is transmitted in the network; when malicious traffic is detected, the alarm information is recorded, including analysis time, source IP, destination IP, traffic type, detected threat level, and malicious traffic type information, and stored in the log file and the data warehouse, and notifications are sent at the same time, alarms are sent through emails and dashboards, and the content of the notification includes the alarm information and the urgency of the threat.
2. The system according to claim 1, It is characterized in that The management of docker cluster is implemented through K8S.
3. The system according to claim 1, It is characterized in that The system is designed based on Django.
4. A real-time malicious traffic detection method implemented using the system as described in claim 1, 2 or 3.
5. The method according to claim 4, It is characterized in that When the dataset management module performs data preprocessing or feature extraction on the uploaded data, it creates interactive diagrams through GoJS components and supports the creation of custom nodes and lines. Each node represents a specific behavior, and the nodes are connected in sequence. The business process composed of multiple complex behaviors is displayed in the form of a diagram, and the code corresponding to the corresponding functional behavior is executed in the order of the connection, so that the function of a complex behavior can be executed by dragging the interactive diagram.
6. The method according to claim 4, It is characterized in that The feature extraction performed by the data set management module includes traffic statistical feature extraction, time series feature extraction, and time-frequency feature extraction; Statistical features include traffic features extracted using Cicflowermeter or Nfstream tools, covering basic traffic statistical features, transport layer features, IP layer features, HTTP features, and SSL / TLS features; Time series feature extraction is to extract the byte stream of the PCAP packet and convert it into hexadecimal code, and then put the extracted feature hexadecimal code into the CNN-LSTM-Attention model to extract the time series features of the traffic; Time-frequency feature extraction is to convert the original traffic PCAP packet into a time-frequency sequence vector and then perform a wavelet transform on it. Then, the low-frequency part is subjected to a second and third wavelet transform to obtain a signal mapped in a suitable frequency range as a time-frequency feature vector.
7. The method according to claim 4, It is characterized in that The operators uploaded to the operator warehouse by the algorithm online development module are zip compressed packages, which include the algorithm source code, Json configuration files, and operator usage instructions. The multi-operator orchestration algorithm sub-function is implemented using GoJS and K8S, and the front-end interface uses the GoJS component to generate an interactive node connection diagram, where the nodes are operators in the operator warehouse that perform different functions, and the lines between the node connection diagrams represent the order in which the operators are executed. When all connected operators are executed successfully in order, it means that the algorithm orchestration is successful and can be stored in the algorithm warehouse.
8. The method according to claim 7, It is characterized in that The Json configuration file contains the operator's input parameters, output parameters, and program startup commands.
9. An application of the system as claimed in claim 1, 2 or 3 in the field of network security application technology.
10. Application of the method according to any one of claims 4 to 8 in the field of network security application technology.
