An AI-based real-time detection method and system for abnormal network traffic

Abstract

This invention relates to the field of network traffic technology, and more particularly to an AI-based real-time abnormal network traffic detection method and system. The system includes: a data processing module that collects network traffic data in real time and sequentially cleans, de-identifies, structures, and extracts features from the data; a traffic detection module that trains an initial model, analyzes the network traffic data, and issues an early warning when anomalies are detected in the analysis results; an executability judgment module that determines whether the executability of the analysis results does not meet requirements based on the quality characterization value of the analysis results; a depth adjustment module that determines the elastic resolution coefficient of the network traffic data parsing depth based on the packet loss rate of the network traffic data; and a granularity adjustment module that determines the triggering dynamic response granularity of the traffic features based on the generation delay of the early warning information. This invention improves the stability of real-time detection.

Description

Technical Field

[0001] This invention relates to the field of network traffic technology, and in particular to an AI-based method and system for real-time detection of abnormal network traffic. Background Technology

[0002] In existing technologies, rule-based matching, signature detection, or supervised machine learning methods have been used to identify anomalies in network traffic, which has achieved the detection of known attacks to a certain extent. However, there are still problems such as difficulty in effectively discovering unknown threats, reliance on a large amount of labeled data, weak model generalization ability, high false alarm rate under high-dimensional heterogeneous traffic characteristics, poor real-time performance, lack of a closed-loop mechanism for dynamic evaluation and adaptive adjustment of the reliability of detection results, and difficulty in meeting the stability requirements of real-time detection.

[0003] Chinese Patent Application Publication No. CN121441643A discloses a holographic traffic monitoring method and system based on AI and BI. The system includes: a streaming data processing module for real-time cleaning, transformation, and calculation of incoming multi-source heterogeneous data streams; an AI large-scale model analysis module for behavioral analysis, detection, and correlation analysis of the data transmitted by the streaming data processing module through machine learning models; a BI visualization analysis module for spatiotemporal visualization, trend prediction, and dynamic risk scoring of the analysis results from the AI ​​large-scale model analysis module; an interactive query engine for providing access-controlled, permission-based monitoring data queries based on natural language; and a data platform module for unified collection, fusion, storage, governance, and auditing of logs, traffic, and threat intelligence data. The AI ​​large-scale model analysis module, BI visualization analysis module, and interactive query engine all interact bidirectionally with the data platform module. It is evident that the AI ​​and BI-based holographic traffic monitoring system suffers from several issues. Because each module only interacts bidirectionally with the data platform, there is a lack of real-time collaboration between streaming data processing and AI large-scale model analysis. This results in cumulative delays when multi-source heterogeneous data is centrally managed by the data platform. Furthermore, the AI ​​model's detection trigger strategy is fixed and cannot be dynamically adjusted according to traffic load, leading to a decrease in the stability of real-time detection. Summary of the Invention

[0004] To address this, the present invention provides an AI-based real-time detection method and system for abnormal network traffic, which overcomes the problems in the prior art where each module only interacts bidirectionally with the data platform, lacks a real-time collaborative mechanism between streaming data processing and AI large model analysis, accumulates delays when multi-source heterogeneous data is centrally managed by the data platform, and has a fixed AI model detection trigger strategy that cannot be dynamically adjusted according to traffic load, resulting in decreased stability of real-time detection.

[0005] To achieve the above objectives, the present invention provides an AI-based real-time abnormal network traffic detection system, comprising: The data processing module includes a data acquisition unit for real-time acquisition of network traffic data and a preprocessing unit connected to the data acquisition unit for sequentially cleaning, desensitizing, structuring, and feature extraction of the network traffic data to obtain traffic features. The traffic detection module, which is connected to the data processing module, includes a model training unit for training an initial model based on the traffic characteristics to obtain a deep learning model, an analysis unit connected to the model training unit for analyzing network traffic data based on the deep learning model to obtain analysis results, and an early warning unit connected to the analysis unit for issuing early warning information when the analysis results are abnormal. An executability determination module, which is connected to the data processing module, is used to determine whether the executability of the analysis results does not meet the requirements based on the quality characterization value of the analysis results determined by the abnormal proportion of network traffic data and the false alarm rate of early warning information. A depth adjustment module, which is connected to the executability determination module, is used to determine the elastic resolution coefficient of the network traffic data parsing depth based on the packet loss rate of the network traffic data when the executability of the analysis results does not meet the requirements. The granularity adjustment module, which is connected to the depth adjustment module, is used to determine the triggering dynamic response granularity of the traffic characteristics based on the generation delay of the early warning information.

[0006] Furthermore, the executability determination module determines the quality characterization value of the analysis result based on the ratio of the abnormal proportion of network traffic data to the false alarm rate of the early warning information, in order to determine whether the executability of the analysis result meets the requirements.

[0007] Furthermore, the executability determination module determines that the executability of the analysis result meets the requirements when the quality characterization value of the analysis result is less than or equal to a preset characterization value. The executability determination module determines that the executability of the analysis result does not meet the requirements if the quality characterization value of the analysis result is greater than the preset characterization value.

[0008] Furthermore, in response to the condition that the executability of the analysis results does not meet the requirements, the depth adjustment module determines whether the integrity of the network traffic data collection meets the requirements based on the packet loss rate of the network traffic data.

[0009] Furthermore, the depth adjustment module determines that the integrity of the network traffic data collection meets the requirements when the packet loss rate of the network traffic data is less than or equal to a preset first packet loss rate. The depth adjustment module determines that the integrity of the network traffic data collection does not meet the requirements when the packet loss rate of the network traffic data is greater than the preset first packet loss rate.

[0010] Furthermore, the depth adjustment module increases the elastic resolution coefficient of the network traffic data parsing depth in response to a packet loss rate of network traffic data that is greater than a preset first packet loss rate and less than or equal to a preset second packet loss rate. The deep adjustment module responds to the network traffic data packet loss rate being greater than the preset second packet loss rate, initially determining that the real-time performance of the deep learning model's early warning does not meet the requirements, and then determines whether the real-time performance of the deep learning model's early warning meets the requirements based on the generation delay of the early warning information.

[0011] Furthermore, the increase in the elastic resolution coefficient of the network traffic data parsing depth is determined by the difference between the packet loss rate of the network traffic data and the preset first packet loss rate.

[0012] Furthermore, in response to the condition that the packet loss rate of network traffic data is greater than a preset second packet loss rate, the granularity adjustment module determines whether the real-time performance of the deep learning model's early warning meets the requirements based on the generation delay of the early warning information.

[0013] Furthermore, if the granularity adjustment module responds to the generation delay of the warning information being less than or equal to the preset delay, it determines that the real-time performance of the warning from the deep learning model meets the requirements. The granularity adjustment module responds when the generation delay of the warning information exceeds the preset delay time, determines that the real-time performance of the warning from the deep learning model does not meet the requirements, and increases the granularity of the dynamic response triggered by the traffic feature. The increase in the dynamic response granularity triggered by the traffic characteristics is determined by the difference between the generation delay of the warning information and the preset delay.

[0014] This invention also provides an AI-based real-time detection method for abnormal network traffic, comprising: Real-time network traffic data is collected, and the network traffic data is sequentially cleaned, desensitized, structured, and feature extracted to obtain traffic features; The initial model is trained based on the traffic characteristics to obtain a deep learning model. The network traffic data is analyzed based on the deep learning model to obtain analysis results, and an early warning message is issued when the analysis results are abnormal. The abnormal proportion of network traffic data and the false alarm rate of early warning information are obtained to determine the quality characterization value of the analysis results, and the executability of the analysis results is determined based on the quality characterization value of the analysis results. If the executability of the analysis results does not meet the requirements, the packet loss rate of the network traffic data is used to determine whether the integrity of the network traffic data collection meets the requirements. If the integrity of the network traffic data collection does not meet the requirements, then determine whether it is necessary to increase the elastic resolution coefficient of the network traffic data parsing depth; If it is not necessary to increase the elastic resolution coefficient of network traffic data parsing depth, then the generation delay of obtaining early warning information determines the granularity of the dynamic response triggered by traffic characteristics.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: The system of this invention, by setting up a data processing module, a traffic detection module, an executability judgment module, a depth adjustment module, and a granularity adjustment module, determines the executability of the analysis results based on the quality characterization value of the analysis results determined by the abnormal proportion of network traffic data and the false alarm rate of early warning information. Because the data distribution of network traffic drifts, the model's generalization ability decreases, and high-frequency false alarms cause overload in the early warning processing link. By determining the executability of the analysis results, the system can promptly identify model output distortion and early warning link congestion, avoiding frequent triggering of invalid early warnings that leads to slow monitoring system response and chaotic handling logic. This ensures the stable and reliable execution of the abnormal detection and early warning handling process. The system also adjusts the elastic resolution coefficient of the network traffic data parsing depth based on the packet loss rate of the network traffic data. Due to uneven task allocation among the acquisition nodes, some acquisition nodes may become overloaded. Packet queuing overflow leads to packet loss. By increasing the elastic resolution coefficient of network traffic data parsing depth, parsing resources can be precisely allocated based on node load and traffic importance, balancing the processing pressure of each collection node, improving overall processing throughput, alleviating packet queuing overflow, reducing packet loss, and ensuring collection integrity. The triggering dynamic response granularity of traffic features can be adjusted according to the generation delay of early warning information. Due to the accumulation of real-time traffic data to be analyzed, the transmission and generation delay of early warning messages increases, the analysis scheduling efficiency is low, and the early warning queue is continuously congested, preventing real network attacks from being detected and dealt with in a timely manner. By increasing the triggering dynamic response granularity of traffic features, the feature output rhythm can be controlled more finely according to node load, reducing data accumulation waiting and cache retention, accelerating feature flow and model inference efficiency, alleviating data backlog, shortening early warning generation time, and improving the stability of real-time detection.

[0016] Furthermore, the system described in this invention determines the executability of the analysis results by setting preset characterization values. Due to the drift in the data distribution of network traffic and the decline in the generalization ability of the model, high-frequency misjudgment results cause overload of the early warning processing link. By determining the executability of the analysis results, the system can promptly identify model output distortion and early warning link congestion problems, avoid frequent triggering of invalid early warnings that leads to slow response of the monitoring system and chaotic handling logic, ensure the stable and reliable execution of the anomaly detection and early warning handling process, and further improve the stability of real-time detection.

[0017] Furthermore, the system of the present invention adjusts the elastic resolution coefficient of the network traffic data parsing depth by setting a preset first packet loss rate and a preset second packet loss rate. Due to uneven task allocation among collection nodes, some collection nodes become overloaded, causing data packet queuing overflow and resulting in data packet loss. By increasing the elastic resolution coefficient of the network traffic data parsing depth, parsing resources can be accurately allocated according to the differences in node load and traffic importance, balancing the processing pressure of each collection node, improving the overall processing throughput, alleviating data packet queuing overflow, reducing packet loss and ensuring collection integrity, and further improving the stability of real-time detection.

[0018] Furthermore, the system described in this invention adjusts the granularity of the dynamic response triggered by traffic features by setting a preset delay duration. Due to the accumulation of real-time traffic data to be analyzed, the transmission and generation delay of early warning messages increases, the analysis and scheduling efficiency is low, and the early warning queue is continuously congested, making it impossible to detect and deal with real network attacks in a timely manner. By increasing the granularity of the dynamic response triggered by traffic features, the feature output rhythm can be controlled more finely according to the node load, reducing data accumulation waiting and cache retention, accelerating feature flow and model inference efficiency, alleviating data backlog, shortening the early warning generation time, and further improving the stability of real-time detection. Attached Figure Description

[0019] Figure 1 This is an overall structural block diagram of the AI-based real-time abnormal network traffic detection system according to an embodiment of the present invention; Figure 2 This is a logical flowchart illustrating the executable process of the judgment and analysis results of the AI-based real-time abnormal network traffic detection system according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the process of determining the elastic resolution coefficient for network traffic data parsing depth in the AI-based real-time abnormal network traffic detection system according to an embodiment of the present invention. Figure 4 This is an overall flowchart of the AI-based real-time abnormal network traffic detection method according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0021] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0022] Please see Figure 1The diagram shown is an overall structural block diagram of the AI-based real-time network traffic detection system according to an embodiment of the present invention.

[0023] An embodiment of the present invention discloses an AI-based real-time abnormal network traffic detection system, comprising: The data processing module includes a data acquisition unit for real-time acquisition of network traffic data and a preprocessing unit connected to the data acquisition unit for sequentially cleaning, desensitizing, structuring, and feature extraction of the network traffic data to obtain traffic features. The traffic detection module, which is connected to the data processing module, includes a model training unit for training an initial model based on the traffic characteristics to obtain a deep learning model, an analysis unit connected to the model training unit for analyzing network traffic data based on the deep learning model to obtain analysis results, and an early warning unit connected to the analysis unit for issuing early warning information when the analysis results are abnormal. An executability determination module, which is connected to the data processing module, is used to determine whether the executability of the analysis results does not meet the requirements based on the quality characterization value of the analysis results determined by the abnormal proportion of network traffic data and the false alarm rate of early warning information. A depth adjustment module, which is connected to the executability determination module, is used to determine the elastic resolution coefficient of the network traffic data parsing depth based on the packet loss rate of the network traffic data when the executability of the analysis results does not meet the requirements. The granularity adjustment module, which is connected to the depth adjustment module, is used to determine the triggering dynamic response granularity of the traffic characteristics based on the generation delay of the early warning information.

[0024] Specifically, network traffic data includes port transmit / receive bandwidth utilization, kernel packet loss count, and raw packet length.

[0025] Specifically, de-identification involves removing or obscuring data fields that involve user privacy, sensitive information, or compliance controls.

[0026] Specifically, traffic characteristics include port bandwidth utilization after cleaning, kernel packet loss after desensitization, and original packet length after structuring.

[0027] Specifically, the process of training the initial model to obtain a deep learning model based on traffic characteristics involves constructing a time-series sample set of traffic characteristics and dividing it into a training set, a validation set, and a test set. The training set is then input into the initial model for forward training and parameter iteration. Model parameters are tuned based on the validation set, and model performance is verified based on the test set to obtain the deep learning model.

[0028] Specifically, the initial model is a basic model framework that already has the ability to analyze and detect traffic features but has not yet undergone parameter iteration optimization using data feature samples.

[0029] Specifically, the deep learning model can be a long short-term memory network model, a gated recurrent unit network model, or a multilayer perceptron model, with the preferred embodiment being a long short-term memory network model.

[0030] Specifically, the process of analyzing network traffic data based on a deep learning model to obtain analysis results involves inputting the network traffic data into the deep learning model, fusing and calculating the traffic features, identifying the current state of the network, and outputting the analysis results.

[0031] Specifically, the analysis results include a single source IP making high-frequency connection attempts to the same destination IP within a short period of time, an abnormally high frequency of requests per unit time, and a transmission destination address that is an unconventional external network address.

[0032] Specifically, anomalies in the analysis results include a surge in target port entropy, a sudden increase in uplink traffic, and a sharp rise in the proportion of specific flag bits.

[0033] Specifically, the warning information includes malicious port scanning, abnormal external data transmission, and DDoS traffic attacks.

[0034] Specifically, the elastic resolution coefficient of network traffic data parsing depth is a quantitative parameter that characterizes the adaptive adjustment of the network traffic data parsing level and the fineness of feature extraction based on the current traffic load.

[0035] Specifically, the triggering dynamic response granularity of traffic features is a quantitative parameter that characterizes the timing and response accuracy of adaptively adjusting traffic feature extraction, matching, and anomaly detection based on real-time traffic load.

[0036] In implementation, the system of this invention sets up a data processing module, a traffic detection module, an executability judgment module, a depth adjustment module, and a granularity adjustment module. It judges the executability of the analysis results based on the quality characterization value determined by the abnormal proportion of network traffic data and the false alarm rate of early warning information. Because the data distribution of network traffic drifts, the model's generalization ability decreases, and high-frequency false alarms cause overload in the early warning processing link. By judging the executability of the analysis results, it can promptly identify model output distortion and early warning link congestion, avoiding frequent triggering of invalid early warnings that leads to slow monitoring system response and chaotic handling logic. This ensures the stable and reliable execution of the abnormality detection and early warning handling process. The system adjusts the elastic resolution coefficient of the network traffic data parsing depth based on the packet loss rate of the network traffic data. Due to uneven task allocation among the collection nodes, some collection nodes become overloaded, causing data packet overflow. This leads to packet loss. By increasing the elastic resolution coefficient of network traffic data parsing depth, parsing resources can be precisely allocated according to node load and traffic importance, balancing the processing pressure of each collection node, improving overall processing throughput, alleviating packet queuing overflow, reducing packet loss and ensuring collection integrity. The dynamic response granularity of traffic features is adjusted according to the generation delay of warning information. Due to the accumulation of real-time traffic data to be analyzed, the transmission and generation delay of warning messages increases, the analysis scheduling efficiency is low, and the warning queue is continuously congested, making it impossible to detect and deal with real network attacks in a timely manner. By increasing the dynamic response granularity of traffic features, the feature output rhythm can be controlled more finely according to node load, reducing data accumulation waiting and cache retention, accelerating feature flow and model inference efficiency, alleviating data backlog, shortening the warning generation time, and improving the stability of real-time detection.

[0037] Please continue reading. Figure 2 The diagram shown is a logical flowchart illustrating the executable process of the judgment and analysis results of the AI-based real-time network traffic detection system according to an embodiment of the present invention.

[0038] Specifically, the executability determination module determines the quality characterization value of the analysis result based on the ratio of the abnormal proportion of network traffic data to the false alarm rate of the early warning information, in order to determine whether the executability of the analysis result meets the requirements.

[0039] Specifically, the executability determination module determines that the executability of the analysis result meets the requirements when the quality characterization value of the analysis result is less than or equal to a preset characterization value. The executability determination module determines that the executability of the analysis result does not meet the requirements if the quality characterization value of the analysis result is greater than the preset characterization value.

[0040] Understandably, in an AI-based real-time anomaly network traffic detection system, the core logic of using preset characterization values ​​to assess the executability of analysis results is to transform the executability of the analysis results into quantifiable characterization values ​​for judgment. The preset characterization value serves as the dividing line for determining whether the executability of the analysis results meets the requirements. The preset characterization value can be set according to actual operating conditions. The preset characterization value aims to ensure the stability and practicality of real-time detection. Optionally, the preset characterization value is determined through a limited number of experiments by evaluating the effect of different characterization values ​​on real-time detection. The determined preset characterization value should satisfy the condition that it is neither too small nor causes excessive interference to the real-time detection process. For example, the preset characterization value is generally selected within the range of [60%, 70%].

[0041] Preferably, the preset characterization value is 65% in the preferred embodiment.

[0042] Specifically, the percentage of abnormal network traffic data is the ratio of the number of abnormal network traffic to the total number of network traffic.

[0043] Specifically, the false alarm rate of early warning information is the ratio of the number of false alarms to the total number of early warning information.

[0044] In practice, the system described in this invention determines the executability of the analysis results by setting preset characterization values. Due to the drift in the data distribution of network traffic and the decline in the generalization ability of the model, high-frequency misjudgment results cause overload of the early warning processing link. By determining the executability of the analysis results, the system can promptly identify model output distortion and early warning link congestion problems, avoid frequent triggering of invalid early warnings that leads to slow response of the monitoring system and chaotic handling logic, ensure the stable and reliable execution of the anomaly detection and early warning handling process, and further improve the stability of real-time detection.

[0045] Please continue reading. Figure 3 The diagram shown is a logical flowchart of the process for determining the elastic resolution coefficient of network traffic data parsing depth in the AI-based real-time network traffic detection system according to an embodiment of the present invention.

[0046] Specifically, in response to the condition that the executability of the analysis results does not meet the requirements, the depth adjustment module determines whether the integrity of the network traffic data collection meets the requirements based on the packet loss rate of the network traffic data.

[0047] Specifically, the depth adjustment module determines that the integrity of the collected network traffic data meets the requirements when the packet loss rate of the network traffic data is less than or equal to a preset first packet loss rate. The depth adjustment module determines that the integrity of the network traffic data collection does not meet the requirements when the packet loss rate of the network traffic data is greater than the preset first packet loss rate.

[0048] Specifically, the depth adjustment module increases the elastic resolution coefficient of the network traffic data parsing depth in response to a packet loss rate of network traffic data that is greater than a preset first packet loss rate and less than or equal to a preset second packet loss rate. The deep adjustment module responds to the network traffic data packet loss rate being greater than the preset second packet loss rate, initially determining that the real-time performance of the deep learning model's early warning does not meet the requirements, and then determines whether the real-time performance of the deep learning model's early warning meets the requirements based on the generation delay of the early warning information.

[0049] Understandably, in an AI-based real-time network traffic detection system, a preset first packet loss rate and a preset second packet loss rate are used to characterize whether the integrity of the collected network traffic data meets the requirements. The core logic is to convert the integrity of the collected network traffic data into a quantifiable packet loss rate for judgment. The preset first packet loss rate serves as the dividing line for determining whether the integrity of the collected network traffic data meets the requirements, and the preset second packet loss rate serves as the dividing line between the two reasons that cause the integrity of the collected network traffic data to fail to meet the requirements. The preset first packet loss rate and the preset second packet loss rate can be set according to actual operating conditions. The preset first packet loss rate and the preset second packet loss rate aim to ensure the stability and practicality of real-time detection. Optionally, the preset first packet loss rate and the preset second packet loss rate are determined through a limited number of experiments by evaluating the effect of different packet loss rates on real-time detection. The determined preset first packet loss rate and the preset second packet loss rate should satisfy the condition that they are neither too small nor cause excessive interference to the real-time detection process. For example, the preset first packet loss rate is generally selected in the range of [1%, 3%], and the preset second packet loss rate is generally selected in the range of [5%, 8%].

[0050] Preferably, the first packet loss rate is 2% in a preferred embodiment, and the second packet loss rate is 6% in a preferred embodiment.

[0051] Specifically, the packet loss rate of network traffic data is the ratio of the number of data packets lost per unit time to the total number of data packets per unit time.

[0052] Specifically, the increase in the elastic resolution coefficient of the network traffic data parsing depth is determined by the difference between the packet loss rate of the network traffic data and the preset first packet loss rate.

[0053] Specifically, when the difference between the packet loss rate of network traffic data and the preset first packet loss rate is within 2%, the elastic resolution coefficient of the network traffic data parsing depth increases to 1.5 times the original value. When the difference between the packet loss rate of network traffic data and the preset first packet loss rate exceeds 2%, in addition to increasing to 1.5 times the original value, for every 1% exceeding 2%, the elastic resolution coefficient of the network traffic data parsing depth increases by 0.2%. For example, if the difference between the packet loss rate of network traffic data and the preset first packet loss rate is 5%, and the current elastic resolution coefficient of the network traffic data parsing depth is 3%, the increased elastic resolution coefficient of the network traffic data parsing depth is 3×1.5+0.2×3=5.1%.

[0054] In implementation, the system of the present invention adjusts the elastic resolution coefficient of the network traffic data parsing depth by setting a preset first packet loss rate and a preset second packet loss rate. Due to uneven task allocation among collection nodes, some collection nodes are overloaded, causing data packet queuing overflow and resulting in data packet loss. By increasing the elastic resolution coefficient of the network traffic data parsing depth, parsing resources can be accurately allocated according to the differences in node load and traffic importance, balancing the processing pressure of each collection node, improving the overall processing throughput, alleviating data packet queuing overflow, reducing packet loss and ensuring collection integrity, and further improving the stability of real-time detection.

[0055] Specifically, the granularity adjustment module, in response to a network traffic data packet loss rate greater than a preset second packet loss rate, determines whether the real-time performance of the deep learning model's early warning meets the requirements based on the generation delay of the early warning information.

[0056] Specifically, the granularity adjustment module determines that the real-time performance of the early warning model meets the requirements when the generation delay of the early warning information is less than or equal to the preset delay time. The granularity adjustment module responds when the generation delay of the warning information exceeds the preset delay duration, determines that the real-time performance of the warning from the deep learning model does not meet the requirements, and increases the granularity of the dynamic response triggered by the traffic features.

[0057] Understandably, in an AI-based real-time abnormal network traffic detection system, using a preset delay duration to characterize whether the real-time performance of the deep learning model's warnings meets the requirements involves converting the real-time performance of the deep learning model's warnings into a quantifiable delay duration for judgment. The preset delay duration serves as the dividing line for determining whether the real-time performance of the deep learning model's warnings meets the requirements. The preset delay duration can be set according to actual operating conditions. The preset delay duration aims to ensure the stability and practicality of real-time detection. Optionally, the preset delay duration is determined through a limited number of experiments by evaluating the effect of different delay durations on real-time detection. The determined preset delay duration should satisfy the condition that it is neither too small nor causes excessive interference to the real-time detection process. For example, the preset delay duration is generally selected in the range of [1s, 5s].

[0058] Preferably, the preset delay duration is 3 seconds.

[0059] Specifically, the generation delay of the warning information is the difference between the actual generation time of the warning information and the generation time of the warning information target.

[0060] Specifically, the increase in the dynamic response granularity triggered by the traffic characteristics is determined by the difference between the generation delay of the warning information and the preset delay.

[0061] Specifically, when the difference between the generation delay of the warning information and the preset delay is within 2 seconds, the dynamic response granularity of the traffic feature is increased to 1.2 times the original value. When the difference between the generation delay of the warning information and the preset delay exceeds 2 seconds, in addition to increasing to 1.2 times the original value, the dynamic response granularity of the traffic feature is increased by 1% for every 1 second exceeding the original value. For example, if the difference between the generation delay of the warning information and the preset delay is 4 seconds, the current dynamic response granularity of the traffic feature is 5%, and the increased dynamic response granularity of the traffic feature is 5×1.2+1×2=8%.

[0062] In practice, the system described in this invention adjusts the granularity of the dynamic response triggered by traffic features by setting a preset delay duration. Due to the accumulation of real-time traffic data to be analyzed, the transmission and generation delay of early warning messages increases, the analysis and scheduling efficiency is low, and the early warning queue is continuously congested, making it impossible to detect and deal with real network attacks in a timely manner. By increasing the granularity of the dynamic response triggered by traffic features, the feature output rhythm can be controlled more finely according to the node load, reducing data accumulation waiting and cache retention, accelerating feature flow and model inference efficiency, alleviating data backlog, shortening the early warning generation time, and further improving the stability of real-time detection.

[0063] Please continue reading. Figure 4 The diagram shown is an overall flowchart of the AI-based real-time detection method for abnormal network traffic according to an embodiment of the present invention.

[0064] This invention provides an AI-based real-time detection method for abnormal network traffic, comprising: Step S1: Real-time collection of network traffic data, and sequential cleaning, desensitization, structuring and feature extraction of the network traffic data to obtain traffic features; Step S2: Train the initial model based on the traffic characteristics to obtain a deep learning model, analyze the network traffic data based on the deep learning model to obtain analysis results, and issue a warning message when the analysis results are abnormal. Step S3: Obtain the abnormal proportion of network traffic data and the false alarm rate of early warning information to determine the quality characterization value of the analysis results, and determine whether the executability of the analysis results meets the requirements based on the quality characterization value of the analysis results; Step S4: If the executability of the analysis results does not meet the requirements, the packet loss rate of the network traffic data is obtained to determine whether the integrity of the network traffic data collection meets the requirements. Step S5: If the integrity of the network traffic data collection does not meet the requirements, determine whether it is necessary to increase the elastic resolution coefficient of the network traffic data parsing depth. Step S6: If it is not necessary to increase the elastic resolution coefficient of the network traffic data parsing depth, then the generation delay time of the early warning information determines the triggering dynamic response granularity of the traffic characteristics.

[0065] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An AI-based real-time abnormal network traffic detection system, characterized in that, include: The data processing module includes a data acquisition unit for real-time acquisition of network traffic data and a preprocessing unit connected to the data acquisition unit for sequentially cleaning, desensitizing, structuring, and feature extraction of the network traffic data to obtain traffic features. The traffic detection module, which is connected to the data processing module, includes a model training unit for training an initial model based on the traffic characteristics to obtain a deep learning model, an analysis unit connected to the model training unit for analyzing network traffic data based on the deep learning model to obtain analysis results, and an early warning unit connected to the analysis unit for issuing early warning information when the analysis results are abnormal. An executability determination module, which is connected to the data processing module, is used to determine whether the executability of the analysis results does not meet the requirements based on the quality characterization value of the analysis results determined by the abnormal proportion of network traffic data and the false alarm rate of early warning information. A depth adjustment module, which is connected to the executability determination module, is used to determine the elastic resolution coefficient of the network traffic data parsing depth based on the packet loss rate of the network traffic data when the executability of the analysis results does not meet the requirements. The granularity adjustment module, which is connected to the depth adjustment module, is used to determine the triggering dynamic response granularity of the traffic characteristics based on the generation delay of the early warning information.

2. The AI-based real-time abnormal network traffic detection system according to claim 1, characterized in that, The executability determination module responds to the ratio of the abnormal proportion of network traffic data to the false alarm rate of early warning information to determine the quality characterization value of the analysis results, thereby determining whether the executability of the analysis results meets the requirements.

3. The AI-based real-time abnormal network traffic detection system according to claim 2, characterized in that, The executability determination module determines that the executability of the analysis result meets the requirements when the quality characterization value of the analysis result is less than or equal to the preset characterization value. The executability determination module determines that the executability of the analysis result does not meet the requirements if the quality characterization value of the analysis result is greater than the preset characterization value.

4. The AI-based real-time abnormal network traffic detection system according to claim 3, characterized in that, The depth adjustment module, in response to the condition that the executability of the analysis results does not meet the requirements, determines whether the integrity of the network traffic data collection meets the requirements based on the packet loss rate of the network traffic data.

5. The AI-based real-time abnormal network traffic detection system according to claim 4, characterized in that, The depth adjustment module determines that the integrity of the network traffic data collection meets the requirements when the packet loss rate of the network traffic data is less than or equal to a preset first packet loss rate. The depth adjustment module determines that the integrity of the network traffic data collection does not meet the requirements when the packet loss rate of the network traffic data is greater than the preset first packet loss rate.

6. The AI-based real-time abnormal network traffic detection system according to claim 5, characterized in that, The depth adjustment module increases the elastic resolution coefficient of the network traffic data parsing depth in response to the network traffic data packet loss rate being greater than a preset first packet loss rate and less than or equal to a preset second packet loss rate. The deep adjustment module responds to the network traffic data packet loss rate being greater than the preset second packet loss rate, initially determining that the real-time performance of the deep learning model's early warning does not meet the requirements, and then determines whether the real-time performance of the deep learning model's early warning meets the requirements based on the generation delay of the early warning information.

7. The AI-based real-time abnormal network traffic detection system according to claim 6, characterized in that, The increase in the elastic resolution coefficient of the network traffic data parsing depth is determined by the difference between the packet loss rate of the network traffic data and the preset first packet loss rate.

8. The AI-based real-time abnormal network traffic detection system according to claim 7, characterized in that, The granularity adjustment module, in response to a network traffic data packet loss rate greater than a preset second packet loss rate, determines whether the real-time performance of the deep learning model's early warning meets the requirements based on the generation delay of the early warning information.

9. The AI-based real-time abnormal network traffic detection system according to claim 8, characterized in that, The granularity adjustment module determines that the real-time performance of the early warning model meets the requirements if the generation delay of the early warning information is less than or equal to the preset delay. The granularity adjustment module responds when the generation delay of the warning information exceeds the preset delay time, determines that the real-time performance of the warning from the deep learning model does not meet the requirements, and increases the granularity of the dynamic response triggered by the traffic feature. The increase in the dynamic response granularity triggered by the traffic characteristics is determined by the difference between the generation delay of the warning information and the preset delay.

10. A detection method applied to the AI-based real-time abnormal network traffic detection system according to any one of claims 1-9, characterized in that, include: Real-time network traffic data is collected, and the network traffic data is sequentially cleaned, desensitized, structured, and feature extracted to obtain traffic features; The initial model is trained based on the traffic characteristics to obtain a deep learning model. The network traffic data is analyzed based on the deep learning model to obtain analysis results, and an early warning message is issued when the analysis results are abnormal. The abnormal proportion of network traffic data and the false alarm rate of early warning information are obtained to determine the quality characterization value of the analysis results, and the executability of the analysis results is determined based on the quality characterization value of the analysis results. If the executability of the analysis results does not meet the requirements, the packet loss rate of the network traffic data is used to determine whether the integrity of the network traffic data collection meets the requirements. If the integrity of the collected network traffic data does not meet the requirements, then determine whether it is necessary to increase the elastic resolution coefficient of the network traffic data parsing depth. If it is not necessary to increase the elastic resolution coefficient of network traffic data parsing depth, then the generation delay of obtaining early warning information determines the granularity of the dynamic response triggered by traffic characteristics.

Images