Network traffic anomaly real-time detection method based on deep learning

Abstract

The invention relates to the technical field of network flow detection, in particular to a real-time network flow anomaly detection method based on deep learning, and the system comprises the following steps: S1, carrying out the real-time collection and preprocessing of multi-modal data; s2, performing dynamic feature engineering and sliding window statistics; s3, carrying out online adaptive threshold initialization; s4, multi-modal deep learning model reasoning is carried out; s5, updating the adaptive threshold in real time; s6, abnormal decision making and confidence coefficient calibration; s7, generating interpretability analysis; and S8, performing real-time feedback and online learning. According to the scheme, the capability of detecting hidden and complex attacks is remarkably improved through multi-modal data fusion and dynamic feature engineering, network traffic, system logs, user behavior data and external threat intelligence are synchronously collected, and traffic statistical features, time sequence change features, frequency domain features and distribution features are extracted in real time by using a sliding window mechanism.

Description

Technical Field

[0001] This invention relates to the field of network traffic detection technology, and in particular to a real-time network traffic anomaly detection method based on deep learning. Background Technology

[0002] With the rapid development of internet technology and the deepening of digital transformation, the internet has become an indispensable infrastructure of modern society. However, the ever-expanding scale and increasing complexity of networks have made cybersecurity issues increasingly prominent. Cyberattack methods are constantly evolving, from traditional viruses and worms to advanced persistent threats and distributed denial-of-service attacks. These attacks not only pose a serious threat to personal privacy and corporate data, but may also endanger national security and social stability.

[0003] Traditional network traffic anomaly detection systems are mostly based on rule matching, statistical analysis, or machine learning methods. Rule matching methods rely on predefined attack signature databases; while offering high accuracy, they cannot handle unknown attack types and require frequent rule database updates to maintain effectiveness. Statistical analysis methods identify abnormal traffic deviating from normal patterns by establishing statistical models of traffic behavior, but their performance is highly dependent on the completeness and representativeness of historical data and they are poorly adaptable to dynamically changing network environments. According to the patent announcement number "CN118353690A," a multimodal fusion feature-based anomaly network traffic detection system and method are disclosed. This system acquires data transmission volume, packet quantity, transmission rate, source IP address, destination IP address, and protocol type from network traffic data at multiple predetermined time points within a predetermined time period in real time. It then uses deep learning technology to analyze and process the network traffic data to determine whether network traffic anomalies exist at the current time point. This approach combines multiple types of feature data to provide more comprehensive and multi-dimensional information, enabling the system to better understand the characteristics and behavior of network traffic and respond quickly to network traffic anomalies, thereby strengthening network security protection and protecting the network from various potential threats and attacks.

[0004] In recent years, deep learning technology has achieved remarkable success in fields such as image recognition and natural language processing. Its powerful feature learning and nonlinear modeling capabilities have also brought new opportunities for network traffic anomaly detection.

[0005] Currently, deep learning models have improved in detection performance, but the following problems still exist when applied on a large scale: 1. Threshold setting is a common bottleneck in traditional anomaly detection systems, with most systems relying on fixed thresholds to determine whether traffic is abnormal. 2. The limited availability of data sources restricts the comprehensiveness and accuracy of the detection system. Existing methods mostly focus on analyzing network traffic data, but complex network attacks often involve multi-stage and multi-dimensional methods, and traffic data alone may not provide sufficient contextual information. 3. The black-box nature of deep learning models makes their decision-making process difficult to explain. Security analysts need to understand why the model marks certain traffic as abnormal in order to verify the results, adjust strategies, or conduct further investigations. The lack of interpretability not only reduces users' trust in the system, but may also lead to misjudgments and response delays. Summary of the Invention

[0006] The purpose of this invention is to address the aforementioned shortcomings in the existing technology by proposing a real-time network traffic anomaly detection method based on deep learning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a real-time network traffic anomaly detection method based on deep learning, wherein the system flow for the real-time network traffic anomaly detection is as follows: S1. Real-time acquisition and preprocessing of multimodal data Simultaneously collect network traffic data, system logs such as Syslog and error logs, user behavior data such as login frequency and operation sequence, and access external threat intelligence streams; S2, Dynamic Feature Engineering and Sliding Window Statistics Traffic statistics features are extracted using a sliding window as a unit, and multimodal correlation features are calculated simultaneously. S3, Online Adaptive Threshold Initialization Based on historical data from the initial time window, a baseline for traffic characteristics is calculated. The mean plus or minus three standard deviations is used as the initial threshold value, and an online update thread is started to achieve dynamic adjustment. S4, Multimodal Deep Learning Model Inference Using a multi-input CNN architecture, the final output is an anomaly probability score; S5, Adaptive threshold real-time update The threshold is dynamically adjusted using the recursive least squares method. S6. Abnormal Decision Making and Confidence Calibration Compare the model output probability with the dynamic threshold: if the probability is higher than the current threshold, an anomaly alarm is triggered, and an integrated confidence assessment mechanism is used to automatically initiate a manual review process for low-confidence samples; S7, Interpretability Analysis Generation The built-in attention mechanism visualization function can highlight key features that affect decision-making, and call the SHAP interpreter for abnormal events to generate a ranking of feature contribution. S8, Real-time Feedback and Online Learning The feedback from security analysts to the alerts is fed into the online learning module in real time. The model weights are fine-tuned through incremental learning technology, and the adaptive threshold module dynamically adjusts the sensitivity based on the feedback.

[0008] Preferably, the method of jointly modeling based on multi-source data in S1 and S4 enhances the detection capability of covert attacks, as detailed below: M1 introduces an adaptive streaming data integration framework, supports dynamic expansion of multi-source input, and deploys edge computing nodes to achieve near-data source preprocessing, reducing the load on the center. M2. The data standardization process is upgraded to an online learning mode: numerical data uses dynamic Z-score normalization and automatically handles distribution offset; text logs use a lightweight BERT variant for real-time sentence embedding; and time series data achieves multi-granularity alignment through dynamic time warping and adaptive resampling. A data quality awareness module is embedded to automatically detect missing or abnormal inputs and trigger the repair process. M3, the architecture is upgraded to a hybrid model of graph neural network and Transformer: traffic, log and behavioral data are constructed into a heterogeneous graph structure, the relationship graph convolutional encoding is used to encode cross-entity dependencies, and the temporal Transformer is introduced to capture long-period patterns to replace the traditional LSTM; M4 designs a hierarchical cross-modal attention mechanism to enhance representation consistency through intermodal interaction in the early stages of feature extraction. During the inference stage, it integrates a continuous learning strategy, adapts to changes in data distribution through incremental model fine-tuning, and introduces an uncertainty quantification module to output anomaly probability and confidence assessment, thereby improving interpretability and decision reliability.

[0009] Preferably, in M2, the passive response is further upgraded to active prediction, and the modularization is upgraded to a systematic approach, as follows: M2.1, intelligent sensing and predictive intervention, transforms passive detection into proactive prediction, nipping data quality problems in the bud; M2.2, collaborative adaptive and federated learning, breaks down processing silos, enabling data processing workflows of different modalities to collaborate with each other, and protecting data privacy; M2.3, closed-loop evaluation and process self-optimization enable the entire system to have the ability to self-evaluate and optimize, forming a continuously improving reinforcement learning loop.

[0010] Preferably, the core of the sliding window in S2 is to use the current time as a reference to extract a continuous segment of data, such as 5 minutes or 100 data packets, and calculate statistical characteristics based on the data within the window: S2.1.1 Basic statistics: the sum, mean, variance, maximum and minimum values ​​of flow within the window, reflecting the central tendency and dispersion of the flow; S2.1.2, Temporal variation characteristics: First-order difference - instantaneous rate of change, moving standard deviation - fluctuation stability, and linear trend slope within the window, used to identify sudden increases or decreases in flow; S2.1.3, Frequency Domain Characteristics: The dominant frequency components are extracted through Fourier transform to detect periodic flow patterns; S2.1.4 Distribution characteristics: Calculate the skewness and kurtosis of the flow values ​​within the window to determine whether the distribution pattern deviates from normality.

[0011] Preferably, the multimodal association features in S2 aim to mine the interaction relationships between different data sources, specifically: S2.2.1 Cross-modal statistical correlation: Calculate the correlation coefficient between the traffic value within the window and the number of user requests within the corresponding time period, or the covariance with the server load, to reveal the impact of external factors on traffic; S2.2.2 Event Triggering Features: If a specific event occurs within the window, a binary flag feature is generated and combined with traffic statistics to construct a composite feature; S2.2.3 Spatial-Temporal Correlation: In IoT scenarios, the sum of window traffic of multiple devices under the same gateway is aggregated, and the traffic similarity between devices is calculated to identify coordination anomalies.

[0012] Preferably, the multi-input CNN architecture in S4 includes: S4.1, Branch 1 uses 1D-CNN to process the temporal features of traffic; Traffic time-series feature processing: Suppose the input time series data is a vector sequence of length L. Features are extracted through one-dimensional convolution operations, and the convolution kernel... The features within a local window are processed by sliding computation, where d is the input dimension and w is the kernel width. The output of the k-th feature mapped at position i is: in: It is a non-linear activation function. The bias term is used, and the final output is a feature tensor composed of all feature maps. S4.2, Branch 2 uses LSTM to process user behavior sequences; User behavior sequence processing: Let the input behavior sequence be The hidden state is updated step-by-step by the loop unit. At time step t, the unit receives the current input. And the previous hidden state Calculate candidate states and gating signals (input gate) Forgotten Gate Output gate ): in: and This is the weight matrix. This indicates element-wise multiplication, and the final output is the hidden state at the last time step or all states of the sequence. S4.3 and Branch 3 use text CNN to process system log embedding vectors.

[0013] System log embedding vector processing: Let the input be the word embedding matrix E (n is the number of words, m is the embedding dimension), local features are extracted through multi-scale convolutional kernels. (r is the kernel height) Sliding calculation on the embedding matrix generates feature maps: Pooling operations, such as max pooling, are performed on the convolution results at different scales to obtain a fixed-dimensional log feature representation.

[0014] Preferably, the threshold adjustment in S5 needs to update the threshold boundary based on the recent traffic characteristic distribution within the window, introduce a decay factor, and give new data higher weight in order to flexibly adapt to sudden traffic patterns. The recursive method for dynamic thresholds is as follows: S5.1 Defining variables: Let t be the current time; For the traffic observation value at time t, such as the number of requests or the traffic volume; The threshold parameter at time t can be a vector or a scalar, such as representing the mean and variance of the threshold. This is the error covariance matrix, used in PLS to adjust the update step size; λ is the decay factor, 0≤λ≤1, usually taken as λ≈0.95-0.99, to give new data a higher weight; S5.2, Recursive Update Steps: For each new data point : Calculate the prior error: in, It is a feature vector, which can be set as follows: Or simply take 1, depending on the model complexity; Update the gain vector : Update parameter estimates: Update the error covariance matrix: S5.3 Extracting dynamic thresholds: Based on the estimated parameters The current threshold boundary is calculated as follows: like = Let be the mean and variance, respectively. Then the threshold can be set as: Where k is a constant, such as 2 or 3, which controls the sensitivity.

[0015] Preferably, in S8, a real-time feedback closed loop based on streaming processing is constructed to achieve efficient and lossless flow and instant learning of feedback data, laying a solid real-time foundation for the system.

[0016] Preferably, in S8, the development of context-aware adaptive threshold and meta-learning adjustment makes the threshold adjustment no longer a simple numerical fluctuation, but an intelligent behavior with scene understanding capabilities.

[0017] Preferably, in S8, an enhanced framework for predictive and collaborative defense is created, upgrading the system from a detection-feedback loop to an enhanced framework for prediction-recommendation-collaboration.

[0018] The design scheme proposed in this invention has the following beneficial effects during application: 1. This solution significantly improves the detection capability of covert and complex attacks through multimodal data fusion and dynamic feature engineering. It simultaneously collects network traffic, system logs, user behavior data and external threat intelligence, and uses a sliding window mechanism to extract traffic statistical features, time-series change features, frequency domain features and distribution features in real time. At the same time, by calculating cross-modal statistical correlation, event trigger features and spatial-temporal correlation features, it deeply explores the interaction relationship between different data sources. The multi-source data joint modeling method can capture attack traces more comprehensively, especially identifying low-slow penetration or coordinated attack behaviors that are difficult to detect from a single data source. 2. As described in 1, the advanced online learning and adaptive optimization mechanism achieves continuous performance improvement and operational efficiency assurance. The built-in incremental learning engine can receive feedback from security analysts in real time and fine-tune the model weights accordingly, enabling the model to quickly adapt to new attacks or changes in data distribution. At the same time, the adaptive threshold module dynamically adjusts the detection sensitivity through recursive least squares and meta-learning strategies, reducing false alarms and improving response accuracy. The entire preprocessing and model update process adopts a microservice and streaming architecture, supporting high throughput and low latency processing requirements. 3. As described in section 2, the efficiency and credibility of security analysis are greatly improved through interpretability enhancement and decision support mechanisms. The system integrates an attention mechanism and a SHAP interpreter, which can automatically highlight the key features that have the greatest impact on anomaly decisions and generate a feature contribution ranking. This enables security analysts to quickly understand the basis behind alerts, accelerate incident investigation and root cause analysis. At the same time, the system has a built-in confidence assessment function that automatically initiates a manual review process for low-confidence samples, avoiding blind automated decision-making. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the network traffic anomaly real-time detection system of the present invention; Figure 2 This is a schematic diagram of multi-source data joint modeling according to the present invention; Figure 3 This is a schematic diagram of a further optimization scheme based on S2 according to the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Example 1 Reference Figures 1-3 A method for real-time detection of network traffic anomalies based on deep learning is described below. The system process for real-time detection of network traffic anomalies is as follows: S1. Real-time acquisition and preprocessing of multimodal data Simultaneously collect network traffic data, system logs such as Syslog and error logs, user behavior data such as login frequency and operation sequence, and access external threat intelligence streams; S2, Dynamic Feature Engineering and Sliding Window Statistics Traffic statistics features are extracted using a sliding window as a unit, and multimodal correlation features are calculated simultaneously. S3, Online Adaptive Threshold Initialization Based on historical data from the initial time window, a baseline for traffic characteristics is calculated. The mean plus or minus three standard deviations is used as the initial threshold value, and an online update thread is started to achieve dynamic adjustment. S4, Multimodal Deep Learning Model Inference Using a multi-input CNN architecture, the final output is an anomaly probability score; S5, Adaptive threshold real-time update The threshold is dynamically adjusted using the recursive least squares method. S6. Abnormal Decision Making and Confidence Calibration Compare the model output probability with the dynamic threshold: if the probability is higher than the current threshold, an anomaly alarm is triggered, and an integrated confidence assessment mechanism is used to automatically initiate a manual review process for low-confidence samples; S7, Interpretability Analysis Generation The built-in attention mechanism visualization function can highlight key features that affect decision-making, and call the SHAP interpreter for abnormal events to generate a ranking of feature contribution. S8, Real-time Feedback and Online Learning The feedback from security analysts to the alerts is fed into the online learning module in real time. The model weights are fine-tuned through incremental learning technology, and the adaptive threshold module dynamically adjusts the sensitivity based on the feedback.

[0022] Based on the joint modeling of multi-source data in S1 and S4, the detection capability of covert attacks is enhanced, as follows: M1 introduces an adaptive streaming data integration framework, supports dynamic expansion of multi-source input, and deploys edge computing nodes to achieve near-data source preprocessing, reducing the load on the center. M2. The data standardization process is upgraded to an online learning mode: numerical data uses dynamic Z-score normalization and automatically handles distribution offset; text logs use a lightweight BERT variant for real-time sentence embedding; and time series data achieves multi-granularity alignment through dynamic time warping and adaptive resampling. A data quality awareness module is embedded to automatically detect missing or abnormal inputs and trigger the repair process. M3, the architecture is upgraded to a hybrid model of graph neural network and Transformer: traffic, log and behavioral data are constructed into a heterogeneous graph structure, the relationship graph convolutional encoding is used to encode cross-entity dependencies, and the temporal Transformer is introduced to capture long-period patterns to replace the traditional LSTM; M4 designs a hierarchical cross-modal attention mechanism to enhance representation consistency through intermodal interaction in the early stages of feature extraction. During the inference stage, it integrates a continuous learning strategy, adapts to changes in data distribution through incremental model fine-tuning, and introduces an uncertainty quantification module to output anomaly probability and confidence assessment, thereby improving interpretability and decision reliability.

[0023] In M2, passive response is further upgraded to proactive prediction, and modularization is upgraded to systematization, as follows: M2.1, intelligent sensing and predictive intervention, transforms passive detection into proactive prediction, nipping data quality problems in the bud; Numerical data: Beyond dynamic Z-score, the system introduces online distribution change warnings. It not only calculates the current mean and standard deviation, but also continuously monitors the shape of the data distribution, such as changes in skewness and kurtosis. When potential concept drift is detected, it provides early warnings or gradually and smoothly switches to new normalization parameters to avoid drastic changes. Text and Time Series Data: Embedded Metadata Quality Scanning. Before processing text logs and time series data, lightweight analysis is performed on their meta-features, such as text length distribution, symbol ratio, and time series frequency stability, to predict potential anomalies in batch data, such as a sudden influx of a large number of ultra-short texts or a large number of duplicate values ​​in time series, and adaptation strategies are pre-triggered, such as adjusting the pooling strategy of the embedding model or the resampling granularity. M2.2, collaborative adaptive and federated learning, breaks down processing silos, enabling data processing workflows of different modalities to collaborate with each other, and protecting data privacy; Cross-modal quality correlation: Establish a unified quality-aware signal bus. When a module detects an anomaly, such as a large number of anomalies reported by a numerical sensor during a certain period, the signal will be broadcast to the text and time-series processing modules in real time. These modules can adjust their own strategies accordingly, such as assigning lower confidence to the log embedding results within that period, or focusing on reviewing the time-series alignment of that segment, to achieve collaborative judgment of multi-modal quality. Federated Embedding Updates: For the lightweight BERT model, a federated learning mechanism is used for online updates. Each data source fine-tunes the model locally using the new data, and only the model update gradient is encrypted and uploaded to the central server for aggregation. The original text data never leaves the local machine. This not only enables the model to adapt to the latest language features, but also strictly protects user privacy and log security. M2.3, closed-loop evaluation and process self-optimization enable the entire system to have the ability to self-evaluate and optimize, forming a continuous improvement reinforcement learning loop; Embedded downstream feedback signals: Introduce downstream model performance feedback as the final evaluation criterion. Input the preprocessed data into the online machine learning model for training or inference, and monitor its performance indicators in real time, such as accuracy and AUC value. If the performance degrades, the system can trace back to the preprocessing stage and automatically adjust relevant parameters, such as the window size of dynamic Z-score, the curved window limit of DTW, and the granularity of resampling. Automated process tuning: The selection of parameters and algorithms for the entire preprocessing process is modeled as a hyperparameter optimization problem. A lightweight meta-controller, such as one based on Bayesian optimization, continuously explores the impact of different parameter combinations on the performance of downstream tasks and automatically finds and switches to the current optimal preprocessing pipeline configuration.

[0024] The core of the sliding window in S2 is to use the current time as a reference to extract a continuous segment of data, such as 5 minutes or 100 data packets, and calculate statistical characteristics based on the data within the window: S2.1.1 Basic statistics: the sum, mean, variance, maximum and minimum values ​​of flow within the window, reflecting the central tendency and dispersion of the flow; S2.1.2, Temporal variation characteristics: First-order difference - instantaneous rate of change, moving standard deviation - fluctuation stability, and linear trend slope within the window, used to identify sudden increases or decreases in flow; S2.1.3, Frequency Domain Characteristics: The dominant frequency components are extracted through Fourier transform to detect periodic flow patterns; S2.1.4 Distribution characteristics: Calculate the skewness and kurtosis of the flow values ​​within the window to determine whether the distribution pattern deviates from normality.

[0025] The multimodal association features in S2 aim to uncover the interaction relationships between different data sources, specifically: S2.2.1 Cross-modal statistical correlation: Calculate the correlation coefficient between the traffic value within the window and the number of user requests within the corresponding time period, or the covariance with the server load, to reveal the impact of external factors on traffic; S2.2.2 Event Triggering Features: If a specific event occurs within the window, a binary flag feature is generated and combined with traffic statistics to construct a composite feature; S2.2.3 Spatial-Temporal Correlation: In IoT scenarios, the sum of window traffic of multiple devices under the same gateway is aggregated, and the traffic similarity between devices is calculated to identify coordination anomalies.

[0026] The multi-input CNN architecture in S4 includes: S4.1, Branch 1 uses 1D-CNN to process the temporal features of traffic; Traffic time-series feature processing: Suppose the input time series data is a vector sequence of length L. Features are extracted through one-dimensional convolution operations, and the convolution kernel... The features within a local window are processed by sliding computation, where d is the input dimension and w is the kernel width. The output of the k-th feature mapped to position i is: in: It is a non-linear activation function. The bias term is used, and the final output is a feature tensor composed of all feature maps. S4.2, Branch 2 uses LSTM to process user behavior sequences; User behavior sequence processing: Let the input behavior sequence be The hidden state is updated step-by-step by the loop unit. At time step t, the unit receives the current input. And the previous hidden state Calculate candidate states and gating signals (input gate) Forgotten Gate Output gate ): in: and This is the weight matrix. This indicates element-wise multiplication, and the final output is either the hidden state at the last time step or all states in the sequence.

[0027] S4.3 and Branch 3 use text CNN to process system log embedding vectors.

[0028] System log embedding vector processing: Let the input be the word embedding matrix E (n is the number of words, m is the embedding dimension), local features are extracted through multi-scale convolutional kernels. (r is the kernel height) Sliding calculation on the embedding matrix generates feature maps: Pooling operations, such as max pooling, are performed on the convolution results at different scales to obtain a fixed-dimensional log feature representation.

[0029] In S5, threshold adjustment requires updating the threshold boundary based on the recent traffic characteristics distribution within the window, introducing a decay factor, and assigning higher weight to new data in order to flexibly adapt to sudden traffic patterns. The recursive method for dynamic thresholds is as follows: S5.1 Defining variables: Let t be the current time; For the traffic observation value at time t, such as the number of requests or the traffic volume; The threshold parameter at time t can be a vector or a scalar, such as representing the mean and variance of the threshold. This is the error covariance matrix, used in PLS to adjust the update step size; λ is the decay factor, 0≤λ≤1, usually taken as λ≈0.95-0.99, to give new data a higher weight; S5.2, Recursive Update Steps: For each new data point : Calculate the prior error: in, It is a feature vector, which can be set as follows: Or simply take 1, depending on the model complexity; Update the gain vector : Update parameter estimates: Update the error covariance matrix: S5.3 Extracting dynamic thresholds: Based on the estimated parameters The current threshold boundary is calculated as follows: like = Let be the mean and variance, respectively. Then the threshold can be set as: Where k is a constant, such as 2 or 3, which controls the sensitivity.

[0030] Example 2 Based on S8, a real-time feedback closed loop based on streaming processing is constructed to achieve efficient and lossless flow and instant learning of feedback data, laying a solid real-time foundation for the system. Specifically: S8.1.1 Lightweight feedback event standardization: Design a minimalist standardized feedback event structure; S8.1.2 High-throughput feedback message queue: Introduces a dual-channel feedback message queue. One channel is used for real-time model updates, transmitting high-confidence feedback, while the other channel is used for long-term model evaluation and retraining, transmitting all feedback and rich contextual information such as analyst survey notes. S8.1.3 Microservice architecture of incremental learning engine: The incremental learning module is encapsulated as an independent, horizontally scalable microservice. Each time a feedback event is received, the service triggers a very small batch of model weight updates.

[0031] Based on S8, we developed context-aware adaptive thresholding and meta-learning adjustment, so that threshold adjustment is no longer a simple numerical fluctuation, but an intelligent behavior with scene understanding capabilities. Specifically: S8.2.1 Dynamic feature weight threshold: Abandoning the global unified threshold, dynamic weights are assigned to different features and independent thresholds are set. The system automatically adjusts the feature weights based on feedback. If a feature appears frequently in false alarms, its weight is reduced; if it is significant in real attacks, its weight is increased. The threshold is then adaptively adjusted according to the new weight distribution. The dynamic update of feature weights is as follows: set up: ; For learning rate, ; ; ; Weight update rules: in: It is the total number of false alarms at time t; The total number of actual attacks at time t; This is the reward coefficient.

[0032] S8.2.2, Meta-learning-based threshold optimization strategy: A lightweight meta-learning model is introduced, whose input is the feedback statistics of recent alarms and whose output is the recommended optimal threshold adjustment strategy. The threshold optimization strategy based on meta-learning is summarized as follows: in: The optimal threshold adjustment strategy; A lightweight meta-learning strategy; This represents the statistical characteristics of recent alarm feedback.

[0033] S8.2.3 Sensitivity adjustment for attack phase perception: The system integrates an attack chain model. When initial signs of intrusion are detected, the sensitivity of subsequent key steps is automatically increased, achieving an active defense posture of "once suspicious, always monitoring". The sensitivity changes dynamically according to the progress of the attack phase.

[0034] Based on S8, an enhanced framework for predictive and collaborative defense is created, upgrading the system from a detection-feedback loop to an enhanced prediction-recommendation-collaboration framework, specifically: S8.3.1 Feedback Prediction and Recommendation System: Train a prediction model to estimate the probability that a new alarm will be confirmed by the analyst as a real attack. For high-probability alarms, the system can automatically increase their priority or directly trigger a predefined containment script. For low-probability but high-uncertainty alarms, the system can recommend relevant investigation contexts to assist analysts in making quick decisions, thereby accelerating the feedback loop. S8.3.2 Cross-model knowledge distillation and sharing: Periodically distill the knowledge of online learning models into a lighter and more stable golden model; S8.3.3 Simulation Adversarial Training Loop: Construct an attack simulation environment, periodically generate simulated attack data using the latest discovered attack techniques and tactics, actively challenge the current detection model with this data, observe its blind spots, and then feed these undetected simulated attacks as negative samples back to the online learning module to achieve proactive, adversarial incremental learning, allowing the model to acquire immunity before real attacks occur.

[0035] Example 3 Application Implementation: Real-time Network Anomaly Detection System for Intelligent Cloud Platform Scenario: A large cloud service provider has deployed a deep learning-based real-time network traffic anomaly detection method to protect its infrastructure security. The system is based on a streaming processing architecture and realizes a closed loop from data collection to proactive defense. First, the system collects network traffic, Syslog logs and user behavior data in real time, and preprocesses them through edge nodes. Numerical traffic metrics are normalized by dynamic Z-score, log data is embedded into vectors through a lightweight BERT model, and time series data is aligned through adaptive resampling. The data quality awareness module monitors distribution offsets in real time, such as sudden changes in traffic kurtosis or abnormal log lengths, and triggers smooth parameter switching. Secondly, the system calculates traffic statistics features and cross-modal correlation features using a 5-minute sliding window, integrates heterogeneous data using a multimodal graph neural network, and captures key dependencies through a hierarchical attention mechanism. The model outputs anomaly probability scores, which are compared with dynamic thresholds updated based on recursive least squares to trigger high-confidence alarms. Finally, the security analyst's feedback flows into the system through a dual-channel message queue. High-confidence feedback triggers the incremental learning microservice, which adjusts the model weights and feature thresholds in real time. The meta-learning module recommends threshold strategies based on feedback history and dynamically adjusts the detection sensitivity in conjunction with the attack chain model. The system also integrates simulation adversarial training and periodically injects simulated attack data to strengthen the model's blind spots. Results and Value: The system achieves real-time detection of DDoS attacks, abnormal logins, and covert penetrations, reducing the false alarm rate by 40%. It also provides readable decision-making support through the SHAP interpreter. The system forms a virtuous cycle of "prediction-detection-feedback-optimization", significantly improving the security resilience of the cloud platform.

[0036] In practice The system achieves comprehensive perception and deep correlation analysis through real-time acquisition of multimodal data and dynamic feature engineering. In the initial stage, the system synchronously collects network traffic data, system logs, and user behavior data, and connects to external threat intelligence streams to form a multi-source data foundation. Subsequently, the system performs dynamic feature extraction in units of sliding windows, not only calculating traffic statistical features, but also deeply mining multimodal correlation features, including cross-modal statistical correlation, event-triggered features, and spatial-temporal correlation. This process ensures that the system can capture network status from multiple dimensions and enhance its ability to identify covert attacks. The system adopts an online adaptive threshold initialization and real-time update mechanism, combined with multimodal deep learning model inference, to achieve intelligent anomaly detection and dynamic adjustment. Based on historical data of the initial time window, the system calculates the traffic feature baseline and uses the mean plus or minus three standard deviations as the initial threshold value. At the same time, an online update thread is started. During the multimodal deep learning model inference stage, the system uses a multi-input CNN architecture to process feature data and output anomaly probability scores. The threshold adjustment adopts the recursive least squares method for dynamic updates to ensure that it adapts to changes in data distribution. During the anomaly decision stage, the system compares the model output probability with the dynamic threshold, triggers alarms, and integrates a confidence assessment mechanism to initiate manual review for low-confidence samples. The system enhances decision-making transparency and continuous learning capabilities through interpretability analysis and real-time feedback loop. It has a built-in attention mechanism to visualize key features and calls the SHAP interpreter to generate feature contribution rankings, providing decision-making basis for security analysts. The real-time feedback loop transmits analyst feedback data through a standardized event structure and a high-throughput message queue, triggering microservice updates of the incremental learning engine to achieve online fine-tuning of model weights. At the same time, the adaptive threshold module dynamically adjusts sensitivity based on feedback and combines meta-learning optimization strategies and attack phase awareness mechanisms to make threshold adjustment intelligent for specific scenarios. The system upgrades from passive detection to proactive defense through predictive intervention, collaborative adaptation, and closed-loop self-optimization. In the data preprocessing stage, the system introduces online distribution change early warning and metadata quality scanning to identify anomalies in numerical, textual, and time-series data in advance and trigger adaptation strategies to avoid quality deterioration. At the same time, the system achieves cross-modal collaboration and privacy protection through a unified quality-aware signal bus and federated embedding updates. Downstream model performance feedback is used to automatically adjust preprocessing parameters, while the meta-controller continuously optimizes the entire pipeline configuration. In addition, the system integrates feedback prediction and recommendation, cross-model knowledge distillation, and simulation adversarial training to form an enhanced framework of prediction-recommendation-collaboration, enabling the system to proactively adapt to threat changes and improve overall resilience.

[0037] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for real-time detection of network traffic anomalies based on deep learning, characterized in that, Includes the following steps: S1. Real-time acquisition and preprocessing of multimodal data, synchronous acquisition of network traffic data, collection of system logs and user behavior data, and access to external threat intelligence streams; S2. Dynamic feature engineering and sliding window statistics: Extract flow statistics features and calculate multimodal correlation features using a sliding window as the unit. S3. Online adaptive threshold initialization: Based on historical data of the initial time window, calculate the traffic characteristic baseline, use the mean plus or minus three times the standard deviation as the initial threshold value, and start the online update thread to achieve dynamic adjustment; S4, Multimodal deep learning model inference, using a multi-input CNN architecture, outputs anomaly probability scores; S5. Adaptive threshold is updated in real time, and the threshold is dynamically adjusted using the recursive least squares method. S6. Abnormal Decision Making and Confidence Calibration; S7: Interpretable analysis generation, built-in attention mechanism visualization function, generating feature contribution ranking; S8 features real-time feedback and online learning, using incremental learning technology to fine-tune model weights, and an adaptive threshold module that dynamically adjusts sensitivity based on feedback.

2. The method for real-time detection of network traffic anomalies based on deep learning according to claim 1, characterized in that: The method of enhancing the detection capability of covert attacks by jointly modeling using multi-source data in S1 and S4 includes the following steps: M1 introduces an adaptive streaming data integration framework and deploys edge computing nodes to achieve near-data source preprocessing; M2. The data standardization process is upgraded to an online learning mode: numerical data is normalized by dynamic Z-score and the distribution offset is automatically handled; text logs are embedded in real-time sentences using a lightweight BERT variant; time series data is aligned to multiple granularities through dynamic time warping and adaptive resampling; and a data quality awareness module is embedded to automatically detect missing or abnormal inputs and trigger the repair process. M3, the architecture is upgraded to a hybrid model of graph neural network and Transformer: traffic, log and behavioral data are constructed into a heterogeneous graph structure, the relationship graph convolutional encoding is used to encode cross-entity dependencies, and the temporal Transformer is introduced to capture long-cycle patterns; M4 designs a hierarchical cross-modal attention mechanism to enhance representation consistency through intermodal interaction in the early stages of feature extraction. It integrates a continuous learning strategy during the inference stage, adapts to changes in data distribution through incremental model fine-tuning, and introduces an uncertainty quantification module to output anomaly probability and confidence assessment.

3. The method for real-time detection of network traffic anomalies based on deep learning according to claim 2, characterized in that: The M2 upgrades passive response to active prediction and from modular to systematic, including the following steps: M2.1, intelligent sensing and predictive intervention, transforming passive detection into proactive prediction; M2.2, collaborative adaptive and federated learning, enables data processing workflows of different modalities to collaborate and protect data privacy; M2.3, closed-loop evaluation and process self-optimization, forming a reinforcement learning loop for continuous improvement.

4. The method for real-time detection of network traffic anomalies based on deep learning according to claim 3, characterized in that: The sliding window in S2 uses the current time as a reference to extract a continuous segment of data from the past, and calculates statistical features based on the data within the window: S2.1.1 Basic statistics: the sum, mean, variance, maximum and minimum values ​​of flow within the window, reflecting the central tendency and dispersion of the flow; S2.1.2, Temporal variation characteristics: First-order difference - instantaneous rate of change, moving standard deviation - fluctuation stability, and linear trend slope within the window, used to identify sudden increases or decreases in flow; S2.1.3, Frequency Domain Characteristics: The dominant frequency components are extracted through Fourier transform to detect periodic flow patterns; S2.1.4 Distribution characteristics: Calculate the skewness and kurtosis of the flow values ​​within the window to determine whether the distribution pattern deviates from normality.

5. The method for real-time detection of network traffic anomalies based on deep learning according to claim 4, characterized in that: The multimodal association features in S2 aim to uncover the interaction relationships between different data sources, including: S2.2.1 Cross-modal statistical correlation: Calculate the correlation coefficient between the traffic value within the window and the number of user requests within the corresponding time period, or the covariance with the server load, and analyze the impact of external factors on traffic; S2.2.2 Event Triggering Features: If a specific event occurs within the window, a binary flag feature is generated and combined with traffic statistics to construct a composite feature; S2.2.3 Spatial-Temporal Correlation: In IoT scenarios, aggregate the total window traffic of multiple devices under the same gateway, calculate the traffic similarity between devices, and identify coordination anomalies.

6. The method for real-time detection of network traffic anomalies based on deep learning according to claim 5, characterized in that: The multi-input CNN architecture in S4 includes: S4.1, Branch 1 uses 1D-CNN to process the temporal features of traffic; Traffic time-series feature processing: Suppose the input time series data is a vector sequence of length L. Features are extracted through one-dimensional convolution operations, and the convolution kernel... The features within a local window are processed by sliding computation, where d is the input dimension and w is the kernel width. The output of the k-th feature mapped at position i is: in: It is a non-linear activation function. The bias term is used, and the final output is a feature tensor composed of all feature maps. S4.2, Branch 2 uses LSTM to process user behavior sequences; User behavior sequence processing: Let the input behavior sequence be The hidden state is updated step-by-step by the loop unit. At time step t, the unit receives the current input. And the previous hidden state Calculate candidate states and gating signals (input gate) Forgotten Gate Output gate ): in: and This is the weight matrix. This indicates element-wise multiplication, and the final output is the hidden state at the last time step or all states of the sequence. S4.3 and Branch 3 use text CNN to process system log embedding vectors; System log embedding vector processing: Let the input be the word embedding matrix E (n is the number of words, m is the embedding dimension), local features are extracted through multi-scale convolutional kernels. (r is the kernel height) Sliding calculation on the embedding matrix generates feature maps: Pooling operations are performed on the convolution results at different scales to obtain a fixed-dimensional log feature representation.

7. The method for real-time detection of network traffic anomalies based on deep learning according to claim 6, characterized in that: The threshold adjustment in S5 updates the threshold boundary based on the recent traffic characteristic distribution within the window, introduces a decay factor, and assigns higher weight to new data in order to flexibly adapt to sudden traffic patterns. The recursive method for dynamic thresholds includes the following steps: S5.1 Defining variables: Let t be the current time; The flow rate observation value at time t; The threshold parameter at time t can be a vector or a scalar; This is the error covariance matrix, used in PLS to adjust the update step size; λ is the decay factor, 0≤λ≤1, usually taken as λ≈0.95-0.99, to give new data a higher weight; S5.2, Recursive Update Steps: For each new data point : Calculate the prior error: in, It is an eigenvector; Update the gain vector : Update parameter estimates: Update the error covariance matrix: S5.3 Extracting dynamic thresholds: Based on the estimated parameters The current threshold boundary is calculated as follows: like = Let be the mean and variance, respectively. Then the threshold can be set as: Where k is a constant.

8. The method for real-time detection of network traffic anomalies based on deep learning according to claim 7, characterized in that: Based on S8, a real-time feedback closed loop based on streaming processing is constructed to achieve efficient, lossless flow and instant learning of feedback data.

9. The method for real-time detection of network traffic anomalies based on deep learning according to claim 8, characterized in that: Based on S8, a context-aware adaptive threshold and meta-learning adjustment are developed to enable the threshold adjustment to have scene understanding capabilities.

10. A method for real-time detection of network traffic anomalies based on deep learning according to claim 9, characterized in that: Based on S8, an enhanced framework for predictive and collaborative defense is created, upgrading the system from a detection-feedback loop to an enhanced framework for prediction-recommendation-collaboration.

Images