Methods, devices and media for identifying abnormal network traffic

Abstract

This disclosure relates to a method, apparatus, and medium for identifying abnormal network traffic, belonging to the field of network security technology. The method includes: collecting real-time network data packets; processing the real-time network data packets to obtain semantic features and time-series feature indicators; encoding the semantic features using an improved BERT model to obtain high-dimensional semantic vectors, wherein the improvement to the BERT model includes: designing corresponding Tokenizer and Embedding layers for the semantic features; modeling and encoding the time-series feature indicators using a Prophet model to obtain time-series feature vectors; generating multimodal feature vectors; and using the model to detect the multimodal feature vectors to obtain identification results. The reinforcement learning network model uses the DDQN algorithm, and the model includes states, actions, and rewards; the parameters of the network model are updated based on the final reward for the action execution. This solution can accurately identify abnormal network traffic and can continuously adapt to new and unknown attack patterns.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of network security, and particularly relates to an abnormal network traffic identification method, device and medium. BACKGROUND

[0002] Currently, network attack means is increasingly complex, especially the carpet bombing network attack with its distributed, mass request characteristics, which is easy to cause the target server to be down and the service to be unavailable. The existing network attack detection system usually relies on traditional statistical features such as request rate, connection number, packet size, or rule-based methods. The detection system usually relies on traditional statistical features such as request rate, connection number, packet size, or rule-based methods.

[0003] Statistical feature-based method: for mass network traffic data, the change of its statistical features may lag behind the outbreak of attack, and is easily disturbed by normal traffic fluctuations, with high false negative rate and false positive rate. In addition, it is difficult to effectively capture the semantic features of the attack, such as specific payload in HTTP request, URL pattern, etc.

[0004] Rule-based method: a large number of attack rules need to be maintained manually, which is difficult to cope with new and variant attacks.

[0005] Existing deep learning method: although some deep learning-based methods such as LSTM and CNN have been used for traffic analysis, they still have limitations in capturing long-distance dependency and understanding complex semantic patterns. It is especially difficult to efficiently handle semantic features and time series features at the same time, and the ability to identify attack behavior in detail is insufficient. SUMMARY

[0006] The present disclosure proposes an abnormal network traffic identification method, device and medium, aiming to solve the problems that the existing technology is difficult to simultaneously handle the semantic features and time series features of network packets, resulting in insufficient carpet bombing network attack detection capability, and unable to identify new and variant attacks.

[0007] To solve the above technical problems, according to the first aspect of the present disclosure, an abnormal network traffic identification method is proposed, comprising:

[0008] Collecting real-time network packets;

[0009] Processing the real-time network packets to obtain semantic features and time series feature indicators;

[0010] Encoding the semantic features by using an improved BERT model to obtain high-dimensional semantic vectors, wherein the BERT model is improved, including: designing corresponding Tokenizer and Embedding for semantic features;

[0011] The Prophet model is used to model and encode time series feature indicators to obtain time series feature vectors;

[0012] High-dimensional semantic vectors and time-series feature vectors are fused to generate multimodal feature vectors;

[0013] An abnormal network traffic identification reinforcement learning network model is used to detect multimodal feature vectors and obtain identification results;

[0014] The reinforcement learning network model for anomaly network traffic identification uses the DDQN algorithm and includes states. ,action and rewards According to the action The final reward for execution Update the parameters of the reinforcement learning network model for identifying abnormal network traffic.

[0015] In some embodiments, encoding semantic features using an improved BERT model to obtain a high-dimensional semantic vector includes:

[0016] The semantic features are processed using the corresponding Tokenizer and Embedding layers and positional encoding layers to obtain a vector sequence. The improved BERT model includes a positional encoding layer.

[0017] By using a multi-head self-attention mechanism and a feedforward neural network to process vector sequences, high-dimensional semantic vectors are obtained.

[0018] In some embodiments, the state It is a multimodal feature vector;

[0019] action include: Normal traffic. Potential attack traffic :Sure attack;

[0020] award Including: Rewards for correctly identifying legitimate traffic for Correctly identify Rewards for attacking for , mistakenly Rewards for identifying attacks as legitimate traffic for Mistakenly identified normal traffic as Rewards for attacking for Timely identification Rewards for attacking for Failed to identify in time Rewards for attacking for .

[0021] In some embodiments, the detection of multimodal feature vectors using an abnormal network traffic identification reinforcement learning network model includes:

[0022] Initialize the main network Weight parameters The DDQN algorithm uses a dual-network architecture, consisting of a main network and a secondary network. and target network ;

[0023] copy Initialize the target network with weight parameters Weight parameters ;

[0024] Obtaining multimodal feature vectors ,in, Indicates in Multimodal feature vectors at time points;

[0025] Utilizing the main network Calculation execution First Q value, execution The second Q value, execution The third Q-value; the Q-value represents the long-term reward of taking an action in a specific state, that is, the value of the state-action pair. It reflects the goal of achieving long-term value in the action pair and is the core basis for agent decision-making.

[0026] Determine the first maximum Q value among the first Q value, second Q value, and third Q value, execute the action corresponding to the first maximum Q value, and then obtain the corresponding recognition result.

[0027] In some embodiments, based on the action The final reward for execution Update the parameters of the reinforcement learning network model for anomaly network traffic identification, including:

[0028] After executing the action corresponding to the first maximum Q value, we obtain... and instant rewards ;

[0029] Utilizing the main network Sure The action corresponding to the second maximum Q value;

[0030] Using target network The second maximum Q value corresponding to the action is calculated by using the target network . The value of the action corresponding to the second maximum Q value is calculated according to the immediate reward and the target value

[0031] . The target value is calculated according to the second maximum Q value and the value of the action corresponding to the second maximum Q value . The error value is calculated according to the target value and the predicted value

[0032] , wherein the predicted value is the first maximum Q value . The first maximum Q value is updated according to the error value and the predicted value . The loss is calculated according to the target value

[0033] and the predicted value , wherein the weight parameters of the main network are updated according to the loss, and the weight parameters of the target network are updated according to the parameters

[0034] . In some embodiments, the target value is calculated according to the immediate reward and the value of the action corresponding to the second maximum Q value , and the value of the action corresponding to the second maximum Q value is calculated by using the target network . The value of the action corresponding to the second maximum Q value is calculated according to the second maximum Q value . In some embodiments, the target value is calculated according to the immediate reward and the value of the action corresponding to the second maximum Q value

[0035] , and the value of the action corresponding to the second maximum Q value is calculated by using the target network . The value of the action corresponding to the second maximum Q value is calculated according to the second maximum Q value . The value of the action corresponding to the second maximum Q value is calculated according to the second maximum Q value

[0036] . The value of the action corresponding to the second maximum Q value is calculated according to the second maximum Q value . In some embodiments, the target value is calculated according to the immediate reward and the value of the action corresponding to the second maximum Q value

[0037] , and the value of the action corresponding to the second maximum Q value is calculated by using the target network . The value of the action corresponding to the second maximum Q value is calculated according to the second maximum Q value .

[0038] In some embodiments, the target value is calculated according to the immediate reward and the value of the action corresponding to the second maximum Q value , and the value of the action corresponding to the second maximum Q value is calculated by using the target network . The target value is calculated according to the formula

[0039] . The target value is calculated according to the formula . In some embodiments, the target value is calculated according to the immediate reward

[0040] and the value of the action corresponding to the second maximum Q value , and the value of the action corresponding to the second maximum Q value is calculated by using the target network .

[0041] In some embodiments, the according to Error value and predicted value renew The first maximum Q value includes:

[0042] The first maximum Q value is determined to be the third Q value;

[0043] According to the formula: ,renew The first maximum Q value;

[0044] in, express Error value, This represents the learning rate.

[0045] According to a second aspect of this disclosure, this disclosure also proposes an abnormal network traffic identification device, comprising:

[0046] Memory; and

[0047] A processor coupled to the memory is configured to execute the abnormal network traffic identification method described above based on instructions stored in the memory.

[0048] According to a third aspect of this disclosure, a computer-readable storage medium is also provided that stores computer program instructions thereon, which, when executed by a processor, implement the abnormal network traffic identification method as described above.

[0049] By adopting the above technical solutions, some embodiments of this disclosure can achieve the following beneficial technical effects:

[0050] First, enhanced semantic understanding: The improved BERT model can deeply understand the text / semantic information in network traffic packets and capture attack patterns that are difficult to detect by traditional methods, such as forged User-Agents and specific URL constructions.

[0051] Second, time-dynamic modeling: The Prophet model can effectively capture the time-series characteristics of network traffic and identify periodic and trend changes, thereby more accurately judging anomalies.

[0052] Third, improve detection accuracy and recall: By fusing multimodal features, traffic behavior can be more comprehensively characterized, reducing the one-sidedness of a single sample; reinforcement learning can dynamically optimize decision-making strategies and improve the ability to identify complex attacks.

[0053] Fourth, reduce reliance on labeled data: Reinforcement learning learns by interacting with the environment, and can utilize unlabeled or semi-labeled data to a certain extent, reducing data labeling costs.

[0054] Fifth, enhanced real-time performance and adaptability: The online learning characteristics of reinforcement learning enable the system to continuously adapt to new and unknown attack patterns without the need for frequent offline retraining.

[0055] Sixth, dealing with unknown attacks: Based on deep learning models, even novel carpet bombing attacks can be identified by the model as long as they exhibit certain semantic or temporal patterns. Attached Figure Description

[0056] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0057] This disclosure can be more clearly understood with reference to the accompanying drawings and the following detailed description.

[0058] Figure 1 This is a flowchart illustrating an abnormal network traffic identification method according to some embodiments of the present disclosure.

[0059] Figure 2 This is a schematic diagram illustrating a reinforcement learning network according to some embodiments of the present disclosure.

[0060] Figure 3 This is a schematic diagram illustrating a DQN network structure according to some embodiments of the present disclosure.

[0061] Figure 4 This is a flowchart illustrating a Q network update process according to some embodiments of the present disclosure.

[0062] Figure 5 This is a block diagram illustrating an abnormal network traffic identification device according to some embodiments of the present disclosure.

[0063] Figure 6 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Detailed Implementation

[0064] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0065] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0066] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0067] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0068] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0069] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0070] like Figure 1 As shown, the method disclosed herein includes:

[0071] Collect and parse real-time network data packets (raw network traffic data), extract key fields such as IP address, port, protocol, HTTP headers (User-Agent, Referer, Cookie, etc.), URL, request body (if available), and record timestamps.

[0072] An improved BERT model is used to encode extracted textual / semantic features (such as User-Agent strings, URLs, HTTP header field values, etc.) to generate high-dimensional semantic vectors. The improvement lies in the ability to perform transfer learning or fine-tuning for specific textual patterns in network traffic, thereby increasing sensitivity to textual features related to network attacks.

[0073] Improvements: Design dedicated tokenizer and embedding layers for specific fields of network protocols (such as HTTP request methods, URLs, header information, DNS query content, etc.), and introduce domain-related pre-trained corpora for further pre-training to enhance its ability to understand network semantics.

[0074] Output: Generate a vector representation of the text / semantic features, "V_text".

[0075] The core of the BERT model is the Transformer's Encoder layer, which contains two main sublayers: Multi-Head Self-Attention and a Feed-Forward Network. The Transformer uses Multi-Head Self-Attention to compute the correlation between any two positions in a sequence (representing different feature dimensions). This self-attention mechanism allows the model to focus on different parts of the input sequence, thereby capturing long-range dependencies and global context in the traffic sequence. For example, when identifying anomalous traffic, the model can notice the temporal correlation between an early series of low-rate probe requests and a later surge in traffic.

[0076] The Transformer Encoder utilizes its multi-head self-attention mechanism to capture complex dependencies within and between sequences. Its basic principle is as follows:

[0077] Self-attention mechanism: For the input sequence Attention weights are obtained by calculating the vectors of query (Q), key (K), and value (V).

[0078] ; ; .in, , , Both are weight matrices. Attention score calculation: ,in It is the dimension of the Key vector.

[0079] Attention weight calculation: .

[0080] Multi-head self-attention: Q, K, and V are projected onto h different subspaces, and self-attention is computed in parallel in each subspace. Finally, the outputs of the h heads are concatenated and linearly projected again. This allows the model to learn information from different representation subspaces.

[0081] .

[0082] .

[0083] in, For the first Secondary attention weights, This is the weight matrix for concatenating the outputs of h heads.

[0084] Feed-Forward Network: The output at each location passes through an identical two-layer fully connected feed-forward network that does not share any parameters.

[0085] Where z is the output of the self-attention mechanism, , These are the weight parameters of the two fully connected network layers. , These are the bias parameters for the two fully connected layers, respectively.

[0086] Output: The context-aware feature representation "V_text" after processing by the Transformer Encoder.

[0087] Using the Prophet model, key time-series characteristic indicators of network traffic (such as the number of connections per unit time, traffic volume, proportion of new connections, port distribution, request rate, mean / variance of packet size, etc.) are modeled and encoded to capture their periodicity, trend and seasonality changes, and generate time-series feature vector "V_time".

[0088] The high-dimensional semantic vector generated by the BERT model and the time-series feature vector generated by the Prophet model are fused to obtain a multimodal feature vector (a comprehensive representation of network traffic data points). The fusion method involves concatenating or weighting the vectors of each time step in "V_text" and "V_time". Further processing will be performed by constructing a reinforcement learning network.

[0089] This application employs a reinforcement learning network model for detecting abnormal traffic and identifying DDoS network attack behaviors.

[0090] Reinforcement learning is a machine learning method that learns by having a model perform actions in an environment and receiving rewards or penalties based on the results. The goal of reinforcement learning is to find a policy that maximizes the cumulative reward of the model in the long run.

[0091] like Figure 2 As shown, the key techniques of reinforcement learning include state, action, reward, and policy. State: State describes information about the environment; the model selects actions based on the current state. Action: An action is the operation the model can perform in the environment. Reward: The reward is the feedback the model receives after performing an action; it guides the model in choosing actions. Policy: The policy is the method by which the model selects actions; it can be deterministic or stochastic.

[0092] The process of reinforcement learning can be represented by the following mathematical formula:

[0093] ,in, It is in state Next action value, It's a reward. It's the learning rate. It is a discount factor. It's a new state. It's a new move.

[0094] Define the state space: the fused multimodal feature vectors, which represent the overall state of the current network traffic.

[0095] Define the Action Space (A): the executable actions of the detection system, such as: A0: Normal traffic; A1: Potential attack traffic; A2: Identify DDoS attack. Define the Reward Function (R): used to guide the agent in learning the optimal strategy. The design of the reward function is crucial, requiring a balance between accuracy, recall, and timeliness.

[0096] Correctly identify normal traffic: R_normal = + ( (Positive weight); Correctly identify DDoS attacks: R_ddos = + ( Positive weights, usually greater than ); Mistakenly identifying DDoS attacks as normal traffic (false negative): R_false_negative = - ( (Negative weighting results in a stronger penalty); misidentifying normal traffic as a DDoS attack (false positive): R_false_positive = - ( (Negative weighting, moderate penalty); Timely identification of DDoS attacks: R_timely = + ( (Positive weighting, rewarding timeliness); Failure to identify DDoS attacks in a timely manner: R_untimely = - ( (Negative weighting, penalty for delay).

[0097] Total reward: R(s, a) = R_normal + R_ddos + R_false_negative + R_false_positive + R_timely + R_untimely (calculated based on actual actions and results).

[0098] Reinforcement learning algorithm selection: The DDQN (Dueling Double DQN) algorithm is adopted to handle the high-dimensional state space. The DDQN algorithm can handle the discrete action space and performs well in terms of stability and accuracy of Q-value estimation.

[0099] Model training: Experience replay: Storage (state) Action , award Next state The system uses a transfer sample to break the correlation between data and improve sample utilization. A separate "target network" is used to calculate the target Q-value, increasing training stability. During operation, the system continuously updates the reinforcement learning model based on real-time traffic data and feedback reward signals, enabling it to adapt to new attack patterns. The response and feedback module executes corresponding response strategies (such as traffic scrubbing, blocking, and alerting) based on the output actions of the DDoS detection reinforcement learning model, and feeds back the detection results (comparison between the real label and the model's predicted label) as reward signals for online learning.

[0100] DQN Algorithm Principle: The DQN algorithm employs a dual-network architecture, with one network used to train the main network. (Q network), another target network The (Target Q network) is used to compute the TD objective. The TD objective is the core of the temporal difference algorithm, used to update the estimated value function, combining immediate rewards with value predictions for subsequent states. Because the Q network (the main network) The target Q network is continuously updated during training. Directly using it to calculate the TD target will cause the target value to change constantly, which may lead to training instability. By using a relatively fixed Target Q network (target network...) This can reduce such instability, thereby alleviating the so-called "overestimation" problem, where the algorithm may overestimate future rewards. For example... Figure 3 As shown, the Q network (main network) The input is the state, and the output is the Q-value corresponding to each action. The more accurate the Q-value, the better the Q-network (main network). The better the training.

[0101] like Figure 4 As shown, the basic steps of the DDQN algorithm are as follows: Initialize the main network (main network) ) and target network (target network) The two networks have the same structure; at each time step, an action is selected using the main network based on the current state to determine the current action. Here we assume it to be ; Perform the selected action (will Input into the environment and get and ), observe the next state and the immediate reward to gain an experience: Then, the experience is added to the experience pool. The next state is calculated using the target network. The action with the maximum Q value, calculate its Q value. Update the Q-value of the current state using the target Q-value of the action with the maximum Q-value of the next state. The update formula is:

[0102] ;

[0103] , .

[0104] Update the parameters of the main network using the mean squared error loss function. Update the parameters of the target network periodically.

[0105] Based on the output of the Transformer model, real-time online decisions are made, and an alarm mechanism is triggered when abnormal traffic or carpet-bombing cyberattacks are detected.

[0106] The concept of this solution will be further illustrated through another embodiment.

[0107] Based on the statistical characteristics of traffic (such as connection count, request frequency, packet size, etc.), metrics suitable for text analysis and time series analysis are calculated. Structured feature extraction: Extracting statistical information such as IP address, port, protocol, packet size, connection duration, packet count, and traffic rate. Unstructured feature extraction: Extracting text data (semantic feature fields) from HTTP requests (such as GET / POST methods, URL paths, query parameters, User-Agent, Referer, Accept fields in request headers, etc.) and potential payloads. Data cleaning: Cleaning the extracted text data, such as removing special characters, URL decoding, case conversion, and word segmentation. Serialization: Organizing the processed traffic data (including structured features and text information) into a sequence format suitable for model input.

[0108] For the extracted text / semantic feature fields, such as User-Agent strings, URLs, and HTTP header values, an improved BERT model is used for encoding. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on Transformers. It is pre-trained through Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks, enabling it to deeply understand the bidirectional semantic information of words in context.

[0109] BERT Model Improvements: Domain-Adaptive Fine-Tuning: Based on the general BERT model, fine-tuning is performed using large-scale network traffic logs, security incident reports, and attack sample descriptions to better understand cybersecurity terminology and attack patterns. Introduction of Specific Category Embeddings: Special embeddings are designed for common attack types (such as SQL injection keywords, XSS keywords, scanning tool user-Agents, etc.) or normal traffic identifiers (such as browser user-Agent types), and combined with word embeddings. Finally, the BERT model outputs a high-dimensional semantic vector.

[0110] The BERT model encoding process involves segmenting the extracted text information (such as the URL of an HTTP request, User-Agent, etc.) from the network traffic and using it as input to the BERT model. The BERT model then constructs customized Tokenizer and Embedding layers based on these features.

[0111] Represent the input text sequence as Each 's' is a TokenID. The input layer of the BERT model generates a combination of word embeddings, positional embeddings, and segment embeddings. Token Embedding: Position Embedding: (Indicates location information); Segment Embedding: (Distinguish between different text fragments); Input representation:

[0112] .

[0113] The multiple encoder layers of the BERT model will process these embeddings through a multi-head self-attention mechanism and a feedforward network, ultimately obtaining the context embedding representation of each token.

[0114] For a given text sequence, extract the output vector corresponding to each token in the sequence, and perform pooling on the output vectors of all tokens to obtain a fixed-dimensional text / semantic feature vector V_text. V_text = BERT_Encoder(E_text).

[0115] For the I-th layer, the embedding vector of the i-th token is represented as: Layer I refers to a specific layer among the multiple Transformer layers stacked in the BERT encoder. layer).

[0116] Multi-head self-attention mechanism calculation: ; —Concatenate h attention points and perform another linear transformation. To construct the weight matrix for h attention operations; —Map Q, K, V to a lower-dimensional space and compute the k-th attention.

[0117] —Calculation of self-attention weights; It is the dimension of the key vector, divided by This can prevent excessive input from causing The function value is close to 0, thus avoiding gradient vanishing. At the same time, this operation makes... The results follow a distribution with a mean of 0 and a variance of 1, similar to a normalization effect. Here, the Q (Query), K (Key), and V (Value) vectors are matrices obtained through linear transformation:

[0118] ; ; .

[0119] in, Let Q be the embedding vector of the j-th token in layer I-1; Q is the query vector, used to find the relevance with other locations; k is the key vector, used to calculate the similarity with the query vector; and V is the value vector, containing the actual information content. , , These represent the weights of the Q, K, and V vectors in the I-th layer, respectively.

[0120] Feedforward network processing: ,in, The result of multi-head self-attention mechanism calculation for the i-th input sequence; Where z is the output of the self-attention mechanism, , These are the weight parameters of the two fully connected network layers. , These are the bias parameters of the two fully connected layers, respectively. Feedforward neural networks are generally two identical fully connected feedforward layers that do not share parameters.

[0121] We obtain the context-related embedding vector of the i-th token.

[0122] The BERT model stacks multiple such encoder layers to ultimately output a context-dependent vector representation of each input token.

[0123] For the Lth text sequence, the final result is... Add a pooling layer to... Perform pooling operations to output a fixed-dimensional semantic feature vector V_text.

[0124] The extracted time-series feature indicators, such as the statistical measures of request rate, number of connections, and data packet size per unit time, are modeled using the Prophet model.

[0125] Prophet is an open-source time series forecasting model from Facebook, particularly suitable for time series with periodicity (daily, weekly, yearly), holiday effects, and missing values. Its model form is as follows:

[0126] Where g(t) is the trend term, simulating non-periodic changes, usually using a piecewise linear model or a logistic growth model; s(t) is the periodic term, simulating weekly or yearly periodic changes, represented using a Fourier series; and h(t) is the holiday term, simulating the impact of special dates (such as holidays). This is the error term.

[0127] The Prophet model can capture the normal fluctuation patterns of traffic indicators. In this system, we do not use it directly to predict future values. Instead, we utilize its fitting ability to train the Prophet model based on normal traffic data over a period of time, and then fit it with new traffic data. The resulting fitting residual (i.e., the difference between the actual value and the model's predicted value) can be used as a time series feature vector. In addition, the future trends and periodic components predicted by the Prophet model are also included as part of the time series feature vector.

[0128] The fusion encoding module concatenates the semantic vectors generated by the BERT model and the time-series vectors generated by the Prophet model, or combines them through other fusion mechanisms (such as attention fusion), to form a multimodal feature vector. The fusion method used in this disclosure is to concatenate or weightedly sum the vectors of each time step in "V_text" and "V_time".

[0129] A reinforcement learning network is constructed to classify abnormal traffic and identify DDoS network attack behaviors.

[0130] The fused feature vector S is input into the "DDoS detection reinforcement learning model". Based on the current state S, the model selects an action A from the defined action space A = {normal traffic, potential attack traffic, confirmed DDoS attack} using an e-greedy policy. Specifically, the model will randomly select an action with probability ε (exploration) and select the action with the highest current Q-value with probability 1-ε (exploitation).

[0131] The selected action A is sent to the "Response and Feedback Module".

[0132] Response execution: If A is "normal traffic", then allow the traffic to pass through "Network egress / security device (G)"; if A is "potential attack traffic", then mild intervention measures such as rate limiting and traffic mirroring analysis may be taken; if A is "confirmed DDoS attack", then strong defense measures such as blocking and traffic redirection to cleaning devices will be implemented.

[0133] Observational environment feedback: The system judges the correctness of the model's decisions based on actual traffic conditions (e.g., subsequent traffic behavior, manual confirmation results, threat intelligence, etc.) and calculates the reward R. For example, identifying normal traffic as a DDoS attack (false positive) will result in a negative reward R_false_positive; successfully identifying and blocking a DDoS attack will result in positive rewards R_ddos and R_timely.

[0134] The current state S, the chosen action A, the reward R obtained, and the observed next state S' are combined into an empirical quadruple (S, A, R, S').

[0135] Store the quadruple in the "Experience Replay Pool".

[0136] When the number of samples in the experience replay pool reaches a certain amount or a specific condition is met, a batch of experience quadruplets is randomly sampled from the replay pool.

[0137] Use the current policy network and target network To calculate the target Q value. The target Q value is usually calculated as follows: ,in It is a discount factor used to measure the importance of future rewards.

[0138] Calculate the loss function and use the mean squared error (MSE) to measure the predicted Q value. The difference between the target Q value y and the target Q value y.

[0139] Use gradient descent algorithms (such as the Adam optimizer) to update the policy network parameters θ to minimize the loss function.

[0140] The parameters θ of the policy network are periodically copied to the target network θ' (e.g., every N steps) to maintain training stability.

[0141] After completing a model update, return to the status input and action selection module to continue processing new traffic.

[0142] The entire process is cyclical. While the system monitors traffic in real time, it continuously learns from experience and optimizes its detection strategies, thereby achieving continuous and intelligent detection of abnormal traffic and DDoS attacks.

[0143] Based on the output of the reinforcement learning network model, if the traffic is determined to be normal, monitoring continues; if it is determined to be abnormal, corresponding alarms are triggered based on further judgment. For example, if it is identified as a DDoS attack, a high-level alarm is issued; if it is identified as other types of abnormal traffic, an alarm of the corresponding level is issued.

[0144] The concept of this solution will be further illustrated through another embodiment below.

[0145] This article details the six steps of the DDQN algorithm through a real-world case study of an e-commerce website experiencing an HTTP Flood attack. The example demonstrates how the algorithm progresses from its initial state to achieving accurate detection.

[0146] Background: During the "Double Eleven" shopping festival, a certain e-commerce platform had a normal traffic of 2,000 requests per minute, but suddenly experienced an abnormal traffic peak.

[0147] Specific parameters: Status characteristics: Request rate: 8500 times / minute (normal 2000 times); User-Agent: Contains the malicious identifier "BotNet_V2"; Source IP: Concentrated in a specific IP range (192.168.1.100-192.168.1.200); Request target: Same API endpoint / api / checkout. Action space: A0: Normal traffic (allow); A1: Potential attack (monitor and observe); A2: DDoS attack (immediately block). Reward settings: Correctly identify DDoS: +5 points; Missed detection (attack misjudged as normal): -10 points; False alarm (normal detected as attack): -3 points; Timely detection: +2 points.

[0148] Network initialization: Operation process: Main network Q1: Randomly initialize weight parameters Target network Q2: Copy the same parameters Network structure: input layer (fusion feature dimension), 3 hidden layers, output layer (Q-values ​​of 3 actions).

[0149] Action Selection: Current state s_t: Fusion Feature Vector = [Request Rate: 8500, Text Features: "BotNet_V2", Source IP Concentration: 0.95, Target API: " / api / checkout"]. Q-value Calculation: Main Network Q1 Output: Q(s_t, A0) = 1.2 (Normal Action); Q(s_t, A1) = 2.8 (Potential Attack); Q(s_t, A2) = 4.5 (DDoS Attack). Action Selection: Select the action corresponding to the maximum Q-value: A2 (DDoS Attack); Exploration Strategy: 10% probability of random selection (to avoid getting trapped in local optima).

[0150] Execute actions and store experience: Execute action A2: Trigger traffic cleaning mechanism; block requests from suspicious IP segments; start deep packet inspection; Environment feedback: Next state s_{t+1}: Request rate drops to 2100 times / minute; Instant reward r_t: +7 points (correct identification +5, timely detection +2); Experience storage; Experience tuple: (s_t, A2, +7, s_{t+1}); Store in experience pool (capacity 10000 records).

[0151] Target network Q-value calculation: Next state analysis: s_{t+1} = [Request rate: 2100, Text characteristics: "Normal", Source IP concentration: 0.15, Target API: "Dispersed"]. Q-value calculation process: Main network Q1 selects action: argmax_aQ1(s_{t+1}, a) = A0; Target network Q2 calculates Q-value: Q2(s_{t+1}, A0) = 1.8.

[0152] Update the Q-value. Parameter settings: learning rate = 0.01; Discount factor = 0.95; Calculation process: target = r_t + • Q2(s_{t+1}, A0) = 7 + 0.95×1.8 = 8.71; predict = Q1(s_t, A2) = 4.5; TD error = target - predict = 8.71 - 4.5 = 4.21; Q value update: Q1(s_t, A2) ← 4.5 + 0.01 × 4.2 = 4.54.

[0153] Update network parameters. Loss calculation: L = (target - predict)² = (8.71 - 4.5)² = 17.72. Backpropagation: Update the main network parameters using the Adam optimizer. Learning rate decay: Decreases to 0.9 of the original rate every 10,000 steps; Target network update: Synchronizes every 1,000 steps. ←0.95 + 0.05 .

[0154] Batch training: Randomly select 32 experiences for batch training. Experience 1: High rate + Bot feature → A2 → +7 reward; Experience 2: Normal rate + Dispersed IP → A0 → +1 reward; Experience 3: Medium rate + Suspicious UA → A1 → +0.5 reward. Breaking correlation: Avoid the model overfitting to recently experienced specific patterns.

[0155] The stabilizing role of the target network. If the attack pattern suddenly changes, such as from HTTP Flood to Slowloris, the target network provides a stable Q-value estimation benchmark, preventing the main network from over-adjusting.

[0156] like Figure 5 As shown, the abnormal network traffic identification device 500 includes a memory 510 and a processor 520 coupled to the memory 510. The memory 510 is used to store instructions for executing embodiments of the abnormal network traffic identification method. The processor 520 is configured to execute the abnormal network traffic identification method in any of the embodiments of this disclosure based on the instructions stored in the memory 510.

[0157] Figure 6 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Figure 6 As shown, the computer system 600 can be represented in the form of a general computing device. The computer system 600 includes a memory 610, a processor 620, and a bus 630 connecting different system components.

[0158] The memory 610 may include, for example, system memory, non-volatile storage media, etc. The system memory may include, for example, storage...

[0159] It stores the operating system, applications, the boot loader, and other programs. System memory can...

[0160] This includes volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for executing at least one of the abnormal network traffic identification methods in a corresponding embodiment. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0161] The processor 620 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Correspondingly, modules can be implemented by executing instructions in the central processing unit (CPU) memory to perform corresponding steps, or by implementing dedicated circuits that perform corresponding steps.

[0162] Bus 1230 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0163] The computer system 600 may also include an input / output interface 640, a network interface 650, a storage interface 660, etc.

[0164] These interfaces 640, 650, and 660, as well as the memory 610 and processor 620, can be connected via bus 630. Input / output interface 640 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 650 provides a connection interface for various networked devices. Storage interface 660 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0165] Here, reference is made to flowchart illustrations and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of this disclosure.

[0166] Various aspects of this disclosure have been described. It should be understood that each block of the flowchart and / or block diagram, as well as combinations of blocks, can be implemented by computer-readable program instructions.

[0167] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0168] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0169] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0170] This concludes the detailed description of the abnormal network traffic identification method, apparatus, and medium according to the present disclosure. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description. Although some specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and not for limiting the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. An abnormal network traffic identification method, characterized by, The method comprises: collecting real-time network data packets; processing the real-time network data packets to obtain semantic features and time series feature indicators; encoding the semantic features using an improved BERT model to obtain high-dimensional semantic vectors, wherein the BERT model is improved, including: designing corresponding Tokenizer and Embedding layers for semantic features; modeling and encoding the time series feature indicators using a Prophet model to obtain time series feature vectors; fusing the high-dimensional semantic vectors and the time series feature vectors to generate multi-modal feature vectors; detecting the multi-modal feature vectors using an abnormal network traffic identification reinforcement learning network model to obtain identification results; The abnormal network traffic identification reinforcement learning network model adopts a DDQN algorithm, and the abnormal network traffic identification reinforcement learning network model comprises a state , an action , and a reward , wherein a final reward of the action is used to update parameters of the abnormal network traffic identification reinforcement learning network model; the state is a multi-modal feature vector; Actions comprise: : normal traffic, : potential attack traffic, : determining attacks; rewards including: reward for correctly identifying normal traffic for correctly identifying attacks for incorrectly identifying attacks as normal traffic for incorrectly identifying normal traffic as attacks for timely identifying attacks for failing to timely identify attacks for ; The abnormal network traffic identification reinforcement learning network model detects the multi-modal feature vectors, comprising: Initialize the main network weight parameters of the main network The DDQN algorithm uses a double network, which includes a main network and a target network ; copying of the target network of the target network ; Acquiring multi-modal feature vectors wherein, denotes a multi-modal feature vector at a time instant t. Utilizing a host network Computational execution a first Q-value of execution a second Q-value of execution a third Q-value of execution wherein Q value represents the value return of state-action pair; determining the first maximum Q value among the first Q value, the second Q value and the third Q value, executing the action corresponding to the first maximum Q value, and then obtaining the corresponding identification result.

2. The method of claim 1, wherein, The improved BERT model encodes the semantic features to obtain high-dimensional semantic vectors, comprising: processing the semantic features using the corresponding Tokenizer and Embedding layers and position encoding layer to obtain vector sequences, wherein the improved BERT model includes a position encoding layer; processing the vector sequences using multi-head self-attention mechanism and feedforward neural network to obtain high-dimensional semantic vectors.

3. The method of claim 1, wherein, According to the action Final reward of execution updating the parameters of the abnormal network traffic identification reinforcement learning network model, comprising: After performing the action corresponding to the first maximum Q value, a next state is obtained and an immediate reward ; Utilizing a host network determining the action corresponding to the second maximum Q value Utilizing a target network Computational execution The action corresponding to the second maximum Q-value Value; According to instant reward and performing the action corresponding to the second maximum Q-value value of the target value ; According to the target value and the predicted value Calculate Error value, wherein the predicted value is the first maximum Q value; According to error value and prediction value update a first maximum Q value; Based on the target value and predicted value Calculate the loss, whereby the main network is updated based on the loss. Weight parameters According to parameters Update target network Weight parameters .

4. The method of claim 3, wherein, The target network Computational execution The second maximum Q value corresponding to the action of Value, comprising: According to the second maximum Q value corresponding to the action value; wherein, represents performing the action corresponding to the second maximum Q-value.

5. The method of claim 4, wherein, The method according to the instant reward and executing the action corresponding to the second maximum Q value The value calculation target value includes: The target value is calculated according to the formula ;​ wherein represents a discount factor.

6. The method of claim 5, wherein, The according to Error values and predicted values Updating A first maximum Q value, comprising: The first maximum Q value is the third Q value; According to the formula: , the first maximum Q value of is updated; wherein, denotes error value, denotes learning rate.

7. An anomalous network traffic identification apparatus characterized by, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the instructions stored in the memory to perform the abnormal network traffic identification method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, A computer program instruction is stored thereon, which is executed by a processor to implement the abnormal network traffic identification method according to any one of claims 1 to 6.

Images