An interactive game-based network traffic anomaly detection method
By training a network traffic anomaly detection model using interactive game theory and reinforcement learning algorithms, the problem of insufficient adaptability of the model in complex network environments is solved, and more efficient anomaly detection and identification are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-09
- Publication Date
- 2026-03-31
AI Technical Summary
Existing network traffic anomaly detection models lack adaptability when facing complex network environments and cannot effectively identify new attack models, leading to an increase in the workload of network operations and maintenance personnel.
An interactive game-based network traffic anomaly detection method is adopted, which combines reinforcement learning algorithm and trains the model through feedback mechanism to establish game model and improve adaptive capability.
By using an interactive game approach, the model is trained with external information feedback, which improves the accuracy and adaptability of detection and reduces the workload of maintenance personnel.
Smart Images
Figure CN116248519B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of network traffic anomaly detection technology, and more specifically to a network traffic anomaly detection method based on interactive game theory. Background Technology
[0002] Any network traffic anomaly detection model must be able to adapt to different network scenarios, changing accordingly as the network environment evolves. Traditional network traffic anomaly detection involves quantitatively collecting data from a specific scenario and using it as input to a pre-designed machine learning algorithm model for training. Once trained, the model is deployed to the production environment. However, this model lacks the ability to adapt to the current network conditions. When a new attack model fails to identify it, an alert is issued, delegating the detection and investigation of the current data to network operations personnel. Given the current larger and more complex network infrastructure, basic services, and online business environments, this process significantly increases the workload of network operations personnel and is no longer sufficient to meet the needs of current monitoring and management tasks.
[0003] Therefore, improving the adaptive capability of network traffic anomaly detection is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] In view of this, the present invention provides a network traffic anomaly detection method based on interactive game theory. By combining reinforcement learning algorithms, the model receives reward and punishment signals from the outside world through feedback during the training process, thereby guiding its own training. It also collects external interaction data, establishes a game theory model, improves training accuracy, and enhances the adaptive capability of network traffic anomaly detection.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A method for detecting network traffic anomalies in interactive game theory includes the following steps:
[0007] Step 1: Collect network data within the local area network, construct a simulated network topology, build a virtual machine in the simulated network topology to simulate network attacks during data packet transmission between the client and server, and collect data packets and attack data;
[0008] Local area network (LAN) data collection involves establishing a simulated network topology within the LAN and using virtual machines to simulate the process of data packet transmission between clients and servers in a specific network environment. The data packets transmitted during this process are then collected. A network attack is simulated during this data packet transmission process, resulting in a specific network attack. Nine clients and one server are simulated. All nine clients act as attackers, while the server appears as the victim in the network topology. On the attacker's machine, a DoS attack is simulated using hping3, sending a data packet frame to the server every 10 milliseconds. On the server side, tcpdump is used to collect the traffic packet data. The traffic packet data collected on the server side is the same type of data as the data packets and data packets sent. The collected data packets contain two parts: TCP connection-related characteristics and time-based network traffic statistics. The main network data collected includes duration (connection duration), service (target host's network service type), flag (connection status: normal / error), src_bytes (number of bytes of data from the target host to the source host), wrong_fragment (number of faulty fragments), mem_defal (default TCP data receive window size), serror_rate (percentage of SYN error connections), and rerror_rate (percentage of REJ error connections).
[0009] Step 2: Based on the network data, train and construct an interactive guided anomaly detection model using reinforcement learning, preset game elements, and use machine learning methods to conduct interactive game on the interactive guided anomaly detection model;
[0010] Based on the game elements, machine learning methods are used to enable the interactive guided anomaly detection model to engage in interactive game with the model's guide entity; the interactive guided anomaly detection model is a neural network model trained based on the network data in step 1.
[0011] The game elements include game strategy and game payoff;
[0012] The game strategy includes the model strategy set and model strategy probability distribution of the interactively guided anomaly detection model, as well as the interaction strategy set and interaction strategy probability distribution;
[0013] The model strategy set is [m0, m1, m2], where m0 means no interaction is initiated, m1 means interaction is initiated and the data is judged as abnormal data, and m2 means interaction is initiated and the data is judged as normal data.
[0014] The probability distribution of the model policy is [p0, p1, p2], and p0 + p1 + p2 = 1;
[0015] The set of operation and maintenance interaction strategies is [a0, a1, a2], where a0 means no feedback, a1 means feedback and the feedback content is data of normal type, and a2 means feedback and the feedback content is data of abnormal type.
[0016] The probability distribution of the operation and maintenance interaction strategy is [q0, q1, q2], q0+q1+q2=1;
[0017] Preset game payouts;
[0018] The payoffs of the game process are pre-defined; the game process includes single-step game model failure, single-step game model victory, model attack success, game silence, interactive attack failure, and interactive attack success;
[0019] The payoff for a single-step game model failure is equal to the cost of the interaction. In addition, a penalty P is applied to the model; the interaction benefit is the total system resources R minus the cost required to provide feedback to the model.
[0020] Winning payoff in a single-step game model: The model's payoff equals the total system resources R minus the cost of interaction. The benefit of interaction equals the corresponding penalty P plus the cost required for feedback.
[0021] Successful model attack benefit: The model benefit is the total system resources R minus the cost required for the interaction. The reward for interaction is the corresponding penalty P;
[0022] Silent payoff in game theory: Both model payoff and interaction payoff are 0;
[0023] Benefits of failed attack during operations and maintenance interaction: The benefits of interaction are The model's return is 0;
[0024] Benefits of a successful attack via operations and maintenance interaction: The benefits of the interaction are... The model's return is 0;
[0025] Step 3: Preset interaction rules, collect environmental rewards and interaction feedback rewards, and train the interactive guidance anomaly detection model based on the environmental rewards, the operation and maintenance interaction feedback rewards and the interaction rules to obtain an adaptive anomaly detection model;
[0026] Interaction rules are preset parameters that need to be set during interactive game theory.
[0027] The interactive guided anomaly detection model receives environmental rewards and interactive feedback rewards, calculates the model reward, and uses the model reward as a loss function to train and optimize the interactive guided anomaly detection model. The expression for the model reward is:
[0028] R all (s t ,a t )=δR A (s t ,a t )+(1-δ)R E (s t ,a t )
[0029] Among them, R E (s t ,a t ) indicates environmental reward; R A (s t ,a t ) represents the interactive feedback reward; δ represents the weighting ratio between environmental rewards and interactive feedback rewards;
[0030] The weight ratio δ between environmental rewards and interactive feedback rewards is dynamically adjusted based on the accuracy of the model judgment results obtained from the adaptive anomaly detection model during training. This adjusts the proportion of direct feedback rewards in the model. When the model performs well in anomaly detection, the value of δ needs to be appropriately reduced to decrease the proportion of model guidance body feedback rewards in the total model reward. When the model performs poorly, the value of δ needs to be appropriately increased to increase the proportion of model guidance body feedback rewards in the total model reward, thereby increasing the guidance strength of the model guidance body.
[0031] Step 4: Use an adaptive anomaly detection model to detect network traffic anomalies in the collected data packets and obtain the detection results;
[0032] The data packets include DDoS, DoS, and Port Scan attacks in simulated network topologies, as well as normal data.
[0033] Preferably, the interactive guided anomaly detection model is a 5-layer neural network model, including one input layer, three hidden layers, and one output layer, with each layer fully connected. The input layer has 12 units, and all features in the dataset are scaled down to 12 using PCA principal component analysis. The output layer represents the data category. After the three hidden layers, the model outputs the reward values corresponding to the four actions. These reward values are used to calculate the convergence of the loss function of the neural network model, and the output result is the category of the current input network data, including DDoS, DoS, Port Scan, and Bengin.
[0034] Preferably, the game elements include participants, game strategies, and game payoffs, wherein:
[0035] The participants in the game process consist of the model guide and the model itself; they rationally decide their next strategy based on the actions of their opponents, thereby maximizing their own gains.
[0036] Game strategy: The actions that participants can choose during the game; the model's strategy set is [m0, m1, m2], where m0 represents not initiating interaction, m1 represents initiating interaction and classifying the data as anomalous, and m2 represents initiating interaction and classifying the data as normal; [p0, p1, p2] represent the probability distributions of the model's strategies, where p0 + p1 + p2 = 1; the strategy set of the model's guiding agent A is [a0, a1, a2], where a0 represents not providing feedback, a1 represents providing feedback with the data category being normal, and a2 represents providing feedback with the data category being anomalous; [q0, q1, q2] represent the probability distributions of the model's guiding agent's strategies, where q0 + q1 + q2 = 1;
[0037] Game payoff: The reward that participants can obtain during the game; when the model chooses to interact, the model's supervisor chooses to provide feedback, and the data category determined by the model is inconsistent with the data category in the feedback result, it is recorded as a failure of the model in a single step of the game, and the model's payoff equals the cost of the interaction. In addition to the corresponding penalty P applied to the model, the benefit of the model guidance entity is the total system resources R minus the cost required to provide feedback to the model.
[0038] When the model chooses to interact, the model's supervisor chooses to provide feedback, and the data category determined by the model matches the data category in the feedback result, the model is considered to have won in the single-step game. The model's payoff equals the total system resource R minus the cost of the interaction. The benefit of the model-guided entity equals the corresponding penalty P plus the cost of feedback.
[0039] If the model chooses interaction and the model's guidance agent chooses not to provide feedback, the model wins, and its benefit is the total system resources R minus the cost of interaction. The model guide's benefit is the corresponding penalty P;
[0040] When the model chooses not to interact and the model guide does not actively provide feedback to the model, the benefits of both the model and the model guide are 0.
[0041] When the model guidance agent believes that the model's prediction for a certain traffic flow is incorrect and actively provides feedback, but the feedback result is also incorrect, the model guidance agent's benefit is... The model's return is 0;
[0042] When the model guidance agent believes that the model's prediction for a certain traffic flow is incorrect, and actively provides feedback that is correct, the model guidance agent's reward is R. The model's return is 0.
[0043] Preferably, the δ value is dynamically adjusted based on metrics such as the accuracy of the model's judgment results, thereby adjusting the proportion of the model's guidance reward. When the model performs well in anomaly detection, the δ value should be appropriately reduced to decrease the proportion of the model's guidance reward in the total model reward; conversely, when the model performs poorly, the δ value should be appropriately increased to increase the proportion of the model's guidance reward in the total model reward, thus increasing the guidance strength of the model's guidance. During training, guidance from the model's guidance can help accelerate the convergence speed of the anomaly detection model, and the interactive guidance process allows the model to dynamically change its perception of anomaly data, thereby strengthening the model's ability to infer the intention of the model's guidance.
[0044] As can be seen from the above technical solution, compared with the prior art, this invention discloses a network traffic anomaly detection method based on interactive game theory. Combining the reward feedback mechanism of reinforcement learning algorithms, compared with traditional AIOps systems, this scenario adopts an interactive game approach. The model supervisor provides feedback and evaluation on the model's analysis and decisions through a visual interactive interface, thereby enabling the model to continuously obtain more accurate training results during the training process using external information provided by the model supervisor. To better facilitate cooperation between the model supervisor and the model, and to establish reasonable interaction and feedback strategies for the model, a reputation evaluation and update mechanism is established between them using a dynamic Bayesian game model. An equilibrium state is established between the model supervisor and the model through game theory, thereby guiding the behavior of both the model and the model supervisor. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0046] Figure 1 The attached figure is a flowchart of the network traffic anomaly detection method based on interactive game theory provided by the present invention;
[0047] Figure 2 The attached figure is a schematic diagram of the neural network model structure provided by the present invention for playing games with a model guide.
[0048] Figure 3The attached figure is a schematic diagram of the interactive game training process of the interactive guided anomaly detection model provided by the present invention. Detailed Implementation
[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0050] like Figure 1 As shown in the figure, a network traffic anomaly detection method based on interactive game theory according to an embodiment of the present invention may include the following steps in its specific implementation process:
[0051] S1: Local Area Network (LAN) Data Acquisition. A simulated network topology is established within the LAN. A virtual machine simulates the process of data packet transmission between clients and servers in a specific network environment, and the data packets sent during this process are collected. A DDoS attack is simulated in the current environment, with 9 simulated clients and 1 server. All 9 clients act as attackers, and the server appears as the victim in the network topology. On the attacker's machine, a DoS attack is simulated using hping3, sending a data packet frame to the server every 10 milliseconds. On the server side, tcpdump is used to collect the traffic packet data. The extracted data includes two parts: TCP connection-related features and time-based network traffic statistics.
[0052] S2: Extraction of elements from the game system;
[0053] (1) Participants: Composed of a model-guided team (operation and maintenance personnel) and an interactive guided anomaly detection model. They rationally decide their next strategy based on the opponent's actions, thereby maximizing their own gains;
[0054] (2) Strategy: The actions that participants can choose during the game. The strategy set of model M is [m0, m1, m2], where m0 means not to initiate interaction, m1 means to initiate interaction and judge the data as abnormal, and m2 means to initiate interaction and judge the data as normal; [p0, p1, p2] represent the probability distribution of the model strategy, where p0+p1+p2=1; the strategy set of model guide A is [a0, a1, a2], where a0 means not to provide feedback, a1 means to provide feedback and the feedback content is normal data, and a2 means to provide feedback and the feedback content is abnormal data; [q0, q1, q2] represent the probability distribution of the model guide strategy, where q0+q1+q2=1;
[0055] (3) Payoff: The gain that participants can obtain during the game. When the model chooses to interact, the model guide chooses to provide feedback, and the data category determined by the model is inconsistent with the data category in the feedback result of the model guide, it is recorded as the model failing in the single-step game. The model's payoff is equal to the cost of interaction. Including the corresponding penalty P applied to the model, the benefit of the model guidance entity is the total system resources R minus the cost required to provide feedback to the model. When the model chooses to interact, the model's supervisor chooses to provide feedback, and the data category determined by the model matches the data category in the feedback result, the model is considered to have won in the single-step game. The model's payoff equals the total system resource R minus the cost of the interaction. The benefit of the model-guided entity equals the corresponding penalty P plus the cost of feedback. If the model chooses interaction and the model's guidance agent chooses not to provide feedback, the model wins, and its benefit is the total system resources R minus the cost of interaction. The model guide's reward is the corresponding penalty P; when the model chooses not to interact and the model guide does not actively provide feedback to the model, the rewards for both the model and the model guide are 0; when the model guide believes that the model's prediction result for a certain traffic is incorrect and actively provides feedback, but its feedback result is incorrect, the model guide's reward is... The model's return is 0; when the model guide believes the model's prediction for a certain traffic flow is incorrect, actively provides feedback, and the feedback result is correct, the model guide's return is R. The model's return is 0;
[0056] S3: Interaction rule formulation. In the interactive guided anomaly detection model based on reinforcement learning algorithm of this invention, the model receives rewards from the environment and feedback rewards from the model's supervisor. The model's interactive game process is as follows: Figure 3 As shown; the reward equation is as shown in formula R. all (s t a t )=δR A (s t a t )+(1-δ)R E (s t a t ), where R E (s t a t R represents the reward received by the model from the environment; it is a reward signal in a machine learning algorithm model. A (s t a tThe value δ represents the interactive feedback reward received by the model from the model's supervisor, which is the reward and penalty obtained by both parties in the game model, namely the model and the model supervisor (operation personnel), during the game. δ represents the weight ratio between environmental rewards and interactive feedback rewards. The value of δ is dynamically adjusted according to indicators such as the accuracy of the model's judgment results, thereby adjusting the proportion of the model supervisor's feedback reward. When the model performs well in anomaly detection, the value of δ needs to be appropriately reduced, thereby reducing the proportion of the model supervisor's feedback reward in the total model reward. When the model performs poorly, the value of δ needs to be appropriately increased, thereby increasing the proportion of the model supervisor's feedback reward in the total model reward, thereby increasing the guidance strength of the model supervisor. During training, the guidance of the model supervisor can help the anomaly detection model converge faster, and during interactive guidance, the model can dynamically change its perception of anomaly data, thereby strengthening the model's ability to infer the intention of the model supervisor.
[0057] In summary, this invention first collects network data within a local area network (LAN), then utilizes Bayesian game theory to model the model guide and the model game process, as well as specify the reward distribution rules under the interaction rules. A mathematical model of a dynamic Bayesian game between the two parties is established, enabling the interaction and feedback behaviors between the model guide and the model to reach a dynamic equilibrium. The prior beliefs of the maintenance personnel influence the model's interaction decisions, and the model's interaction behavior, through the established Nash equilibrium, reacts to the model's prior beliefs, thus achieving a dynamic equilibrium between the two parties. Ultimately, this achieves the goal of identifying abnormal traffic in network traffic based on the interactive game model. The network traffic data processing method of this invention employs an interactive game approach. The model guide provides feedback and evaluation on the model's analysis and decisions through a visual interactive interface, thereby enabling the model to continuously obtain more accurate training results during the training process using external information provided by the model guide.
[0058] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An interactive game-based network traffic anomaly detection method, characterized in that, The method comprises the following steps: Step 1: collecting network data in a local area network, establishing an analog network topology in the local area network according to the network data, constructing network attacks in the process of sending data packets between a virtual machine simulation client and a server according to the analog network topology, and collecting data packets; Step 2: training an interactive guided anomaly detection model based on reinforcement learning according to the network data, presetting game elements, and performing interactive gambling on the interactive guided anomaly detection model by using a machine learning method; Step 3: presetting interaction rules, collecting environmental rewards and interactive feedback rewards in the process of interactive gambling, training the interactive guided anomaly detection model according to the environmental rewards, the interactive feedback rewards and the interaction rules, and obtaining an adaptive anomaly detection model; Step 4: using the adaptive anomaly detection model to detect network traffic anomalies of the collected data packets, and obtaining a detection result; In step 3, the interactive guided anomaly detection model receives environmental rewards and interactive feedback rewards, calculates model rewards, trains the interactive guided anomaly detection model according to the model rewards, and the expression of the model rewards is: ； wherein, represents an environmental reward; represents an interaction feedback reward; represents a weight ratio between the environmental reward and the interaction feedback reward; During the training process, the correct rate of the model judgment result obtained by the adaptive anomaly detection model is used to dynamically adjust the weight ratio between the environmental rewards and the interactive feedback rewards. 2.The network traffic anomaly detection method based on interactive game according to claim 1, characterized in that, The game elements include game strategies and game benefits; The game strategies include a model strategy set and a model strategy probability distribution of the interactive guided anomaly detection model, and an interaction strategy set and an interaction strategy probability distribution; The model policy set is wherein represents not initiating interaction, represents initiating interaction and judging the data as abnormal data by itself, represents initiating interaction and judging the data as normal data by itself; The model policy probability distribution is , ; The interaction policy set is wherein represents no feedback, represents feedback and the feedback content is that the data category is normal data, represents feedback and the feedback content is that the data category is abnormal data; The interaction policy probability distribution is , ; The game benefits include single-step game model failure benefits, single-step game model victory benefits, model attack success benefits, game silence benefits, interaction attack failure benefits and interaction attack success benefits; Single-step game model failure reward, model reward equals the cost of interaction Plus the corresponding penalty P given to the model; interaction reward is the total system resources R minus the cost of giving the model feedback ; The single-step game model winning payoff, the model payoff is equal to the total resources R of the system minus the cost of interaction ; the interaction payoff is equal to the corresponding penalty P plus the cost of feedback required ; Model attack success revenue, model revenue is the total system resources R minus the cost of interaction required , interaction revenue is the corresponding penalty P; The game silence benefits, model benefits and interaction benefits are all 0; The operation and maintenance interaction attack failure benefit, the interaction benefit is , the model benefit is 0; The operation and maintenance interaction attack success benefit, the interaction benefit is , the model benefit is 0. 3.The network traffic anomaly detection method based on interactive game according to claim 1, characterized in that, The interactive guided anomaly detection model is a 5-layer neural network model, which includes an input layer, three hidden layers and an output layer, and full connection is used between each layer; the input layer has 12 units, all features in the data set are scaled to 12 by PCA principal component analysis; the output layer is the category of data; After the three hidden layers, the output of the model is the reward value corresponding to four actions, which is used for the convergence calculation of the Loss function of the neural network model, and the output result is the category of the current input network data, including DDoS, DoS, Port Scan and Bengin.