A method and device for detecting abnormal flow data of Internet of Things

Abstract

The present invention discloses a method for detecting anomalies in IoT traffic data, comprising the steps of: constructing an anomaly detection agent for IoT traffic data based on an anomaly detection framework of deep reinforcement learning; setting an internal reward function for learning potential anomalies in unlabeled data based on a distributed autoencoder, setting an external reward function for learning labeled data and unlabeled data based on the internal reward function, calculating a total reward value based on the internal reward value and the external reward value, and designing a reward optimization mechanism; setting an abnormal bias sampling function for selecting the state of the next stage; taking a data set containing labeled abnormal data and unlabeled data as input, generating an experience group based on the interaction between the anomaly detection agent and the environment, and training a neural network using the experience group; and performing anomaly detection on IoT traffic data using the trained neural network. The present invention can be applied to anomaly detection scenarios with a small amount of labeled data and a large amount of unlabeled data.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things data security technology, and in particular to an Internet of Things flow data anomaly detection method and device. Background Art

[0002] Industrial Internet of Things, or IIoT for short, refers to the use of sensors, devices, and other IoT devices to provide network interconnection for intelligent industrial systems to achieve data collection, data analysis, and automated control in industrial environments. It is the basis for the digital transformation of traditional manufacturing, energy, and power industries in recent years. The application of edge computing, computing offload, big data analysis, and intelligent sensing technologies has greatly promoted the development of the Industrial Internet of Things. Its functions are becoming more and more complete, and the network complexity is also increasing. With the help of various types of sensor devices and massive network data transmission, the Industrial Internet of Things can achieve the interconnection of all things, human-computer interaction, information transmission, environmental perception, and intelligent analysis. Based on the comprehensive and in-depth perception of industrial production information, massive data exchange, real-time and fast calculation, and precise modeling and analysis, it can achieve intelligent production control, operation strategy optimization, and management model innovation. However, a large number of attacks on the communication network of the Industrial Internet of Things, such as zombie network viruses, worm Trojan injection, and DDoS attacks, will cause abnormal network traffic data, which often brings huge economic losses. Therefore, analyzing network traffic to perceive network trends and detect abnormal network traffic is of great significance for improving network emergency response capabilities, timely discovering network attack behaviors, and maintaining industrial Internet of Things network security.

[0003] In actual industrial IoT scenarios, due to the large-scale nature of network traffic data, high security requirements, and limited labor costs, only a small portion of network traffic data is usually clearly labeled as abnormal or normal. These abnormal data labeled as specific types are called known anomalies. Most network traffic data do not have labels. These unlabeled data mainly consist of a large amount of normal data and a part of abnormal data of known and unknown types. These unknown types of anomalies are called unknown anomalies. Most of the existing research on network traffic anomaly detection at home and abroad uses unsupervised or semi-supervised learning methods. Some of the work ignores the exploration and utilization of potential abnormal information in unlabeled data, that is, the information of unknown abnormal data, and cannot take into account the learning of labeled and unlabeled data. At the same time, these methods cannot guarantee detection accuracy at low labor costs, and there is a gap with actual applications. Summary of the invention

[0004] In view of this, it is necessary to provide a method and device for detecting anomaly in IoT traffic data, so as to effectively solve the technical problem that the existing technology cannot take into account the learning and identification of both labeled data and unlabeled data.

[0005] The present invention provides a method for detecting anomaly in flow data of the Internet of Things, comprising the following steps:

[0006] Step S1: construct an anomaly detection agent for IoT traffic data based on an anomaly detection framework based on deep reinforcement learning;

[0007] Step S2: Based on the distributed autoencoder, an internal reward function for learning potential anomalies in unlabeled data is set; based on the internal reward function, an external reward function for learning labeled data and unlabeled data is set; a total reward value is calculated based on the internal reward value and the external reward value, and a reward optimization mechanism is designed;

[0008] Step S3, setting an abnormal bias sampling function to select the state of the next stage;

[0009] Step S4, taking a data set including labeled anomaly data and unlabeled data as input, generating an experience group based on the interaction between the anomaly detection agent and the environment, and using the experience group to train the neural network;

[0010] Step S5: Use the trained neural network to perform anomaly detection on IoT traffic data.

[0011] Preferably, the step S1 is specifically:

[0012] Define the optimal action value function, which follows the maximum expected return that can be achieved by policy π:

[0013]

[0014] Among them, Q * (s,a) is the optimal action value function, st represents the current state, at represents the current action value of the anomaly detection agent, and R t represents the total reward value, r t' is the reward value at time t', t'=t,t+1,…,T, T is the total number of moments, γ is a constant, It means returning the expected maximum total reward value achieved under strategy π;

[0015] The output of the anomaly detection agent is:

[0016]

[0017] Where Q(s,a;θ,α,β) is the output of the anomaly detection agent, s represents the state, a represents the action, θ is the convolutional layer parameter of the neural network, a' is all possible action values, V(s;θ,β) and A(s,a;θ,α) are the scalar and vector of the two fully connected layers of the dueling Q network, α and β are the parameters of the two fully connected layers, respectively. is the vector dimension.

[0018] Preferably, in step S2, based on the distributed autoencoder, an internal reward function for learning potential anomalies of unlabeled data is set, specifically:

[0019] Based on the model structure of the distribution autoencoder, a KL loss function and a reconstruction loss function are defined, a loss function for neural network training is defined based on the KL loss function and the reconstruction loss function, and the internal reward function for learning potential anomalies is set based on the loss function.

[0020] Preferably, in step S2, based on the internal reward function, an external reward function for learning the labeled data and the unlabeled data is set, specifically:

[0021] When in the labeled dataset D a When abnormal data is identified in the labeled data set D, a positive reward is given. a Give negative rewards when normal data is identified;

[0022] When the data instance belongs to the unlabeled dataset D u When the abnormality and normality of the unlabeled data are calculated based on the internal reward function, a corresponding reward is given based on the difference between the abnormality and normality;

[0023] The total reward value during training is calculated based on the intrinsic reward value and the extrinsic reward value.

[0024] Preferably, the step S2 further comprises: setting an MSR reward optimizer to optimize the total reward value, the total reward value being the sum of the internal reward value and the external reward value;

[0025] The MSR reward optimizer is set to optimize the total reward value, specifically: when the change degree between the reward of the previous state and the reward of the current state is greater than the set value, it is a positive jump, otherwise it is a negative jump; the reward for positive jumps is amplified, and the reward for negative jumps is reduced.

[0026] Preferably, the reward for positive jumps is amplified, and the reward for negative jumps is reduced, specifically:

[0027] Define the change degree parameter used to measure the change degree between the current state reward and the previous state reward:

[0028]

[0029] Among them, ρ is the variation parameter, r t is the current state reward, r t-1 is the previous state reward, r t '=r t +σ,r t-1 '=r t-1 +σ, σ is a constant;

[0030] Set a monotonically increasing bounded function to amplify the current reward:

[0031]

[0032] Among them, f(x) is a bounded function, x = ρ + λ, ρ is the degree of change parameter, λ = sgn(r t -r t-1 ), sgn() is a step function, r t is the current state reward, r t-1 is the reward of the previous state, η is the threshold of change degree;

[0033] The total reward value is optimized based on the bounded function:

[0034] r t * =r t '+(f(x)-λ)*|r t '|;

[0035] Among them, r t * is the total reward value after optimization.

[0036] Preferably, the step S3 is specifically:

[0037] The abnormal bias sampling function includes a first sampling function and a second sampling function; the first sampling function and the second sampling function are respectively adopted with corresponding probabilities;

[0038] The first sampling function is used to sample a piece of data from the labeled data set as the state of the next stage, and the probability of each piece of data in the labeled data set being sampled is equal;

[0039] The second sampling function selects the next stage state based on the action value of the anomaly detection agent: when the data is judged as abnormal data by the anomaly detection agent, the data with the smallest Euclidean distance to the current data is selected as the state of the next stage; when the data is judged as normal data by the anomaly detection agent, the data with the largest Euclidean distance to the current data is selected as the state of the next stage.

[0040] Preferably, the step S4 is specifically:

[0041] Input the data samples in the data set into the anomaly detection agent to output the action value, combine the current state, the next stage state selected by the anomaly bias sampling function, the optimized total reward value, and the action value of the anomaly detection agent to form an experience group, and set an experience replay pool to store the experience group;

[0042] A random sampling function is set to sample the experience group in the experience replay pool, and the neural network is trained through the sampled experience group. The neural network updates parameters by iteratively minimizing the loss value.

[0043] Preferably, the step S5 is specifically:

[0044] The IoT traffic data is preprocessed and input into the trained neural network to obtain the corresponding network output value, and then it is judged whether the IoT traffic data is abnormal data according to the network output value.

[0045] The present invention also provides an Internet of Things traffic data anomaly detection device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the Internet of Things traffic data anomaly detection method is implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects: the present invention adopts a weakly supervised anomaly detection method, that is, based on a deep reinforcement learning framework, designs an anomaly detection agent based on Dueling DQN to interact with the environment, and realizes the detection of known anomalies and unknown anomalies when there is only a small part of labeled data. At the same time, a reward mechanism based on a distributed autoencoder is introduced, which comprehensively considers the potential anomalies of labeled abnormal data and unlabeled data, improves the ability of the model to identify anomalies, and improves the detection accuracy; finally, an abnormal bias sampling function that is more inclined to select abnormal data as the next stage state is set to select the next state. The above method solves the existing problems of anomaly detection of network traffic data, achieves the balance between labeled data and unlabeled data, and has a good guiding reference value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. Among them:

[0048] Figure 1 A flowchart of an embodiment of a method for detecting abnormality in Internet of Things traffic data provided by the present invention;

[0049] Figure 2 yes Figure 1 Application scenario diagram of the embodiment in the embodiment;

[0050] Figure 3 yes Figure 1 Detection principle diagram of the embodiment in FIG.

[0051] Figure 4a yes Figure 1 A comparison chart of the AUC-PR index when the unsw-analysis data set is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0052] Figure 4b yes Figure 1 A comparison chart of the AUC-PR index when the unsw-backdoors dataset is detected by the embodiment and the prior art respectively as the pollution rate of the unlabeled dataset changes;

[0053] Figure 4c yes Figure 1 A comparison chart of the AUC-PR index when the unsw-dos data set is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0054] Figure 4d yes Figure 1 A comparison chart of the AUC-PR index when the unsw-exploits dataset is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled dataset changes;

[0055] Figure 4e yes Figure 1 A comparison chart of the AUC-PR index when the unsw-fuzzers data set is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0056] Figure 4f yes Figure 1 A comparison chart of the AUC-PR index when the unsw-generic data set is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0057] Figure 4g yes Figure 1 A comparison chart of the AUC-PR index when the unsw-reconnaissance data set is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0058] Figure 4h yes Figure 1 A comparison chart of the AUC-PR index when the data set nslkdd-dos is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0059] Figure 4i yes Figure 1 A comparison chart of the AUC-PR index when the data set nslkdd-probe is detected by the embodiment and the prior art respectively as the contamination rate of the unlabeled data set changes;

[0060] Figure 5a yes Figure 1 A comparison chart of the AUC-PR index when the data set nslkdd-dos is detected by the embodiment and the prior art respectively as the number of anomalies is changed;

[0061] Figure 5b yes Figure 1 A comparison chart of the AUC-PR index and the number of anomalies when the data set nslkdd-probe is detected in the embodiment and the prior art respectively;

[0062] Figure 6a yes Figure 1 A comparison chart showing the change in the AUC-PR index as the number of anomaly types is changed when the unsw-analysis data set is detected in the embodiment and the prior art;

[0063] Figure 6b yes Figure 1 A comparison chart of the AUC-PR index when the unsw-backdoors dataset is detected by the embodiment and the prior art respectively as the number of anomaly types is marked;

[0064] Figure 6c yes Figure 1 A comparison chart of the AUC-PR index when the unsw-dos data set is detected by the embodiment and the prior art respectively as the number of anomaly types is changed;

[0065] Figure 6d yes Figure 1 A comparison chart showing the change in the AUC-PR index as the number of anomaly types is changed when the unsw-exploits data set is detected by the embodiment and the prior art respectively;

[0066] Figure 6e yes Figure 1 A comparison chart showing the change of the AUC-PR index with the number of anomaly types marked when the unsw-fuzzers data set is detected by the embodiment and the prior art respectively;

[0067] Figure 6f yes Figure 1 A comparison chart of the AUC-PR index when the unsw-generic dataset is detected by the embodiment and the prior art as the number of anomaly types is changed;

[0068] Figure 6g yes Figure 1A comparison chart showing the change in the AUC-PR index as the number of anomaly types is labeled when the unsw-reconnaissance dataset is detected in the embodiment and the prior art respectively. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0070] Embodiment 1

[0071] See also Figure 1 In this embodiment, a method for detecting abnormality in IoT traffic data includes the following steps:

[0072] Step S1: construct an anomaly detection agent for IoT traffic data based on an anomaly detection framework based on deep reinforcement learning;

[0073] Step S2: Based on the distributed autoencoder, an internal reward function for learning potential anomalies in unlabeled data is set; based on the internal reward function, an external reward function for learning labeled data and unlabeled data is set; a total reward value is calculated based on the internal reward value and the external reward value, and a reward optimization mechanism is designed;

[0074] Step S3, setting an abnormal bias sampling function to select the state of the next stage;

[0075] Step S4, taking a data set including labeled anomaly data and unlabeled data as input, generating an experience group based on the interaction between the anomaly detection agent and the environment, and using the experience group to train the neural network;

[0076] Step S5: Use the trained neural network to perform anomaly detection on IoT traffic data.

[0077] This embodiment discloses a method for detecting anomalies in IoT traffic data based on deep reinforcement learning, the main purpose of which is to identify potential network attacks and ensure the security and reliability of the industrial IoT network environment. According to the actual scenarios of the industrial IoT, network traffic data usually exhibits static properties and contains multiple features, of which only a small part of the data has been labeled as normal or abnormal. These abnormal data labeled as specific categories are regarded as known anomalies. On the other hand, large-scale unlabeled data includes known anomaly classes and new anomaly classes. These unknown new types of anomalies are called unknown anomalies.

[0078] The application scenarios of this embodiment are as follows Figure 2As shown, only a small part of the network traffic data is labeled, and the rest is a large amount of unlabeled data. This embodiment detects abnormal data by performing anomaly detection on abnormal data, detects abnormal data, realizes identification of network attacks, and ensures the reliability of traffic data.

[0079] like Figure 3 As shown, the present invention mainly consists of two modules: an anomaly detection environment E and an anomaly detection agent A based on Dueling DQN.

[0080] Specifically, the anomaly detection environment E includes an anomaly bias sampling function g, an external reward function h, an internal reward function f, and an MSR reward optimizer M. Among them, the anomaly detection environment E selects the next moment state s through the anomaly bias sampling function g t+1 , the abnormal bias sampling function g includes g a and g u The internal reward function f is used to evaluate the unlabeled data D u The potential anomalies in the learning are used to obtain the internal reward value r t i The internal reward function f is based on the distributed autoencoder VAE, which includes the encoder network, mean vector, standard deviation vector, sampling potential vector and decoder network. The external reward function h is used to annotate the anomaly data D a Study and strengthen D u Learning potential anomalies in the , and obtaining the external reward value r t e The MSR reward optimizer M for the current state s t The total reward r t =r t i +r t e Optimize to generate the final optimized reward r t * The anomaly detection environment E achieves efficient learning of anomaly data through the reward optimization mechanism combining f, h and M, and converts the obtained experience group (s t ,a t ,r t * ,s t+1 ) is stored in the experience pool for interaction with A.

[0081] The anomaly detection agent A is used to select the given state s at the current moment t The best action t The neural network of the anomaly detection agent A includes a Q network and a target Q network. The output of the Q network is Q(s t ,a t;θ,α,β), the target Q network output is Q(s t+1 ,a';θ - ,α - ,β - ), the optimal Q network can be obtained through neural network parameter update and back propagation. This network is used to estimate the abnormality of the data. The network parameter update is iteratively updated through the loss function L(θ).

[0082] The anomaly detection environment E interacts with the anomaly detection agent A and uses the action value a selected by A t , the experience group is stored in the experience pool, and the random sampling batch function is used for sampling to obtain the minimum batch experience group for training A.

[0083] Under the premise of considering the security, low cost, and large-scale requirements of industrial Internet of Things data transmission, this embodiment adopts a weakly supervised anomaly detection framework based on deep reinforcement learning, constructs Dueling deep Q learning as an anomaly detection agent, and the anomaly detection agent interacts with the environment to learn abnormal network traffic data patterns. In the case of only a small part of labeled data, the detection of known anomalies and unknown anomalies is achieved; the internal reward function is designed to use a distributed autoencoder to learn potential anomalies in unlabeled data, and the external reward function is designed to learn labeled and unlabeled data at the same time, comprehensively considering the anomalies of labeled data and the potential anomalies of unlabeled data, and construct a reward optimization mechanism, which improves the model's ability to identify anomalies and improves detection accuracy; an abnormally biased sampling function design method is used to select the next state. The above measures solve the existing problems of network traffic data anomaly detection and have good guiding reference value.

[0084] Furthermore, the step S1 is specifically:

[0085] Define the optimal action value function, which follows the maximum expected return that can be achieved by policy π:

[0086]

[0087] Among them, Q * (s,a) is the optimal action value function, st represents the current state, at represents the current action value of the anomaly detection agent, and R t represents the total reward value, r t' is the reward value at time t', t'=t, t+1, ..., T, T is the total number of time, γ is a constant, which is set to 0.99 in the present invention, It means returning the expected maximum total reward value achieved under strategy π;

[0088] Step S1 is used to design an anomaly detection agent, which is used to select action values. The network structure of Dueling Deep Q Learning (Dueling DQN) is constructed. The goal of the anomaly detection agent is to learn the optimal action value function. The optimal action value function is the maximum expected return that can be achieved by following the strategy π. The optimal action value function also follows the Bellman equation, which is as follows:

[0089]

[0090] Where s' is the state of a sequence, a' is the value of all possible actions, r is the current reward, and E represents the mean.

[0091] The output of the anomaly detection agent is:

[0092]

[0093] Where Q(s,a;θ,α,β) is the output of the anomaly detection agent, s represents the state, a represents the action, θ is the convolutional layer parameter of the neural network, a' is all possible action values, V(s;θ,β) and A(s,a;θ,α) are the scalar and vector of the two fully connected layers of the dueling Q network, α and β are the parameters of the two fully connected layers, respectively. is the vector dimension.

[0094] All possible states constitute the state space, which is defined as the data instance of the training data set D. The state s at each moment t It is selected by the abnormal bias sampling function g at the corresponding time t. The action space is defined as [a 0 ,a 1 ], where a 0 Indicates state s t The action value when marked as normal data, a 1 Indicates state s t The action value when an instance of data is marked as an anomaly.

[0095] At each step, choose action a t Make the current state s t Output Q(s t ,a t θ j ,α j ,β j ) is maximized. When evaluating the test data, * (s,a;θ * ,α * ,β * ) value, that is, data with high anomaly scores are considered to be abnormally biased data, where θ * , α* and β * are the parameters of the neural network after training.

[0096] Furthermore, in step S2, based on the distributed autoencoder, an internal reward function for learning potential anomalies in unlabeled data is set, specifically:

[0097] Based on the model structure of the distributed autoencoder, the KL loss function and reconstruction loss function are defined:

[0098]

[0099] L recon =MSE(x recon ,x)

[0100] Among them, L KL represents the KL loss function, KL(N(μ,σ 2 )||N(0,1)) means to calculate the normal distribution of the original data N(μ,σ 2 ) and the expected value of the logarithmic difference between the standard normal distribution N(0,1), μ and σ are the mean and variance of the probability distribution of the original data, respectively, L recon represents the reconstruction loss function, MSE(x recon ,x) represents the reconstructed sample x recon The average square error between the original sample x and the original sample x is expressed in the present invention using the reconstruction loss function L recon Indicates the degree of abnormality of the data, L recon The higher the value, the more the data tends to be abnormal data;

[0101] In order to continuously increase L during training recon The value of , based on the KL loss function and the reconstruction loss function, defines the loss function for neural network training:

[0102] L=-L recon +L KL

[0103] Where L is the loss function used for neural network training;

[0104] The reconstruction loss function is used to indicate the degree of abnormality of the data. The higher the value of the reconstruction loss function, the more the data tends to be abnormal data. The internal reward function is designed based on the reconstruction loss function:

[0105] r t i =f(s t )=L recon (s t )

[0106] Among them, rt i is the internal reward value, f(s t ) is the internal reward function, L recon (s t ) represents the reconstruction loss function value at state st.

[0107] Furthermore, in step S2, based on the internal reward function, an external reward function for learning the labeled abnormal data and the unlabeled data is set, specifically:

[0108] When in the labeled dataset D a When abnormal data is identified in the labeled data set D, a positive reward is given. a Give negative rewards when normal data is identified;

[0109] When the data instance belongs to the unlabeled dataset D u When the abnormality and normality of the unlabeled data are calculated based on the internal reward function, a corresponding reward is given based on the difference between the abnormality and normality;

[0110] The external reward function is specifically expressed as:

[0111]

[0112] Among them, r t e is the external reward value, h(s t ,a t ,r t i ) is the external reward function, a t =a 1 It indicates that the anomaly detection agent identifies an abnormal action value, a t =a 0 It indicates that the anomaly detection agent recognizes the normal action value, s t ∈D a Represents data t The labeled dataset D a The labeled data in s t ∈D u Represents data t The unlabeled dataset D u The unlabeled data in t Indicates the abnormality of unlabeled data, l t =r t i , r t i is the internal reward value, l′ t Indicates the normality of unlabeled data, l′ t =1-rt i .

[0113] Design an external reward function to achieve learning for labeled anomaly data and strengthen learning for unlabeled anomaly data.

[0114] Calculate the total reward value during training Internal reward value and external reward value sum.

[0115] The reward acts as a feedback signal to help the anomaly detection agent learn and improve its decision-making strategy to maximize the reward. The internal reward drives the anomaly detection agent to learn potential anomalies in the unlabeled dataset. The external reward encourages the anomaly detection agent to automatically and interactively learn known anomalies from the labeled dataset, and at the same time amplify the learning of unknown anomalies in the unlabeled dataset based on the internal reward, thereby strengthening the detection and identification of unknown anomalies.

[0116] Furthermore, the step S2 further includes: setting an MSR reward optimizer to optimize the total reward value, where the total reward value is the sum of the internal reward value and the external reward value;

[0117] The MSR reward optimizer is set to optimize the total reward function, specifically: when the change degree between the reward of the previous state and the reward of the current state is greater than the set value, it is a positive jump, otherwise it is a negative jump; the reward for positive jumps is amplified, and the reward for negative jumps is reduced.

[0118] Furthermore, the rewards for positive jumps are amplified, and the rewards for negative jumps are reduced, specifically:

[0119] Define the change degree parameter used to measure the change degree between the current state reward and the previous state reward:

[0120]

[0121] Among them, ρ is the variation parameter, r t is the current state t The reward, r t-1 is the previous state s t-1 The reward, r t '=r t +σ,r t-1 '=r t-1 +σ, σ is a constant close to 0;

[0122] Set a monotonically increasing bounded function to amplify the current reward:

[0123]

[0124] Among them, f(x) is a bounded function, x = ρ + λ, ρ is the degree of change parameter, λ = sgn(r t -r t-1 ), sgn() is a step function, r t is the current state reward, r t-1 is the reward of the previous state, η is the threshold of change degree;

[0125] If x>η or x<-η, the total reward value is optimized based on the bounded function:

[0126] r t * =r t '+(f(x)-λ)*|r t '|;

[0127] Among them, r t * is the total reward value after optimization.

[0128] This embodiment adopts the concept of reward jumping. The phenomenon that the reward of the previous state is very different from the current reward is defined as reward jumping. The current reward is considered to be a positive jump when it is good, and it is considered to be a negative jump when it is poor. Experiments show that reward jumping also exists in the anomaly detection task using deep reinforcement learning. For this reason, the MSR algorithm is used to design a reward optimizer to dynamically adjust the current total reward. Specifically, the total reward is amplified when a positive reward jump occurs, and the total reward is reduced when a negative jump occurs.

[0129] Specifically, define the change degree parameter ρ to measure the current reward and the reward at the previous moment, and design a bounded function to ensure the appropriate amplification of the current reward: set a threshold η, if x>η or x<-η, optimize the current total reward and finally obtain the optimized reward.

[0130] This embodiment designs an MSR reward optimizer to amplify reward jumps, fully utilizes reward change information, appropriately optimizes rewards, and improves the model's ability to learn anomalies.

[0131] Furthermore, the step S3 is specifically as follows:

[0132] The abnormal bias sampling function includes a first sampling function and a second sampling function; the first sampling function and the second sampling function are respectively adopted with corresponding probabilities;

[0133] The first sampling function is used to sample a piece of data from the labeled data set as the state of the next stage, and the probability of each piece of data in the labeled data set being sampled is equal;

[0134] The second sampling function selects the next stage state based on the action value of the anomaly detection agent: when the data is judged as abnormal data by the anomaly detection agent, the data with the smallest Euclidean distance to the current data is selected as the state of the next stage; when the data is judged as normal data by the anomaly detection agent, the data with the largest Euclidean distance to the current data is selected as the state of the next stage;

[0135] The abnormal bias sampling function is specifically:

[0136] s t+1 =g a (D a )~U(D a )

[0137]

[0138] Among them, s t+1 is the state of the next stage selected by the abnormal bias sampling function, g a (D a ) is the first sampling function, U(D a ) represents a random a The data samples collected in the dataset D have been labeled a The probability of each data sample being sampled is equal, g u (s t ,a t ,θ e ) is the second sampling function, Indicates the current state of distance s t The data sample with the smallest Euclidean distance is Indicates the current state of distance s t The data sample with the largest Euclidean distance, d(s t ,s,θ e ) function is used to calculate the current state s t The Euclidean distance between the state s and other states, s∈S, S is the set of other states, a t represents the action value of the anomaly detection agent, a t =a 1 Indicates that the data is judged as abnormal data by the anomaly detection agent, a t =a 0 Indicates that the data is judged as normal data by the anomaly detection agent.

[0139] Euclidean distance: measures the absolute distance between two points or vectors in multidimensional space, that is, the square root of the difference between the vectors. i ,y i ) to point B(x j ,y j ) is

[0140] Design an anomaly-biased sampling function, select the state of the next stage, guide the anomaly detection agent to make the best decision in the current state, and achieve continuous learning and strategy optimization through interaction with the environment. In order to achieve a balance between development and exploration of the entire dataset, two sampling functions g a (D a ) and g u (s t ,a t ,θ e ) have a 50% probability of being used. Through the above design, it is ensured that the next state selected is more likely to be abnormal data, which strengthens the model's learning of abnormal data.

[0141] Furthermore, the step S4 is specifically as follows:

[0142] Input the data samples in the dataset into the anomaly detection agent to output the action value a t , combined with the current state s t , the next stage state s selected by the abnormal bias sampling function t+1 , the total reward value after optimization The action value a of the anomaly detection agent t Form an experience group Setting an experience replay pool for storing the experience group;

[0143] Setting a random sampling function to sample the experience group in the experience replay pool, and training the neural network through the sampled experience group;

[0144] The neural network updates its parameters by iteratively minimizing the loss value. The loss function is defined as:

[0145] L j (θ j , α j , β j )=E (s，a，r，s′) [(y i -Q(s,a;θ j , α j , β j )) 2 ]

[0146]

[0147] Among them, L j (θ j ,α j ,β j ) is the loss function, (s,a,r,s′) is an experience group in the experience replay pool, y iis the target network value, r is the total reward value, γ is the adjustment coefficient, and a′ is the next state action value. The Dueling DQN network structure consists of two neural networks, one is the main network and the other is the target network. is the target network output, Q(s,a;θ j ,α j ,β j ) is the main neural network output, parameter is the target network parameter, θ j , α j , β j are the main neural network parameters.

[0148] Set up an experience replay pool to store the current state, next stage state, current reward, and current action value as a set of interactive information. Set up a random sampling strategy to sample the minimum batch for training the neural network. The experience replay pool stores a set of interactive information at each step. Use a small batch of experience groups randomly extracted from the experience replay pool to train the neural network.

[0149] Specifically, the first 10,000 steps are set as the warm-up phase. For each data instance, the action value is calculated, the total reward is calculated, the next stage state is selected, the neural network parameters are updated, and the experience group obtained at each step is stored in the experience replay pool. After the 10,000-step warm-up phase, the minimum batch of data is randomly sampled from the experience replay pool for neural network training, the neural network parameters are updated, and the environment is interacted with to strengthen the learning of abnormal data. At the same time, new experience groups are continuously added to the experience replay pool.

[0150] Furthermore, the step S5 is specifically as follows:

[0151] The IoT traffic data is preprocessed. Specifically, one-hot encoding is used to convert non-numeric data into numerical data, and the maximum and minimum normalization method is used to normalize all data. After preprocessing, the preprocessed IoT traffic data is input into the trained neural network to obtain the corresponding network output value. According to the network output value, it is judged whether the IoT traffic data is abnormal data.

[0152] The trained neural network is used as an anomaly detection model, and the corresponding optimal action value function value, referred to as Q value, is given for each test set data. Data with high Q value is judged as abnormal data, thereby completing anomaly detection of network traffic data.

[0153] In general, compared with the prior art, this embodiment has the following beneficial effects:

[0154] 1. Low cost. It only relies on a small amount of labeled abnormal data and a large amount of unlabeled data for training, which greatly reduces the cost of manually labeling data labels, while ensuring the accuracy of anomaly detection and reducing the false positive rate;

[0155] 2. Strong anomaly learning ability. The model can fully learn the anomaly data and achieve better performance improvement when the number of anomaly labels increases. It can also identify both known and unknown anomalies.

[0156] 3. Strong stability. By strengthening the learning of potential outliers in unlabeled data, the stability of the model is improved. When the unlabeled data is affected by noise, it can still maintain good anomaly detection performance.

[0157] In order to further verify the detection effect of the present invention, the present invention and several commonly used detection methods in the prior art are used to detect different data sets respectively. By changing the contamination rate of the unlabeled data set, the number of anomalies in the labeled data set, and the types of abnormal data in the labeled data set, the detection results are compared using the AUC-PR indicator to compare the impact of the changes in the parameters on the AUC-PR indicator. The detection method provided by the present invention is referred to as WAND, and several detection methods in the prior art are DPLAN, DevNet, FEAWAD, and Prenet. In the accompanying drawings, WAND is an example of the present invention, DPLAN, DevNet, FEAWAD, and Prenet are four comparison methods, and WAND * The variants of the present invention were used for control experiments.

[0158] Specifically, Figure 4a , Figure 4b , Figure 4c , Figure 4d , Figure 4e , Figure 4f , Figure 4g , Figure 4h , Figure 4i It is a schematic diagram of the comparison of AUC-PR indicators for the detection of the data sets unsw-analysis, unsw-backdoors, unsw-dos, unsw-exploits, unsw-fuzzers, unsw-generic, unsw-reconnaissance, nslkdd-dos, and nslkdd-probe. The horizontal axis is the contamination rate and the vertical axis is the AUC-PR value.

[0159] Figure 5a , Figure 5bThe following are schematic diagrams showing the comparison of the AUC-PR indicators for the nslkdd-do and nslkdd-probe datasets. The horizontal axis is the number of labeled anomalies, and the vertical axis is the AUC-PR value.

[0160] Figure 6a , Figure 6b , Figure 6c , Figure 6d , Figure 6e , Figure 6f , Figure 6g Schematic diagram of the comparison of AUC-PR indicators for detecting the datasets unsw-analysis, unsw-backdoors, unsw-dos, unsw-exploits, unsw-fuzzers, unsw-generic, and unsw-reconnaissance; the horizontal axis is the number of labeled abnormal data types known classes, and the vertical axis is the AUC-PR value.

[0161] The above comparison proves the versatility of this method, which can maintain good performance under the influence of abnormal data types, quantities and unlabeled data contamination rate.

[0162] Embodiment 2

[0163] This embodiment provides an IoT traffic data anomaly detection device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the IoT traffic data anomaly detection method described in Embodiment 1 is implemented.

[0164] The IoT traffic data anomaly detection device provided in this embodiment is used to implement the IoT traffic data anomaly detection method. Therefore, the IoT traffic data anomaly detection method has the same technical effects as the IoT traffic data anomaly detection method, which will not be repeated here.

[0165] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present invention should be covered by the present invention.

Claims

1. A method for detecting anomaly in Internet of Things traffic data, characterized in that: The following steps are involved: Step S1: construct an anomaly detection agent for IoT traffic data based on an anomaly detection framework based on deep reinforcement learning; Step S2: Based on the distributed autoencoder, an internal reward function for learning potential anomalies in unlabeled data is set; based on the internal reward function, an external reward function for learning labeled data and unlabeled data is set; a total reward value is calculated based on the internal reward value and the external reward value, and a reward optimization mechanism is designed; Step S3, setting an abnormal bias sampling function to select the state of the next stage; Step S4, taking a data set including labeled abnormal data and unlabeled data as input, generating an experience group based on the interaction between the anomaly detection agent and the environment, and using the experience group to train the neural network; Step S5: Use the trained neural network to perform anomaly detection on IoT traffic data; In step S2, based on the distributed autoencoder, an internal reward function for learning potential anomalies of unlabeled data is set, specifically: Based on the model structure of the distribution autoencoder, a KL loss function and a reconstruction loss function are defined, a loss function for neural network training is defined based on the KL loss function and the reconstruction loss function, and the internal reward function for learning potential anomalies is set based on the loss function.

2. The method for detecting anomaly in Internet of Things traffic data according to claim 1, characterized in that: The step S1 is specifically: Define the optimal action value function, which follows the maximum expected return that can be achieved by policy π: Among them, Q * (s,a) is the optimal action value function, s t Indicates the current state, a t represents the current action value of the anomaly detection agent, R t represents the total reward value, rt' is the reward value at time t', t'=t,t+1,…,T, T is the total number of times, γ is a constant, It means returning the expected maximum total reward value achieved under strategy π; The output of the anomaly detection agent is: Where Q(s,a;θ,α,β) is the output of the anomaly detection agent, s represents the state, a represents the action, θ is the convolutional layer parameter of the neural network, a' is all possible action values, V(s;θ,β) and A(s,a;θ,α) are the scalar and vector of the two fully connected layers of the dueling Q network, α and β are the parameters of the two fully connected layers, respectively. is the vector dimension.

3. The method for detecting anomaly in Internet of Things traffic data according to claim 1, characterized in that: In step S2, based on the internal reward function, an external reward function for learning the labeled data and the unlabeled data is set, specifically: When in the labeled dataset D a When abnormal data is identified in the labeled data set D, a positive reward is given. a Give negative rewards when normal data is identified; When the data instance belongs to the unlabeled dataset D u When the abnormality and normality of the unlabeled data are calculated based on the internal reward function, a corresponding reward is given based on the difference between the abnormality and normality; The total reward value during training is calculated based on the intrinsic reward value and the extrinsic reward value.

4. The method for detecting anomaly in Internet of Things traffic data according to claim 1, characterized in that: The step S2 further includes: setting an MSR reward optimizer to optimize the total reward value, where the total reward value is the sum of the internal reward value and the external reward value; The MSR reward optimizer is set to optimize the total reward value, specifically: when the change degree between the reward of the previous state and the reward of the current state is greater than the set value, it is a positive jump, otherwise it is a negative jump; the reward for positive jumps is amplified, and the reward for negative jumps is reduced.

5. The method for detecting anomaly in Internet of Things traffic data according to claim 4, characterized in that: The rewards for positive jumps are amplified, and the rewards for negative jumps are reduced, specifically: Define the change degree parameter used to measure the change degree between the current state reward and the previous state reward: Among them, ρ is the variation parameter, r t is the current state reward, r t-1 is the previous state reward, r t '=r t +σ,r t-1 '=r t-1 +σ, σ is a constant; Set a monotonically increasing bounded function to amplify the current reward: Among them, f(x) is a bounded function, x = ρ + λ, ρ is the degree of change parameter, λ = sgn(r t -r t-1 ), sgn() is a step function, r t is the current state reward, r t-1 is the reward of the previous state, η is the threshold of change degree; The total reward value is optimized based on the bounded function: r t * =r t '+(f(x)-λ)*|r t '|; Among them, r t * is the total reward value after optimization.

6. The method for detecting anomaly in Internet of Things traffic data according to claim 1, characterized in that: The step S3 is specifically as follows: The abnormal bias sampling function includes a first sampling function and a second sampling function; the first sampling function and the second sampling function are respectively adopted with corresponding probabilities; The first sampling function is used to sample a piece of data from the labeled data set as the state of the next stage, and the probability of each piece of data in the labeled data set being sampled is equal; The second sampling function selects the next stage state based on the action value of the anomaly detection agent: when the data is judged as abnormal data by the anomaly detection agent, the data with the smallest Euclidean distance to the current data is selected as the state of the next stage; when the data is judged as normal data by the anomaly detection agent, the data with the largest Euclidean distance to the current data is selected as the state of the next stage.

7. The method for detecting anomaly in Internet of Things traffic data according to claim 1, characterized in that: The step S4 is specifically as follows: Input the data samples in the data set into the anomaly detection agent to output the action value, combine the current state, the next stage state selected by the anomaly bias sampling function, the optimized total reward value, and the action value of the anomaly detection agent to form an experience group, and set an experience replay pool to store the experience group; A random sampling function is set to sample the experience group in the experience replay pool, and the neural network is trained through the sampled experience group. The neural network updates parameters by iteratively minimizing the loss value.

8. The method for detecting anomaly in Internet of Things traffic data according to claim 1, characterized in that: The step S5 is specifically as follows: The IoT traffic data is preprocessed and input into the trained neural network to obtain the corresponding network output value, and then it is judged whether the IoT traffic data is abnormal data according to the network output value.

9. An IoT traffic data anomaly detection device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method for detecting anomaly in Internet of Things traffic data as described in any one of claims 1 to 8 is implemented.

Images