A method and device for detecting abnormal data of Internet of Things equipment

Through the neural network method combined with Markov decision-making process and deep deterministic strategy gradient algorithm, the problem of identifying known, unknown and new anomaly types in the data abnormality detection of IoT devices is solved, and efficient abnormality detection is achieved to reduce errors Reporting underreports has improved the quality of IoT services.

CN118378192BActive Publication Date: 2024-11-22HUAZHONG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410661325.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-27
Publication Date
2024-11-22
Estimated Expiration
2044-05-27

AI Technical Summary

Technical Problem

It is difficult to effectively identify known, unknown and new abnormal data in IoT device data abnormality detection, and existing methods rely heavily on known abnormal data, resulting in false positives and missed reports.

Method used

Using Markov decision-making process framework and deep deterministic strategy gradient algorithm, the interactive process between agents and the environment is constructed, data anomaly detection is performed through neural networks, and the combination of labeled and unlabeled data sets is used for training to achieve known and unknown , joint detection of new abnormal type data.

Benefits of technology

It improves the generalization and accuracy of data anomaly detection, reduces false positives and missed reports, and can detect abnormalities on large-scale data sets under a small number of labels, improving the quality and detection accuracy of IoT services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118378192B_ABST
    Figure CN118378192B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for detecting data anomalies of an Internet of Things device, wherein the method comprises the following steps: step S1, collecting historical data of Internet of Things devices to establish an Internet of Things data set, dividing the Internet of Things data set into a first subset with annotated anomaly types and a second subset without annotations; step S2, constructing an interaction process between an intelligent agent and an environment for data anomaly detection according to a Markov decision process framework, and designing environmental states, actions, state transfer functions, and reward functions; step S3, constructing a neural network for data anomaly detection based on a deep deterministic policy gradient algorithm framework; step S4, generating experience tuples by using the interaction between the intelligent agent and the environment, and training the neural network by using the experience tuples; step S5, using the trained neural network to perform anomaly detection on Internet of Things device data. The present invention can realize the joint detection of known, unknown, and new types of anomaly data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things data detection, and in particular to a method and device for detecting anomaly in Internet of Things device data. Background Art

[0002] IoT abnormal data detection aims to identify information that is uncommon or significantly different from most data. With the large-scale application of smart IoT, the scale of its equipment and production data are growing exponentially. The abnormal data collected by IoT devices may cause security vulnerabilities in IoT services. Therefore, in order to avoid the consequences of the resulting adverse consequences, effective detection of abnormal data is crucial to ensure the credibility of data and the normal operation of IoT services.

[0003] The difficulties of data anomaly detection in IoT devices are mainly reflected in three aspects. First, in actual IoT applications, due to the complexity of the network environment and the difficulty of data collection, many abnormal data often do not have clear labels, or only have a very small number of labels; second, various applications in the IoT environment generate massive data, and the types of abnormal data are numerous, making it difficult to obtain labels for all abnormal data types; third, when the known abnormal data types are incomplete, the existing abnormal data detection methods rely heavily on the supervisory signals of the known abnormal data, which leads to overfitting of known specific types of abnormal data, resulting in false positives or omissions of abnormal data. Summary of the invention

[0004] In view of this, it is necessary to provide a method and device for detecting anomaly in IoT device data to effectively solve the technical problem that known anomaly types, unknown anomaly types and new anomaly types of data cannot be jointly detected.

[0005] The present invention provides a method for detecting anomaly in data of an Internet of Things device, comprising the following steps:

[0006] Step S1, collecting historical data of IoT devices to establish an IoT data set, and dividing the IoT data set into a first subset with annotated abnormal types and a second subset without annotations;

[0007] Step S2: Based on the Markov decision process framework, construct the interaction process between the agent and the environment for data anomaly detection, and design the environment state, action, state transfer function and reward function;

[0008] Step S3: constructing a neural network for data anomaly detection based on a deep deterministic policy gradient algorithm framework;

[0009] Step S4, using the intelligent agent to interact with the environment to generate an experience tuple, and using the experience tuple to train the neural network;

[0010] Step S5: Use the trained neural network to perform anomaly detection on IoT device data.

[0011] Preferably, the step S2 is specifically:

[0012] Setting a state space, wherein the state space is set based on all data samples in the IoT data set, each data sample in the IoT data set represents an environmental state, and all environmental states constitute the state space;

[0013] Set the action of the agent. The action space of the agent is a continuous space [-1,1]. When the action value is greater than or equal to 0, the current environment state is judged as abnormal data. When the action value is less than 0, the current state is judged as normal data.

[0014] Setting a state transfer function, wherein the state transfer function includes a first state transfer function for the first subset and a second state transfer function for the second subset;

[0015] The first state transfer function is a probability function of uniform distribution. Randomly collect a data sample from the first subset as the environment state at the next moment, and the probability of each data sample in the first subset being sampled is equal;

[0016] The second state transfer function samples the environment state at the next moment from the second subset according to the current environment state and the current action. When the agent determines that the current environment state is normal data, the second state transfer function uses the data sample with the largest Euclidean distance from the current environment state as the environment state at the next moment. When the agent determines that the current environment state is abnormal data, the second state transfer function uses the data sample with the smallest Euclidean distance from the current environment state as the environment state at the next moment.

[0017] Setting a reward function, wherein the reward function includes an external reward function for prompting the agent to learn using labeled abnormal data and an internal reward function for prompting the agent to learn using unlabeled data; setting a balance coefficient, wherein the balance coefficient is used to balance the size of the external reward and the internal reward;

[0018] The external reward function provides feedback by coupling the agent action value, and the internal reward function provides feedback by calculating the difference between the current state and the set of states that have appeared before.

[0019] Preferably, the step S3 is specifically:

[0020] The network structure of the deep deterministic policy gradient algorithm framework includes two parts: an Actor network and a Critic network. The Actor network is used to output a certain action, and the Critic network is used to output an evaluation value of the action output by the Actor network in the current state. Both the Actor network and the Critic network are provided with a target network.

[0021] Preferably, the step S4 is specifically:

[0022] Batch extract experience tuples and input them into the Actor network and the Critic network respectively, and calculate the loss function value, and use the loss function value to perform gradient update on the neural network parameters:

[0023] The parameter update of the Actor network follows the following loss function:

[0024]

[0025] The parameter update of the Critic network follows the following loss function:

[0026]

[0027] Where J is the loss function of the Actor network, B represents the number of experience tuples extracted from the experience replay pool for network parameter update, i = 1, 2, ..., B, ▽ a Indicates action a t The gradient operator, Q(s i ,μ(s i θ μ );θ Q ) is μ(s i θ μ ) The corresponding Critic network outputs the evaluation value, μ(s i θ μ ) is the Actor network output action value corresponding to the i-th experience tuple, ▽ θμ Denotes the parameter θ μ The gradient operator, L is the Critic network loss function, Q(s i ,a i θ Q ) is the evaluation value of the Critic network output corresponding to the i-th experience tuple, y i It is the target network value of the Critic network.

[0028] Preferably, the step S4 further comprises constraining the actions of each of the agents when training the neural network:

[0029]

[0030] Among them, a t is the action of the agent, μ(s t θ μ ) is an Actor network, g is a random number in the range of [0,1], ε is a variable parameter, ε gradually decreases from its upper limit to its lower limit, and U(-1,1) represents a set consisting of -1 and 1.

[0031] Preferably, step S4 also includes: calculating the time differential error of the experience tuple, setting the priority of each experience tuple based on the time differential error, and sampling the experience tuple based on the priority and using a random priority method when using the experience tuple to train the neural network.

[0032] Preferably, the time difference error of the experience tuple is calculated, specifically:

[0033] δ t =r t +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )

[0034] Among them, δ t is the time difference error, r t is the reward function value, γ is the adjustment coefficient, Q(s t+1 ,a t+1 ) represents the Critic network in the environment state s t+1 Action is a t+1 The evaluation value output when maxQ(s t+1 ,a t+1 ) means taking Q(s t+1 ,a t+1 ), Q(s t ,a t ) represents the Critic network in the environment state s t Action is a t The evaluation value output when ;

[0035] The priority of the experience tuple is set based on the time search error, specifically:

[0036] p t =|δ t |+σ

[0037] Among them, p t is the priority, σ is a constant greater than 0, used to ensure that the time difference error δ t Experience tuples of 0 can also be sampled.

[0038] Preferably, the step S4 further includes: setting sampling weights, and correcting the loss function of the Critic network by using the sampling weights;

[0039] The loss function of the modified Critic network is:

[0040]

[0041] Among them, L is the Critic network loss function, Q(s i ,a i θ Q ) is the evaluation value of the Critic network output corresponding to the i-th experience tuple, y i is the target network value of the Critic network;

[0042] The sampling weight is:

[0043]

[0044] Among them, ω i is the sampling weight, α is the priority factor; β is the importance sampling weight correction coefficient, p i The priority of the i-th experience tuple, N is the number of experience tuples, j=1,2,…,N.

[0045] Preferably, the step S5 is specifically as follows:

[0046] The IoT device data is preprocessed, and the preprocessed IoT device data is input into a trained neural network to obtain detection results and evaluation indicators.

[0047] The present invention also provides an IoT device data anomaly detection device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the IoT device data anomaly detection method is implemented.

[0048] Compared with the prior art, the present invention has the following beneficial effects: after establishing the Internet of Things data set, the present invention divides it into two subsets, labeled and unlabeled, and sets the five-tuple of the Markov decision process for the two subsets respectively. Therefore, the experience tuple generated based on the interaction of the five-tuple includes both the experience tuple obtained based on the labeled data and the experience tuple obtained based on the unlabeled data. After the neural network is trained based on the experience tuple, it can realize the joint detection of known, unknown and new abnormal type data. The present invention has strong versatility and can perform anomaly detection on large-scale data sets with only a small number of labels, without consuming a lot of manpower and material resources to label the Internet of Things device data; at the same time, it makes full use of the limited data information of labeled abnormal types, learns from it, actively explores the known and unknown types of abnormal data in the unlabeled data, and has high generalization of data anomaly detection; finally, the joint detection method adopted by the present invention can more accurately identify various anomalies, reduce false positives and false negatives, improve the quality of Internet of Things services, and has high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0050] Figure 1 A flowchart of an embodiment of an IoT device data anomaly detection method provided by the present invention;

[0051] Figure 2a yes Figure 1 A schematic diagram of AUC-ROC comparison when the embodiment and the prior art respectively detect data with abnormal type Analysis;

[0052] Figure 2b yes Figure 1 A schematic diagram of AUC-ROC comparison when the embodiment and the prior art respectively detect data of abnormal type Backdoor;

[0053] Figure 2c yes Figure 1 AUC-ROC comparison diagram of the embodiment and the prior art when detecting data with Dos as an abnormality type;.

[0054] Figure 2d yes Figure 1 A schematic diagram of AUC-ROC comparison when the embodiment and the prior art respectively detect data of anomaly type Exploits;

[0055] Figure 2e yes Figure 1AUC-ROC comparison diagram of the embodiment and the prior art when detecting data of abnormal type Fuzzers respectively;

[0056] Figure 2f yes Figure 1 AUC-ROC comparison diagram of the embodiment and the prior art when detecting data of Generic abnormality type respectively;

[0057] Figure 2g yes Figure 1 AUC-ROC comparison diagram of the embodiment and the prior art when detecting data of abnormal type Reconnaissance respectively;

[0058] Figure 2h yes Figure 1 A schematic diagram showing the comparison of AUC-ROC when the embodiment and the prior art respectively detect data with an abnormal type of High;

[0059] Figure 2i yes Figure 1 A schematic diagram showing the comparison of AUC-ROC when the embodiment and the prior art respectively detect data with an abnormal type of Bypass;

[0060] Figure 2j yes Figure 1 AUC-ROC comparison diagram of the embodiment and the prior art when detecting data with abnormal type Upstairs respectively;

[0061] Figure 2k yes Figure 1 AUC-ROC comparison diagram of the embodiment and the prior art when detecting data with an abnormal type of Downstairs;

[0062] Figure 2l yes Figure 1 AUC-ROC comparison diagram of the embodiment and the prior art when detecting data of abnormal type Hypothyroid respectively;

[0063] Figure 2m yes Figure 1 A schematic diagram showing the comparison of AUC-ROC when the embodiment and the prior art respectively detect data of subnormal abnormality type;

[0064] Figure 3a yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data of the abnormal type Analysis;

[0065] Figure 3b yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data of the abnormal type Backdoor;

[0066] Figure 3c yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data with DoS as anomaly type;.

[0067] Figure 3d yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data of anomaly type Exploits;

[0068] Figure 3e yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data of the anomaly type Fuzzers;

[0069] Figure 3f yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data of Generic abnormality type;

[0070] Figure 3g yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data of abnormality type Reconnaissance;

[0071] Figure 3h yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data with an abnormal type of High;

[0072] Figure 3i yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data with an abnormal type of Bypass;

[0073] Figure 3j yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data with an abnormal type of Upstairs;

[0074] Figure 3k yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art respectively detect data with the abnormal type of Downstairs;

[0075] Figure 3l yes Figure 1 A schematic diagram showing the comparison of AUC-PR when the embodiment and the prior art are used to detect data of abnormal type Hypothyroid;

[0076] Figure 3m yes Figure 1 The illustrated embodiment and the prior art are schematic diagrams comparing AUC-PR when detecting data of subnormal type. DETAILED DESCRIPTION

[0077] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0078] Embodiment 1

[0079] See also Figure 1 In this embodiment, a method for detecting abnormal data of an IoT device includes the following steps:

[0080] Step S1, collecting historical data of IoT devices to establish an IoT data set, and dividing the IoT data set into a first subset with annotated abnormal types and a second subset without annotations;

[0081] Step S2: Based on the Markov decision process framework, construct the interaction process between the agent and the environment for data anomaly detection, and design the environment state, action, state transfer function and reward function;

[0082] Step S3: constructing a neural network for data anomaly detection based on a deep deterministic policy gradient algorithm framework;

[0083] Step S4, using the intelligent agent to interact with the environment to generate an experience tuple, and using the experience tuple to train the neural network;

[0084] Step S5: Use the trained neural network to perform anomaly detection on IoT device data.

[0085] Therefore, the present invention has strong versatility and can perform anomaly detection on large-scale data sets with only a small number of labels, without consuming a lot of manpower and material resources to label the IoT device data; at the same time, it makes full use of limited data information of labeled anomaly types, learns from them, and actively explores known and unknown types of anomaly data in unlabeled data, and the generalization of data anomaly detection is high; finally, the joint detection method adopted by the present invention can more accurately identify various anomalies, reduce false alarms and missed alarms, improve the quality of IoT services, and has high detection accuracy.

[0086] This embodiment provides an IoT device data anomaly detection method based on deep reinforcement learning, which aims to effectively detect anomalies in IoT device data, improve the accuracy and efficiency of IoT device data anomaly detection, and ensure the normal and safe operation of IoT services.

[0087] First, the data set is divided according to the actual IoT device data; the IoT data set is divided into two subsets: labeled and unlabeled. Taking full advantage of the extremely limited data labels, based on the Markov decision process framework, the interaction process between the agent and the environment for data anomaly detection is constructed, and the five-tuple of the Markov decision process is set, including the environment state, action, state transfer function, and reward function. The neural network algorithm is designed using the deep deterministic policy gradient framework. The experience tuple includes both the experience tuple obtained based on labeled data and the experience tuple obtained based on unlabeled data. After training based on the experience tuple, the neural network realizes the joint detection of known, unknown, and new types of abnormal data.

[0088] This embodiment has the following beneficial effects:

[0089] Strong versatility; it can detect anomalies in large-scale data sets with only a small number of labels, without spending a lot of manpower and material resources to label IoT device data;

[0090] It can jointly detect known, unknown, and new types of abnormal data; make full use of limited labeled abnormal data information, learn from it, and actively explore known and unknown types of abnormal data in unlabeled data, which improves the generalization of data anomaly detection and has good guiding reference value;

[0091] High detection accuracy; through joint detection of IoT device data of known, unknown, and new abnormal types, it can more accurately identify various anomalies, reduce false alarms and missed alarms, and improve the quality of IoT services.

[0092] Further, step S1 is specifically as follows: according to the data type of the IoT device, the data subsets are divided, and the IoT data set D is divided into two data subsets, namely, the first subset D a and the second subset D u The first subset D a contains known abnormal type data, the second subset D u It contains known abnormal type data, unknown abnormal type data and a large number of normal data. Different codes are used to mark the first subset D of known abnormal types. a and the unlabeled second subset D u .

[0093] Furthermore, the step S2 is specifically as follows: based on the Markov decision process framework, constructing the interaction process between the intelligent agent and the environment for data anomaly detection, and designing the key parts of the Markov decision process including environmental state, action, state transfer function, and reward function.

[0094] Set a state space S, and set the state space S based on all data samples in the IoT dataset, where each data sample in the IoT dataset represents an environmental state s t , all environmental states s t Constitute the state space S.

[0095] Set the agent's action a t In order to preserve the decision uncertainty of the data anomaly detection algorithm during training, the action space of the agent is a continuous space, represented by a t ∈[-1,1], when a t When the value is greater than or equal to 0, the current environment state s t It is judged as abnormal data. When a t When the value is less than 0, the current state s t The data is judged to be normal.

[0096] Set the state transfer function In order to improve the interaction process between the data anomaly detection agent and the environment, and ensure that the agent can fully explore the IoT device dataset D, the state transition function Two parts are designed, namely, for the first subset D a The first state transition function and for the second subset D u The second state transition function During the training phase of the data anomaly detection algorithm, and They are selected for execution with probability q and 1-q respectively.

[0097] First state transfer function is a probability function of uniform distribution, the first state transfer function Randomly from the first subset D a Collect a data sample as the environment state s at the next moment t+1 , the first subset D a The probability of each data sample being sampled is equal.

[0098] The first state transfer function It is expressed as:

[0099]

[0100] Among them, U(D a ) is a data sample randomly collected from the first subset, s t+1 Represents the environmental state s at the next moment t+1 .

[0101] The second state transfer function According to the current environment status t and the current action a t From the second subset D u Sample the next moment's environment state s t+1 When the agent determines the current environment state s t When the data is normal, the second state transfer function The distance from the current environment state s t The data sample with the largest Euclidean distance is taken as the environment state s at the next moment t+1 ; When the agent judges the current environment state s t When it is abnormal data, the second state transfer function The distance from the current environment state s t The data sample with the smallest Euclidean distance is taken as the environment state s at the next moment t+1 .

[0102] Specifically, the second state transfer function It is expressed as:

[0103]

[0104] in, The second state transition function is shown The relevant parameters of the second state transfer function According to the current environment status t and the current action a t From the second subset D u Sample the next moment's environment state s t+1 , Indicates the distance from the current environment state s t The data sample corresponding to the maximum Euclidean distance is Indicates the distance from the current environment state s t The data sample corresponding to the Euclidean minimum distance, Ed(·) function is used to calculate and The Euclidean distance between express and The Euclidean distance between It is a hidden layer network extracted from the neural network used to output the action of the intelligent agent. It is a neural network The network weight parameters are Indicates the hidden layer network when the environment state is s The output, Indicates that the environment state is s t Hidden layer network The output, is a random sample of capacity from the state space S A subset of data, a t ∈[-1,0) indicates that the agent judges the current environment state s t is normal data, a t ∈[0,1] indicates that the agent judges the current environment state s t is abnormal data.

[0105] Setting the reward function. In order to realize the joint detection of known abnormal type data and unknown abnormal type data, the reward function is designed to have two parts: external reward function and internal reward function. Specifically:

[0106]

[0107] Among them, r t (s t ,a t ) is the reward function, is the external reward function, is the internal reward function, and τ is a positive coefficient used to balance the size of the external reward and the internal reward so that the data anomaly detection algorithm can be better trained.

[0108] The external reward function is used to encourage the agent to learn valuable information using limited labeled abnormal type data. The external reward function helps the agent obtain more detailed feedback by coupling the agent action value into the external reward function. The external reward function is specifically expressed as:

[0109]

[0110] Among them, the external reward function The function parameters include the environment state s t and action a t , D a is the first subset, D u The second subset.

[0111] The internal reward function is used to encourage the agent to actively explore the unlabeled data and mine the data that exist in the first subset D. a The internal reward function is specifically expressed as:

[0112]

[0113] in, is the internal reward function, It is a neural network that calculates the difference between the current state and the set of previously encountered states; It is a neural network The network weight parameters and the hidden layer network In The parameters are consistent. The greater the difference between the current state observed by the agent and the previous state observed, the greater the difference between the neural network The larger the calculated difference value, the greater the internal reward function The returned reward value will be larger, which helps the agent explore the hidden u The data does not have an abnormal type or whether it is abnormal.

[0114] Furthermore, the step S3 is specifically as follows:

[0115] The network structure of the deep deterministic policy gradient algorithm framework includes two parts: an Actor network and a Critic network. The Actor network is used to output a certain action, and the Critic network is used to output an evaluation value of the action output by the Actor network in the current state. Both the Actor network and the Critic network are provided with a target network.

[0116] Actor network μ(s t θ μ ) Output a certain action a t , Critic network Q(s t ,a t θ Q ) outputs a t Next, implement action a t The evaluation value Q. Both the Actor network and the Critic network have corresponding target networks, namely μ′(s i θ μ′ ) and Q′(s t ,a t θ Q′ ), used to improve the stability of algorithm training.

[0117] Furthermore, the step S4 is specifically as follows: utilizing a large number of experience tuples generated by the interaction between the agent and the environment and storing them in an experience replay pool for training a neural network of a data anomaly detection algorithm.

[0118] The experience tuple is represented by the following five-tuple:

[0119] (s t ,a t ,s t+1 ,r t ,p t )

[0120] Among them, s t is the current state of the environment, a t is the action of the agent at the current moment, r t is the reward value obtained by the agent at the current moment, s t+1 is the environmental state at the next moment, p t is the priority of the experience tuple. The experience replay pool stores a large number of experience tuples in a moment.

[0121] The experience tuples are extracted in batches and input into the Actor network and the Critic network respectively, and the loss function value is calculated, and the loss function value is used to perform gradient update on the neural network parameters.

[0122] By applying the chain rule to the expected returns of the starting distribution J, the actor network parameters θ μ The update is done as follows:

[0123]

[0124] The parameter update of the Critic network follows the following loss function:

[0125]

[0126] Where J is the loss function of the Actor network, B represents the number of experience tuples extracted from the experience replay pool for network parameter update, i = 1, 2, ..., B, ▽ a Indicates action a t The gradient operator of θμ Denotes the parameter θ μ The gradient operator, μ(s i θ μ ) is the Actor network output action value corresponding to the i-th experience tuple, Q(s i ,μ(s i θ μ );θ Q ) is μ(s i θ μ ) corresponds to the Critic network output evaluation value, L is the Critic network loss function, Q(s i ,a i θ Q ) is the evaluation value of the Critic network output corresponding to the i-th experience tuple, y iis the target network value.

[0127] Target network value y i , the specific calculation is:

[0128] y i =r i +γQ′(s i+1 ,μ′(s i+1 θ μ′ );θ Q′ )

[0129] Among them, r t is the reward function value, γ is the adjustment coefficient, μ′(s i+1 θ μ′ ) is the Actor network in the environment state s i+1 The target network value when Q′(s i+1 ,μ′(s i+1 θ μ′ );θ Q′ ) is the Critic network in the environment state s i+1 The target network value at time.

[0130] Furthermore, the step S4 also includes further constraining the action a of the data anomaly detection agent t For each agent action a at a moment t , the following rules are adopted to constrain them:

[0131]

[0132] Among them, a t is the action of the agent, μ(s t θ μ ) is the Actor network, g is a random number in the range of [0,1], ε is a variable parameter, ε gradually decreases from its upper limit to its lower limit, and U(-1,1) represents a set consisting of -1 and 1.

[0133] Furthermore, the step S4 also includes: calculating the time differential error of the experience tuple, setting the priority of each experience tuple based on the time differential error, and when using the experience tuple to train the neural network, sampling the experience tuple using a random priority method based on the priority.

[0134] In order to more effectively utilize the experience tuples stored in the experience pool and improve the efficiency of the learning process of the data anomaly detection agent, a priority experience replay mechanism is further introduced to improve the data anomaly detection method, and the priority of experience is measured by calculating the time difference error TD-error.

[0135] Furthermore, the time difference error of the experience tuple is calculated, specifically:

[0136] δ t =r t +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )

[0137] Among them, δ t is the time difference error, r t is the reward function value, γ is the adjustment coefficient, Q(s t+1 ,a t+1 ) represents the Critic network in the environment state s t+1 Action is a t+1 The evaluation value output when maxQ(s t+1 ,a t+1 ) means taking Q(s t+1 ,a t+1 ), Q(s t ,a t ) represents the Critic network in the environment state s t Action is a t The evaluation value output when ;

[0138] The priority of the experience tuple is set based on the time search error, specifically:

[0139] p t =|δ t |+σ

[0140] Among them, p t is the priority, σ is a constant greater than 0, used to ensure that the time difference error δ t The experience tuple of 0 can also be sampled, p t It will serve as an important basis for extracting small batches of experience tuples for experience playback and updating network parameters.

[0141] Further, a sampling weight is set, and the loss function of the Critic network is corrected by the sampling weight;

[0142] The loss function of the modified Critic network is:

[0143]

[0144] Among them, L is the Critic network loss function, Q(s i ,a i θ Q ) is the evaluation value of the Critic network output corresponding to the i-th experience tuple, yi is the target network value of the Critic network;

[0145] The sampling weights are:

[0146]

[0147] Among them, ω i is the sampling weight, α is the priority factor; β is the importance sampling weight correction coefficient, p i The priority of the i-th experience tuple, N is the number of experience tuples, j=1,2,…,N.

[0148] Since the above priority sampling method based on priority will lead to the loss of diversity of the extracted small batch data, the random prioritization method is used to reduce the loss, and the importance sampling weight ω is used i To correct the error introduced by diversity loss.

[0149] Furthermore, the step S5 is specifically as follows: preprocessing the IoT device data, removing or completing missing data, and processing the data format using a Z-Score method. The preprocessed IoT device data is input into a trained neural network to obtain detection results and evaluation results.

[0150] Specifically, the data is input into the trained Actor network to detect abnormal data in the data, and two evaluation indicators, the area under the receiver operating characteristic curve AUC-ROC and the area under the precision and recall curve AUC-PR, are further calculated.

[0151] In order to verify the present invention, the detection method provided by the present invention, hereinafter referred to as PANDA, is compared with three commonly used detection methods in the prior art, which are DPLAN, FEAWAD and SLAD.

[0152] The above four detection methods are used to detect different types of abnormal data respectively, and a comparison diagram of the two evaluation indicators, the area under the receiver operating characteristic curve AUC-ROC and the area under the precision and recall curve AUC-PR, is drawn to achieve a comparison of different detection methods.

[0153] In all the drawings, the same reference numerals are used to represent the same elements or structures, wherein: the circular straight line represents the performance of the DPLAN of the present invention; the inverted triangle straight line represents the performance of the prior art FEAWAD; the cross straight line represents the performance of the prior art SLAD; the square straight line represents the performance of the present technology PANDA;

[0154] Specifically, Figure 2a , Figure 2b , Figure 2c , Figure 2d , Figure 2e , Figure 2f , Figure 2g , Figure 2h , Figure 2i , Figure 2j , Figure 2k , Figure 2l , Figure 2m The AUC-ROC comparison diagrams of the above four detection methods for the Analysis type, Backdoor type, Dos type, Exploits type, Fuzzers type, Generic type, Reconnaissance type, High type, Bypass type, Upstairs type, Downstairs type, Hypothyroid type, and Subnormal type data are respectively shown. In the AUC-ROC comparison diagram, the horizontal axis is the abnormal contamination rate, and the vertical axis is the AUC-ROC value. It can be seen from the figure that no matter which type of abnormal data is detected, the present invention has obvious advantages in the average AUC-ROC value.

[0155] Specifically, Figure 3a , Figure 3b , Figure 3c , Figure 3d , Figure 3e , Figure 3f , Figure 3g , Figure 3h , Figure 3i , Figure 3j , Figure 3k , Figure 3l , Figure 3m The AUC-PR comparison diagrams are shown when the above four detection methods are used to detect Analysis type, Backdoor type, Dos type, Exploits type, Fuzzers type, Generic type, Reconnaissance type, High type, Bypass type, Upstairs type, Downstairs type, Hypothyroid type, and Subnormal type data. In the AUC-PR comparison diagram, the horizontal axis is the abnormal contamination rate, and the vertical axis is the AUC-PR value. It can be seen from the figure that no matter which type of abnormal data is detected, the present invention has obvious advantages in the AUC-PR value.

[0156] Embodiment 2

[0157] This embodiment provides an IoT device data anomaly detection device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the IoT device data anomaly detection method described in Embodiment 1 is implemented.

[0158] The IoT device data anomaly detection device provided in this embodiment is used to implement the IoT device data anomaly detection method. Therefore, the IoT device data anomaly detection device also has the technical effects possessed by the IoT device data anomaly detection method, which will not be repeated here.

[0159] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present invention should be covered by the present invention.

Claims

1. A method for detecting anomaly in data of an Internet of Things device, characterized in that: The following steps are involved: Step S1, collecting historical data of IoT devices to establish an IoT data set, and dividing the IoT data set into a first subset with annotated abnormal types and a second subset without annotations; Step S2: Based on the Markov decision process framework, construct the interaction process between the agent and the environment for data anomaly detection, and design the environment state, action, state transfer function and reward function; Step S3: constructing a neural network for data anomaly detection based on a deep deterministic policy gradient algorithm framework; Step S4, using the intelligent agent to interact with the environment to generate an experience tuple, and using the experience tuple to train the neural network; Step S5: Use the trained neural network to perform anomaly detection on IoT device data; The step S2 is specifically as follows: Setting a state space, wherein the state space is set based on all data samples in the IoT data set, each data sample in the IoT data set represents an environmental state, and all environmental states constitute the state space; Set the action of the agent. The action space of the agent is a continuous space [-1,1]. When the action value is greater than or equal to 0, the current environment state is judged as abnormal data. When the action value is less than 0, the current state is judged as normal data. Setting a state transfer function, wherein the state transfer function includes a first state transfer function for the first subset and a second state transfer function for the second subset; The first state transfer function is a probability function of uniform distribution. a Randomly collect a data sample from the first subset as the environment state at the next moment, and the probability of each data sample in the first subset being sampled is equal; The second state transfer function samples the environment state at the next moment from the second subset according to the current environment state and the current action. When the agent determines that the current environment state is normal data, the second state transfer function uses the data sample with the largest Euclidean distance from the current environment state as the environment state at the next moment. When the agent determines that the current environment state is abnormal data, the second state transfer function uses the data sample with the smallest Euclidean distance from the current environment state as the environment state at the next moment. Setting a reward function, wherein the reward function includes an external reward function for prompting the agent to learn using the labeled abnormal data and an internal reward function for prompting the agent to learn using the unlabeled data; Setting a balance coefficient, wherein the balance coefficient is used to balance the size of external rewards and internal rewards; The external reward function provides feedback by coupling the agent action value, and the internal reward function provides feedback by calculating the difference between the current state and the set of states that have appeared before.

2. The method for detecting anomaly in IoT device data according to claim 1, characterized in that: The step S3 is specifically as follows: The network structure of the deep deterministic policy gradient algorithm framework includes two parts: an Actor network and a Critic network. The Actor network is used to output a certain action, and the Critic network is used to output an evaluation value of the action output by the Actor network in the current state. Both the Actor network and the Critic network are provided with a target network.

3. The method for detecting anomaly in IoT device data according to claim 2, characterized in that: The step S4 is specifically as follows: Batch extract experience tuples and input them into the Actor network and the Critic network respectively, and calculate the loss function value, and use the loss function value to perform gradient update on the neural network parameters: The parameter update of the Actor network follows the following loss function: The parameter update of the Critic network follows the following loss function: Where J is the loss function of the Actor network, B represents the number of experience tuples extracted from the experience replay pool for network parameter update, i = 1, 2, ..., B, Indicates action a t The gradient operator, Q(s i ,μ(s i θ μ );θ Q ) is μ(s i θ μ ) The corresponding Critic network outputs the evaluation value, μ(s i θ μ ) is the Actor network output action value corresponding to the i-th experience tuple, Denotes the parameter θ μ The gradient operator, L is the Critic network loss function, Q(s i ,a i θ Q ) is the evaluation value of the Critic network output corresponding to the i-th experience tuple, y i It is the target network value of the Critic network.

4. The method for detecting anomaly in IoT device data according to claim 1, characterized in that: The step S4 also includes constraining the actions of each of the agents when training the neural network: Among them, a t is the action of the agent, μ(s t θ μ ) is an Actor network, g is a random number in the range of [0,1], ε is a variable parameter, ε gradually decreases from its upper limit to its lower limit, and U(-1,1) represents a set consisting of -1 and 1.

5. The method for detecting anomaly in data of IoT devices according to claim 1, characterized in that: The step S4 also includes: calculating the time differential error of the experience tuple, setting the priority of each experience tuple based on the time differential error, and sampling the experience tuple based on the priority and using a random priority method when using the experience tuple to train the neural network.

6. The method for detecting anomaly in IoT device data according to claim 5, characterized in that: Calculate the time difference error of the experience tuple, specifically: d t =r t +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t ) Among them, δ t is the time difference error, r t is the reward function value, γ is the adjustment coefficient, Q(s t+1 ,a t+1 ) represents the Critic network in the environment state s t+1 Action is a t+1 The evaluation value output when maxQ(s t+1 ,a t+1 ) means taking Q(s t+1 ,a t+1 ), Q(s t ,a t ) represents the Critic network in the environment state s t Action is a t The evaluation value output when ; The priority of the experience tuple is set based on the time search error, specifically: p t =|δ t |+s Among them, p t is the priority, σ is a constant greater than 0, used to ensure that the time difference error δ t Experience tuples of 0 can also be sampled.

7. The method for detecting anomaly in data of IoT devices according to claim 5, characterized in that: The step S4 further includes: setting sampling weights, and correcting the loss function of the Critic network by using the sampling weights; The loss function of the modified Critic network is: Among them, L is the Critic network loss function, Q(s i ,a i θ Q ) is the evaluation value of the Critic network output corresponding to the i-th experience tuple, y i is the target network value of the Critic network; The sampling weight is: Among them, ω i is the sampling weight, α is the priority factor; β is the importance sampling weight correction coefficient, p i The priority of the i-th experience tuple, N is the number of experience tuples, j=1,2,…,N.

8. The method for detecting anomaly in IoT device data according to claim 1, characterized in that: The step S5 is specifically as follows: The IoT device data is preprocessed, and the preprocessed IoT device data is input into a trained neural network to obtain detection results and evaluation indicators.

9. An IoT device data anomaly detection device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the IoT device data anomaly detection as described in any one of claims 1 to 8 is implemented.