Anomaly detection, anomaly detection model training method, device and electronic equipment

Abstract

The application provides an anomaly detection method, an anomaly detection model training method and device, and electronic equipment, relates to the technical field of network security, and is used for improving the security of Internet of Things equipment. The anomaly detection method comprises the following steps: collecting network traffic data of a target Internet of Things equipment within a preset time window; extracting a first feature data set from the network traffic data, wherein the first feature data set comprises a first feature data subset of M pairs of data combinations; inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model; and determining whether the target Internet of Things equipment is abnormal according to a difference degree between network traffic data indicated by the detection value and reference network traffic data. The anomaly detection model is trained based on a sample feature data set obtained by performing noise processing on a second feature data set, the second feature data set is extracted from sample network traffic, and the sample feature data set comprises at least one abnormal situation.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to anomaly detection, anomaly detection model training method, device and electronic equipment. Background Art

[0002] With the rapid development of IoT technology, it has been widely applied in smart homes, smart cities, industry, healthcare, and other fields. However, due to weak security mechanisms, IoT devices have become a major target of cyberattacks. Therefore, improving the security of IoT devices has become a pressing technical issue. Summary of the Invention

[0003] The present application provides an anomaly detection, an anomaly detection model training method, device and electronic equipment for improving the security of Internet of Things devices.

[0004] In a first aspect, an embodiment of the present application provides an anomaly detection method, comprising: collecting network traffic data of a target Internet of Things device within a preset time window; extracting a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M being a positive integer; inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, the detection value being used to indicate the degree of difference between the network traffic data and reference network traffic data; determining whether the target Internet of Things device is abnormal based on the detection value; wherein the anomaly detection model is trained based on a sample feature data set, the sample feature data set being obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set including at least one abnormal situation.

[0005] In the embodiment of the present application, since the second feature data set is subjected to noise processing during training, the noise types included in the obtained sample feature data set are richer, and the various abnormal situations existing in network traffic data in real scenarios can be more realistically simulated. Furthermore, the anomaly detection model trained based on this sample feature data set can better identify and process noise in network traffic data during actual use, and improve the ability to reconstruct normal network traffic data, thereby making it more accurate in detecting abnormal network traffic data, which is conducive to improving the accuracy of anomaly detection of IoT devices, thereby improving the security of IoT devices.

[0006] In one possible embodiment, extracting a first feature data set from the network traffic data includes: determining M pairs of data combinations for generating the first feature data set based on preconfigured business requirements; determining, based on the network traffic data, data statistics generated by each pair of data combinations in the M pairs of data combinations within a unit time of the preset time window; determining, based on the data statistics generated by each pair of data combinations within a unit time of the preset time window, feature vectors of each pair of data combinations within a unit time, and forming a first feature data subset for each pair of data combinations within the preset time window; determining the first feature data set based on the first feature data subset of the M pairs of data combinations within the preset time window.

[0007] In one possible embodiment, the M pairs of data combinations include one or more pairs of data combinations: the average value of the number of data packets and the size of data packets in the network traffic data per unit time; the data volume corresponding to each data protocol in all data protocols in the network traffic data per unit time and the usage frequency corresponding to each data protocol; the proportion of encrypted traffic in the total traffic of the network traffic data per unit time and the data volume of encrypted traffic output by the target IoT device; the number of Internet Protocol IP addresses connected to the target IoT device per unit time in a preset time window and the frequency of failure to connect to the IP address.

[0008] In a possible embodiment, the method further includes: extracting the second feature data set from the sample network traffic data, the second feature data set including second feature data subsets of N pairs of data combinations, where N is a positive integer; determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations based on the second feature data subset and a preset calculation rule; determining the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; performing noise processing on the second feature data subset of each pair of data combinations based on the noise parameter corresponding to each pair of data combinations to obtain a sample feature data set; training an initial anomaly detection model based on the sample feature data set to obtain a detection result of the initial anomaly detection model; adjusting the model parameters of the initial anomaly detection model based on the error value between the detection result and the reference data set until the error value is less than the preset error value to obtain the trained anomaly detection model.

[0009] In a possible implementation, determining the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations includes: determining the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations and a preset strategy, the sparsity value being used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include Gaussian noise, salt and pepper noise, and masking noise; determining the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter based on the noise type corresponding to each pair of data combinations; and determining the noise parameter corresponding to each pair of data combinations based on the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0010] In a possible embodiment, the preset strategy includes a first threshold and a second threshold; based on the variance and sparsity value of the second feature data subset of each pair of data combinations and the preset strategy, the noise type corresponding to each pair of data combinations is determined, including: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, then the noise type corresponding to each pair of data combinations is determined to be Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value is greater than the second threshold, then the noise type corresponding to each pair of data combinations is determined to be salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value does not exceed the second threshold, then the noise type corresponding to each pair of data combinations is determined to be masking noise.

[0011] In a second aspect, the embodiments of the present application provide an anomaly detection model training method, comprising: extracting a second feature data set from sample network traffic data, the second feature data set comprising N pairs of data combination second feature data subsets, N being a positive integer; determining variance and / or sparsity of each pair of data combination second feature data subset according to the second feature data subset of each pair of data combination and a preset calculation rule; determining a noise parameter corresponding to each pair of data combination according to the variance and / or sparsity of each pair of data combination second feature data subset; performing noise processing on the second feature data subset of each pair of data combination according to the noise parameter corresponding to each pair of data combination, to obtain a sample feature data set, the sample feature data set comprising at least one abnormal situation; training an initial anomaly detection model according to the sample feature data set, to obtain a detection result of the initial anomaly detection model; adjusting model parameters of the initial anomaly detection model according to an error value between the detection result and a reference data set, until the error value is less than a preset error value, to obtain a trained anomaly detection model, the trained anomaly detection model being used to determine whether a target Internet of Things device is abnormal according to network traffic data of the target Internet of Things device.

[0012] In a possible implementation, the noise parameter corresponding to each pair of data combination is determined according to the variance and / or sparsity of the second feature data subset of each pair of data combination, comprising: determining a noise type corresponding to each pair of data combination according to the variance and sparsity of the second feature data subset of each pair of data combination and a preset strategy, the sparsity being used to indicate sparsity of the second feature data subset of each pair of data combination; wherein the noise type comprises Gaussian noise, salt and pepper noise and masking noise; determining the variance or sparsity of the second feature data subset of each pair of data combination as a first parameter according to the noise type corresponding to each pair of data combination; determining the noise parameter corresponding to each pair of data combination according to the noise type corresponding to each pair of data combination, the first parameter and a first formula corresponding to the noise type.

[0013] In a possible embodiment, the preset strategy includes a first threshold and a second threshold; based on the variance and sparsity value of the second feature data subset of each pair of data combinations and the preset strategy, the noise type corresponding to each pair of data combinations is determined, including: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, then the noise type corresponding to each pair of data combinations is determined to be Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value is greater than the second threshold, then the noise type corresponding to each pair of data combinations is determined to be salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value does not exceed the second threshold, then the noise type corresponding to each pair of data combinations is determined to be masking noise.

[0014] In a third aspect, an embodiment of the present application provides an anomaly detection device, comprising: an acquisition module for collecting network traffic data of a target Internet of Things device within a preset time window; an extraction module for extracting a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M being a positive integer; a detection module for inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, the detection value being used to indicate the degree of difference between the network traffic data and the reference network traffic data; the detection module is also used to determine whether the target Internet of Things device is abnormal based on the detection value; wherein the anomaly detection model is trained based on a sample feature data set, the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

[0015] In one possible embodiment, the extraction module is specifically used to: determine M pairs of data combinations used to generate the first feature data set based on preconfigured business requirements; determine the data statistics generated by each pair of data combinations in the M pairs of data combinations within the unit time of the preset time window based on the network traffic data; determine the feature vectors of each pair of data combinations within the unit time based on the data statistics generated by each pair of data combinations within the unit time of the preset time window, and form the first feature data subset of each pair of data combinations within the preset time window; determine the first feature data set based on the first feature data subset of the M pairs of data combinations within the preset time window.

[0016] In a possible implementation, the M pairs of data combinations include one or more of the following pairs of data combinations: an average value of a data packet amount and a data packet size of the network traffic data in a unit time; an amount of data and a use frequency of each data protocol corresponding to all data protocols in a unit time of the network traffic data; a proportion of encrypted traffic in total traffic in a unit time of the network traffic data and an amount of data of encrypted traffic output by the target Internet of Things device; a number of Internet Protocol (IP) addresses connected by the target Internet of Things device in a unit time of a preset time window and a frequency of connection IP address failure.

[0017] In a possible implementation, the anomaly detection apparatus further includes a training module, configured to: extract the second feature data set from the sample network traffic data, the second feature data set including N pairs of second feature data subsets of data combinations, N being a positive integer; determine a variance and / or a sparsity value of each pair of second feature data subsets of data combinations according to the pair of second feature data subsets of data combinations and a preset calculation rule; determine a noise parameter corresponding to each pair of data combinations according to the variance and / or the sparsity value of each pair of second feature data subsets of data combinations; perform noise processing on the second feature data subset of each pair of data combinations according to the noise parameter corresponding to each pair of data combinations, to obtain a sample feature data set; train an initial anomaly detection model according to the sample feature data set, to obtain a detection result of the initial anomaly detection model; and adjust a model parameter of the initial anomaly detection model according to an error value between the detection result and a reference data set, until the error value is less than a preset error value, to obtain the trained anomaly detection model.

[0018] In a possible implementation, the training module is specifically configured to: determine a noise type corresponding to each pair of data combinations according to the variance and the sparsity value of each pair of second feature data subsets of data combinations and a preset strategy, the sparsity value being used to indicate sparsity of the second feature data subset of each pair of data combinations; the noise type includes Gaussian noise, salt and pepper noise, and masking noise; determine the variance or the sparsity value of the second feature data subset of each pair of data combinations as a first parameter according to the noise type corresponding to each pair of data combinations; and determine the noise parameter corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0019] In one possible embodiment, the preset strategy includes a first threshold and a second threshold; the training module is specifically used to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determine that the noise type corresponding to each pair of data combinations is salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, determine that the noise type corresponding to each pair of data combinations is masking noise.

[0020] In a fourth aspect, an embodiment of the present application provides an anomaly detection model training device, comprising: an extraction module for extracting a second feature data set from sample network traffic data, wherein the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; a determination module for determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations based on the second feature data subset of each pair of data combinations and a preset calculation rule; the determination module is further used to determine the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; a noise processing module for determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; Corresponding noise parameters are used to add noise to the second feature data subset of each pair of data combinations to obtain a sample feature data set, and the sample feature data set includes at least one abnormal situation; a training module is used to train an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; the training module is also used to adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than a preset error value, to obtain a trained anomaly detection model, and the trained anomaly detection model is used to determine whether the target Internet of Things device is abnormal based on the network traffic data of the target Internet of Things device.

[0021] In a possible implementation, the determining module is specifically configured to: determine the noise type corresponding to each pair of data combinations according to the variance of the second feature data subset of each pair of data combinations, a sparsity value, and a preset strategy, the sparsity value being used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise type includes Gaussian noise, salt and pepper noise, and masking noise; determine the variance or the sparsity value of the second feature data subset of each pair of data combinations as a first parameter according to the noise type corresponding to each pair of data combinations; and determine the noise parameter corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0022] In a possible implementation, the preset strategy includes a first threshold and a second threshold; and the determining module is specifically configured to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value is greater than the second threshold, determine that the noise type corresponding to each pair of data combinations is salt and pepper noise; and if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value does not exceed the second threshold, determine that the noise type corresponding to each pair of data combinations is masking noise.

[0023] In a fifth aspect, an embodiment of the present application provides an electronic device, including at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the method in the first aspect or the second aspect and any possible implementation manner thereof by executing the instructions stored in the memory.

[0024] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and when the computer instructions run on a computer, the computer executes the method in the first aspect or the second aspect and any possible implementation manner thereof.

[0025] In a seventh aspect, an embodiment of the present application provides a computer program product, which contains computer instructions, and when the computer instructions run on a computer, the method in the first aspect or the second aspect and any possible implementation manner thereof is implemented.

[0026] The beneficial effects of the second aspect to the seventh aspect can refer to the content described in the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A schematic diagram of an application scenario of an anomaly detection method provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of an anomaly detection method provided in an embodiment of the present application;

[0029] Figure 3 A flowchart of an anomaly detection model training method provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the structure of an anomaly detection device provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of the structure of an anomaly detection model training device provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to better understand the technical solution provided by this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0034] Please refer to Figure 1 , is a schematic diagram of an application scenario of an anomaly detection method provided in an embodiment of the present application. Figure 1 As shown, the scenario includes a target IoT device 110, an anomaly detection apparatus 120, and other devices 130. The target IoT device 110 can perform wired or wireless communication with the anomaly detection apparatus 120 and other devices 130, respectively.

[0035] Among them, the target IoT device 110 refers to an intelligent IoT device that is connected to the Internet in real time, such as smart home devices, smart wearable devices, smart city devices, and industrial IoT devices. The anomaly detection device 120 refers to a device or equipment with data processing capabilities and anomaly detection capabilities. The anomaly detection device 120 may specifically refer to a device or equipment configured with an anomaly detection model, and the anomaly detection model can be used to detect whether the target IoT device 110 is abnormal. The device may specifically be a functional module or detection device in the target IoT device 110 or other detection devices; the device may specifically be a terminal device or server, and the terminal device includes but is not limited to a mobile phone, a personal computer (PC), a tablet computer, a laptop computer, a PDA, and a mobile internet device (MID). Other devices 130 refer to devices that can communicate with the target IoT device 110, such as terminal devices or servers. The terminal devices may correspond to the content described above and will not be repeated here. Among them, the other devices 130 are Figure 1 A server is used as an example, but in actual applications, the other devices 130 may refer to all devices that communicate with the target IoT device, that is, the number of other devices 130 may be one or more, which is not limited in the embodiments of the present application.

[0036] For example, anomaly detection device 120 may collect network traffic data between target IoT device 110 and other devices 130 in real time, extract a first feature dataset from the network traffic data, input the first feature dataset into a trained anomaly detection model to obtain a detection value, and then determine whether target IoT device 110 is abnormal based on the detection value. The specific implementation of anomaly detection device 120 determining whether target IoT device 110 is abnormal based on network traffic data will be described in detail below.

[0037] Please refer to Figure 2 , is a flow chart of an abnormality detection method provided in an embodiment of the present application. The following is a flow chart of an abnormality detection method performed by an abnormality detection device. Figure 2 The steps shown are described below. The abnormality detection device is, for example, Figure 1 The abnormality detection device 120 shown, Figure 2 The target IoT devices involved are, for example, Figure 1 Target IoT device 110 is shown.

[0038] S201, collecting network traffic data of target IoT devices within a preset time window.

[0039] Among them, the preset time window can be configured in advance in the anomaly detection device, and the length of the preset time window can be set according to actual needs. For example, the preset time window is 10 seconds, and this embodiment of the present application is not limited to this. The network traffic data can be all network traffic data received and sent by the target IoT device within the preset time window. On the one hand, the network traffic data may include traffic data of multiple protocols, such as the message queuing telemetry transport (MQTT) protocol, the hypertext transfer protocol (HTTP) protocol, and the constrained application protocol (Coap) protocol. On the other hand, the network traffic data may also include encrypted traffic data and non-encrypted traffic data.

[0040] Specifically, the anomaly detection device can monitor the target IoT device and obtain the network traffic data between the target IoT and other devices in real time within a preset time window. Figure 1 The other devices 130 shown are not described in detail here.

[0041] S202, extracting a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, where one pair of data combinations corresponds to traffic data of one dimension in the network traffic data, and M is a positive integer.

[0042] The M pairs of data combinations may include one or more of the following pairs of data combinations:

[0043] 1. Network traffic data: the average value of the number of packets and packet size per unit time.

[0044] 2. Network traffic data includes the amount of data corresponding to each data protocol in all data protocols per unit time and the usage frequency of each data protocol.

[0045] 3. Network traffic data: the proportion of encrypted traffic in the total traffic per unit time and the volume of encrypted traffic output by the target IoT device.

[0046] 4. The number of Internet Protocol IP addresses connected by the target IoT device per unit time within the preset time window and the frequency of failed connection to the IP addresses.

[0047] For example, the unit time in the embodiments of the present application may refer to time per second.

[0048] It should be noted that the above-mentioned data combination is the data combination exemplified in the embodiment of the present application. However, in actual applications, since the dimensions of traffic data related to different businesses are different, other data combinations can be selected according to business needs, that is, traffic data of other dimensions can be selected for anomaly detection. The embodiment of the present application does not limit this.

[0049] The first feature data subset corresponding to each pair of data combinations in the M pairs of data combinations can be understood as a set of feature vectors of each pair of data combinations within a preset time window.

[0050] The following describes in detail the specific implementation method of the anomaly detection device extracting the first feature data set from network traffic data.

[0051] Based on preconfigured business requirements, the anomaly detection device determines M pairs of data combinations for generating a first feature data set, and determines, based on network traffic data, the data statistics generated per unit time of each pair of data combinations in the M pairs of data combinations. In other words, the data statistics generated per unit time of each pair of data combinations are counted, for example, the number of data packets per second. Furthermore, based on the data statistics generated per unit time of each pair of data combinations in the preset time window, the anomaly detection device can determine a feature vector for each pair of data combinations per unit time. For example, if a pair of data combinations includes data A and data B, the feature vector for the pair of data combinations per unit time can be expressed as: [data statistics for data A per unit time, data statistics for data B per unit time]. Based on this operation, the anomaly detection device can obtain all feature vectors for each pair of data combinations within the preset time window, thereby forming a first feature data subset for each pair of data combinations within the preset time window. Ultimately, the anomaly detection device can determine a first feature data set based on the first feature data subsets of the M pairs of data combinations within the preset time window.

[0052] In one possible implementation, since network traffic data may contain missing data or abnormal data, before determining the first feature data subset for each pair of data combinations, the feature vectors may be cleaned to process missing values ​​and abnormal values ​​in the feature vectors. Specifically, missing values ​​and abnormal values ​​may be replaced using a preset strategy, which can be set based on actual needs. For example, the missing values ​​may be replaced with the average value of the data combination corresponding to the missing value, although this embodiment of the present application is not limited to this.

[0053] In one possible implementation, to ensure that the first feature data set meets subsequent detection requirements, before determining the first feature data subset for each data pair within a preset time window, the anomaly detection device determines the feature vector for each data pair within a unit time and then normalizes each feature vector. Exemplarily, the anomaly detection device may use Min-Max scaling to normalize the feature vectors to a value between [0, 1].

[0054] For example, in order to better understand the specific implementation method of the abnormality detection device to determine the first feature data set, the following takes the preset time window of 5 seconds and the unit time of one second as an example, and combines the data combinations 1-4 of the above examples to illustrate.

[0055] 1. Network traffic data: the average value of the number of packets and packet size per unit time.

[0056] The anomaly detection device collects the number of packets per second and the average packet size within a preset time window, where the number of packets per second from 1 to 5 seconds is: [10, 20, 15, 25, 30], and the average packet size from 1 to 5 seconds is: [145, 200, 160, 180, 220]. The anomaly detection device combines the number of packets per second and the average packet size per second into a feature vector, i.e., feature vector = [number of packets per second, average packet size per second].

[0057] For example, the first (first second) feature vector is [10, 145]; the second (second second) feature vector is [20, 200], and so on.

[0058] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0059] Optionally, the anomaly detection device may normalize the above-mentioned feature vector to [0, 1] using Min-Max. For example, the normalized feature vector is [0, 0] in the first time window and [0.5, 0.733] in the second time window.

[0060] 2. Network traffic data includes the amount of data corresponding to each data protocol in all data protocols per unit time and the usage frequency of each data protocol.

[0061] The anomaly detection device collects raw data on the data volume ratio and usage frequency of different protocols within a preset time window, including the following data statistics:

[0062] HTTP protocol data volume ratio; [0.6, 0.5, 0.7, 0.8, 0.6];

[0063] MQTT protocol data volume ratio: [0.3, 0.4, 0.2, 0.1, 0.2];

[0064] CoAP protocol data volume ratio: [0.1, 0.1, 0.1, 0.1, 0.2];

[0065] HTTP protocol usage frequency: [50, 60, 70, 80, 60];

[0066] MQTT protocol usage frequency: [30, 40, 20, 10, 20];

[0067] CoAP protocol usage frequency: [10, 10, 10, 10, 20];

[0068] The anomaly detection device combines the data volume ratio and usage frequency of all protocols in each time window (per second) into a feature vector, that is, the feature vector = [HTTP data volume ratio, MQTT data volume ratio, CoAP data volume ratio, HTTP usage frequency, MQTT usage frequency, CoAP usage frequency]

[0069] For example, the first (first second) eigenvector is: [0.6, 0.3, 0.1, 50, 30, 10], the second eigenvector is: [0.5, 0.4, 0.1, 60, 40, 10]... and so on.

[0070] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0071] Optionally, the anomaly detection device may normalize the above-mentioned feature vector to [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0.33, 0.67, 0, 0, 0.67, 0].

[0072] 3. Network traffic data: the proportion of encrypted traffic in the total traffic per unit time and the volume of encrypted traffic output by the target IoT device.

[0073] The anomaly detection device collects the proportion of encrypted traffic to total traffic and the amount of data output by the encryption protocol within a preset time window, including the following data statistics:

[0074] The proportion of encrypted traffic to total traffic: [0.8, 0.6, 0.9, 0.75, 0.7];

[0075] The amount of data output by the encryption protocol: [500, 400, 600, 550, 450] (unit: MB);

[0076] The anomaly detection device combines the proportion of encrypted traffic to total traffic per second and the amount of data output by the encryption protocol into a feature vector, that is, feature vector = [proportion of encrypted traffic to total traffic, amount of data output by the encryption protocol].

[0077] For example, the first eigenvector = [0.8, 500]; the second eigenvector = [0.6, 400], and so on.

[0078] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0079] Optionally, the anomaly detection device normalizes the above feature vector to [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0.67, 0.5]; and the second time window is: [0, 0].

[0080] 4. The number of Internet Protocol IP addresses connected by the target IoT device per unit time within the preset time window and the frequency of failed connection to the IP addresses.

[0081] The anomaly detection device collects the number of connections to different IP addresses within a preset time window and counts the frequency of failed connections, including the following data statistics:

[0082] Number of connections to different IP addresses: [10, 15, 20, 25, 30]

[0083] Statistics of the frequency of failed connections to IP addresses: [1, 2, 1.5, 3, 2.5] (unit: times / second)

[0084] The anomaly detection device combines the number of connections to different IP addresses per second and the frequency of statistically failed connections into a feature vector, that is, feature vector = [number of connections to different IP addresses, frequency of statistically failed connections].

[0085] For example, the first eigenvector = [10, 1]; the second eigenvector = [15, 2], and so on.

[0086] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0087] Optionally, the anomaly detection device normalizes the above feature vector to [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0, 0]; and for the second time window is: [0.25, 5].

[0088] In summary, the anomaly detection device can combine the first feature data subsets of the above four pairs of data into a first feature data set.

[0089] S203, input the first feature data set into the trained anomaly detection model to obtain a detection value output by the anomaly detection model, where the detection value is used to indicate the degree of difference between the network traffic data and the reference network traffic data. The anomaly detection model is trained based on a sample feature data set, and the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set. The sample feature data set includes at least one abnormal situation.

[0090] The anomaly detection model may be a denoising autoencoder (DAE) model.

[0091] Because the second feature data set is denoised when training the anomaly detection model, the anomaly detection model has better processing capabilities for various noises in real scenarios. Therefore, when denoising the network traffic data of the target IoT device, the denoising accuracy is better and the data reconstruction capability is better, so that the degree of difference between the network traffic data and the reference network traffic data can be better identified, and more accurate detection values ​​can be output.

[0092] To better understand the training process of the anomaly detection model, the following describes the training process of the anomaly detection model in detail.

[0093] The anomaly detection device may extract a second feature data set from the sample network traffic data. The second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer. The N pairs of data combinations may be one or more of the multiple data combinations described above. The N pairs of data combinations may be completely identical or partially identical to the M pairs of data combinations, and this is not limited in this embodiment of the present application.

[0094] Among them, the specific method in which the anomaly detection device extracts the second feature data set from the sample network traffic data can correspond to the specific method in which the anomaly detection device extracts the first feature data set from the network traffic data mentioned above, and will not be repeated here.

[0095] After obtaining the second feature data set, the variance and / or sparsity value of the second feature data subset of each pair of data combinations is determined based on the second feature data subset of each pair of data combinations and the preset calculation rule. The sparsity value is used to indicate the sparsity of the second feature data subset of each pair of data combinations. The preset calculation rule can be a calculation formula related to the variance and the sparsity value. Exemplarily, a calculation formula for calculating the variance of the second feature data subset of each pair of data combinations is as follows:

[0096]

[0097] Where Var(X) represents the variance of the second feature data subset of each pair of data combinations, n represents the number of feature vectors included in the second feature data subset, and X i represents the i-th eigenvector in the second feature data subset in each pair of data combinations, and μ represents the mean of the second feature data subset.

[0098] A calculation formula for calculating the sparse value of the second feature data subset for each pair of data combinations is as follows:

[0099]

[0100] Wherein, Sparsity(X) represents the sparse value of the second feature data subset for each pair of data combinations, n represents the number of eigenvectors included in the second feature data subset, and m represents the number of columns of the matrix corresponding to each eigenvector in the second feature data subset.

[0101] 1. The anomaly detection device determines the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations. The following describes three methods for determining the noise parameter corresponding to each pair of data combinations.

[0102] Method 1: The anomaly detection device determines the noise parameter corresponding to each pair of data combinations based on the variance of the second feature data subset of each pair of data combinations.

[0103] In this way, since the variance can reflect the degree of discreteness of the data and thus measure the volatility of the data, Gaussian noise can be selected as the added noise to simulate random fluctuations or measurement errors in network traffic data. Based on this, the anomaly detection device can determine the noise parameters of the Gaussian noise for each pair of data combinations based on the variance of the second feature data subset of each pair of data combinations. The noise parameters of the Gaussian noise are, for example, the standard deviation of the Gaussian noise. Exemplarily, a calculation formula for determining the noise parameters of the Gaussian noise for each pair of data combinations is as follows:

[0104] sigma=α1*Var(X)

[0105] Among them, sigma represents the standard layer of Gaussian noise, α1 is a custom constant, and Var(X) represents the variance of each pair of data combinations.

[0106] Method 2: The anomaly detection device determines the noise parameter corresponding to each pair of data combinations based on the sparse value of the second feature data subset of each pair of data combinations.

[0107] In this approach, since the sparsity value reflects the proportion of zero elements in the feature data and can measure the sparsity of the data, salt and pepper noise or masking noise can be selected as the added noise. Salt and pepper noise can simulate abnormal data caused by sensor failure or transmission errors, while masking noise can simulate abnormal conditions caused by missing data. Based on this, the anomaly detection device can determine the noise parameter corresponding to each data pair based on the sparsity value of the second feature data subset for each data pair. The noise parameter can be, for example, the salt and pepper noise probability or the masking noise probability.

[0108] For example, if the added noise is salt and pepper noise, a calculation formula for determining the noise parameters of the salt and pepper noise for each pair of data combinations is as follows:

[0109] P(Salt)=α2*Sparsity(X)

[0110] Among them, P(Salt) is the probability of salt and pepper noise, α2 is a custom constant, and Sparsity(X) is the sparse value of each pair of data combinations.

[0111] If the added noise is masking noise, a calculation formula for determining the noise parameter of the masking noise for each pair of data combinations is as follows:

[0112] P(Masking)=α3*(1-Sparsity(X))

[0113] Among them, P (Masking) is the probability of masking noise, α3 is a custom constant, and Sparsity (X) is the sparse value of each pair of data combinations.

[0114] Method three: The anomaly detection device determines the noise parameter corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations.

[0115] Because the variance and sparsity values ​​of the second feature data subsets for each of the N data pairs are different, the data characteristics reflected by the second feature data subsets are also different. Therefore, based on the variance and sparsity values ​​of the second feature data subsets for each data pair, noise data of a noise type that matches the data characteristics can be determined as the added noise data.

[0116] Specifically, the anomaly detection device may determine the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations, as well as a preset strategy. Noise types may include Gaussian noise, salt and pepper noise, and masking noise. The preset strategy may include a first threshold and a second threshold, and the preset strategy is used to indicate the relationship between the variance and sparsity value of the second feature data subset of each pair of data combinations and the noise type. The following describes in detail the specific manner in which the anomaly detection device determines the noise type corresponding to each pair of data combinations based on the preset strategy and the variance and sparsity value of the second feature data subset of each pair of data combinations.

[0117] If the anomaly detection device determines that the variance of the second feature data subset for each pair of data combinations is greater than a first threshold, the noise type corresponding to each pair of data combinations is determined to be Gaussian noise. If the anomaly detection device determines that the variance of the second feature data subset for each pair of data combinations does not exceed the first threshold and the sparsity value is greater than a second threshold, the noise type corresponding to each pair of data combinations is determined to be salt and pepper noise. If the anomaly detection device determines that the variance of the second feature data subset for each pair of data combinations does not exceed the first threshold and the sparsity value does not exceed the second threshold, the noise type corresponding to each pair of data combinations is determined to be masking noise.

[0118] After determining the noise type corresponding to each pair of data combinations in N pairs of data combinations, the first parameter can be determined. The first parameter is the variance or sparse value of the second feature data subset of each pair of data combinations. For example, if the noise type is Gaussian noise, the first parameter is the variance of the second feature data subset of each pair of data combinations. If the noise type is salt and pepper noise or masking noise, the first parameter is the sparse value of the second feature data subset of each pair of data combinations.

[0119] The anomaly detection device can then determine the noise parameter corresponding to each pair of data combinations based on the noise type, the first parameter, and the first formula corresponding to the noise type. The first formula is the calculation formula for calculating the noise parameter provided in Methods 1 and 2 above. Depending on the corresponding noise type, the corresponding calculation formula can be selected to calculate the noise parameter, and no further details are given here.

[0120] It should be noted that the embodiments of the present application are illustrated using the above three types of noise. In order to better simulate the real environment, other noises may also be used as added noise, such as random packet loss noise, which is not limited in the embodiments of the present application.

[0121] The anomaly detection device performs noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations, obtains the second feature data subset after noise processing for each pair of data combinations, and combines the second feature data subsets after noise processing in N pairs of data combinations into a sample feature data set.

[0122] Finally, the anomaly detection device can input the sample feature data set into the initial anomaly detection model, train the initial anomaly detection model, obtain the detection results output by the initial anomaly detection model, and adjust the model parameters of the initial anomaly detection model based on the error value between the detection results and the reference data set. The above training process is repeated until the error value between the detection results output by the initial anomaly detection model and the reference data set is less than the preset error value, thereby obtaining a trained anomaly detection model.

[0123] Furthermore, the anomaly detection device can input the first feature data set into the trained anomaly detection model, denoise the first feature data set through the trained anomaly detection model, and reconstruct the damaged feature data to obtain clean original data, compare the difference between the original data and the learned reference network traffic data, and output the detection value.

[0124] S204: Determine whether the target IoT device is abnormal based on the detection value.

[0125] Among them, the abnormality detection device can be pre-configured with a preset threshold, which is used to measure whether the target IoT device is abnormal. The preset threshold can be set according to actual needs, for example, to 0.5, and this embodiment of the present application does not limit this.

[0126] Based on this, the anomaly detection device can determine whether the target IoT device is abnormal based on the detection value and a preset threshold. Specifically, the anomaly detection device determines whether the detection value exceeds the preset threshold. If the detection value exceeds the preset threshold, the target IoT device is determined to be abnormal. If the detection value does not exceed the preset threshold, the target IoT device is determined to be normal.

[0127] For example, taking the detection value as 0.7 and the preset threshold as 0.5 as an example, if the abnormality detection device determines that the detection value exceeds the preset threshold, it determines that the target IoT device is abnormal.

[0128] In one possible implementation, when the anomaly detection device determines that a target IoT device is abnormal, it can generate an alarm and send it to the operation and maintenance device, which displays the alarm on an interface for the operation and maintenance personnel to review. This allows the personnel to quickly take appropriate measures to address the abnormality of the target IoT device. Furthermore, when the anomaly detection device determines that a target IoT device is abnormal, it can also protect the network security of the target IoT device based on preconfigured protective measures.

[0129] To improve the accuracy of an anomaly detection model in detecting whether a target IoT device is abnormal, an embodiment of the present application provides an anomaly detection model training method. In this method, by performing noise processing on a second feature dataset, the sample feature dataset includes at least one abnormal condition. Furthermore, after training the anomaly detection model using the sample feature dataset, the trained anomaly detection model is capable of recognizing and processing various types of noise data, thereby improving the anomaly detection model's ability to process noise data and thus improving the anomaly detection model's accuracy in detecting anomalies in the target IoT device.

[0130] The following combination Figure 3 The flowchart of the anomaly detection model training method shown in FIG. 1 illustrates the anomaly detection model training method provided in the embodiment of the present application. Figure 3 The steps shown are described below. The training device may be, for example, Figure 1 The abnormality detection device 120 is shown.

[0131] S301 , extracting a second feature data set from sample network traffic data, where the second feature data set includes a second feature data subset of N pairs of data combinations.

[0132] The specific method for extracting the second feature data set from the sample network traffic data by the training device can refer to the specific method for extracting the first feature data set from the network traffic data by the anomaly detection device described above, and will not be repeated here. The specific contents of the second feature data set, N pairs of data combinations, and the second feature data subset can also refer to the specific contents of the first feature data set, N pairs of data combinations, and the first feature data subset described above, respectively, and will not be repeated here.

[0133] S302 : Determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations according to the second feature data subset of each pair of data combinations in the N pairs of data combinations and a preset calculation rule.

[0134] Among them, the specific method in which the training device determines the variance and / or sparsity value of the second feature data subset for each pair of data combinations can correspond to the specific method in which the anomaly detection device determines the variance and / or sparsity value of the second feature data subset for each pair of data combinations mentioned above, and will not be repeated here.

[0135] S303 : Determine the noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations.

[0136] Among them, the specific way in which the training device determines the noise parameters corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations can refer to the specific way in which the abnormality detection device determines the noise parameters corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations mentioned above, which will not be repeated here.

[0137] S304 , performing noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations to obtain a sample feature data set.

[0138] The sample feature data set includes at least one abnormal situation.

[0139] The training device performs noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations. The specific method of obtaining the sample feature data set can refer to the specific method of determining the sample feature data set by the anomaly detection device mentioned above, which will not be repeated here.

[0140] S305: Train the initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model.

[0141] S306, adjusting the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than the preset error value, thereby obtaining a trained anomaly detection model. The trained anomaly detection model is used to determine whether the target IoT device is abnormal based on the network traffic data of the target IoT device.

[0142] The training device adjusts the model parameters of the initial anomaly detection model based on the error value between the detection result and the reference data set, and repeats the above training process until the error value between the detection result output by the initial anomaly detection model and the reference data set is less than the preset error value, thereby obtaining a trained anomaly detection model.

[0143] The trained anomaly detection model can be used to detect whether the target IoT device is abnormal. The specific method for determining whether the target IoT device is abnormal based on the trained anomaly detection model can be referred to the contents described in S201-S204 above and will not be repeated here.

[0144] In the embodiment of the present application, due to the addition of noise data that matches the data characteristics based on different data characteristics, the feature data after the addition of noise data can more realistically simulate the abnormal situation of network traffic data in real scenarios. The anomaly detection model obtained by training based on the feature data enables the trained anomaly detection model to perform better when processing noise and abnormal data in a real environment, and can better process and recover data contaminated by noise, which is conducive to improving the robustness of the anomaly detection model and improving the stability of anomaly detection. Therefore, even if there are occasional sudden interference data in real network traffic data, the trained anomaly detection model can still effectively identify and remove these noises, thereby improving the anomaly detection model's ability to reconstruct normal data, thereby improving the accuracy of anomaly detection of target IoT devices.

[0145] Based on the same inventive concept, the embodiment of the present application provides an abnormality detection device, which is used to implement any of the above abnormality detection methods, for example, Figure 2 The abnormality detection method shown in the figure can also realize the functions of the abnormality detection device mentioned above.

[0146] Please refer to Figure 4 , is a structural diagram of an abnormality detection device provided in an embodiment of the present application. Figure 4 As shown, the anomaly detection device 400 includes a collection module 401, an extraction module 402, and a detection module 403. Optionally, the anomaly detection device 400 may further include a training module 404.

[0147] Exemplarily, the collection module 401 is used to collect network traffic data of the target Internet of Things device within a preset time window; the extraction module 402 is used to extract a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M is a positive integer; the detection module 403 is used to input the first feature data set into the trained anomaly detection model to obtain a detection value output by the anomaly detection model, the detection value is used to indicate the degree of difference between the network traffic data and the reference network traffic data; the detection module 403 is also used to determine whether the target Internet of Things device is abnormal based on the detection value; wherein the anomaly detection model is trained based on a sample feature data set, the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

[0148] In one possible embodiment, the extraction module 402 is specifically used to: determine M pairs of data combinations for generating a first feature data set based on preconfigured business needs; determine the data statistics generated by each pair of data combinations in the M pairs of data combinations within a unit time of a preset time window based on network traffic data; determine the feature vectors of each pair of data combinations within a unit time based on the data statistics generated by each pair of data combinations within a unit time of the preset time window, and form the first feature data subset for each pair of data combinations within the preset time window; determine the first feature data set based on the first feature data subset of the M pairs of data combinations within the preset time window.

[0149] In one possible embodiment, the M pairs of data combinations include one or more pairs of data combinations: the average value of the number of data packets and the size of data packets in the network traffic data per unit time; the data volume corresponding to each data protocol in all data protocols in the network traffic data per unit time and the usage frequency corresponding to each data protocol; the proportion of encrypted traffic in the total traffic of the network traffic data per unit time and the data volume of encrypted traffic output by the target IoT device; the number of Internet Protocol IP addresses connected by the target IoT device per unit time in a preset time window and the frequency of failure to connect to the IP address.

[0150] In one possible embodiment, the anomaly detection device also includes a training module 404, which is used to extract a second feature data set from the sample network traffic data, where the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations based on the second feature data subset of each pair of data combinations in the N pairs of data combinations and a preset calculation rule; determine the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; perform noise processing on the second feature data subset of each pair of data combinations based on the noise parameter corresponding to each pair of data combinations to obtain a sample feature data set; train an initial anomaly detection model based on the sample feature data set to obtain a detection result of the initial anomaly detection model; adjust the model parameters of the initial anomaly detection model based on the error value between the detection result and the reference data set until the error value is less than the preset error value, thereby obtaining a trained anomaly detection model.

[0151] In one possible embodiment, the training module 404 is specifically used to: determine the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations, and a preset strategy, and the sparsity value is used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; according to the noise type corresponding to each pair of data combinations, determine the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter; and determine the noise parameter corresponding to each pair of data combinations based on the noise type, the first parameter, and the first formula corresponding to the noise type corresponding to each pair of data combinations.

[0152] In one possible embodiment, the preset strategy includes a first threshold and a second threshold; the training module 404 is specifically used to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value is greater than the second threshold, determine that the noise type corresponding to each pair of data combinations is salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value does not exceed the second threshold, determine that the noise type corresponding to each pair of data combinations is masking noise.

[0153] Based on the same inventive concept, the embodiment of the present application provides an anomaly detection model training device, which is used to implement any of the above anomaly detection model training methods, for example, Figure 3 The anomaly detection model training method shown in the figure can also realize the functions of the training device mentioned above.

[0154] Please refer to Figure 5 , is a structural diagram of an anomaly detection model training device provided in an embodiment of the present application. Figure 5 As shown, the anomaly detection model training device 500 includes an extraction module 501, a determination module 502, a noise processing module 503 and a training module 504.

[0155] Exemplarily, the extraction module 501 is configured to extract a second feature data set from the sample network traffic data, the second feature data set including N second feature data subsets of data combinations, N being a positive integer; the determination module 502 is configured to determine a variance and / or a sparsity value of each second feature data subset of data combinations according to the second feature data subset of each data combination and a preset calculation rule; the determination module 502 is further configured to determine a noise parameter corresponding to each data combination according to the variance and / or the sparsity value of the second feature data subset of each data combination; the noise processing module 503 is configured to perform noise processing on the second feature data subset of each data combination according to the noise parameter corresponding to each data combination, to obtain a sample feature data set, the sample feature data set including at least one abnormal situation; the training module 504 is configured to train an initial anomaly detection model according to the sample feature data set, to obtain a detection result of the initial anomaly detection model; the training module 504 is further configured to adjust a model parameter of the initial anomaly detection model according to an error value between the detection result and a reference data set, until the error value is less than a preset error value, to obtain a trained anomaly detection model, the trained anomaly detection model being configured to determine whether a target Internet of Things device is abnormal according to network traffic data of the target Internet of Things device.

[0156] In a possible implementation, the determination module 502 is specifically configured to determine a noise type corresponding to each data combination according to the variance and the sparsity value of the second feature data subset of each data combination and a preset strategy, the sparsity value being used to indicate sparsity of the second feature data subset of each data combination; the noise type includes Gaussian noise, salt and pepper noise and masking noise; the variance or the sparsity value of the second feature data subset of each data combination is determined as a first parameter according to the noise type corresponding to each data combination; and the noise parameter corresponding to each data combination is determined according to the noise type corresponding to each data combination, the first parameter and a first formula corresponding to the noise type.

[0157] In a possible implementation, the preset strategy includes a first threshold and a second threshold; and the determination module 502 is specifically configured to determine that the noise type corresponding to each data combination is Gaussian noise if the variance of the second feature data subset of each data combination is greater than the first threshold; determine that the noise type corresponding to each data combination is salt and pepper noise if the variance of the second feature data subset of each data combination does not exceed the first threshold and the sparsity value is greater than the second threshold; and determine that the noise type corresponding to each data combination is masking noise if the variance of the second feature data subset of each data combination does not exceed the first threshold and the sparsity value does not exceed the second threshold.

[0158] Based on the same inventive concept, an electronic device is provided. Please refer to Figure 6Fig. 1 shows a schematic diagram of an electronic device according to an embodiment of the present application. As shown in Fig. 1, the electronic device 600 includes at least one processor 601 and a memory 602 connected to the at least one processor 601. Figure 6

[0159] The processor 601 can be a general purpose processor or a special purpose processor, etc. The processor 601 includes, for example, a baseband processor or a central processing unit, etc. The baseband processor can be configured to process communication protocols and communication data. The central processing unit can be configured to control the electronic device 600, execute software programs and / or process data. Different processors can be independent devices or can be arranged in one or more processing circuits, for example, integrated on one or more application specific integrated circuits.

[0160] In one embodiment, the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 implements the functions of the anomaly detection apparatus as described above by executing the instructions stored in the memory 602. Accordingly, the at least one processor 601 can also implement the steps performed by the anomaly detection apparatus as described above.

[0161] In this embodiment, the electronic device 600 can also implement the functions of the anomaly detection apparatus 400 as described above. In addition, the at least one processor 601 in the electronic device 600 can also implement the functions of the acquisition module 401, the extraction module 402, the detection module 403 and the training module 404 as described above.

[0162] In another embodiment, the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 implements the functions of the training apparatus as described above by executing the instructions stored in the memory 602. Accordingly, the at least one processor 601 can also implement the steps performed by the training apparatus as described above.

[0163] In this embodiment, the electronic device 600 can also implement the functions of the anomaly detection model training apparatus 500 as described above. In addition, the at least one processor 601 in the electronic device 600 can also implement the functions of the extraction module 501, the determination module 502, the noise processing module 503 and the training module 504 as described above.

[0164] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium storing computer instructions, when the computer instructions are run on a computer, the computer executes the anomaly detection method or the anomaly detection model training method as described above. The anomaly detection method is, for example, the anomaly detection method as shown in Fig. 2, and the anomaly detection model training method is, for example, the anomaly detection model training method as shown in Fig. 5. Figure 2 Figure 3 The anomaly detection model training method is, for example, the anomaly detection model training method as shown in Fig. 5.

[0165] ​​Based on the same inventive concept, the embodiment of the present application provides a computer program product comprising computer instructions which, when executed on a computer, cause the above-mentioned abnormality detection method or abnormality detection model training method to be implemented, the abnormality detection method being, for example, the abnormality detection method shown in Figure 2 the abnormality detection model training method being, for example, the abnormality detection model training method shown in Figure 3 .

[0166] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0167] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0168] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in one or more flows and / or blocks.

[0170] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for detecting anomalies, characterized in that: include: Collect network traffic data of target IoT devices within a preset time window; Extracting a first feature data set from the network traffic data, the first feature data set comprising a first feature data subset of M pairs of data combinations, where one pair of data combinations corresponds to traffic data of one dimension in the network traffic data, and M is a positive integer; Inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, wherein the detection value is used to indicate a degree of difference between the network traffic data and reference network traffic data; Determining whether the target IoT device is abnormal based on the detection value; Among them, the anomaly detection model is trained based on a sample feature data set, and the sample feature data set is obtained by extracting a second feature data set from sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

2. The method according to claim 1, characterized in that Extracting a first feature data set from the network traffic data includes: Determining, according to preconfigured business requirements, M pairs of data combinations for generating the first feature data set; Determine, based on the network traffic data, a data statistic value generated by each pair of data combinations in the M pairs of data combinations within a unit time of the preset time window; Determine, based on the data statistics generated by each pair of data combinations within the unit time of the preset time window, the feature vectors of each pair of data combinations within the unit time, and form a first feature data subset of each pair of data combinations within the preset time window; The first feature data set is determined according to a first feature data subset of the M pairs of data combinations within the preset time window.

3. The method according to claim 2, characterized in that The M pairs of data combinations include one or more pairs of data combinations below: The average value of the number of data packets and the size of data packets per unit time of the network traffic data; The network traffic data has a data volume corresponding to each data protocol in all data protocols per unit time and a usage frequency corresponding to each data protocol; The proportion of encrypted traffic in the total traffic per unit time of the network traffic data and the data volume of the encrypted traffic output by the target IoT device; The number of Internet Protocol IP addresses connected by the target IoT device within a preset time window and the frequency of failure to connect to the IP addresses.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Extracting the second feature data set from the sample network traffic data, where the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; Determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; Determining a noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; performing noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations to obtain a sample feature data set; Training an initial anomaly detection model based on the sample feature data set to obtain a detection result of the initial anomaly detection model; According to the error value between the detection result and the reference data set, the model parameters of the initial anomaly detection model are adjusted until the error value is less than a preset error value, thereby obtaining the trained anomaly detection model.

5. The method according to claim 4, characterized in that Determining the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparsity value of the second feature data subset of each pair of data combinations includes: Determining the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations and a preset strategy; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; Determining the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter according to the noise type corresponding to each pair of data combinations; The noise parameter corresponding to each pair of data combinations is determined according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

6. The method according to claim 5, characterized in that The preset strategy includes a first threshold and a second threshold; Determining the noise type corresponding to each pair of data combinations according to the variance and the sparsity value of the second feature data subset of each pair of data combinations and a preset strategy includes: If the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determining that the noise type corresponding to each pair of data combinations is Gaussian noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value is greater than the second threshold, determining that the noise type corresponding to each pair of data combinations is salt and pepper noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, it is determined that the noise type corresponding to each pair of data combinations is masking noise.

7. A method for training an anomaly detection model, characterized in that: include: Extracting a second feature data set from the sample network traffic data, wherein the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; Determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; Determining a noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; performing noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations to obtain a sample feature data set, wherein the sample feature data set includes at least one abnormal situation; Training an initial anomaly detection model based on the sample feature data set to obtain a detection result of the initial anomaly detection model; According to the error value between the detection result and the reference data set, the model parameters of the initial anomaly detection model are adjusted until the error value is less than the preset error value, thereby obtaining a trained anomaly detection model. The trained anomaly detection model is used to determine whether the target IoT device is abnormal based on the network traffic data of the target IoT device.

8. The method according to claim 7, characterized in that Determining the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparsity value of the second feature data subset of each pair of data combinations includes: Determining the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations and a preset strategy; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; Determining the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter according to the noise type corresponding to each pair of data combinations; The noise parameter corresponding to each pair of data combinations is determined according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

9. The method according to claim 8, characterized in that The preset strategy includes a first threshold and a second threshold; Determining the noise type corresponding to each pair of data combinations according to the variance and the sparsity value of the second feature data subset of each pair of data combinations and a preset strategy includes: If the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determining that the noise type corresponding to each pair of data combinations is Gaussian noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparsity value is greater than the second threshold, determining that the noise type corresponding to each pair of data combinations is salt and pepper noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, it is determined that the noise type corresponding to each pair of data combinations is masking noise.

10. An abnormality detection device, characterized in that: include: A collection module, used to collect network traffic data of target IoT devices within a preset time window; an extraction module, configured to extract a first feature data set from the network traffic data, the first feature data set comprising a first feature data subset of M pairs of data combinations, where one pair of data combinations corresponds to traffic data of one dimension in the network traffic data, and M is a positive integer; a detection module, configured to input the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, wherein the detection value is used to indicate a degree of difference between the network traffic data and reference network traffic data; The detection module is further configured to determine whether the target IoT device is abnormal based on the detection value; Among them, the anomaly detection model is trained based on a sample feature data set, and the sample feature data set is obtained by extracting a second feature data set from sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

11. An anomaly detection model training device, characterized in that: include: An extraction module is used to extract a second feature data set from the sample network traffic data, where the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; a determination module, configured to determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations based on the second feature data subset of each pair of data combinations and a preset calculation rule; The determining module is further configured to determine the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; a noise processing module, configured to perform noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations, to obtain a sample feature data set, wherein the sample feature data set includes at least one abnormal condition; A training module, configured to train an initial anomaly detection model based on the sample feature data set to obtain a detection result of the initial anomaly detection model; The training module is further used to adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than a preset error value, thereby obtaining a trained anomaly detection model. The trained anomaly detection model is used to determine whether the target IoT device is abnormal based on the network traffic data of the target IoT device.

12. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 9 by executing the instructions stored in the memory.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 9.

14. A computer program product, characterized in that The invention comprises computer instructions, which, when executed on a computer, enable the method according to any one of claims 1 to 9 to be implemented.

Images