Abnormal detection method and device, abnormal detection model training method and device and electronic equipment

Abstract

The invention provides an anomaly detection method and device, an anomaly detection model training method and device and electronic equipment, relates to the technical field of network security, and is used for improving the security of Internet of Things equipment. The anomaly detection method comprises the following steps: collecting network traffic data of target Internet of Things equipment in a preset time window; extracting a first feature data set from the network traffic data, wherein the first feature data set comprises a first feature data subset of M pairs of data combinations; inputting the first feature data set into a trained anomaly detection model, obtaining a detection value output by the anomaly detection model, and determining whether the target Internet of Things equipment is abnormal according to the difference degree between the network traffic data indicated by the detection value and the reference network traffic data; the anomaly detection model is obtained by training a sample feature data set obtained by carrying out noise addition processing on a second feature data set, the second feature data set is extracted from sample network traffic, and the sample feature data set comprises at least one anomaly condition.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to anomaly detection, anomaly detection model training method, device and electronic equipment. Background Art

[0002] With the rapid development of IoT technology, IoT technology has been widely used in smart homes, smart cities, industry, medical health and other fields. However, due to the weak security mechanism of IoT devices, they have become one of the main targets of network attacks. Therefore, how to improve the security of IoT devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides an anomaly detection, anomaly detection model training method, device and electronic equipment for improving the security of Internet of Things devices.

[0004] In a first aspect, an embodiment of the present application provides an anomaly detection method, including: collecting network traffic data of a target Internet of Things device within a preset time window; extracting a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M is a positive integer; inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, the detection value being used to indicate the degree of difference between the network traffic data and reference network traffic data; determining whether the target Internet of Things device is abnormal based on the detection value; wherein the anomaly detection model is trained based on a sample feature data set, the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

[0005] In the embodiment of the present application, since the second feature data set is subjected to noise processing during training, the types of noise included in the obtained sample feature data set are richer, and various abnormal situations existing in network traffic data in real scenes can be simulated more realistically. Then, the anomaly detection model obtained by training based on the sample feature data set can better identify and process the noise in the network traffic data when actually used, and improve the ability to reconstruct normal network traffic data, so that it can detect abnormal network traffic data more accurately, that is, it is conducive to improving the accuracy of anomaly detection of IoT devices, thereby helping to improve the security of IoT devices.

[0006] In one possible implementation, extracting a first feature data set from the network traffic data includes: determining, based on preconfigured business requirements, M pairs of data combinations used to generate the first feature data set; determining, based on the network traffic data, data statistics generated by each pair of data combinations in the M pairs of data combinations within a unit time of the preset time window; determining, based on the data statistics generated by each pair of data combinations within a unit time of the preset time window, feature vectors of each pair of data combinations within a unit time, and forming a first feature data subset for each pair of data combinations within the preset time window; determining the first feature data set based on the first feature data subset of the M pairs of data combinations within the preset time window.

[0007] In one possible implementation, the M pairs of data combinations include one or more pairs of data combinations: the average value of the number of data packets and the size of data packets in the network traffic data per unit time; the data volume corresponding to each data protocol in all data protocols in the network traffic data per unit time and the usage frequency corresponding to each data protocol; the proportion of encrypted traffic in the total traffic of the network traffic data per unit time and the data volume of encrypted traffic output by the target IoT device; the number of Internet Protocol IP addresses connected to the target IoT device per unit time in a preset time window and the frequency of failure to connect to the IP address.

[0008] In a possible implementation, the method further includes: extracting the second feature data set from the sample network traffic data, the second feature data set including second feature data subsets of N pairs of data combinations, where N is a positive integer; determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; determining the noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; performing noise processing on the second feature data subset of each pair of data combinations according to the noise parameter corresponding to each pair of data combinations to obtain a sample feature data set; training an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; adjusting the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than a preset error value to obtain the trained anomaly detection model.

[0009] In a possible implementation, determining the noise parameters corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations includes: determining the noise type corresponding to each pair of data combinations according to the variance and sparsity value of the second feature data subset of each pair of data combinations and a preset strategy, the sparse value being used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; determining the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter according to the noise type corresponding to each pair of data combinations; and determining the noise parameters corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0010] In a possible implementation, the preset strategy includes a first threshold and a second threshold; according to the variance and sparsity value of the second feature data subset of each pair of data combinations, and the preset strategy, the noise type corresponding to each pair of data combinations is determined, including: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, then the noise type corresponding to each pair of data combinations is determined to be Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, then the noise type corresponding to each pair of data combinations is determined to be salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, then the noise type corresponding to each pair of data combinations is determined to be masking noise.

[0011] In a second aspect, an embodiment of the present application provides an anomaly detection model training method, comprising: extracting a second feature data set from sample network traffic data, the second feature data set including a second feature data subset of N pairs of data combinations, where N is a positive integer; determining the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; determining the noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; performing noise processing on the second feature data subset of each pair of data combinations according to the noise parameter corresponding to each pair of data combinations to obtain a sample feature data set, the sample feature data set including at least one abnormal situation; training an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; adjusting the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than the preset error value, to obtain a trained anomaly detection model, the trained anomaly detection model being used to determine whether the target Internet of Things device is abnormal according to the network traffic data of the target Internet of Things device.

[0012] In a possible implementation, determining the noise parameters corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations includes: determining the noise type corresponding to each pair of data combinations according to the variance and sparsity value of the second feature data subset of each pair of data combinations and a preset strategy, the sparse value being used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; determining the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter according to the noise type corresponding to each pair of data combinations; and determining the noise parameters corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0013] In a possible implementation, the preset strategy includes a first threshold and a second threshold; according to the variance and sparsity value of the second feature data subset of each pair of data combinations, and the preset strategy, the noise type corresponding to each pair of data combinations is determined, including: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, then the noise type corresponding to each pair of data combinations is determined to be Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, then the noise type corresponding to each pair of data combinations is determined to be salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, then the noise type corresponding to each pair of data combinations is determined to be masking noise.

[0014] In a third aspect, an embodiment of the present application provides an anomaly detection device, comprising: an acquisition module, used to collect network traffic data of a target Internet of Things device within a preset time window; an extraction module, used to extract a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M is a positive integer; a detection module, used to input the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, the detection value being used to indicate the degree of difference between the network traffic data and the reference network traffic data; the detection module is also used to determine whether the target Internet of Things device is abnormal based on the detection value; wherein the anomaly detection model is trained based on a sample feature data set, the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

[0015] In a possible implementation, the extraction module is specifically used to: determine M pairs of data combinations for generating the first feature data set according to preconfigured business requirements; determine the data statistics generated by each pair of data combinations in the M pairs of data combinations within the unit time of the preset time window according to the network traffic data; determine the feature vectors of each pair of data combinations within the unit time according to the data statistics generated by each pair of data combinations within the unit time of the preset time window, and form the first feature data subset for each pair of data combinations within the preset time window; determine the first feature data set according to the first feature data subset of the M pairs of data combinations within the preset time window.

[0016] In one possible implementation, the M pairs of data combinations include one or more pairs of data combinations: the average value of the number of data packets and the size of data packets in the network traffic data per unit time; the data volume corresponding to each data protocol in all data protocols in the network traffic data per unit time and the usage frequency corresponding to each data protocol; the proportion of encrypted traffic in the total traffic of the network traffic data per unit time and the data volume of encrypted traffic output by the target IoT device; the number of Internet Protocol IP addresses connected to the target IoT device per unit time in a preset time window and the frequency of failure to connect to the IP address.

[0017] In a possible implementation, the anomaly detection device further includes a training module, which is used to extract the second feature data set from the sample network traffic data, the second feature data set including a second feature data subset of N pairs of data combinations, where N is a positive integer; determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; determine the noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; perform noise processing on the second feature data subset of each pair of data combinations according to the noise parameter corresponding to each pair of data combinations to obtain a sample feature data set; train an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than a preset error value to obtain the trained anomaly detection model.

[0018] In a possible implementation, the training module is specifically used to: determine the noise type corresponding to each pair of data combinations according to the variance and sparsity value of the second feature data subset of each pair of data combinations, and a preset strategy, and the sparse value is used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; according to the noise type corresponding to each pair of data combinations, determine the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter; determine the noise parameter corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0019] In a possible implementation, the preset strategy includes a first threshold and a second threshold; the training module is specifically used to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to the each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determine that the noise type corresponding to the each pair of data combinations is salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, determine that the noise type corresponding to the each pair of data combinations is masking noise.

[0020] In a fourth aspect, an embodiment of the present application provides an anomaly detection model training device, comprising: an extraction module, used to extract a second feature data set from sample network traffic data, the second feature data set comprising a second feature data subset of N pairs of data combinations, N being a positive integer; a determination module, used to determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations based on the second feature data subset of each pair of data combinations and a preset calculation rule; the determination module is also used to determine the noise parameter corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; a noise processing module, used to determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations; Corresponding noise parameters are used to perform noise processing on the second feature data subset of each pair of data combinations to obtain a sample feature data set, and the sample feature data set includes at least one abnormal situation; a training module is used to train an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; the training module is also used to adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than a preset error value, so as to obtain a trained anomaly detection model, and the trained anomaly detection model is used to determine whether the target Internet of Things device is abnormal according to the network traffic data of the target Internet of Things device.

[0021] In a possible implementation, the determination module is specifically used to: determine the noise type corresponding to each pair of data combinations according to the variance and sparsity value of the second feature data subset of each pair of data combinations, and a preset strategy, and the sparse value is used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; according to the noise type corresponding to each pair of data combinations, determine the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter; determine the noise parameter corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

[0022] In a possible implementation, the preset strategy includes a first threshold and a second threshold; the determination module is specifically used to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to the each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determine that the noise type corresponding to the each pair of data combinations is salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, determine that the noise type corresponding to the each pair of data combinations is masking noise.

[0023] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the method described in the first aspect or the second aspect and any possible implementation manner above by executing the instructions stored in the memory.

[0024] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect or the second aspect and any possible implementation manner as described above.

[0025] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed on a computer, enables the method described in the first aspect or the second aspect and any possible implementation manner as described above to be implemented.

[0026] Regarding the beneficial effects of the second to seventh aspects, reference can be made to the contents described in the first aspect above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of an application scenario of an anomaly detection method provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of an anomaly detection method provided in an embodiment of the present application;

[0029] Figure 3 A flowchart of an anomaly detection model training method provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the structure of an abnormality detection device provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of the structure of an anomaly detection model training device provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to better understand the technical solution provided by the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0034] Please refer to Figure 1 , is a schematic diagram of an application scenario of an anomaly detection method provided in an embodiment of the present application. Figure 1 As shown, the scenario includes a target IoT device 110, an anomaly detection device 120, and other devices 130. The target IoT device 110 can perform wired or wireless communication with the anomaly detection device 120 and other devices 130, respectively.

[0035] Among them, the target IoT device 110 refers to an intelligent IoT device that is connected to the Internet in real time, such as smart home devices, smart wearable devices, smart city devices, and industrial IoT devices. The anomaly detection device 120 refers to a device or equipment with data processing capabilities and anomaly detection capabilities. The anomaly detection device 120 may specifically refer to a device or equipment configured with an anomaly detection model, and the anomaly detection model can be used to detect whether the target IoT device 110 is abnormal. The device may specifically be a functional module or detection device in the target IoT device 110 or other detection devices; the device may specifically be a terminal device or server, and the terminal device includes but is not limited to mobile phones, personal computers (PC), tablet computers, laptops, PDAs, and mobile Internet devices (MID). Other devices 130 refer to devices that can communicate with the target IoT device 110, such as terminal devices or servers. The terminal devices may correspond to the content described above and will not be repeated here. Among them, other devices 130 are Figure 1 A server is used as an example, but in actual applications, the other device 130 may refer to all devices that communicate with the target IoT device, that is, the number of other devices 130 may be one or more, which is not limited in the embodiments of the present application.

[0036] Exemplarily, the anomaly detection device 120 can collect network traffic data between the target IoT device 110 and other devices 130 in real time, extract a first feature data set from the network traffic data, input the first feature data set into the trained anomaly detection model, obtain a detection value, and then determine whether the target IoT device 110 is abnormal based on the detection value. The specific implementation method of the anomaly detection device 120 determining whether the target IoT device 110 is abnormal based on the network traffic data will be described in detail below.

[0037] Please refer to Figure 2 , is a flow chart of an abnormality detection method provided in an embodiment of the present application. The following is a flow chart of an abnormality detection method performed by an abnormality detection device. Figure 2 The steps shown are described below. For example, the abnormality detection device is Figure 1 The abnormality detection device 120 shown, Figure 2 The target IoT devices involved are, for example, Figure 1 The target IoT device 110 is shown.

[0038] S201, collecting network traffic data of target IoT devices within a preset time window.

[0039] Among them, the preset time window may be configured in advance in the anomaly detection device, and the length of the preset time window may be set according to actual needs. For example, the preset time window is 10 seconds, which is not limited in the embodiment of the present application. The network traffic data may be all network traffic data received and sent by the target IoT device within the preset time window. On the one hand, the network traffic data may include traffic data of multiple protocols, such as the message queuing telemetry transport (MQTT) protocol, the hypertext transfer protocol (HTTP) protocol, and the constrained application protocol (Coap) protocol. On the other hand, the network traffic data may also include encrypted traffic data and non-encrypted traffic data.

[0040] Specifically, the anomaly detection device can monitor the target IoT device and obtain the network traffic data between the target IoT and other devices in real time within a preset time window. Figure 1 The other devices 130 shown are not described in detail here.

[0041] S202, extracting a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M is a positive integer.

[0042] The M pairs of data combinations may include one or more of the following pairs of data combinations:

[0043] 1. Network traffic data refers to the average amount of data packets and the size of data packets per unit time.

[0044] 2. Network traffic data includes the amount of data corresponding to each data protocol among all data protocols in unit time and the usage frequency of each data protocol.

[0045] 3. The proportion of encrypted traffic in the total traffic per unit time and the amount of encrypted traffic output by the target IoT device.

[0046] 4. The number of Internet Protocol IP addresses that the target IoT device connects to per unit time in the preset time window and the frequency of failure to connect to the IP addresses.

[0047] Exemplarily, the unit time in the embodiments of the present application may refer to time per second.

[0048] It should be noted that the above-mentioned data combination is the data combination exemplified in the embodiment of the present application. However, in actual applications, since the dimensions of traffic data related to different businesses are different, other data combinations can be selected according to business needs, that is, traffic data of other dimensions can be selected for anomaly detection. The embodiment of the present application does not limit this.

[0049] The first feature data subset corresponding to each pair of data combinations in the M pairs of data combinations can be understood as a set of feature vectors of each pair of data combinations within a preset time window.

[0050] The specific implementation method of the anomaly detection device extracting the first feature data set from network traffic data is described in detail below.

[0051] The anomaly detection device determines M pairs of data combinations for generating the first feature data set according to the preconfigured business requirements, and determines the data statistics generated by each pair of data combinations in the M pairs of data combinations within the unit time of the preset time window according to the network traffic data. In other words, the data statistics generated by each pair of data combinations within the unit time are counted, for example, the number of data packets per second is counted. Then, the anomaly detection device can determine the feature vector of each pair of data combinations within the unit time according to the data statistics generated by each pair of data combinations within the unit time of the preset time window. For example, if a pair of data combinations includes A data and B data, the feature vector of the pair of data combinations within the unit time can be expressed as: [data statistics of A data within the unit time, data statistics of B data within the unit time]. Based on this operation, the anomaly detection device can obtain all feature vectors of each pair of data combinations within the preset time window, and then form the first feature data subset of each pair of data combinations within the preset time window. Finally, the anomaly detection device can determine the first feature data set according to the first feature data subset of the M pairs of data combinations within the preset time window.

[0052] In a possible implementation, since there may be missing data or abnormal data in the network traffic data, before determining the first feature data subset for each pair of data combinations, the feature vector may be cleaned to process the missing values ​​and abnormal values ​​in the feature vector. Specifically, the missing values ​​and abnormal values ​​may be replaced by a preset strategy, which may be set according to actual needs. For example, the missing values ​​may be replaced by the average value of the data combination corresponding to the missing value, which is not limited in the present embodiment of the application.

[0053] In a possible implementation, in order to make the first feature data set meet the subsequent detection requirements, before determining the first feature data subset of each pair of data combination within the preset time window, the anomaly detection device determines the feature vector of each pair of data combination within the unit time, and then normalizes each feature vector. Exemplarily, the anomaly detection device may use Min-Max to normalize the feature vector to between [0, 1].

[0054] Exemplarily, in order to better understand the specific implementation method of the anomaly detection device determining the first feature data set, the preset time window is 5 seconds and the unit time is per second as an example, combined with the data combinations 1-4 of the above example, an example is given below.

[0055] 1. Network traffic data refers to the average amount of data packets and the size of data packets per unit time.

[0056] The anomaly detection device collects the number of packets per second and the average packet size within a preset time window, wherein the number of packets per second within 1 to 5 seconds is: [10, 20, 15, 25, 30], and the average packet size per second within 1 to 5 seconds is: [145, 200, 160, 180, 220]. The anomaly detection device combines the number of packets per second and the average packet size per second into a feature vector, i.e., feature vector = [number of packets per second, average packet size per second].

[0057] Exemplarily, the first (first second) feature vector is [10, 145]; the second (second second) feature vector = [20, 200]... and so on.

[0058] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0059] Optionally, the anomaly detection device may normalize the above-mentioned feature vector to [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0, 0]; and for the second time window is: [0.5, 0.733].

[0060] 2. Network traffic data includes the amount of data corresponding to each data protocol among all data protocols in unit time and the usage frequency of each data protocol.

[0061] The anomaly detection device collects raw data on the data volume ratio and usage frequency of different protocols within a preset time window, including the following data statistics:

[0062] HTTP protocol data volume ratio; [0.6, 0.5, 0.7, 0.8, 0.6];

[0063] MQTT protocol data volume ratio: [0.3, 0.4, 0.2, 0.1, 0.2];

[0064] CoAP protocol data volume ratio: [0.1, 0.1, 0.1, 0.1, 0.2];

[0065] HTTP protocol usage frequency: [50, 60, 70, 80, 60];

[0066] MQTT protocol usage frequency: [30, 40, 20, 10, 20];

[0067] CoAP protocol usage frequency: [10, 10, 10, 10, 20];

[0068] The anomaly detection device combines the data volume ratio and usage frequency of all protocols in each time window (per second) into a feature vector, that is, the feature vector = [HTTP data volume ratio, MQTT data volume ratio, CoAP data volume ratio, HTTP usage frequency, MQTT usage frequency, CoAP usage frequency]

[0069] Exemplarily, the first (first second) feature vector is: [0.6, 0.3, 0.1, 50, 30, 10], the second feature vector is: [0.5, 0.4, 0.1, 60, 40, 10]... and so on.

[0070] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0071] Optionally, the anomaly detection device may normalize the above feature vector to [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0.33, 0.67, 0, 0, 0.67, 0].

[0072] 3. The proportion of encrypted traffic in the total traffic per unit time and the amount of encrypted traffic output by the target IoT device.

[0073] The anomaly detection device collects the proportion of encrypted traffic to total traffic and the amount of data output by the encryption protocol within a preset time window, including the following data statistics:

[0074] The proportion of encrypted traffic to total traffic: [0.8, 0.6, 0.9, 0.75, 0.7];

[0075] The amount of data output by the encryption protocol: [500, 400, 600, 550, 450] (unit: MB);

[0076] The anomaly detection device combines the proportion of encrypted traffic to total traffic per second and the amount of data output by the encryption protocol into a feature vector, that is, feature vector = [the proportion of encrypted traffic to total traffic, the amount of data output by the encryption protocol].

[0077] For example, the first eigenvector = [0.8, 500]; the second eigenvector = [0.6, 400], and so on.

[0078] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0079] Optionally, the anomaly detection device normalizes the above feature vector to between [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0.67, 0.5]; and the second time window is: [0, 0].

[0080] 4. The number of Internet Protocol IP addresses that the target IoT device connects to per unit time in the preset time window and the frequency of failure to connect to the IP addresses.

[0081] The anomaly detection device collects the number of connections to different IP addresses within a preset time window and counts the frequency of failed connections, including the following data statistics:

[0082] Number of connections to different IP addresses: [10, 15, 20, 25, 30]

[0083] Statistics of the frequency of failed connections to IP addresses: [1, 2, 1.5, 3, 2.5] (unit: times / second)

[0084] The anomaly detection device combines the number of connections to different IP addresses per second and the frequency of statistically failed connections into a feature vector, that is, feature vector = [number of connections to different IP addresses, frequency of statistically failed connections].

[0085] For example, the first eigenvector = [10, 1]; the second eigenvector = [15, 2], and so on.

[0086] Based on this, the anomaly detection device can obtain the feature vectors from the 1st second to the 5th second, and then determine these 5 feature vectors as the first feature data subset of the data combination.

[0087] Optionally, the anomaly detection device normalizes the above feature vector to between [0, 1] using Min-Max. For example, the normalized feature vector for the first time window is: [0, 0]; and the second time window is: [0.25, 5].

[0088] In summary, the anomaly detection device can combine the first feature data subsets of the above four pairs of data into a first feature data set.

[0089] S203, input the first feature data set into the trained anomaly detection model to obtain a detection value output by the anomaly detection model, the detection value is used to indicate the degree of difference between the network traffic data and the reference network traffic data, the anomaly detection model is trained based on a sample feature data set, the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

[0090] Among them, the anomaly detection model can be a denoising autoencoder (DAE) model.

[0091] Since the second feature data set is denoised when training the anomaly detection model, the anomaly detection model has better processing capabilities for various noises in real scenarios. Therefore, when denoising the network traffic data of the target IoT device, the denoising accuracy is better and the data reconstruction capability is better, so that the degree of difference between the network traffic data and the reference network traffic data can be better identified, and a more accurate detection value can be output.

[0092] In order to better understand the training process of the anomaly detection model, the training process of the anomaly detection model is described in detail below.

[0093] The anomaly detection device can extract a second feature data set from the sample network traffic data, and the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer. Among them, the N pairs of data combinations can be one or more of the multiple data combinations described above. The N pairs of data combinations can be completely the same or partially the same as the M pairs of data combinations, and the embodiments of the present application are not limited to this.

[0094] Among them, the specific method in which the anomaly detection device extracts the second feature data set from the sample network traffic data can correspond to the specific method in which the anomaly detection device extracts the first feature data set from the network traffic data in the previous text, and will not be repeated here.

[0095] After obtaining the second feature data set, the variance and / or sparseness value of the second feature data subset of each pair of data combinations is determined according to the second feature data subset of each pair of data combinations in the N pairs of data combinations and the preset calculation rule, and the sparseness value is used to indicate the sparsity of the second feature data subset of each pair of data combinations. Among them, the preset calculation rule can be a calculation formula related to the variance and the sparseness value. Exemplarily, a calculation formula for calculating the variance of the second feature data subset of each pair of data combinations is as follows:

[0096]

[0097] Wherein, Var(X) represents the variance of the second feature data subset of each pair of data combinations, n represents the number of feature vectors included in the second feature data subset, and X i represents the i-th eigenvector in the second feature data subset in each pair of data combinations, and μ represents the mean of the second feature data subset.

[0098] A calculation formula for calculating the sparse value of the second feature data subset for each pair of data combinations is as follows:

[0099]

[0100] Wherein, Sparsity(X) represents the sparse value of the second feature data subset for each pair of data combinations, n represents the number of eigenvectors included in the second feature data subset, and m represents the number of columns of the matrix corresponding to each eigenvector in the second feature data subset.

[0101] 1. The anomaly detection device determines the noise parameter corresponding to each pair of data combinations according to the variance and / or sparse value of the second feature data subset of each pair of data combinations. The following describes the three methods for determining the noise parameter corresponding to each pair of data combinations.

[0102] Method 1: The anomaly detection device determines the noise parameter corresponding to each pair of data combinations according to the variance of the second feature data subset of each pair of data combinations.

[0103] In this way, since the variance can reflect the discreteness of the data and thus measure the volatility of the data, Gaussian noise can be selected as the added noise, and Gaussian noise can be used to simulate the random fluctuations or measurement errors in the network traffic data. Based on this, the anomaly detection device can determine the noise parameters of the Gaussian noise of each pair of data combinations according to the variance of the second feature data subset of each pair of data combinations. The noise parameters of the Gaussian noise are, for example, the standard deviation of the Gaussian noise. Exemplarily, a calculation formula for determining the noise parameters of the Gaussian noise of each pair of data combinations is as follows:

[0104] sigma=α1*Var(X)

[0105] Among them, sigma represents the standard layer of Gaussian noise, α1 is a custom constant, and Var(X) represents the variance of each pair of data combinations.

[0106] Method 2: The anomaly detection device determines the noise parameter corresponding to each pair of data combinations according to the sparse value of the second feature data subset of each pair of data combinations.

[0107] In this way, since the sparse value can reflect the proportion of zero elements in the feature data and can measure the sparsity of the data, salt and pepper noise or masking noise can be selected as the added noise, wherein the salt and pepper noise can simulate abnormal data caused by sensor failure and transmission error. Masking noise can simulate the abnormal situation of missing data. Based on this, the abnormality detection device can determine the noise parameter corresponding to each pair of data combinations according to the sparse value of the second feature data subset of each pair of data combinations. The noise parameter can be, for example, the salt and pepper noise probability or the masking noise probability.

[0108] Exemplarily, if the added noise is salt and pepper noise, a calculation formula for determining the noise parameter of the salt and pepper noise for each pair of data combinations is as follows:

[0109] P(Salt)=α2*Sparsity(X)

[0110] Among them, P(Salt) is the probability of salt and pepper noise, α2 is a custom constant, and Sparsity(X) is the sparse value of each pair of data combinations.

[0111] If the added noise is masking noise, a calculation formula for determining the noise parameter of the masking noise for each pair of data combinations is as follows:

[0112] P(Masking)=α3*(1-Sparsity(X))

[0113] Among them, P (Masking) is the probability of masking noise, α3 is a custom constant, and Sparsity (X) is the sparse value of each pair of data combinations.

[0114] Method three: The anomaly detection device determines the noise parameter corresponding to each pair of data combinations according to the variance and the sparsity value of the second feature data subset of each pair of data combinations.

[0115] Since the variance and sparseness value of the second feature data subset of each pair of data combinations are different in the N pairs of data combinations, the data characteristics reflected by the second feature data subset are also different. Therefore, according to the variance and sparseness value of the second feature data subset of each pair of data combinations, noise data of a noise type matching its data characteristics can be determined as the added noise data.

[0116] Specifically, the anomaly detection device may determine the noise type corresponding to each pair of data combinations according to the variance and sparseness value of the second feature data subset of each pair of data combinations and the preset strategy, wherein the noise type may include Gaussian noise, salt and pepper noise, and masking noise. The preset strategy may include a first threshold and a second threshold, and the preset strategy is used to indicate the relationship between the variance and sparseness value of the second feature data subset of each pair of data combinations and the noise type. The specific manner in which the anomaly detection device determines the noise type corresponding to each pair of data combinations according to the preset strategy and the variance and sparseness value of the second feature data subset of each pair of data combinations is described in detail below.

[0117] If the anomaly detection device determines that the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, the noise type corresponding to each pair of data combinations is determined to be Gaussian noise. If the anomaly detection device determines that the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, the noise type corresponding to each pair of data combinations is determined to be salt and pepper noise. If the anomaly detection device determines that the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, the noise type corresponding to each pair of data combinations is determined to be masking noise.

[0118] After determining the noise type corresponding to each pair of data combinations in N pairs of data combinations, the first parameter can be determined. The first parameter is the variance or sparse value of the second characteristic data subset of each pair of data combinations. For example, if the noise type is Gaussian noise, the first parameter is the variance of the second characteristic data subset of each pair of data combinations. If the noise type is salt and pepper noise or masking noise, the first parameter is the sparse value of the second characteristic data subset of each pair of data combinations.

[0119] Then, the abnormality detection device can determine the noise parameter corresponding to each pair of data combinations according to the noise type corresponding to each pair of data combinations, the first parameter and the first formula corresponding to the noise type. The first formula is the calculation formula for calculating the noise parameter provided in the above-mentioned method 1 and method 2. According to the corresponding noise type, the calculation formula corresponding to it can be selected to calculate the noise parameter, which will not be repeated here.

[0120] It should be noted that the embodiments of the present application are illustrated using the above three types of noise. In order to better simulate the real environment, other noises may also be used as added noises, such as random packet loss noise, which is not limited in the embodiments of the present application.

[0121] The anomaly detection device performs noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations, obtains the second feature data subset after noise processing for each pair of data combinations, and combines the second feature data subsets after noise processing in N pairs of data combinations into a sample feature data set.

[0122] Finally, the anomaly detection device can input the sample feature data set into the initial anomaly detection model, train the initial anomaly detection model, obtain the detection result output by the initial anomaly detection model, adjust the model parameters of the initial anomaly detection model based on the error value between the detection result and the reference data set, and repeat the above training process until the error value between the detection result output by the initial anomaly detection model and the reference data set is less than the preset error value, thereby obtaining a trained anomaly detection model.

[0123] Furthermore, the anomaly detection device can input the first feature data set into the trained anomaly detection model, denoise the first feature data set through the trained anomaly detection model, and reconstruct the damaged feature data to obtain clean original data, compare the difference between the original data and the learned reference network traffic data, and output the detection value.

[0124] S204: Determine whether the target IoT device is abnormal based on the detection value.

[0125] Among them, a preset threshold may be pre-configured in the anomaly detection device, and the preset threshold is used to measure whether the target IoT device is abnormal. The preset threshold can be set according to actual needs, for example, 0.5, and the embodiment of the present application does not limit this.

[0126] Based on this, the abnormality detection device can determine whether the target IoT device is abnormal based on the detection value and the preset threshold. Specifically, the abnormality detection device determines whether the detection value exceeds the preset threshold. If it is determined that the detection value exceeds the preset threshold, the target IoT device is determined to be abnormal. If it is determined that the detection value does not exceed the preset threshold, the target IoT device is determined to be normal.

[0127] Exemplarily, taking the detection value as 0.7 and the preset threshold as 0.5 as an example, if the anomaly detection device determines that the detection value exceeds the preset threshold, it determines that the target IoT device is abnormal.

[0128] In a possible implementation, when the anomaly detection device determines that the target IoT device is abnormal, an alarm message may be generated and sent to the operation and maintenance device, so that the operation and maintenance device displays it on the interface and provides it to the operation and maintenance personnel for viewing, so that the operation and maintenance personnel can quickly take corresponding measures to deal with the abnormal situation of the target IoT device. In addition, when the anomaly detection device determines that the target IoT device is abnormal, the network security of the target IoT device can also be protected based on pre-configured protection measures.

[0129] In order to improve the accuracy of the anomaly detection model in detecting whether the target IoT device is abnormal, an embodiment of the present application provides an anomaly detection model training method. In this method, by performing noise processing on the second feature data set, the sample feature data set includes at least one abnormal situation, and then after the anomaly detection model is trained by the sample feature data set, the trained anomaly detection model can recognize and process various types of noise data, thereby improving the anomaly detection model's ability to process noise data, thereby improving the accuracy of the anomaly detection model in detecting anomalies of the target IoT device.

[0130] Combine the following Figure 3 The flowchart of a method for training an anomaly detection model shown in FIG. 1 illustrates the method for training an anomaly detection model provided in an embodiment of the present application. Figure 3 The steps shown are described in detail, and the training device can be, for example, Figure 1 The abnormality detection device 120 is shown.

[0131] S301, extracting a second feature data set from sample network traffic data, where the second feature data set includes a second feature data subset of N pairs of data combinations.

[0132] Among them, the specific method of extracting the second feature data set from the sample network traffic data by the training device can correspond to the specific method of extracting the first feature data set from the network traffic data by the anomaly detection device mentioned above, which will not be repeated here. The specific contents of the second feature data set, N pairs of data combinations, and the second feature data subset can also correspond to the specific contents of the first feature data set, N pairs of data combinations, and the first feature data subset mentioned above, which will not be repeated here.

[0133] S302: Determine the variance and / or sparse value of the second characteristic data subset of each pair of data combinations in the N pairs of data combinations according to the second characteristic data subset of each pair of data combinations and a preset calculation rule.

[0134] Among them, the specific manner in which the training device determines the variance and / or sparsity value of the second feature data subset for each pair of data combinations can correspond to the specific manner in which the anomaly detection device determines the variance and / or sparsity value of the second feature data subset for each pair of data combinations, and will not be repeated here.

[0135] S303: Determine the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparsity value of the second feature data subset of each pair of data combinations.

[0136] Among them, the specific manner in which the training device determines the noise parameters corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations can refer to the specific manner in which the abnormality detection device determines the noise parameters corresponding to each pair of data combinations based on the variance and / or sparsity value of the second feature data subset of each pair of data combinations, which will not be repeated here.

[0137] S304, performing noise processing on the second characteristic data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations, to obtain a sample characteristic data set.

[0138] Among them, the sample feature data set includes at least one abnormal situation.

[0139] The training device performs noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations. The specific method of obtaining the sample feature data set can correspond to the specific method of determining the sample feature data set by the anomaly detection device mentioned above, which will not be repeated here.

[0140] S305: Train the initial anomaly detection model according to the sample feature data set to obtain the detection result of the initial anomaly detection model.

[0141] S306, according to the error value between the detection result and the reference data set, adjust the model parameters of the initial anomaly detection model until the error value is less than the preset error value, and obtain the trained anomaly detection model. The trained anomaly detection model is used to determine whether the target IoT device is abnormal based on the network traffic data of the target IoT device.

[0142] The training device adjusts the model parameters of the initial anomaly detection model based on the error value between the detection result and the reference data set, and repeats the above training process until the error value between the detection result output by the initial anomaly detection model and the reference data set is less than the preset error value, thereby obtaining a trained anomaly detection model.

[0143] The trained anomaly detection model can be used to detect whether the target IoT device is abnormal. The specific method of determining whether the target IoT device is abnormal based on the trained anomaly detection model can correspond to the contents described in S201-S204 above, which will not be repeated here.

[0144] In the embodiment of the present application, due to the addition of noise data that matches the data characteristics based on different data characteristics, the feature data after adding the noise data can more realistically simulate the abnormal situation of network traffic data in real scenes. Then, the anomaly detection model obtained by training based on the feature data makes the trained anomaly detection model perform better when processing noise and abnormal data in a real environment, and can better process and recover data contaminated by noise, which is conducive to improving the robustness of the anomaly detection model and improving the stability of anomaly detection. Therefore, even if there are occasional sudden interference data in real network traffic data, the trained anomaly detection model can still effectively identify and remove these noises, improve the anomaly detection model's ability to reconstruct normal data, and thus improve the accuracy of anomaly detection of target IoT devices.

[0145] Based on the same inventive concept, an embodiment of the present application provides an abnormality detection device, which is used to implement any of the above abnormality detection methods, for example, Figure 2 The abnormality detection method shown, and the device can also realize the functions of the abnormality detection device mentioned above.

[0146] Please refer to Figure 4 , is a schematic diagram of the structure of an abnormality detection device provided in an embodiment of the present application. Figure 4 As shown, the anomaly detection device 400 includes a collection module 401, an extraction module 402, and a detection module 403. Optionally, the anomaly detection device 400 may further include a training module 404.

[0147] Exemplarily, the collection module 401 is used to collect network traffic data of the target IoT device within a preset time window; the extraction module 402 is used to extract a first feature data set from the network traffic data, the first feature data set including a first feature data subset of M pairs of data combinations, one pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M is a positive integer; the detection module 403 is used to input the first feature data set into the trained anomaly detection model to obtain a detection value output by the anomaly detection model, and the detection value is used to indicate the degree of difference between the network traffic data and the reference network traffic data; the detection module 403 is also used to determine whether the target IoT device is abnormal based on the detection value; wherein the anomaly detection model is trained based on a sample feature data set, the sample feature data set is obtained by extracting a second feature data set from the sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

[0148] In one possible implementation, the extraction module 402 is specifically used to: determine M pairs of data combinations for generating a first feature data set based on preconfigured business requirements; determine the data statistics generated by each pair of data combinations in the M pairs of data combinations within a unit time of a preset time window based on network traffic data; determine the feature vectors of each pair of data combinations within a unit time based on the data statistics generated by each pair of data combinations within a unit time of the preset time window, and form a first feature data subset for each pair of data combinations within the preset time window; determine the first feature data set based on the first feature data subset of the M pairs of data combinations within the preset time window.

[0149] In one possible implementation, the M pairs of data combinations include one or more pairs of data combinations: the average value of the number of data packets and the size of data packets in the network traffic data per unit time; the data volume corresponding to each data protocol in all data protocols in the network traffic data per unit time and the usage frequency corresponding to each data protocol; the proportion of encrypted traffic in the total traffic of the network traffic data per unit time and the data volume of encrypted traffic output by the target IoT device; the number of Internet Protocol IP addresses connected to the target IoT device per unit time in a preset time window and the frequency of failure to connect to the IP address.

[0150] In a possible implementation, the anomaly detection device further includes a training module 404, which is used to extract a second feature data set from the sample network traffic data, where the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; determine the noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; perform noise processing on the second feature data subset of each pair of data combinations according to the noise parameter corresponding to each pair of data combinations to obtain a sample feature data set; train an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than the preset error value to obtain a trained anomaly detection model.

[0151] In one possible implementation, the training module 404 is specifically used to: determine the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations and a preset strategy, and the sparse value is used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; according to the noise type corresponding to each pair of data combinations, determine the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter; determine the noise parameter corresponding to each pair of data combinations based on the noise type corresponding to each pair of data combinations, the first parameter, and the first formula corresponding to the noise type.

[0152] In one possible implementation, the preset strategy includes a first threshold and a second threshold; the training module 404 is specifically used to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determine that the noise type corresponding to each pair of data combinations is salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, determine that the noise type corresponding to each pair of data combinations is masking noise.

[0153] Based on the same inventive concept, the embodiment of the present application provides an anomaly detection model training device, which is used to implement any of the above anomaly detection model training methods, for example Figure 3 The anomaly detection model training method shown, and the device can also realize the function of the training device mentioned above.

[0154] Please refer to Figure 5 , is a schematic diagram of the structure of an anomaly detection model training device provided in an embodiment of the present application. Figure 5 As shown, the anomaly detection model training device 500 includes an extraction module 501, a determination module 502, a noise processing module 503 and a training module 504.

[0155] Exemplarily, the extraction module 501 is used to extract a second feature data set from the sample network traffic data, the second feature data set including a second feature data subset of N pairs of data combinations, where N is a positive integer; the determination module 502 is used to determine the variance and / or sparsity value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; the determination module 502 is also used to determine the noise parameter corresponding to each pair of data combinations according to the variance and / or sparsity value of the second feature data subset of each pair of data combinations; the noise processing module 503 is used to determine the noise parameter corresponding to each pair of data combinations according to the noise parameter corresponding to each pair of data combinations , performing noise processing on the second feature data subset of each pair of data combinations to obtain a sample feature data set, wherein the sample feature data set includes at least one abnormal situation; a training module 504 is used to train the initial anomaly detection model according to the sample feature data set to obtain the detection result of the initial anomaly detection model; the training module 504 is also used to adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than the preset error value, thereby obtaining a trained anomaly detection model, and the trained anomaly detection model is used to determine whether the target IoT device is abnormal according to the network traffic data of the target IoT device.

[0156] In a possible implementation, the determination module 502 is specifically used to: determine the noise type corresponding to each pair of data combinations based on the variance and sparsity value of the second feature data subset of each pair of data combinations, and a preset strategy, and the sparse value is used to indicate the sparsity of the second feature data subset of each pair of data combinations; wherein the noise types include: Gaussian noise, salt and pepper noise, and masking noise; according to the noise type corresponding to each pair of data combinations, determine the variance or sparsity value of the second feature data subset of each pair of data combinations as a first parameter; determine the noise parameter corresponding to each pair of data combinations based on the noise type corresponding to each pair of data combinations, the first parameter, and the first formula corresponding to the noise type.

[0157] In a possible implementation, the preset strategy includes a first threshold and a second threshold; the determination module 502 is specifically used to: if the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determine that the noise type corresponding to each pair of data combinations is Gaussian noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determine that the noise type corresponding to each pair of data combinations is salt and pepper noise; if the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, determine that the noise type corresponding to each pair of data combinations is masking noise.

[0158] Based on the same inventive concept, the present application embodiment provides an electronic device. Figure 6, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 600 includes at least one processor 601 and a memory 602 communicatively connected to the at least one processor 601 .

[0159] The processor 601 may be a general-purpose processor or a dedicated processor. The processor 601 may include, for example, a baseband processor or a central processing unit. The baseband processor may be used to process communication protocols and communication data. The central processing unit may be used to control the electronic device 600, execute software programs and / or process data. Different processors may be independent devices or may be arranged in one or more processing circuits, for example, integrated in one or more dedicated integrated circuits.

[0160] In one embodiment, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 implements the functions of the above-mentioned abnormality detection device by executing the instructions stored in the memory 602. Accordingly, the steps performed by the above-mentioned abnormality detection device can also be implemented.

[0161] In this embodiment, the electronic device 600 can also implement the function of the preceding text abnormality detection device 400. In addition, at least one processor 601 in the electronic device 600 can also implement the functions of the preceding text acquisition module 401, the extraction module 402, the detection module 403 and the training module 404.

[0162] In another embodiment, the memory 602 stores instructions that can be executed by at least one processor 601, and the at least one processor 601 implements the functions of the training device mentioned above by executing the instructions stored in the memory 602, and accordingly, can also implement the steps performed by the training device mentioned above.

[0163] In this embodiment, the electronic device 600 can also implement the functions of the preceding anomaly detection model training device 500. In addition, at least one processor 601 in the electronic device 600 can also implement the functions of the preceding text extraction module 501, the determination module 502, the noise processing module 503 and the training module 504.

[0164] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the anomaly detection method or anomaly detection model training method as described above. The anomaly detection method is, for example, Figure 2 The anomaly detection method shown, the anomaly detection model training method is, for example, Figure 3 The anomaly detection model training method shown.

[0165] Based on the same inventive concept, the embodiment of the present application provides a computer program product, which includes computer instructions. When the computer program product is executed on a computer, the above-mentioned anomaly detection method or anomaly detection model training method is implemented. The anomaly detection method is, for example, Figure 2 The anomaly detection method shown, the anomaly detection model training method is, for example, Figure 3 The anomaly detection model training method shown.

[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An anomaly detection method, characterized in that: include: Collect network traffic data of target IoT devices within a preset time window; Extracting a first feature data set from the network traffic data, the first feature data set comprising a first feature data subset of M pairs of data combinations, a pair of data combinations corresponding to traffic data of one dimension in the network traffic data, and M being a positive integer; Inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, wherein the detection value is used to indicate a degree of difference between the network traffic data and reference network traffic data; Determine whether the target IoT device is abnormal according to the detection value; Among them, the anomaly detection model is trained based on a sample feature data set, and the sample feature data set is obtained by extracting a second feature data set from sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

2. The method according to claim 1, characterized in that Extracting a first feature data set from the network traffic data includes: Determine, according to preconfigured business requirements, M pairs of data combinations for generating the first feature data set; Determine, according to the network traffic data, a data statistic value generated by each pair of data combinations in the M pairs of data combinations within a unit time of the preset time window; Determine the feature vector of each pair of data combinations within the unit time according to the data statistics value generated by each pair of data combinations within the unit time of the preset time window, and form a first feature data subset of each pair of data combinations within the preset time window; The first feature data set is determined according to a first feature data subset of the M pairs of data combinations within the preset time window.

3. The method according to claim 2, characterized in that The M pairs of data combinations include one or more pairs of data combinations below: The average value of the amount of data packets and the size of data packets in a unit time of the network traffic data; The network traffic data has a data volume corresponding to each data protocol in all data protocols per unit time and a usage frequency corresponding to each data protocol; The proportion of the encrypted traffic of the network traffic data in the total traffic per unit time and the data volume of the encrypted traffic output by the target IoT device; The number of Internet Protocol IP addresses connected by the target IoT device within a unit time of a preset time window and the frequency of failure to connect to the IP addresses.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Extracting the second feature data set from the sample network traffic data, the second feature data set comprising a second feature data subset of N pairs of data combinations, where N is a positive integer; Determine the variance and / or sparse value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; Determine the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparse value of the second feature data subset of each pair of data combinations; According to the noise parameters corresponding to each pair of data combinations, a second feature data subset of each pair of data combinations is subjected to noise adding processing to obtain a sample feature data set; Training an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; According to the error value between the detection result and the reference data set, the model parameters of the initial anomaly detection model are adjusted until the error value is less than a preset error value, thereby obtaining the trained anomaly detection model.

5. The method according to claim 4, characterized in that Determining the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparse value of the second feature data subset of each pair of data combinations includes: Determine the noise type corresponding to each pair of data combinations according to the variance and the sparse value of the second feature data subset of each pair of data combinations and a preset strategy; wherein the noise type includes: Gaussian noise, salt and pepper noise, and masking noise; According to the noise type corresponding to each pair of data combinations, the variance or sparse value of the second feature data subset of each pair of data combinations is determined as the first parameter; The noise parameter corresponding to each pair of data combinations is determined according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

6. The method according to claim 5, characterized in that The preset strategy includes a first threshold and a second threshold; Determining the noise type corresponding to each pair of data combinations according to the variance and the sparse value of the second feature data subset of each pair of data combinations and a preset strategy includes: If the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determining that the noise type corresponding to each pair of data combinations is Gaussian noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determining that the noise type corresponding to each pair of data combinations is salt and pepper noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, it is determined that the noise type corresponding to each pair of data combinations is masking noise.

7. A method for training an anomaly detection model, characterized in that: include: Extracting a second feature data set from the sample network traffic data, wherein the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; Determine the variance and / or sparse value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; Determine the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparse value of the second feature data subset of each pair of data combinations; According to the noise parameters corresponding to each pair of data combinations, a second feature data subset of each pair of data combinations is subjected to noise adding processing to obtain a sample feature data set, wherein the sample feature data set includes at least one abnormal situation; Training an initial anomaly detection model according to the sample feature data set to obtain a detection result of the initial anomaly detection model; According to the error value between the detection result and the reference data set, the model parameters of the initial anomaly detection model are adjusted until the error value is less than the preset error value, so as to obtain a trained anomaly detection model. The trained anomaly detection model is used to determine whether the target IoT device is abnormal based on the network traffic data of the target IoT device.

8. The method according to claim 7, characterized in that Determining the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparse value of the second feature data subset of each pair of data combinations includes: Determine the noise type corresponding to each pair of data combinations according to the variance and the sparse value of the second feature data subset of each pair of data combinations and a preset strategy; wherein the noise type includes: Gaussian noise, salt and pepper noise, and masking noise; According to the noise type corresponding to each pair of data combinations, the variance or sparse value of the second feature data subset of each pair of data combinations is determined as the first parameter; The noise parameter corresponding to each pair of data combinations is determined according to the noise type corresponding to each pair of data combinations, the first parameter, and a first formula corresponding to the noise type.

9. The method according to claim 8, characterized in that The preset strategy includes a first threshold and a second threshold; Determining the noise type corresponding to each pair of data combinations according to the variance and the sparse value of the second feature data subset of each pair of data combinations and a preset strategy includes: If the variance of the second feature data subset of each pair of data combinations is greater than the first threshold, determining that the noise type corresponding to each pair of data combinations is Gaussian noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value is greater than the second threshold, determining that the noise type corresponding to each pair of data combinations is salt and pepper noise; If the variance of the second feature data subset of each pair of data combinations does not exceed the first threshold and the sparse value does not exceed the second threshold, it is determined that the noise type corresponding to each pair of data combinations is masking noise.

10. An abnormality detection device, characterized in that: include: A collection module, used to collect network traffic data of target IoT devices within a preset time window; An extraction module is used to extract a first feature data set from the network traffic data, wherein the first feature data set includes a first feature data subset of M pairs of data combinations, wherein a pair of data combinations corresponds to traffic data of one dimension in the network traffic data, and M is a positive integer; A detection module, used for inputting the first feature data set into a trained anomaly detection model to obtain a detection value output by the anomaly detection model, wherein the detection value is used to indicate a degree of difference between the network traffic data and reference network traffic data; The detection module is further used to determine whether the target IoT device is abnormal based on the detection value; Among them, the anomaly detection model is trained based on a sample feature data set, and the sample feature data set is obtained by extracting a second feature data set from sample network traffic data and performing noise processing on the second feature data set, and the sample feature data set includes at least one abnormal situation.

11. An anomaly detection model training device, characterized in that: include: An extraction module, used to extract a second feature data set from the sample network traffic data, wherein the second feature data set includes a second feature data subset of N pairs of data combinations, where N is a positive integer; A determination module, configured to determine the variance and / or sparseness value of the second feature data subset of each pair of data combinations in the N pairs of data combinations according to the second feature data subset of each pair of data combinations and a preset calculation rule; The determination module is further configured to determine the noise parameter corresponding to each pair of data combinations according to the variance and / or the sparsity value of the second feature data subset of each pair of data combinations; A noise processing module, configured to perform noise processing on the second feature data subset of each pair of data combinations according to the noise parameters corresponding to each pair of data combinations, to obtain a sample feature data set, wherein the sample feature data set includes at least one abnormal situation; A training module, used to train the initial anomaly detection model according to the sample feature data set to obtain the detection result of the initial anomaly detection model; The training module is also used to adjust the model parameters of the initial anomaly detection model according to the error value between the detection result and the reference data set until the error value is less than a preset error value, thereby obtaining a trained anomaly detection model. The trained anomaly detection model is used to determine whether the target IoT device is abnormal based on the network traffic data of the target IoT device.

12. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 9 by executing the instructions stored in the memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

14. A computer program product, characterized in that The invention comprises computer instructions, which, when executed on a computer, enable the method according to any one of claims 1 to 9 to be implemented.

Images