Method and system for mining abnormal data in industrial internet big data

Abstract

The invention discloses an abnormal data mining method and system in industrial internet big data. The method comprises the steps that 1, a mutual k neighbor set MkNN of each sample in a data set is calculated; step 2, calculating a change score VS of each sample based on the MkNN set; 3, calculating the maximum distance from each sample to the k neighbor as an isolation score IS; step 4, calculating a local change degree LVD based on the change score VS and the isolation score IS; step 5, selecting the sample with the maximum LVD value to carry out category labeling, and setting the LVD value of the sample and the k neighbor thereof to be invalid; and step 6, repeating the step 5 until an iteration termination condition is met, and outputting the marked abnormal data set. According to the method, the local change degree is introduced, the abrupt change of local data distribution can be fully utilized to position the abnormal area, and the inquiry frequency required for finding all abnormal data in the data set is remarkably reduced, so that the efficiency and the accuracy of mining the abnormal data in the industrial internet big data are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data mining, and particularly relates to an abnormal data mining method and system in industrial internet big data. BACKGROUND

[0002] The rapid development of industrial internet has led to high-frequency collection and storage of massive heterogeneous data, including numerical database tables such as device sensor time series data, production line operation parameters, and supply chain logistics records. Among these data, abnormal data (such as device failure precursors, production line process deviations, and malicious network intrusion behaviors) are often hidden in the noise of normal production patterns, but their small deviations can cause major production accidents or safety hazards. Therefore, precise mining of abnormal data in the industrial internet scenario has high application value.

[0003] When the distribution of abnormal data is very different from that of normal data and the data volume is small, existing traditional clustering and classification techniques can be used for mining, but in industrial internet big data, abnormal data usually exhibit local feature mutations (such as transient over-limiting of a sensor under certain working conditions or significant differences in data distribution between a production line section and adjacent areas), rather than global deviations. For example, at the initial stage of bearing wear, only a small tight cluster of vibration signals is formed in a certain speed range, or a small number of malicious instructions disguised as legitimate protocols in an industrial control network. Such abnormalities are easily ignored by traditional methods. Industrial scenarios require low-latency responses, and the computational efficiency of traditional algorithms under massive data is difficult to meet the real-time monitoring requirements. In order to quickly and accurately find abnormal data from industrial internet big data, it is necessary to combine the local feature adaptability of industrial internet big data with a lightweight computing architecture for abnormal data mining. SUMMARY

[0004] To solve the above technical problems, the application provides an abnormal data mining device and method in industrial internet big data. Given an industrial internet big data set The method introduces LVD (Local Variation Degree), uses the characteristics of abnormal data to identify it from the industrial internet big data. Specifically, LVD uses the difference in the number of mutual k-nearest neighbors (MkNN) set to distinguish abnormal data from normal data points, and identifies abnormal data from normal data by calculating the distance between each data sample and its k-nearest neighbors (kNN). In addition, in order to maximize the LVD value of each abnormal data, the invention automatically optimizes and selects the k value required by LVD based on the histogram density estimation method. Finally, the invention selects the data sample with the maximum LVD value for labeling, thereby realizing effective mining of abnormal data in industrial internet big data.

[0005] To solve the above technical problems, the present application provides the following technical solutions: An abnormal data mining method in industrial internet big data, comprising the following steps: Step 1. Calculate the mutual k-nearest neighbor set MkNN of each sample in the data set; Step 2. Calculate the variation score VS of each sample based on the MkNN set; Step 3. Calculate the maximum distance of each sample to its k-nearest neighbors as the isolation score IS; Step 4. Calculate the local variation degree LVD based on the variation score VS and the isolation score IS; Step 5. Select the sample with the maximum LVD value for class labeling, and set the LVD value of the sample and its k-nearest neighbors to be invalid; Step 6. Repeat step 5 until the iteration termination condition is met, and output the labeled abnormal data set.

[0006] Further, the step 1 of calculating the mutual k-nearest neighbor set MkNN of each sample in the data set comprises:

[0007] Wherein, MkNN value of a data sample , is an industrial internet big data set, is another data sample, k-nearest neighbors of data sample , k-nearest neighbors of data sample , .

[0008] Furthermore, the calculation formula of the change score VS is:

[0009] in, Represents a data sample of VS value, Represents a data sample The absolute value of the MkNN value, Represents data samples The MkNN value, k is the number of nearest neighbors, Indicates the number of elements in the collection.

[0010] Furthermore, the k value is automatically selected based on the optimal bandwidth partition in the histogram density estimation, and the steps are as follows: Calculation Dataset D The optimal bandwidth of each dimension for each data sample in ; Based on optimal bandwidth and data set D In the partition range of each dimension, determine the number of partitions for partitioning; The maximum number of samples in the partition is taken as the k value.

[0011] Furthermore, the dataset D Middle Dimensional optimal bandwidth , the calculation formula is as follows

[0012] in for D The number of data samples in , is the first in histogram density estimation The number of samples in a partition, In the dataset D i The bandwidth parameter used to partition the data samples.

[0013] Furthermore, the number of partitions for:

[0014] in, c is a hyperparameter, For the dataset D In the The distribution range in dimension.

[0015] Furthermore, the calculation formula of the isolation score IS is:

[0016] in, , is a matrix norm.

[0017] Further, the calculation formula of the local variation degree LVD is:

[0018] wherein, represents the value of the data sample . LVD .

[0019] Further, the iteration termination condition in the step 6 is that the number of category label inquiries reaches a preset threshold.

[0020] In another aspect, the present application provides an industrial internet big data anomaly data mining system, comprising: a mutual k-neighbor calculation module for calculating a mutual k-neighbor set MkNN of each sample in the data set; a variation score calculation module for calculating a variation score VS of each sample based on the MkNN set; an isolation score calculation module for calculating the maximum distance of each sample to its k-neighbor as an isolation score IS; a local variation degree calculation module for calculating a local variation degree LVD based on the variation score VS and the isolation score IS; a category label name module for selecting a sample with the largest LVD value for category labeling, and setting the LVD values of the sample and its k-neighbor as invalid; a result output module for repeating the above module execution steps until the iteration termination condition is met, and outputting the labeled anomaly data set.

[0021] Compared with the prior art, the present application has the following beneficial effects: By introducing the local variation degree, the present application can fully utilize the mutation of local data distribution to locate the abnormal area, and significantly reduce the number of inquiries required to find all the anomaly data in the data set, thereby improving the efficiency and accuracy of anomaly data mining in industrial internet big data. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0023] Figure 1 is a flowchart of an embodiment of the present application.

[0024] Figure 2 Schematic diagram of abnormal data distribution in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0026] Example 1 The present invention will be further described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, in this embodiment, the present invention provides an abnormal data mining device and method in industrial Internet big data. In this embodiment, given an industrial Internet big data D, the data distribution is as follows Figure 2 The specific values ​​are shown in Table 1. The algorithm outputs the selected query data sample set Q and the selected sample category label set L , as shown in Table 1.

[0028] Table 1 Experimental data

[0029] Step 1: For a given Industrial Internet big data set , a positive integer k (calculated according to the above preferred method, its value is 2), calculate the data set The MkNN value of each data sample in The MkNN value is used It is calculated as follows:

[0030] in, Represents a data sample k nearest neighbors, data samples .

[0031] Step 2: Given a positive integer In this case, the data set is calculated Change score VS Value, where a data sample of VS Value It is calculated as follows:

[0032] where k is automatically selected by the method, denotes the number of elements in the set.

[0033] Step 3: Given a positive integer k, calculate the isolation score value of the dataset , where the value of a data sample IS is denoted by , which is calculated as follows: IS

[0034] where , is the matrix norm.

[0035] Step 4: Given a positive integer k, calculate the local variation degree value of the dataset , where the value of a data sample LVD is denoted by , which is calculated as follows: LVD

[0036] Step 5: Find the data sample with the maximum local variation degree in the dataset LVD , and ask for its class label . Add the data sample and its class label to the selected query data sample set and the selected sample class label set Q , respectively. L

[0037] Step 6: For any data sample D in the dataset , if , then . That is, for any data sample , if it is the k-nearest neighbor of the sample with the maximum value in step 4 LVD , then set the value of the sample to . LVD

[0038] Step 7: If the number of class label inquiries is greater than 0, then jump to step 5 for execution, otherwise return the selected query data sample set Q and the selected sample class label set L . ​​​​​​​

[0039] Embodiment 2 The embodiment provides an abnormal data mining system in an industrial internet big data, comprising: a mutual k-neighbor computing module configured to compute a mutual k-neighbor set MkNN of each sample in a data set; a variation score computing module configured to compute a variation score VS of each sample based on the MkNN set; an isolation score computing module configured to compute a maximum distance of each sample to its k-neighbor as an isolation score IS; a local variation degree computing module configured to compute a local variation degree LVD based on the variation score VS and the isolation score IS; a class label naming module configured to select a sample with the maximum LVD value for class label naming, and set the LVD value of the sample and its k-neighbor as invalid; a result output module configured to repeat the above module execution steps until an iteration termination condition is met, and output a set of labeled abnormal data.

[0040] It should be understood that parts not described in detail in the specification are all prior art.

[0041] It should be understood that the above description of the preferred embodiments is more detailed and should not be considered as a limitation on the scope of patent protection of the present application. It is not necessary or possible to enumerate all the embodiments here. Those skilled in the art can make substitutions or modifications without departing from the scope of the claims, which are all within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for mining abnormal data in industrial Internet big data, characterized in that: The following steps are involved: Step 1. Calculate the mutual k-nearest neighbor set MkNN of each sample in the dataset; Step 2. Calculate the change score VS of each sample based on the MkNN set; Step 3. Calculate the maximum distance from each sample to its k nearest neighbors as the isolation score IS; Step 4. Calculate the local variation degree LVD based on the variation score VS and the isolation score IS; Step 5. Select the sample with the largest LVD value for category labeling, and set the LVD values ​​of this sample and its k nearest neighbors to invalid; Step 6. Repeat step 5 until the iteration termination condition is met and output the labeled abnormal data set.

2. The industrial Internet big data rapid clustering method according to claim 1 is characterized in that: The calculation of the mutual k-nearest neighbor set MkNN of each sample in the data set in step 1 includes: in, Represents a data sample The MkNN value of For the Industrial Internet Big Data Set, For another data sample, Represents data samples k nearest neighbors, Represents data samples k nearest neighbors, .

3. The industrial Internet big data rapid clustering method according to claim 2, characterized in that: The calculation formula of the change score VS is: in, Represents a data sample of VS value, Represents a data sample The absolute value of the MkNN value, Represents data samples The MkNN value, k is the number of nearest neighbors, Indicates the number of elements in the collection.

4. The industrial Internet big data rapid clustering method according to claim 3 is characterized in that: The k value is automatically selected based on the optimal bandwidth partition in the histogram density estimate, and the steps are as follows: Calculation Dataset D The optimal bandwidth of each dimension for each data sample in ; Based on optimal bandwidth and data set D In the partition range of each dimension, determine the number of partitions for partitioning; The maximum number of samples in the partition is taken as the k value.

5. The industrial Internet big data rapid clustering method according to claim 4 is characterized in that: Dataset D Middle Dimensional optimal bandwidth , the calculation formula is as follows in for D The number of data samples in , is the first in histogram density estimation The number of samples in a partition, In the dataset D i The bandwidth parameter used to partition the data samples.

6. The industrial Internet big data rapid clustering method according to claim 4, characterized in that: The number of partitions for: in, c is a hyperparameter, For the dataset D In the The distribution range in dimension.

7. The industrial Internet big data rapid clustering method according to claim 3 is characterized in that: The calculation formula of the isolation fraction IS is: in, , is the matrix norm.

8. The industrial Internet big data rapid clustering method according to claim 7, characterized in that: The calculation formula of the local variation degree LVD is: in, Represents data samples of LVD value.

9. The industrial Internet big data rapid clustering method according to claim 6, characterized in that: The iteration termination condition in step 6 is that the number of category label inquiries reaches a preset threshold.

10. An abnormal data mining system in industrial Internet big data, characterized by: include: The mutual nearest neighbor calculation module is used to calculate the mutual k-nearest neighbor set MkNN of each sample in the data set; A change score calculation module is used to calculate the change score VS of each sample based on the MkNN set; Isolation score calculation module. It is used to calculate the maximum distance from each sample to its k nearest neighbors as the isolation score IS; Local variation calculation module. It is used to calculate the local variation LVD based on the variation score VS and the isolation score IS; Category labeling module. It is used to select the sample with the largest LVD value for category labeling, and set the LVD values ​​of the sample and its k nearest neighbors to invalid; Result output module. It is used to repeat the above module execution steps until the iteration termination condition is met and output the anomaly data set; The abnormal data mining system in industrial Internet big data is used to execute the steps in the abnormal data mining method in industrial Internet big data described in any one of claims 1-9.

Images