Industrial big data outlier cleaning method based on double-layer KNN rule collaboration

Abstract

The invention discloses an industrial big data outlier cleaning method based on cooperation of multiple KNN rules, and belongs to the technical field of industrial big data preprocessing. The core of the method is that a double-layer detection framework is adopted to consider both efficiency and precision: firstly, the relative difference degree between samples is calculated based on variable coefficient weighted distance measurement, a suspicious sample set with abnormal density is quickly screened out, and the calculation scale is remarkably reduced; and secondly, for a suspicious sample set, innovatively fusing and transmitting three types of rules of K neighbor distance, reachable distance and mutual neighbor number, and constructing a mixed factor index to carry out accurate identification. The method effectively solves the problems that a traditional method is high in calculation complexity when coping with massive high-dimensional industrial data and unbalanced in local and global benefit group point detection capacity, has the advantages of being high in precision, high in efficiency and high in universality, and is suitable for improving the data quality of an industrial monitoring system.

Description

Technical Field

[0001] This invention relates to the field of industrial big data preprocessing and equipment status monitoring technology, and in particular to an outlier detection method based on a two-layer K-nearest neighbor algorithm. It is applicable to the massive high-dimensional but low-quality data collected by industrial monitoring platforms such as Supervisory Control and Data Acquisition (SCADA) systems, and effectively cleans outliers caused by complex and changeable industrial environments, hardware equipment failures, or changes in operating status. Background Technology

[0002] In the context of the Industrial Internet and intelligent manufacturing, systems such as SCADA continuously collect massive amounts of equipment operating parameters (such as temperature, pressure, and speed). This data forms the basis for equipment health prediction, fault diagnosis, and energy efficiency optimization. However, due to sensor drift, communication interference, and sudden changes in operating conditions, outliers are inevitably mixed in with the collected raw data. If these outliers are not effectively cleaned up, they will seriously interfere with the accuracy of subsequent data analysis models, leading to false alarms or missed alarms, thereby affecting operation and maintenance decisions.

[0003] Existing outlier detection methods mainly include:

[0004] 1. Methods based on statistical distribution, such as the 3σ criterion and quartile method. These methods are simple and efficient, but they require the data to follow a specific distribution, such as a normal distribution. They are less adaptable to complex multimodal and nonlinear industrial data and are prone to false positives and false negatives.

[0005] 2. Distance-based methods: such as algorithms based on K-Nearest Neighbor (KNN) distance, which simply assume that outliers are far from most samples. Traditional K-distance-based methods suffer from high computational complexity, achieving a computational complexity of O(n^2). 2 Furthermore, it only considers the one-way distance relationship between samples, which is insufficient for detecting local outliers located at the edge of the data distribution.

[0006] 3. Density-based methods: such as the Local Outlier Factor (LOF) algorithm, which identifies anomalies by comparing local densities, but this method is unstable in datasets with large density differences.

[0007] Therefore, existing technologies generally suffer from problems such as insufficient mining of the characteristics of complex industrial data, unbalanced global and local anomaly detection capabilities, and the need to improve computational efficiency. There is an urgent need for an outlier detection method that can comprehensively consider the multi-dimensional characteristics of data and balance detection accuracy and efficiency. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a high-precision, widely applicable method for detecting outliers in industrial data based on a two-layer K-nearest neighbor rule. This method can efficiently identify global and local outliers in massive amounts of industrial data, significantly improving data quality and providing a reliable data foundation for subsequent industrial equipment condition monitoring models. The specific technical solution is as follows:

[0009] A method for cleaning outliers in industrial big data based on two-layer KNN rule collaboration includes the following steps:

[0010] Step S1: Data Preprocessing and Weighted Distance Measurement

[0011] S1.1: For the original industrial dataset D m×n (m samples, n variables) are normalized to eliminate the influence of dimensions;

[0012] S1.2: Calculate the coefficient of variation for each variable, and use it as the weight ω of that variable. j ;

[0013] S1.3: Calculate the distance between samples using weighted Euclidean distance:

[0014]

[0015] Where, ω p Let be the weight of the p-th variable.

[0016] Step S2: First-layer detection – Screening of suspicious samples based on relative differences

[0017] S2.1: For each sample x i Calculate the average distance d(x) of its K nearest neighbors. i );

[0018] S2.2: Calculate any two samples x i and x j The relative difference between them:

[0019]

[0020] S2.3: Construct the relative dissimilarity matrix Ψ and calculate the first-order detection index for each sample.

[0021] S2.4: Set a primary screening threshold T1. Statistical tests, quantile methods, etc., can be used, combined with the characteristics of industrial data, to screen out ρ. i The samples <T1 constitute the density anomaly sample set D dou This sample set includes samples located in both low-density and high-density regions.

[0022] Step S3: Second-layer detection – Outlier identification based on multiple K-nearest neighbor rules

[0023] For density anomaly sample set D dou For each sample x in the dataset, perform the following operations:

[0024] S3.1: Calculate the ratio w1 = d1(x) / d2(x) of the transit distance d1(x) (the distance from the Kth nearest neighbor of the sample to its Kth nearest neighbor) to the K distance d2(x) (the distance from the sample x to its Kth nearest neighbor);

[0025] S3.2: Calculate the reachability density of sample x Where d3(x, x) i ) represents the reachability distance of x to other samples;

[0026] S3.3: The number of nearest neighbors N(x) of the statistical sample x is used as a weighting factor q = N(x) + f (f is a small constant to avoid 0 in the denominator);

[0027] S3.4: Calculate the secondary detection index for sample x:

[0028] J(x) = q × w1 × w2

[0029] S3.5: Set a secondary screening threshold T2. If J(x) < T2, then sample x is determined to be an outlier.

[0030] Step S4: Output the cleaned dataset

[0031] All outliers identified in step S3 are removed from the original dataset D, and the cleaned dataset is output for subsequent industrial equipment condition monitoring modeling and analysis.

[0032] Preferably, the K value in step S2 can be set empirically or through cross-validation based on the size of the dataset;

[0033] Preferably, the thresholds T1 and T2 in steps S2.4 and S3.5 are determined by combining actual data, and thus have an adaptive nature.

[0034] The advantages and beneficial effects of this invention are as follows:

[0035] Compared with the prior art, the present invention has the following significant advantages:

[0036] (1) This invention innovatively combines multiple KNN-related concepts such as relative difference, transmission distance, reachability density and neighbor relationship, and comprehensively evaluates the abnormality of samples from multiple dimensions such as local distribution characteristics, global distribution characteristics and neighbor relationship. It can effectively detect global outliers and local outliers, and the detection accuracy and AUC value are better than traditional methods.

[0037] (2) The present invention obtains a suspicious sample set through the first screening, which greatly narrows the scope of subsequent precise calculations and reduces the overall computational complexity, making it particularly suitable for large-scale industrial data scenarios.

[0038] (3) The present invention uses the coefficient of variation weighted distance to adapt to the differences in the importance of different variables; the threshold is adaptively set based on the data quantile, which does not require too much prior knowledge and has good robustness to different data distributions.

[0039] (4) The core method of this invention does not rely on specific equipment or processes and can be widely used in the quality cleaning and preprocessing of SCADA data and sensor network data in various industrial fields such as wind power, photovoltaics, chemical industry, and manufacturing. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the process of the outlier cleaning method for industrial big data based on two-layer KNN rule collaboration of the present invention;

[0041] Figure 2 This is a schematic diagram of the K-distance and the K-distance of the present invention, where x_k is the Kth nearest neighbor sample of sample x, d_k is the K-distance of sample x, and d_c is the K-distance of sample x.

[0042] Figure 3 This is a schematic diagram of the reachability distance of samples in this invention. When K = 3, the reachability distance between sample x and x1 is d_r1, and the reachability distance between sample x and samples x2 and x3 is the K distance of sample x. Detailed Implementation

[0043] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0044] like Figure 1 As shown, the outlier cleaning method for industrial big data based on two-layer KNN rule collaboration of the present invention includes data preprocessing and weighted distance measurement, first-layer filtering and second-layer filtering.

[0045] During implementation, the data preprocessing steps mainly include data normalization and sample-weighted distance measurement based on the coefficient of variation. First, for raw data collected on-site with missing data (such as values ​​recorded as "NAN"), unusable data is removed to ensure that the remaining data are all valid values. Second, the filtered valid data is normalized according to the corresponding variables to maintain uniformity in the units of measurement. Finally, the coefficient of variation for each variable is calculated to measure its relative volatility, and the coefficient of variation of each variable is used as the weight for improving the distance measurement, thereby obtaining more accurate distance values ​​between samples.

[0046] For the first layer of detection, the specific implementation of suspicious sample screening based on relative dissimilarity is as follows: For the preprocessed and weighted distance-measured industrial dataset, the hyperparameter K, i.e., the number of nearest neighbors of a sample, is determined. This step can be reasonably set based on the dataset size through industry experience or cross-validation to balance detection coverage and computational efficiency. Then, for each sample in the dataset, all other samples are traversed, and the calculated weighted Euclidean distance is used to select the K nearest neighbors. The distance values ​​of these K samples are summed and divided by K to obtain the average K nearest neighbor distance for each sample. Next, based on any two samples x... i and x j The average distance between their respective K-nearest neighbors is used to calculate their relative dissimilarity using the formula for calculating relative dissimilarity, as follows:

[0047]

[0048] This formula reflects the relative deviation trend of samples in the data distribution by comparing the differences in the nearest neighbor distances between two samples. Then, using all samples as rows and columns, the calculated relative difference between any two samples is filled into the corresponding positions to construct a relative difference matrix between samples. Then, the relative difference values ​​of all rows in the matrix for each sample are summed, and the result is the primary detection index of that sample. This index can comprehensively reflect the overall difference between a single sample and all other samples. Finally, combined with the actual distribution characteristics of industrial data and business needs, a primary detection threshold is set using statistical testing methods or quantile methods (such as selecting the 90th or 95th quantile). Samples with primary detection indices greater than this threshold are screened out to form a density anomaly sample set. This sample set covers samples located in low-density areas (globally possible outlier candidates) and locally deviating in high-density areas (locally possible outlier candidates) in the data, narrowing the scope for subsequent second-layer accurate detection.

[0049] For the second-layer detection, a specific implementation method for accurate outlier identification based on the collaboration of multiple KNN rules is used. For the density anomaly sample set obtained from the first-layer screening, multi-dimensional index calculations and comprehensive judgments are performed sequentially for each sample: First, the propagation distance d_c of the sample is calculated, such as... Figure 2 As shown, the distance from the Kth nearest neighbor of a sample to its own Kth nearest neighbor is d_k, which is the distance from K to d_k. Figure 2 As shown, the ratio between the distance of a sample to its Kth nearest neighbor sample and the distance of the sample to its nearest neighbor sample reflects the deviation characteristics of the sample in the nearest neighbor transit relationship; next, the reachability density of the sample is calculated, as shown. Figure 3 As shown, this density is derived from the reachable distance d_r from the sample to other samples, and is used to characterize the density distribution characteristics of the local region where the sample is located. Simultaneously, the number of mutual neighbors between the sample and other samples is counted, and a small constant is introduced to avoid zero in the denominator during subsequent calculations. The number of mutual neighbors is transformed into a mutual neighbor factor, which serves as a weighted index to measure the closeness of the sample's neighbor relationship. Subsequently, combining the mutual neighbor factor, the ratio of transmission distance to K-distance, and the reachable density obtained above, the secondary detection index of the sample is calculated according to the following formula:

[0050] J(x) = q × w1 × w2

[0051] This indicator comprehensively quantifies the degree of anomaly of a sample from three dimensions: nearest neighbor relationship, local density, and transmission characteristics. Finally, combined with the actual business needs and data distribution characteristics of industrial data, an adaptive method such as quantile method is used to set a secondary screening threshold. If the mixing factor of a sample is greater than the threshold, the sample is determined to be an outlier, thus completing the accurate identification of abnormal data in the suspicious sample set.

Claims

1. A method for industrial big data outlier detection based on double-layer KNN rule cooperation, characterized in that, The method comprises the following steps: Step S1: exporting raw data from an industrial monitoring system and preprocessing, and calculating weighted Euclidean distance based on coefficient of variation; Step S2: performing first screening based on relative difference of samples, and screening out a suspicious sample set; Step S3: performing second detection on the suspicious sample set, calculating a hybrid factor by fusing transmission K-neighbor distance, reachable distance and mutual-neighbor number, and accurately identifying outliers; Step S4: outputting cleaned data after removing outliers. 2.The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 1, characterized in that, The step S1 specifically comprises the following steps: S1.1: Normalization is performed on the original industrial dataset D m×n , where m is the number of samples and n is the number of variables. S1.2: Calculate the coefficient of variation for each variable as the coefficient of variation weight ω for that variable j ; S1.3: calculating distance between samples by using weighted Euclidean distance: where ω p is the weight of the pth variable. 3.The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 2, characterized in that, In the step S1.2, the coefficient of variation weight where c j is the coefficient of variation of the jth variable. 4.The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 1, characterized in that, The step S2 specifically comprises the following steps: S2.1 : For each sample x i , compute the average distance of its K nearest neighbors S2.2: Calculate the relative difference between any two samples x i and x j : S2.3: Construct the relative difference matrix Ψ, and calculate the one-time detection index of each sample S2.4: Set a one-time screening threshold T1, and screen out p i The sample set D of the density anomaly samples is composed of samples located in the low-density area and the high-density area. dou , and the sample set D of the density anomaly samples is composed of samples located in the low-density area and the high-density area. 5.The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 1, characterized in that, The K value in the step S3 is set by experience or cross-validation according to the size of the data set. 6.The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 4, characterized in that, The step S3 performs the following operations on each sample x in the set of density anomaly samples D dou ​ S3.1: calculating the ratio w1=d1(x) / d2(x) of the transmission distance d1(x) of the sample x and the K-distance d2(x), wherein d1(x) is the distance from the Kth nearest neighbor sample of the sample to its Kth nearest neighbor, and d2(x) is the distance from the sample x to its Kth nearest neighbor sample; S3.2: Compute the reachable density of sample x where d3(x, x i ) is the reachable distance of x to other samples; S3.3: counting the mutual-neighbor number N(x) of the sample x as a weight factor q=N(x)+f, wherein f is a small constant; S3.4: calculating the second detection index of the sample x: J(x)=q×w1×w2 S3.5: setting a second screening threshold T2, if J(x)<T2, the sample x is determined as an outlier.

7. The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 6, characterized in that, In the step S3, the calculation formula of the hybrid factor S(x) is: S(x) = q x [d1(x) / d2(x)] x [1 / ∑d3(x,x i )], where q is a mutual proximity factor. 8.The industrial big data outlier detection method based on double-layer KNN rule cooperation according to claim 7, characterized in that, In the step S3, outliers are distinguished by comparing the hybrid factor S(x) with the quantile threshold T2.

Images