SIP-LOF-based observation data anomaly identification method

Abstract

An SIP-LOF-based observation data anomaly identification method comprises the following steps: firstly, extracting all data points from observation data to be detected through a PLR-SIP method, then calculating a point with the largest y-direction distance between two segmentation points by taking two data points of a starting point and an end point as the segmentation points, and calculating a point with the largest y-direction distance between the two segmentation points; sequentially calculating a point with the largest y-direction distance between two adjacent segmentation points to obtain a plurality of segmentation points, and forming a subsequence between the two adjacent segmentation points; extracting a plurality of feature values of the subsequences, constructing a multi-dimensional space according to all the feature values, then mapping all the feature values in the multi-dimensional space, and calculating the distance between each feature value; after the distance values are arranged according to the size, finding the kth distance and the first k neighbors of each sample, then calculating the reachable distance and the local reachable density from each sample to the k neighbors, and then carrying out local outlier factor LOF calculation to obtain an LOF value; according to the method, the abnormal data can be quickly and automatically positioned in mass data.

Description

Technical Field

[0001] The invention relates to an improvement of a seismic abnormal data recognition technology, belongs to the field of data processing, and in particular to an observation data abnormality recognition method based on SIP-LOF. Background Art

[0002] The geophysical observation network consists of three major disciplines: topographic deformation, geomagnetic and underground fluid. Among them, topographic deformation measurement is one of the indispensable and important means in earthquake monitoring. The crust deformation instrument conducts scientific observations from multiple angles such as ground tilt, stress, strain, gravity, spatial large-scale topographic deformation measurement, and fault deformation measurement. Common topographic deformation instruments include but are not limited to borehole strain gauges, water pipe inclinometers, vertical pendulum inclinometers, telescopes, and body strain gauges.

[0003] With the dense deployment of digital earthquake monitoring networks, the use of topographic deformation instruments has also increased, resulting in an exponential growth in the total amount of observation data. This provides more substantial and rich data samples for earthquake prediction research, but also greatly increases the difficulty of determining abnormal data. However, in the process of topographic deformation instrument observation, it is inevitably affected by various factors, such as solid tides, natural environment, human interference, observation systems or earthquake precursors. These factors will cause data samples to show abnormal dynamics based on the original periodic changes, such as data glitches, large jumps, step jumps, missing data, etc. In this case, traditional processing modes have difficulty in quickly and automatically locating anomalies in massive observation data, which will affect the quality of observation data.

[0004] A Chinese patent application with application number CN202310937865.X and application date July 28, 2023 discloses a method and system for constructing a geomagnetic anomaly data set, the method comprising: extracting and removing the linear trend component in the preprocessed minute data of the geomagnetic vertical component of each station, to obtain a residual geomagnetic sequence containing periodic fluctuations of the geomagnetic signal and local abnormal fluctuations; constructing a reference signal of the geomagnetic data of each station based on the residual geomagnetic sequence; calculating the window-weighted correlation of the daily geomagnetic data of each station with the reference signal; extracting the geomagnetic anomaly waveform based on the size of the window-weighted correlation, and forming a geomagnetic anomaly data set, which can improve the accuracy and sensitivity of geomagnetic anomaly waveform detection so as to timely discover possible geomagnetic anomaly signals, but the above scheme does not solve the problem of not being able to quickly and automatically locate abnormal data in massive data.

[0005] The information disclosed in this background technology section is only intended to increase the understanding of the overall background of this patent application, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention

[0006] The purpose of the present invention is to overcome the problem in the prior art that abnormal data cannot be quickly and automatically located in massive data, and to provide an observation data anomaly identification method based on SIP-LOF that can quickly and automatically locate abnormal data in massive data.

[0007] To achieve the above objectives, the technical solution of the present invention is: a method for identifying anomalies in observed data based on SIP-LOF, the method for identifying anomalies in observed data based on SIP-LOF comprising the following steps:

[0008] Step 1: First, extract all data points from the observation data to be detected by the PLR-SIP method, then take all the data points as a time series, and then use the starting point and the end point as the first-generation segmentation points, connect the two first-generation segmentation points as the first-generation baseline, calculate the y-distance between the remaining data points and the first-generation baseline, and find the point with the largest y-distance, which is the important point. If the distance is less than the threshold, the segmentation point calculation ends. If it is greater than the threshold, the important point is the second-generation segmentation point, and then connect the second-generation segmentation point with the starting point and the end point to form two second-generation baselines, and then calculate the new y-distance between the remaining data points and the two second-generation baselines, and find the point with the largest y-distance and compare it with the threshold. If it is greater than the threshold, the third-generation segmentation point is obtained, and then the first-generation segmentation point, the second-generation segmentation point, and the third-generation segmentation point are connected to obtain three third-generation baselines, and then the y-distance between the remaining data points and the third-generation baselines is calculated according to the above method. This cycle continues to form segmentation points until the largest y-distance is less than the threshold, and the segmentation point calculation ends. At this time, a subsequence is formed between two adjacent segmentation points;

[0009] Step 2: extract multiple eigenvalues ​​of the subsequence, construct a multidimensional space based on all the eigenvalues, then map all the eigenvalues ​​in the multidimensional space, calculate the distance between each eigenvalue, and obtain the corresponding distance value;

[0010] Step 3: After arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding distance value and density value. Then, calculate the local outlier factor LOF based on all the distance values ​​and density values ​​to obtain all the LOF values.

[0011] Step 4: If the LOF value is close to 1, the data point is a normal point. If the LOF value is far from 1, the data point is an outlier.

[0012] In the step 1, if the distance is less than the threshold, the segmentation point calculation ends specifically as follows: when the distance is less than the threshold, the point cannot become a new segmentation point, the point is merged into the line segment formed by the previous segmentation point, and the segmentation ends.

[0013] In the step 2, the multiple eigenvalues ​​include length, extreme value difference, standard deviation and mean, and these four eigenvalues ​​are the four elements of the feature space;

[0014] The length is the subsequence variation trend, the extreme value difference is the difference between the maximum value and the minimum value, the standard deviation is the degree of dispersion of the sequence data, and the mean is the average trend.

[0015] The threshold is 0.001-0.005 times the standard deviation.

[0016] In the step 2, multiple eigenvalues ​​of the subsequence are extracted, a multidimensional space is constructed according to all the eigenvalues, and then all the eigenvalues ​​are mapped in the multidimensional space, and the distance between each eigenvalue is calculated to obtain the corresponding distance value:

[0017] The time series is mapped to the feature space. Each subsequence corresponds to a point in the feature space. The four elements consisting of the eigenvalues ​​are used to represent the position of each subsequence in the feature space. On this basis, the distance between each pattern is calculated to obtain the corresponding distance value.

[0018] Since the ranges of the four characteristic quantities vary greatly, different magnitudes will affect the abnormal detection results of the observed data. Therefore, the four sets of characteristic values ​​are normalized separately, specifically:

[0019] Let x1=(x 11 , x 12 , x 13 , ..., x 1n ) is one of the subsequences, x max With x min Respectively represent the maximum and minimum values ​​of each eigenvalue;

[0020]

[0021] In the step 3, after arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding density value, and then calculate the local outlier factor LOF according to all the density values ​​to obtain all the LOF values. Specifically, use the density-based local anomaly factor algorithm to perform abnormal pattern detection, and the specific steps are as follows:

[0022] A. Calculate the Euclidean distance of the eigenvalues ​​between any two subsequences;

[0023] B. After arranging the Euclidean distances, find the kth distance and the first k nearest neighbors of each sample;

[0024] C. Calculate the reachable distance and local reachable density of each sample to its k nearest neighbors respectively.

[0025] The reachable distance k is calculated as follows:

[0026] reach-dist(p, q)=max(k-distance(q), d(p, q));

[0027] Where point p is a data point in the sample, point q is one of the k nearest neighbors of point p, d(p,q) represents the actual distance between point p and point q, and k-distance(q) is the distance from point q to its kth nearest neighbor.

[0028] The local reachable density is calculated as follows:

[0029]

[0030] In step 3, the local outlier factor LOF is then calculated based on all distance values ​​and density values ​​to obtain all LOF values, specifically:

[0031]

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. In a method for identifying abnormal observation data based on SIP-LOF of the present invention, all data points are first extracted from the observation data to be detected by the PLR-SIP method, and then all the data points are used as a time series, and then the first and last two data points are used as segmentation points, and then the point with the largest y-direction distance between the two segmentation points is calculated. If the distance is less than a threshold value, the two ends are merged, and if it is greater than the threshold value, the important point is the segmentation point, and then the point with the largest y-direction distance between two adjacent segmentation points is calculated in sequence to obtain multiple segmentation points until all data are segmented. At this time, a subsequence is formed between two adjacent segmentation points. When applied, the time series analysis technology in data mining is introduced, and the observation data of typical events such as equipment failure, site environment, and earthquake precursors are regarded as time series, and the sequence is further segmented and the subsequence feature quantity is extracted. Finally, the subsequence LOF value is calculated, which can accurately and effectively identify typical interference events and earthquake precursor information, thereby quickly and automatically locating abnormal data in massive data, improving the availability of instrument data, and improving the fault discrimination efficiency of operation and maintenance personnel. Therefore, the present invention can quickly and automatically locate abnormal data in massive data.

[0034] 2. In the observation data anomaly identification method based on SIP-LOF of the present invention, the SIP-LOF algorithm pre-processes the time series data through the statistical interpolation method to remove the long-term trend and seasonal fluctuations, so as to make the data more stable, which helps the algorithm to more accurately distinguish the real abnormal signal from the periodic changes caused by natural changes, reduce false positives and false negatives, and at the same time, the data after detrending can better reflect the abnormal changes in the short term, and the SIP-LOF algorithm can capture these rapid changes, and can timely discover potential abnormal events even when the overall trend of the data is stable. Therefore, the present invention reduces false positives and false negatives, and can also discover abnormal events under stable conditions.

[0035] 3. In the observation data anomaly identification method based on SIP-LOF of the present invention, the SIP-LOF method does not rely on the assumption of data distribution, can handle abnormal points of various shapes and sizes, is applicable to a variety of complex data structures and monitoring scenarios, and improves the universality and adaptability of the method. Although the LOF method has a high computational complexity when processing large-scale data sets, the preprocessing step of SIP can simplify the subsequent anomaly detection process to a certain extent, thereby improving efficiency, which is particularly important in real-time monitoring systems. By combining statistical interpolation and local anomaly factor algorithms, the ability to distinguish noise and abnormal values ​​is enhanced. Therefore, the present invention has strong adaptability and high detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a data processing schematic diagram of the present invention.

[0037] Figure 2 It is a schematic diagram of typical event detection results of Mudanjiang seismic station in the present invention.

[0038] Figure 3 It is a schematic diagram of the typical event detection results of Jiaxiang seismic station in the present invention.

[0039] Figure 4 It is a schematic diagram of the typical event detection results of Zhongshan Station in the present invention.

[0040] Figure 5 It is a schematic diagram of the typical event detection results of Kuancheng seismic station in the present invention.

[0041] Figure 6 It is a schematic diagram of the typical event detection results of the Nanshan seismic station in the present invention.

[0042] Figure 7 It is a schematic diagram of the typical event detection results of Yixian seismic station in the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] See also Figures 1 to 7 , a method for identifying anomalies in observed data based on SIP-LOF, the method for identifying anomalies in observed data based on SIP-LOF comprising the following steps:

[0045] Step 1: First, extract all data points from the observation data to be detected by the PLR-SIP method, then take all the data points as a time series, and then use the starting point and the end point as the first-generation segmentation points, connect the two first-generation segmentation points as the first-generation baseline, calculate the y-distance between the remaining data points and the first-generation baseline, and find the point with the largest y-distance, which is the important point. If the distance is less than the threshold, the segmentation point calculation ends. If it is greater than the threshold, the important point is the second-generation segmentation point, and then connect the second-generation segmentation point with the starting point and the end point to form two second-generation baselines, and then calculate the new y-distance between the remaining data points and the two second-generation baselines, and find the point with the largest y-distance and compare it with the threshold. If it is greater than the threshold, the third-generation segmentation point is obtained, and then the first-generation segmentation point, the second-generation segmentation point, and the third-generation segmentation point are connected to obtain three third-generation baselines, and then the y-distance between the remaining data points and the third-generation baselines is calculated according to the above method. This cycle continues to form segmentation points until the largest y-distance is less than the threshold, and the segmentation point calculation ends. At this time, a subsequence is formed between two adjacent segmentation points;

[0046] Step 2: extract multiple eigenvalues ​​of the subsequence, construct a multidimensional space based on all the eigenvalues, then map all the eigenvalues ​​in the multidimensional space, calculate the distance between each eigenvalue, and obtain the corresponding distance value;

[0047] Step 3: After arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding distance value and density value. Then, calculate the local outlier factor LOF based on all the distance values ​​and density values ​​to obtain all the LOF values.

[0048] Step 4: If the LOF value is close to 1, the data point is a normal point. If the LOF value is far from 1, the data point is an outlier.

[0049] In the step 1, if it is less than the threshold, the two ends are merged. Specifically, when it is less than the threshold, the point cannot become a new segmentation point, and the point is merged into the line segment formed by the previous segmentation point.

[0050] In the step 2, the multiple eigenvalues ​​include length, extreme value difference, standard deviation and mean, and these four eigenvalues ​​are the four elements of the feature space;

[0051] The length is the subsequence variation trend, the extreme value difference is the difference between the maximum value and the minimum value, the standard deviation is the degree of dispersion of the sequence data, and the mean is the average trend.

[0052] The threshold is 0.001-0.005 times the standard deviation.

[0053] In the step 2, multiple eigenvalues ​​of the subsequence are extracted, a multidimensional space is constructed according to all the eigenvalues, and then all the eigenvalues ​​are mapped in the multidimensional space, and the distance between each eigenvalue is calculated to obtain the corresponding distance value:

[0054] The time series is mapped to the feature space. Each subsequence corresponds to a point in the feature space. The four elements consisting of the eigenvalues ​​are used to represent the position of each subsequence in the feature space. On this basis, the distance between each pattern is calculated to obtain the corresponding distance value.

[0055] Since the ranges of the four characteristic quantities vary greatly, different magnitudes will affect the abnormal detection results of the observed data. Therefore, the four sets of characteristic values ​​are normalized separately, specifically:

[0056] Let x1=(x 11 , x 12 , x 13 , ..., x 1n ) is one of the subsequences, x max With x min Respectively represent the maximum and minimum values ​​of each eigenvalue;

[0057]

[0058] In the step 3, after arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding density value, and then calculate the local outlier factor LOF according to all the density values ​​to obtain all the LOF values. Specifically, use the density-based local anomaly factor algorithm to perform abnormal pattern detection, and the specific steps are as follows:

[0059] A. Calculate the Euclidean distance of the eigenvalues ​​between any two subsequences;

[0060] B. After arranging the Euclidean distances, find the kth distance and the first k nearest neighbors of each sample;

[0061] C. Calculate the reachable distance and local reachable density of each sample to its k nearest neighbors respectively.

[0062] The reachable distance k is calculated as follows:

[0063] reach-dist(p, q)=max(k-distance(q), d(p, q));

[0064] Where point p is a data point in the sample, point q is one of the k nearest neighbors of point p, d(p,q) represents the actual distance between point p and point q, and k-distance(q) is the distance from point q to its kth nearest neighbor.

[0065] The local reachable density is calculated as follows:

[0066]

[0067] In step 3, the local outlier factor LOF is then calculated based on all distance values ​​and density values ​​to obtain all LOF values, specifically:

[0068]

[0069] The supplementary description of the present invention is as follows:

[0070] The time series observation data of the geophysical network used are high-dimensional and have a wide range of data volumes, and the curve shapes are different. It is often inefficient to directly apply data mining technology to the original data point by point. Therefore, the pattern representation is used to replace the original time series and reduce the dimension of the time series, which can compress the data while maintaining the basic shape of the time series.

[0071] Embodiment 1:

[0072] A method for identifying anomalies in observed data based on SIP-LOF, the method comprising the following steps:

[0073] Step 1: First, extract all the data points of the observation data to be detected by the PLR-SIP method, then treat all the data points as a time series, and then use the starting point and the end point as the first-generation segmentation points, connect the two first-generation segmentation points as the first-generation baseline, calculate the y-distance between the remaining data points and the first-generation baseline, and find the point with the largest y-distance, which is the important point. If the distance is less than the threshold, the segmentation point calculation ends. If it is greater than the threshold, the important point is the second-generation segmentation point. Then connect the second-generation segmentation point with the starting point and the end point respectively to form two second-generation baselines, and then calculate the remaining points respectively. The new y-distance between the remaining data points and the two second-generation baselines is calculated, and the point with the largest y-distance is found and compared with the threshold. If it is greater than the threshold, the third-generation segmentation point is obtained, and then the first-generation segmentation point, the second-generation segmentation point, and the third-generation segmentation point are connected to obtain three third-generation baselines. Then, the y-distance between the remaining data points and the three-generation baselines is calculated according to the above method. This cycle continues to form segmentation points until the maximum y-distance is less than the threshold, and the segmentation point calculation is completed. The final compressed segmentation time series can accurately reflect the overall trend and characteristics of the original data; at this time, a subsequence is formed between two adjacent segmentation points;

[0074] Step 2: extract multiple eigenvalues ​​of the subsequence, construct a multidimensional space based on all the eigenvalues, then map all the eigenvalues ​​in the multidimensional space, calculate the distance between each eigenvalue, and obtain the corresponding distance value;

[0075] Step 3: After arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding distance value and density value. Then, calculate the local outlier factor LOF based on all the distance values ​​and density values ​​to obtain all the LOF values.

[0076] Step 4: If the LOF value is close to 1, the data point is a normal point; if the LOF value is far from 1, the data point is an outlier.

[0077] Embodiment 2:

[0078] Embodiment 2 is substantially the same as Embodiment 1, except that:

[0079] In the step 2, multiple eigenvalues ​​of the subsequence are extracted, a multidimensional space is constructed according to all the eigenvalues, and then all the eigenvalues ​​are mapped in the multidimensional space, and the distance between each eigenvalue is calculated to obtain the corresponding distance value. Specifically, the time series is mapped to the feature space, each subsequence corresponds to a point in the feature space, and the four elements composed of the eigenvalues ​​are used to represent the position of each subsequence in the feature space. On this basis, the distance between each mode is calculated to obtain the corresponding distance value; because the value ranges of the four feature quantities are very different, different magnitudes will cause the abnormal detection results of the observed data to be affected, so the four groups of eigenvalues ​​are normalized respectively, specifically as follows:

[0080] Let x1=(x 11 , x 12 , x 13 , ..., x 1n ) is one of the subsequences, x max With x min Respectively represent the maximum and minimum values ​​of each eigenvalue;

[0081]

[0082] Embodiment 3:

[0083] Embodiment 3 is substantially the same as Embodiment 1, except that:

[0084] In the step 3, after arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding density value, and then calculate the local outlier factor LOF according to all the density values ​​to obtain all the LOF values. Specifically, use the density-based local anomaly factor algorithm to perform abnormal pattern detection, and the specific steps are as follows:

[0085] A. Calculate the Euclidean distance of the eigenvalues ​​between any two subsequences;

[0086] B. After arranging the Euclidean distances, find the kth distance and the first k nearest neighbors of each sample;

[0087] C. Calculate the reachable distance and local reachable density of each sample to its k nearest neighbors respectively.

[0088] The reachable distance k is calculated as follows:

[0089] reach-dist(p, q)=max(k-distance(q), d(p, q));

[0090] Where point p is a data point in the sample, point q is one of the k nearest neighbors of point p, d(p,q) represents the actual distance between point p and point q, and k-distance(q) is the distance from point q to its kth nearest neighbor.

[0091] The local reachable density is calculated as follows:

[0092]

[0093] In step 3, the local outlier factor LOF is then calculated based on all distance values ​​and density values ​​to obtain all LOF values, specifically:

[0094]

[0095] Embodiment 4:

[0096] Embodiment 4 is substantially the same as Embodiment 1, except that:

[0097] The original observation data of deformation instruments at multiple seismic stations were selected, and six typical events with obvious curve characteristics were used as templates to explain and analyze the abnormal detection research content of typical events such as site environment, equipment failure, and earthquake. The data used were all from the precursor observation database of the Discipline Center of Hubei Provincial Seismological Bureau; the time sampling rate of the observation data was minute sampling;

[0098] Data source and measurement information

[0099]

[0100] Directly detecting anomalies on raw data will greatly reduce the accuracy of the algorithm. First, the data must be preprocessed and the moving average method must be used to convert the non-stationary sequence into a stationary sequence. The results show that the trend and cycle effects of most precursor observation data can be eliminated by the moving average method.

[0101] First, select an appropriate window length, calculate the moving average trend, and continuously slide the window forward and take the average, such as Figure 1 As shown in the figure, the black dotted line is the moving average trend of the curve. The moving average trend is subtracted from the original data, and the threshold is set to twice the standard deviation of the data after trend removal. If the detrended data is greater than twice the standard deviation, it is considered abnormal. At the same time, interpolation is used to replace the abnormal value, so as to smooth the short-term fluctuation data and retain the long-term trend, forming preprocessed data, which helps to improve the quality and reliability of precursor data and better serve the subsequent anomaly detection;

[0102] Apply the SIP-LOP algorithm to the preprocessed data and output the LOF value and detection curve of the subsequence. Figure 2-Figure 7The detection results of various typical events are given. The upper part of the figure is the observation curve after preprocessing, and the lower part is the abnormal detection result curve. The specific values ​​of the LOF detection value reflect the abnormal degree of each part of the original observation curve;

[0103] Figure 2 and Figure 4 There are steps at the red marks, and there are obvious data mutations. The abnormal detection result chart shows that the LOF value has increased significantly. Figure 2 The step phenomenon has only a small vertical upward trend, which proves that the method proposed in this paper can also easily detect small sudden jump anomalies. Figure 3 It was confirmed that the trend turning point of the curve was caused by unstable operation of the pendulum, and its corresponding LOF curve also changed significantly.

[0104] Figure 5 and Figure 6 The situation is similar. V-shaped and irregular distortions appeared on the originally smooth and stable periodic solid tide curve, which were judged to be caused by air pressure interference and interference from surrounding infrastructure construction. The algorithm in this paper also successfully discovered these two events. Figure 7 The water pipe instrument at Yixian Station recorded relatively clear co-seismic changes, which showed an oscillatory pattern and returned to calm after the earthquake. During the oscillation period, its LOF detection value also continued to increase.

[0105] From the above six test results, it can be seen that when the observation curve is in a stable and regularly changing solid tide form, the corresponding LOF detection value is always around 1. When the observation curve is distorted by typical events such as steps, jumps, and trend reversals, the corresponding LOF detection value will increase significantly to between 2 and 10. In order to facilitate the abnormal judgment of actual observation work, this plan selects the deformation instrument observation data of Yixian Seismic Station in 2022 to further quantify the abnormal judgment, that is, to set the LOF detection value limit. When the LOF detection value exceeds the limit, it is determined that an abnormality is detected.

[0106] The selection of LOF limit is the key to determine whether the observed data is abnormal. If the LOF limit is too small, normal fluctuations that do not belong to abnormal events will be misjudged as abnormal. If the LOF limit is too large, it will miss local micro-distortions caused by microseismic events, site environment or natural interference, etc. Therefore, based on comprehensive consideration of recall and precision, this paper preliminarily selects the lower limit of the range 2-10 mentioned above as the limit, and conducts statistical experiments around the value 2. The statistical results can be seen from the table below. The average accuracy of the algorithm for the four measurement items reaches 90.1%. When the LOF limit is 2.5, the average anomaly judgment accuracy is the highest;

[0107] Anomaly detection statistics

[0108]

[0109] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A method for identifying abnormal observation data based on SIP-LOF, characterized in that: The observation data anomaly identification method based on SIP-LOF comprises the following steps: Step 1: First, extract all data points from the observation data to be detected by the PLR-SIP method, then take all the data points as a time series, and then use the starting point and the end point as the first-generation segmentation points, connect the two first-generation segmentation points as the first-generation baseline, calculate the y-distance between the remaining data points and the first-generation baseline, and find the point with the largest y-distance, which is the important point. If the distance is less than the threshold, the segmentation point calculation ends. If it is greater than the threshold, the important point is the second-generation segmentation point, and then connect the second-generation segmentation point with the starting point and the end point to form two second-generation baselines, and then calculate the new y-distance between the remaining data points and the two second-generation baselines, and find the point with the largest y-distance and compare it with the threshold. If it is greater than the threshold, the third-generation segmentation point is obtained, and then the first-generation segmentation point, the second-generation segmentation point, and the third-generation segmentation point are connected to obtain three third-generation baselines, and then the y-distance between the remaining data points and the third-generation baselines is calculated according to the above method. This cycle continues to form segmentation points until the largest y-distance is less than the threshold and the segmentation point calculation ends; at this time, a subsequence is formed between two adjacent segmentation points; Step 2: extract multiple eigenvalues ​​of the subsequence, construct a multidimensional space based on all the eigenvalues, then map all the eigenvalues ​​in the multidimensional space, calculate the distance between each eigenvalue, and obtain the corresponding distance value; Step 3: After arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding distance value and density value. Then, calculate the local outlier factor LOF based on all the distance values ​​and density values ​​to obtain all the LOF values. Step 4: If the LOF value is close to 1, the data point is a normal point. If the LOF value is far from 1, the data point is an outlier.

2. The method for identifying anomaly in observed data based on SIP-LOF according to claim 1, characterized in that: In the step 1, if the distance is less than the threshold, the segmentation point calculation ends specifically as follows: when the distance is less than the threshold, the point cannot become a new segmentation point, the point is merged into the line segment formed by the previous segmentation point, and the segmentation ends.

3. The method for identifying abnormal observation data based on SIP-LOF according to claim 2, characterized in that: The threshold is 0.001-0.005 times the standard deviation.

4. The method for identifying anomaly in observed data based on SIP-LOF according to claim 1, characterized in that: In the step 2, the multiple eigenvalues ​​include length, extreme value difference, standard deviation and mean, and these four eigenvalues ​​are the four elements of the feature space; The length is the subsequence variation trend, the extreme value difference is the difference between the maximum value and the minimum value, the standard deviation is the degree of dispersion of the sequence data, and the mean is the average trend.

5. The method for identifying abnormal observation data based on SIP-LOF according to claim 4, characterized in that: In the step 2, multiple eigenvalues ​​of the subsequence are extracted, a multidimensional space is constructed according to all the eigenvalues, and then all the eigenvalues ​​are mapped in the multidimensional space, and the distance between each eigenvalue is calculated to obtain the corresponding distance value: The time series is mapped to the feature space. Each subsequence corresponds to a point in the feature space. The four elements consisting of the eigenvalues ​​are used to represent the position of each subsequence in the feature space. On this basis, the distance between each pattern is calculated to obtain the corresponding distance value.

6. The method for identifying abnormal observation data based on SIP-LOF according to claim 5, characterized in that: Since the ranges of the four characteristic quantities vary greatly, different magnitudes will affect the abnormal detection results of the observed data. Therefore, the four sets of characteristic values ​​are normalized separately, specifically: Let x1=(x 11 , x 12 , x 13 , ..., x 1n ) is one of the subsequences, x max With x min Respectively represent the maximum and minimum values ​​of each eigenvalue; 7. The method for identifying abnormal observation data based on SIP-LOF according to claim 1, characterized in that: In the step 3, after arranging the distance values ​​by size, find the kth distance and the first k nearest neighbors of each sample, then calculate the reachable distance and local reachable density of each sample to the k nearest neighbors, and obtain the corresponding density value, and then calculate the local outlier factor LOF according to all the density values ​​to obtain all the LOF values. Specifically, use the density-based local anomaly factor algorithm to perform abnormal pattern detection, and the specific steps are as follows: A. Calculate the Euclidean distance of the eigenvalues ​​between any two subsequences; B. After arranging the Euclidean distances, find the kth distance and the first k nearest neighbors of each sample; C. Calculate the reachable distance and local reachable density of each sample to its k nearest neighbors respectively.

8. The method for identifying abnormal observation data based on SIP-LOF according to claim 7, characterized in that: The reachable distance k is calculated as follows: reach-dist(p, q)=max(k-distance(q), d(p, q)); Where point p is a data point in the sample, point q is one of the k nearest neighbors of point p, d(p,q) represents the actual distance between point p and point q, and k-distance(q) is the distance from point q to its kth nearest neighbor.

9. The method for identifying abnormal observation data based on SIP-LOF according to claim 8, characterized in that: The local reachable density is calculated as follows:

10. The method for identifying abnormal observation data based on SIP-LOF according to claim 9, characterized in that: In step 3, the local outlier factor LOF is then calculated based on all distance values ​​and density values ​​to obtain all LOF values, specifically:

Images