Multi-source massive monitoring data preprocessing method

Abstract

The application provides a multi-source massive monitoring data preprocessing method, sample quantity analysis, distribution test and local density evaluation are carried out on time series monitoring data, and then an adaptive abnormality detection criterion is selected from a criterion library based on the analysis result, so that the defects of the prior art using a single criterion are avoided, and the problems of missed detection and false detection of abnormality detection, low processing efficiency and poor repair effect caused by insufficient method adaptability of monitoring data preprocessing are solved. In combination with the specific detection criterion, an abnormal point is identified and is interpolated, so that the abnormality detection accuracy and the processing efficiency are improved, the continuity and accuracy of the data after repair are ensured, and the multi-source massive data characteristics are adapted.

Description

Technical Field

[0001] This invention relates to the field of data preprocessing technology, and in particular to a method for preprocessing multi-source massive monitoring data. Background Technology

[0002] In fields such as underground space construction and health monitoring of large-scale engineering structures, the acquisition of monitoring data is the core foundation for determining the structural safety status and providing early warning of construction risks. Its accuracy and completeness directly determine the reliability of subsequent data analysis results and the scientific nature of engineering decisions.

[0003] With the development of automated monitoring technology, monitoring systems are increasingly characterized by multi-source and massive data collection. Data acquisition covers multiple dimensions, including structural deformation, strain, and land subsidence. The interaction of various factors, such as acquisition equipment, transmission links, and complex monitoring environments, inevitably leads to problems like outliers, random noise, and data loss in the monitoring data. Outliers manifest as abrupt numerical changes with disordered patterns, making them impossible to identify directly using simple thresholds. Random noise fluctuates around the data's trend, interfering with the accuracy of individual data points while not altering the overall trend. Data loss disrupts the continuity of time-series data. All these issues severely impact the accuracy of structural safety assessments. Therefore, effective preprocessing of multi-source, massive monitoring data has become a crucial aspect of automated monitoring systems.

[0004] Existing data preprocessing techniques fail to select appropriate anomaly detection criteria based on the sample size, distribution characteristics, and local density characteristics of the monitoring data. Instead, they blindly employ a single criterion to process complex data, resulting in poor data preprocessing performance. Summary of the Invention

[0005] This invention provides a method for preprocessing massive amounts of multi-source monitoring data to solve the problem of poor monitoring data preprocessing performance in existing technologies.

[0006] On the one hand, the present invention provides a method for preprocessing multi-source massive monitoring data, including: Acquire the time-series monitoring data to be processed; The time-series monitoring data are subjected to sample size analysis, distribution test, and local density assessment to generate analysis results; Based on the analysis results, at least one anomaly detection criterion is selected from a pre-defined anomaly identification criterion library; the anomaly identification criterion library includes the Leida criterion, the Chauville criterion, anomaly detection methods based on proximity, and anomaly detection methods based on local outlier factors; Anomalies are detected in the time-series monitoring data using the aforementioned anomaly detection criteria to obtain abnormal data points. The abnormal data points are interpolated to obtain the repaired time-series monitoring data.

[0007] Optionally, the time-series monitoring data is subjected to sample size analysis, distribution test, and local density assessment to generate analysis results, including: Calculate the total number of data points in the time-series monitoring data. If the total number of data points is greater than the preset sample size threshold, it is determined to be a large data sample; otherwise, it is determined to be a small data sample. The time-series monitoring data is subjected to a normality test. If the p-value corresponding to the test statistic is greater than the preset significance level, the time-series monitoring data is determined to approximately follow a normal distribution; otherwise, it is determined to be non-normally distributed. Based on the time-series monitoring data, the average distance between each data point and its k nearest neighbor data points is calculated to obtain the local density characterization value; Based on the statistical distribution of the local density characterization values ​​of all data points, it is determined whether the time-series monitoring data is of uniform density distribution type or non-uniform density distribution type.

[0008] Optionally, based on the analysis results, at least one anomaly detection criterion is selected from a preset anomaly identification criterion library, including: When the time-series monitoring data is a large data sample and the time-series monitoring data approximately follows a normal distribution, the Leida criterion is selected from the preset anomaly identification criterion library as the anomaly detection criterion.

[0009] Optionally, anomaly detection is performed on the time-series monitoring data using the anomaly detection criteria to obtain anomalous data points, including: The Leida criterion was chosen as the anomaly detection criterion. Calculate the mean and standard deviation of the time-series monitoring data; The absolute value of the difference between the value of each data point in the time-series monitoring data and the mean value is taken. If the absolute value is greater than three times the standard deviation, the data point is determined to be an outlier.

[0010] Optionally, based on the analysis results, at least one anomaly detection criterion is selected from a preset anomaly identification criterion library, further comprising: When the time-series monitoring data is a large data sample, the Schauville criterion is selected from the preset anomaly identification criterion library as the anomaly detection criterion.

[0011] Optionally, anomaly detection is performed on the time-series monitoring data using the anomaly detection criteria to obtain anomalous data points, including: Choosing the Schauville criterion as the anomaly detection criterion; Calculate the mean and standard deviation of the time-series monitoring data; The absolute value of the difference between the value of each data point in the time-series monitoring data and the mean value is taken. If the absolute value is greater than the product of the Schauville coefficient and the standard deviation, then the data point is determined to be an outlier.

[0012] Optionally, based on the analysis results, at least one anomaly detection criterion is selected from a preset anomaly identification criterion library, further comprising: When the time-series monitoring data is of uniform density distribution type, a proximity-based anomaly detection method is selected as the anomaly detection criterion.

[0013] Optionally, anomaly detection is performed on the time-series monitoring data using the anomaly detection criteria to obtain anomalous data points, including: Proximity-based anomaly detection methods were selected as the anomaly detection criteria. Based on the time-series monitoring data, the length of the adjacent region and the judgment threshold are determined; Calculate the average of the absolute differences between the data to be detected and all data points within the length of the adjacent region, and use this as the proximity difference of the data to be detected. If the proximity difference is greater than the judgment threshold, the data point to be detected is determined to be an abnormal data point.

[0014] Optionally, based on the analysis results, at least one anomaly detection criterion is selected from a preset anomaly identification criterion library, further comprising: When the time-series monitoring data is of uniform density distribution type, an anomaly detection method based on local outlier factors is selected as the anomaly detection criterion.

[0015] Optionally, anomaly detection is performed on the time-series monitoring data using the anomaly detection criteria to obtain anomalous data points, including: An anomaly detection method based on local outlier factors was selected as the anomaly detection criterion. Set the neighborhood size parameter k; For each data point to be measured in the time series monitoring data, calculate the distance from the data point to be measured to any other data point, and sort the distances in ascending order to obtain the sorting result; Based on the sorting results, the k-th distance of the data point to be tested is determined; the k-th distance is the distance from the data point to be tested to its k-th nearest neighbor. Determine the k-th distance neighborhood of the data point to be tested; the k-th distance neighborhood is the set of all data points whose distance to the data point to be tested is not greater than the k-th distance. Calculate the local outlier factor of the data point to be tested. If the local outlier factor is greater than a preset outlier threshold, the data point to be tested is determined to be an outlier data point.

[0016] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-source massive monitoring data preprocessing method as described above.

[0017] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-source massive monitoring data preprocessing method as described above.

[0018] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-source massive monitoring data preprocessing method as described above.

[0019] The multi-source massive monitoring data preprocessing method provided by this invention performs sample size analysis, distribution testing, and local density assessment on time-series monitoring data. Based on the analysis results, it selects suitable anomaly detection criteria from a criterion library. This avoids the shortcomings of existing technologies that use a single criterion, and solves the problems of missed and false anomaly detection, low processing efficiency, and poor repair effects caused by insufficient method adaptability in monitoring data preprocessing. By combining targeted detection criteria to identify and interpolate anomalies, it improves the accuracy and efficiency of anomaly detection while ensuring the continuity and accuracy of the repaired data, thus adapting to the characteristics of multi-source massive data. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the multi-source massive monitoring data preprocessing method provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a flowchart illustrating the multi-source massive monitoring data preprocessing method provided in this embodiment of the invention.

[0024] like Figure 1 As shown in the figure, the multi-source massive monitoring data preprocessing method provided in this embodiment of the invention mainly includes the following steps: 101. Obtain the time series monitoring data to be processed.

[0025] Among them, time-series monitoring data is continuous time-series data that reflects the state of structural deformation, strain, and surface subsidence, which is continuously collected by multiple sensors in scenarios such as underground space construction.

[0026] Time-series monitoring data is affected by factors such as acquisition equipment, transmission equipment, and monitoring environment, and is prone to problems such as outliers, random noise, and data loss. In addition, time-series monitoring data is multi-source, with sampling intervals of thousands to tens of thousands for a single monitoring section, and the whole data includes thousands to tens of thousands or even more sampling points. It is a multi-source massive time-series data, and targeted preprocessing is required to ensure the effectiveness of time-series monitoring data.

[0027] 102. Perform sample size analysis, distribution test and local density assessment on the time series monitoring data, and generate analysis results.

[0028] Among them, the sample size analysis needs to count the total number of sampling points of the time series monitoring data and determine whether the time series monitoring data meets the sample size requirements of different anomaly detection criteria.

[0029] Distribution tests are used to determine whether time-series monitoring data conforms to a normal distribution, providing a basis for selecting detection criteria that depend on distribution characteristics.

[0030] Local density assessment analyzes the density and differences of local point distribution in time-series monitoring data, identifies areas with uneven density distribution, and ultimately forms a comprehensive analysis result that includes sample size, data distribution type, and local density characteristics.

[0031] Specifically, sample size analysis, distribution tests, and local density assessments are performed on time-series monitoring data to generate analysis results, including: The total number of data points in the time-series monitoring data is calculated. If the total number of data points is greater than the preset sample size threshold, it is determined to be a large data sample; otherwise, it is determined to be a small data sample.

[0032] Calculating the total number of data points in time-series monitoring data requires clarifying the data acquisition logic. For example, if data is continuously collected by multiple sensors at fixed sampling intervals, the total number of data points n is the total number of valid data points collected by all sensors within the monitoring time domain, which is the product of the number of data points collected by a single sensor m and the number of sensors. If the data collected by a single sensor is denoted as {x1, x2, ..., x...} m}, then the total number of data points from multiple sensors .in, Assign sensor numbers. Set the application boundary of the Leyda criterion. The criterion applies when the amount of data collected by a single sensor is less than 10. The preset sample size threshold can be set according to the actual situation. The judgment rule is: if the total number of data points n is greater than the preset sample size threshold, it is judged as a large data sample; if the total number of data points n ≤ the preset sample size threshold, it is judged as a small data sample.

[0033] A normality test is performed on the time-series monitoring data. If the p-value corresponding to the test statistic is greater than the preset significance level, the time-series monitoring data is determined to be approximately normally distributed; otherwise, it is determined to be non-normally distributed.

[0034] When performing a normality test on time-series monitoring data, the determination is made by comparing the p-value of the test statistic with a preset significance level. First, a suitable normality test statistical method is selected for the time-series data to test whether the data conforms to a normal distribution. The preset significance level is α=0.05, and the p-value corresponding to the test statistic is calculated using the appropriate test algorithm.

[0035] The judgment rule is as follows: if the p-value is greater than the preset significance level, it indicates that the data does not have the characteristics of significantly deviating from the normal distribution and is judged to be approximately following the normal distribution; if the p-value is ≤0.05, it is judged to be non-normal and anomaly detection methods such as LOF that do not rely on the assumption of normal distribution should be used.

[0036] Based on time-series monitoring data, the average distance between each data point and its k nearest neighbor data points is calculated to obtain the local density characterization value.

[0037] The local density representation value is obtained by calculating the proximity relationship between data points. For example, the value of k is determined, which represents the number of nearest neighbor data points and needs to be adjusted according to the data sampling interval and the characteristics of data changes. Secondly, for each data point p in the time-series monitoring data, according to the definition of d(p,o), where d(p,o) represents the distance between point p and point o, the distance between p and all other data points in the dataset is calculated, and the k data points closest to p are selected, i.e., the k nearest neighbors of p. Finally, the average distance between the k nearest neighbors and p is calculated and used as the local density representation value of the data point p.

[0038] Based on the statistical distribution of the local density characterization values ​​of all data points, determine whether the time-series monitoring data is of uniform or non-uniform density distribution type.

[0039] The determination of density distribution type requires based on the statistical distribution characteristics of the local density representation values ​​of all data points, combined with data distribution difference analysis. For example, calculating the local density representation values ​​of all data points... Statistical parameters, including standard deviation Range Interquartile range Set the judgment criteria; if the statistical parameters display all data points... The numerical differences are small, such as the standard deviation. and The ratio of the mean to the range is less than a preset threshold, or the range is less than a preset threshold. A relatively small value indicates that the average distance between the data points in each region of the time-series monitoring data and their k nearest neighbors is similar, and the local density distribution is relatively consistent, thus classifying it as a uniform density distribution type; if the statistical parameters show... The numerical differences are significant, such as the standard deviation. and The ratio of the mean to the mean is greater than or equal to 0.1, or there are partial data points. If the density of data points is significantly greater than or less than that of other data points, it indicates that there are significant differences in the local density of data points in different regions, and there is a distinction between high-density and low-density regions, which is judged as a non-uniform density distribution type.

[0040] 103. Based on the analysis results, select at least one anomaly detection criterion from the preset anomaly identification criterion library.

[0041] The anomaly detection criterion library includes the Leida criterion, the Chauville criterion, proximity-based anomaly detection methods, and local outlier-based anomaly detection methods.

[0042] Specifically, if the time-series monitoring data is a large data sample and the time-series monitoring data approximately follows a normal distribution, then the Leida criterion is selected as the anomaly detection criterion from the preset anomaly identification criterion library.

[0043] When the time-series monitoring data is a large data sample, the Schauville criterion is selected as the anomaly detection criterion from the preset anomaly identification criterion library.

[0044] When the time series monitoring data is of uniform density distribution type, the anomaly detection method based on proximity is selected as the anomaly detection criterion.

[0045] When the time series monitoring data is of uniform density distribution type, the anomaly detection method based on local outlier factors is selected as the anomaly detection criterion.

[0046] 104. Anomaly detection criteria are used to detect anomalies in time-series monitoring data to obtain abnormal data points.

[0047] In some embodiments, anomaly detection criteria are used to detect anomalies in time-series monitoring data to obtain anomalous data points, including: The Leida criterion was chosen as the anomaly detection criterion. Calculate the mean and standard deviation of the time-series monitoring data; The absolute value of the difference between the value of each data point in the time series monitoring data and the mean is taken. If the absolute value is greater than three times the standard deviation, the data point is considered an outlier.

[0048] Specifically, the Leida criterion is also known as... The outlier detection criterion is one of the commonly used methods for removing outliers in monitoring data processing. Extensive experimental research has shown that when the sample monitoring data is large, this criterion is simple, convenient, and highly efficient for detecting outliers; however, when the sample monitoring data is small, using this criterion to detect outliers is unreliable.

[0049] Assume a set of time-series monitoring data The characteristics of the changes in time-series monitoring data are as follows: ; It is known that Based on the observation data, it can be obtained that... indivual Value. Let and These are the changes in the data sequence. The expressions for the mean and standard deviation of the change are: L; ; Then, calculate the result for each data sequence. The ratio of the deviation of the value to the mean square error ,in, The expression is: ; If along When the value is greater than 3, it is considered that... If the value is an outlier, the data point is considered an anomaly and should be discarded.

[0050] The Leida criterion requires no table lookup and is easy to use. It is suitable for use when there is a large amount of time series monitoring data or when the accuracy requirement is not high. The Leida criterion is invalid when the time series monitoring data is less than 10.

[0051] In some embodiments, anomaly detection is performed on time-series monitoring data using anomaly detection criteria to obtain anomalous data points, including: Choosing the Schauville criterion as the anomaly detection criterion; Calculate the mean and standard deviation of the time-series monitoring data; The absolute value of the difference between the value of each data point in the time series monitoring data and the mean is taken. If the absolute value is greater than the product of the Schauville coefficient and the standard deviation, the data point is considered an outlier.

[0052] Specifically, when choosing the Chauville criterion as the anomaly detection criterion, the first step in anomaly detection is to determine the Chauville coefficients of a set of data. When in... In a measurement, if a certain error occurs less than half the time, the outlier is discarded. For a normal distribution, the probability that it cannot occur is... ; Then, based on the normal distribution table and known... The value can be used to obtain the Schauville coefficients. In engineering applications, to improve calculation efficiency, the coefficients can also be directly looked up from the corresponding coefficient table, as shown in Table 1 below. When the error requirement is not very strict, the following approximate formula can also be used for calculation: .

[0053] If the absolute value of the difference between a data point and the mean is greater than the product of the sample standard deviation and the Schauville coefficient, then the data point is discarded. Otherwise, the data point is considered a normal measurement.

[0054] ; The Schauville criterion is suitable for real-time data acquisition and processing. In addition to the size of the allowable error limit, the accuracy is also related to the accuracy of the first two measured values. If the change law of the measured physical quantity is not monotonically increasing or monotonically decreasing, this method will produce a large error at the inflection point of the function, and in severe cases, it will be unusable.

[0055] Table 1. Partial Schauweiler Coefficients

[0056] In some embodiments, anomaly detection is performed on time-series monitoring data using anomaly detection criteria to obtain anomalous data points, including: Proximity-based anomaly detection methods were selected as the anomaly detection criteria. Based on time-series monitoring data, determine the length of the adjacent area and the judgment threshold; Calculate the average of the absolute differences between the data to be detected and all data points within the length of the neighboring region, and use this as the proximity difference of the data point to be detected. If the proximity difference is greater than the judgment threshold, the data point to be detected is determined to be an abnormal data point.

[0057] Among them, proximity-based anomaly detection methods calculate the difference between values ​​in neighboring regions of monitored data and combine this with a set threshold to determine whether a data point is abnormal. This approach is suitable for time-series monitoring scenarios where data changes are relatively stable and density distribution differences are small. When choosing a proximity-based anomaly detection method, it is essential to ensure that the sample data originates from the time-series monitoring data itself and that the data exhibits relatively stable changes. Furthermore, it is important to be aware of the inherent characteristics of proximity-based anomaly detection methods: they are sensitive to the length of the neighboring region and have a time complexity of O(m²), where m is the number of data collection points. This makes them unsuitable for large-scale data scenarios and datasets with highly variable density distributions. Therefore, it is necessary to assess in advance whether the scale and distribution characteristics of the time-series monitoring data are compatible.

[0058] Based on time-series monitoring data, when determining the length of the nearest neighbor region and the judgment threshold, a relatively stable portion of the time-series monitoring data is selected as sample data. Simultaneously, the length L of the variable nearest neighbor region is determined. The length L of the variable nearest neighbor region needs to be reasonably set in conjunction with the data sampling interval and the frequency of change of the monitoring indicators: if the length L of the variable nearest neighbor region is too small, it may not be able to fully capture the correlation characteristics of the adjacent data, leading to the missed detection of some anomalies; if the length L of the variable nearest neighbor region is too large, it is easy to include normally fluctuating adjacent data in the calculation, causing normal points to be misclassified as anomalies. Then, according to the nearest neighbor difference formula: Calculate the nearest neighbor difference for each sample point in the sample data, and then calculate the arithmetic mean of the nearest neighbor differences for all sample points. The average value is the threshold for anomaly detection. The rationality of the threshold directly depends on the stationarity of the sample data and the accuracy of the length L of the variable neighborhood.

[0059] For the dataset to be detected in the time series monitoring data, calculations are performed on each data point to be detected one by one, with the determined range of the adjacent region length L.

[0060] For the data points to be detected at any sampling time Based on the calculation logic of the nearest neighbor difference formula, the data points to be detected are extracted. Calculate the data point to be detected from L adjacent data points. The difference between the value of each of the L data points and the value of the absolute value of all differences is calculated to eliminate the influence of the sign on the degree of difference.

[0061] Finally, the arithmetic mean of these absolute differences is calculated, and the arithmetic mean is the data point to be tested. The proximity difference value directly reflects the proximity difference of the data points to be detected. The degree of deviation from neighboring data.

[0062] The proximity difference of each data point to be detected is compared one by one with the judgment threshold calculated from the sample data. If the proximity difference of a data point to be detected is greater than the judgment threshold, it indicates that the data point to be detected differs significantly from the stable data in the surrounding area, exceeding the fluctuation range of normal data. Based on the anomaly detection criterion based on proximity, the data point to be detected is judged as an anomaly data point. If the proximity difference is less than or equal to the judgment threshold, it indicates that the deviation of the data point to be detected from the surrounding data is within a reasonable range, and it is judged as a normal data point.

[0063] In some embodiments, anomaly detection is performed on time-series monitoring data using anomaly detection criteria to obtain anomalous data points, including: An anomaly detection method based on local outlier factors was selected as the anomaly detection criterion. Set the neighborhood size parameter k; For each data point to be measured in the time series monitoring data, calculate the distance from the data point to any other data point, and sort the distances in ascending order to obtain the sorting results; Based on the sorting results, determine the k-th distance of the data point to be tested; the k-th distance is the distance from the data point to be tested to its k-th nearest neighbor. Determine the k-th distance neighborhood of the data point to be tested; the k-th distance neighborhood is the set of all data points whose distance to the data point to be tested is not greater than the k-th distance. Calculate the local outlier factor of the data point to be tested. If the local outlier factor is greater than the preset outlier threshold, the data point to be tested is determined to be an outlier.

[0064] Specifically, the Local Outlier Factor (LOF)-based anomaly detection method is a density-based anomaly detection technique. It identifies anomalies by comparing the density differences between a data point and its neighbors. The greater the density difference between a point and its neighbors, the more likely it is to be an anomaly, and density is quantified by the distance between points. The LOF-based anomaly detection method does not rely on the normal distribution assumption of the data and is suitable for time-series monitoring scenarios where density distributions differ, especially for situations where local sparse anomalies exist in the data.

[0065] To use an anomaly detection method based on local outlier factors, the neighborhood size parameter k must first be obtained. The neighborhood size parameter k is an adjustment parameter of the LOF algorithm. The value of the neighborhood size parameter k directly affects the results of neighborhood point screening and the accuracy of subsequent density calculation. It needs to be reasonably set in combination with the sampling interval, data fluctuation frequency and distribution characteristics of time series monitoring data.

[0066] For example, the setting of k must satisfy the following condition in the time series monitoring dataset: for any data point p to be tested, there must be at least k data points that do not contain p itself, and at most k-1 data points that do not contain p itself.

[0067] After setting the neighborhood size parameter k, for each data point p in the time-series monitoring data, the distance between p and all other data points in the dataset is calculated according to the definition of d(p,o) representing the distance between p and o. After calculation, all distance values ​​are sorted in ascending order from smallest to largest to obtain the sorted results. Sort the distance values ​​to select the k data points closest to the data point p, providing an ordered distance basis for determining the k-th distance and dividing the k-th distance neighborhood.

[0068] Based on the definition of the k-th distance, the k-th distance of the data point p to be tested is determined by combining the sorting results. The k-th distance must satisfy two conditions: first, in the time series monitoring dataset, there must be at least k data points o such that the distance d(p,o) between data point o and p is less than or equal to the k-th distance; second, in the dataset, there must be at most k-1 data points o such that the distance d(p,o) between k-1 data points o and p is less than the k-th distance. From the sorting results, the distance value corresponding to the k-th position is extracted, and the distance value corresponding to the k-th position is the k-th distance of the data point p to be tested.

[0069] After determining the k-th distance of the data point to be tested, it is necessary to determine the k-th distance neighborhood of the data point. When determining the k-th distance neighborhood, a k-th distance neighborhood of the data point p is constructed based on the k-th distance. The k-th distance neighborhood is defined as the set of all data points in the time-series monitoring data whose distance d(p,o) to p is less than or equal to the k-th distance, and which do not include p itself. The construction of this set uses the k-th distance as the boundary, ensuring that the neighborhood includes the nearest neighboring data point to p.

[0070] After determining the k-th distance and its neighborhood, it is necessary to calculate the local outlier factor of the data point to be tested. To calculate the local outlier factor, firstly, the k-th reachable distance is calculated according to the formula for the k-th reachable distance, which is: ; in, for arrive The Reachable distance for arrive The time distance.

[0071] For each point in the k-th distance neighborhood , arrive The The reachable distance is max( The distance, ).

[0072] Subsequently, based on the formula for calculating local reachability density, the following calculations were performed. Locally achievable density The formula for calculating the locally reachable density is as follows: ; in, For point The k-th distance neighborhood.

[0073] Locally achievable density For all points in the neighborhood arrive The reciprocal of the average of the k-th reachable distances.

[0074] Finally, the local outlier factor (LOF(p)) of p is calculated using the local outlier factor formula. The local outlier factor formula is as follows: ; The Local Outlier Factor (LOF) (p) is the arithmetic mean of the ratios of the local reachability density lrd(o) of all points o in their neighborhood to the local reachability density lrd(p) of point p. The preset outlier threshold is set to 1, but can be adjusted based on data characteristics. If LOF(p) is greater than the preset outlier threshold, it indicates that the local density of point p is significantly lower than the density of its neighboring points. Based on the anomaly detection logic of the LOF algorithm, p is determined to be an outlier data point. If LOF(p) ≤ the threshold, then p is a normal data point.

[0075] 105. Perform interpolation on the abnormal data points to obtain the repaired time series monitoring data.

[0076] Interpolation of outlier data points can be achieved using methods such as linear interpolation, spline interpolation, and Lagrange interpolation.

[0077] For example, when using linear interpolation, it is assumed that the data changes linearly between two adjacent normal data points. For outlier data points, if there are normal data points before and after the outlier data point... and Then abnormal data points It can be calculated using a linear interpolation formula, where the linear interpolation formula is:

[0078] in, for Sampling time, for The sampling time.

[0079] Figure 2 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0080] like Figure 2 As shown, the electronic device may include a processor 210, a communications interface 220, a memory 230, and a communication bus 240. The processor 210, communications interface 220, and memory 230 communicate with each other via the communication bus 240. The processor 210 can call logical instructions stored in the memory 230 to execute a multi-source, massive monitoring data preprocessing method.

[0081] Furthermore, the logical instructions in the aforementioned memory 230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multi-source massive monitoring data preprocessing method provided by the above methods.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-source massive monitoring data preprocessing method X provided by the above methods.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for preprocessing multi-source massive monitoring data, characterized in that, include: Acquire the time-series monitoring data to be processed; The time-series monitoring data are subjected to sample size analysis, distribution test, and local density assessment to generate analysis results; Based on the analysis results, at least one anomaly detection criterion is selected from a pre-defined anomaly identification criterion library; the anomaly identification criterion library includes the Leida criterion, the Chauville criterion, anomaly detection methods based on proximity, and anomaly detection methods based on local outlier factors; Anomalies are detected in the time-series monitoring data using the aforementioned anomaly detection criteria to obtain abnormal data points. The abnormal data points are interpolated to obtain the repaired time-series monitoring data.

2. The multi-source massive monitoring data preprocessing method according to claim 1, characterized in that, The time-series monitoring data are subjected to sample size analysis, distribution test, and local density assessment to generate analysis results, including: Calculate the total number of data points in the time-series monitoring data. If the total number of data points is greater than the preset sample size threshold, it is determined to be a large data sample; otherwise, it is determined to be a small data sample. The time-series monitoring data is subjected to a normality test. If the p-value corresponding to the test statistic is greater than the preset significance level, the time-series monitoring data is determined to approximately follow a normal distribution; otherwise, it is determined to be non-normally distributed. Based on the time-series monitoring data, the average distance between each data point and its k nearest neighbor data points is calculated to obtain the local density characterization value; Based on the statistical distribution of the local density characterization values ​​of all data points, it is determined whether the time-series monitoring data is of uniform density distribution type or non-uniform density distribution type.

3. The multi-source massive monitoring data preprocessing method according to claim 2, characterized in that, Based on the analysis results, at least one anomaly detection criterion is selected from a pre-defined anomaly identification criterion library, including: When the time-series monitoring data is a large data sample and the time-series monitoring data approximately follows a normal distribution, the Leida criterion is selected from the preset anomaly identification criterion library as the anomaly detection criterion.

4. The multi-source massive monitoring data preprocessing method according to claim 3, characterized in that, Anomalies are detected in the time-series monitoring data using the aforementioned anomaly detection criteria to obtain anomalous data points, including: The Leida criterion was chosen as the anomaly detection criterion. Calculate the mean and standard deviation of the time-series monitoring data; The absolute value of the difference between the value of each data point in the time-series monitoring data and the mean value is taken. If the absolute value is greater than three times the standard deviation, the data point is determined to be an outlier.

5. The multi-source massive monitoring data preprocessing method according to claim 2, characterized in that, Based on the analysis results, at least one anomaly detection criterion is selected from a pre-defined anomaly identification criterion library, and the method further includes: When the time-series monitoring data is a large data sample, the Schauville criterion is selected from the preset anomaly identification criterion library as the anomaly detection criterion.

6. The multi-source massive monitoring data preprocessing method according to claim 5, characterized in that, Anomaly detection is performed on the time-series monitoring data using the aforementioned anomaly detection criteria to obtain anomalous data points, including: Choosing the Schauville criterion as the anomaly detection criterion; Calculate the mean and standard deviation of the time-series monitoring data; The absolute value of the difference between the value of each data point in the time-series monitoring data and the mean value is taken. If the absolute value is greater than the product of the Schauville coefficient and the standard deviation, then the data point is determined to be an outlier.

7. The multi-source massive monitoring data preprocessing method according to claim 2, characterized in that, Based on the analysis results, at least one anomaly detection criterion is selected from a pre-defined anomaly identification criterion library, and the method further includes: When the time-series monitoring data is of uniform density distribution type, a proximity-based anomaly detection method is selected as the anomaly detection criterion.

8. The multi-source massive monitoring data preprocessing method according to claim 7, characterized in that, Anomaly detection is performed on the time-series monitoring data using the aforementioned anomaly detection criteria to obtain anomalous data points, including: Proximity-based anomaly detection methods were selected as the anomaly detection criteria. Based on the time-series monitoring data, the length of the adjacent region and the judgment threshold are determined; Calculate the average of the absolute differences between the data to be detected and all data points within the length of the adjacent region, and use this as the proximity difference of the data to be detected. If the proximity difference is greater than the judgment threshold, the data point to be detected is determined to be an abnormal data point.

9. The multi-source massive monitoring data preprocessing method according to claim 2, characterized in that, Based on the analysis results, at least one anomaly detection criterion is selected from a pre-defined anomaly identification criterion library, and the method further includes: When the time-series monitoring data is of uniform density distribution type, an anomaly detection method based on local outlier factors is selected as the anomaly detection criterion.

10. The multi-source massive monitoring data preprocessing method according to claim 9, characterized in that, Anomaly detection is performed on the time-series monitoring data using the aforementioned anomaly detection criteria to obtain anomalous data points, including: An anomaly detection method based on local outlier factors was selected as the anomaly detection criterion. Set the neighborhood size parameter k; For each data point to be measured in the time series monitoring data, calculate the distance from the data point to be measured to any other data point, and sort the distances in ascending order to obtain the sorting result; Based on the sorting results, the k-th distance of the data point to be tested is determined; the k-th distance is the distance from the data point to be tested to its k-th nearest neighbor. Determine the k-th distance neighborhood of the data point to be tested; the k-th distance neighborhood is the set of all data points whose distance to the data point to be tested is not greater than the k-th distance. Calculate the local outlier factor of the data point to be tested. If the local outlier factor is greater than a preset outlier threshold, the data point to be tested is determined to be an outlier data point.

Images