Sewage treatment anomaly detection and early warning system based on big data analysis
By classifying and optimizing the K-value of wastewater treatment data indicators, the problem of misjudgment in wastewater treatment by the KNN algorithm was solved, achieving higher detection accuracy and robustness.
Patent Information
- Application Number
- CN202411186906.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-28
AI Technical Summary
Existing KNN algorithms are prone to misjudgment in wastewater treatment anomaly detection due to the diversity of data types and large data volumes, which reduces the accuracy and robustness of detection.
The data analysis module categorizes data indicators, constructs scatter plots, uses Spearman correlation coefficient to classify categories, and optimizes the K value through Euclidean distance and deviation coefficient to adaptively obtain the optimal K value, eliminate interference from data deviation points, and improve detection accuracy.
In the detection of anomalies in wastewater treatment, the optimal K value is obtained adaptively, which reduces the interference of data deviation points on the detection and improves the accuracy and reliability of the detection.
Smart Images

Figure CN118709065B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a sewage treatment anomaly detection and early warning system based on big data analysis. Background Art
[0002] At present, sewage treatment anomaly detection and early warning play an increasingly important role in the field of modern sewage treatment. The application of big data makes it possible to process large amounts of real-time data and historical data. It can monitor various parameters of sewage treatment in real time, such as water quality, flow, temperature, etc., and perform complex data analysis, pattern recognition and anomaly detection. Once an abnormal situation is detected, an early warning can be issued in time to help operators take timely measures to avoid further problems.
[0003] In the prior art, various monitoring data in the sewage treatment process are collected and the KNN algorithm is used to detect abnormal data in the various monitoring data, thereby achieving the purpose of abnormal warning of sewage treatment.
[0004] However, in the traditional way, when using the KNN algorithm for abnormal data detection, the sum or average distance between each point and its K nearest neighbor points is usually calculated, and a distance threshold is set. Points whose sum or average distance is greater than the distance threshold are considered abnormal points. However, in the actual process, due to the large type and amount of data in sewage treatment, there may be very few nearest neighbor points among the K nearest neighbor points of a data point that are far away from the data point, or there may be one or more points among the nearest neighbor points of the point that have a large degree of dispersion for the overall data point distribution, which increases the sum or average distance between the data point and the K nearest neighbor points and exceeds the distance threshold. In this case, the data point will be misjudged and mistakenly identified as an abnormal point, resulting in reduced detection accuracy.
[0005] Therefore, how to improve the accuracy and robustness of sewage treatment abnormality warning using KNN algorithm has become an urgent problem to be solved. Summary of the invention
[0006] In view of this, an embodiment of the present invention provides a sewage treatment anomaly detection and early warning system based on big data analysis to solve the problem of how to improve the accuracy and robustness of sewage treatment anomaly early warning using the KNN algorithm.
[0007] An embodiment of the present invention provides a sewage treatment anomaly detection and early warning system based on big data analysis, the system comprising:
[0008] A data analysis module, used to respectively obtain at least two data indicators of sewage treatment within a preset period, obtain a data sequence for each data indicator, and divide the data sequences of all data indicators into at least one category according to the correlation between the data sequences;
[0009] A data processing module is used to normalize all data indicators in any category to obtain normalized data indicators, construct a scatter plot of all normalized data indicators, obtain an initial K value of the KNN algorithm when classifying all data points in the scatter plot using the KNN algorithm, and obtain the abnormality degree of each data point in the scatter plot according to the initial K value;
[0010] A parameter acquisition module, used to acquire an abnormal point according to the abnormal degree of all data points in the scatter plot, and adaptively acquire an optimal K value for each data point in the scatter plot according to the abnormal point and the initial K value;
[0011] The anomaly detection module is used to classify all data points in the scatter plot according to the optimal K value of each data point in the scatter plot using the KNN algorithm to obtain classification results corresponding to the categories, and to provide abnormal warning for sewage treatment according to the classification results corresponding to each category.
[0012] Furthermore, the data analysis module divides the data sequences of all data indicators into at least one category according to the correlation between the data sequences, including:
[0013] For any two data sequences, the Spearman correlation coefficient between the two data sequences is calculated, and the absolute value of the Spearman correlation coefficient is used as the trend change characteristic value of the two data sequences;
[0014] If the trend change characteristic value is within a preset range, the two data series are classified into the same category, and the data series of all data indicators are divided into at least one category according to the trend change characteristic value between every two data series.
[0015] Furthermore, the data processing module obtains the abnormality degree of each data point in the scatter plot according to the initial K value, including:
[0016] For any data point in the scatter plot, respectively calculate the Euclidean distance between the data point and each other data point, sort all the Euclidean distances from small to large, and take the other data points corresponding to the first K Euclidean distances among the sorted Euclidean distances as the nearest neighbor data points of the data point;
[0017] According to the distribution characteristics of each of the nearest neighbor data points, the deviation coefficient of each of the nearest neighbor data points is obtained, and the Euclidean distance between the data point and each of the nearest neighbor data points is optimized by using the deviation coefficient of each of the nearest neighbor data points to obtain the characteristic distance between the data point and each of the nearest neighbor data points;
[0018] The abnormality degree of the data point is obtained according to the characteristic distance and the Euclidean distance between the data point and each of the nearest neighboring data points.
[0019] Furthermore, the data processing module obtains the deviation coefficient of each of the nearest neighbor data points according to the distribution characteristics of each of the nearest neighbor data points, including:
[0020] Performing curve fitting on all data points in the scatter plot to obtain a data trend distribution fitting curve;
[0021] For any nearest neighbor data point, obtain the straight line between the data point and the nearest neighbor data point, obtain the intersection of the straight line and the data trend distribution fitting curve, obtain the tangent of the data trend distribution fitting curve at the intersection, obtain the sine value of the minimum angle between the tangent and the straight line, and add the sine value to the constant 1 to obtain the deviation coefficient of the nearest neighbor data point.
[0022] Furthermore, the data processing module uses the deviation coefficient of each of the nearest neighbor data points to optimize the Euclidean distance between the data point and each of the nearest neighbor data points to obtain the characteristic distance between the data point and each of the nearest neighbor data points, including:
[0023] For the Euclidean distance between the data point and any nearest neighbor data point, the product of the Euclidean distance and the deviation coefficient of the nearest neighbor data point is used as the characteristic distance between the data point and the nearest neighbor data point.
[0024] Furthermore, the data processing module obtains the abnormality degree of the data point according to the characteristic distance and the Euclidean distance between the data point and each of the nearest neighbor data points, including:
[0025] Obtaining a maximum value of characteristic distances between the data point and each of the nearest neighboring data points, accumulating the characteristic distances between the data point and each of the nearest neighboring data points to obtain an accumulation result, calculating a ratio of the maximum value to the accumulation result as a first variable, and taking a hyperbolic tangent function value of the first variable as a first function value;
[0026] Calculate the average value of the Euclidean distance between the data point and each of the nearest neighbor data points to obtain an average distance, calculate the standard deviation of the Euclidean distance between the data point and each of the nearest neighbor data points, use the ratio of the standard deviation to the average distance as a second variable, and use the hyperbolic tangent function value of the second variable as a second function value;
[0027] The first function value and the second function value are weightedly summed to obtain the abnormality degree of the data point.
[0028] Furthermore, the parameter acquisition module acquires abnormal points according to the abnormality degree of all data points in the scatter plot, including:
[0029] For any abnormality level, if the abnormality level is within a preset abnormality level range, the data point corresponding to the abnormality level is marked as an abnormal point.
[0030] Furthermore, the parameter acquisition module adaptively acquires the optimal K value for each data point in the scatter plot according to the abnormal point and the initial K value, including:
[0031] For any abnormal point in the scatter plot, the initial K value is reduced and adjusted using a preset adjustment value to obtain an adjusted K value. According to the adjusted K value, the abnormality degree of the abnormal point is obtained. If the abnormality degree of the abnormal point is within the preset abnormality degree range, the adjusted K value is further reduced and adjusted using the adjustment value to obtain a new adjusted K value. The new adjusted K value is used as the adjusted K value, and the step of obtaining the abnormality degree of the abnormal point is repeated until the abnormality degree of the abnormal point is no longer within the preset abnormality degree range. The corresponding adjusted K value is used as the optimal K value of the abnormal point.
[0032] The initial K value is used as the optimal K value for each non-abnormal point in the scatter plot.
[0033] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0034] The present invention provides a sewage treatment anomaly detection and early warning system based on big data analysis, comprising a data analysis module, which is used to respectively obtain at least two data indicators of sewage treatment within a preset time period, obtain a data sequence of each data indicator, and divide the data sequences of all data indicators into at least one category according to the correlation between the data sequences; a data processing module, which is used to normalize all data indicators in any category to obtain normalized data indicators, construct a scatter plot of all normalized data indicators, obtain an initial K value of the KNN algorithm when classifying all data points in the scatter plot using a KNN algorithm, and obtain the degree of abnormality of each data point in the scatter plot according to the initial K value; a parameter acquisition module, which is used to obtain abnormal points according to the degree of abnormality of all data points in the scatter plot, and adaptively obtain the optimal K value of each data point in the scatter plot according to the abnormal points and the initial K value; an abnormality detection module, which is used to classify all data points in the scatter plot using the KNN algorithm according to the optimal K value of each data point in the scatter plot, obtain the classification result corresponding to the category, and perform an abnormal early warning of sewage treatment according to the classification result corresponding to each category. Among them, since sewage treatment anomaly detection contains many types of data and a large amount of data, if a scatter plot is constructed directly, the scatter plot will be very crowded, making it difficult to distinguish the distribution of each data indicator, and will cover up the correlation or pattern between the data. Therefore, the data series of each data indicator are first classified, and then a scatter plot is constructed based on the classification results. Since the sewage treatment anomaly detection process contains a large amount of real-time data and historical data, there will be some data deviation points that are far away from the sample points in the K nearest neighbor data points. These data deviation points will cause great interference in the operation of the KNN algorithm, causing the sample points to be mistaken for abnormal points. Therefore, according to the distribution characteristics of each data point in the scatter plot, the degree of abnormality of each data point is obtained, and the optimal K value is adaptively obtained according to the degree of abnormality. By reducing the K value, the interference of the data deviation points is eliminated. When using the KNN algorithm for sewage treatment anomaly detection, the optimal K value is selected to analyze and identify the data, eliminating the interference of the data deviation points during the operation of the KNN algorithm, and improving the accuracy of data point detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0036] Figure 1It is a structural block diagram of a sewage treatment anomaly detection and early warning system based on big data analysis provided in Example 1 of the present invention. DETAILED DESCRIPTION
[0037] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.
[0038] It should be noted that the terms "first", "second", etc. in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.
[0039] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.
[0040] See also Figure 1 , is a structural block diagram of a sewage treatment anomaly detection and early warning system based on big data analysis provided in the first embodiment of the present invention, such as Figure 1 As shown, the system may include:
[0041] The data analysis module 11 is used to respectively obtain at least two data indicators of sewage treatment within a preset time period, obtain a data sequence for each data indicator, and divide the data sequences of all data indicators into at least one category according to the correlation between the data sequences.
[0042] First, determine at least two data indicators that need to be obtained in sewage treatment anomaly detection, such as pH value, total suspended solids, turbidity, water quality, flow rate, temperature, etc., set the preset time period to 1 hour, there is no restriction here, the implementer can set it according to the implementation scenario, and obtain the data sequence corresponding to each data indicator within 1 hour according to the determined data indicators. In order to ensure the accuracy and consistency of each data in the data sequence, after obtaining all data sequences, each data in all data sequences is preprocessed to achieve the purpose of data cleaning for all data sequences, wherein data preprocessing includes but is not limited to: processing outliers, identifying and adjusting outliers, ensuring the accuracy and consistency of data; processing missing values, filling or deleting missing data points; data conversion, such as normalizing or standardizing the data to ensure that different indicators are comparable. Among them, data preprocessing technology belongs to the prior art and will not be repeated here.
[0043] In sewage treatment anomaly detection, when using the KNN algorithm, a scatter plot is usually directly constructed for multiple data indicators, and multiple data indicators are detected together instead of detecting each data indicator separately. This method can capture the relationship between data indicators and improve the accuracy of anomaly detection. However, in the actual process, since there are many types of data indicators and a large amount of data included in sewage treatment anomaly detection, if a scatter plot is directly constructed, the scatter plot will be very crowded, and it will be difficult to distinguish the distribution of each data indicator, which will cover up the correlation or pattern between the data. Therefore, after preprocessing all data sequences, each data indicator is classified according to the correlation between each data indicator.
[0044] Among them, the method of classifying various data indicators according to the correlation between each data indicator includes:
[0045] For any two data sequences, the Spearman correlation coefficient between the two data sequences is calculated, and the absolute value of the Spearman correlation coefficient is used as the trend change characteristic value of the two data sequences;
[0046] If the trend change characteristic value is within a preset range, the two data series are classified into the same category, and the data series of all data indicators are divided into at least one category according to the trend change characteristic value between every two data series.
[0047] In one embodiment, all data indicators are paired in pairs, and for any two data sequences corresponding to a pair of data indicators, the Spearman correlation coefficient between the two data sequences is calculated, and the absolute value of the Spearman correlation coefficient is used as the trend change characteristic value of the two data sequences, wherein the Spearman correlation coefficient is a prior art and will not be repeated here, and the calculation formula of the trend change characteristic value is:
[0048]
[0049] in, Represents the trend change characteristic value of the j-th pair of data indicators, represents the Spearman correlation coefficient between the jth pair of data indicators, Represents the absolute value symbol.
[0050] The closer it is to 1, the stronger the correlation between the j-th pair of data indicators. Therefore, the preset range is set to [0.8, 1]. There is no restriction here. The implementer can set it according to the implementation scenario. If the trend change characteristic value of the j-th pair of data indicators is within [0.8, 1], the two data series corresponding to the j-th pair of data indicators are divided into the same initial category. Similarly, the trend change characteristic values between the two data series corresponding to all pairs of data indicators are obtained, and the data series corresponding to a pair of data indicators with trend change characteristic values within [0.8, 1] are divided into one initial category. According to transitivity, if there is at least one data series corresponding to the same data indicator between at least two initial categories, these initial categories are divided into one category. The data series corresponding to all data indicators that are not within the preset range are divided into one category. For example: there are data sequences corresponding to 7 data indicators, namely A, B, C, D, E, F, and G. If the trend change characteristic value between A and B is within [0.8, 1], the trend change characteristic value between B and C is within [0.8, 1], and the trend change characteristic value between D and E is within [0.8, 1], then the data sequences corresponding to the three data indicators A, B, and C are divided into one category, the data sequences corresponding to the two data indicators D and E are divided into one category, and the data sequences corresponding to the remaining two data indicators F and G are divided into one category.
[0051] At this point, the data series of all data indicators are divided into at least one category.
[0052] The data processing module 12 is used to normalize all data indicators in any category to obtain normalized data indicators, construct a scatter plot of all normalized data indicators, and when using the KNN algorithm to classify all data points in the scatter plot, obtain the initial K value of the KNN algorithm, and obtain the abnormality degree of each data point in the scatter plot based on the initial K value.
[0053] After the data sequences corresponding to all data indicators are divided into multiple categories, Min-Max normalization processing can be performed on each data indicator in any category to obtain a normalized data indicator, and a scatter plot is constructed based on the normalized data indicator. Among them, Min-Max normalization is an existing technology and will not be repeated here.
[0054] In the prior art, the KNN algorithm is usually used to classify data points in the scatter plot, so as to provide anomaly warning for sewage treatment according to the classified anomaly points. However, when using KNN to classify data points in the traditional way, a single K value is used to calculate the sum or average distance of the distance from the data point to the K nearest neighboring data points, and the data points are classified according to the sum or average distance. However, in the actual process, since the sewage treatment anomaly detection process contains a large amount of real-time data and historical data, there will be some data deviation points. The data deviation points are manifested in the scatter plot of data indicators as follows: the data deviation points in the K nearest neighboring data points of the data point The distance from the point to the data point is much larger than the distance from other points to the data point, or in the data trend distribution fitting curve of the scatter plot, it is manifested as a greater degree of deviation. These data deviation points increase the sum of the distances or the average distance between the data point and the K nearest neighboring data points, resulting in the misclassification of the data point when the KNN algorithm is running. Therefore, in the example of the present invention, firstly, based on the scatter plot, the initial K value in the KNN algorithm is defined by cross-validation or the use of a validation set, and then the degree of abnormality of each data point in the scatter plot is obtained according to the initial K value, so as to adaptively obtain the optimal K value according to the degree of abnormality to improve the accuracy of data point classification.
[0055] Among them, according to the initial K value, the specific process of obtaining the abnormality degree of each data point in the scatter plot is:
[0056] (1) For any data point in the scatter plot, respectively calculate the Euclidean distance between the data point and each other data point, sort all the Euclidean distances from small to large, and take the other data points corresponding to the first K Euclidean distances among the sorted Euclidean distances as the nearest neighbor data points of the data point;
[0057] In one embodiment, for any data point in the scatter plot, the data point is used as a sample point, and the Euclidean distance between the sample point and each data point in the scatter plot except the sample point is calculated respectively, and all Euclidean distances are arranged in ascending order, and the data points corresponding to the first K Euclidean distances in the sorted Euclidean distances are taken as the nearest neighbor data points of the sample point, wherein the Euclidean distance is a prior art and will not be described in detail here.
[0058] (2) obtaining a deviation coefficient of each of the nearest neighbor data points according to the distribution characteristics of each of the nearest neighbor data points, and optimizing the Euclidean distance between the data point and each of the nearest neighbor data points using the deviation coefficient of each of the nearest neighbor data points to obtain a characteristic distance between the data point and each of the nearest neighbor data points;
[0059] Wherein, obtaining the deviation coefficient of each of the nearest neighbor data points according to the distribution characteristics of each of the nearest neighbor data points includes:
[0060] Performing curve fitting on all data points in the scatter plot to obtain a data trend distribution fitting curve;
[0061] For any nearest neighbor data point, obtain the straight line between the data point and the nearest neighbor data point, obtain the intersection of the straight line and the data trend distribution fitting curve, obtain the tangent of the data trend distribution fitting curve at the intersection, obtain the sine value of the minimum angle between the tangent and the straight line, and add the sine value to the constant 1 to obtain the deviation coefficient of the nearest neighbor data point.
[0062] In one embodiment, all data points in the scatter plot are curve-fitted using a triangular kernel function to obtain a data trend distribution fitting curve, wherein the triangular kernel function is a prior art and will not be described in detail herein.
[0063] For the i-th nearest neighbor data point of the sample point, connect the sample point and the i-th nearest neighbor data point to obtain the straight line where the sample point and the i-th nearest neighbor data point are located, obtain the intersection of the straight line and the data trend distribution fitting curve, make a tangent line at the intersection on the data trend distribution fitting curve, obtain the minimum angle value between the tangent line and the straight line, and obtain the deviation coefficient of the i-th nearest neighbor data point according to the minimum angle value. The calculation formula of the deviation coefficient is:
[0064]
[0065] in, represents the deviation coefficient of the i-th nearest neighbor data point, It represents the minimum angle between the straight line where the i-th nearest neighbor data point is located and the tangent line, sin() represents the sine function, and 1 represents a constant.
[0066] It should be noted that The value range of is [0°, 90°], and then The value range of is [1, 2]. The closer it is to 2, the greater the deviation between the straight line between the sample point and the i-th nearest neighbor data point and the data trend distribution fitting curve, the greater the deviation coefficient of the corresponding i-th nearest neighbor data point, and the greater the possibility that the corresponding i-th nearest neighbor data point is a data deviation point.
[0067] Similarly, the deviation coefficients of the K nearest neighboring data points of the sample point are obtained.
[0068] The step of optimizing the Euclidean distance between the data point and each of the nearest neighbor data points by using the deviation coefficient of each of the nearest neighbor data points to obtain the characteristic distance between the data point and each of the nearest neighbor data points includes:
[0069] For the Euclidean distance between the data point and any nearest neighbor data point, the product of the Euclidean distance and the deviation coefficient of the nearest neighbor data point is used as the characteristic distance between the data point and the nearest neighbor data point.
[0070] In one embodiment, for the i-th nearest neighbor data point of the sample point, the Euclidean distance between the sample point and the i-th nearest neighbor data point is obtained, and the product of the Euclidean distance and the deviation coefficient of the i-th nearest neighbor data point is used as the characteristic distance between the sample point and the i-th nearest neighbor data point. Then, the calculation formula between the characteristic distance between the sample point and the i-th nearest neighbor data point is:
[0071]
[0072] in, Represents the characteristic distance between the i-th nearest neighbor data point and the sample point among the K nearest neighbor data points of the sample point, Represents the deviation coefficient of the i-th nearest neighbor data point among the K nearest neighbor data points of the sample point, Represents the Euclidean distance between the i-th nearest neighbor data point and the sample point among the K nearest neighbor data points of the sample point.
[0073] Similarly, the characteristic distances between the sample point and the K nearest neighbor data points are obtained respectively.
[0074] (3) Obtaining the abnormality degree of the data point based on the characteristic distance and the Euclidean distance between the data point and each of the nearest neighboring data points.
[0075] Specifically, a maximum value of the characteristic distances between the data point and each of the nearest neighboring data points is obtained, the characteristic distances between the data point and each of the nearest neighboring data points are accumulated to obtain an accumulated result, a ratio of the maximum value to the accumulated result is calculated as a first variable, and a hyperbolic tangent function value of the first variable is used as a first function value;
[0076] Calculate the average value of the Euclidean distance between the data point and each of the nearest neighbor data points to obtain an average distance, calculate the standard deviation of the Euclidean distance between the data point and each of the nearest neighbor data points, use the ratio of the standard deviation to the average distance as a second variable, and use the hyperbolic tangent function value of the second variable as a second function value;
[0077] The first function value and the second function value are weightedly summed to obtain the abnormality degree of the data point.
[0078] In one embodiment, the maximum value of the characteristic distance is obtained from the characteristic distances between the sample point and the K nearest neighbor data points, the characteristic distances between the sample point and each nearest neighbor data point are accumulated to obtain an accumulated result, the ratio of the maximum value to the accumulated result is calculated as the first variable, the average value of the Euclidean distance between the sample point and each nearest neighbor data point is calculated to obtain the average distance, the standard deviation of the Euclidean distance between the sample point and each nearest neighbor data point is calculated, the ratio of the standard deviation to the average distance is used as the second variable, the abnormality degree of the sample point is obtained according to the first variable and the second variable, and the calculation formula of the abnormality degree of the sample point is:
[0079]
[0080] Among them, γ represents the abnormality of the sample point, represents the first weight value, represents the second weight value, tanh() represents the hyperbolic tangent function, Represents the maximum value of the feature distance between the sample point and the K nearest neighbor data points, Represents the cumulative result of the characteristic distance between the sample point and each nearest neighbor data point, represents the standard deviation of the Euclidean distance between the sample point and each nearest neighbor data point, The average distance representing the Euclidean distance between the sample point and each of the nearest neighbor data points.
[0081] It should be noted that tanh is used to fix the values of the first variable and the second variable between [0,1]. The coefficient of variation of the Euclidean distance between the K nearest neighbor data points of the sample point and the sample point reflects the degree of dispersion of the Euclidean distance between the sample point and each nearest neighbor data point relative to the average distance. The larger the coefficient of variation, the higher the degree of dispersion between the sample point and each nearest neighbor data point, that is, the larger the deviation of the standard deviation of the Euclidean distance between the sample point and each nearest neighbor data point from the average distance, the greater the possibility that the sample point is an outlier, and the greater the degree of abnormality of the corresponding sample point. On the contrary, if the coefficient of variation is small, it means that the standard deviation of the Euclidean distance between the sample point and each nearest neighbor data point is relatively close to the average distance, and the degree of dispersion between the sample point and each nearest neighbor data point is low, then the possibility that the sample point is an outlier is smaller, and the degree of abnormality of the corresponding sample point is smaller. It represents the ratio of the maximum value of the characteristic distance between the sample point and the K nearest neighbor data points to the sum of all characteristic distances. The larger the ratio, the larger the characteristic distance corresponding to the numerator. There is a large difference between the corresponding K characteristic distances, which means that the probability of data deviation points appearing in the K nearest neighbor data points is greater, and the corresponding sample point has a greater degree of abnormality. is 0.7, set is 0.3, and there is no restriction here. The implementer can set it according to the implementation scenario, and the value range of γ is (0, 1). When γ is closer to 1, it means that there are K nearest neighbor data points of the sample point, and the Euclidean distance between the data point and the sample point is larger than the Euclidean distance between the sample point and other nearest neighbor data points, and the straight line between the sample point and each nearest neighbor data point deviates more from the data trend distribution fitting curve, and the corresponding sample point is more likely to be an outlier.
[0082] Similarly, obtain the abnormality degree of all data points in the scatter plot.
[0083] The parameter acquisition module 13 is used to acquire abnormal points according to the abnormality levels of all data points in the scatter plot, and adaptively acquire the optimal K value for each data point in the scatter plot according to the abnormal points and the initial K value.
[0084] For any data point in the scatter plot, the greater the abnormality of the data point, the greater the possibility that the data point is an abnormal point when running the KNN algorithm. Therefore, in the embodiment of the present invention, the abnormal point is obtained according to the abnormality of all data points in the scatter plot, specifically:
[0085] For any abnormality level, if the abnormality level is within a preset abnormality level range, the data point corresponding to the abnormality level is marked as an abnormal point.
[0086] In one embodiment, the preset abnormality degree range is set to (0.7, 1), which is not limited here. The implementer can set it according to the implementation scenario. For any abnormality degree, if the abnormality degree is within (0.7, 1), the data point corresponding to the abnormality degree is marked as an abnormal point. Similarly, all abnormal points in the scatter plot are obtained.
[0087] In the defined initial K value, there may be data deviation points among the K nearest neighboring data points of any data point in the scatter plot. Affected by the data deviation points, the sum of the distances between the K nearest neighboring data points of the data point and the data point or the average distance will increase, resulting in the data point being misclassified. That is, there is a high probability that there are data deviation points among the K nearest neighboring data points corresponding to the data points with an abnormal degree within (0.7, 1). Therefore, it is necessary to adjust the initial K value according to each abnormal point in the scatter plot, and adaptively obtain the optimal K value for each data point in the scatter plot, so as to eliminate the influence of the data deviation points on the sum of the distances between the K nearest neighboring data points and the data point or the average distance, so as to improve the accuracy of data point classification.
[0088] The optimal K value for each data point in the scatter plot is adaptively obtained, including:
[0089] For any abnormal point in the scatter plot, the initial K value is reduced and adjusted using a preset adjustment value to obtain an adjusted K value. According to the adjusted K value, the abnormality degree of the abnormal point is obtained. If the abnormality degree of the abnormal point is within the preset abnormality degree range, the adjusted K value is further reduced and adjusted using the adjustment value to obtain a new adjusted K value. The new adjusted K value is used as the adjusted K value, and the step of obtaining the abnormality degree of the abnormal point is repeated until the abnormality degree of the abnormal point is no longer within the preset abnormality degree range. The corresponding adjusted K value is used as the optimal K value of the abnormal point.
[0090] The initial K value is used as the optimal K value for each non-abnormal point in the scatter plot.
[0091] In one embodiment, the preset adjustment value is set to 1, which is not limited here. The implementer can set it according to the implementation scenario and reduce the initial K value of the abnormal point by 1 to obtain ,according to Get the abnormality degree of the abnormal point. If the abnormality degree of the abnormal point is within (0.7, 1), Continue to reduce by 1, and get ,Will As , repeat the steps of obtaining the abnormal degree of the abnormal point until the abnormal degree of the abnormal point is not within (0.7, 1), then As the optimal K value for this outlier point.
[0092] The initial K value of each normal point (ie, non-abnormal point) in the scatter plot is used as the optimal K value of each normal point, thereby obtaining the optimal K value of all data points in the scatter plot.
[0093] The anomaly detection module 14 is used to classify all data points in the scatter plot according to the optimal K value of each data point in the scatter plot using the KNN algorithm to obtain classification results corresponding to the categories, and to provide anomaly warning for sewage treatment according to the classification results corresponding to each category.
[0094] In the KNN algorithm, the optimal K value is applied to classify the data points in the scatter plot corresponding to the category to obtain the corresponding classification results, which include abnormal data and normal data. Similarly, a scatter plot of each category is constructed, and the optimal K value of all data points in each scatter plot is obtained according to each scatter plot, which is used to classify the data points of each category using the KNN algorithm to obtain the classification results. Then, according to the data indicators corresponding to the abnormal data in the classification results of each category, the abnormal type and severity are analyzed, and an alarm is issued in time to notify the relevant operators or systems, and corresponding treatment measures and emergency responses are taken, thereby completing the abnormal warning of sewage treatment. Under the optimal K value, the interference of the data deviation point on the sum of the distance between the data point and the K nearest neighbor data points or the average distance is eliminated, and the probability of the data point being misclassified by the KNN algorithm is relatively reduced, which improves the accuracy and credibility of the KNN algorithm in detecting data points.
[0095] It is worth noting that the focus of the present invention is on how to adaptively obtain the optimal K value for each data point when classifying all data points in a scatter plot using the KNN algorithm. Abnormal warning of sewage treatment based on the classification results corresponding to each category belongs to the prior art and will not be elaborated here.
[0096] In summary, the present invention provides a sewage treatment anomaly detection and early warning system based on big data analysis, including a data analysis module, which is used to respectively obtain at least two data indicators of sewage treatment within a preset time period, and obtain a data sequence corresponding to each data indicator, and divide the data sequences of all data indicators into at least one category according to the correlation between the data sequences; a data processing module, which is used to normalize all data indicators in any category to obtain normalized data indicators, construct a scatter plot of all normalized data indicators, and obtain an initial K value of the KNN algorithm when classifying all data points in the scatter plot using the KNN algorithm, and obtain the abnormality of each data point in the scatter plot according to the initial K value; a parameter acquisition module, which is used to obtain abnormal points according to the abnormality of all data points in the scatter plot, and adaptively obtain the optimal K value of each data point in the scatter plot according to the abnormal point and the initial K value; an abnormality detection module, which is used to classify all data points in the scatter plot using the KNN algorithm according to the optimal K value of each data point in the scatter plot, obtain the classification result corresponding to the category, and perform an abnormal early warning of sewage treatment according to the classification result corresponding to each category. Among them, since sewage treatment anomaly detection contains many types of data and a large amount of data, if a scatter plot is constructed directly, the scatter plot will be very crowded, making it difficult to distinguish the distribution of each data indicator, and will cover up the correlation or pattern between the data. Therefore, the data series of each data indicator are first classified, and then a scatter plot is constructed based on the classification results. Since the sewage treatment anomaly detection process contains a large amount of real-time data and historical data, there will be some data deviation points that are far away from the sample points in the K nearest neighbor data points. These data deviation points will cause great interference in the operation of the KNN algorithm, causing the sample points to be mistaken for abnormal points. Therefore, according to the distribution characteristics of each data point in the scatter plot, the degree of abnormality of each data point is obtained, and the optimal K value is adaptively obtained according to the degree of abnormality. By reducing the K value, the interference of the data deviation points is eliminated. When using the KNN algorithm for sewage treatment anomaly detection, the optimal K value is selected to analyze and identify the data, eliminating the interference of the data deviation points during the operation of the KNN algorithm, and improving the accuracy of data point detection.
[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. The sewage treatment anomaly detection and early warning system based on big data analysis is characterized by: The sewage treatment anomaly detection and early warning system based on big data analysis includes: A data analysis module, used to respectively obtain at least two data indicators of sewage treatment within a preset period, obtain a data sequence for each data indicator, and divide the data sequences of all data indicators into at least one category according to the correlation between the data sequences; A data processing module is used to normalize all data indicators in any category to obtain normalized data indicators, construct a scatter plot of all normalized data indicators, obtain an initial K value of the KNN algorithm when classifying all data points in the scatter plot using the KNN algorithm, and obtain the abnormality degree of each data point in the scatter plot according to the initial K value; A parameter acquisition module, used to acquire an abnormal point according to the abnormal degree of all data points in the scatter plot, and adaptively acquire an optimal K value for each data point in the scatter plot according to the abnormal point and the initial K value; An anomaly detection module, used to classify all data points in the scatter plot using the KNN algorithm according to the best K value of each data point in the scatter plot, obtain classification results corresponding to the categories, and perform abnormal warning of sewage treatment according to the classification results corresponding to each category; The data processing module obtains the abnormality degree of each data point in the scatter plot according to the initial K value, including: For any data point in the scatter plot, respectively calculate the Euclidean distance between the data point and each other data point, sort all the Euclidean distances from small to large, and take the other data points corresponding to the first K Euclidean distances among the sorted Euclidean distances as the nearest neighbor data points of the data point; According to the distribution characteristics of each of the nearest neighbor data points, the deviation coefficient of each of the nearest neighbor data points is obtained, and the Euclidean distance between the data point and each of the nearest neighbor data points is optimized by using the deviation coefficient of each of the nearest neighbor data points to obtain the characteristic distance between the data point and each of the nearest neighbor data points; Obtaining the abnormality degree of the data point according to the characteristic distance and the Euclidean distance between the data point and each of the nearest neighboring data points; The data processing module obtains the deviation coefficient of each of the nearest neighbor data points according to the distribution characteristics of each of the nearest neighbor data points, including: Performing curve fitting on all data points in the scatter plot to obtain a data trend distribution fitting curve; For any nearest neighbor data point, obtain the straight line between the data point and the nearest neighbor data point, obtain the intersection of the straight line and the data trend distribution fitting curve, obtain the tangent of the data trend distribution fitting curve at the intersection, obtain the sine value of the minimum angle between the tangent and the straight line, and add the sine value to the constant 1 to obtain the deviation coefficient of the nearest neighbor data point.
2. The sewage treatment anomaly detection and early warning system based on big data analysis according to claim 1 is characterized in that: The data analysis module divides the data sequences of all data indicators into at least one category according to the correlation between the data sequences, including: For any two data sequences, the Spearman correlation coefficient between the two data sequences is calculated, and the absolute value of the Spearman correlation coefficient is used as the trend change characteristic value of the two data sequences; If the trend change characteristic value is within a preset range, the two data series are classified into the same category, and the data series of all data indicators are divided into at least one category according to the trend change characteristic value between every two data series.
3. The sewage treatment anomaly detection and early warning system based on big data analysis according to claim 1 is characterized in that: The data processing module optimizes the Euclidean distance between the data point and each of the nearest neighbor data points by using the deviation coefficient of each of the nearest neighbor data points to obtain the characteristic distance between the data point and each of the nearest neighbor data points, including: For the Euclidean distance between the data point and any nearest neighbor data point, the product of the Euclidean distance and the deviation coefficient of the nearest neighbor data point is used as the characteristic distance between the data point and the nearest neighbor data point.
4. The sewage treatment anomaly detection and early warning system based on big data analysis according to claim 1 is characterized in that: The data processing module obtains the abnormality degree of the data point according to the characteristic distance and the Euclidean distance between the data point and each of the nearest neighboring data points, including: Obtaining a maximum value of characteristic distances between the data point and each of the nearest neighboring data points, accumulating the characteristic distances between the data point and each of the nearest neighboring data points to obtain an accumulation result, calculating a ratio of the maximum value to the accumulation result as a first variable, and taking a hyperbolic tangent function value of the first variable as a first function value; Calculate the average value of the Euclidean distance between the data point and each of the nearest neighbor data points to obtain an average distance, calculate the standard deviation of the Euclidean distance between the data point and each of the nearest neighbor data points, use the ratio of the standard deviation to the average distance as a second variable, and use the hyperbolic tangent function value of the second variable as a second function value; The first function value and the second function value are weightedly summed to obtain the abnormality degree of the data point.
5. The sewage treatment anomaly detection and early warning system based on big data analysis according to claim 1 is characterized in that: The parameter acquisition module acquires abnormal points according to the abnormality degree of all data points in the scatter plot, including: For any abnormality level, if the abnormality level is within a preset abnormality level range, the data point corresponding to the abnormality level is marked as an abnormal point.
6. The sewage treatment anomaly detection and early warning system based on big data analysis according to claim 1 is characterized in that: The parameter acquisition module adaptively acquires the optimal K value for each data point in the scatter plot according to the abnormal point and the initial K value, including: For any abnormal point in the scatter plot, the initial K value is adjusted to be reduced by using a preset adjustment value to obtain an adjusted K value, and the abnormality degree of the abnormal point is obtained according to the adjusted K value. If the abnormality degree of the abnormal point is within the preset abnormality degree range, the adjusted K value is further adjusted to be reduced by using the adjustment value to obtain a new adjusted K value, and the new adjusted K value is used as the adjusted K value, and the step of obtaining the abnormality degree of the abnormal point is repeated until the abnormality degree of the abnormal point is not within the preset abnormality degree range, and the corresponding adjusted K value is used as the optimal K value of the abnormal point; The initial K value is used as the optimal K value for each non-abnormal point in the scatter plot.
Citation Information
Cited By
Sewage detection data aggregation and analysis early warning method
CN119557815A
A sewage detection data aggregation and analysis early warning method
CN119557815B
