A method for detecting abnormal data of effluent BOD concentration based on time series anomaly factor algorithm

Abstract

A method for detecting abnormal effluent BOD concentration data based on a time-series anomaly factor algorithm is being developed. Detecting abnormal effluent BOD concentration data is an important branch of advanced manufacturing technology, encompassing both the control and water treatment fields. By designing a time-series anomaly factor algorithm, this method detects abnormal effluent BOD concentration data based on data collected during the sewage treatment process. This solves the problem of difficulty in accurately detecting abnormal effluent BOD concentration data during sewage treatment, and improves the real-time monitoring of water quality in urban sewage treatment plants.

Description

Technical field:

[0001] The present invention relates to a method for detecting abnormal data of effluent BOD concentration based on a time series abnormality factor algorithm. Detecting abnormal data of effluent BOD concentration is an important branch of the field of advanced manufacturing technology, belonging to both the control field and the water treatment field. Background technology:

[0002] Wastewater treatment is an important measure to protect the environment and human health. In the sewage treatment process, abnormal data detection of effluent BOD concentration is one of the most important steps.

[0003] Biochemical oxygen demand (BOD) refers to the amount of dissolved oxygen consumed by microorganisms to decompose organic matter within a specified time period. It is an important indicator for evaluating sewage quality. Abnormal effluent BOD concentration data may indicate problems in the sewage treatment process, such as pipe blockage, sensor failure, or abnormal operating conditions. Therefore, the use of effective methods to detect it is crucial to ensure effluent quality. Currently, the commonly used methods for detecting abnormal effluent BOD concentration data include statistical methods, machine learning methods, etc., but these methods not only require a large amount of historical data for calculation, but also do not take into account the temporal nature of the time series. Therefore, how to detect abnormal effluent BOD concentration data at low cost and high efficiency is a difficult problem facing the sewage treatment process.

[0004] The Time Series Anomaly Factor algorithm, based on statistics and machine learning, analyzes the density and temporal nature of each point and calculates the Time Series Anomaly Factor, thereby identifying abnormal data in time series. This provides an efficient and rapid solution for detecting abnormal data in key water quality parameters during wastewater treatment. This paper designs a method for detecting abnormal effluent BOD concentration data based on the Time Series Anomaly Factor algorithm, achieving effective detection of abnormal effluent BOD concentration data. Summary of the invention:

[0005] The present invention has obtained a method for detecting abnormal effluent BOD concentration data based on a time series anomaly factor algorithm. The method realizes the detection of abnormal effluent BOD concentration data based on data collected during the sewage treatment process, solves the problem of the difficulty in accurately detecting abnormal effluent BOD concentration data during the sewage treatment process, and improves the level of real-time monitoring of water quality in urban sewage treatment plants.

[0006] A method for detecting abnormal data of effluent BOD concentration based on a time series abnormality factor algorithm comprises the following steps:

[0007] Step 1: Data preprocessing;

[0008] First, the effluent BOD concentration data were normalized to [0,1] according to formula (1);

[0009]

[0010] Where y is the time series data consisting of effluent BOD concentration, y(t) is the effluent BOD concentration at time t, max(y) and min(y) represent the maximum and minimum values ​​in the time series y, respectively;

[0011] Step 2: Calculate the timing anomaly factor TOF;

[0012] Step 2.1: Adaptively select the optimal cutoff distance;

[0013] In order to reduce the impact of manual iterative selection of the truncation distance, the optimal truncation distance is adaptively determined according to the set rules. The specific selection process is as follows: First, initialize the truncation distance d cs , d cs The value range is (0,1), according to d cs Calculate the density of each data point, and then calculate the threshold τ based on the average value μ and variance σ of the distance between two adjacent points in the data set. The calculation of the threshold τ is shown in formula (2):

[0014] τ=μ+ασ (2)

[0015] Among them, the value range of α is (0,1);

[0016] Calculate the maximum density point x pmax The average distance between the data points and the center of the circle with τ as the center and τ as the radius is the optimal cutoff distance d c , for the entire data set C, the cutoff distance d c The calculation formula is shown in (3):

[0017]

[0018] Step 2.2: Calculate the density of data points;

[0019] The density ρ of each data point i (i∈[1, N], N is the size of the data set) in the dataset i Defined as:

[0020]

[0021]

[0022] Among them, d ij is the Euclidean distance between data point i and data point j, d c is the optimal cutoff distance of data set C;

[0023] Step 2.3: Calculate the deviation coefficient of the data point;

[0024] In view of the temporal nature of data, data points that are adjacent in time often have greater similarity. Then, for data point i, find the point in the time neighborhood (size L i ) of the deviation, where L i The value range is an integer in the range of (2,20) and the deviation coefficient of data point i is calculated As shown in the formula:

[0025]

[0026] in, represents the average value of all data except the maximum and minimum values ​​in the time neighborhood of point i, σ i represents the variance of the data in the time neighborhood of point i;

[0027] When the deviation coefficient The smaller it is, the more stable the point is. The larger the value is, the greater the deviation of the point is, the greater the abnormality is.

[0028] Step 2.4: Define the timing anomaly factor TOF;

[0029] The density ρ for data point i i and the coefficient of deviation The definition of timing anomaly is as follows:

[0030]

[0031] When the density ρ of data point i i Relatively small, the degree of deviation If it is larger, then tod i The larger the value, the greater the degree of abnormality, and the point appears as a global abnormality; when the density ρ i Close to the normal point, but the degree of deviation When it is still relatively large, tod i The value will also be relatively large, in which case the point will show a local anomaly, otherwise it will be stable;

[0032] The ratio of the abnormality of each data point to the average abnormality of the point in the time neighborhood is defined as the time series abnormality factor TOF, and the formula is as follows:

[0033]

[0034] where i∈[1,N], Represents each data point j in the time neighborhood of data point i;

[0035] The greater the TOF value of a data point is, the greater the degree of abnormality of the data is. When the TOF value range is between 0 and 1, it is considered normal data.

[0036] Step 3: Design an anomaly detection algorithm based on TOF;

[0037] Determine the adaptive threshold TS, the formula is as follows:

[0038]

[0039]

[0040] Where Q is the set of TOF values ​​greater than 1, and the value range of β is (1,9);

[0041] When TOF i When (i∈[1, N]) is greater than TS, it indicates that the data point i is abnormal in the data set, otherwise it is normal.

[0042] Step 4: Detection of abnormal effluent BOD concentration data;

[0043] The test sample data is used as the input of the time series anomaly factor algorithm, and the TOF value of each sample data is compared with the threshold value, and finally the abnormal data of the effluent BOD concentration is obtained;

[0044] Based on the detected anomaly data, the accuracy rate (DR) and the false alarm rate (FAR) are used as evaluation indicators for the anomaly data detection of the proposed method. DR represents the proportion of actual anomaly data among the considered anomaly data, and FAR represents the proportion of data that are mistakenly considered anomaly to all normal data. The specific formula is as follows:

[0045]

[0046]

[0047] Among them, TP, FP and TN represent the number of actual abnormal data detected, the number of false abnormal data detected and the number of normal data not detected, respectively. According to the formula, the larger the DR and the smaller the FAR, the better the detection effect and the higher the detection accuracy;

[0048] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0049] (1) Aiming at the problem that abnormal data of BOD concentration, a key water quality parameter, is difficult to detect accurately in the current sewage treatment process, the present invention proposes a method for detecting abnormal data of effluent BOD concentration based on a time series anomaly factor algorithm, which has the characteristics of high detection rate, good stability and strong generalization ability.

[0050] (2) The present invention addresses the problem that traditional density-based abnormal data detection methods are unable to detect abnormal data caused by time series. By combining the characteristics of density and time series, the present invention proposes a time series anomaly factor algorithm to detect abnormal data, which can accurately detect abnormal data in time series. Description of the drawings:

[0051] Figure 1 is a sample data distribution diagram of the present invention;

[0052] Figure 2 It is a result diagram of the time series anomaly factor (TOF) of the effluent BOD concentration abnormal data detection method of the present invention; Specific implementation method:

[0053] The present invention has obtained a method for detecting abnormal effluent BOD concentration data based on a time series anomaly factor algorithm. The method realizes the detection of abnormal effluent BOD concentration data based on data collected during the sewage treatment process, solves the problem of the difficulty in accurately detecting abnormal effluent BOD concentration data during the sewage treatment process, and improves the level of real-time monitoring of water quality in urban sewage treatment plants.

[0054] The experimental data comes from the water quality analysis data of a sewage treatment plant in 2011, which contains a total of 365 effluent BOD concentration samples. All the data are selected as test samples.

[0055] A method for detecting abnormal data of effluent BOD concentration based on a time series abnormality factor algorithm comprises the following steps:

[0056] Step 1: Data preprocessing;

[0057] First, the effluent BOD concentration data were normalized to [0,1] according to formula (1);

[0058]

[0059] Where y is the time series data consisting of effluent BOD concentration, y(t) is the effluent BOD concentration at time t, max(y) and min(y) represent the maximum and minimum values ​​in the time series y, respectively;

[0060] Step 2: Calculate the timing anomaly factor TOF;

[0061] Step 2.1: Adaptively select the optimal cutoff distance;

[0062] In order to reduce the impact of manual iterative selection of the truncation distance, the optimal truncation distance is adaptively determined according to the set rules. The specific selection process is as follows: First, initialize the truncation distance d cs , d cs The value range is (0,1). In this embodiment, d csTake 0.8, according to d cs Calculate the density of each data point, and then calculate the threshold τ based on the average value μ and variance σ of the distance between two adjacent points in the data set. The calculation of the threshold τ is shown in formula (2):

[0063] τ=μ+ασ (2)

[0064] The value range of α is (0, 1). In this embodiment, α is 0.2.

[0065] Calculate the maximum density point x pmax The average distance between the data points and the center of the circle with τ as the center and τ as the radius is the optimal cutoff distance d c , for the entire data set C, the cutoff distance d c The calculation formula is shown in (3):

[0066]

[0067] Step 2.2: Calculate the density of data points;

[0068] The density ρ of each data point i (i∈[1, N], N is the size of the data set) in the dataset i Defined as:

[0069]

[0070]

[0071] Among them, d ij is the Euclidean distance between data point i and data point j, d c is the optimal cutoff distance of data set C;

[0072] Step 2.3: Calculate the deviation coefficient of the data point;

[0073] In view of the temporal nature of data, data points that are adjacent in time often have greater similarity. Then, for data point i, find the point in the time neighborhood (size L i ) of the deviation, where L i The value range is an integer in the range of (2, 20). In this embodiment, L i Take 9 and calculate the deviation coefficient of data point i As shown in the formula:

[0074]

[0075] in, represents the average value of all data except the maximum and minimum values ​​in the time neighborhood at point i, σ i represents the variance of the data in the time neighborhood of point i;

[0076] When the deviation coefficient The smaller it is, the more stable the point is. The larger the value is, the greater the deviation of the point is, the greater the abnormality is.

[0077] Step 2.4: Define the timing anomaly factor TOF;

[0078] The density ρ for data point i i and the coefficient of deviation The definition of timing anomaly is as follows:

[0079]

[0080] When the density ρ of data point i i Relatively small, the degree of deviation If it is larger, then tod i The larger the value, the greater the degree of abnormality, and the point appears as a global abnormality; when the density ρ i Close to the normal point, but the degree of deviation When it is still relatively large, tod i The value will also be relatively large, in which case the point will show a local anomaly, otherwise it will be stable;

[0081] The ratio of the abnormality of each data point to the average abnormality of the point in the time neighborhood is defined as the time series abnormality factor TOF, and the formula is as follows:

[0082]

[0083] where i∈[1,N], Represents each data point j in the time neighborhood of data point i;

[0084] The greater the TOF value of a data point is, the greater the degree of abnormality of the data is. When the TOF value range is between 0 and 1, it is considered normal data.

[0085] Step 3: Design an anomaly detection algorithm based on TOF;

[0086] Determine the adaptive threshold TS, the formula is as follows:

[0087]

[0088]

[0089] Where Q is a set of TOF values ​​greater than 1, and the value range of β is (1, 9). According to the definition of the time series outlier factor (TOF), β should be greater than 1. The growth step of β is set to 1, so that the detection accuracy is not less than 90%. Finally, in this embodiment, β is set to 5;

[0090] When TOF i When (i∈[1, N]) is greater than TS, it indicates that the data point i is abnormal in the data set, otherwise it is normal.

[0091] Step 4: Detection of abnormal effluent BOD concentration data;

[0092] The test sample data is used as the input of the time series anomaly factor algorithm, and the TOF value of each sample data is compared with the threshold value, and finally the abnormal data of the effluent BOD concentration is obtained. In this embodiment, after the sample data is input, TS=5 is calculated;

[0093] Based on the detected anomaly data, the accuracy rate (DR) and the false alarm rate (FAR) are used as evaluation indicators for the anomaly data detection of the proposed method. DR represents the proportion of actual anomaly data among the considered anomaly data, and FAR represents the proportion of data that are mistakenly considered anomaly to all normal data. The specific formula is as follows:

[0094]

[0095]

[0096] Among them, TP, FP and TN represent the number of actual abnormal data detected, the number of false abnormal data detected and the number of normal data not detected, respectively. According to the formula, the larger the DR and the smaller the FAR, the better the detection effect and the higher the detection accuracy;

[0097] In this embodiment, the test results are as follows: Figure 2 As shown, the X-axis represents sample points (unit: sample), and the Y-axis represents the TOF value. The solid line represents the TOF value for each point, and the dashed line represents the abnormal data threshold. In this experiment, DR = 100% and FAR = 0 were calculated. This result demonstrates the effectiveness of the effluent BOD concentration abnormal data detection method based on the time series anomaly factor algorithm.

[0098] Table 1 shows the BOD concentration data used in the experiment of the present invention, and Table 2 shows the abnormal data and abnormality degree set in the experiment of the present invention.

[0099] Table 1. Effluent BOD concentration data (mg / L)

[0100]

[0101]

[0102] Table 2. Abnormal data and degree of abnormality

[0103] Serial number Original data Abnormal data Abnormality 1 11.02 12.13 +10％ 2 10.85 11.94 +10％ 3 10.77 8.62 -20％ 4 11.40 12.54 +10％ 5 10.54 12.65 +20％ 6 12.24 11.02 -10％ 7 13.10 10.48 -20％ 8 14.20 9.78 -30％ 9 12.64 17.70 +40％ 10 12.45 7.47 -40％

Claims

1. A method for detecting abnormal data of effluent BOD concentration based on a time series abnormality factor algorithm, characterized in that: The following steps are involved: Step 1: Data preprocessing; First, the effluent BOD concentration data were normalized to [0,1] according to formula (1); Where y is the time series data consisting of effluent BOD concentration, y(t) is the effluent BOD concentration at time t, max(y) and min(y) represent the maximum and minimum values ​​in the time series y, respectively; Step 2: Calculate the timing anomaly factor TOF; Step 2.1: Adaptively select the optimal cutoff distance; First initialize the cutoff distance d cs , d cs The value range is (0,1), according to d cs Calculate the density of each data point, and then calculate the threshold τ based on the average value μ and variance σ of the distance between two adjacent points in the data set. The calculation of the threshold τ is shown in formula (2): τ=μ+ασ (2) Among them, the value range of α is (0,1); Calculate the maximum density point x pmax The average distance between the data points and the center of the circle with τ as the center and τ as the radius is the optimal cutoff distance d c , for the entire data set C, the cutoff distance d c The calculation formula is shown in (3): Step 2.2: Calculate the density of data points; The density ρ of each data point i (i∈[1, N], N is the size of the data set) in the dataset i Defined as: Among them, d ij is the Euclidean distance between data point i and data point j, d c is the optimal cutoff distance of data set C; Step 2.3: Calculate the deviation coefficient of the data point; In view of the temporal nature of data, data points that are adjacent in time often have greater similarity. Then, for data point i, the degree of deviation of the point in the time neighborhood is calculated. The size of the time neighborhood is L. i , where L i The value range is an integer in the range of (2,20) and the deviation coefficient of data point i is calculated As shown in the formula: in, represents the average value of all data except the maximum and minimum values ​​in the time neighborhood at point i, σ i represents the variance of the data in the time neighborhood of point i; When the deviation coefficient The smaller it is, the more stable the point is. The larger the value is, the greater the deviation of the point is, the greater the abnormality is. Step 2.4: Define the timing anomaly factor TOF; The density ρ for data point i i and the coefficient of deviation The definition of timing anomaly is as follows: When the density ρ of data point i i Relatively small, the degree of deviation If it is larger, then tod i The larger the value, the greater the degree of abnormality, and the point appears as a global abnormality; when the density ρ i Close to the normal point, but the degree of deviation When it is still relatively large, tod i The value will also be relatively large, in which case the point will show a local anomaly, otherwise it will be stable; The ratio of the abnormality of each data point to the average abnormality of the point in the time neighborhood is defined as the time series abnormality factor TOF, and the formula is as follows: where i∈[1,N], Represents each data point j in the time neighborhood of data point i; The greater the TOF value of a data point is, the greater the degree of abnormality of the data is. When the TOF value range is between 0 and 1, it is considered normal data. Step 3: Design an anomaly detection algorithm based on TOF; Determine the adaptive threshold TS, the formula is as follows: Where Q is the set of TOF values ​​greater than 1, and the value range of β is (1,9); When TOF i When it is greater than TS, it indicates that the data point i is abnormal data in the data set, otherwise it is normal data; where i∈[1, N]; Step 4: Detection of abnormal effluent BOD concentration data; The test sample data is used as the input of the time series anomaly factor algorithm, the TOF value of each sample data is compared with the threshold, and finally the abnormal data of the effluent BOD concentration is obtained.

Images