A method and system for detecting abnormal data in environmental monitoring

Abstract

The present application relates to the technical field of data processing, and in particular to a kind of environmental monitoring's metering abnormal data detection method and system, the technical problem that the accuracy of water quality index data is lower in the prior art due to neglecting periodicity and trend of water quality data, leading to abnormal data monitoring has been solved.This method comprises: obtaining water quality data, and decomposing water quality data into periodic sequence, trend sequence and residual sequence by a preset timing decomposition algorithm;The similarity of the periodic change law between multiple segments in the periodic sequence is compared, and the segment with a similarity less than a similarity threshold is marked as a suspected abnormal segment;In the time range corresponding to the suspected abnormal segment, the volatility measure of the data in the trend sequence is corrected according to the change trend of the trend sequence, and the abnormal time is detected according to the corrected volatility measure;According to the residual sequence, the abnormal time is verified to determine the real abnormality or false abnormality.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and system for detecting abnormal measurement data in environmental monitoring. Background Technology

[0002] Rivers, as the lifeblood of the earth, play a vital role in human life and the maintenance of the ecological environment. However, in recent years, river pollution problems have remained prominent. Water quality monitoring can not only effectively prevent and reduce water pollution accidents, but also significantly improve the efficiency of water resource management and utilization.

[0003] During water quality monitoring, water quality data do not exhibit random fluctuations. Instead, they are influenced by a combination of factors, including climate, geographical location, and human activities, resulting in significant seasonal and periodic variations, as well as long-term trends. These periodicities and trends together constitute important structural characteristics of water quality data.

[0004] However, existing statistical outlier detection methods often overlook the overall trend of water quality data and fail to fully consider the inherent periodicity and trend characteristics of water quality data. This makes them susceptible to interference from structural fluctuations in the data (such as seasonal cycles and long-term trends) during actual monitoring. For example, after obtaining residual data through seasonal and trend decomposition using loess (STL) algorithms, directly applying statistical outlier detection methods to the residual data often misjudges normal structural fluctuations as random fluctuations (such as sudden sewage discharge), thus incorrectly identifying them as outliers. Ultimately, this results in low accuracy in monitoring water quality indicators and makes it difficult to meet the precise requirements of actual water quality monitoring. Summary of the Invention

[0005] To address the problem of low accuracy in monitoring water quality indicators due to the neglect of the periodicity and trends in water quality data in existing technologies, the present invention aims to provide a method and system for detecting metrological anomalies in environmental monitoring. The specific technical solution adopted is as follows:

[0006] Firstly, a method for detecting measurement anomalies in environmental monitoring is provided, comprising: acquiring water quality data and decomposing the water quality data into a periodic sequence, a trend sequence, and a residual sequence using a preset time-series decomposition algorithm; the periodic sequence is used to characterize the periodic changes in water quality data; the trend sequence is used to characterize the long-term trend of water quality data; and the residual sequence is used to characterize the random fluctuations of water quality data; comparing the similarity of the periodic change patterns among multiple segments in the periodic sequence, and marking segments with similarity less than a similarity threshold as suspected anomaly segments; within the time range corresponding to the suspected anomaly segments, correcting the volatility measure of the data within the trend sequence according to the changing trend of the trend sequence, and detecting anomaly moments based on the corrected volatility measure; anomaly moments are used to indicate when the data values ​​in the trend sequence deviate from the normal range; and verifying the anomaly moments based on the residual sequence to determine whether they are real or false anomalies.

[0007] Based on the above technical solution, in the method for detecting measurement anomalies in environmental monitoring provided by this invention, water quality data is first acquired and separated into a periodic sequence representing the periodic changes in water quality, a trend sequence representing the long-term trend, and a residual sequence representing random fluctuations using a preset time-series decomposition algorithm, effectively capturing the structural characteristics of the water quality data. Then, by comparing the similarity between each segment in the periodic sequence and the overall periodic pattern, suspected anomaly segments are marked, narrowing the scope of subsequent analysis and reducing invalid calculations to improve monitoring efficiency. Next, within the time range of the suspected anomaly segments, the deviation of the data is corrected by combining the trend sequence change trend to avoid misjudging normal trend fluctuations and accurately locate the anomaly moment. Finally, the anomaly moment is verified by the residual sequence, filtering out false anomalies caused by system errors such as sensor drift. Ultimately, this achieves efficient and accurate identification of water quality measurement anomalies, significantly improving the accuracy and reliability of anomaly data monitoring in environmental monitoring.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the method of comparing the similarity of periodic change patterns among multiple segments in a periodic sequence and marking segments with similarity less than a similarity threshold as suspected anomalous segments specifically includes: performing curve fitting on the periodic sequence to obtain a periodic curve, and dividing the periodic curve into multiple segments based on the extreme points of the periodic curve; extracting multiple statistical features for each segment and constructing a feature vector; using statistical features to describe the morphology and distribution characteristics of the segments; determining the similarity between the feature vector of each segment and the feature vectors of other segments using a preset similarity measurement method; and marking segments with similarity less than a similarity threshold as suspected anomalous segments.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the method for extracting multiple statistical features for each segment specifically includes: extracting multiple feature data of the periodic curve corresponding to the segment; the feature data includes variance, mean curvature, kurtosis, mean, maximum value, and minimum value; and normalizing each feature data to obtain multiple statistical features.

[0010] In conjunction with the first aspect above, in one possible implementation, the method of correcting the volatility measure of data within a trend sequence based on the changing trend of the trend sequence within the time range corresponding to the suspected anomaly segment, and detecting anomaly moments based on the corrected volatility measure, specifically includes: dividing the data in the trend sequence corresponding to the suspected anomaly segment into multiple window data by a preset window size; performing linear fitting on each window to obtain the trend slope of the window; the trend slope is used to characterize the direction and magnitude of change of data within the window; determining the relative deviation between windows based on the trend slopes of multiple windows; the relative deviation is used to characterize the volatility level of the window slope change; identifying anomaly windows based on the relative deviation, and determining anomaly moments from the anomaly windows.

[0011] In conjunction with the first aspect above, in one possible implementation, the method for determining the relative deviation between windows based on the trend slope of multiple windows specifically includes: determining the overall deviation based on the trend slope of multiple windows; determining the absolute deviation of each window based on the trend slope of each window and the trend slope of the previous adjacent window; and correcting the absolute deviation based on the overall deviation to obtain the relative deviation of each window.

[0012] In conjunction with the first aspect above, in one possible implementation, the method for identifying abnormal windows based on the degree of relative deviation specifically includes: determining the overall relative deviation degree based on the relative deviation degrees of multiple windows; correcting a basic threshold for the distribution range of the relative deviation degree based on the overall relative deviation degree; and marking windows with a relative deviation degree greater than the corrected threshold as abnormal windows.

[0013] In conjunction with the first aspect mentioned above, in one possible implementation, the method for verifying abnormal moments based on the residual sequence to determine whether they are real or false anomalies specifically includes: extracting a preset number of residual data before and after the abnormal moment as a verification window, and determining the fluctuation range of the residual data within the verification window using a preset statistical method; if the residual value within the verification window exceeds the fluctuation range, the abnormal moment is determined to be a real anomaly; if the residual value within the verification window is within the fluctuation range, the abnormal moment is determined to be a false anomaly.

[0014] In conjunction with the first aspect above, in one possible implementation, the method for obtaining water quality data specifically includes: collecting raw data through deployed sensors; the raw data includes at least one of pH, dissolved oxygen content, turbidity, temperature, ammonia nitrogen content, and permanganate index; and normalizing the raw data to obtain water quality data.

[0015] In conjunction with the first aspect mentioned above, in one possible implementation, the method of decomposing water quality data into periodic sequences, trend sequences, and residual sequences using a preset time-series decomposition algorithm specifically includes: sorting water quality data by time to form a continuous time series; performing weighted fitting within a sliding window on the time series and extracting trend components reflecting the long-term direction of water quality changes to obtain a trend sequence; performing periodic pattern recognition and fitting on the intermediate data after removing the trend components from the time series to extract periodic components exhibiting fixed-period repetition characteristics to obtain a periodic sequence; and determining the remaining components of the time series after removing the trend and periodic components as the residual sequence.

[0016] Secondly, an environmental monitoring measurement anomaly data detection system is provided, comprising: a data preprocessing module for acquiring water quality data and decomposing the water quality data into periodic sequences, trend sequences, and residual sequences using a preset time-series decomposition algorithm; the periodic sequence characterizes the periodic changes in water quality data; the trend sequence characterizes the long-term trend of water quality data; and the residual sequence characterizes the random fluctuations in water quality data; a periodic sequence analysis module for comparing the similarity of periodic change patterns among multiple segments in the periodic sequence and marking segments with similarity less than a similarity threshold as suspected anomaly segments; a trend sequence analysis module for correcting the volatility measurement of data within the trend sequence based on the changing trend of the trend sequence within the time range corresponding to the suspected anomaly segments, and detecting abnormal moments based on the corrected volatility measurement; abnormal moments indicate the moments when data values ​​in the trend sequence deviate from the normal range; and a residual sequence verification module for verifying abnormal moments based on the residual sequence to determine whether they are real or false anomalies.

[0017] Thirdly, an environmental monitoring measurement anomaly data detection device is provided, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to perform the actions described in the first aspect and any possible implementation thereof. This environmental monitoring measurement anomaly data detection device may be an electronic device or a chip within an electronic device.

[0018] Fourthly, a computer-readable storage medium is provided, in which instructions are stored, which, when executed on an environmental monitoring metering anomaly data detection device, cause the environmental monitoring metering anomaly data detection device to perform the actions described in the first aspect and any possible implementation thereof.

[0019] Fifthly, a computer program product containing instructions is provided, which, when run on an environmental monitoring metering anomaly data detection device, causes the environmental monitoring metering anomaly data detection device to perform the actions described in the first aspect and any possible implementation thereof.

[0020] The present invention has the following beneficial effects:

[0021] By first acquiring water quality data and then using a pre-defined time-series decomposition algorithm to separate periodic sequences representing periodic changes in water quality, trend sequences representing long-term trends, and residual sequences representing random fluctuations, the structural characteristics of the water quality data are effectively captured. Next, by comparing the similarity between each segment of the periodic sequence and the overall periodic pattern, suspected anomalous segments are marked, narrowing the scope of subsequent analysis and reducing unnecessary calculations to improve monitoring efficiency. Then, within the time range of suspected anomalous segments, the deviation of the data is corrected by combining the trend sequence changes to avoid misjudging normal trend fluctuations and accurately locate the anomalous moments. Finally, the anomalous moments are verified using the residual sequences, filtering out false anomalies caused by systematic errors such as sensor drift. Ultimately, this achieves efficient and accurate identification of anomalous water quality measurement data, significantly improving the accuracy and reliability of anomalous data monitoring in environmental monitoring. Attached Figure Description

[0022] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A system structure diagram of an environmental monitoring metrological anomaly data detection system provided in one embodiment of the present invention;

[0024] Figure 2 A flowchart of a method for detecting abnormal measurement data in environmental monitoring, provided in one embodiment of the present invention;

[0025] Figure 3 A flowchart illustrating another method for detecting abnormal measurement data in environmental monitoring, provided as an embodiment of the present invention;

[0026] Figure 4 A flowchart illustrating another method for detecting abnormal measurement data in environmental monitoring, provided as an embodiment of the present invention;

[0027] Figure 5A flowchart illustrating another method for detecting abnormal measurement data in environmental monitoring, provided as an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of the hardware structure of an environmental monitoring metering anomaly data detection device provided in one embodiment of the present invention. Detailed Implementation

[0029] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a method and system for detecting metrological anomalies in environmental monitoring according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] The following description, in conjunction with the accompanying drawings, details the specific scheme of the method and system for detecting abnormal measurement data in environmental monitoring provided by the present invention.

[0032] Please see Figure 1 The diagram illustrates a system structure of an environmental monitoring metrological anomaly data detection system according to an embodiment of the present invention. The environmental monitoring metrological anomaly data detection system includes: a data preprocessing module 1, a periodic sequence analysis module 2, a trend sequence analysis module 3, and a residual sequence verification module 4.

[0033] In some implementations, the environmental monitoring measurement anomaly data detection system also includes: a data storage module 5 and an anomaly alarm module 6.

[0034] Among them, the data preprocessing module 1 is the basic data support unit of the system, responsible for completing the collection, standardization processing and structural decomposition of water quality data.

[0035] In some implementations, the data preprocessing module 1 includes a real-time data acquisition unit 11, a normalization processing unit 12, and a sequence decomposition unit 13.

[0036] The data acquisition unit 11 collects raw water quality data (including pH, dissolved oxygen, turbidity, temperature, ammonia nitrogen, and permanganate index) in real time at a preset frequency (e.g., once every 10 minutes) by deploying multiple types of sensors (including pH, dissolved oxygen, turbidity, temperature, ammonia nitrogen, and permanganate index) at monitoring stations. At the same time, it records the timestamp corresponding to each set of raw data to ensure the temporal correlation of the data. After the collection is completed, the raw data and timestamps are packaged and transmitted to the normalization processing unit 12.

[0037] The normalization processing unit 12 receives the raw data transmitted by the data acquisition unit 11. Due to the dimensional differences of different water quality indicators (such as temperature in °C, ammonia nitrogen content in mg / L, and pH being dimensionless), the raw data is standardized using a preset normalization algorithm to eliminate the interference of dimensional and value range differences on subsequent analysis and obtain water quality data of a uniform scale. The processed water quality data will be transmitted to the sequence decomposition unit 13.

[0038] The sequence decomposition unit 13 receives the water quality data output by the normalization processing unit 12, first arranges the water quality data in time stamp order to form a continuous water quality time series, and then uses a preset time series decomposition algorithm to structurally decompose the water quality time series to obtain a trend series, a periodic series, and a residual series. After the decomposition is completed, the periodic series is transmitted to the periodic series analysis module 2, the trend series is transmitted to the trend series analysis module 3, and the residual series is transmitted to the residual series verification module 4. At the same time, all data is temporarily stored in the data storage module 5.

[0039] The periodic sequence analysis module 2 is the coarse screening unit of the system. It is responsible for performing in-depth analysis on the periodic sequence output by the data preprocessing module 1 to locate suspected abnormal segments that do not conform to the overall periodic pattern.

[0040] In some implementations, the periodic sequence analysis module 2 includes a curve fitting unit 21, a feature construction unit 22, and an anomaly labeling unit 23.

[0041] The curve fitting unit 21 receives the periodic sequence transmitted by the data preprocessing module 1, performs curve fitting on the periodic sequence, and obtains a periodic curve that reflects the overall fluctuation pattern of the periodic sequence. Then, it calculates the minimum point of the periodic curve and divides the periodic curve into multiple independent segments, each segment corresponding to a continuous periodic data segment. The segmented data is then transmitted to the feature construction unit 22.

[0042] The feature construction unit 22 receives segmented data output from the curve fitting unit 21 and calculates core feature data for each segment (including variance characterizing the segment's volatility, average curvature characterizing the curve's curvature, kurtosis characterizing the probability of extreme values, mean value characterizing the overall level of the segmented data, and the segment's maximum and minimum values). To eliminate dimensional differences between feature data, each feature data is normalized again to obtain standardized statistical features. Finally, the statistical features of each segment are combined in a preset order to construct segmented feature vectors, and the feature vectors of all segments are transmitted to the anomaly labeling unit 23.

[0043] The anomaly marking unit 23 receives the segmented feature vectors output by the feature construction unit 22, and uses a preset similarity measurement algorithm (such as cosine similarity algorithm) to calculate the similarity between the feature vector of each segment and the feature vectors of all other segments one by one. The average of all similarities is taken as the overall similarity of the segment. The overall similarity is compared with a preset similarity threshold (such as 0.7 determined based on historical normal water quality data). If the overall similarity of a segment is less than the threshold, the segment is determined to be a suspected anomaly segment. After marking is completed, the suspected anomaly segments and their corresponding time range information are transmitted to the trend sequence analysis module 3, and the marking results are stored in the data storage module 5.

[0044] The trend sequence analysis module 3 is the system's fine positioning unit. Based on the suspected abnormal segments output by the periodic sequence analysis module 2, it performs targeted analysis on the trend sequence output by the data preprocessing module 1, corrects data fluctuation interference, and accurately locates abnormal moments.

[0045] In some implementations, the trend sequence analysis module 3 includes a segmentation filtering unit 31, a trend analysis unit 32, a deviation correction unit 33, and an anomaly time location unit 34.

[0046] The segmentation filtering unit 31 receives the suspected anomaly segments and their corresponding time ranges output by the periodic sequence analysis module 2. Combined with the trend sequence transmitted by the data preprocessing module 1, it filters the trend sequence according to the time dimension, extracting trend data that perfectly matches the time range of the suspected anomaly segments to form suspected anomaly trend segments. The filtered suspected anomaly trend segments are then transmitted to the trend analysis unit 32.

[0047] The trend analysis unit 32 receives the suspected abnormal trend segments output by the segmentation and filtering unit 31, and first divides the suspected abnormal trend segments into sliding windows to obtain multiple continuous window data. For each window data, the trend slope of each window is calculated. The calculated trend slopes of all windows are transmitted to the deviation correction unit 33.

[0048] Deviation correction unit 33 receives the window trend slope output by trend analysis unit 32, first calculates the overall deviation based on the trend slopes of all windows, then calculates the absolute deviation of each window from the previous adjacent window, and normalizes and corrects the absolute deviation based on the overall deviation to obtain the relative deviation characterizing the trend difference between a single window and its adjacent windows. Subsequently, the mean of the relative deviations of all windows is used to determine the overall relative deviation. Based on the overall relative deviation, the original standard deviation (characterizing the uncorrected original fluctuation level) of the suspected abnormal trend segments is corrected to obtain a corrected standard deviation that reflects the true fluctuation of water quality. The corrected standard deviation and the suspected abnormal trend segment data are then transmitted to abnormal time location unit 34.

[0049] The abnormal moment location unit 34 receives the corrected standard deviation and suspected abnormal trend segment data output by the deviation correction unit 33, and identifies outliers in the suspected abnormal trend segment data. At the same time, it retrieves the timestamp bound to the trend sequence from the data storage module 5, transmits the abnormal moment information to the residual sequence verification module 4, and stores the abnormal moment and the corresponding outlier in the data storage module 5.

[0050] The residual sequence verification module 4 is the system's anti-spurious unit. Based on the residual sequence output by the data preprocessing module 1 and the abnormal moments output by the trend sequence analysis module 3, it verifies the authenticity of the abnormal moments and filters out false anomalies.

[0051] In some implementations, the residual sequence verification module 4 includes a verification window construction unit 41, a fluctuation range determination unit 42, and a true anomaly determination unit 43.

[0052] The verification window construction unit 41 receives the abnormal moments output by the trend sequence analysis module 3 and the residual sequences output by the data preprocessing module 1. For each abnormal moment, it extracts a preset number of residual data (e.g., 144, corresponding to 24 hours of data collection) before and after that moment to form a verification window that can cover the complete fluctuation situation around the abnormal moment. The constructed verification window will be transmitted to the fluctuation range determination unit 42.

[0053] The fluctuation range determination unit 42 receives the verification window output by the verification window construction unit 41, analyzes the residual data within the verification window, and forms the normal fluctuation range of the residual data. The determined fluctuation range and the residual data within the verification window are then transmitted to the real anomaly determination unit 43.

[0054] The real anomaly determination unit 43 receives the fluctuation range output by the fluctuation range determination unit 42 and the residual data within the verification window, and compares the relationship between the residual data and the fluctuation range. If the residual data within the verification window exceeds the fluctuation range, it indicates that the trend anomaly at the abnormal moment is caused by a real water quality problem, and the abnormal moment is determined to be a real anomaly. If all the residual data within the verification window are within the fluctuation range, it indicates that the trend anomaly is caused by system errors such as sensor zero-point drift, and the abnormal moment is determined to be a false anomaly. After the determination is completed, the real anomaly result (including the abnormal moment, corresponding water quality index, and anomaly degree) is transmitted to the anomaly alarm module 6, and the determination results of real anomalies and false anomalies and the complete analysis process are stored in the data storage module 5.

[0055] Data storage module 5 is the system's data hub, responsible for receiving and storing raw data, intermediate data, and final results transmitted from each module. Specifically, this includes: raw data, normalized water quality data, periodic sequences, trend sequences, and residual sequences output by data preprocessing module 1; suspected anomaly segments and time ranges output by periodic sequence analysis module 2; window trend slope, corrected standard deviation, and anomaly timestamps output by trend sequence analysis module 3; and verification window data, fluctuation range, and true / false anomaly determination results output by residual sequence verification module 4. Furthermore, data storage module 5 supports each module in retrieving required historical data during operation (such as timestamps retrieved by anomaly time location unit 34 and intermediate calculation results retrieved by true anomaly determination unit 43), providing support for stable system operation and subsequent data traceability.

[0056] The anomaly alarm module 6 is the system's result output unit. It is responsible for receiving the real anomaly results output by the residual sequence verification module 4, processing the real anomaly information (including the time of anomaly occurrence, the location of the monitoring station, the name of the abnormal water quality indicator, the deviation of the abnormal value from the normal range, and a preliminary judgment of the anomaly type), and promptly pushing the real anomaly information through preset alarm methods (such as sending SMS alarms to the monitoring personnel's terminals, generating pop-up alarms on the environmental monitoring platform, and triggering on-site audible and visual alarms). This ensures that monitoring personnel can quickly be aware of and intervene in the handling of the anomaly, achieving seamless integration between anomaly data monitoring and emergency response.

[0057] Please see Figure 2 The diagram illustrates a flowchart of a method for detecting abnormal measurement data in environmental monitoring according to an embodiment of the present invention. This method includes:

[0058] S1. Acquire water quality data and decompose the water quality data into periodic series, trend series and residual series through a preset time series decomposition algorithm.

[0059] Among them, periodic sequences are used to characterize the periodic changes in water quality data, trend sequences are used to characterize the long-term trends in water quality data, and residual sequences are used to characterize the random fluctuations in water quality data.

[0060] In some implementations, the method for acquiring water quality data may specifically include: collecting raw data through deployed sensors. The raw data includes at least one of the following: turbidity (characterizing the content of suspended particles in the water), ammonia nitrogen content (characterizing the degree of nitrogen pollution in the water), water temperature (characterizing the temperature state of the water), permanganate index (characterizing the content of oxidizable organic and inorganic matter in the water), pH value (characterizing the acidity or alkalinity of the water), and dissolved oxygen content (characterizing the concentration of dissolved oxygen in the water).

[0061] The raw data underwent a unified format conversion and normalization process to obtain water quality data at multiple time points. First, the collected real-time raw data and the queried historical raw data were converted to a unified format. Heterogeneous formats, such as binary data output from different sensors and JavaScript object notation (JSON) data stored in the database, were all converted to CSV format files. The converted files must contain nine fields: date, time, sampling location number, turbidity, ammonia nitrogen content, water temperature, permanganate index, pH, and dissolved oxygen content, ensuring that each data point accurately corresponds to a specific time, location, and parameter. Then, the six water quality data items in the CSV file, excluding date, time, and sampling location number, were normalized. For example, a min-max normalization method was used, mapping parameter values ​​to the [0, 1] interval using a formula to eliminate the interference of different parameter unit differences (such as water temperature in degrees Celsius and turbidity in NTU) on subsequent data calculations, ultimately yielding standardized water quality data.

[0062] In some implementations, methods for decomposing water quality data using a pre-defined time-series decomposition algorithm (such as the STL algorithm) may specifically include:

[0063] First, the water quality data is sorted by time to form a continuous time series.

[0064] Then, a weighted fitting within a sliding window is performed on the time series, and the trend component reflecting the long-term direction of water quality changes is extracted to obtain the trend sequence. Specifically, the sliding window size is first determined based on the water quality data collection frequency (e.g., once every 10 minutes, for a total of 144 data points in 24 hours), selecting a window that covers 24 hours of data (i.e., the window contains 144 data points). This window size can cover short-term fluctuations within a single day while avoiding discontinuous trend extraction due to an excessively small window. Subsequently, local weighted regression fitting is performed on the water quality data within each sliding window (data closer to the center of the window is given higher weights to reduce the interference of outliers on the fitting results), obtaining the local fitting curve for each window. The local fitting curves of all windows are concatenated in chronological order to form an overall fitting curve covering the entire water quality time series; this curve is the trend component of the water quality data. Finally, this trend component is output as an independent sequence, denoted as the trend sequence. Trend sequences can clearly characterize the long-term changing trends of water quality indicators, such as the gradual increase in ammonia nitrogen content at a certain monitoring section from 0.5 mg / L to 1.2 mg / L within one month.

[0065] Next, periodic pattern recognition and fitting are performed on the intermediate data after removing the trend component from the time series to extract periodic components exhibiting fixed-period repetition characteristics, thus obtaining the periodic series. Specifically, the difference between the water quality time series and the trend series is first calculated, i.e., the intermediate data after eliminating the influence of long-term trends is obtained by subtracting the trend series value at the corresponding moment from the water quality time series value. The intermediate data only contains the periodic and random fluctuations of water quality data, allowing for focused analysis of the periodic characteristics of water quality. Subsequently, combined with the water quality influencing factors in the monitoring area (such as the water consumption patterns of surrounding residents, the cycle of industrial sewage discharge, and seasonal changes in natural hydrology), an iterative fitting method is used to mine the repetitive fluctuation patterns in the intermediate data. For example, by comparing the intermediate data at the same time every day, it is found that turbidity peaks between 7:00-9:00 and 18:00-20:00 (corresponding to the increased sewage discharge caused by peak residential water consumption), thus determining that the intermediate data has a fluctuation pattern with a 24-hour cycle. Based on the identified periodic pattern, periodic fitting is performed on the intermediate data to generate a periodic fitting curve that is consistent with the period of the intermediate data and matches the fluctuation trend. This curve is the periodic component of the water quality data. The periodic component is output as an independent sequence, denoted as a periodic sequence. Periodic sequences can accurately reflect the fixed periodic repetitive characteristics of water quality indicators, such as the periodic fluctuations of temperature with seasonal changes and the regular changes of dissolved oxygen with the alternation of day and night.

[0066] Finally, the remaining components after removing trend and periodic components from the time series are determined as the residual series. Specifically, the trend series value at the corresponding time point is subtracted from the periodic series value at the corresponding time point from the water quality time series value. This yields the remaining fluctuation data after eliminating trend and periodic components. This remaining data is output as an independent sequence and denoted as the residual series. The residual series only contains the random fluctuation component of the water quality data. The main sources include sudden sewage discharge events (such as a sudden increase in water quality indicators caused by a factory's instantaneous discharge of high-concentration wastewater), small sensor errors (such as instantaneous value fluctuations caused by pH sensor zero-point drift), and occasional interference from the natural environment (such as temporary turbidity anomalies caused by heavy rain). These components cannot be explained by trend or periodic patterns and provide key evidence for subsequent verification of the authenticity of trend anomalies.

[0067] S2. Compare the similarity of the periodic change patterns among multiple segments in the periodic sequence, and mark segments with similarity less than the similarity threshold as suspected abnormal segments.

[0068] In some implementations, a periodic curve is first fitted to the periodic sequence, and the sequence is divided into multiple segments based on the curve's minimum points. Then, features such as variance, mean curvature, and kurtosis of each segment are extracted, normalized, and used to construct feature vectors. Next, the average similarity between each segment's feature vector and those of other segments is calculated. Finally, segments with an average similarity less than a preset threshold are marked as suspected anomalous segments. This method can accurately filter out segments that do not conform to the overall periodic pattern, narrowing the scope of subsequent trend sequence analysis, avoiding misjudging normal periodic fluctuations as anomalies, providing effective constraints for accurately locating anomalies, and improving the initial screening efficiency and accuracy of water quality anomaly monitoring.

[0069] S3. Within the time range corresponding to the suspected abnormal segment, correct the volatility measure of the data in the trend sequence according to the changing trend of the trend sequence, and detect abnormal moments based on the corrected volatility measure.

[0070] Among them, abnormal moments are used to indicate the times when data values ​​in a trend sequence deviate from the normal range.

[0071] In some implementations, suspected anomaly trend segments are first selected from the trend sequence based on the time range of the suspected anomaly segments. Then, sliding windows are divided according to preset rules, and the trend slope of each window is calculated to quantify local trend changes. Subsequently, the overall relative deviation is obtained by calculating the overall deviation, absolute deviation, and relative deviation, and the original standard deviation (i.e., volatility measure) of the suspected anomaly trend segments is corrected based on this value. Finally, the 3sigma (3σ) principle is used, combined with the corrected standard deviation, to detect outliers within the trend segments, and the anomaly moment is located through the mapping between the trend sequence and timestamps. This method restores the true fluctuation of water quality data by correcting the volatility measure, avoiding misjudging normal trend fluctuations as anomalies. It also focuses on the time range of suspected anomalies to reduce invalid calculations, and the accurately located anomaly moment provides a reliable basis for subsequent residual sequence verification, improving the accuracy and efficiency of water quality anomaly monitoring.

[0072] S4. Based on the residual sequence, verify the abnormal moments to determine whether they are real or false anomalies.

[0073] In some implementations, for anomaly moments, a verification window is constructed by extracting a predetermined number of residual data points before and after that moment from the residual sequence. The distribution of the residual data within the window is analyzed to determine the normal fluctuation range of the residuals. The residual data within the window is compared with the normal fluctuation range; if any residuals exceed the range, the anomaly moment is determined to be a genuine anomaly; if all residuals are within the range, it is determined to be a false anomaly. This method utilizes the random fluctuation characteristics of the residual sequence to verify anomalies, effectively filtering out false anomalies caused by system errors such as sensor drift and data transmission interference, avoiding misjudgments of non-water quality issues, significantly improving the reliability of water quality anomaly monitoring results, and providing accurate basis for subsequent alarm responses to genuine anomalies and equipment maintenance for false anomalies.

[0074] Based on the above technical solution, water quality data is first acquired, and a preset time-series decomposition algorithm is used to separate periodic sequences representing periodic changes in water quality, trend sequences representing long-term trends, and residual sequences representing random fluctuations, effectively capturing the structural characteristics of the water quality data. Then, by comparing the similarity between each segment in the periodic sequence and the overall periodic pattern, suspected abnormal segments are marked, narrowing the scope of subsequent analysis and reducing unnecessary calculations to improve monitoring efficiency. Next, within the time range of suspected abnormal segments, the deviation of the data is corrected by combining the trend sequence change trend to avoid misjudging normal trend fluctuations and accurately locate the abnormal moment. Finally, the abnormal moment is verified by the residual sequence, filtering out false anomalies caused by systematic errors such as sensor drift. Ultimately, this achieves efficient and accurate identification of abnormal water quality measurement data, significantly improving the accuracy and reliability of abnormal data monitoring in environmental monitoring.

[0075] In one possible implementation, combining Figure 2 ,like Figure 3As shown, the method in S2 above can be specifically implemented through the following steps S21 to S24, which are explained in detail below:

[0076] S21. Perform curve fitting on the periodic sequence to obtain the periodic curve, and divide the periodic curve into multiple segments according to the extreme points of the periodic curve.

[0077] Specifically, the validity of the periodic sequence is first validated and completed to avoid affecting the fitting accuracy due to missing data or outliers. All data points in the periodic sequence are traversed. If data is missing (e.g., a temporary sensor malfunction causing data to be empty at a certain moment), linear interpolation (based on the linear trend of three valid data points before and after the missing point) is used to complete the missing values. If there are extreme values ​​that significantly exceed the reasonable range (e.g., pH = 12, far exceeding the normal pH range of 6-9 for water quality), they are marked as invalid and replaced using linear interpolation. Then, the temporal continuity of the periodic sequence is confirmed, ensuring that all data points are arranged sequentially according to their collection timestamps, without any temporal order disorder (e.g., the timestamp of one set of data is earlier than the previous set). If there are temporal order anomalies, they need to be reordered according to the timestamps. Finally, a valid periodic sequence without missing data or outliers and with temporal continuity is obtained, which serves as the input data for curve fitting.

[0078] For the preprocessed effective periodic sequence, a polynomial fitting algorithm is used for curve fitting. The core is to select the optimal polynomial order to ensure that the fitted periodic curve accurately reflects the overall fluctuation pattern of the periodic sequence. The polynomial order is obtained by combining the periodic characteristics of the water quality data (such as daily, weekly, and monthly cycles). For example, if the periodic sequence is daily periodic data (24-hour fluctuation), the fluctuation pattern is relatively simple, and the candidate order range can be set to a polynomial of degree 1-5 (degree 1 polynomial is linear fitting and suitable for smooth fluctuations; degree 5 polynomial can fit complex multi-peak and multi-valley fluctuations). Then, the optimal order is selected by the Akaike Information Criterion (AIC).

[0079] Based on a polynomial fitting model of optimal order, the fitted value corresponding to each data point in the periodic sequence is calculated. All fitted values ​​are connected in time stamp order to form a smooth and continuous periodic curve. This curve can completely characterize the overall periodic variation law of the periodic sequence. For example, the daily periodic curve of turbidity at a certain monitoring station shows typical fluctuation characteristics: trough in the early morning, rise in the morning, peak at noon, fall in the afternoon, and remain stable at night.

[0080] Finally, because minimum values ​​better reflect the start and end boundaries of a cycle in water quality cyclical fluctuations—for example, the minimum turbidity value at dawn each day in a diurnal cycle can be considered the start of a cycle—the extreme points of the curve are defined using these minimum values ​​as the core. All minimum values ​​are located by traversing the numerical changes of the cycle curve. Then, using these located minimum values ​​as dividing points, the cycle curve is divided into multiple independent segments according to the principles of chronological order and boundary integrity. The cycle curve between the starting point and the first minimum value is considered the first segment; subsequent segments are defined as the intervals between adjacent minimum values; and the segment between the last minimum value and the end point is considered the final segment.

[0081] S22. Extract multiple statistical features for each segment and construct a feature vector.

[0082] Statistical features are used to describe the morphology and distribution characteristics of the segments, including the degree of fluctuation, curvature of the shape, steepness of the distribution, overall level, and fluctuation boundaries. Specifically, this can be achieved through six types of feature data, including variance (used to quantify the fluctuation amplitude of periodic curve data points within a segment around the segment mean; the more severe the fluctuation, the greater the variance), mean curvature (used to quantify the curvature of the periodic curve within a segment; the greater the curvature, the more obvious the curve curvature), kurtosis (used to quantify the steepness of the data distribution within a segment, reflecting the density of extreme values ​​in the data; the greater the kurtosis, the steeper the distribution), mean (reflecting the overall numerical level of water quality indicators within a segment), and maximum and minimum values ​​(characterizing the fluctuation boundaries of the segment data).

[0083] In some implementations, multiple feature data points corresponding to the segmented periodic curves are first extracted, including variance, mean curvature, kurtosis, mean, maximum, and minimum values. Then, each feature data point is normalized to obtain multiple statistical features.

[0084] Specifically, for each segment of the divided periodic curve, six types of feature data are calculated one by one to obtain variance, mean curvature, kurtosis, mean, maximum, and minimum values. Since the dimensions and value ranges of the six types of feature data differ significantly (e.g., the variance may range from 0.01 to 0.2, the segmented mean curvature from 0.01 to 0.3, and the segmented mean from 0.1 to 0.9), directly using them for subsequent similarity calculations would lead to biased results due to dimensional interference. Therefore, the Min-Max normalization method, consistent with the data preprocessing steps, is adopted to uniformly map each type of feature data to the [0,1] interval, eliminating the influence of dimensions and obtaining standardized statistical features.

[0085] Furthermore, the six normalized statistical features of the current segment are combined in a fixed order (e.g., normalized variance, normalized mean curvature, normalized kurtosis, normalized mean, normalized maximum, and normalized minimum) to construct a 6-dimensional segment feature vector. Each dimension of this vector corresponds to a standardized index describing the segment's shape or distribution characteristics, uniquely representing the local periodic variation pattern of the current segment.

[0086] For example, the six statistical features obtained after normalization of a certain segment are: normalized variance 0.38, normalized mean curvature 0.09, normalized kurtosis 0.25, normalized mean 0.62, normalized maximum value 0.85, and normalized minimum value 0.18. Then its eigenvector is [0.38, 0.09, 0.25, 0.62, 0.85, 0.18].

[0087] For all the periodic curve segments, feature data are extracted, normalized, and feature vectors are constructed according to the above process, and finally the feature vectors of all segments are obtained.

[0088] S23. For each segment, determine the similarity between the feature vector of the segment and the feature vector of other segments using a preset similarity measurement method.

[0089] In some implementations, since the feature data has been normalized and the vector values ​​are uniformly within the range of [0, 1], consistent vector directions can represent consistent segmentation patterns. Therefore, the cosine similarity algorithm can be used as the preset similarity measurement method. By measuring the cosine of the angle between two feature vectors, it focuses on the consistency of vector directions (i.e., the degree of pattern matching) rather than the difference in numerical values. The range of cosine similarity is [0, 1]. The closer the value is to 1, the more consistent the directions of the two vectors are, and the more similar the periodic change patterns of the corresponding segments are; the closer the value is to 0, the greater the difference in directions, and the less similar the periodic patterns of the corresponding segments are.

[0090] If there are N segments, calculate the feature vector similarity between the current segment and each of the other segments, obtaining N-1 one-to-one similarities. Then, use the average of these N-1 one-to-one similarities as the overall similarity of the current segment. The average similarity reflects the average similarity level between the current segment and the entire curve. If there are no outliers in the segment data, the segment is highly similar to the whole curve, reflected in a high average similarity value. If the average similarity is low, it indicates a significant difference between the current segment and the overall curve, potentially suggesting data anomalies that require further analysis to determine.

[0091] For all the periodic curve segments, the overall similarity is calculated according to the above process, and finally the similarity of all segments is obtained.

[0092] S24. Mark segments with similarity less than the similarity threshold as suspected abnormal segments.

[0093] In some implementations, setting a similarity threshold may involve: retrieving historical periodic sequence data without water quality anomalies (verified manually to confirm the absence of actual anomalies), calculating the overall similarity of each segment, and obtaining a historical normal similarity set (containing the overall similarity values ​​of historical normal segments). Statistical analysis is then performed on the historical normal overall similarity set, calculating its mean and standard deviation. The base threshold is determined based on the difference between the mean and twice the standard deviation. Based on the normal distribution characteristics, this value covers most segments in the historical normal data, ensuring that the overall similarity of the vast majority of normal segments will not fall below this value.

[0094] For example, if the mean of the historical normal overall similarity set is 0.82 and the standard deviation is 0.06, then the basic threshold is 0.82 - 2 × 0.06 = 0.70.

[0095] In addition, considering the slight fluctuations that may occur in the water quality cycle (such as short-term regular changes caused by seasonal alternation), the basic threshold needs to be fine-tuned based on engineering experience. If the water quality stability of the monitoring area is high (such as enclosed water bodies), the threshold can be maintained at the basic value; if the water quality is easily affected by short-term disturbances (such as the impact of rainy season runoff), the threshold can be appropriately reduced (down by 0.05) to avoid misjudging normal fluctuations as abnormalities.

[0096] Based on the above examples, if the monitoring area is an urban river with moderate water quality stability, the similarity threshold can be determined to be 0.70.

[0097] Furthermore, if the overall similarity of a segment is greater than or equal to the similarity threshold, it indicates that the local periodicity of the segment matches the overall pattern well and conforms to normal periodicity characteristics. It is judged as a normal segment and is not marked. If the similarity of a segment is less than the similarity threshold, it indicates that the local periodicity of the segment differs significantly from the overall pattern and deviates from normal periodicity characteristics. It is judged as a suspected abnormal segment and needs to be marked.

[0098] Based on the above technical solution, by performing curve fitting and extreme point segmentation on the periodic sequence, an independent analysis unit reflecting local periodic patterns is constructed. Then, by extracting segmented feature data and normalizing it to construct feature vectors, the abstract periodic variation pattern is transformed into a quantifiable vector index. Subsequently, the cosine similarity algorithm is used to calculate the pattern matching degree between each segment and other segments. Combined with a similarity threshold calibrated based on historical data, suspected anomaly identification and labeling are completed. This approach can accurately identify suspected anomalous segments that significantly differ from the overall periodic pattern, effectively avoiding misjudging normal periodic fluctuations as anomalies. Furthermore, by clearly defining the time range for subsequent trend analysis, it significantly reduces unnecessary computation.

[0099] In one possible implementation, combining Figure 2 ,like Figure 4 As shown, the method in S3 above can be specifically implemented through the following S31 to S34, which are explained in detail below:

[0100] S31. Divide the data in the trend sequence corresponding to the suspected abnormal segments into sliding windows with a preset window size to obtain multiple window data.

[0101] First, based on the time range corresponding to the suspected abnormal segments in the periodic sequence, trend data that perfectly matches the time range is selected from the complete trend sequence to form suspected abnormal trend segments.

[0102] Then, by combining the total number of data points in the suspected abnormal trend segment with the time scale of the trend change, a preset window size is set to ensure that each window can fully capture the change characteristics of a local trend while avoiding the local trend being smoothed due to an excessively large window or the trend characteristics being fragmented due to an excessively small window.

[0103] In some implementations, the suspected abnormal trend segments identified through screening can be divided into segments, and the data points can be counted, denoted as the total data volume m. Then, considering the typical time scale of water quality trend changes (e.g., short-term trend changes in water quality indicators typically take 1-2 hours to manifest, corresponding to 6-12 data points at 10-minute intervals), the window size can be set as a preset proportion of the total data volume m and rounded up. The preset proportion is related to the total data volume.

[0104] For example, if the suspected anomaly segment corresponding to the suspected anomaly trend segment in 24 hours contains 144 data points, i.e., m=144, the window size can be set to 5% of the total data volume m and rounded up. m×5%=7.2, and after rounding up, the window size is 8, meaning each window contains 8 data points, corresponding to 80 minutes of trend data. At this point, the 5% ratio ensures that the window size contains enough data points to support subsequent trend slope calculations, meeting the needs of capturing short-term water quality trend changes. Simultaneously, it allows the number of windows to be maintained at around 20, ensuring that each window contains enough data points to fit a stable trend, while also allowing multiple windows to cover the entire suspected anomaly trend segment, achieving a fine depiction of local trends.

[0105] Based on a defined window size, the sliding step size is set to be consistent with the window size (i.e., non-overlapping sliding, to avoid redundant trend features caused by repeated data calculation). Suspected abnormal trends are segmented into continuous sliding windows in chronological order to obtain multiple window data. Each window data corresponds to a local trend data of a continuous time period.

[0106] If the total data volume *m* of a suspected anomaly segment is not divisible by the window size, the remaining data points in the last segment that are less than the window size are merged into the previous window or treated as a separate window to avoid data omission. For example, when *m*=144, 144 ÷ 8 = 18, which is exactly divisible, resulting in 18 windows, each containing 8 data points. If *m*=145, the first 17 windows each contain 8 data points (a total of 136 data points), and the remaining 9 data points are used as the 18th window, ensuring that all data points in the entire suspected anomaly segment are included within the window coverage.

[0107] S32. For each window, perform linear fitting to obtain the trend slope of the window.

[0108] The trend slope is used to characterize the direction and magnitude of changes in the data within the window.

[0109] The time sequence of data points within a window is used as the independent variable to characterize the temporal order of the data points within the window. For example, the 8 data points in window 1 are assigned numbers 0, 1, 2, ..., 7 in chronological order (the unit is the data point interval, corresponding to an actual time interval of 10 minutes / interval), ensuring that the independent variable is consistent with the temporal correlation of the data points. The trend sequence value corresponding to each data point within the window is used as the dependent variable, taken from the window data. The trend data within the window is fitted with a straight line, and the slope of the line (slope k) reflects the change of the trend data over time.

[0110] S33. Determine the relative deviation between windows based on the trend slope of multiple windows.

[0111] The relative deviation is used to characterize the level of fluctuation in the window slope.

[0112] In some implementations, the overall deviation can be determined first based on the trend slopes of multiple windows. The standard deviation of the absolute values ​​of the trend slopes of all windows is calculated as the overall deviation, used to characterize the average fluctuation range of the trend slopes around the mean. Here, the sign of the trend slope only represents the direction of the trend (upward or downward), and is unrelated to the magnitude of the fluctuation (for example, a slope of 0.02 represents an upward trend, and a slope of -0.02 represents a downward trend; although the directions are opposite, the fluctuation ranges are exactly the same). If the absolute values ​​are not taken first, the positive and negative slopes will cancel each other out when calculating the standard deviation because of their opposite directions, resulting in the final standard deviation not accurately reflecting the overall level of fluctuation. Therefore, it is necessary to first take the absolute value of each slope (discarding the direction), and then calculate the standard deviation of these absolute values ​​to obtain the overall deviation of the trend slope as a benchmark reflecting the average fluctuation range globally. The larger the standard deviation, the more drastic the slope fluctuations globally; the smaller the standard deviation, the more stable the global slope fluctuations.

[0113] Simultaneously, the absolute deviation of each window is determined based on its trend slope and the trend slope of the preceding adjacent window. The absolute value of the difference between the trend slope of each window and the trend slope of the preceding adjacent window is calculated as the absolute deviation of each window, used to characterize the magnitude of the change in the trend slope of a single window relative to the preceding adjacent window. Calculating the difference without disregarding direction allows for precise quantification of the drastic change in the slope of adjacent windows, including not only increases or decreases in amplitude within the same direction but also the cumulative amplitude caused by directional reversals (e.g., from rising to falling, or from falling to rising), the latter often being an important signal of trend anomalies. Therefore, calculating the difference first and then taking the absolute value fully preserves the combined effect of changes in direction and amplitude, accurately reflecting actual fluctuations. Furthermore, since the first window has no preceding adjacent windows, its absolute deviation is set to 0.

[0114] Then, the absolute deviation is corrected based on the overall deviation to obtain the relative deviation for each window. Since the absolute deviation only reflects the original differences between adjacent windows and does not consider the influence of global fluctuations (e.g., adjacent changes may be considered normal fluctuations in a highly volatile global environment, but abnormal fluctuations in a less volatile environment), the absolute deviation needs to be normalized and corrected using the overall deviation to obtain a standardized relative deviation, denoted as:

[0115]

[0116] In the formula, This represents the absolute deviation of the i-th window. When i=1, =0; when i is greater than 1 .in, This represents the trend slope of the i-th window. This represents the trend slope of the preceding adjacent window of the i-th window.

[0117] This indicates the overall degree of deviation, specifically the standard deviation of the slopes across all windows. Divide by Used to eliminate the interference of global fluctuation amplitude on the judgment of local differences.

[0118] It should be a very small value, such as 0.001, to avoid the error of the denominator being zero.

[0119] Let represent the normalization function. It is normalized to the [0, 1] interval using Min-Max (where the minimum and maximum values ​​are the extreme values ​​of the calculated results for all windows), yielding the final standardized relative deviation. This results in the relative deviation of the i-th window. The closer the value is to 1, the more significant the slope change of the i-th window is, and the greater the possibility that it exceeds the normal global fluctuation; the closer the value is to 0, the smoother the fluctuation is, which is consistent with the characteristics of the normal global fluctuation.

[0120] S34. Identify abnormal windows based on the degree of relative deviation, and determine the abnormal time from the abnormal windows.

[0121] In some implementations, the overall relative deviation can be determined first based on the relative deviations of multiple windows. The average of the relative deviations of multiple windows is then calculated as the overall relative deviation, used to quantify the comprehensive level of global slope fluctuation.

[0122] Then, based on the overall relative deviation, the basic threshold of the relative deviation distribution range is corrected.

[0123] In some implementations, the 3sigma (3σ) principle can be used to calculate the mean of the relative deviations of all windows. and standard deviation Set a base threshold T, expressed as:

[0124]

[0125] Then based on the overall relative deviation Construct correction coefficients and correct the base threshold T to obtain the corrected threshold. , represented as:

[0126]

[0127] In the formula, This is the parameter tuning coefficient, for example, 0.5.

[0128] Follow An increase in the overall trend, especially when it fluctuates dramatically, indicates that the data is in a period of rapid change and is generally unstable. In this case, the detection threshold should be relaxed to avoid misinterpreting normal large changes as anomalies. This can eliminate invalid interference from fluctuations in the trend slope and restore the true fluctuations in water quality. Specifically, this refers to the overall relative deviation. The larger the value, the more significant the slope fluctuations between most windows and their adjacent windows within the suspected abnormal trend segment, indicating drastic fluctuations in the global trend slope. This suggests interference with the screening based on the basic threshold T, necessitating a relaxation of the screening criteria for the severity of trend changes to increase the likelihood of detection. ,Right now The closer to 1, The closer it is to 1.5 Overall relative deviation The smaller the value, the more gradual the overall trend change, the less the basic threshold T is affected by interference, and the lower the detection threshold. ,Right now The closer to 0, The closer .

[0129] Next, anomaly detection is performed on the relative deviation of each window. If the relative deviation of a window exceeds the corrected threshold, the window is marked as an abnormal window. For each abnormal window, the time corresponding to the center of the window can be selected as the abnormal moment. If multiple consecutive windows are marked as abnormal, they can be merged into an abnormal time period, and the time corresponding to the window with the largest relative deviation can be taken as the abnormal moment.

[0130] Based on the above technical solution, the trend sequence of suspected anomalies is divided into sliding windows. The trend slope of each window is calculated and the relative deviation is determined. Then, a threshold for the distribution range of the relative deviation is corrected. Finally, based on the corrected threshold, the abnormal window is identified and the abnormal moment is located. This approach can focus on the suspected anomaly time range to reduce invalid calculations, and by eliminating the interference of trend slope fluctuations, it accurately restores the true fluctuations in water quality, effectively avoiding misjudgments of normal trend fluctuations, and significantly improving the accuracy of anomaly moment detection.

[0131] In one possible implementation, combining Figure 2 ,like Figure 5 As shown, the method in S4 above can be specifically implemented through the following steps S41 to S43, which are explained in detail below:

[0132] S41. Extract a preset number of residual data before and after the abnormal moment as a verification window, and determine the fluctuation range of the residual data in the verification window through a preset statistical method.

[0133] The core function of the validation window is to capture the natural fluctuation characteristics of the residuals before and after the anomaly. It needs to cover the residual data at the time of the anomaly and for a period of time before and after it to ensure that it can reflect the trend of residual changes before and after the anomaly occurs.

[0134] In some implementations, based on the water quality data collection frequency (one data point every 10 minutes), the preset number of residual data points before and after the anomaly is 144. This means the verification window includes 144 data points before the anomaly, the data point at the anomaly, and 144 data points after the anomaly, corresponding to a time range of one day before and one day after the anomaly. This time span covers short-term residual changes before and after the anomaly, while avoiding the introduction of irrelevant fluctuations (such as unrelated residual changes caused by environmental interference) due to an excessively large range, ensuring the correlation between the residual data within the window and the anomaly.

[0135] Furthermore, for the residual data within the verification window, box plots are used. Figure 4The interquartile range (IQR) algorithm determines the fluctuation range. Specifically, it involves: first, sorting the residual data within the validation window by numerical value in ascending order; then calculating the first quartile Q1 (the value at the 25th percentile after sorting) and the third quartile Q3 (the value at the 75th percentile), obtaining the interquartile range IQR = Q3 − Q1; finally, determining the fluctuation range of the residuals, with an upper limit threshold of Q3 + 1.5 × IQR and a lower limit threshold of Q1 − 1.5 × IQR. This interval represents the normal fluctuation range of the residual data within the validation window. Here, 1.5 is the outlier determination coefficient, which can be adjusted according to the outlier detection requirements. A coefficient of 1.0 is used in strict mode, and 2.0 is used in lenient mode. A coefficient of 1.5 is preferred to balance false negatives and false negatives.

[0136] S42. If the residual value in the verification window exceeds the fluctuation range, the abnormal moment is determined to be a real abnormality.

[0137] Among them, the essence of true anomalies is that outliers detected in the trend sequence and abnormal fluctuations in the residual sequence occur simultaneously. The correlation between the two indicates that the anomalies are not caused by systematic errors such as sensor drift and data interference, but by abnormal changes in the water quality itself.

[0138] S43. If the residual value within the verification window is within the fluctuation range, the abnormal moment is determined to be a false anomaly.

[0139] Among them, the essence of false anomalies is that the anomalies in the trend sequence are caused by systematic errors such as sensor zero drift and data transmission interference, and the residual sequence does not show synchronous abnormal fluctuations. Typical scenarios include long-term sensor non-calibration causing overall drift of trend data, or short-term electromagnetic interference causing distortion of individual trend data.

[0140] Based on the above technical solution, a verification window is constructed by extracting residual data before and after the anomaly, determining the normal fluctuation range of the residual, and then judging the authenticity of the anomaly based on whether the residual value exceeds this range. By leveraging the characteristics of the residual sequence (random fluctuations after removing trends and periods), false anomalies caused by systematic errors such as sensor drift and data interference are effectively filtered out. Only abnormal fluctuations in residuals caused by real water quality issues are judged as real anomalies. This avoids false alarms for non-water quality issues and ensures accurate identification of real water quality anomalies. It improves the closed loop of water quality anomaly monitoring from the residual dimension, significantly enhancing the reliability and practicality of anomaly monitoring results.

[0141] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0143] In this embodiment of the invention, the environmental monitoring measurement anomaly data detection device can be divided into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0144] This invention also provides a hardware structure diagram of an environmental monitoring metering anomaly data detection device, see [link / reference]. Figure 6 The environmental monitoring measurement anomaly data detection device 600 includes a processor 601, and optionally, a memory 602 connected to the processor 601.

[0145] In the first possible implementation, see Figure 6 The environmental monitoring metering anomaly data detection device 600 also includes a transceiver 603. The processor 601, memory 602, and transceiver 603 are connected via a bus. The transceiver 603 is used to communicate with other devices or communication networks. Optionally, the transceiver 603 may include a transmitter and a receiver. The device in the transceiver 603 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of the present invention. The device in the transceiver 603 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of the present invention.

[0146] Based on the first possible implementation method Figure 6 The schematic diagram shown can be used to illustrate the structure of the environmental monitoring measurement anomaly data detection device involved in the above embodiments.

[0147] in, Figure 6 This can also be illustrated by the system chip in the environmental monitoring metering anomaly data detection device. In this case, the actions performed by the aforementioned environmental monitoring metering anomaly data detection device can be implemented by this system chip. The specific actions performed can be found above and will not be repeated here.

[0148] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in this embodiment can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0149] The processor in this invention may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a System-on-a-Chip (SoC), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or logic circuits that implement dedicated logic operations.

[0150] The memory in the embodiments of the present invention may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0151] This invention also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0152] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0153] This invention also provides a chip, which includes a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0154] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0155] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In this invention, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several of the functions listed in this invention.

[0156] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for detecting abnormal measurement data in environmental monitoring, characterized in that, include: Water quality data is acquired and decomposed into periodic sequences, trend sequences, and residual sequences using a preset time-series decomposition algorithm. The periodic sequence is used to characterize the periodic changes in water quality data; The trend sequence is used to characterize the long-term trend of water quality data; The residual sequence is used to characterize the random fluctuations in water quality data; By comparing the similarity of the periodic change patterns among multiple segments in the periodic sequence, segments with similarity less than a similarity threshold are marked as suspected abnormal segments. The data corresponding to the suspected abnormal segments in the trend sequence are divided into multiple window data by a sliding window of a preset window size; For each window, a linear fit is performed to obtain the trend slope of the window; The trend slope is used to characterize the direction and magnitude of data change within the window; The overall deviation is determined by the trend slope of multiple windows. The absolute deviation of each window is determined by the trend slope of each window and the trend slope of the previous adjacent window. The absolute deviation is then corrected based on the overall deviation to obtain the relative deviation of each window. The relative deviation is used to characterize the level of fluctuation in the window slope change; The overall relative deviation is determined based on the relative deviation of multiple windows. The basic threshold of the relative deviation distribution range is corrected based on the overall relative deviation. Windows with relative deviation greater than the corrected threshold are marked as abnormal windows, and abnormal moments are determined from the abnormal windows. The abnormal moments are used to indicate the moments when data values ​​in the trend sequence deviate from the normal range. Based on the residual sequence, the abnormal moments are verified to determine whether they are real or false anomalies.

2. The method for detecting abnormal measurement data according to claim 1, characterized in that, By comparing the similarity of the periodic variation patterns among multiple segments in the periodic sequence, segments with similarity scores less than a similarity threshold are marked as suspected abnormal segments, including: The periodic sequence is curve-fitted to obtain a periodic curve, and the periodic curve is divided into multiple segments based on the extreme points of the periodic curve. For each segment, multiple statistical features are extracted and a feature vector is constructed; the statistical features are used to describe the shape and distribution characteristics of the segment. For each segment, the similarity between the feature vector of that segment and the feature vector of other segments is determined by a preset similarity measurement method. Segments with similarity scores less than the similarity threshold are marked as suspected abnormal segments.

3. The method for detecting abnormal measurement data according to claim 2, characterized in that, For each segment, extract multiple statistical features, including: Extract multiple feature data of the segmented periodic curves; the feature data includes variance, mean curvature, kurtosis, mean, maximum and minimum values; Each feature data point is normalized to obtain multiple statistical features.

4. The method for detecting abnormal measurement data according to claim 1, characterized in that, Based on the residual sequence, the abnormal moments are verified to determine whether they are real or false anomalies, including: Extract a preset number of residual data before and after the abnormal moment as a verification window, and determine the fluctuation range of the residual data within the verification window using a preset statistical method; If the residual value within the verification window exceeds the fluctuation range, the abnormal moment is determined to be a true anomaly. If the residual value within the verification window is within the fluctuation range, the abnormal moment is determined to be a false anomaly.

5. The method for detecting abnormal measurement data according to claim 1, characterized in that, Obtain water quality data, including: Raw data is collected through deployed sensors; the raw data includes at least one of pH, dissolved oxygen content, turbidity, temperature, ammonia nitrogen content, and permanganate index. The water quality data is obtained by normalizing the original data.

6. The method for detecting abnormal measurement data according to claim 5, characterized in that, The water quality data is decomposed into periodic sequences, trend sequences, and residual sequences using a preset time-series decomposition algorithm, including: The water quality data is sorted by time to form a continuous time series; The time series is subjected to weighted fitting within a sliding window, and the trend component reflecting the long-term change direction of water quality is extracted to obtain the trend series; The intermediate data after removing the trend component from the time series are subjected to periodic pattern recognition and fitting to extract the periodic components that exhibit fixed periodic repetition characteristics, thereby obtaining the periodic sequence. The remaining components after removing the trend and periodic components from the time series are determined as the residual series.

7. A system for detecting abnormal measurement data in environmental monitoring, characterized in that, include: The data preprocessing module is used to acquire water quality data and decompose the water quality data into periodic sequences, trend sequences, and residual sequences using a preset time-series decomposition algorithm; the periodic sequences are used to characterize the periodic changes in water quality data. The trend sequence is used to characterize the long-term trend of water quality data; The residual sequence is used to characterize the random fluctuations in water quality data; The periodic sequence analysis module is used to compare the similarity of the periodic change patterns among multiple segments in the periodic sequence, and to mark segments with similarity less than a similarity threshold as suspected abnormal segments. The trend sequence analysis module is used to divide the data corresponding to the suspected abnormal segments in the trend sequence into sliding windows with a preset window size to obtain multiple window data. For each window, a linear fit is performed to obtain the trend slope of the window; The trend slope is used to characterize the direction and magnitude of data change within the window; The overall deviation is determined by the trend slope of multiple windows. The absolute deviation of each window is determined by the trend slope of each window and the trend slope of the previous adjacent window. The absolute deviation is then corrected based on the overall deviation to obtain the relative deviation of each window. The relative deviation is used to characterize the fluctuation level of the window slope change; the overall relative deviation is determined based on the relative deviation of multiple windows, the basic threshold of the relative deviation distribution range is corrected based on the overall relative deviation, windows with relative deviation greater than the corrected threshold are marked as abnormal windows, and abnormal moments are determined from the abnormal windows; the abnormal moments are used to indicate the moments when data values ​​in the trend sequence deviate from the normal range. The residual sequence verification module is used to verify the abnormal time based on the residual sequence to determine whether it is a real abnormality or a false abnormality.

Images