Lake water quality monitoring data quality control method and system

Abstract

The application provides a lake water quality monitoring data quality control method and system, the scheme comprises obtaining lake water quality monitoring data, and corresponding pretreatment is carried out;For the pretreated lake water quality monitoring data, according to whether a special event occurs, classification is carried out, the lake water quality monitoring data corresponding to the special event and the lake water quality monitoring data corresponding to the non-special event are obtained;The lake water quality monitoring data corresponding to the non-special event is subjected to seasonal outlier analysis, and the initial abnormal data is obtained;For the obtained initial abnormal data, based on the mixed test strategy, the true abnormal value is obtained and the abnormal value is removed;For the original missing value in the lake water quality monitoring data and the missing value generated due to the removal of the abnormal value, the missing value is completed based on the adjacent data of the missing value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data quality control, and particularly relates to a lake water quality monitoring data quality control method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The buoy automatic monitoring system is one of important technical means to realize water quality continuous monitoring. Through continuous observation of the buoy, the water quality change characteristics are gradually changed from qualitative description to accurate quantitative measurement. At the same time, with long-term accumulation of monitoring data, these data will provide valuable basic information for large-scale water environment spatio-temporal variation research. As an inland water body, the water quality monitoring of the lake plays an important role in environmental protection and water resources management. The buoy automatic monitoring system can continuously provide water quality data, but due to long-term operation of the equipment, some parameters may drift, and combined with the uniqueness of the lake environment, such as slow water flow, low water exchange capacity and significant seasonal variation, data quality control faces many challenges.

[0004] The inventor finds that the existing water quality data quality control method mainly has the following problems: in the quality control process, many methods often involve human-set detection standards or thresholds, these thresholds cannot be applied to all water areas, and the human-set thresholds may be too sensitive to the data distribution form, resulting in inaccurate determination of abnormal values in the case of uneven data distribution or existence of extreme values, and at the same time, such scheme depends on prior knowledge, that is, it needs to rely on prior knowledge or historical data to set thresholds, which limits the universality and flexibility of the quality control method; secondly, the existing water quality data quality control method mainly aims at marine water type, but due to the particularity of the lake environment, direct application in the lake water area will result in poor data control effect, and there is a lack of data quality control strategy with pertinence and good effect for the lake water area. SUMMARY

[0005] The embodiments of the present application provide a lake water quality monitoring data quality control method and system to solve the problems that the traditional scheme needs human participation and relies on prior knowledge, resulting in poor accuracy and applicability of the quality control result; and the particularity of the lake environment cannot be directly used in the existing method.

[0006] According to a first aspect of the embodiments of the present application, a lake water quality monitoring data quality control method is provided, comprising:

[0007] Obtaining lake water quality monitoring data and performing corresponding preprocessing;

[0008] For the pretreated lake water quality monitoring data, the lake water quality monitoring data corresponding to the special event and the lake water quality monitoring data corresponding to the non-special event are obtained according to whether the special event occurs, wherein the special event is an event of abnormal lake water quality caused by human factors and environmental factors.

[0009] The lake water quality monitoring data corresponding to the non-special event is subjected to seasonal abnormal value analysis to obtain initial abnormal data; wherein the seasonal abnormal value analysis is specifically: the lake water quality monitoring data is subjected to seasonal division, 3σ analysis and box plot analysis are respectively performed on the data of each season, and initial abnormal data is obtained.

[0010] For the obtained initial abnormal data, real abnormal values are obtained based on a mixed test strategy, and the abnormal values are removed.

[0011] For the original missing values in the lake water quality monitoring data and the missing values generated due to the removal of abnormal values, the missing values are completed based on the adjacent data of the missing values.

[0012] Further, the 3σ analysis and the box plot analysis are respectively performed on the data of each season to obtain the initial abnormal data, specifically: the mean and the standard deviation of the lake water quality monitoring data of each season are calculated respectively; based on the monitoring data in the lake water quality monitoring data deviating from the mean by more than three times the standard deviation, a first initial abnormal data set is constructed; for the lake water quality monitoring data of each season, box plot analysis is performed to determine the upper quartile, the lower quartile and the interquartile range, and based on the upper quartile, the lower quartile and the interquartile range, a second initial abnormal data set is determined; based on the intersection of the first initial abnormal data set and the second initial abnormal data set, the initial abnormal data is obtained.

[0013] Further, based on the mixed test strategy, real abnormal values are obtained, specifically: based on the obtained initial abnormal data, range test method, mean square error test method and sharp peak test Rein Da method are respectively performed to obtain a first abnormal data set, a second abnormal data set and a third abnormal data set; based on the union of the first abnormal data set, the second abnormal data set and the third abnormal data set, real abnormal data is obtained.

[0014] Further, the special event includes but is not limited to pollution events or red tide events.

[0015] Further, the missing values are completed based on the adjacent data of the missing values, specifically: the preset number of adjacent data of the missing value data is determined based on the K-nearest neighbor algorithm, and the missing values are completed based on the mean of the adjacent data.

[0016] Further, the determination of the neighbor data is based on the position relationship in the time sequence of the monitoring data, the neighbor data is determined by combining the K-neighbor algorithm, or the neighbor data is determined based on the similarity calculation result between different monitoring data by performing multi-dimensional feature extraction on the monitoring data.

[0017] According to a second aspect of the embodiments of the present application, a lake water quality monitoring data quality control system is provided, comprising:

[0018] a data acquisition unit configured to acquire lake water quality monitoring data and perform corresponding preprocessing;

[0019] a data classification unit configured to classify the preprocessed lake water quality monitoring data according to whether a special event occurs, to obtain lake water quality monitoring data corresponding to the special event and lake water quality monitoring data corresponding to a non-special event, wherein the special event is an event in which lake water quality anomaly occurs due to human factors and environmental factors;

[0020] an initial abnormal data acquisition unit configured to perform seasonal outlier analysis on the lake water quality monitoring data corresponding to the non-special event to obtain initial abnormal data, wherein the seasonal outlier analysis specifically comprises: performing seasonal division on the lake water quality monitoring data, performing 3σ analysis and box plot analysis on the data of each season respectively to obtain the initial abnormal data; and performing independent analysis on the lake water quality monitoring data corresponding to the special event to determine the initial abnormal data;

[0021] an outlier rejection unit configured to obtain true outliers and perform outlier rejection based on a hybrid test strategy for the obtained initial abnormal data;

[0022] a missing value filling unit configured to fill missing values in the lake water quality monitoring data based on neighbor data of the missing values.

[0023] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor implements the lake water quality monitoring data quality control method when executing the program.

[0024] According to a fourth aspect of the embodiments of the present application, a non-transitory computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the lake water quality monitoring data quality control method.

[0025] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the lake water quality monitoring data quality control method.

[0026] The one or more technical solutions above have the following beneficial effects:

[0027] (1) The present application provides a lake water quality monitoring data quality control method and system, which does not need to set threshold parameters manually and does not need the assistance of prior knowledge, and based on the particularity of the lake environment, the data is divided seasonally by using seasonal anomaly analysis, and the initial abnormal data is judged respectively; and through the mixed verification strategy, the obtained initial abnormal value is verified again, which effectively ensures the accuracy of abnormal data identification, and based on the neighborhood data, the abnormal data is completed, which effectively ensures the data quality of the lake water quality monitoring data.

[0028] (2) The scheme of the present application classifies the lake water quality monitoring data according to whether a special event occurs, independently analyzes the data under the special event, effectively avoids the situation that the data under the special event is misidentified as abnormal data, and improves the adaptability of the scheme of the present application.

[0029] (3) The scheme of the present application can adapt to the lake water body type with complex inland water quality characteristics while ensuring the accuracy of data quality control, has better data quality control effect, and has important significance for future inland lake buoy data quality control development.

[0030] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0031] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application.

[0032] Figure 1 A flow chart of a lake water quality monitoring data quality control method described in an embodiment of the present application. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0034] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application.

[0035] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0036] like Figure 1 As shown, this embodiment of the invention provides a method for quality control of lake water quality monitoring data, including:

[0037] Step 1: Obtain lake water quality monitoring data and perform appropriate preprocessing;

[0038] In specific implementation, the preprocessing includes finding duplicate data, empty records, and missing values, deleting duplicate records, and marking empty records and missing values.

[0039] Step 2: For the pre-processed lake water quality monitoring data, classify them according to whether special events have occurred, and obtain the lake water quality monitoring data corresponding to special events and the lake water quality monitoring data corresponding to non-special events. The special events are events that cause abnormal lake water quality due to human factors and environmental factors.

[0040] In practice, the special events include, but are not limited to, pollution events or red tide events, and the data from the periods when these special events occurred are analyzed independently.

[0041] Step 3: Perform seasonal outlier analysis on the lake water quality monitoring data corresponding to non-special events to obtain initial outlier data; wherein, the seasonal outlier analysis specifically involves: dividing the lake water quality monitoring data into seasonal categories, and performing 3σ analysis and box plot analysis on the data for each season to obtain initial outlier data;

[0042] In specific implementation, the method of performing 3σ analysis and box plot analysis on the data for each season to obtain initial anomaly data is as follows: The mean and standard deviation of the lake water quality monitoring data for each season are calculated; a first initial anomaly dataset is constructed based on the monitoring data in the lake water quality monitoring data whose deviation from the mean exceeds three times the standard deviation; a box plot analysis is performed on the lake water quality monitoring data for each season to determine the upper quartile, lower quartile, and interquartile range; a second initial anomaly dataset is determined based on the upper quartile, lower quartile, and interquartile range; and initial anomaly data is obtained based on the intersection of the first and second initial anomaly datasets.

[0043] Specifically, the seasonal 3σ analysis is as follows:

[0044] Under the original 3σ principle, the outliers are often defined as the values deviating from the average value by more than 3 times the standard deviation in a set of measured values; but in the inland lake water area, due to the gentle water flow, the limited water exchange capacity, etc., the water quality changes significantly seasonally, etc., so the same data has different results in different seasons, therefore, according to the time information, the data is divided by season or month, and seasonal 3σ analysis is carried out respectively.

[0045] The seasonal box plot analysis is specifically:

[0046] The box plot identifies outliers, which are usually defined as values less than QL-1.5IQR or QU+1.5IQR, QL is the lower quartile, QU is the upper quartile, and IQR is the interquartile range, which is represented as QU-QL; according to the time information, the data is divided by season or month, and seasonal box plot analysis is carried out respectively.

[0047] Step 4: Based on the mixed test strategy, the real outliers are obtained and the outliers are removed;

[0048] In specific implementation, based on the mixed test strategy, the real outliers are obtained, specifically: based on the obtained initial abnormal data, range test method, mean square difference test method and peak test Rein Da method are respectively executed, first, second and third abnormal data sets are obtained; based on the union set of the first, second and third abnormal data sets, real abnormal data is obtained.

[0049] Specifically, the range test method is specifically: the purpose of this method is to preliminarily screen the original observation data according to the range of the observation element in the target area, and mark the data outside the range as abnormal quality control. Through the analysis of historical data and other observation data, if the data is detected outside the range, it is marked as abnormal data. The processing process is: according to the historical measurement data or public other data set of a certain water quality observation element, the threshold range is set, the data beyond the range is identified as abnormal data.

[0050] The mean square difference test is specifically: when a value at a certain time is greater than the average value of the previous and next day at that time by n times the standard deviation, it is considered as an outlier, where n is 5; wherein, a certain time is defined as t time of d day, and the previous and next day is defined as t time of d-1 day to t time of d+1 day;

[0051] The peak test Rein Da method is specifically: the calculation formula of the peak test Rein Da method, assuming that there is a set of observation data X i =(X1, X2, X3,..., X N ), the previous adjacent data of X i is Xi-1 , P i = |X i - X i-1 |, wherein P i is the adjacent difference, assuming P0 is 0, when the residual of P i and P i+1 are both greater than 3σ, i.e. |P i - P| > 3σ and |P i+1 - P| > 3σ, it is considered that the X i data is abnormal, wherein P is the mean value, and σ is the standard deviation,

[0052] Step 5: For the original missing values in the lake water quality monitoring data and the missing values generated due to the removal of abnormal values, the missing values are completed based on the adjacent data of the missing values.

[0053] In a specific implementation, the missing values are completed based on the adjacent data of the missing values, specifically: the preset number of adjacent data of the missing value data is determined based on the K-Nearest Neighbor algorithm, and the missing values are completed based on the mean value of the adjacent data.

[0054] Specifically, in this embodiment, the near neighbors are determined based on the positional relationship in the time sequence of the monitoring data, and in more embodiments, the near neighbors can also be determined based on the similarity calculation results between different monitoring data by performing multi-dimensional feature extraction on the monitoring data.

[0055] It can be understood that the determination of the near neighbors can be determined according to actual needs.

[0056] In one or more embodiments, the acquisition of the initial abnormal data can also adopt the Dixon test method, which is a non-parametric method and can effectively find abnormal observation values in the data set.

[0057] Compared with seasonal 3σ analysis and seasonal box plot analysis, the Dixon test method can effectively and quickly detect outliers in the data, and is relatively simple, but has requirements for the number of samples, and the test needs to refer to the threshold setting.

[0058] In one or more embodiments, in the mixed test strategy, the traditional spike test method can be used to replace the spike test Lindauer method, that is, the observation elements are subjected to spike test, and if a large mutation occurs, it is abnormal. The observation elements are subjected to test value operation, and if the test value exceeds the standard range, it is determined to be an abnormal value.

[0059] Compared with the spike test Lindauer method, the traditional spike test method can detect mutations in the data, and is relatively easy to implement, but due to the large standard range, a large part of the spike values cannot be effectively detected.

[0060] In one or more embodiments, for the above method, the embodiment provides a lake water quality monitoring data quality control system, comprising:

[0061] a data acquisition unit configured to acquire lake water quality monitoring data and perform corresponding preprocessing;

[0062] a data classification unit configured to classify the preprocessed lake water quality monitoring data according to whether a special event occurs, to obtain lake water quality monitoring data corresponding to the special event and lake water quality monitoring data corresponding to a non-special event, wherein the special event is an event in which lake water quality anomaly occurs due to human factors and environmental factors;

[0063] an initial abnormal data acquisition unit configured to perform seasonal outlier analysis on the lake water quality monitoring data corresponding to the non-special event to obtain initial abnormal data, wherein the seasonal outlier analysis specifically comprises: performing seasonal division on the lake water quality monitoring data, performing 3σ analysis and box plot analysis on the data of each season respectively to obtain the initial abnormal data; and performing independent analysis on the lake water quality monitoring data corresponding to the special event to determine the initial abnormal data;

[0064] an outlier elimination unit configured to obtain true outliers and eliminate the outliers based on a hybrid test strategy for the obtained initial abnormal data;

[0065] a missing value filling unit configured to fill missing values based on adjacent data of the missing values for original missing values in the lake water quality monitoring data and missing values generated due to the elimination of the outliers, wherein the lake water quality monitoring data corresponding to the special event is independently analyzed to determine abnormal data and to perform elimination and missing value filling of the abnormal data.

[0066] It can be understood that the system described in the embodiment corresponds to the method described in the above embodiment, and the technical details have been described in detail in Embodiment One, which will not be repeated here.

[0067] In more embodiments, there is also provided:

[0068] An electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the method described in the above embodiments is completed. For brevity, this will not be repeated here.

[0069] It should be understood that the processor in the embodiments can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0070] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0071] A computer readable storage medium for storing computer instructions, the computer instructions being executed by a processor to complete the method described in the above embodiments.

[0072] A computer program product comprising a computer program, the computer program being executed by a processor to implement the method described in the above embodiments.

[0073] The method in the above embodiments can be directly embodied as a hardware processor to complete, or be completed by a combination of hardware and software modules in the processor. The software modules can be located in storage media in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, or the like. The storage medium is located in the memory, and the processor reads information in the memory to combine hardware to complete the steps of the above method. To avoid repetition, no longer detailed description here.

[0074] Those skilled in the art can realize that the units of the examples described in combination with the embodiments, i.e. the algorithm steps, can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software mode depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0075] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for quality control of lake water quality monitoring data, characterized in that, include: Acquire lake water quality monitoring data and perform appropriate preprocessing; The pre-processed lake water quality monitoring data is classified according to whether a special event has occurred, to obtain lake water quality monitoring data corresponding to special events and lake water quality monitoring data corresponding to non-special events. The special events are events that cause abnormal lake water quality due to human factors and environmental factors. Seasonal outlier analysis is performed on lake water quality monitoring data corresponding to non-special events to obtain initial outlier data. Specifically, the seasonal outlier analysis involves: dividing the lake water quality monitoring data into seasonal categories; calculating the mean and standard deviation for each season; constructing a first initial outlier dataset based on monitoring data whose deviation from the mean exceeds three times the standard deviation; performing box plot analysis on the lake water quality monitoring data for each season to determine the upper quartile, lower quartile, and interquartile range; determining a second initial outlier dataset based on the upper quartile, lower quartile, and interquartile range; and obtaining initial outlier data based on the intersection of the first and second initial outlier datasets. In addition, independent analysis of lake water quality monitoring data corresponding to special events is conducted to determine initial abnormal data; For the initial abnormal data obtained, the true outliers are obtained and outliers are removed based on a hybrid testing strategy. Specifically, based on the initial abnormal data, the range test, the mean square error test, and the peak test (Lynda method) are performed respectively to obtain a first abnormal dataset, a second abnormal dataset, and a third abnormal dataset. The true abnormal data are obtained based on the union of the first abnormal dataset, the second abnormal dataset, and the third abnormal dataset. For the missing values ​​in the original lake water quality monitoring data and the missing values ​​caused by outlier removal, the missing values ​​are filled in based on the neighboring data of the missing values.

2. The method for quality control of lake water quality monitoring data as described in claim 1, characterized in that, The special events mentioned include, but are not limited to, pollution events or red tide events.

3. The method for quality control of lake water quality monitoring data as described in claim 1, characterized in that, Missing value completion is performed based on neighboring data of missing values. Specifically, a preset number of neighboring data of missing value data are determined based on the K-nearest neighbor algorithm, and missing value completion is performed based on the mean of the neighboring data.

4. The method for quality control of lake water quality monitoring data as described in claim 3, characterized in that, The determination of neighboring data is based on the positional relationship in the time series of monitoring data, combined with the K-nearest neighbor algorithm, or by extracting multi-dimensional features from the monitoring data and determining neighboring data based on the similarity calculation results between different monitoring data.

5. A lake water quality monitoring data quality control system, characterized in that, include: The data acquisition unit is used to acquire lake water quality monitoring data and perform corresponding preprocessing. The data classification unit is used to classify the pre-processed lake water quality monitoring data according to whether a special event has occurred, and to obtain the lake water quality monitoring data corresponding to the special event and the lake water quality monitoring data corresponding to the non-special event. The special event is an event in which lake water quality abnormalities occur due to human factors and environmental factors. An initial anomaly data acquisition unit is used to perform seasonal anomaly analysis on lake water quality monitoring data corresponding to non-special events to obtain initial anomaly data. Specifically, the seasonal anomaly analysis involves: dividing the lake water quality monitoring data into seasonal categories; calculating the mean and standard deviation for each season's lake water quality monitoring data; constructing a first initial anomaly dataset based on monitoring data whose deviation from the mean exceeds three times the standard deviation; performing box plot analysis on the lake water quality monitoring data for each season to determine the upper quartile, lower quartile, and interquartile range; determining a second initial anomaly dataset based on the upper quartile, lower quartile, and interquartile range; and obtaining the initial anomaly data based on the intersection of the first and second initial anomaly datasets. Furthermore, the system independently analyzes lake water quality monitoring data corresponding to special events to determine initial anomaly data; the outlier removal unit is used to obtain true anomalies and remove them based on a hybrid testing strategy, specifically: based on the obtained initial anomaly data, the system performs range testing, mean square error testing, and peak testing (Lyndahl method) to obtain a first anomaly dataset, a second anomaly dataset, and a third anomaly dataset; the true anomaly data is obtained based on the union of the first, second, and third anomaly datasets. The missing value filling unit is used to fill in the missing values ​​in the original lake water quality monitoring data and the missing values ​​caused by outlier removal, based on the neighboring data of the missing values.

6. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements a lake water quality monitoring data quality control method as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a lake water quality monitoring data quality control method as described in any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a lake water quality monitoring data quality control method as described in any one of claims 1-4.

Images