Lake water quality monitoring data quality control method and system

Abstract

The invention provides a lake water quality monitoring data quality control method and system, and the scheme comprises the steps: obtaining lake water quality monitoring data, and carrying out the corresponding preprocessing; classifying the preprocessed lake water quality monitoring data according to whether a special event occurs, and obtaining lake water quality monitoring data corresponding to the special event and lake water quality monitoring data corresponding to a non-special event; performing seasonal abnormal value analysis on the lake water quality monitoring data corresponding to the non-special event to obtain initial abnormal data; for the obtained initial abnormal data, a real abnormal value is obtained based on a hybrid inspection strategy, and abnormal value elimination is carried out; and for original missing values in the lake water quality monitoring data and missing values generated due to abnormal value elimination, performing missing value completion based on adjacent data of the missing values.

Description

Technical Field

[0001] The present invention belongs to the technical field of data quality control, and in particular relates to a lake water quality monitoring data quality control method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The buoy automatic monitoring system is one of the important technical means to achieve continuous water quality monitoring. Through continuous observation of the buoy, the characteristics of water quality changes have gradually changed from qualitative descriptions in the past to precise quantitative measurements. At the same time, with the long-term accumulation of monitoring data, these data will provide valuable basic information for conducting large-scale research on spatiotemporal changes in water environment. As an inland water body, lake water quality monitoring plays an important role in environmental protection and water resources management. The buoy automatic monitoring system can continuously provide water quality data, but due to the long-term operation of the equipment, some parameters may drift. In addition, due to the uniqueness of the lake environment, such as slow water flow, low water exchange capacity and significant seasonal changes, data quality control faces many challenges.

[0004] The inventors have found that the existing water quality data quality control methods mainly have the following problems: in the quality control process, many methods often involve artificially set detection standards or thresholds, which cannot be applied to all waters. Artificially set thresholds may also be too sensitive to the distribution pattern of the data, resulting in inaccurate judgment of outliers when the data distribution is uneven or there are extreme values. At the same time, such schemes rely on prior knowledge, that is, they need to rely on prior knowledge or historical data to set thresholds, which limits the universality and flexibility of the quality control method; secondly, the existing water quality data quality control method process is mainly aimed at marine water types, but due to the particularity of the lake environment, direct application to lake waters will result in poor data control effects. There is a lack of targeted and effective data quality control strategies for lake waters. Summary of the invention

[0005] The embodiments of the present invention provide a lake water quality monitoring data quality control method and system to solve the problem that traditional solutions require human participation and rely on prior knowledge, resulting in poor accuracy and applicability of quality control results; and the problem that the particularity of the lake environment cannot directly utilize existing methods.

[0006] According to a first aspect of an embodiment of the present invention, a method for controlling lake water quality monitoring data quality is provided, comprising:

[0007] Obtain lake water quality monitoring data and perform corresponding preprocessing;

[0008] The pre-processed lake water quality monitoring data are classified according to whether special events occur, and lake water quality monitoring data corresponding to special events and lake water quality monitoring data corresponding to non-special events are obtained, wherein the special events are events in which abnormal lake water quality occurs due to human factors and environmental factors;

[0009] Perform seasonal outlier analysis on lake water quality monitoring data corresponding to non-special events to obtain initial outlier data; wherein the seasonal outlier analysis specifically includes: seasonal division of lake water quality monitoring data, performing 3σ analysis and box plot analysis on the data of each season to obtain initial outlier data;

[0010] For the initial abnormal data obtained, based on the mixed inspection strategy, the real outliers are obtained and the outliers are eliminated;

[0011] For the original missing values ​​in the lake water quality monitoring data and the missing values ​​caused by the removal of outliers, the missing values ​​are filled based on the neighboring data of the missing values.

[0012] Furthermore, the 3σ analysis and box plot analysis are performed on the data of each season to obtain initial abnormal data, specifically: the mean and standard deviation are calculated for the lake water quality monitoring data of each season; a first initial abnormal data set is constructed based on the monitoring data of the lake water quality monitoring data whose deviation from the mean exceeds three times the standard deviation; a box plot analysis is performed on the lake water quality monitoring data of each season to determine the upper quartile, lower quartile and interquartile range, and a second initial abnormal data set is determined based on the upper quartile, lower quartile and interquartile range; initial abnormal data is obtained based on the intersection of the first initial abnormal data set and the second initial abnormal data set.

[0013] Furthermore, based on the hybrid inspection strategy, the real outlier value is obtained, specifically: based on the initial abnormal data obtained, the range test method, the mean square error test method and the peak test Rheinda method are respectively executed to obtain the first abnormal data set, the second abnormal data set and the third abnormal data set; based on the union of the first abnormal data set, the second abnormal data set and the third abnormal data set, the real abnormal data is obtained.

[0014] Furthermore, the special events include but are not limited to pollution events or red tide events.

[0015] Furthermore, missing values ​​are filled based on neighboring data of the missing values, specifically: a preset number of neighboring data of the missing value data is determined based on a K-nearest neighbor algorithm, and the missing values ​​are filled based on the mean of the neighboring data.

[0016] Furthermore, the neighbor data is determined based on the positional relationship in the monitoring data time series, combined with the K nearest neighbor algorithm to determine the neighbor data, or, by performing multi-dimensional feature extraction on the monitoring data, the neighbor data is determined based on the similarity calculation results between different monitoring data.

[0017] According to a second aspect of an embodiment of the present invention, a lake water quality monitoring data quality control system is provided, comprising:

[0018] A data acquisition unit, which is used to acquire lake water quality monitoring data and perform corresponding preprocessing;

[0019] A data classification unit, which is used to classify the pre-processed lake water quality monitoring data according to whether a special event occurs, and obtain lake water quality monitoring data corresponding to a special event and lake water quality monitoring data corresponding to a non-special event, wherein the special event is an event in which abnormal lake water quality occurs due to human factors and environmental factors;

[0020] An initial abnormal data acquisition unit is used to perform seasonal abnormal value analysis on lake water quality monitoring data corresponding to non-special events to obtain initial abnormal data; wherein the seasonal abnormal value analysis specifically comprises: seasonal division of lake water quality monitoring data, performing 3σ analysis and box plot analysis on data of each season respectively, to obtain initial abnormal data; and, independently analyzing lake water quality monitoring data corresponding to special events to determine initial abnormal data;

[0021] An outlier elimination unit is used to obtain the real outlier value and eliminate the outlier value based on the hybrid inspection strategy for the obtained initial outlier data;

[0022] The missing value filling unit is used to fill in the original missing values ​​in the lake water quality monitoring data and the missing values ​​caused by the removal of outliers based on the neighboring data of the missing values.

[0023] According to a third aspect of an embodiment of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored and running on the memory, wherein when the processor executes the program, the method for quality control of lake water quality monitoring data is implemented.

[0024] According to a fourth aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for quality control of lake water quality monitoring data is implemented.

[0025] According to a fifth aspect of an embodiment of the present invention, there is provided a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method for quality control of lake water quality monitoring data is implemented.

[0026] One or more of the above technical solutions have the following beneficial effects:

[0027] (1) The present invention provides a lake water quality monitoring data quality control method and system. The scheme does not require manual setting of threshold parameters and does not require the assistance of prior knowledge. Based on the particularity of the lake environment, the scheme adopts seasonal anomaly analysis to divide the data into seasonal categories and judge the initial abnormal data separately. The scheme also performs secondary verification on the initial abnormal values ​​obtained through a hybrid verification strategy, effectively ensuring the accuracy of abnormal data identification, and supplementing the abnormal data based on its neighborhood data, effectively ensuring the data quality of the lake water quality monitoring data.

[0028] (2) The scheme described in the present invention classifies the lake water quality monitoring data according to whether a special event occurs and independently analyzes the data under the special event, thereby effectively avoiding the situation where the data under the special event is mistakenly identified as abnormal data, thereby improving the adaptability of the scheme described in the present invention.

[0029] (3) The scheme described in the present invention can adapt to inland lake water types with complex water quality characteristics while ensuring the accuracy of data quality control, and has better data quality control effect, which is of great significance for the future development of inland lake buoy data quality control.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 This is a flow chart of a lake water quality monitoring data quality control method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0034] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0035] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0036] like Figure 1 As shown, an embodiment of the present invention provides a lake water quality monitoring data quality control method, comprising:

[0037] Step 1: Obtain lake water quality monitoring data and perform corresponding preprocessing;

[0038] In a specific implementation, the preprocessing specifically includes finding duplicate data, empty records and missing values, deleting duplicate records, and marking empty records and missing values.

[0039] Step 2: The pre-processed lake water quality monitoring data are classified according to whether a special event occurs, and lake water quality monitoring data corresponding to a special event and lake water quality monitoring data corresponding to a non-special event are obtained, wherein the special event is an event in which abnormal lake water quality occurs due to human factors and environmental factors;

[0040] In a specific implementation, the special events include but are not limited to pollution events or red tide events, and the data during the periods when these special events occur are independently analyzed.

[0041] Step 3: Perform seasonal outlier analysis on the lake water quality monitoring data corresponding to non-special events to obtain initial outlier data; wherein the seasonal outlier analysis specifically includes: seasonal division of the lake water quality monitoring data, performing 3σ analysis and box plot analysis on the data of each season to obtain initial outlier data;

[0042] In a specific implementation, 3σ analysis and box plot analysis are performed on the data of each season to obtain initial abnormal data, specifically: the mean and standard deviation of the lake water quality monitoring data of each season are calculated; a first initial abnormal data set is constructed based on the monitoring data of the lake water quality monitoring data whose deviation from the mean exceeds three times the standard deviation; a box plot analysis is performed on the lake water quality monitoring data of each season to determine the upper quartile, lower quartile and interquartile range, and a second initial abnormal data set is determined based on the upper quartile, lower quartile and interquartile range; initial abnormal data is obtained based on the intersection of the first initial abnormal data set and the second initial abnormal data set.

[0043] Specifically, seasonal 3σ analysis is as follows:

[0044] Under the original 3σ principle, outliers are often defined as values ​​in a group of measured values ​​that deviate from the mean by more than 3 times the standard deviation. However, in inland lakes, due to the slow flow and limited water exchange capacity, the water quality changes significantly with seasonality. The same data may produce different results when analyzed in different seasons. Therefore, the data are divided into seasons or months according to time information, and seasonal 3σ analysis is performed separately.

[0045] The seasonal box plot analysis is as follows:

[0046] Box plots are used to identify outliers, which are usually defined as values ​​less than QL-1.5IOR or QU+1.5IQR, where QL is the lower quartile, QU is the upper quartile, and IQR is the interquartile range, expressed as QU-QL. The data are divided into seasons or months according to time information, and seasonal box plot analysis is performed separately.

[0047] Step 4: For the initial abnormal data obtained, based on the hybrid inspection strategy, obtain the true outliers and remove the outliers;

[0048] In a specific implementation, based on a hybrid test strategy, the true outlier value is obtained, specifically: based on the initial abnormal data obtained, the range test method, the mean square error test method and the peak test Rheinland method are respectively executed to obtain the first abnormal data set, the second abnormal data set and the third abnormal data set; based on the union of the first abnormal data set, the second abnormal data set and the third abnormal data set, the true abnormal data is obtained.

[0049] Specifically, the range test method is as follows: The purpose of this method is to conduct a preliminary screening of the original observation data based on the range of the observation element in the target area, and to mark the data outside the regional range as abnormal quality control. Through the analysis of historical data and other observation data, if the data is detected to be outside the regional range, it is marked as abnormal data. The processing process is: according to the historical measurement data of a certain water quality observation element or other public data sets, the threshold range is set, and the data outside the regional range is identified as abnormal data.

[0050] The specific mean square error test is as follows: when the value at a certain moment is greater than the standard deviation of the average value of the day before and after the moment by n times, it is regarded as an outlier, where n is 5; a certain moment is defined as the t moment of the dth day, and the day before and after the moment is defined as the t moment of the d-1th day to the t moment of the d+1th day;

[0051] The specific formula of the spike test Rheinda method is as follows: i =(X1, X2, X3, ..., X N ), X i The previous adjacent data is Xi-1 , P i =|X i -X i-1 |, where P i is the neighbor difference, assuming P0 is 0, when P i and P i+1 When the residuals of are all greater than 3σ, that is, |P i -P|>3σ and |P i+1 -P|>3σ, then it is considered that X i The data is abnormal, where P is the mean and σ is the standard deviation.

[0052] Step 5: For the original missing values ​​in the lake water quality monitoring data and the missing values ​​caused by the removal of outliers, the missing values ​​are filled based on the neighboring data of the missing values.

[0053] In a specific implementation, missing values ​​are filled based on the neighboring data of the missing value, specifically: a preset number of neighboring data of the missing value data is determined based on a K-nearest neighbor algorithm, and the missing value is filled based on the mean of the neighboring data.

[0054] Specifically, in this embodiment, the nearest neighbor is determined based on the positional relationship in the monitoring data time series. In more implementations, multi-dimensional feature extraction can be performed on the monitoring data, and the nearest neighbor data can be determined based on the similarity calculation results between different monitoring data.

[0055] It can be understood that the neighbor data can be determined according to actual needs.

[0056] In one or more embodiments, the initial abnormal data may also be obtained by using the Dixon test method, which is a non-parametric method that can effectively find abnormal observations in a data set.

[0057] Compared with seasonal 3σ analysis and seasonal box plot analysis, the Dixon test can effectively and quickly detect outliers in the data, and is relatively simple, but it has requirements on the number of samples, and the test needs to refer to the threshold setting.

[0058] In one or more embodiments, in the hybrid inspection strategy, the peak inspection Rheinland method can be replaced by the traditional peak inspection method, that is, the observed element is subjected to a peak inspection, and if a large mutation occurs, it is abnormal. The inspection value operation is performed on the observed element, and if the inspection value exceeds the standard range, it is determined to be an abnormal value.

[0059] Compared with the spike test Rheinda method, the traditional spike test method can detect sudden changes in the data and is relatively easy to implement. However, due to the large standard range, a large part of the spike values ​​cannot be effectively detected.

[0060] In one or more implementations, with respect to the above method, this embodiment provides a lake water quality monitoring data quality control system, including:

[0061] A data acquisition unit, which is used to acquire lake water quality monitoring data and perform corresponding preprocessing;

[0062] A data classification unit, which is used to classify the pre-processed lake water quality monitoring data according to whether a special event occurs, and obtain lake water quality monitoring data corresponding to a special event and lake water quality monitoring data corresponding to a non-special event, wherein the special event is an event in which abnormal lake water quality occurs due to human factors and environmental factors;

[0063] An initial abnormal data acquisition unit is used to perform seasonal abnormal value analysis on lake water quality monitoring data corresponding to non-special events to obtain initial abnormal data; wherein the seasonal abnormal value analysis specifically comprises: seasonal division of lake water quality monitoring data, performing 3σ analysis and box plot analysis on data of each season respectively, to obtain initial abnormal data; and, independently analyzing lake water quality monitoring data corresponding to special events to determine initial abnormal data;

[0064] An outlier elimination unit is used to obtain the real outlier value and eliminate the outlier value based on the hybrid inspection strategy for the obtained initial outlier data;

[0065] The missing value filling unit is used to fill in the missing values ​​in the lake water quality monitoring data and the missing values ​​caused by the removal of outliers, based on the neighboring data of the missing values; wherein, the lake water quality monitoring data corresponding to special events are independently analyzed to determine the abnormal data and execute the removal of abnormal data and the missing value filling.

[0066] It can be understood that the system described in this embodiment corresponds to the method described in the above embodiment, and its technical details have been described in detail in Embodiment 1 and will not be repeated here.

[0067] In further embodiments, there is also provided:

[0068] An electronic device includes a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the above embodiment. For the sake of brevity, no further description is given here.

[0069] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0070] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0071] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in the above embodiment is completed.

[0072] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the above embodiment is implemented.

[0073] The method in the above embodiment can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0074] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A lake water quality monitoring data quality control method, characterized in that: include: Obtain lake water quality monitoring data and perform corresponding preprocessing; The pre-processed lake water quality monitoring data are classified according to whether special events occur, and lake water quality monitoring data corresponding to special events and lake water quality monitoring data corresponding to non-special events are obtained, wherein the special events are events in which abnormal lake water quality occurs due to human factors and environmental factors; Perform seasonal outlier analysis on lake water quality monitoring data corresponding to non-special events to obtain initial outlier data; wherein the seasonal outlier analysis specifically includes: seasonal division of lake water quality monitoring data, performing 3σ analysis and box plot analysis on the data of each season to obtain initial outlier data; For the initial abnormal data obtained, based on the mixed inspection strategy, the real outliers are obtained and the outliers are eliminated; For the original missing values ​​in the lake water quality monitoring data and the missing values ​​caused by the removal of outliers, the missing values ​​are filled based on the neighboring data of the missing values.

2. A lake water quality monitoring data quality control method as claimed in claim 1, characterized in that: The 3σ analysis and box plot analysis are respectively performed on the data of each season to obtain the initial abnormal data, specifically: the mean and standard deviation are calculated for the lake water quality monitoring data of each season; based on the monitoring data of the lake water quality monitoring data whose deviation from the mean exceeds three times the standard deviation, a first initial abnormal data set is constructed; for the lake water quality monitoring data of each season, a box plot analysis is performed to determine the upper quartile, the lower quartile and the interquartile range, and based on the upper quartile, the lower quartile and the interquartile range, a second initial abnormal data set is determined; Initial abnormal data is obtained based on the intersection of the first initial abnormal data set and the second initial abnormal data set.

3. A lake water quality monitoring data quality control method as claimed in claim 1, characterized in that: Based on the hybrid test strategy, the real outliers are obtained, specifically: based on the obtained initial outlier data, the range test method, the mean square error test method and the spike test Rheinda method are respectively performed to obtain the first outlier data set, the second outlier data set and the third outlier data set; Based on the union of the first abnormal data set, the second abnormal data set and the third abnormal data set, real abnormal data is obtained.

4. A lake water quality monitoring data quality control method as claimed in claim 1, characterized in that: The special events include, but are not limited to, pollution events or red tide events.

5. A lake water quality monitoring data quality control method as claimed in claim 1, characterized in that: The missing value is filled based on the neighboring data of the missing value, specifically: a preset number of neighboring data of the missing value data is determined based on the K nearest neighbor algorithm, and the missing value is filled based on the mean value of the neighboring data.

6. A lake water quality monitoring data quality control method as claimed in claim 5, characterized in that: The neighbor data is determined based on the positional relationship in the monitoring data time series, combined with the K nearest neighbor algorithm to determine the neighbor data, or, by performing multi-dimensional feature extraction on the monitoring data, the neighbor data is determined based on the similarity calculation results between different monitoring data.

7. A lake water quality monitoring data quality control system, characterized in that: include: A data acquisition unit, which is used to acquire lake water quality monitoring data and perform corresponding preprocessing; A data classification unit, which is used to classify the pre-processed lake water quality monitoring data according to whether a special event occurs, and obtain lake water quality monitoring data corresponding to a special event and lake water quality monitoring data corresponding to a non-special event, wherein the special event is an event in which abnormal lake water quality occurs due to human factors and environmental factors; An initial abnormal data acquisition unit is used to perform seasonal abnormal value analysis on lake water quality monitoring data corresponding to non-special events to obtain initial abnormal data; wherein the seasonal abnormal value analysis specifically comprises: seasonal division of lake water quality monitoring data, performing 3σ analysis and box plot analysis on data of each season respectively, to obtain initial abnormal data; and, independently analyzing lake water quality monitoring data corresponding to special events to determine initial abnormal data; An outlier elimination unit is used to obtain the real outlier value and eliminate the outlier value based on the hybrid inspection strategy for the obtained initial outlier data; The missing value filling unit is used to fill in the original missing values ​​in the lake water quality monitoring data and the missing values ​​caused by the removal of outliers based on the neighboring data of the missing values.

8. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, a lake water quality monitoring data quality control method as described in any one of claims 1-6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a lake water quality monitoring data quality control method as described in any one of claims 1-6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, a lake water quality monitoring data quality control method as described in any one of claims 1 to 6 is implemented.

Images