Drinking water detection result calibration method based on multi-source data

Abstract

The invention discloses a drinking water detection result calibration method based on multi-source data, and relates to the technical field related to water quality detection.The method comprises the steps that the multi-source data are collected, and the multi-source data comprise laboratory data and online monitoring data; aligning the laboratory data and the online monitoring data according to the data acquisition time, space and index consistency relationship, and constructing a data alignment matrix; according to the data alignment matrix, contrast data is extracted for deviation analysis, error sources are determined, and the error sources comprise a system error source and a random error source; based on an error source, performing calibration learning according to the contrast data, and constructing a calibration model; and deploying the calibration model in a detection system to calibrate a detection result. The technical problem that in the prior art, effective butt joint of laboratory detection data and actual monitoring data cannot be achieved, data integration is improper, and consequently the error recognition and calibration accuracy is insufficient is solved, and the technical effect of improving the accuracy and reliability of the drinking water detection result is achieved.

Description

Technical Field

[0001] The present application relates to the technical field related to water quality detection, and in particular to a method for calibrating drinking water detection results based on multi-source data. Background Art

[0002] Drinking water can be monitored and tested in real time through a variety of means to ensure its safety. However, although the existing water quality testing has become mature, there are still some challenges, especially in the integration of different types of monitoring data, data accuracy, calibration of detection errors, etc. The traditional water quality monitoring process relies on laboratory test data and online monitoring data. Laboratory data usually has high accuracy and precision, but its disadvantages are low sampling frequency, complex collection process, time lag, etc.; while online monitoring data has the advantages of strong real-time and frequent data collection, but it is affected by factors such as sensor accuracy and environmental interference, and may produce large errors; how to effectively integrate laboratory data and online monitoring data and reduce the impact of errors is a key link in current drinking water testing.

[0003] At the current stage, relevant technologies have technical problems such as the inability to achieve effective connection between laboratory test data and actual monitoring data and improper data integration, which in turn leads to insufficient accuracy of error identification and calibration. Summary of the invention

[0004] This application solves the technical problems in the prior art of being unable to achieve effective connection between laboratory test data and actual monitoring data and improper data integration, which in turn leads to insufficient accuracy of error identification and calibration, by providing a drinking water test result calibration method based on multi-source data, thereby achieving the technical effect of improving the accuracy and reliability of drinking water test results.

[0005] The present application provides a drinking water test result calibration method based on multi-source data, including: collecting multi-source data, including laboratory data and online monitoring data; aligning the laboratory data and online monitoring data according to the data collection time, space and indicator consistency relationship, and constructing a data alignment matrix; extracting control data according to the data alignment matrix to perform deviation analysis and determine the source of error, wherein the error source includes systematic error and random error source; based on the error source, performing calibration learning according to the control data and constructing a calibration model; and deploying the calibration model in the detection system to calibrate the detection results.

[0006] In a possible implementation, the drinking water test result calibration method based on multi-source data also performs the following processing: aligning the laboratory data and online monitoring data according to the consistency of the data acquisition time and space, and establishing a spatiotemporal acquisition relationship; converting the units of the spatiotemporal acquisition relationship and marking the test indicators according to the indicator consistency target, and establishing an indicator alignment relationship; aligning the test indicators of the indicator alignment relationship according to the indicator marking, and constructing the data alignment matrix.

[0007] In a possible implementation, the drinking water test result calibration method based on multi-source data also performs the following processing: calculating the difference between the laboratory data and the online monitoring data based on the control data to obtain the absolute error; calculating the residual standard deviation between the laboratory data and the online monitoring data based on the control data to obtain the relative error; determining the systematic error based on the absolute error, and determining the random error based on the relative error.

[0008] In a possible implementation, the drinking water test result calibration method based on multi-source data also performs the following processing: extracting the control data based on the test index, and arranging the control data in time series according to the data acquisition time; extracting the time series arranged control data according to the set time window according to the data acquisition time, and obtaining a test index deviation analysis sample; calculating the absolute error and relative error of the time series data based on the test index deviation analysis sample, and obtaining the source of the time series error of the test index.

[0009] In a possible implementation, the drinking water detection result calibration method based on multi-source data further performs the following processing: determining the indicator error increment according to the timing error source of the detection indicator; and fitting the indicator error increment to the error source.

[0010] In a possible implementation, the drinking water detection result calibration method based on multi-source data also performs the following processing: setting a dynamic time window, the dynamic time window including a time step for adjusting the time length of the time window; calculating the error sources of the control data based on the dynamic time window, and using the error source calculation results to fit the long and short time series deviations to obtain error trend characteristics; adding the error trend characteristics to the indicator error increment.

[0011] In a possible implementation, the drinking water detection result calibration method based on multi-source data also performs the following processing: constructing a linear model for the systematic error, and training the linear model by constructing a systematic error training set to obtain a systematic error calibration model, and the systematic error calibration model is used to calibrate and analyze the systematic error and output calibration parameters; constructing a random forest model for the random error, and obtaining a random error calibration model by constructing a random error training and test set for model training convergence; integrating the systematic error calibration model and the random error calibration model to obtain the calibration model.

[0012] In a possible implementation, the drinking water detection result calibration method based on multi-source data also performs the following processing: respectively perform performance evaluation on the systematic error calibration model and the random error calibration model, including mean square error, root mean square error, absolute error mean, and goodness of fit; construct a performance matrix according to the performance evaluation results of the systematic error calibration model and the random error calibration model, wherein the rows of the performance matrix represent model categories and the columns are performance indicators; remove the indicator dimension of the performance matrix and assign weights to each performance indicator; calculate the performance distance of the systematic error calibration model and the random error calibration model based on the performance matrix and the weights of each performance indicator; determine the fusion weights of the systematic error calibration model and the random error calibration model according to the performance distance, and integrate the systematic error calibration model and the random error calibration model based on the fusion weights to obtain the calibration model.

[0013] The method for calibrating drinking water test results based on multi-source data proposed in this application is intended to collect multi-source data, including laboratory data and online monitoring data; align laboratory data and online monitoring data according to data collection time, space and indicator consistency, and construct a data alignment matrix; extract control data according to the data alignment matrix for deviation analysis to determine the source of error, which includes systematic error and random error sources; based on the error source, perform calibration learning based on the control data to construct a calibration model; deploy the calibration model in the detection system to calibrate the test results. This solves the technical problems in the prior art that the effective connection between laboratory test data and actual monitoring data and improper data integration lead to insufficient error identification and calibration accuracy, and achieves the technical effect of improving the accuracy and reliability of drinking water test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0015] Figure 1 A schematic diagram of a flow chart of a drinking water test result calibration method based on multi-source data provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of the process of constructing a data alignment matrix in the drinking water test result calibration method based on multi-source data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0018] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0019] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of the objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, product, or server comprising a series of steps is not necessarily limited to those steps clearly listed, but may include other steps that are not clearly listed or inherent to these processes, methods, products, or devices, and unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field to which this application belongs. The terms used herein are for the purpose of describing the embodiments of this application only.

[0020] The present application embodiment provides a drinking water test result calibration method based on multi-source data, such as Figure 1 As shown, the method includes:

[0021] Step S100, collecting multi-source data, including laboratory data and online monitoring data.

[0022] Preferably, in the process of drinking water testing, water quality information is obtained through different data sources to ensure the comprehensiveness and accuracy of the data. The data sources include laboratory data and online monitoring data. Specifically, laboratory data refers to data obtained by manual sampling and sending it to the laboratory for analysis and testing. It usually has high accuracy and precision and is suitable for some water quality indicators that require fine measurement. Common water quality analysis items include pH value, dissolved oxygen (DO), suspended solids (SS), heavy metals (such as lead, mercury, copper, etc.), bacteria, virus detection (such as Escherichia coli, total bacterial count, etc.), chemical oxygen demand (COD), biochemical oxygen demand (BOD) and pesticide residues, etc. Laboratory data usually takes a certain amount of time to process after a sampling, and the data update frequency is low; online monitoring data refers to the data of various water quality parameters in drinking water collected and recorded continuously and in real time through real-time sensors or automated monitoring equipment. These devices are generally installed in water sources, water supply networks or terminal water plants. Compared with laboratory data, online monitoring data has higher real-time performance and can be used for real-time alarm, data analysis and automatic processing. Common indicators of online monitoring equipment include water temperature, turbidity, oxidation-reduction potential (ORP), conductivity, pH value (real-time monitoring), dissolved oxygen (DO), total nitrogen (TN) and total phosphorus (TP), as well as chloride, fluoride, etc. The collection and integration of these two types of data can provide more comprehensive water quality information. Laboratory data can provide more accurate analysis results, while online monitoring data can provide real-time and dynamic water quality conditions. The integration, comparison and calibration of these two types of data can effectively improve the accuracy and reliability of drinking water testing.

[0023] Step S200, aligning the laboratory data and online monitoring data according to the data collection time, space and indicator consistency relationship to construct a data alignment matrix.

[0024] Preferably, laboratory data and online monitoring data are connected and synchronized in time, space and monitoring indicators through specific methods to ensure that data from different sources can be effectively compared and analyzed. Specifically, time alignment refers to ensuring that the timestamps of laboratory data and online monitoring data can correspond. Since laboratory data are usually collected periodically and there may be sampling time delays, and online monitoring data are collected in real time, it is necessary to match data at different time points through the time axis, fill in the gaps in laboratory data collection through interpolation (such as linear interpolation or time window aggregation), or match the timestamps of online monitoring data according to the collection time points; spatial alignment refers to spatially unifying laboratory data and online monitoring data from different monitoring locations. The water quality of drinking water will vary due to factors such as geographical location and water supply method. Spatial alignment is performed to ensure that water quality data from different locations can be compared at the same spatial coordinates. The data can be spatially mapped according to the geographical coordinates of the monitoring points (such as longitude and latitude, altitude, etc.). If the laboratory If the sampling points of the laboratory data and the online monitoring data are not exactly the same, spatial interpolation (such as Kriging interpolation or inverse distance weighted method) can be used to spatially complete the data; the indicator consistency relationship refers to ensuring the uniformity and consistency of the laboratory data and the online monitoring data in terms of monitoring indicators. Different data sources may monitor similar or different indicators. Therefore, it is necessary to establish a standard indicator comparison relationship to ensure the comparability of the data. For the same or similar water quality indicators, such as pH value, dissolved oxygen, etc., it is necessary to ensure that the units and measurement methods of the two are consistent. If there are differences, they need to be adjusted through conversion or standardization; by aligning the laboratory data and the online monitoring data, a data alignment matrix is ​​constructed, in which the rows of the data alignment matrix represent time, space or monitoring points, and the columns represent various monitoring indicators. Each item represents a one-to-one correspondence between laboratory data and online monitoring data at a specific time, space or location, that is, in the data matrix, each column represents a water quality indicator (such as pH value, dissolved oxygen, chlorine content, etc.), and each row represents the corresponding data at a time point or spatial location. Through this data alignment matrix, multi-source data from laboratories and online monitoring are uniformly organized and mapped according to the consistent relationship of time, space and indicators, so that the data can be compared, analyzed and calibrated within the same framework, and the differences between different data sources can be analyzed for deviations and error sources can be located, which helps to improve the accuracy and reliability of test results.

[0025] Further, such as Figure 2As shown, step S200 also includes step S210, aligning the laboratory data and online monitoring data according to the consistency of the data collection time and space to establish a spatiotemporal collection relationship; step S220, converting the units of the spatiotemporal collection relationship and detecting indicator marking according to the indicator consistency target to establish an indicator alignment relationship; step S230, aligning the indicator alignment relationship according to the indicator marking to construct the data alignment matrix.

[0026] Preferably, the laboratory data and the online monitoring data are sorted and matched according to the correspondence between time and space to ensure that the data can be compared at the same time point and spatial location. Specifically, the collection time of the laboratory data and the online monitoring data is determined, and the data with different time collection frequencies are unified. For example, the high-frequency data of the online monitoring data is aggregated through a time window (such as hours, days), and matched with the collection time point of the laboratory data; determine whether the geographical locations of the laboratory data collection point and the online monitoring equipment installation point are consistent. If there is an offset, it is necessary to align the data through a spatial mapping method (such as an interpolation algorithm or a coordinate transformation) to ensure that the laboratory data and the online monitoring data belong to the same spatial location; and then establish a spatiotemporal collection relationship, including the time point, spatial location and the matching relationship between the two types of data.

[0027] Preferably, after the spatiotemporal alignment is completed, the specific detection indicators of the data are processed for consistency to ensure that they can be directly compared under the same standards, including unit consistency conversion. If the units of laboratory data and online monitoring data are different (for example, dissolved oxygen is expressed in ppm and mg / L respectively), they need to be unified through conversion formulas, and different expressions of the same indicator (such as the mass concentration and molar concentration of ammonia nitrogen NH3-N) are converted; detection indicator marking is performed to mark the indicators of laboratory data and online monitoring data to ensure that each indicator corresponds to the same parameter, and that the meaning and unit of the same indicator in the two types of data are consistent, and then the indicator alignment relationship is output, that is, the alignment relationship after completing unit conversion and marking, and a one-to-one mapping of each indicator is established.

[0028] Preferably, after the establishment of the spatiotemporal acquisition relationship and the indicator alignment relationship is completed, the two types of data are integrated into a data alignment matrix by detecting the indicator alignment, so as to perform comparative analysis and deviation calibration. Specifically, according to the indicator label, the indicators of the laboratory data and the online monitoring data are filled into the corresponding positions. The rows of the matrix represent the time points, and the columns of the matrix represent the various monitoring indicators (such as NH3-N, temperature, pH value, dissolved oxygen, etc.). The values ​​in the matrix are the specific values ​​of the laboratory data and the online monitoring data at the corresponding time points and spatial positions. The values ​​of the laboratory data and the online monitoring data in the matrix are compared to ensure that their alignment relationship is correct. The exemplary data alignment matrix is ​​as follows:

[0029] Time Spatial location <![CDATA[NH3-N (Laboratory)]]> <![CDATA[NH3-N (Online Monitoring)]]> Temperature(℃) pH Dissolved oxygen (DO, mg / L) Chemical oxygen demand (COD, mg / L) Turbidity (NTU) 01 / 1009:00 S1 0.23 0.25 20 7.6 8.5 15 3.2 01 / 1109:00 S1 0.27 0.29 19 7.4 7.9 18 2.8 01 / 1209:00 S1 0.21 0.22 18 7.5 8.1 14 3.1

[0030] Step S300, extracting control data according to the data alignment matrix to perform deviation analysis and determine the error source, which includes systematic error and random error source.

[0031] Preferably, deviation analysis refers to the quantitative and qualitative analysis of the difference (i.e., deviation) between the laboratory data and the online monitoring data in the data alignment matrix. By comparing the two sets of data at the same time point, spatial location and index, the deviation value is calculated, and the trend, distribution and cause of the deviation are analyzed to determine the source of the error, which is usually divided into systematic error and random error. For example, by comparing the data deviations at multiple time points or spatial locations, if the deviation value always remains within a certain range or the trend is fixed, it can be determined as a systematic error; by fitting the laboratory data and the online monitoring data using the linear regression method, if there is a systematic deviation in the regression result (such as the intercept significantly deviates from 0), it can be determined as a systematic error. Specifically, the systematic error is an error caused by a fixed deviation in the measuring equipment, method or environment, which has regularity and predictability, and is usually manifested as a systematically high or low measurement result. Possible sources include sensor calibration problems (the sensors of the online monitoring equipment have not been calibrated for a long time, resulting in the measurement results systematically deviating from the true value), measurement method differences (experimental The standard method used in the laboratory is different from the method of the online monitoring equipment, such as chemical analysis and optical sensor detection), environmental interference (such as the equipment is affected by environmental factors such as temperature, humidity, pressure, etc., resulting in data deviation) and unit conversion error (inaccurate unit conversion in data processing, resulting in fixed deviation); random error is an error caused by accidental factors, which is usually random and unpredictable. Its value may be large or small in measurement. For example, comparing the distribution of deviation values ​​at multiple time points, if the deviation value has no fixed pattern but is randomly distributed and has a large fluctuation range, it may be a random error. Through statistical methods (such as variance analysis), it is judged whether the fluctuation range of the deviation conforms to the characteristics of random distribution. Possible sources include transient environmental changes (such as transient fluctuations of factors such as water flow rate and flow), short-term instability of equipment (short-term noise or fluctuation of online monitoring equipment or laboratory instruments), operation errors (irregular operation or human error during laboratory testing) and sampling problems (uneven distribution of particulate matter in water quality, resulting in sampling differences between different batches). Through data deviation analysis, the specific cause of the error is finally located to provide support for subsequent data calibration.

[0032] Furthermore, step S300 also includes step S310, calculating the difference between the laboratory data and the online monitoring data based on the control data to obtain the absolute error; step S320, calculating the residual standard deviation between the laboratory data and the online monitoring data based on the control data to obtain the relative error; step S330, determining the systematic error based on the absolute error, and determining the random error based on the relative error.

[0033] Preferably, the difference between the laboratory data and the online monitoring data is calculated based on the control data to obtain the absolute error. The absolute error is the basis for judging the systematic error, which is usually manifested as a fixed deviation between the laboratory data and the online monitoring data. For example, the online monitoring data is always higher or lower than the laboratory data by a certain fixed value. If the absolute error shows a consistent deviation trend at multiple data points, that is, the error value is relatively stable (such as the above results are all around 0.02), it indicates that there is a systematic error; the residual standard deviation of the laboratory data and the online monitoring data is calculated based on the control data, and the formula is as follows:

[0034] Residual = laboratory data - online monitoring data;

[0035] Residual standard deviation ;

[0036] in, is the number of data samples, is the mean value of the residuals; the residual standard deviation is used to judge the random error, reflecting the volatility of the error between the laboratory data and the online monitoring data. If the standard deviation is large, the random error is large; finally, the absolute error is used to determine the systematic error, that is, if the absolute error value is basically stable, the systematic error is determined, and the random error is determined by the residual standard deviation, that is, the error between the laboratory and online monitoring data is volatile, then the random error is determined, among which the random error is manifested as random fluctuations between the data and cannot be predicted. Through error analysis, the error sources of laboratory data and online monitoring data can be comprehensively determined.

[0037] Furthermore, step S300 also includes step S340, extracting the control data based on the detection index, and chronologically arranging the control data according to the data acquisition time; step S350, extracting the time-series arranged control data according to the set time window according to the data acquisition time, and obtaining the detection index deviation analysis sample; step S360, calculating the absolute error and relative error of the time series data according to the detection index deviation analysis sample, and obtaining the source of the time series error of the detection index.

[0038] Preferably, data related to the target detection index are extracted from the data alignment matrix, and these data are arranged according to time points to form a time series, the purpose of which is to integrate the data into a time dimension for trend and error analysis. Specifically, the index to be analyzed (NH3-N, pH value, dissolved oxygen, etc.) is selected, and the laboratory data and online monitoring data of the index are extracted from the data alignment matrix. According to the data collection time (such as the specific year / month / day / time), the laboratory data and online monitoring data are arranged in chronological order; then the time-series arranged data are arranged according to the set time window (such as 1 hour, 1 day, 1 week). The deviations of water quality testing experiments are identified and determined according to the data changes of continuous experiments, that is, the absolute and relative errors of the time series data are calculated according to the detection index deviation analysis samples, and the sources of the errors in the time dimension are analyzed in combination with the calculation results of the absolute and relative errors, so as to distinguish between systematic errors and random errors, clarify their time series distribution characteristics, and thus more clearly reveal the dynamic changes of errors in time.

[0039] Furthermore, step S360 also includes step S361, determining the indicator error increment according to the timing error source of the detection indicator; and step S362, fitting the indicator error increment to the error source.

[0040] Preferably, the error of the experimental detection equipment is identified based on the continuous error and then added to the error source, that is, the error source compensation of the equipment is considered when calibrating, and the indicator error increment is determined based on the timing error source of the detection indicator, that is, the error change amount caused by the system error or random error at different time points or time windows of the detection indicator. If the incremental value is small and stable, it may be mainly caused by the system error. If the incremental value changes greatly, it may be mainly caused by the random error. The data pattern (incremental trend) of the indicator error increment is fit to the error source to verify whether the source of the error is consistent with the characteristics of the system error or random error, and the online monitoring data is compensated for the error to reduce the impact of the system error. If the random error accounts for a large proportion, the sampling method or detection method can be optimized to reduce the interference of environmental factors and accidental factors, thereby improving the reliability and accuracy of the online monitoring system.

[0041] Furthermore, step S361 also includes step A, setting a dynamic time window, wherein the dynamic time window includes a time step for adjusting the time length of the time window; step B, calculating the error sources of the control data based on the dynamic time window, and using the error source calculation results to fit the long and short time series deviations to obtain error trend characteristics; and step C, adding the error trend characteristics to the indicator error increment.

[0042] Preferably, by setting a dynamic time window, the error sources of the data at different time spans are calculated and fitted, so as to extract the error trend characteristics and integrate them into the indicator error increment. Specifically, the dynamic time window refers to automatically adjusting the length of the time window to adapt to the data characteristics at different times according to the analysis requirements. The setting of the dynamic time window is mainly achieved through the time step, that is, according to the time series change characteristics of the data, the time window is flexibly expanded or shortened. The time step is the fixed time length (for example, 1 hour, 1 day) that the time window is increased or decreased each time. The short time window is suitable for capturing high-frequency error changes, and the long time window is suitable for capturing long-term error trends. The dynamic time window realizes multi-scale analysis of errors by combining short-term and long-term data analysis; within the dynamic time window, the error source calculation is performed on the control data of the laboratory data and the online monitoring data, and the systematic error and random error are distinguished, and their changing characteristics in the short-term and long-term time series are analyzed, wherein the systematic error calculation is to calculate the fixed deviation between the laboratory data and the online monitoring data in each time window, and the random error calculation is to calculate the standard deviation of the error in each time window.

[0043] Preferably, long and short time series deviation fitting is performed based on the calculation results of the error source, that is, by fitting the error source data in the short time window and the long time window, the trend characteristics of the error are extracted, that is, the dynamic change pattern of the error over time, including mainly capturing the rapid change trend of random errors and capturing the long-term change trend of system errors. The short time series and long time series data are combined to obtain the error trend characteristics, and finally the error trend characteristics obtained by fitting the long and short time series are integrated into the indicator error increment to provide dynamic support for the calibration and correction of online monitoring data, thereby improving the accuracy and adaptability of error correction.

[0044] Step S400: Based on the error source, calibration learning is performed according to the control data to construct a calibration model.

[0045] Step S400 further includes step S410, constructing a linear model for the system error, learning and training the linear model by constructing a system error training set, and obtaining a system error calibration model, wherein the system error calibration model is used to calibrate and analyze the system error and output calibration parameters; step S420, constructing a random forest model for the random error, and obtaining a random error calibration model by constructing a random error training and test set for model training convergence; and step S430, integrating the system error calibration model and the random error calibration model to obtain the calibration model.

[0046] Preferably, based on the source of the error, calibration learning is performed using control data to construct a calibration model, including constructing a linear model for the system error, that is, by establishing a linear model, using the fixed trend and regular characteristics of the system error (such as linear change), a calibration model is constructed to predict and correct the system error. Specifically, the system error value is extracted from the time series control data as the training set input (output target), and time, environmental parameters (such as temperature, humidity), measurement equipment parameters, etc. are used as input features to construct a system error training set to learn and train the linear model to obtain a system error calibration model for calibrating and analyzing the system error and outputting calibration parameters, that is, it can output calibration parameters (i.e., the prediction results of the linear model) for correcting the system errors in the online monitoring data.

[0047] Preferably, it also includes constructing a random forest model for random errors, that is, by constructing a random forest model, using the nonlinear and complex characteristics of random errors, predicting random errors and correcting them, specifically, extracting random error values ​​from control data, generating sample sets, including taking environmental parameters (such as temperature, humidity, water quality parameter fluctuations), equipment performance indicators, etc. as input features, taking actual random error values ​​as output targets, and then dividing the constructed sample sets into random error training sets and random error test sets, constructing a prediction model based on random forests, and then training and testing the prediction model according to the constructed random error training sets and random error test sets, so that the model converges to the optimal prediction performance, and obtaining a random error calibration model, which can output random error prediction values ​​for correcting random errors in online monitoring data. Finally, the linear model for systematic errors and the random forest model for random errors are integrated to generate a comprehensive calibration model for simultaneously correcting systematic errors and random errors, realizing comprehensive correction of systematic errors and random errors in online monitoring data, and providing higher-precision data correction capabilities, improving the reliability and accuracy of online water quality detection.

[0048] Furthermore, step S430 also includes step S431, respectively performing performance evaluation on the system error calibration model and the random error calibration model, including mean square error, root mean square error, mean absolute error, and goodness of fit; step S432, constructing a performance matrix according to the performance evaluation results of the system error calibration model and the random error calibration model, wherein the rows of the performance matrix represent model categories and the columns are performance indicators; step S433, removing the indicator dimension of the performance matrix and assigning weights to each performance indicator; step S434, calculating the performance distance of the system error calibration model and the random error calibration model based on the performance matrix and the weights of each performance indicator; step S435, determining the fusion weights of the system error calibration model and the random error calibration model according to the performance distance, integrating the system error calibration model and the random error calibration model based on the fusion weights to obtain the calibration model.

[0049] Preferably, the performance of the systematic error calibration model and the random error calibration model is evaluated respectively, including selecting appropriate evaluation indicators for the systematic error calibration model and the random error calibration model, including mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), and goodness of fit (R²), wherein the mean square error is the mean square error between the calibration and the laboratory data, the smaller the better; the root mean square error is the square root of the error after calibration, the smaller the better; the mean absolute error is the mean absolute value of the error after calibration, the smaller the better; the goodness of fit is the degree of fit between the calibration result and the laboratory data, the larger the better; then, according to the performance evaluation results of the systematic error calibration model and the random error calibration model, a performance matrix is ​​constructed, wherein the rows of the performance matrix represent the model categories and the columns are performance indicators. An example performance matrix is ​​as follows:

[0050] Model Category MSE RMSE MAE R² System Error Calibration Model 0.004 0.063 0.050 0.92 Random Error Calibration Model 0.006 0.078 0.060 0.89

[0051] Preferably, the indicator dimension is removed from the performance matrix, and the weight of each performance indicator is assigned, that is, the indicators of the performance matrix are standardized, and the weight corresponding to each indicator is assigned according to the importance of the performance indicator, and then the performance distance between the system error calibration model and the random error calibration model is calculated, including quantitatively evaluating the performance of the system error model and the random error model through the multi-attribute decision-making method TOPSIS, calculating the relative proximity, determining the ideal solution (the best value of each indicator) and the negative ideal solution (the worst value of each indicator), and calculating the distance between each scheme and the ideal solution and the negative ideal solution to obtain the closeness, that is, the performance distance. The larger the value, the better the scheme, reflecting the comprehensive difference of the model in the standardized performance indicators, which is used to compare the performance of the system error calibration model and the random error calibration model. Finally, the weights of the system error calibration model and the random error calibration model are determined according to the performance distance, and a larger fusion weight is assigned to the better model. The fusion weight is used to perform weighted integration on the system error calibration model and the random error calibration model to obtain the calibration model, which significantly improves the accuracy and adaptability of the calibration model and ensures that the error correction effect is excellent in different scenarios.

[0052] Step S500: deploy the calibration model in the detection system to calibrate the detection results.

[0053] Preferably, the integrated calibration model is deployed in the actual detection system to calibrate the detection data to improve the accuracy and reliability of the detection results. Specifically, the calibration model is embedded in the data processing module of the detection system to support real-time calling and correction of the detection data. According to the needs of the detection system, an appropriate calibration mode is selected to correct the detection results, such as real-time calibration (integrating the calibration model with the real-time data stream of the online monitoring equipment) or batch calibration (suitable for periodic sampling scenarios, the calibration model centrally corrects the data set after the detection task is completed). The calibration process includes receiving online monitoring raw data (such as pH value, dissolved oxygen, turbidity, etc.), inputting it into the calibration model to calibrate its systematic error and random error, and returning the corrected test results, thereby improving the reliability, accuracy and practicality of water quality detection.

[0054] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for calibrating drinking water test results based on multi-source data, characterized in that: include: Collect multi-source data, including laboratory data and online monitoring data; Align the laboratory data and online monitoring data according to the data collection time, space and indicator consistency relationship to construct a data alignment matrix; Extracting control data according to the data alignment matrix to perform deviation analysis and determine the source of error, wherein the source of error includes systematic error and random error source; Based on the error source, calibration learning is performed according to the control data to construct a calibration model; The calibration model is deployed in the detection system to calibrate the detection results.

2. The drinking water test result calibration method based on multi-source data according to claim 1, characterized in that: The step of aligning the laboratory data and online monitoring data according to the data collection time, space and indicator consistency relationship to construct a data alignment matrix includes: Aligning the laboratory data and online monitoring data according to the consistency of the data collection time and space, and establishing a spatiotemporal collection relationship; According to the indicator consistency target, the time-space acquisition relationship is converted into units for consistency and the detection indicator is marked to establish an indicator alignment relationship; The indicator alignment relationship is detected and aligned according to the indicator marks to construct the data alignment matrix.

3. The drinking water test result calibration method based on multi-source data according to claim 1, characterized in that: Extract the control data according to the data alignment matrix to perform deviation analysis and determine the source of error, which includes systematic error and random error source, including: Calculate the difference between the laboratory data and the online monitoring data based on the control data to obtain the absolute error; Calculate the residual standard deviation of the laboratory data and the online monitoring data based on the control data to obtain the relative error; A systematic error is determined based on the absolute error, and a random error is determined based on the relative error.

4. The drinking water detection result calibration method based on multi-source data according to claim 3, characterized in that: Extracting control data according to the data alignment matrix to perform deviation analysis and determine the source of error, including: Extracting the control data based on the detection index, and arranging the control data in time series according to the data collection time; According to the data collection time, the time series comparison data is extracted according to the set time window to obtain the detection index deviation analysis sample; The absolute error and relative error of the time series data are calculated according to the detection index deviation analysis sample to obtain the source of the time series error of the detection index.

5. The drinking water detection result calibration method based on multi-source data according to claim 4, characterized in that: The sources of timing errors of the detection indicators are obtained, which include: Determining an indicator error increment according to a source of a timing error of the detection indicator; The indicator error increment is fitted to the error source.

6. The drinking water test result calibration method based on multi-source data according to claim 5, characterized in that: Also includes: Setting a dynamic time window, wherein the dynamic time window includes a time step for adjusting the time length of the time window; Calculating the error sources of the control data based on the dynamic time window, and fitting the long and short time series deviations using the error source calculation results to obtain error trend characteristics; The error trend feature is added to the indicator error increment.

7. The drinking water test result calibration method based on multi-source data according to claim 1, characterized in that: Based on the error source, calibration learning is performed according to the control data to construct a calibration model, including: A linear model is constructed for the system error, and a system error training set is constructed to train the linear model to obtain a system error calibration model, wherein the system error calibration model is used to calibrate and analyze the system error and output calibration parameters; A random forest model is constructed for random errors, and the random error calibration model is obtained by constructing random error training and test sets for model training convergence; The systematic error calibration model and the random error calibration model are integrated to obtain the calibration model.

8. The drinking water test result calibration method based on multi-source data according to claim 7, characterized in that: Integrating the system error calibration model and the random error calibration model to obtain the calibration model includes: Performing performance evaluation on the systematic error calibration model and the random error calibration model respectively, including mean square error, root mean square error, absolute error mean, and goodness of fit; According to the performance evaluation results of the system error calibration model and the random error calibration model, a performance matrix is ​​constructed, wherein the rows of the performance matrix represent model categories and the columns represent performance indicators; removing the indicator dimension from the performance matrix and assigning weights to the performance indicators; Based on the performance matrix and the weights of each performance indicator, calculating the performance distance of the system error calibration model and the random error calibration model; The fusion weights of the system error calibration model and the random error calibration model are determined according to the performance distance, and the system error calibration model and the random error calibration model are integrated based on the fusion weights to obtain the calibration model.

Images