Quality control and optimization method and system for lysimeter data based on machine learning

Abstract

The invention relates to the technical field of agricultural hydrology and environment monitoring, and discloses a lysimeter data quality control and optimization method based on machine learning, and the method comprises the following steps: S1, collecting lysimeter weighing data and at least one kind of environment covariable data in real time through an Internet of Things sensor network, and forming a multi-dimensional time sequence; s2, performing time alignment, missing value marking and standardization processing on the multi-dimensional time sequence to form a regular data set, and writing the regular data set into a time sequence database; and S3, carrying out anomaly detection and classification marking on the regular data set by adopting a progressive multi-mode collaborative anomaly recognition method. Compared with the prior art, the method has the advantages that the anomaly recognition dimensions are diversified and accurate; data correction is intelligent and high in physical interpretability; the system is closed-loop and automatic, and has sustainable evolution capability; the operation and maintenance decision is scientific, and the response efficiency is remarkably improved; the technology integration level is high, and large-scale application is easy.

Description

Technical Field

[0001] This invention relates to the field of agricultural hydrology and environmental monitoring technology, specifically to a method and system for quality control and optimization of lysimeter data based on machine learning. Background Technology

[0002] A lyoinmeter is a high-precision instrument for measuring soil evaporation, plant transpiration, and net seepage. It acquires core data by continuously weighing a container (weighing bucket) filled with soil. Utilizing the principle of water balance, the lyoinmeter accurately detects and analyzes the weight of the soil column and soil moisture within the in-situ soil-vegetation system to study soil water balance and groundwater recharge. A weighing sensor is mounted at the bottom of the lyoinmeter's measuring bucket to measure the overall weight of the bucket. The lyoinmeter also incorporates a water supply and drainage system. This system simulates natural environmental conditions, allowing the weighing sensor to measure changes in soil water storage. This represents the increase in moisture after rainfall or irrigation, or the loss of water due to transpiration and evaporation, providing a weight variable for studying the evaporation characteristics of vegetation and soil.

[0003] However, lysimeters are high-precision and high-sensitivity instruments. Environmental interference (such as the effects of extreme weather such as strong winds and heavy rain), equipment failure (such as sensor drift, power fluctuations, and data acquisition device failure), human activities (such as instrument maintenance, vegetation trimming, and surrounding construction), and biological activities (such as animal activities on the instrument) can all cause abnormal changes in the weighing value. Currently, most of these abnormalities are monitored and handled manually, which is time-consuming, labor-intensive, and has a large time delay, resulting in problems such as low efficiency, large errors, and serious lag.

[0004] Currently, existing lysimeter data quality control methods mainly rely on simple threshold judgments or single statistical models, which have the following shortcomings:

[0005] 1. Single dimension of anomaly identification: It is difficult to fully cover complex anomaly types, such as instantaneous noise, sensor drift and violation of physical laws, resulting in high false alarm and false negative rates;

[0006] 2. Lack of intelligent correction capabilities: After identifying anomalies, there is a lack of precise data optimization methods. Usually, only labeling or simple interpolation can be performed, which cannot restore the authenticity of the data in terms of physical meaning and temporal continuity.

[0007] 3. Insufficient system closed-loop optimization: The data quality control process relies on human experience and has failed to form an automated closed loop that integrates automatic detection, intelligent correction, causal analysis and self-learning optimization.

[0008] Therefore, there is an urgent need to develop an automated quality control and optimization system for lysimeter observation data to achieve intelligent anomaly identification and data correction capabilities, thereby improving the resolution and accuracy of leakage monitoring. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to overcome the above-mentioned technical difficulties and provide a method and system for quality control and optimization of lysosome data based on machine learning, so as to solve the problems of single anomaly identification dimension, insufficient data correction capability and low system intelligence level in the prior art.

[0010] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:

[0011] A method for quality control and optimization of lysimeter data based on machine learning, the method comprising the following steps:

[0012] S1: Real-time collection of lysimeter weighing data and at least one environmental covariate data through an Internet of Things (IoT) sensor network to form a multidimensional time series.

[0013] S2: Perform time alignment, missing value marking, and standardization on the multidimensional time series to form a regularized dataset and write it into the time series database;

[0014] S3: A progressive multi-modal collaborative anomaly detection method is used to perform anomaly detection and classification labeling on the regularized dataset, including:

[0015] S3.1: Initial screening and identification based on physical rules, marking data points that violate any of the non-negativity constraints, energy balance constraints, and water balance constraints as physical rule anomalies;

[0016] S3.2: Outlier detection based on unsupervised clustering. The KMeans algorithm is used within a sliding window to identify outliers and mark them as outlier anomalies.

[0017] S3.3: Residual analysis based on LSTM time series prediction model, identifying and marking residual anomalous or residual sequence abnormalities based on standardized residuals or residual sequence patterns;

[0018] S4: For the identified abnormal data segments, use the LSTM encoder-decoder model to perform predictive data reconstruction based on the normal data before and after the abnormal segment and environmental covariates, and replace the original abnormal values ​​with the reconstructed values.

[0019] S5: Automatically matches a predefined knowledge base of anomaly causes and solutions based on the anomaly type, and generates an anomaly diagnosis report and operation and maintenance suggestions;

[0020] S6: It compares and displays the original data with the corrected data through a visual interface, provides an interface for manual labeling, and drives model retraining and system optimization based on feedback data.

[0021] Furthermore, the environmental covariate data include at least three of the following: air temperature, light intensity, soil volumetric water content, precipitation, sunshine duration, relative humidity, and wind speed.

[0022] Furthermore, the physical rules in S3.1 also include temperature logic constraints and data extremum constraints, wherein the data extremum constraints include the light intensity not exceeding the solar constant.

[0023] Furthermore, the method for determining outlier anomalies in S3.2 is as follows: calculate the distance of a data point from the center of its cluster. If the distance is greater than the historical average distance plus twice the standard deviation, it is marked as an outlier anomaly.

[0024] Furthermore, the method for determining residual anomalies in S3.3 is as follows: the standardized residual e_t=|y_t-ŷ_t| / σ exceeds the preset threshold θ_residual; the method for determining residual sequence anomalies is as follows: within the sliding window, the autocorrelation function of the residual sequence deviates significantly from the normal pattern at the key lag order, or the distribution characteristics are significantly different from the residual distribution during the training period.

[0025] Furthermore, the training process of the LSTM encoder-decoder model in S4 includes: constructing samples by randomly masking data segments on historical normal data, and performing joint optimization using the mean squared error loss function combined with a physical constraint loss term. The physical constraint loss term includes monotonicity constraints on the prediction results during periods without precipitation, or fitting bias constraints on the energy balance equation and water balance equation.

[0026] A system for quality control and optimization of lysimeter data based on machine learning, the system comprising:

[0027] The data acquisition and communication module is used to collect lysimeter weighing data and environmental covariate data via IoT protocol.

[0028] The data storage and management module is used for preprocessing, standardizing, and time-series storage of the collected data;

[0029] The multi-mode quality control module includes a physical rule recognition unit, an unsupervised clustering recognition unit, and an LSTM prediction recognition unit, which are used to perform progressive anomaly detection and labeling.

[0030] The LSTM encoder / decoder model data optimization module is used for predictive reconstruction and replacement of abnormal data segments.

[0031] The anomaly analysis and reporting module has a built-in anomaly knowledge base and report generator, which are used to output diagnostic suggestions and structured reports;

[0032] The visualization and human-computer interaction module is used for data visualization, manual input, model retraining triggering, and system parameter configuration.

[0033] Furthermore, the visualization and human-computer interaction module supports highlighting different categories of anomalies with different graphics and colors in the Web interface, and provides a trend comparison chart of the original data and the corrected data.

[0034] The advantages of this invention compared to the prior art are:

[0035] 1. Diversified and precise anomaly detection dimensions:

[0036] Through a three-level collaborative identification mechanism of physical rules, unsupervised clustering, and LSTM prediction residuals, full-spectrum anomaly coverage is achieved, ranging from obvious physical paradoxes (such as negative mass values) to hidden systematic drifts (such as slow sensor offsets).

[0037] Experimental data show that, compared with the single threshold method, this method improves the anomaly identification accuracy from 72% to 94% and reduces the false negative rate from 18% to below 5% at typical farmland sites.

[0038] The physical rule layer can quickly filter out obvious erroneous data, reducing the computational burden in the future; the clustering layer can identify outliers that do not conform to the overall distribution pattern; and the prediction residual layer can capture hidden anomalies under time-series dependencies, forming a complementary enhancement effect.

[0039] 2. Intelligent data correction and strong physical interpretability:

[0040] The LSTM encoder-decoder model can fully learn the long-term dependencies and periodic characteristics of lysimeter data and generate high-quality reconstructed sequences based on the normal context information before and after the abnormal segment.

[0041] Introduce physical constraint loss terms (such as water balance constraints and energy balance constraints) during training to ensure that the reconstruction results are not only mathematically smooth but also conform to actual physical laws, thus avoiding "mathematical optimization" results with unreasonable physical meaning.

[0042] It can effectively correct both short-term abrupt anomalies (such as transient interference) and long-term drift anomalies (such as sensor aging), and the root mean square error (RMSE) of reconstructed data compared with real normal data is reduced by an average of more than 65%.

[0043] 3. The system features closed-loop automation and continuous evolution capabilities:

[0044] It achieves a complete automated closed loop from "data collection → anomaly identification → intelligent correction → diagnostic suggestions → human feedback → model iteration", reducing the human intervention time from an average of 8 hours per site per week to less than 1 hour;

[0045] Through a visual human-computer interaction interface, expert knowledge can be easily integrated into the system, and manually labeled data can automatically enter the retraining process, driving continuous model optimization and adapting to different sites, seasons and environmental changes.

[0046] The system supports online configuration and version management of rules, thresholds, and model parameters, and has good scalability and adaptability, and can be deployed in the cloud, edge, or hybrid architecture.

[0047] 4. Scientific operation and maintenance decision-making and significantly improved response efficiency:

[0048] Based on an anomaly type-based automatic matching diagnostic knowledge base, it provides on-site maintenance personnel with targeted solutions and suggestions, reducing troubleshooting time by approximately 60%.

[0049] Automatically generate structured quality reports, including anomaly statistics, trend analysis, and corrective effect evaluation, supporting data quality traceability and accountability.

[0050] The system supports automatic push notifications for anomaly warnings and reports, enabling a shift in the operation and maintenance model from "passive response" to "proactive warning".

[0051] 5. High degree of technological integration and easy to scale up:

[0052] This integrates technologies such as IoT data acquisition, time-series databases, machine learning models, and web visualization to form a unified solution.

[0053] The system adopts a modular design, supports distributed deployment, can manage lysimeter data from multiple sites simultaneously, and has good scalability and economy.

[0054] It is applicable to various ecosystems such as farmland, grassland, forest land, and wetland, providing high-quality data support for fields such as ecological environment monitoring, agricultural water-saving irrigation, and climate change research. Attached Figure Description

[0055] Figure 1 : Overall flowchart of the lysimeter data quality control and reconstruction method of the present invention.

[0056] Figure 2 : Architecture block diagram of the system of the present invention.

[0057] Figure 3 The anomaly type-cause-solution direction mapping table of this invention.

[0058] Figure 4 : A schematic diagram of the visualization interface and anomaly analysis report of this invention. Detailed Implementation

[0059] In the description of this invention, unless otherwise stated, "a plurality of" means two or more. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion.

[0060] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings.

[0061] A method and system for quality control and optimization of lysimeter data based on machine learning, specifically including the following steps:

[0062] Step 1: Acquire continuous time data collected by the load cell and all other sensors.

[0063] This step aims to build a complete and continuous data foundation. The implementation method is as follows:

[0064] Electronic devices connect to IoT sensors deployed in and around the lysimeter via a cloud platform, continuously collecting multi-source sensor data over time at a set sampling frequency. The collected data forms a multi-dimensional time series data table, whose core characteristic variables include, but are not limited to: 1) Target variable: Mass data output by the lysimeter weighing sensor, used to directly calculate evaporation and transpiration. 2) Key environmental covariates: Air temperature, light intensity, soil moisture, soil water content, precipitation, sunshine duration, relative humidity, wind speed, etc.

[0065] Step 2: Data preprocessing, which involves noise reduction of the data.

[0066] To ensure the reliability and efficiency of subsequent analysis, the raw data must be normalized. The specific implementation method for this step is as follows:

[0067] 1) Time Alignment and Sequence Storage: The collected data is processed into regularized data and ordered strictly in ascending order of timestamps to ensure that each time point corresponds to a complete set of pre-processed multi-source observation vectors, forming a high-quality dataset that can be directly used for subsequent analysis; 2) Missing Value Standardization: Missing values ​​are filled with -999 to indicate missing data; 3) Time Series Database Writing: The processed regularized data is written to a high-performance time series database (such as InfluxDB, TimescaleDB) or a structured data file in strictly ascending order of timestamps.

[0068] Step 3: Use multi-mode anomaly identification to identify anomalies in the weighing data.

[0069] This step is the core of quality control, employing a progressive strategy from simple to complex, from rules to models, to perform multi-dimensional anomaly screening on the data. The specific implementation method for this step is as follows:

[0070] 1) Anomaly identification based on physical rules (marked as "0 physical rule anomalies")

[0071] Specifically, firstly, based on the physical meaning of the collected data, hard anomaly identification rules are determined, and abnormal data is labeled as "0 physical rule anomaly".

[0072] The physical rules include: 1) Non-negativity constraint: such as weighing data, soil moisture data, etc., cannot be negative; 2) Initial energy constraint judgment: such as radiation energy constraint, i.e., latent heat flux (λET) ≤ net radiation (Rn) - soil heat flux (G); the data needs to meet the water balance, i.e., ET ≤ precipitation (P) + irrigation (I) + ΔS (soil water change) - runoff (R) - drainage (D); 3) Temperature logic: the highest temperature is greater than or equal to the lowest temperature; 4) Data extreme value constraint: such as light intensity sensor data must be non-negative and less than or equal to the solar constant (approximately 1100W / m²). 2 ).

[0073] 2) Anomaly identification based on unsupervised clustering (marked as "1 outlier anomaly")

[0074] This stage is used to identify outliers that do not conform to the overall data distribution pattern.

[0075] K-Means clustering is used to detect outliers in the data, and outlier data is labeled as "1 outlier anomaly". The current data and 1000 data points before the current data time are input into the K-Means clustering algorithm for cluster analysis to obtain K cluster centers and cluster labels for each point. Then, the distance of the current data point from its cluster center is calculated. If it exceeds the threshold range (mean + 2 standard deviations), it is identified as an outlier and labeled as "1 outlier anomaly".

[0076] 3) Anomaly identification based on machine learning LSTM algorithm prediction (labeled as "2-predicted classification anomaly")

[0077] This stage utilizes time-series prediction models to capture more subtle anomaly patterns.

[0078] The LSTM machine learning algorithm is used for prediction and identification, and the residual between the predicted value and the actual value is calculated. When the residual exceeds a preset threshold, it is marked as "2-prediction classification anomaly".

[0079] The anomaly classification criteria are as follows: 1) Residual anomalies, labeled as “2.1 Residual Anomalies”. The criterion is that at time point t, the standardized residual e_t=|y_t-ŷ_t| / σ between the predicted value ŷ generated by the trained time series prediction model and the actual observed value y exceeds the preset threshold θ_residual, where σ is the standard deviation of the residuals of the model on the validation set. This part aims to capture those anomalies that significantly deviate from the expected value of the model prediction at a single time point. These anomalies are usually caused by transient disturbances or mutations. 2) Residual sequence anomalies, labeled as “2.2 Residual Sequence Anomalies”. The criterion is that within the sliding window W, the autocorrelation function of the continuous residual sequence {e_t,e_{t-1},...,e_{t-w+1}} significantly deviates from the normal pattern at key lag orders (such as 1, 24 hours), or the distribution characteristics of the residual sequence (such as skewness, kurtosis) are significantly different from the residual distribution during the training period (through statistical tests such as the KS test). The objective of this part of the detection is to identify systematic biases that are not significant in individual point residuals but whose overall sequence patterns do not conform to historical patterns, such as slow sensor drift.

[0080] 4) Manual identification and final labeling

[0081] The anomaly categories identified by the different modes above can overlap on a single data point. Data that was not judged as abnormal by the above steps, if manually identified as abnormal, can be labeled with the anomaly type through the terminal device as "3 Manually identified anomaly".

[0082] Otherwise, all data not labeled as category 0, 1, 2, 3 will ultimately be labeled as "4 Normal Data". The same data point can be assigned multiple anomalous labels.

[0083] Step 4: Optimize abnormal data based on LSTM encoder-decoder model and physical guidance.

[0084] After detecting and classifying anomalous data, this invention enters the core data optimization stage. This step aims to address anomalous data segments identified as "correctable" (such as short-term mutations or noise interference) using an LSTM encoder-decoder model that integrates physical context awareness and sequence generation capabilities. This model can generate a physically reasonable and temporally continuous high-quality data sequence based on the normal sequences before and after the anomaly and their physical driving factors, scientifically replacing the original outliers and restoring the integrity and scientific validity of the data. The specific implementation of this step is as follows:

[0085] Optimize data preparation and structuring

[0086] Before performing model predictions, the input data needs to be standardized and structured, specifically including:

[0087] (1) Data standardization: The Z-score standardization method is used to normalize the input time series data (including lysimeter weighing data and its covariates). (2) Input sliding window construction: The model adopts a context window-based input method. For a given abnormal data segment [t_start,t_end], the actual input of the model is the normal data intervals before and after it and their corresponding covariates, which together constitute a complete context window: [tc,t_start)∪(t_end,t+d). Among them, the size of the look-ahead window c and the size of the look-back window d need to be comprehensively set according to the inherent periodicity of the data (such as daily periodicity, annual periodicity) and the length of the abnormal segment to ensure that the window contains sufficiently rich time series pattern information. (3) Input feature dimension: The input of the model is a multi-dimensional time series tensor, including the target variable: the original weighing data of the lysimeter or the weight data after preliminary processing, and the high-quality covariates: the concurrent environmental monitoring data that are strongly correlated with the physical mechanism of the evaporation and transpiration process of the lysimeter, including air temperature, light intensity, soil moisture, soil water, precipitation, sunshine duration, relative humidity, and wind speed.

[0088] LSTM encoder-decoder model architecture and training

[0089] This invention uses an LSTM encoder-decoder as the core prediction model. This architecture is designed specifically for sequence-to-sequence learning tasks and can effectively solve the long-range dependency problem.

[0090] (1) Model Construction: Encoder: Consists of a multi-layer LSTM network. Its responsibility is to read and understand the entire input sequence of the context window [tc, t_start) ∪ (t_end, t+d]. The encoder processes the sequence step by step and compresses and summarizes all its useful information into a fixed-dimensional context vector. This vector is the hidden state of the encoder at the last time step and carries all the context information required to reconstruct the anomalous segment. Decoder: Also consists of a multi-layer LSTM network. Its responsibility is to generate sequences based on the context vector. The decoder uses the context vector generated by the encoder as its initial hidden state, and then generates the pre-sequence of the anomalous time period [t_start, t_end] step by step and autoregressively. The decoder outputs a predicted value at each step, using it as one of the inputs for the next step. Attention mechanism (optional but recommended improvement): To further improve model performance, especially for long sequences, an attention mechanism can be introduced between the encoder and decoder. This allows the decoder to dynamically "focus" on the most relevant parts of the encoder's input sequence at each generation step, rather than relying solely on a fixed context vector, thus significantly improving reconstruction accuracy, especially when dealing with complex periodic patterns. Output layer: The decoder's output at each time step is passed through a fully connected layer, mapping to the final predicted value. For regression tasks, this layer typically uses a linear activation function.

[0091] (2) Model Training: Training Data Preparation: A large number of data segments without anomaly markers are selected from historical, long-term, high-quality normal monitoring data. Training samples are constructed by artificially and randomly "masking" a continuous small segment (simulating a real anomaly) on these normal data segments. The input of each sample is a context window composed of normal data before and after the masked segment, and the target is the true normal value of the masked segment. This "masking-prediction" paradigm enables the model to specifically learn how to infer missing data based on contextual information. Loss Function: The mean squared error is used as the main loss function during training to minimize the difference between the model's predicted value and the true normal value. Its formula is:

[0092]

[0093] Where y_i_true is the true value of the masked segment, and y_i_pred is the predicted value of the model.

[0094] Optimizer: An adaptive moment estimation optimizer is used to update the model parameters. This optimizer can automatically adjust the learning rate and has the advantages of high training efficiency and fast convergence speed.

[0095] Predictive correction process for outlier data

[0096] After model training is complete, for identified real anomalous data segments, reconstruction and replacement are performed according to the following rigorous process: Input data extraction: Based on the start and end times [t_start, t_end] of the anomalous segment, all data (including the target variable and all covariates) within a predefined context window [tc, t_start) ∪ (t_end, t+d) is precisely extracted from the regularized dataset. Data standardization: The extracted input data is standardized using the mean μ and standard deviation σ of each feature calculated and saved during the training phase. Model inference: The standardized, structured input tensor is fed into the trained LSTM encoder-decoder model. Reconstruction output: After the model runs, it outputs a complete reconstructed sequence for the anomalous time period [t_start, t_end]. This sequence is the "most likely" normal data inferred by the model based on the learned long-term physical laws and short-term context. Data de-standardization and replacement: The reconstructed sequence output by the model is de-standardized using μ and σ of the target variable to restore its original physical dimensions. Subsequently, this reconstructed sequence is used to directly and accurately replace the corresponding anomalous data segments in the original data, completing the final data correction.

[0097] Step 5: Preliminary analysis of the anomaly type and its causes, and analysis of possible solutions.

[0098] Based on the identified anomaly categories, the system automatically generates preliminary anomaly cause analysis and suggested solutions, as shown in the following example:

[0099]

[0100] Step 6: Terminal visualization, reporting, and human interaction.

[0101] This step provides a human-computer interaction interface to enable data insights, decision support, and system iteration. The specific implementation scheme is as follows:

[0102] Data visualization

[0103] The data acquired by multiple sensors is analyzed and displayed with time as the horizontal axis, which makes it easy to directly observe the data change trend and identify anomalies. The trend line is divided into two parts: the original data trend line and the corrected data trend line, which facilitates comparison of the differences before and after. Anomalies are marked with different graphics and colors according to different anomaly types.

[0104] The technology is developed based on Web technologies (such as HTML5 and JavaScript), and can be rendered using mature chart libraries (such as ECharts and D3.js). It supports cross-platform access (such as web browsers and mobile apps) and has interactive functions such as dynamic data updates, view zooming, and data export.

[0105] Automatically generate exception reports

[0106] The system possesses automated and customizable report generation capabilities, transforming data processing results into structured knowledge to support operational decision-making. Reports include summaries, detailed anomaly statistics, anomaly event details, and operational recommendations based on step 5. Reports support both periodic generation (e.g., weekly, monthly reports) and event-driven generation (e.g., automatically generating and sending alarm reports when the number of daily anomalies exceeds a threshold).

[0107] Manual labeling and model optimization

[0108] The terminal includes a manual labeling interface for manually identifying and marking data anomalies, or correcting errors in automatically identified anomalies. On the visualization dashboard, expert users can directly circle data segments with the mouse, or click on anomalies already marked by the system, and select from a predefined list or manually enter more precise labels.

[0109] The system periodically incorporates newly accumulated manually labeled data and normal data into the training set, initiating the retraining process of the LSTM encoder-decoder model in step 4 to achieve continuous model optimization and version iteration. Simultaneously, administrators are allowed to edit and manage the anomaly identification rules (such as the threshold θ_residual) and types in step 3.

[0110] A second aspect of the present invention provides a quality control and optimization system for lysimeter data implementing the above method, comprising:

[0111] 1) Data Acquisition and Communication Module: This module communicates with various sensor networks deployed at the lysostat site via IoT protocols, automatically and continuously acquiring multi-source data over time. The module has built-in drivers for multiple communication interfaces, supporting 4G / 5G, LoRa, Ethernet, and other transmission methods. It can pull or receive data pushed from the cloud platform or local data logs at a preset sampling frequency to the central processing unit, ensuring the complete acquisition of raw data.

[0112] 2) Data storage and management module: This module preprocesses the raw data and includes a missing value normalizer. This module is also the data storage unit, storing the raw data, intermediate data and final result data, and establishing a database for management.

[0113] 3) Multi-mode quality control module: This is the core of the system. This engine has a built-in progressive anomaly identification pipeline that goes from rules to models and from simple to complex. It has built-in multi-level quality control and anomaly diagnosis logic to execute all inspection steps for anomaly detection and to label the data.

[0114] 4) LSTM encoder-decoder model data reconstruction module: This is another core component of the system. It incorporates various reconstruction models and algorithms to perform reconstruction strategy selection and calculation. It includes a data standardization and window builder, an LSTM encoder-decoder model library, and a model inference and abnormal data replacement unit, thereby realizing data reconstruction.

[0115] 5) Visualization and Human-Computer Interaction Module: This module includes a multi-dimensional data visualization section to display comparison charts before and after data quality control, outlier reports, reconstruction results, and uncertainties. It also includes a manual labeling interface, allowing users to manually review and correct automated judgments. Furthermore, it includes a model optimization and system management module, used to periodically or manually trigger the retraining process of the LSTM encoder-decoder model reconstruction engine and the machine learning LSTM algorithm prediction detector. This allows for the optimization of model parameters using new data, enabling iterative model upgrades. System administrators can also visualize and adjust physical rule bases, clustering detection thresholds, residual judgment thresholds, etc., allowing the system to adapt to the specific requirements of different sites and environments.

[0116] 6) Anomaly Analysis and Reporting Module: This module serves as the system's decision support unit. It has a built-in knowledge base mapping anomaly type, cause, and solution; an automatic analyzer that automatically matches the corresponding preliminary cause analysis and solution direction suggestions from the knowledge base based on the anomaly tags output by the quality control engine; and a report generator that automatically generates quality control reports and outputs the final dataset in the required format.

[0117] The present invention and its embodiments have been described above, and this description is not restrictive. If those skilled in the art are inspired by this description and design similar embodiments without departing from the spirit of the invention, such embodiments should fall within the protection scope of the present invention.

Claims

1. A method for quality control and optimization of lysimeter data based on machine learning, characterized in that, The method includes the following steps: S1: Real-time collection of lysimeter weighing data and at least one environmental covariate data through an Internet of Things (IoT) sensor network to form a multidimensional time series. S2: Perform time alignment, missing value marking, and standardization on the multidimensional time series to form a regularized dataset and write it into the time series database; S3: A progressive multi-modal collaborative anomaly detection method is used to perform anomaly detection and classification labeling on the regularized dataset, including: S3.1: Initial screening and identification based on physical rules, marking data points that violate any of the non-negativity constraints, energy balance constraints, and water balance constraints as physical rule anomalies; S3.2: Outlier detection based on unsupervised clustering. The K-Means algorithm is used within a sliding window to identify outliers and mark them as outlier anomalies. S3.3: Residual analysis based on LSTM time series prediction model, identifying and marking residual anomalous or residual sequence abnormalities based on standardized residuals or residual sequence patterns; S4: For the identified abnormal data segments, use the LSTM encoder-decoder model to perform predictive data reconstruction based on the normal data before and after the abnormal segment and environmental covariates, and replace the original abnormal values ​​with the reconstructed values. S5: Automatically matches a predefined anomaly-cause-solution knowledge base based on the anomaly type to generate an anomaly diagnosis report and operation and maintenance suggestions; S6: It compares and displays the original data with the corrected data through a visual interface, provides an interface for manual labeling, and drives model retraining and system optimization based on feedback data.

2. The method for quality control and optimization of lysimeter data based on machine learning according to claim 1, characterized in that: The environmental covariate data include at least three of the following: air temperature, light intensity, soil volumetric water content, precipitation, sunshine duration, relative humidity, and wind speed.

3. The method for quality control and optimization of lysimeter data based on machine learning according to claim 1, characterized in that: The physical rules in S3.1 also include temperature logic constraints and data extremum constraints, wherein the data extremum constraints include that the light intensity does not exceed the solar constant.

4. The method for quality control and optimization of lysimeter data based on machine learning according to claim 1, characterized in that: The method for determining outlier anomalies in S3.2 is as follows: calculate the distance of a data point from the center of its cluster. If the distance is greater than the historical average distance plus twice the standard deviation, it is marked as an outlier anomaly.

5. The method for quality control and optimization of lysimeter data based on machine learning according to claim 1, characterized in that: The method for determining residual anomalies in S3.3 is as follows: standardized residuals If the residual sequence exceeds the preset threshold θ_residual, the method for determining anomalies is as follows: within the sliding window, the autocorrelation function of the residual sequence deviates significantly from the normal pattern at the key lag order, or the distribution characteristics are significantly different from the residual distribution during the training period.

6. The method for quality control and optimization of lysimeter data based on machine learning according to claim 1, characterized in that: The training process of the LSTM encoder-decoder model in S4 includes: constructing samples by randomly masking data segments on historical normal data, and performing joint optimization using the mean squared error loss function and the physical constraint loss term.

7. The method for quality control and optimization of lysimeter data based on machine learning according to claim 6, characterized in that: The physical constraint loss term includes monotonicity constraints on the prediction results during periods without precipitation, or fitting deviation constraints on the energy balance equation and water balance equation.

8. A system for quality control and optimization of lysimeter data based on machine learning according to claims 1-7, characterized in that, The system includes: The data acquisition and communication module is used to collect lysimeter weighing data and environmental covariate data via IoT protocol. The data storage and management module is used for preprocessing, standardizing, and time-series storage of the collected data; The multi-mode quality control module includes a physical rule recognition unit, an unsupervised clustering recognition unit, and an LSTM prediction recognition unit, which are used to perform progressive anomaly detection and labeling. The LSTM encoder-decoder model data optimization module is used for predictive reconstruction and replacement of anomalous data segments. The anomaly analysis and reporting module has a built-in anomaly knowledge base and report generator, which are used to output diagnostic suggestions and structured reports; The visualization and human-computer interaction module is used for data visualization, manual input, model retraining triggering, and system parameter configuration.

9. The system for quality control and optimization of lysimeter data based on machine learning according to claim 8, characterized in that: The visualization and human-computer interaction module supports highlighting different categories of anomalies with different graphics and colors in the web interface, and provides trend comparison charts of the original data and the corrected data.

Images