An irrigation monitoring data preprocessing method and system

Abstract

The application relates to the technical field of water conservancy monitoring data processing, and discloses an irrigation area monitoring data preprocessing method and system. The method comprises the following steps: adopting a time step fusion method, unifying multi-element monitoring data of different sampling frequencies of various monitoring stations in an irrigation area to the same time step, forming a multi-source aligned time sequence matrix, carrying out multi-scale decomposition on the time sequence matrix, obtaining a trend component sequence and a detail component sequence; for the detail component sequence, respectively adopting corresponding abnormality judgment rules for abnormality detection according to element types, generating an abnormal point position marker, and based on the abnormal point position marker, a spatial correlation relationship between the monitoring stations and a cross-element correlation relationship between the elements, the trend component and the detail component corresponding to the abnormal point are cooperatively repaired, and the repaired values are checked and corrected by hydrological physical constraints, finally, a standardized monitoring data set with a processing state label and a confidence score is output, and the accuracy and reliability of irrigation area multi-element monitoring data preprocessing can be improved.

Description

Technical Field

[0001] This invention relates to the field of water conservancy monitoring data processing technology, and in particular to a method and system for preprocessing irrigation district monitoring data. Background Technology

[0002] With the deepening of smart water conservancy and digital twin irrigation district construction, irrigation districts are generally equipped with a large number of online monitoring sensors for water level, flow, soil moisture, and rainfall, collecting real-time data on rainfall, water levels, and soil moisture to provide data support for irrigation water demand forecasting, water resource allocation decisions, and water situation early warning and forecasting. However, due to the influence of the complex outdoor environment, the collected monitoring data generally suffers from a large amount of noise interference and data gaps. Moreover, the three elements of rainfall, water levels, and soil moisture exhibit strong spatiotemporal correlations, and anomalies in a single element can easily affect subsequent analysis. Currently, common monitoring data preprocessing methods based on fixed thresholds or statistical rules are simple to implement but have poor adaptability and limited ability to identify complex noise collected by sensors. Methods based on single time series models usually assume that the data meets stationarity or a specific distribution, making it difficult to capture non-stationary changes and seasonal abrupt changes in irrigation district data between irrigation and non-irrigation periods. Furthermore, existing methods often process single elements independently, and the reliability of the repair results needs to be improved. How to improve the accuracy and reliability of irrigation district monitoring data preprocessing is a problem that needs to be solved. Summary of the Invention

[0003] To address the technical issues of accuracy and reliability in the preprocessing of irrigation district monitoring data, this invention provides a method and system for preprocessing irrigation district monitoring data.

[0004] The first aspect of this invention discloses a method for preprocessing irrigation district monitoring data, the method comprising:

[0005] Multi-element online monitoring data from various monitoring stations within the irrigation district are collected. A time-step fusion method is used to unify monitoring data with different sampling frequencies to the same time step, forming a multi-source aligned time series matrix. The multi-element data includes rainfall, water level, and soil moisture.

[0006] The time series matrix is ​​decomposed into multiple scales to obtain the trend component sequence representing long-term change characteristics and the detail component sequence representing local fluctuation characteristics.

[0007] For the detailed component sequence, anomaly detection is performed according to the corresponding anomaly judgment rules based on the element type, and anomaly point location markers are generated.

[0008] Based on the location markers of the anomalies, the spatial relationships between monitoring stations, and the cross-element relationships between various elements, the trend components and detail components corresponding to the anomalies are collaboratively repaired. Based on the hydrophysical constraints of the irrigation district, the repaired values ​​are verified and corrected to obtain the repaired monitoring time series.

[0009] A standardized monitoring dataset is generated based on the repaired monitoring time series, and a corresponding processing status label and confidence score are generated for each data point in the standardized monitoring dataset.

[0010] As an optional implementation, in the first aspect of the present invention, the method of using time step fusion to unify monitoring data with different sampling frequencies to the same time step to form a multi-source aligned time series matrix includes:

[0011] Set the target time step;

[0012] For any monitoring station's online monitoring data within the target time window:

[0013] If there is a unique monitoring value within the time window, then that monitoring value will be used as the data value at the target time.

[0014] If there are multiple monitoring values ​​within the time window, the arithmetic mean (or cumulative value) of the multiple monitoring values ​​shall be used as the data value at the target time.

[0015] If there are no monitored values ​​within the time window, and the time span between the nearest valid monitored values ​​before and after the target time is less than the preset maximum interpolation window, then the linear interpolation result of the valid monitored values ​​before and after the target time is used as the data value of the target time.

[0016] If there are no monitored values ​​within the time window, and the time span between the nearest valid monitored values ​​before and after the target time is greater than or equal to the maximum interpolation window, then the target time is marked as missing data.

[0017] The multi-source aligned time series matrix is ​​formed based on the data values ​​and missing data markers at each target time.

[0018] As an optional implementation, in the first aspect of the present invention, the step of performing anomaly detection and generating anomaly point location markers for the detail component sequence according to the feature type using corresponding anomaly judgment rules includes:

[0019] For the detailed components corresponding to rainfall data, a detection rule based on spatial correlation and physical threshold is adopted;

[0020] For the detailed components corresponding to the hydrological data, a cascaded detection rule based on unsupervised anomaly scoring and deep learning time series prediction is adopted;

[0021] For the detailed components corresponding to soil moisture data, detection rules based on local statistical features and cross-factor response relationships are adopted;

[0022] The results of anomaly assessments of various elements are collected to determine the location markers of anomaly points.

[0023] As an optional implementation, in the first aspect of the present invention, the step of collaboratively repairing the trend component and detail component corresponding to the anomaly point based on the anomaly point location marker, the spatial correlation between monitoring stations, and the cross-element correlation between various elements, and verifying and correcting the repaired value based on the hydrophysical constraints of the irrigation district to obtain the repaired monitoring time series includes:

[0024] Based on the spatial distance between monitoring stations, the connectivity of canals, and the correlation of historical monitoring data, a spatial correlation matrix is ​​constructed to determine the spatial neighborhood range and neighborhood weight of each monitoring station.

[0025] Based on the historical response relationships between rainfall, water levels, and soil moisture, a cross-factor correlation matrix is ​​constructed to determine the response direction and intensity among each factor.

[0026] The location to be repaired is determined based on the location marker of the anomaly point. Spatial interpolation repair is performed using the effective monitoring values ​​of similar elements in the spatial neighborhood and the neighborhood weight to obtain a first repair value. Cross-element estimation repair is performed using the effective monitoring values ​​of other elements with response relationships in the same monitoring station, the response direction and response intensity to obtain a second repair value. A preliminary repair value is determined based on the first repair value and the second repair value.

[0027] Using differentiable physical constraints constructed based on irrigation district hydrophysical equations, the preliminary repair values ​​are physically consistent, and repair values ​​that do not meet the physical constraints are iteratively corrected. The corrected trend components and detail components are then reconstructed into the repaired monitoring time series.

[0028] As an optional implementation, in the first aspect of the present invention, the step of generating a standardized monitoring dataset based on the repaired monitoring time series, and generating corresponding processing status labels and confidence scores for each data point in the standardized monitoring dataset, includes:

[0029] The repaired monitoring time series is organized according to a unified time index and monitoring station code to form the standardized monitoring dataset;

[0030] Based on whether each data point has undergone anomaly detection and repair processing, a processing status label is assigned to each data point; wherein, the processing status label includes at least original, interpolated repair, physical correction, and questionable.

[0031] The confidence score for each data point is determined based on the anomaly determination results of each data point, the consistency between spatial interpolation and cross-factor estimation in the collaborative repair, and the physical residuals output by the differentiable physical constraint verification.

[0032] As an optional implementation, in a first aspect of the present invention, determining the preliminary repair value based on the first repair value and the second repair value includes:

[0033] Using the first and second repair values ​​corresponding to the anomaly as input, a graph attention coding network is used to perform message passing along the spatial correlation relationship represented by the spatial correlation matrix and the cross-factor correlation relationship represented by the cross-factor correlation matrix. The repair value features of related nodes in the spatial neighborhood of the anomaly are aggregated to generate the latent space representation of the anomaly. The latent space representation is decoded using a differentiable physical constraint decoding network to output the preliminary repair value.

[0034] As an optional implementation, in the first aspect of the present invention, the step of constructing a spatial correlation matrix based on the spatial distance between monitoring stations, the canal connectivity, and the correlation of historical monitoring data, and determining the spatial neighborhood range and neighborhood weight of each monitoring station, includes:

[0035] Based on the topology of the irrigation canal system and the geographical location of the monitoring stations, determine the physical edges of the water flow that are directly connected by the canal sections;

[0036] Meteorological and hydrological correlation edges are determined based on the spatial distance between monitoring stations and the correlation of historical monitoring data;

[0037] Based on the physical edges of water flow and the related edges of meteorology and hydrology, an IoT map of water information in the irrigation area is constructed;

[0038] The set of nodes in the graph that are connected to the target monitoring station by an edge is taken as the spatial neighborhood of the monitoring station, and the weight of the edge is taken as the neighborhood weight.

[0039] A second aspect of this invention discloses an irrigation district monitoring data preprocessing system, comprising:

[0040] The data acquisition module is used to collect multi-element online monitoring data from various monitoring stations within the irrigation area;

[0041] The time series processing module is used to unify monitoring data with different sampling frequencies to the same time step using a time step fusion method, forming a multi-source aligned time series matrix; wherein, the multi-factors include rainfall, water level, and soil moisture.

[0042] The multi-scale decomposition module is used to perform multi-scale decomposition on the time series matrix to obtain the trend component sequence representing long-term change characteristics and the detail component sequence representing local fluctuation characteristics.

[0043] Anomaly detection module is used to perform anomaly detection on the detailed component sequence according to the feature type using corresponding anomaly judgment rules, and generate anomaly point location markers.

[0044] The collaborative repair module is used to collaboratively repair the trend component and detail component corresponding to the anomaly point based on the anomaly point location marker, the spatial correlation between monitoring stations, and the cross-element correlation between various elements, and to verify and correct the repaired value based on the hydrophysical constraints of the irrigation area to obtain the repaired monitoring time series.

[0045] The result output module is used to generate a standardized monitoring dataset based on the repaired monitoring time series, and to generate corresponding processing status labels and confidence scores for each data point in the standardized monitoring dataset.

[0046] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0047] The method provided by this invention includes the following steps: A time-step fusion method is used to unify multi-element monitoring data from various monitoring stations within the irrigation district with different sampling frequencies to the same time step, forming a multi-source aligned time series matrix; the time series matrix is ​​decomposed into multi-scale sequences to obtain trend component sequences and detail component sequences; for the detail component sequences, anomaly detection is performed according to the corresponding anomaly judgment rules based on the element type, generating anomaly point location markers; based on the anomaly point location markers, the spatial correlation between monitoring stations, and the cross-element correlation between various elements, the trend component and detail component corresponding to the anomaly points are collaboratively repaired, and the repaired values ​​are verified and corrected using irrigation district hydrophysical constraints; finally, a standardized monitoring dataset with processing status labels and confidence scores is output.

[0048] This method can improve the accuracy and reliability of preprocessing multi-element monitoring data in irrigation areas.

[0049] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0051] Figure 1 This is a flowchart illustrating a method for preprocessing irrigation district monitoring data according to an exemplary embodiment;

[0052] Figure 2 This is a block diagram illustrating an irrigation district monitoring data preprocessing system according to an exemplary embodiment. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0054] Figure 1 This is a flowchart illustrating a method for preprocessing irrigation district monitoring data according to an exemplary embodiment. Figure 1 As shown, this method includes the following steps.

[0055] Step 101: Collect multi-element online monitoring data from each monitoring station in the irrigation area, and use the time step fusion method to unify the monitoring data with different sampling frequencies to the same time step to form a multi-source aligned time series matrix; wherein, the above-mentioned multi-element includes rainfall, water level and soil moisture.

[0056] In this embodiment of the invention, multiple rain gauges, water level / flow rate stations, and soil moisture monitoring stations are typically deployed within the irrigation area. Each sensor collects and uploads data via a remote terminal unit at its own set frequency. Rainfall data comes from each rain gauge station, and fields may include, but are not limited to, station code, data collection time, and rainfall amount (unit: mm). Water level data comes from each water level / flow rate station, and fields may include, but are not limited to, station code, data collection time, water level (unit: m) or flow rate (unit: m³ / s). Soil moisture data comes from each soil moisture monitoring station, and fields may include, but are not limited to, station code, data collection time, and layered soil volumetric moisture content (unit: %).

[0057] As an optional implementation, the above-mentioned time-step fusion method, which unifies monitoring data with different sampling frequencies to the same time step to form a multi-source aligned time series matrix, may include:

[0058] Set a target time step; for online monitoring data (hereinafter referred to as monitoring data) of any monitoring station within the time window of the target time: if there is a unique monitoring value within the time window, then the monitoring value is used as the data value of the target time; if there are multiple monitoring values ​​within the time window, then the arithmetic mean (or cumulative value) of the multiple monitoring values ​​is used as the data value of the target time; if there is no monitoring value within the time window, and the time span between the nearest valid monitoring values ​​before and after the target time is less than the preset maximum interpolation window, then the linear interpolation result of the valid monitoring values ​​before and after the target time is used as the data value of the target time; if there is no monitoring value within the time window, and the time span between the nearest valid monitoring values ​​before and after the target time is greater than or equal to the maximum interpolation window, then the target time is marked as missing data; based on the data values ​​of each target time and the missing data markings, a multi-source aligned time series matrix is ​​formed.

[0059] In this embodiment of the invention, since different sensors have different acquisition frequencies (e.g., some devices acquire data every 5 minutes, while others acquire data every 15 minutes or 1 hour), it is necessary to unify the acquired multi-source data to the same time index. Assume the target time step is... For example, it can be standardized to 1 hour. For any measuring point... The original sampling sequence, the unified sequence at time The value selection rules are as follows: If the monitoring data is within the time window If a unique monitoring value exists, it is used directly; if multiple monitoring values ​​exist, the arithmetic mean (or cumulative value) is taken as the representative value within that step size; if there is no monitoring value and the time span between the nearest valid monitoring values ​​to the target time is less than the preset maximum interpolation window W, linear interpolation is used to obtain a temporary filling value. The interpolation formula is:

[0060] (1)

[0061] In the formula, and They are respectively The nearest valid observation time before and after it, and These are the corresponding valid monitoring values. If the time span between consecutive valid monitoring values ​​is greater than or equal to the maximum interpolation window W, the target time point is marked as missing data and will be handled in subsequent steps. Based on the data values ​​and missing data markers at each target time point, a multi-source aligned time series matrix is ​​formed.

[0062] Optionally, the above target time step The maximum interpolation window W can be set according to the actual monitoring needs of the irrigation area, for example... Take 1 hour, W takes 2 times. The embodiments of the present invention are not limited.

[0063] Step 102: Perform multi-scale decomposition on the time series matrix to obtain the trend component sequence representing long-term change characteristics and the detail component sequence representing local fluctuation characteristics.

[0064] In this embodiment of the invention, multi-scale decomposition can be used to separate each single-station, single-element sequence in the time series matrix into trend components and detail components. The trend component reflects the long-term, gradual changes in data under the influence of factors such as the transition between irrigation and non-irrigation periods and seasonal variations, while the detail component focuses on capturing local, rapid fluctuations caused by short-term rainfall events, gate operations, and instantaneous sensor disturbances. Multi-scale decomposition can be implemented using wavelet transform, empirical mode decomposition, or seasonal trend decomposition methods based on local weighted regression. Taking an additive model as an example, after multi-scale decomposition, the aforementioned time series matrix equals the sum of the trend component sequence and the detail component sequence.

[0065] Step 103: For the detail component sequence, anomaly detection is performed according to the corresponding anomaly judgment rules based on the feature type, and anomaly point location markers are generated.

[0066] In this embodiment of the invention, anomaly detection is performed on the detailed component sequence because the detailed components reflect local rapid fluctuations caused by short-term rainfall events, gate operations, instantaneous sensor disturbances, etc. By separating the long-term gradual trend from the local fluctuations in the monitoring data, anomaly detection only applies to the fluctuating components and is not affected by non-stationary changes such as the overall rise in water level during irrigation season or the trend of increasing flow during flood season. Therefore, anomaly detection based on the decomposed detailed components can avoid misjudging trending water level rises or falls as anomalies, thus improving detection robustness.

[0067] As an optional implementation, the above-described method for detecting anomalies in detail component sequences, based on feature type and using corresponding anomaly detection rules to generate anomaly point location markers, may include:

[0068] For the detailed components corresponding to rainfall data, a detection rule based on spatial correlation and physical threshold is adopted;

[0069] For the detailed components corresponding to the hydrological data, a cascaded detection rule based on unsupervised anomaly scoring and deep learning time series prediction is adopted;

[0070] For the detailed components corresponding to soil moisture data, detection rules based on local statistical features and cross-factor response relationships are adopted;

[0071] The results of anomaly assessments of various elements are collected to determine the location markers of anomaly points.

[0072] In this embodiment of the invention, differentiated anomaly detection rules are adopted for the detailed component sequences of different elements. Specifically, for the detailed components corresponding to rainfall data, detection rules based on spatial correlation and physical thresholds are adopted. Specifically, for the detailed component values ​​of measuring point P, a hard rule is first used for screening: the reconstructed rainfall corresponding to the detailed component value must meet the following conditions. ,in This is the upper limit of the extreme value of the local historical maximum hourly rainfall, calculated by step size. After screening, the spatial weighted deviation of the detail component values ​​of measuring point P and its K nearest neighboring rain gauges at the same time is calculated, and the determination formula is:

[0073] (2)

[0074] In the above formula, The weights are calculated using the inverse of the distance. For the detailed component values ​​of neighboring station i, This is the weighted standard deviation of the detail component values ​​of neighboring stations at that time. This is an adaptive coefficient, typically ranging from 2.5 to 3.5. Values ​​exceeding this threshold are considered spatially inconsistent anomalies. For data points marked as anomalous, spatial inverse distance weighting is used for interpolation and repair. The interpolation formula is:

[0075] (3)

[0076] In the formula, Let P be the distance between station P and its neighboring station i. The detailed component values ​​are for neighboring station i.

[0077] For the detailed components corresponding to the hydrological data, a cascaded detection rule based on unsupervised anomaly scoring and deep learning time-series prediction is adopted. In the first stage, the hydrological detailed component values, first-order differences, second-order differences, time period codes (represented by sine / cosine codes of daily ordinal numbers), and irrigation period identifiers are organized as feature vectors. Anomaly scoring is performed using the isolated forest algorithm, and a dynamic quantile threshold (e.g., the 95th percentile) is calculated within a sliding window. Points exceeding this threshold are marked as suspected anomalies. In the second stage, a lightweight gated recurrent unit network is trained online for non-anomaly data segments. Its forward propagation process is as follows:

[0078] (4)

[0079] (5)

[0080] (6)

[0081] (7)

[0082] In the formula, Input features (including hydrological detail component values ​​and corresponding rainfall data).

[0083] In the hidden state, ⊙ represents element-wise multiplication. For suspected outliers, the trained network is used to recursively predict the predicted values. Calculate the deviation .like If the deviation exceeds three times the standard deviation of the predicted residual multiple times consecutively, it is confirmed as a true anomaly, and the predicted value is used as the standard deviation. It replaces the detailed component values ​​corresponding to outliers and can fill in short-term data gaps.

[0084] For the detailed components corresponding to soil moisture data, a detection rule based on local statistical features and cross-factor response relationships is adopted. The mean value of the detailed component is calculated within a moving window of length L. and standard deviation The judgment condition is:

[0085] (8)

[0086] In the formula, Here, α represents the soil moisture detail component value, and α is an adaptive coefficient related to crop growth stage and irrigation status, which can be 3.0 during non-irrigation periods and 3.5 during irrigation periods. To exclude reasonable steep increases caused by rainfall or irrigation, a rainfall-soil moisture response logic can be further introduced for secondary screening: if the rainfall detail component corresponding to the steep increase period indicates that the hourly rainfall exceeds the runoff threshold (e.g., 40 mm), or the water detail component indicates that irrigation is underway, the corresponding anomaly is marked as suspicious and retained, triggering only the labeling without removal. For data points confirmed as anomalies, an exponentially weighted moving average recursive value is used to reconstruct the data by weighting and fusing it with the theoretical soil moisture balance values ​​before and after the detected segment, preventing long-term interpolation drift.

[0087] Step 104: Based on the location markers of anomalies, the spatial relationships between monitoring stations, and the cross-element relationships between various elements, the trend components and detail components corresponding to the anomalies are collaboratively repaired. Based on the hydrological and physical constraints of the irrigation area, the repaired values ​​are verified and corrected to obtain the repaired monitoring time series.

[0088] As an optional implementation method, the above-mentioned method, based on anomaly location markers, spatial correlations between monitoring stations, and cross-element correlations between various elements, collaboratively repairs the trend and detail components corresponding to anomalies. Then, based on irrigation district hydrophysical constraints, the repaired values ​​are verified and corrected to obtain the repaired monitoring time series, including:

[0089] Based on the spatial distance between monitoring stations, the connectivity of canals, and the correlation of historical monitoring data, a spatial correlation matrix is ​​constructed to determine the spatial neighborhood range and neighborhood weight of each monitoring station.

[0090] Based on the historical response relationships between rainfall, water levels, and soil moisture, a cross-factor correlation matrix is ​​constructed to determine the response direction and intensity among each factor.

[0091] The location to be repaired is determined by marking the location of the anomaly point. Spatial interpolation repair is performed using the effective monitoring values ​​of similar elements in the spatial neighborhood and the neighborhood weight to obtain the first repair value. Cross-element estimation repair is performed using the effective monitoring values, response direction and response intensity of other elements with response relationship in the same monitoring station to obtain the second repair value. The preliminary repair value is determined based on the first repair value and the second repair value.

[0092] By utilizing differentiable physical constraints constructed based on the hydrophysical equations of the irrigation district, the physical consistency of the preliminary repair values ​​is verified, and the repair values ​​that do not meet the physical constraints are iteratively corrected. The corrected trend components and detail components are then reconstructed into the repaired monitoring time series.

[0093] In this embodiment of the invention, a spatial correlation matrix is ​​constructed. Specifically, based on the spatial distance between monitoring stations (such as Euclidean distance or distance along the canal system), the connectivity of the canal system (such as whether they are located in the same main canal or branch canal, or whether they are controlled by the same gate), and the statistical correlation of historical monitoring data (such as the Pearson correlation coefficient of water level sequences in the same time period), the correlation strength between each pair of monitoring stations is calculated to form a spatial correlation matrix. Based on this matrix, the spatial neighborhood range and neighborhood weight of each monitoring station are determined. Furthermore, a cross-factor correlation matrix is ​​constructed. Based on the historical response relationships and physical laws between rainfall and water conditions (rainfall-runoff), rainfall and soil moisture (rainfall infiltration), and water conditions and soil moisture (irrigation replenishment) within the irrigation area, the response direction and response intensity between different factors are determined. This matrix provides a quantitative basis for cross-factor estimation in collaborative restoration.

[0094] Further, as an optional implementation, the above-mentioned determination of the preliminary repair value based on the first repair value and the second repair value includes:

[0095] Using the first and second repair values ​​corresponding to the anomaly as input, a graph attention coding network is used to pass messages along the spatial correlation matrix and the cross-factor correlation matrix to aggregate the repair value features of related nodes in the spatial neighborhood of the anomaly to generate the latent space representation of the anomaly. A differentiable physical constraint decoding network is used to decode the latent space representation and output the preliminary repair value.

[0096] In this embodiment of the invention, optionally, the first repair value and the second repair value are obtained through the aforementioned spatial dimension and cross-element dimension, respectively. Specifically, the first repair value can be obtained by extracting the effective monitoring values ​​of unlabeled anomalies of the same type of element within the spatial neighborhood of the monitoring station where the anomaly point is located, and then performing weighted interpolation using neighborhood weights. The second repair value can be estimated by estimating the expected value of the anomaly point within the same monitoring station based on the effective monitoring values ​​of other elements in the cross-element correlation matrix that have a response relationship with the element corresponding to the anomaly point, as well as the corresponding response direction and intensity. The first and second repair values ​​can be determined by fusion methods such as weighted averaging, Kalman fusion, or graph attention networks. Among them, the differentiable physical constraint decoding network utilizes a decoding network combined with differentiable physical constraints. When using a graph attention encoding network as the fusion method, the first and second repair values ​​are used as inputs, message passing is performed along the spatial correlation matrix and the cross-element correlation matrix, the features of related nodes in the neighborhood are aggregated, a latent space representation is generated, and then the decoding network outputs the preliminary repair value. In addition, the hydrophysical equations of the irrigation area can be constructed as differentiable functions as constraints to verify the preliminary repair value. Constraints may include, but are not limited to, water level-flow relationship constraints and water balance constraints. For each water level station, using its pre-processed water level-flow scatter plot, a currently valid water level-flow relationship curve Q=f(Z) is fitted. A residual loss term is constructed using the relative error between the measured flow and the flow estimated from the curve. Gain compensation is applied to sections where system deviations exist. Water balance constraints construct a balance residual loss for the inflow / outflow and storage changes in the canal system control body. Specifically, for adjacent upstream and downstream sections of the canal system, a water balance term is constructed using the difference in measured flow over a given time period, interval water intake, and rainfall runoff. A residual loss term is constructed using the closure error. These physical residuals, as part of the optimization objective, are iteratively corrected based on the initial repair value, ensuring that the repair value conforms to both the temporal regularity of the data and the hydrological and physical laws of the irrigation area. Optionally, as another possible implementation, the above-mentioned spatial correlation matrix is ​​constructed based on the spatial distance between monitoring stations, the canal system connectivity, and the correlation of historical monitoring data. This matrix determines the spatial neighborhood range and neighborhood weight of each monitoring station and may include:

[0097] Based on the topology of the irrigation canal system and the geographical location of the monitoring stations, determine the physical edges of the water flow directly connected by the canal sections;

[0098] Meteorological and hydrological correlation edges are determined based on the spatial distance between monitoring stations and the correlation of historical monitoring data;

[0099] Based on the physical edges of water flow and the correlation edges of meteorological and hydrological data, an IoT map of water information in the irrigation area is constructed.

[0100] The set of nodes in the graph that are connected to the target monitoring station by an edge is taken as the spatial neighborhood of the monitoring station, and the weight of the edge is taken as the neighborhood weight.

[0101] In this embodiment of the invention, each monitoring station can be mapped to a node in the map based on the topology of the irrigation canal system and the geographical location of each monitoring station. If two monitoring stations are directly connected in the canal system without any gates separating them, a physical edge for water flow is established; if the spatial distance between two stations is less than a preset threshold and the correlation of historical monitoring data is higher than a preset value, a meteorological-hydrological correlation edge is established. The weight of the edge can be initialized based on the distance decay coefficient or the correlation coefficient and can be dynamically adjusted in subsequent training. The set of nodes in the map that are connected to the target monitoring station by an edge is taken as the spatial neighborhood of the monitoring station, and the weight of the edge is taken as the neighborhood weight, thereby connecting with the spatial correlation matrix.

[0102] Step 105: Generate a standardized monitoring dataset based on the repaired monitoring time series, and generate corresponding processing status labels and confidence scores for each data point in the standardized monitoring dataset.

[0103] In this embodiment of the invention, the restored full-element monitoring time series can be organized according to a unified time index and monitoring station code to generate a standardized monitoring dataset. Each data point in the dataset, in addition to the monitoring value, also includes a processing status label and a confidence score. The processing status label identifies the data processing path and can include four types: original, interpolated repair, physical correction, and questionable. The confidence score can be calculated based on the anomaly score in the anomaly detection stage of step 103, the consistency between spatial interpolation and cross-element estimation, and the physical residual output of the differentiable physical constraint verification; this embodiment does not limit this calculation.

[0104] It is evident that implementation Figure 1 The described irrigation district monitoring data preprocessing method unifies multi-source monitoring data with different sampling frequencies to the same time step and performs multi-scale decomposition, separating trend and detail components to provide a foundation for differentiated anomaly detection. Adaptive anomaly judgment rules are adopted for the different variation characteristics of rainfall, water levels, and soil moisture, effectively reducing missed and false detections caused by fixed thresholds and single model assumptions. The method utilizes spatial correlations between monitoring stations and cross-element correlations between elements to collaboratively repair anomalies, avoiding information loss caused by independent interpolation of single elements. Simultaneously, differentiable physical constraints are introduced during the repair process, ensuring that the repaired values ​​satisfy the temporal regularity of the monitoring data while also considering the physical consistency of the irrigation district, thus improving the rationality of the repair results. Finally, a standardized dataset with processing status labels and confidence scores is output, providing a complete and reliable data foundation for subsequent analyses such as irrigation water demand prediction, water resource allocation, and water situation early warning, thereby effectively improving the accuracy and reliability of multi-element monitoring data preprocessing in irrigation districts.

[0105] Figure 2 This is a block diagram illustrating a preprocessing system for irrigation district monitoring data according to an exemplary embodiment. Figure 2As shown, the system includes:

[0106] The data acquisition module 201 is used to collect multi-element online monitoring data from various monitoring stations within the irrigation area and provide it to the time series processing module 202.

[0107] The time series processing module 202 is used to unify monitoring data with different sampling frequencies to the same time step using a time step fusion method, forming a multi-source aligned time series matrix; among which, the multiple elements include rainfall, water level and soil moisture.

[0108] The multi-scale decomposition module 203 is used to perform multi-scale decomposition on the above time series matrix to obtain the trend component sequence representing long-term change characteristics and the detail component sequence representing local fluctuation characteristics.

[0109] The anomaly detection module 204 is used to perform anomaly detection on the detail component sequence according to the feature type and the corresponding anomaly judgment rules, and generate anomaly point location markers.

[0110] The collaborative repair module 205 is used to collaboratively repair the trend component and detail component corresponding to the anomaly point based on the anomaly point location marker, the spatial correlation between monitoring stations, and the cross-element correlation between various elements. Based on the hydrological and physical constraints of the irrigation area, the repaired value is verified and corrected to obtain the repaired monitoring time series.

[0111] The result output module 206 is used to generate a standardized monitoring dataset based on the repaired monitoring time series, and to generate corresponding processing status labels and confidence scores for each data point in the standardized monitoring dataset.

[0112] It is evident that implementation Figure 2 The described system can unify multi-source monitoring data with different sampling frequencies to the same time step and perform multi-scale decomposition to separate trend and detail components, providing a foundation for differentiated anomaly detection. Adaptive anomaly judgment rules are adopted for the different variation characteristics of rainfall, water levels, and soil moisture, effectively reducing missed and false detections caused by fixed thresholds and single model assumptions. The system utilizes spatial correlations between monitoring stations and cross-element correlations between elements to collaboratively repair anomalies, avoiding information loss caused by independent interpolation of single elements. Simultaneously, differentiable physical constraints are introduced during the repair process, ensuring that the repaired values ​​satisfy the temporal regularity of the monitoring data while also considering the physical consistency of the irrigation area, thus improving the rationality of the repair results. Finally, a standardized dataset with processing status labels and confidence scores is output, providing a complete and reliable data foundation for subsequent analyses such as irrigation water demand prediction, water resource allocation, and water situation early warning, thereby effectively improving the accuracy and reliability of multi-element monitoring data preprocessing in the irrigation area.

[0113] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for preprocessing irrigation district monitoring data, characterized in that, include: Multi-element online monitoring data from various monitoring stations within the irrigation district are collected. A time-step fusion method is used to unify monitoring data with different sampling frequencies to the same time step, forming a multi-source aligned time series matrix. The multi-element data includes rainfall, water level, and soil moisture. The time series matrix is ​​decomposed into multiple scales to obtain the trend component sequence representing long-term change characteristics and the detail component sequence representing local fluctuation characteristics. For the detailed component sequence, anomaly detection is performed according to the corresponding anomaly judgment rules based on the element type, and anomaly point location markers are generated. Based on the location markers of the anomalies, the spatial relationships between monitoring stations, and the cross-element relationships between various elements, the trend components and detail components corresponding to the anomalies are collaboratively repaired. Based on the hydrophysical constraints of the irrigation district, the repaired values ​​are verified and corrected to obtain the repaired monitoring time series. A standardized monitoring dataset is generated based on the repaired monitoring time series, and a corresponding processing status label and confidence score are generated for each data point in the standardized monitoring dataset.

2. The irrigation district monitoring data preprocessing method according to claim 1, characterized in that, The method employing time-step fusion to unify monitoring data from different sampling frequencies to the same time step, forming a multi-source aligned time series matrix, includes: Set the target time step; For any monitoring station's online monitoring data within the target time window: If there is a unique monitoring value within the time window, then that monitoring value will be used as the data value at the target time. If there are multiple monitoring values ​​within the time window, the arithmetic mean (or cumulative value) of the multiple monitoring values ​​shall be used as the data value at the target time. If there are no monitored values ​​within the time window, and the time span between the nearest valid monitored values ​​before and after the target time is less than the preset maximum interpolation window, then the linear interpolation result of the valid monitored values ​​before and after the target time is used as the data value of the target time. If there are no monitored values ​​within the time window, and the time span between the nearest valid monitored values ​​before and after the target time is greater than or equal to the maximum interpolation window, then the target time is marked as missing data. The multi-source aligned time series matrix is ​​formed based on the data values ​​and missing data markers at each target time.

3. The irrigation district monitoring data preprocessing method according to claim 1, characterized in that, The step of performing anomaly detection on the detailed component sequence, based on the feature type and corresponding anomaly judgment rules, and generating anomaly point location markers includes: For the detailed components corresponding to rainfall data, a detection rule based on spatial correlation and physical threshold is adopted; For the detailed components corresponding to the hydrological data, a cascaded detection rule based on unsupervised anomaly scoring and deep learning time series prediction is adopted; For the detailed components corresponding to soil moisture data, detection rules based on local statistical features and cross-factor response relationships are adopted; The results of anomaly assessments of various elements are collected to determine the location markers of anomaly points.

4. The irrigation district monitoring data preprocessing method according to claim 1, characterized in that, Based on the anomaly location markers, spatial relationships between monitoring stations, and cross-element relationships among various elements, the trend and detail components corresponding to the anomalies are collaboratively repaired. Then, based on irrigation district hydrophysical constraints, the repaired values ​​are verified and corrected to obtain the repaired monitoring time series, including: Based on the spatial distance between monitoring stations, the connectivity of canals, and the correlation of historical monitoring data, a spatial correlation matrix is ​​constructed to determine the spatial neighborhood range and neighborhood weight of each monitoring station. Based on the historical response relationships between rainfall, water levels, and soil moisture, a cross-factor correlation matrix is ​​constructed to determine the response direction and intensity among each factor. The location to be repaired is determined based on the location marker of the anomaly point. Spatial interpolation repair is performed using the effective monitoring values ​​of similar elements in the spatial neighborhood and the neighborhood weight to obtain a first repair value. Cross-element estimation repair is performed using the effective monitoring values ​​of other elements with response relationships in the same monitoring station, the response direction and response intensity to obtain a second repair value. A preliminary repair value is determined based on the first repair value and the second repair value. Using differentiable physical constraints constructed based on irrigation district hydrophysical equations, the preliminary repair values ​​are physically consistent, and repair values ​​that do not meet the physical constraints are iteratively corrected. The corrected trend components and detail components are then reconstructed into the repaired monitoring time series.

5. The irrigation district monitoring data preprocessing method according to claim 1, characterized in that, The process involves generating a standardized monitoring dataset based on the repaired monitoring time series, and generating corresponding processing status labels and confidence scores for each data point in the standardized monitoring dataset, including: The repaired monitoring time series is organized according to a unified time index and monitoring station code to form the standardized monitoring dataset; Based on whether each data point has undergone anomaly detection and repair processing, a processing status label is assigned to each data point; wherein, the processing status label includes at least original, interpolated repair, physical correction, and questionable. The confidence score for each data point is determined based on the anomaly determination results of each data point, the consistency between spatial interpolation and cross-factor estimation in the collaborative repair, and the physical residuals of the differentiable physical constraint verification output.

6. The irrigation district monitoring data preprocessing method according to claim 4, characterized in that, The step of determining the preliminary repair value based on the first repair value and the second repair value includes: Using the first and second repair values ​​corresponding to the anomaly as input, a graph attention coding network is used to pass messages along the spatial association relationship represented by the spatial association matrix and the cross-element association relationship represented by the cross-element association matrix, and the repair value features of related nodes in the spatial neighborhood of the anomaly are aggregated to generate the latent space representation of the anomaly. The latent space representation is decoded using a differentiable physical constraint decoding network to output the preliminary repair value.

7. The irrigation district monitoring data preprocessing method according to claim 4, characterized in that, The process involves constructing a spatial correlation matrix based on the spatial distance between monitoring stations, canal connectivity, and the correlation of historical monitoring data to determine the spatial neighborhood range and neighborhood weight of each monitoring station, including: Based on the topology of the irrigation canal system and the geographical location of the monitoring stations, the physical boundaries of the water flow directly connected by the canal sections are determined; The meteorological and hydrological association boundaries are determined based on the spatial distance between monitoring stations and the correlation of historical monitoring data. Based on the physical boundary of water flow and the meteorological and hydrological correlation boundary, an IoT map of water information in the irrigation area is constructed. The set of nodes in the graph that are connected to the target monitoring station by a boundary is taken as the spatial neighborhood of the monitoring station, and the weight of the edge is taken as the neighborhood weight.

8. A preprocessing system for irrigation district monitoring data, characterized in that, include: The data acquisition module is used to collect multi-element online monitoring data from various monitoring stations within the irrigation area; The time series processing module is used to unify monitoring data with different sampling frequencies to the same time step using a time step fusion method, forming a multi-source aligned time series matrix; wherein, the multi-factors include rainfall, water level, and soil moisture. The multi-scale decomposition module is used to perform multi-scale decomposition on the time series matrix to obtain the trend component sequence representing long-term change characteristics and the detail component sequence representing local fluctuation characteristics. Anomaly detection module is used to perform anomaly detection on the detailed component sequence according to the feature type using corresponding anomaly judgment rules, and generate anomaly point location markers. The collaborative repair module is used to collaboratively repair the trend component and detail component corresponding to the anomaly point based on the anomaly point location marker, the spatial correlation between monitoring stations, and the cross-element correlation between various elements. Based on the hydrophysical constraints of the irrigation area, the repaired value is verified and corrected to obtain the repaired monitoring time series. The results output module is used to generate a standardized monitoring dataset based on the repaired monitoring time series, and to generate corresponding processing status labels and confidence scores for each data point in the standardized monitoring dataset.

Images