Soil moisture sensor data quality inspection and interpolation method

Abstract

The invention provides a soil moisture sensor data quality inspection and interpolation method, which comprises the following steps: screening multivariable soil moisture time sequence data based on a preset core physical feature list, carrying out abnormal value detection through physical rule constraint and an isolation forest algorithm, and carrying out data labeling by creating a complete time axis; generating a training sample from the preprocessed data through sliding window sampling, and performing deep feature learning by using a denoising network based on a space-time diffusion probability model; performing interpolation on missing values in the original data by using the trained space-time diffusion probability model, generating a noisy data sample through a forward noise adding process, and performing conditional data interpolation based on a known observation value and a mask matrix in a reverse denoising process to generate a preliminary interpolation result; and performing post-processing correction on the preliminary interpolation result, wherein the post-processing correction comprises clamping correction based on a monthly historical range and correction based on interlayer physical logic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agricultural water and soil engineering data processing, and particularly relates to a soil moisture sensor data quality checking and interpolation method. BACKGROUND

[0002] In the field of agricultural water and soil, the crop irrigation system based on soil moisture content decision-making is a modern agricultural development trend with scientific irrigation as the core. The soil moisture collector based on the Internet of Things can realize real-time continuous monitoring of soil moisture content of different depth soil layers (for example, 20 cm, 40 cm, 60 cm, 80 cm) and uploading to the cloud. However, in complex actual environments such as fields, due to various factors such as physical wear of the sensor itself, continuous rainy days leading to limited charging of the battery by the solar energy, resulting in depletion of the battery power or interruption of wireless signal transmission, etc., the obtained data has quality problems such as missing of a single data value, abnormality and even long-time continuous data missing. This affects the accuracy of subsequent data analysis, model construction and the reliability of irrigation decision-making.

[0003] At present, in order to solve the problem of data missing, the existing technology often adopts a data interpolation method. Traditional interpolation methods include mean filling, forward / backward filling and linear interpolation, etc. The advantages of such methods are simple calculation and easy operation. However, their technical defects are that they mainly rely on a few measured values of the missing point in the local area, and fail to effectively utilize the change rule of the data in the time dimension and ignore the internal correlation between different depth sensors (spatial dimension). Therefore, when the data missing rate is high or the soil moisture fluctuates more complex due to irrigation and rainfall events, the interpolation accuracy and reliability are difficult to meet the needs of high-precision applications. In recent years, deep generative models represented by GAN (Generative Adversarial Network) and diffusion models have been applied to data interpolation, but they also have technical bottlenecks. These models are completely data-driven, and the results generated by them may be mathematically smooth, but not reasonable in physical or real logic. For example, in the absence of effective constraints, the model may interpolate the soil temperature below zero in the summer crop growing season, or interpolate the dramatic change of the water content of the deep soil (such as 60 cm), while the water content of the shallow soil (such as 20 cm) remains unchanged, which shows that it lacks the integration ability of domain knowledge and physical common sense.

[0004] In summary, the existing technology has the technical problems of fragmented process, low utilization rate of time and space dimension data and lack of physical common sense constraints when processing the soil moisture data based on the Internet of Things. SUMMARY

[0005] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0006] To this end, a first object of the present application is to provide a soil moisture sensor data quality checking and interpolation method.

[0007] A second object of the present application is to provide a soil moisture sensor data quality checking and interpolation device.

[0008] A third object of the present application is to provide an electronic device.

[0009] A fourth object of the present application is to provide a computer-readable storage medium.

[0010] A fifth object of the present application is to provide a computer program product.

[0011] To achieve the above objects, the first aspect of the present application provides a soil moisture sensor data quality checking and interpolation method, comprising:

[0012] Based on the preset core physical feature list, the multivariate soil moisture time series data is filtered, the outlier detection is performed through the physical rule constraint and the isolation forest algorithm, and the data labeling is performed through the creation of a complete time axis;

[0013] The training samples are generated from the preprocessed data through sliding window sampling, and the deep feature learning is performed using the denoising network based on the spatiotemporal diffusion probability model;

[0014] The missing values in the original data are interpolated using the trained spatiotemporal diffusion probability model, the noisy data samples are generated through the forward noise adding process, and the conditional data interpolation is performed based on the known observation values and the mask matrix in the reverse denoising process to generate the preliminary interpolation results;

[0015] The preliminary interpolation results are post-processed and corrected, including the clipping correction based on the monthly historical range and the correction based on the interlayer physical logic, to ensure that the interpolated data conforms to the actual soil moisture variation law in terms of time evolution and spatial distribution.

[0016] Optionally, the filtering of the multivariate soil moisture time series data based on the preset core physical feature list, the outlier detection through the physical rule constraint and the isolation forest algorithm, and the data labeling through the creation of a complete time axis, comprise:

[0017] The multivariate time series data is extracted from the original soil moisture data, and the feature selection is performed according to the preset core physical feature list to filter out the features related to the soil moisture variation;

[0018] The filtered data is constrained using the physical rules to exclude the data points that do not conform to the actual physical law;

[0019] Anomaly value detection is performed on the data using the Isolation Forest algorithm to identify and label abnormal data points, and further eliminate or correct these abnormal values.

[0020] According to the data timestamp and other time information, a complete time axis is created, and each data point is labeled with its corresponding time to ensure the time consistency of the data.

[0021] Optionally, the denoising network based on the spatiotemporal diffusion probability model is a hybrid model containing spatial correlation and temporal dependence, wherein the spatial dimension is propagated through the graph neural network to the feature nodes, and the temporal dimension is modeled through the Transformer self-attention mechanism to form a spatiotemporal fusion denoising network, wherein:

[0022] In the graph neural network, the nodes represent soil moisture sensors or related environmental variables, and the edges are constructed based on the water conduction relationship between soil layers or statistical correlation to form a graph structure with physical semantics.

[0023] The Transformer self-attention mechanism calculates the correlation weight between all time steps in the time dimension to capture the lag response and periodic fluctuations caused by irrigation or rainfall events.

[0024] Optionally, in the training process of the spatiotemporal diffusion probability model, the following steps are included:

[0025] The forward noise adding process uses preset noise scheduling parameters to gradually add Gaussian noise to the original data at T discrete time steps until it becomes isotropic Gaussian noise.

[0026] The reverse denoising process performs the inverse process of Markov chain through iteration, and at each step, the known observation value and the mask matrix are used for conditional injection to guide the generated path to conform to the physical logic.

[0027] Optionally, the post-processing correction of the preliminary interpolation result includes clipping correction based on the monthly historical range and correction based on the interlayer physical logic to ensure that the interpolated data conforms to the actual soil moisture variation law in terms of time evolution and spatial distribution, including:

[0028] The clipping correction based on the monthly historical range limits the interpolated value within the boundary by calculating the historical maximum and minimum value of each core feature in each month to avoid generating unreasonable data that exceeds the seasonal experience range.

[0029] The correction based on the interlayer physical logic corrects the deep soil moisture value if the shallow soil moisture change rate is greater than the deep soil moisture change rate to make it conform to the physical law of water conduction downward.

[0030] To achieve the above object, the second aspect of the present application provides a soil moisture sensor data quality checking and interpolation device, comprising:

[0031] A preprocessing module is configured to filter multivariate soil moisture time series data based on a preset core physical feature list, detect outliers through physical rule constraints and isolated forest algorithms, and label data by creating a complete time axis;

[0032] A spatio-temporal fusion modeling module is configured to generate training samples from preprocessed data through sliding window sampling, and perform deep feature learning using a denoising network based on a spatio-temporal diffusion probability model;

[0033] An interpolation generation module is configured to interpolate missing values in original data using a trained spatio-temporal diffusion probability model, generate noisy data samples through a forward noise adding process, and perform conditional data interpolation based on known observations and a mask matrix in a reverse denoising process to generate preliminary interpolation results;

[0034] A post-processing correction module is configured to perform post-processing correction on the preliminary interpolation results, including clipping correction based on monthly historical ranges and correction based on interlayer physical logic, to ensure that the interpolated data conforms to the actual soil moisture variation law in terms of time evolution and spatial distribution.

[0035] To achieve the above object, the third aspect of the present application provides an electronic device, comprising a processor and a memory in communication with the processor;

[0036] The memory stores computer execution instructions;

[0037] The processor executes the computer execution instructions stored in the memory to implement the method of any one of the first aspect.

[0038] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method of any one of the first aspect.

[0039] To achieve the above object, the fifth aspect of the present application provides a computer program product, wherein the computer program is executed by a processor to implement the method of any one of the first aspect.

[0040] The technical scheme provided by the embodiments of the present application at least brings the following beneficial effects: the abnormal value detection and missing data completion can be seamlessly integrated in a unified framework, and the interpolation data precision is ensured, and the decision-making precision of determining crop irrigation based on soil moisture is improved.

[0041] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 This is a flowchart illustrating a method for checking and interpolating soil moisture sensor data quality according to an embodiment of the present invention.

[0044] Figure 2 This is a model fitting effect diagram provided in Embodiment 1 of the present invention;

[0045] Figure 3 This is a model fitting effect diagram provided in Embodiment 2 of the present invention;

[0046] Figure 4 A schematic diagram of the accuracy dashboard for generating soil data at a depth of 20cm provided in Embodiment 3 of the present invention;

[0047] Figure 5 A schematic diagram of the accuracy dashboard for generating soil data at a depth of 40cm provided in Embodiment 3 of the present invention;

[0048] Figure 6 A schematic diagram of the accuracy dashboard for generating soil data at a depth of 60cm provided in Embodiment 3 of the present invention;

[0049] Figure 7 A schematic diagram of the accuracy dashboard for generating soil data at a depth of 80cm provided in Embodiment 3 of the present invention;

[0050] Figure 8 This is a schematic diagram of a soil moisture sensor data quality inspection and interpolation device provided in an embodiment of the present invention. Detailed Implementation

[0051] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0052] Figure 1 This is a schematic flowchart illustrating a method for checking and interpolating soil moisture sensor data quality according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0053] Step S1: Based on the preset core physical feature list, the multivariate soil moisture time series data is filtered, outlier detection is performed through physical rule constraints and the isolated forest algorithm, and the data is labeled by creating a complete time axis.

[0054] This step aims to comprehensively clean and complete the timeline of IoT soil moisture data, and impose constraints based on the data's own logical knowledge, providing high-quality input for subsequent model training.

[0055] First, in this embodiment of the invention, multivariate time series data is loaded from the original data source (such as a CSV file), and then based on a preset list of core physical characteristics F... c ore = f1, f2, ..., f k Filter out the target data columns to be used for enhancement.

[0056] Then, this embodiment of the invention performs a systematic rationality judgment and outlier detection on the screened core feature data, specifically including: a. Hard constraints based on physical rules: Using domain prior knowledge, the data is initially cleaned. Any data point that violates such hard rules is considered a deterministic anomaly and is immediately replaced with an invalid value (NaN) for subsequent imputation. b. Association anomaly detection based on unsupervised learning: To identify the inherent, deeper-hidden multivariate coupling anomalies in the data, this step uses the Isolation Forest algorithm. This algorithm is an efficient, non-parametric anomaly detection method. Its basic principle is that outliers are easier to isolate in the feature space due to their sparsity. The algorithm is trained on all multivariate data points without missing values, and outputs an anomaly score s(x) for each data point x. Based on this score, points with high anomaly probability (e.g., s(x) close to 1) are identified as anomalies and replaced with invalid values ​​(NaN). In this embodiment, data points with anomaly scores s(x) higher than 0.6 are identified as anomalies and replaced with invalid values ​​(NaN). This step can effectively identify outliers where individual features seem reasonable, but the combination of multiple features does not conform to the overall data distribution pattern.

[0057] Furthermore, this embodiment of the invention generates a complete and uninterrupted timeline index based on the user-defined start and end dates at a preset time frequency. The quality-controlled data is then resampled (reindexed) onto this complete timeline, thereby transforming the problem of missing discontinuous dates into a standardized problem of missing time point values. A "Data Source" column is further created, labeling time points existing in the original data as "Original Data" and time points added through timeline completion as "Model Supplement".

[0058] Step 102: Training samples are generated from the preprocessed data by sliding window sampling, and deep feature learning is performed using a denoising network based on a spatiotemporal diffusion probability model.

[0059] The core task of this step is to train a generative model capable of deeply understanding the inherent spatiotemporal relationships of the data, using the high-quality dataset that has undergone preprocessing and common-sense injection in step S1. This model will then be used to impute missing values ​​in the original data with high precision and high fidelity.

[0060] It is understandable that, in order for the model to effectively learn the dynamic evolution patterns of time series data, continuous time series data needs to be transformed into sample pairs suitable for supervised learning. In this embodiment of the invention, the "pseudo-real" dataset X obtained after preprocessing is... g t (a matrix of shape [number of time points, number of features]) is sampled using a sliding window of fixed length L. Considering that the response of soil moisture to meteorological changes typically lasts from several hours to several days, this invention uses a sliding window of length L = 72 (hours) to capture the complete multi-day dynamic pattern. The data X within each window... w =X t ,X t+1 ,...,X t+L-1 Each sample constitutes an independent training sample. This method ensures that each sample contains sufficiently long, continuous temporal context information. Therefore, the final training set can be represented as D. t rain = X w1 ,X w2 ,..., where each sample X w They are all tensors of shape [L,M], where M is the total feature dimension (including core physical features and all injected common sense features).

[0061] In this invention, the denoising network based on the spatiotemporal diffusion probability model is a hybrid model incorporating spatial correlation and temporal dependence. The spatial dimension uses a graph neural network to propagate information from feature nodes, while the temporal dimension uses a Transformer self-attention mechanism to model the time series, forming a spatiotemporally fused denoising network. Specifically:

[0062] (1) Graph construction and GNN spatial modeling.

[0063] In this invention, graph representation is defined as follows: M input features are constructed into a graph G = (V, E), where the size of the node set V is M, and each node represents a feature. Edges E are constructed based on prior knowledge (such as physical associations) or statistical correlations between features. For soil moisture data, a fully connected graph can be constructed to represent potential associations between all features. When constructing the graph structure, edges are first established between feature nodes in adjacent soil layers based on prior physical knowledge. Subsequently, the time-series Pearson correlation coefficients between all features are calculated, and edges are added for feature pairs with correlation coefficients greater than 0.7 to capture potential remote sensing coupling relationships between non-adjacent layers.

[0064] In this invention, the Graph Neural Network (GNN) layer is specifically implemented as a Graph Convolutional Network (GCN) and used as a spatial information aggregator. In each processing step of the denoising network, data (shape [L, M]) within a time window propagates information along the feature dimension through the GCN layer. For any node (feature) i, its updated representation vector h′... i This can be formally represented as:

[0065]

[0066] Among them, h j Let be the representation vector of neighboring node j (and node i itself), W be the learnable weight matrix, deg(i) and deg(j) be the degrees of nodes i and j, respectively, and σ be the non-linear activation function. This operation updates the feature representation of the center node by aggregating the normalized weighted neighborhood information.

[0067] (2) Transformer time modeling.

[0068] In this invention, a self-attention mechanism is used as a temporal information aggregator. After processing by the GCN layer, the feature sequence H′∈R with spatial context information is obtained. L×M It is fed into a standard Transformer encoder layer. The self-attention mechanism can dynamically calculate the correlation weights between all L time steps in the sequence, and its core calculation is as follows:

[0069]

[0070] Where Q, K, and V are the query, key, and value matrices, respectively. This is a scaling factor used to stabilize the gradient; the softmax function converts scores into probabilistic weights. This formula assigns weights to values ​​by calculating the similarity between the query and the key, allowing the model to focus on the most relevant parts of the input sequence.

[0071] (3) Diffusion model generation framework.

[0072] For the forward noise addition process, this embodiment of the invention defines a fixed Markov chain, which gradually adds Gaussian noise to the original data x0 over T discrete time steps until it becomes pure isotropic Gaussian noise x0. T The noise addition process at any step t can be directly calculated using the following formula:

[0073]

[0074] Where x0 is the original clear data, x t These are noisy data at time t. These are predefined noise scheduling parameters. This represents a Gaussian distribution. This operation defines arbitrary intermediate states in the forward diffusion process by scaling the original data and adding Gaussian noise of a specific intensity.

[0075] For the reverse denoising process (the goal of model learning), the model's task is to learn a neural network ε. θ (x t The inverse process p of this noise-adding process can be approximated by ,t). θ (x t-1 |x t ), that is, from noisy data x t The added noise ε is predicted in the model. The "GNN+Transformer" spatiotemporal module designed in this invention plays the role of this noise reduction network ε. θ The role.

[0076] It should be noted that the interpolation task of this invention is a conditional generation problem. During training, this invention randomly masks the "pseudo-real" data sample x0 to generate a mask matrix Mask. The unmasked portion x0 [Mask=1] will serve as the main condition for the generation process.

[0077] In this invention, the optimization objective of the model is to predict the noise ε at all time steps t and for all samples x0. θ The difference between the noise ε added and the actual noise is minimized. The loss function uses the mean squared error (MSE), as shown in the following formula:

[0078]

[0079] Where ∈ is real random noise, ∈ θ This is a neural network used to predict the noise, where x0 is the original data and Mask is additional conditional information. The objective function trains the model to master the inverse denoising ability to recover data from noise by minimizing the difference between the predicted noise and the actual noise.

[0080] During training, this invention optimizes the parameters θ using stochastic gradient descent (SGD) or its variants (such as the Adam optimizer) until the loss function converges. During training, this invention retains the model parameters that result in the lowest loss on the validation set as the final model used for imputation.

[0081] Step S3: Use the trained spatiotemporal diffusion probability model to impute missing values ​​in the original data, generate noisy data samples through the forward noise addition process, and perform conditional data imputation based on known observations and mask matrix during the reverse noise reduction process to generate preliminary imputation results.

[0082] This step aims to apply the optimal spatiotemporal diffusion model trained in S2 to perform high-fidelity imputation on the data processed in S1, which contains various missing values ​​(missing values, discontinuous dates, outlier markings), and through a key post-processing correction stage, ensure that the final output data has high physical and logical credibility.

[0083] It is understandable that the conditional generative interpolation process is essentially a conditional generation task based on known observations. This embodiment of the invention utilizes a pre-trained denoising network ε. θ Missing data is imputed by performing an iterative reverse denoising process. The specific steps are as follows:

[0084] First, initialization is performed by taking the preprocessed dataset X containing missing locations. proc This includes known observations X. obs And a missing mask. Then, from a standard Gaussian noise distribution with the same dimensions as the target data. Mid-sampling yields the initial noise tensor x. T Next, iterative denoising sampling is performed, from t=T to t=1, executing T steps of Markov chain reversal process p. θ (x t-1 |x t At each time step t, the core operations are as follows:

[0085] Conditional injection: To ensure that the generation process is based on known data, x is first... t The part corresponding to the known observations is represented by the forward process q(x). t The noisy observation q(x) calculated by |x0) t |X obs Replace with ). This operation can be represented as:

[0086]

[0087] Where Mask is a binary mask specifying a known region, xt It is the noisy image at the current moment, x obs is a known, clear image, and ⊙ is the element-wise product. This formula guides the reverse generation process of the diffusion model by re-injecting correctly noisy observation data into a known region, and is often used in image restoration or editing tasks.

[0088] Noise prediction: conditionally injected The time step t, the mask, and the graph structure E are input into the trained spatiotemporal denoising network ε. θ In the middle, the noise is predicted

[0089] Denoising step: Using the predicted noise, calculate clearer data x according to the following formula. t-1 :

[0090]

[0091] in, It is the noisy image at the current moment, ∈ θ It is a neural network for predicting noise, α t and It is the noise dispatch coefficient, σ t z is the random noise term. This formula performs a step-by-step denoising operation by subtracting the predicted noise from the current image and adding a small amount of randomness, thereby gradually recovering a clear image from the noise.

[0092] Output: When t=0, the final dataset X with preliminary interpolation completed is obtained. imputed .

[0093] Step S4: Perform post-processing correction on the preliminary interpolation results, including clamping correction based on the monthly historical range and correction based on interlayer physical logic, to ensure that the interpolation data conforms to the actual soil moisture change pattern in terms of temporal evolution and spatial distribution.

[0094] To address the issue of deep learning models potentially generating physically unreasonable results, this invention introduces a crucial post-processing correction stage to correct the initial interpolation result X. imputed Implement mandatory constraints.

[0095] The principle of correction based on monthly historical range is to utilize each core feature f calculated in S1. k The historical maximum value M of m in each month k,m and minimum value m k,m This serves as the effective data boundary for that month. During execution, for any data point x that is imputed by the model... k,t Its final correction value x′ k,t Determined by the clamping function:

[0096] x′ k,t =clip(x) k,t ,m k,m(t) M k,m(t) )

[0097] Where, x k,t is the input value in dimension k at time t, clip is a function of the constraint range, and m and M are the upper and lower bounds that dynamically change with time t and dimension k. This operation applies dynamic constraints or hard guidance to the iterative process by forcibly constraining the values ​​within a time-varying range.

[0098] This operation ensures that all interpolated values ​​(such as soil moisture content) do not exceed their historical experience range for that season, effectively avoiding counterintuitive results such as interpolating soil moisture saturation during a continuous drought season.

[0099] The principle of correction based on physical logic is to perform a consistency check on a set of features with inherent physical logical relationships. In the execution process, taking the observations of a multi-layer soil moisture sensor as an example, the data must follow basic physical laws. This invention performs the following logical judgment and correction on the interpolation results for each day: Generally, the moisture changes in shallow soil (e.g., 20cm) are more drastic and respond faster than in deeper soil (e.g., 100cm). A key physical logic is that, without bottom water replenishment, the moisture content of deeper soil should not be significantly higher than that of its adjacent shallower soil.

[0100] To enable those skilled in the art to better understand the technical solutions in this application, the present invention also provides several embodiments for reference.

[0101] The experimental data used in the following examples all come from IoT soil moisture collectors deployed in a standardized experimental field in Fukang, Xinjiang. This dataset records data from 22:38:00 on June 18, 2024 to 14:54:00 on September 10, 2024.

[0102] Soil volumetric water content information was collected hourly using sensors. The monitoring indicators covered five vertical depth layers from the surface to the deepest layers: 20cm, 40cm, 60cm, 80cm, and 100cm. The data from each depth layer not only exhibited individual trends over time, but also showed significant correlations and coupling relationships in their trends among different depth layers. This provides a data foundation for the spatiotemporal modeling of this invention.

[0103] Example 1: Performance verification of imputation for randomly missing data.

[0104] Objective: This embodiment aims to verify the interpolation accuracy and reliability of the method described in this invention in handling the most common scenario of missing random and discrete data points in practical applications.

[0105] Data Preparation and Missing Data Scenario Construction: To construct a quantifiable test set, soil moisture content data at a depth of 60cm from June 18th to July 18th, 2024, was first extracted from the original dataset as a complete, missing data subset. Subsequently, random masking was employed, randomly selecting 30% of the data points in this subset and replacing their values ​​with null values ​​(NaN). This operation aims to accurately simulate random data loss caused by factors such as transient network packet loss and single-sample failures of sensors.

[0106] Model imputation process: The spatiotemporal imputation model described in this invention is activated to process the data with 30% missing values. For each target data point set to null, the model's imputation process fully utilizes information from two dimensions:

[0107] Temporal context information: Through the Transformer module inside the model, the temporal evolution trend shown by the time point data that was not removed in the 60cm layer (i.e., the known data points before and after the missing point) is analyzed and utilized.

[0108] Spatial context information: Through the graph neural network module inside the model, the observation data of the other four depth layers (20cm, 40cm, 80cm, and 100cm) at the same time point are analyzed simultaneously. Since there is a physical correlation between the water content of different soil layers, GNN can effectively capture this interlayer (i.e., spatial dimension) correlation, providing strong corroborating information for the interpolation of the target layer.

[0109] Quantitative analysis of results: The data generated by model interpolation is compared point-by-point with the original real data before it was removed in step 1. The results are as follows: Figure 2 As shown. The coefficient of determination (R²) between the two was calculated. 2 The coefficient of determination (R²) reached 0.89. 2 The value of 0.89 characterizes the extent to which the model's predicted values ​​explain the variance of the true values. This indicates that the interpolated values ​​generated using the method of this invention can explain 89% of the variation in the original data, demonstrating extremely high goodness of fit. This result fully demonstrates that the method described in this invention can efficiently fuse spatiotemporal dual-dimensional information to achieve high-precision recovery of randomly missing data points.

[0110] Example 2: Robustness verification of imputation of data with continuous long-term missing data.

[0111] Objective: This embodiment aims to verify the performance limits and robustness of the method described in this invention under more challenging scenarios involving continuous long-term complete data loss. In practical applications, this scenario corresponds to situations such as sensors being offline for several consecutive days, or equipment maintenance or replacement.

[0112] Data preparation and missing data scenario construction: Soil moisture content data at a depth of 40cm was selected. In this data, a continuous missing data scenario was constructed by artificially removing all data points from 10 consecutive days from June 24 to July 4, 2024, creating a long-term "data vacuum".

[0113] Model imputation process: The same pre-trained spatiotemporal imputation model as in Example 1 is used to impute the missing data for these 10 consecutive days. In this scenario, since the target layer has no local temporal context information available for reference over a period of 10 days, traditional time series imputation methods (such as linear interpolation, ARIMA, etc.) will completely fail or produce huge deviations. However, the core advantage of the model in this invention lies in its ability to shift the focus of information dependence. In the case of severe lack of local temporal context information, it effectively utilizes synchronous observation data from other deep layers (i.e., spatial dimensions) within these 10 days as a key information source. The GNN module captures the changing trends of data from other layers, providing crucial constraints and guidance for the generation of the target layer.

[0114] Quantitative analysis of results: such as Figure 3 As shown, the model-interpolated data for 10 consecutive days is compared with the original real data for that period. The coefficient of determination (R²) between the two is calculated. 2 The accuracy still reached 0.85. This result strongly demonstrates the superiority of the method of this invention: even under extreme conditions where the target sequence itself has severely insufficient information, the model can still infer the dynamic changes of missing data from other relevant variables and perform high-precision imputation, thanks to its powerful spatiotemporal information fusion capability. This demonstrates excellent robustness and generalization ability, effectively solving the technical problem that existing technologies suffer from significant accuracy degradation or even inability to handle continuous long-term missing data.

[0115] Example 3: Validation of synergistic interpolation of soil moisture content at multiple depths.

[0116] Objective: This embodiment aims to verify that the method described in this invention can perform high-precision and coordinated interpolation of missing data at multiple depth layers within the target monitoring point when the sensor network of the target monitoring point (Daejeon 1) experiences overall data loss.

[0117] Data preparation and missing scenario construction: Three geographically adjacent experimental fields with similar environmental conditions were selected as data sources. Soil moisture sensors were installed in each experimental field at four depths: 20cm, 40cm, 60cm, and 80cm.

[0118] This embodiment constructs a simulated scenario where the sensor network of Daejeon-1 experiences a sudden failure, such as a power outage, resulting in a complete interruption of signal transmission and a loss of synchronous data across all monitoring levels. To simulate this scenario, an artificially constructed data loss period (e.g., December 25, 2024 to December 31, 2024) is created. During this period, all observation data from Daejeon-1 is masked to create a "data vacuum." Concurrently, data from Daejeon-2 and Daejeon-3 remain complete, serving as the spatial reference base required for the interpolation process.

[0119] Model interpolation process: The spatiotemporal interpolation model described in this invention, which has been trained using historical data, is used to perform parallel interpolation on all four deep-layered missing data segments in Daejeon-1. During this process, the model of this invention demonstrates its unique spatiotemporal information coupling processing capability: its GNN module is responsible for capturing and modeling the spatial correlation between the references Daejeon-2 and Daejeon-3 and the target Daejeon-1; simultaneously, its Transformer module encodes the temporal dynamic evolution of each data sequence.

[0120] The core innovation of this model lies in its ability to efficiently integrate the dynamic features learned from spatial reference sources with the laws of temporal evolution, and then transfer this integrated spatiotemporal knowledge representation to the data generation task of the target experimental field A, thereby accurately inferring the most likely data evolution trajectory within the missing period.

[0121] Results Quantitative Analysis and Illustration: To objectively evaluate the interpolation performance, four sets of interpolated data generated for the Daejeon-1 model were compared point-by-point with pre-masked real data. The visualized evaluation results are as follows: Figure 4 , 5 As shown in Figures 6 and 7.

[0122] Calculations showed that the interpolation accuracy quantification index (coefficient of determination R²) for each depth layer reached a high level, as detailed below:

[0123] 20cm soil depth layer: coefficient of determination R² = 0.81;

[0124] 40cm soil depth layer: coefficient of determination R² = 0.80;

[0125] Soil depth layer of 60cm: coefficient of determination R² = 0.82;

[0126] 80cm soil depth layer: coefficient of determination R2 = 0.82.

[0127] like Figure 4 , 5 As shown in Figures 6 and 7, the interpolation results of each data group exhibit high fidelity in two key dimensions: First, in terms of trend, the dynamic change trajectory of the generated data (shown by the orange dashed line in the figure) shows a very high consistency with the peak and valley characteristics and change rate of the original data (shown by the blue solid line in the figure); Second, in terms of distribution, the probability density curves of the two data are highly overlapping, indicating that the generated data completely reproduces the statistical distribution characteristics of the original data.

[0128] The combined results of the determination coefficients, all above 0.80, and the significant visual consistency strongly demonstrate that the method of this invention possesses the ability to perform high-precision and high-reliability collaborative interpolation when faced with multivariate and synchronous data loss problems, showcasing its outstanding practical application value and technical superiority.

[0129] Through the verification of the above three embodiments, the data augmentation method based on the spatiotemporal diffusion model described in this invention exhibits high accuracy and robustness in both scenarios facing random discrete missing data and continuous long-term missing data. The results of the embodiments fully demonstrate the feasibility and advancement of the technical solution of this invention, which can provide high-quality, highly reliable, and complete time-series data for the agricultural field, and has significant practical application value.

[0130] To achieve the above embodiments, the present invention also proposes a soil moisture sensor data quality inspection and interpolation device. Figure 8 This is a schematic diagram of a soil moisture sensor data quality inspection and interpolation device provided in an embodiment of the present invention. Figure 8 As shown, the device includes:

[0131] The preprocessing module 100 is used to filter multivariate soil moisture time series data based on a preset list of core physical features, detect outliers through physical rule constraints and the isolated forest algorithm, and annotate data by creating a complete time axis.

[0132] The spatiotemporal fusion modeling module 200 is used to generate training samples from preprocessed data through sliding window sampling and to perform deep feature learning using a denoising network based on a spatiotemporal diffusion probability model.

[0133] The interpolation generation module 300 is used to interpolate missing values ​​in the original data using a trained spatiotemporal diffusion probability model. It generates noisy data samples through a forward noise addition process and performs conditional data interpolation based on known observations and a mask matrix during the reverse denoising process to generate preliminary interpolation results.

[0134] The post-processing correction module 400 is used to perform post-processing correction on the preliminary interpolation results, including clamping correction based on the monthly historical range and correction based on interlayer physical logic, to ensure that the interpolation data conforms to the actual soil moisture change pattern in terms of temporal evolution and spatial distribution.

[0135] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0136] To implement the above embodiments, the present invention also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0137] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0138] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0139] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0141] This invention is intended to provide implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0142] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0143] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0144] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0146] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0147] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0148] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0149] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0150] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for checking and interpolating soil moisture sensor data quality, characterized in that, Includes the following steps: Multivariate soil moisture time series data are filtered based on a pre-defined list of core physical features, outlier detection is performed through physical rule constraints and the isolated forest algorithm, and data is labeled by creating a complete time axis. Training samples are generated from preprocessed data through sliding window sampling, and deep feature learning is performed using a denoising network based on a spatiotemporal diffusion probability model. The missing values ​​in the original data are imputed using a trained spatiotemporal diffusion probability model. Noisy data samples are generated through a forward noise addition process, and conditional data imputation is performed based on known observations and a mask matrix during the reverse noise reduction process to generate preliminary imputation results. The preliminary interpolation results are post-processed and corrected, including clamping correction based on monthly historical range and correction based on interlayer physical logic, to ensure that the interpolation data conforms to the actual soil moisture change patterns in terms of temporal evolution and spatial distribution.

2. The method according to claim 1, characterized in that, The process involves filtering multivariate soil moisture time series data based on a pre-defined list of core physical features, detecting outliers using physical rule constraints and the isolated forest algorithm, and annotating the data by creating a complete time axis, including: Multivariate time series data are extracted from the raw soil moisture data, and features are selected based on a pre-defined list of core physical features to filter out features related to soil moisture changes. Use physical rules to constrain the filtered data and exclude data points that do not conform to actual physical laws; The isolated forest algorithm is used to detect outliers in the data, identify and mark outlier data points, and further remove or correct these outliers. Create a complete timeline based on data timestamps and other time information, and label each data point with its corresponding time to ensure data consistency over time.

3. The method according to claim 2, characterized in that, The denoising network based on the spatiotemporal diffusion probability model is a hybrid model incorporating spatial correlation and temporal dependence. The spatial dimension uses a graph neural network to propagate information from feature nodes, while the temporal dimension uses a Transformer self-attention mechanism to model the time series, forming a spatiotemporally fused denoising network. In the graph neural network, nodes represent soil moisture sensors or related environmental variables, and edges are constructed based on the interlayer moisture conduction relationship or statistical correlation to form a graph structure with physical semantics. The Transformer self-attention mechanism calculates correlation weights between all time steps in the time dimension to capture hysteresis responses and periodic fluctuations caused by irrigation or rainfall events.

4. The method according to claim 3, characterized in that, The training process of the spatiotemporal diffusion probability model includes: The forward noise addition process uses preset noise scheduling parameters to gradually add Gaussian noise to the original data in T discrete time steps until it becomes isotropic Gaussian noise. The reverse denoising process iteratively executes the inverse Markov chain process, injecting conditions at each step using known observations and a mask matrix to guide the generated path to conform to physical logic.

5. The method according to claim 4, characterized in that, The post-processing correction of the preliminary interpolation results includes clamping correction based on monthly historical range and correction based on interlayer physical logic, ensuring that the interpolation data conforms to the actual soil moisture change patterns in both temporal evolution and spatial distribution, including: The clamping correction based on the monthly historical range calculates the historical maximum and minimum values ​​of each core feature in each month to limit the boundaries of the interpolated values, thereby avoiding the generation of unreasonable data that exceeds the seasonal experience range. The correction based on interlayer physical logic determines whether the rate of change of shallow soil moisture is greater than that of deep soil moisture. If not, the deep soil moisture value is corrected to conform to the physical law of downward water conduction.

6. A soil moisture sensor data quality inspection and interpolation device, characterized in that, Includes the following steps: The preprocessing module is used to filter multivariate soil moisture time series data based on a preset list of core physical features, detect outliers through physical rule constraints and the isolated forest algorithm, and annotate data by creating a complete time axis. The spatiotemporal fusion modeling module is used to generate training samples from preprocessed data through sliding window sampling and to perform deep feature learning using a denoising network based on a spatiotemporal diffusion probability model. The interpolation generation module is used to interpolate missing values ​​in the original data using a trained spatiotemporal diffusion probability model. It generates noisy data samples through a forward noise addition process and performs conditional data interpolation based on known observations and a mask matrix during the reverse noise reduction process to generate preliminary interpolation results. The post-processing correction module is used to perform post-processing correction on the preliminary interpolation results, including clamping correction based on the monthly historical range and correction based on interlayer physical logic, to ensure that the interpolation data conforms to the actual soil moisture change pattern in terms of temporal evolution and spatial distribution.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images