Analysis method and system for automatically measuring soil moisture based on data processing

Abstract

The invention provides an analysis method and system for automatically measuring soil moisture based on data processing, and relates to the technical field of soil moisture measurement. The method comprises the following steps: deploying uniquely-identified multi-parameter sensing nodes and carrying out ad hoc networking; soil and meteorological multi-modal data are synchronously collected; quality control such as missing value filling, drift correction and abnormity elimination is carried out in real time; executing space-time preprocessing to construct multi-source features; and automatically generating a soil moisture estimation result of the target area based on the integrated learning model and the space time sequence information. According to the invention, a closed loop from data acquisition to accurate prediction can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of soil moisture measurement, in particular to an analysis method and system for automatically measuring soil moisture based on data processing. BACKGROUND

[0002] At present, soil moisture monitoring technology is roughly divided into three categories: in-situ measurement, wireless sensor network (WSN) and remote sensing inversion. The point measurement method has high accuracy but limited coverage; remote sensing can obtain large-scale data but the resolution is limited; WSN realizes real-time monitoring and data uploading in a certain range. With the maturity of 5G and low-power wide-area network, field sensor nodes can push volume moisture content, temperature, conductivity and other flux information at a second level, providing basic data for fine irrigation.

[0003] In the past five years, researchers have tended to improve estimation accuracy through multi-source data fusion and machine learning, and to weaken the dependence on single-source noise and calibration. The trends include: ① fusion of near-surface sensing, meteorology, satellite / Synthetic Aperture Radar (SAR) and crop growth stage data; ② light model inference on the edge side to reduce cloud bandwidth; ③ automatic evaluation of data quality and online calibration; ④ closed-loop control, combined with intelligent irrigation systems to realize "measurement-estimation-control" integrated management.

[0004] Despite the progress made in the above technologies, there are still: ① sensor drift caused by long-term service, lack of general data quality self-checking and dynamic calibration mechanism; ② multi-source data of different scales and heterogeneous formats, high degree of manual involvement in the fusion process; ③ remote sensing methods can only invert the moisture content of the top 5cm, lacking deep information; ④ real-time analysis relies on stable network, and edge-cloud collaboration is insufficient; ⑤ lack of unified abnormality detection and alarm logic, making it difficult to form a transferable automatic analysis framework. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide an analysis method and system for automatically measuring soil moisture based on data processing, which builds a closed loop from high-quality data acquisition to multi-dimensional fusion modeling, and can automatically estimate the soil moisture of the target area in real time, accurately and reliably.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] An analysis method for automatically measuring soil moisture based on data processing, comprising:

[0008] Deploying nodes of soil-environment multi-parameter sensors with unique identification, and completing self-identification and topology construction of the nodes through self-organizing network protocol;

[0009] synchronously collecting volumetric water content data, temperature data, dielectric constant data and meteorological data within a preset sampling period using the node to generate original multi-modal data stream;

[0010] performing real-time data quality control on the original multi-modal data stream to obtain quality control data; the data quality control includes missing value detection, drift correction and outlier removal;

[0011] spatial and temporal heterogeneity preprocessing is performed on the quality control data to obtain preprocessed multi-source features;

[0012] The multi-source features are input into a soil moisture estimation model based on ensemble learning to generate soil moisture estimation results for the target area.

[0013] Preferably, it further comprises:

[0014] The soil moisture estimation results are compared with a preset crop water threshold to generate an irrigation decision signal, which is pushed to an irrigation execution terminal through an edge-cloud collaborative mechanism;

[0015] Real-time soil moisture data returned after irrigation is recorded, and the soil moisture estimation model is trained online and adaptively based on a Bayesian update algorithm and the real-time soil moisture data to realize self-evolution of the soil moisture estimation model.

[0016] Preferably, nodes with soil-environment multi-parameter sensors with unique identifiers are deployed, and self-identification and topology construction of the nodes are completed through a self-organizing network protocol, including:

[0017] In the deployment stage, a unique identifier based on EUI-64 format is written for each node, and the unique identifier is encrypted and stored in the non-volatile memory of the node;

[0018] After the node is powered on, a broadcast message carrying the unique identifier is periodically sent within a preset handshake window, and broadcast messages of adjacent nodes are synchronously listened to;

[0019] The unique identifiers of adjacent nodes and the corresponding received signal strength indication values RSSI are generated to form a neighbor list;

[0020] Based on the neighbor list, a parent node is determined according to a preset parent node selection algorithm to generate a local routing table, realizing node self-identification;

[0021] The neighbor list and the local routing table are reported to an edge gateway through the self-organizing network protocol, and the edge gateway aggregates to form a network topology structure of the target monitoring area, and sends a synchronization confirmation message to all nodes to complete topology construction.

[0022] Preferably, real-time data quality control is performed on the original multi-modal data stream to obtain quality control data, comprising:

[0023] The original multi-modal data stream is time-series merged according to the unique identifier of the node within a preset monitoring time window to generate a to-be-detected data sequence;

[0024] A sampling interval threshold determination is performed on the to-be-detected data sequence, missing sampling points are marked, and missing data is interpolated and filled based on a similar time period historical mean-KNN regression fusion algorithm to obtain completed missing value data;

[0025] A sliding baseline is calculated for the completed missing value data, a point-by-point recursive slope significance test is used to detect sensor drift, if drift is detected, a recursive least square method is used to estimate the drift amount in real time and complete dynamic correction, and drift correction data is output;

[0026] The three-sigma principle is applied to the drift correction data in combination with an isolation forest model to perform anomaly discrimination, data points exceeding the confidence interval are removed or replaced with a local weighted median, and the quality control data is output.

[0027] Preferably, the calculation formula of the similar time period historical mean-KNN regression fusion algorithm is:

[0028]

[0029] Wherein, α i,t is a mixing coefficient, ω ij,t is a weight, and the mixing coefficient and the weight are adaptively determined by and ; is the missing soil moisture interpolation result of node i at time t; X i,s is the measured soil moisture of node i in the historical similar time period s; is a set of historical time points in the same daily phase as time t; is the number of elements in the set ; K is the number of neighbor nodes for KNN regression; ω ij,t is the similarity weight of node j for node i at time t; d ij is the spatial distance between node i and node j; ΔT ij,t is the soil temperature difference between node i and node j at time t; λ d , λ T are attenuation scale parameters of the spatial distance and the temperature difference, respectively; α i,t is an adaptive fusion coefficient of the historical mean and the KNN estimation, and takes a value of 0-1; is the sample variance of the soil moisture of node i in the set ; is the sample variance of the K neighboring soil moisture values of node i at time t; is the sample variance of the K neighboring soil moisture values of node i at time t; is the sample variance of the K neighboring soil moisture values of node i at time t; is the sample variance of the K neighboring soil moisture values of node i at time t; is the sample variance of the K neighboring soil moisture values of node i at time t; β is a positive adjustment parameter controlling the sensitivity of the fusion coefficient.

[0030] Preferably, a sliding baseline is calculated for the data complementing the missing values, a point-by-point recursive slope significance test is used to detect sensor drift, if drift is detected, a recursive least square method is used to estimate the drift amount in real time and complete dynamic correction, and drift-corrected data is output, including:

[0031] A time sliding window with a length of L b is set, and a sliding median operation is performed on the data complementing the missing values X t to obtain a baseline sequence B , wherein is a set of consecutive L b data points from t-L b +1 to t;

[0032] Based on the recursive least square formula k t = k t-1 + g t ((X t -B t )-(k t-1 )), The current slope k t is updated in real time; λ is a forgetting factor; P t is the error covariance; P t-1 is the error covariance at the previous time; B t is the median result, i.e., the sliding baseline at time t; k t is the current slope estimated by recursion, i.e., the drift trend coefficient; k t-1 is the slope estimate at the previous time; g t is the recursive least square gain coefficient;

[0033] A one-point t-test is performed on k t , and if , it is determined that the node sensor has drifted and correction is triggered; t α is a preset significance threshold;

[0034] After drift triggering, a recursive least square method δt =δ t-1 +g t ((X t -B t )-δ t-1 ) for the drift offset δ t Perform online estimation and update P synchronously t With g t ; Among them, δ t-1 is the drift offset of the previous moment;

[0035] Generate drift correction data in real time and in The calibration mode is automatically released after M sampling cycles have passed. M is the number of sampling cycles that must continuously meet the steady-state condition to determine drift elimination.

[0036] Preferably, the quality control data is preprocessed for temporal and spatial heterogeneity to obtain preprocessed multi-source features, including:

[0037] Aligning the quality control data corresponding to the nodes of each soil-environment multi-parameter sensor with timestamps according to a unified reference sampling interval, and performing linear interpolation or nearest neighbor extension on missing time slices to obtain a continuous aligned data sequence;

[0038] Based on the latitude and longitude information of the nodes, the geographic coordinates corresponding to the aligned data sequences are projected into a unified coordinate system, and the elevation and soil depth labels are marked in the same survey area to form a data block with spatial attributes;

[0039] For the data blocks of the same time section, a weighted interpolation method is used to generate complete grid data on a regular grid of a set resolution;

[0040] Different physical quantities in the grid data are normalized using interval scaling rules, and then organized according to node-time-parameter three-dimensional indexes to obtain preprocessed multi-source features.

[0041] Preferably, different physical quantities in the grid data are normalized using interval scaling rules, and then organized according to node-time-parameter three-dimensional indexes to obtain pre-processed multi-source features, including:

[0042] For each physical quantity in the raster data, extracting a minimum value and a maximum value within the data range of the most recent complete crop growing season to establish a scaling interval for the physical quantity;

[0043] Converting the values ​​of the physical quantities of each of the grid data to a unified standard interval according to the scaling interval, retaining the original resolution and spatial position;

[0044] a three-dimensional index key composed of a unique identifier of a node, an observation timestamp, and a code of a physical quantity is assigned to each of the normalized grid data to form a key-value pair record;

[0045] The key-value pair records of all physical quantities are aggregated in parallel according to the three-dimensional index key to generate a node-time-parameter three-dimensional feature tensor, and the node-time-parameter three-dimensional feature tensor is stored in a feature database to obtain the multi-source feature.

[0046] Preferably, the soil moisture estimation model comprises an input layer, a spatial feature integration layer, a time-dependent cycle layer, an attention fusion layer, and a regression output layer; the input layer is connected in parallel with the spatial feature integration layer and the time-dependent cycle layer, respectively; the outputs of the spatial feature integration layer and the time-dependent cycle layer are jointly connected to the attention fusion layer; the output of the attention fusion layer is independently connected to the regression output layer; the input layer is used to receive the preprocessed multi-source feature, and the multi-source feature is respectively split in the spatial dimension and the time dimension to form parallel input streams; the spatial feature integration layer is composed of multiple gradient boosting decision tree sub-models in parallel; the spatial feature integration layer is used to generate a spatial representation vector of a node scale using the spatial dimension data in the parallel input stream; the time-dependent cycle layer is composed of a bidirectional long short-term memory network; the time-dependent cycle layer is used to receive the time series data in the parallel input stream and extract time-dependent features of continuous observations; the attention fusion layer is used to input the spatial representation vector and the time-dependent features after aligning them by node index, dynamically allocate weights of the two, and generate a fusion feature vector; the regression output layer is used to perform a full connection operation on the fusion feature vector and output the soil moisture estimation result of the target area.

[0047] An analysis system for automatically measuring soil moisture based on data processing, comprising:

[0048] A node deployment unit is configured to deploy nodes of soil-environment multi-parameter sensors with unique identifiers, and complete self-identification and topology construction of the nodes through a self-organizing network protocol;

[0049] A data acquisition unit is configured to acquire volume moisture content data, temperature data, dielectric constant data, and meteorological data synchronously within a preset sampling period through the nodes to generate an original multi-modal data stream;

[0050] A quality control unit is configured to perform real-time data quality control on the original multi-modal data stream to obtain quality control data; the data quality control comprises missing value detection, drift correction, and outlier removal;

[0051] a data preprocessing unit configured to perform spatiotemporal heterogeneity preprocessing on the quality control data to obtain preprocessed multi-source features;

[0052] an integrated learning unit configured to input the multi-source features into a soil moisture estimation model based on integrated learning to generate a soil moisture estimation result of the target region.

[0053] According to the specific embodiments of the present application, the following technical effects are disclosed:

[0054] A complete data closed loop is constructed through an integrated architecture of "end-network-cloud". Firstly, a self-organizing network is formed by a plurality of parameter nodes with unique identification, and fast self-recognition and robust topology are achieved. Then, soil-weather multi-modal data are synchronously acquired within the same sampling period, and hardware errors and environmental noise are removed through quality control processes such as missing value detection, drift correction and outlier removal. Then, time alignment, spatial interpolation and dimension unification are completed through spatiotemporal heterogeneity preprocessing to generate structured multi-source features. Finally, the multi-source features are input into an integrated learning estimator which combines tree models and recurrent networks, so as to significantly improve the prediction accuracy and timeliness of soil moisture at different scales, and provide a high-reliability data foundation for precise irrigation and drought-flood risk assessment. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 A method flowchart is provided for the embodiments of the present application.

[0057] Figure 2 A system structure schematic diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] The purpose of the present application is to provide an analysis method and system for automatically measuring soil moisture based on data processing, which constructs a closed loop chain from high-quality data acquisition to multi-dimensional fusion modeling, and can automatically estimate the soil moisture of the target region in real time, accurately and reliably.

[0060] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0061] Figure 1 The method flowchart provided by the embodiment of the present application is shown in Figure 1 The present application provides an analysis method for automatically measuring soil moisture based on data processing, which comprises:

[0062] Step 100: deploying a node of a soil-environment multi-parameter sensor with a unique identifier, and completing self-identification and topology construction of the node through a self-organizing network protocol;

[0063] Step 200: synchronously collecting volume moisture content data, temperature data, dielectric constant data and meteorological data in a preset sampling period by using the node, to generate an original multi-modal data stream;

[0064] Step 300: performing real-time data quality control on the original multi-modal data stream to obtain quality control data; the data quality control comprises missing value detection, drift correction and outlier removal;

[0065] Step 400: performing spatio-temporal heterogeneity preprocessing on the quality control data to obtain preprocessed multi-source features;

[0066] Step 500: inputting the multi-source features into a soil moisture estimation model based on ensemble learning to generate a soil moisture estimation result of a target area.

[0067] Preferably, it further comprises:

[0068] comparing the soil moisture estimation result with a preset crop water requirement threshold to generate an irrigation decision signal, and pushing the irrigation decision signal to an irrigation execution terminal through an edge-cloud collaborative mechanism;

[0069] recording real-time soil moisture data returned after irrigation, and performing online adaptive training on the soil moisture estimation model based on a Bayesian update algorithm and the real-time soil moisture data, to realize self-evolution of the soil moisture estimation model.

[0070] Specifically, in the present embodiment, after the soil moisture estimation result is calculated and generated, it will be automatically sent into a set of water requirement management strategy matched with the regional crop type, the system presets a target moisture range threshold for different crops, and dynamically adjusts the comparison interval in combination with the current growth stage. By comparing the estimated soil moisture value with the threshold interval, it is determined whether there is a risk of insufficient water supply or oversaturation.

[0071] After the comparison is completed, the embodiment generates an irrigation decision signal according to the result, which contains parameters such as irrigation start time, irrigation target depth, recommended water volume, duration, etc. The signal is preliminarily calculated and compressed by the lightweight scheduling unit deployed on the edge side, and then transmitted to the cloud platform through the edge-cloud collaborative mechanism. After secondary optimization according to large-scale weather prediction and regional irrigation historical data, the cloud platform issues operation instructions to specific irrigation control terminals for execution.

[0072] After the irrigation operation is completed, the sensing nodes configured in the embodiment will continue to collect the real-time changes of soil moisture and mark the collection time and the state before and after execution. The edge gateway preliminarily cleanses these returned data and uploads them to the cloud for subsequent model performance tracking and dynamic verification. At the same time, by comparing with the estimated results before irrigation, it can be judged whether the actual irrigation execution response effect meets the prediction expectation.

[0073] To realize the dynamic adaptability of the model, the embodiment constructs an observation-prediction residual sequence from the above-mentioned returned data and the original estimated data, and uses the Bayesian updating algorithm to recursively correct the key parameters of the soil moisture estimation model, such as model weight, error covariance, prior confidence, etc. This process is performed once a day in a set update window without interrupting the main prediction process, ensuring that the model can continuously adapt to environmental changes and sensor aging.

[0074] Finally, the updated model parameters are written into the model cache unit and automatically versioned and archived, facilitating model backtracking and evolution path analysis. The "model self-evolution" process described in the embodiment supports micro-level adjustment without retraining the full model, effectively improving the generalization ability and robustness of the model across seasons and plots, and significantly enhancing the stability and reliability of the soil moisture estimation results in long-term operation.

[0075] Preferably, nodes with soil-environment multi-parameter sensors with unique identifiers are deployed, and self-identification and topology construction of the nodes are completed through a self-organizing network protocol, including:

[0076] In the deployment stage, a unique identifier based on EUI-64 format is written for each node, and the unique identifier is encrypted and stored in the non-volatile memory of the node;

[0077] After the node is powered on, a broadcast message carrying the unique identifier is periodically sent within a preset handshake window, and broadcast messages of adjacent nodes are synchronously listened to;

[0078] The unique identifiers of adjacent nodes and the corresponding received signal strength indication values RSSI are generated to form a neighbor list;

[0079] Determine a parent node according to a preset parent node selection algorithm based on the neighbor list, generate a local routing table, and realize node self-recognition;

[0080] Report the neighbor list and the local routing table to an edge gateway through the self-organizing network protocol, aggregate the network topology structure of the target monitoring area by the edge gateway, and issue a synchronization confirmation message containing the parent-child relationship to each node to complete the self-recognition and topology construction of the entire monitoring area network.

[0081] Specifically, in the embodiment, the node generates a 64-bit unique identifier (UID) based on the EUI-64 format for each node through a special configuration tool at the factory or on-site deployment stage, then encrypts the UID using a symmetric encryption algorithm (such as AES-128), and writes the encrypted UID into the non-volatile memory (such as Flash or EEPROM) built in the MCU of the node. At the same time, a decryption key is pre-set in the node firmware to ensure subsequent secure handshake with the network.

[0082] In the embodiment, after the node is powered on, it enters a preset handshake window (for example, the first 60s after startup), and sends a broadcast message carrying the encrypted UID every fixed time interval (such as 500ms) through a broadcast message in the window. The message format includes: message type flag, message serial number, encrypted UID, and the message header carries the transmission power level information. At the same time, the node opens the receiving channel, synchronously listens to and caches the broadcast messages of adjacent nodes heard in the same channel, extracts the encrypted UID, received signal strength indication value (RSSI), and time stamp, and saves them to the local temporary neighbor list.

[0083] In the embodiment, the local temporary neighbor list is processed according to a preset parent node selection algorithm. The algorithm can select a parent node based on RSSI value priority, residual power or link quality metric comprehensive score method, and write the selected UID and corresponding metric value into the local routing table. Subsequently, the node packages and reports the local routing table and the neighbor list to the edge gateway in a specified message format through the self-organizing network protocol (such as RPL or autonomous lightweight protocol). After receiving all the node reports, the edge gateway aggregates the entire network topology structure diagram, and issues a synchronization confirmation message containing the parent-child relationship to each node to complete the self-recognition and topology construction of the entire monitoring area network.

[0084] Preferably, real-time data quality control is performed on the original multi-modal data stream to obtain quality control data, including:

[0085] Time series merging is performed on the original multi-modal data stream according to the unique identifier of the node within a preset monitoring time window to generate a to-be-detected data sequence;

[0086] Performing sampling interval threshold judgment on the to-be-detected data sequence, marking missing sampling points, and filling in the missing data based on a similar time period historical mean-KNN regression fusion algorithm to obtain completed data of missing values;

[0087] Calculating a sliding baseline for the completed data of missing values, detecting sensor drift by point-by-point recursive slope significance test, if drift is detected, estimating the drift amount in real time by using a recursive least square method and completing dynamic correction, and outputting drift-corrected data;

[0088] Applying a three-sigma principle combined with an isolation forest model to perform abnormality discrimination on the drift-corrected data, removing or replacing data points exceeding a confidence interval with a locally weighted median, and outputting the quality control data.

[0089] Optionally, within each preset monitoring time window (for example, 10 consecutive minutes), the volume moisture content, temperature, dielectric constant and meteorological parameters collected by each node are merged according to the unique node identifier to form a multi-modal data stream arranged in ascending order of time stamp. In specific implementation, the edge gateway first removes duplicates from the received raw messages, and then aggregates all data of the same node within the time window into a to-be-detected data sequence according to the millisecond-level time stamp, to ensure the data integrity and orderliness of each node.

[0090] In this embodiment, the generated to-be-detected data sequence is first subjected to sampling interval threshold judgment: if the time difference between two adjacent data exceeds a preset threshold (for example, 1s±100ms), the missing time point is marked; then, a similar time period historical mean-KNN regression fusion algorithm is called to interpolate and fill in the missing points, the historical data time window can be set as the average value of the corresponding time period within the previous and subsequent 24 hours, and the data of the adjacent 5 nodes are selected as KNN candidates to fuse the space-time characteristics, so as to achieve high-precision filling of the missing sampling points, and output the completed data of missing values.

[0091] In this embodiment, the sliding baseline method and the recursive least square algorithm are combined to detect and correct sensor drift for the completed data of missing values. Specifically, a median baseline is calculated with a sliding window of 1 hour, and the drift trigger is judged by recursive slope significance test when each new data point arrives; if the slope exceeds the set significance threshold (for example, α=0.05), the recursive least square subroutine is started to estimate and remove the drift offset in real time, and the drift-corrected data is generated; then, the three-sigma principle combined with the isolation forest model is used to discriminate abnormal points for the corrected data, and the single point or short pulse exceeding the confidence interval is removed or replaced with the weighted median of adjacent data, and finally the high-confidence quality control data is output to ensure the data reliability of the subsequent analysis module.

[0092] Preferably, the calculation formula of the similar time period history mean-KNN regression fusion algorithm is:

[0093]

[0094] wherein, α i,t is a mixing coefficient, ω ij,t is a weight, the mixing coefficient and the weight are adaptively determined by and ; is the missing soil moisture interpolation result of node i at time t; X i,s is the measured soil moisture of node i at the historical similar time period s; is a set of historical time points in the same daily phase as time t; is the number of elements in the set ; K is the number of neighbor nodes for KNN regression; ω ij,t is the similarity weight of node j to node i at time t; d ij is the spatial distance between node i and node j; ΔT ij,t is the soil temperature difference between node i and node j at time t; λ d , λ T are the attenuation scale parameters of spatial distance and temperature difference respectively; α i,t is the adaptive fusion coefficient of the history mean and KNN estimation, taking a value of 0-1; is the sample variance of soil moisture of node i in the set ; is the sample variance of the K neighbor soil moisture values of node i at time t; is the sample variance of the K neighbor soil moisture values of node i at time t; is the sample variance of the K neighbor soil moisture values of node i at time t; is the sample variance of the K neighbor soil moisture values of node i at time t; is the sample variance of the K neighbor soil moisture values of node i at time t; β is a positive adjustment parameter for controlling the sensitivity of the fusion coefficient.

[0095] Specifically, each of the above parameters is obtained and determined by a combination of systematic offline calibration and online adaptation. Specifically, the time window range of the set of historical similar time periods is pre-set according to the typical daily variation period of the actual soil moisture and verified by multi-site data analysis; the number of neighbor nodes K and the spatial and temperature attenuation scale parameters λ d , λ T are optimized by an offline cross-validation method - different K, λ d , λ TThe statistical interpolation error is combined, and the parameter set with the minimum average absolute error is selected; the β parameter for controlling the sensitivity of the fusion coefficient is subjected to sensitivity analysis experiments under various soil types and weather conditions to ensure smooth transition of α between the historical mean value and the KNN estimation; during system operation, all parameters can also be fine-tuned according to the cumulative online measured data through algorithms such as least squares or maximum likelihood estimation, to ensure the continuous reliability and robustness of the interpolation performance under different fields and seasonal conditions.

[0096] Preferably, a sliding baseline is calculated for the missing value completion data, a point-by-point recursive slope significance test is used to detect sensor drift, if drift is detected, the drift amount is estimated in real time using a recursive least squares method and dynamic correction is completed, and drift correction data is output, including:

[0097] A time sliding window with a length of L b is set, and a sliding median operation is performed on the missing value completion data X t to obtain a baseline sequence , wherein is a set of consecutive L b data points from t-L b +1 to t;

[0098] Based on the recursive least squares formula k t =k t-1 +g t ((X t -B t )-(k t-1 )), The current slope k t is updated in real time; wherein λ is a forgetting factor; P t is an error covariance; P t-1 is the error covariance at the previous moment; B t is the median result, i.e., the sliding baseline at moment t; k t is the current slope estimated by recursive estimation, i.e., the drift trend coefficient; k t-1 is the slope estimation value at the previous moment; g t is the recursive least squares gain coefficient;

[0099] A single-point t-test is performed on k t , and if , it is determined that the node sensor has drifted and correction is triggered; wherein t α is a preset significance threshold;

[0100] After drift triggering, a recursive least squares method is used to estimate δ t =δ t-1 +g t ((Xt -B t )-δ t-1 ) for the drift offset δ t Perform online estimation and update P synchronously t With g t ; Among them, δ t-1 is the drift offset of the previous moment;

[0101] Generate drift correction data in real time and in The calibration mode is automatically released after M sampling cycles have passed. M is the number of sampling cycles that must continuously meet the steady-state condition to determine drift elimination.

[0102] Specifically, in this embodiment, the missing value completion data reported by the node is first median-calculated in a local buffer within a pre-set sliding window length to generate a continuous sliding baseline sequence. Whenever a new data point arrives, the median is calculated for that data point along with the historical data within the current window, dynamically updating the sliding baseline to reflect changes in the sensor's baseline level under normal conditions.

[0103] In this embodiment, a recursive least squares algorithm continuously tracks the instantaneous slope change of each new data point relative to the sliding baseline and compares it locally with the previous slope estimate. When the real-time estimated slope exceeds a pre-set significance threshold, drift detection is triggered, indicating that the sensor is drifting and initiating a correction process.

[0104] In this embodiment, after drift is triggered, the node enters the dynamic correction mode, uses the recursive least squares method to estimate the drift offset online, and subtracts the offset from each new data point in real time to generate drift correction data; at the same time, it continuously monitors the slope change after correction. After multiple sampling cycles that continuously meet the steady-state conditions, it automatically exits the correction mode and resumes normal sliding baseline and slope tracking to ensure that sensor drift is eliminated in a timely manner and the data output is continuous and reliable.

[0105] Preferably, the quality control data is preprocessed for temporal and spatial heterogeneity to obtain preprocessed multi-source features, including:

[0106] Aligning the quality control data corresponding to the nodes of each soil-environment multi-parameter sensor with timestamps according to a unified reference sampling interval, and performing linear interpolation or nearest neighbor extension on missing time slices to obtain a continuous aligned data sequence;

[0107] Based on the latitude and longitude information of the nodes, the geographic coordinates corresponding to the aligned data sequences are projected into a unified coordinate system, and the elevation and soil depth labels are marked in the same survey area to form a data block with spatial attributes;

[0108] For the same time section of the data block, a weighted interpolation method is used to generate complete grid data on a regular grid with a set resolution;

[0109] The different physical quantities in the grid data are normalized by interval scaling rules, and then organized by node-time-parameter three-dimensional index to obtain the pretreated multi-source features.

[0110] In this embodiment, after obtaining the quality-controlled multi-parameter data, each node first aligns the time stamps of all data records according to the pre-set unified reference sampling interval. The specific method is to compare the time stamps reported by different nodes with the reference time grid, mark the missing reference time, and use linear interpolation or nearest neighbor extension to smoothly fill the adjacent real data to ensure that each node corresponds to a set of continuous aligned data values at each reference time.

[0111] In this embodiment, after the aligned data sequence is associated with the latitude and longitude information of each node, the geographic coordinates are mapped to a unified plane coordinate system (such as UTM or latitude and longitude projection) using a general projection algorithm. In the same survey area, in combination with geographic information such as digital elevation model (DEM), the corresponding elevation value is labeled for each projected point, and the soil profile label is added according to different soil depth requirements, thereby forming a data block with spatial position, elevation and soil depth attributes.

[0112] In this embodiment, for all spatial attribute data blocks at the same observation time, a weighted interpolation method based on inverse distance or neighborhood weight is used to generate complete grid layers on a regular grid with a pre-set resolution (such as 1m x 1m or 5m x 5m). Subsequently, for each physical quantity in the grid data, interval scaling normalization processing is performed according to the previously determined minimum-maximum interval, and the normalized value is combined with the corresponding node unique identifier, time stamp and physical quantity code to form a three-dimensional index key for storage, finally forming a pretreated multi-source feature set that can be directly called by subsequent models.

[0113] Preferably, the different physical quantities in the grid data are normalized by interval scaling rules, and then organized by node-time-parameter three-dimensional index to obtain the pretreated multi-source features, including:

[0114] For each physical quantity in the grid data, the minimum value and the maximum value are extracted within the data range of the nearest complete crop growing season to determine the scaling interval of the physical quantity;

[0115] The values of the physical quantities of each grid data are converted to a unified standard interval according to the scaling interval, and the original resolution and spatial position are preserved;

[0116] a three-dimensional index key composed of a node unique identifier, an observation timestamp, and a code of a physical quantity is assigned to each of the normalized grid data, forming a key-value pair record;

[0117] The key-value pair records of all physical quantities are aggregated in parallel according to the three-dimensional index key, a node-time-parameter three-dimensional feature tensor is generated, and the node-time-parameter three-dimensional feature tensor is stored in a feature database, obtaining the multi-source feature.

[0118] In this embodiment, first, the minimum and maximum values of each physical quantity (such as soil moisture, temperature, conductivity, and light intensity) are calculated according to the historical grid data of each monitoring element in the last complete crop growing season (for example, from sowing to harvesting), to establish the scaling interval of the physical quantity, to ensure that the normalized data of different dimensions are comparable within the same standard interval, and to take into account the coverage requirements of extreme environments.

[0119] In this embodiment, the original values of the physical quantities corresponding to each grid cell at each time are mapped to a unified standard interval (such as 0-1 or -1-1) based on the scaling interval described above, but the original spatial resolution and latitude and longitude projection coordinates remain unchanged. Linear interval mapping is used in the normalization process to ensure that all grid points accurately reflect their original spatial distribution characteristics after normalization, laying the foundation for subsequent homogenization processing.

[0120] In this embodiment, for each normalized grid cell, a three-dimensional index key composed of a node unique identifier (representing the sensor node or virtual grid ID to which the cell belongs), an observation timestamp, and a physical quantity code (such as WM for moisture content and TP for temperature) is generated, and the key and the corresponding normalized value form a key-value pair record. Subsequently, all physical quantity records are aggregated in parallel according to the three-dimensional index key to construct a node-time-parameter three-dimensional feature tensor. The tensor is stored and managed in the database as the smallest access granularity, which is the multi-source feature.

[0121] Exemplarily, the multi-source feature refers to a high-dimensional, comprehensive data representation that can be directly used for model input, which is structured and combined in three dimensions of node, time, and parameter after being obtained from different physical measurement channels and being spatio-temporally preprocessed and uniformly normalized.

[0122] Preferably, the soil moisture estimation model comprises an input layer, a spatial feature integration layer, a time-dependent cycle layer, an attention fusion layer, and a regression output layer; the input layer is connected in parallel with the spatial feature integration layer and the time-dependent cycle layer respectively, the outputs of the spatial feature integration layer and the time-dependent cycle layer are jointly connected to the attention fusion layer, and the output of the attention fusion layer is independently connected to the regression output layer; the input layer is used to receive the preprocessed multi-source features, and the multi-source features are respectively split in the spatial dimension and the time dimension to form parallel input streams; the spatial feature integration layer is composed of multiple gradient boosting decision tree sub-models in parallel, and the spatial feature integration layer is used to generate a node-scale spatial feature vector by using the spatial dimension data in the parallel input streams; the time-dependent cycle layer is composed of a bidirectional long short-term memory network; the time-dependent cycle layer is used to receive the time series data in the parallel input streams, and extract time-dependent features of continuous observations; the attention fusion layer is used to input the spatial feature vector and the time-dependent features after aligning them according to node indexes, and dynamically allocate weights of the two to generate a fusion feature vector; and the regression output layer is used to perform full connection operation on the fusion feature vector, and output the soil moisture estimation result of the target area.

[0123] Further, the soil moisture estimation model in the embodiment realizes efficient collaborative modeling of multi-source spatio-temporal data through parallel branch design of spatial feature integration and time-dependent representation, which is creative. On the one hand, the spatial feature integration layer is composed of several parallel gradient boosting decision tree sub-models, and the depth, leaf node number, and learning rate of each sub-model and other hyperparameters can be pre-set according to soil types and sensor network density; on the other hand, the time-dependent cycle layer adopts a bidirectional long short-term memory network, and the number of layers, the number of hidden units, and the dropout rate of the network are all optimized for the typical change rhythm of the soil moisture time series. The parallel-fusion architecture breaks through the limitation of traditional single models on single spatio-temporal dimension, and significantly improves the prediction accuracy and generalization ability.

[0124] In the embodiment, the main parameters of each model layer are as follows: the input layer maps the normalized node-time-parameter tensor to the spatial input stream and the time input stream, and the mapping dimension is consistent with the number of feature channels; in the spatial feature integration layer, the maximum depth of each decision tree is generally set to 6-10 layers, the number of leaf nodes is set to 32-64, and the learning rate is set in the range of 0.05-0.2; the time-dependent cycle layer includes 2 layers of bidirectional LSTM, and the number of hidden units in each layer is generally 64-128, and the dropout rate is set to 0.1-0.3; the attention fusion layer adopts a single-head or multi-head attention mechanism, and the number of attention heads is usually 4-8, and the dimension of the fusion vector is consistent with that of the input vector; and the regression output layer is a layer of full connection, which maps the fusion features to a single soil moisture prediction value.

[0125] In this embodiment, the training process of the model includes: first, dividing the multi-season, multi-plot multi-source features and the corresponding measured soil moisture into a training set, a validation set and a test set, the ratio can be set to 70:15:15; using mean square error (MSE) as the loss function, combined with L2 regularization to prevent overfitting; using adaptive moment estimation (Adam) optimizer during training, the learning rate can be initially set to 1e-3 and gradually reduced according to the decay strategy when the validation loss stagnates; set early stopping mechanism, when the validation loss does not decrease continuously for several rounds, the training can be terminated and the optimal model parameters are retained; finally, evaluate the model performance on the test set, and further integrate the multiple training results through model ensemble technology to enhance the robustness and reliability of the model under different seasons and different soil conditions.

[0126] Corresponding to the above method, as shown in Figure 2 The embodiment also provides an analysis system for automatically measuring soil moisture based on data processing, which comprises:

[0127] A node deployment unit is configured to deploy nodes of soil-environment multi-parameter sensors with unique identifiers, and to complete self-identification and topology construction of the nodes through a self-organizing network protocol.

[0128] A data acquisition unit is configured to acquire volume moisture content data, temperature data, dielectric constant data and meteorological data synchronously within a preset sampling period through the nodes, and to generate original multi-modal data streams.

[0129] A quality control unit is configured to perform real-time data quality control on the original multi-modal data streams, and to obtain quality control data; the data quality control includes missing value detection, drift correction and outlier removal.

[0130] A data preprocessing unit is configured to perform spatio-temporal heterogeneity preprocessing on the quality control data, and to obtain preprocessed multi-source features.

[0131] An ensemble learning unit is configured to input the multi-source features into a soil moisture estimation model based on ensemble learning, and to generate soil moisture estimation results of a target area.

[0132] The beneficial effects of the present application are as follows:

[0133] (1) The present application realizes automatic discovery and topology construction of nodes by deploying multi-parameter sensor nodes with unique identifiers in the field and combining a self-organizing network protocol, eliminates the cumbersome process of manual inspection and network configuration, and improves the deployment efficiency and reliability of the monitoring network.

[0134] (2) The application uses real-time data quality control process to fill in missing values, correct drift and remove outliers for the collected multi-modal data, effectively eliminates measurement errors caused by sensor drift, environmental interference and communication packet loss, and ensures the integrity, consistency and high reliability of the data used for downstream analysis.

[0135] (3) The application unifies time series and spatial information of different scales and different parameters into structured tensor input through spatio-temporal heterogeneity preprocessing and multi-source feature construction, so that the double-branch fusion model based on ensemble learning can simultaneously mine spatial distribution rules and time dynamic characteristics, greatly improving the precision and generalization ability of soil moisture estimation.

[0136] (4) The application links the estimation results with the crop water threshold, combines the edge-cloud collaborative decision-making and real-time feedback mechanism, not only realizes the prediction-driven precision irrigation, but also relies on the Bayesian online update to adaptively train the model, ensuring the continuous reliability and long-term stable operation of the model under different seasons, different soils and sensor aging conditions.

[0137] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0138] In this paper, specific examples are used to illustrate the principles and implementation methods of the application. The above description of the embodiments is only used to help understand the method and its core idea; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for automatically measuring soil moisture based on data processing, characterized in that: include: Deploy nodes of soil-environment multi-parameter sensors with unique identifiers, and complete self-identification and topology construction of the nodes through a self-organizing network protocol; Using the nodes to synchronously collect volumetric water content data, temperature data, dielectric constant data, and meteorological data within a preset sampling period to generate an original multimodal data stream; Performing real-time data quality control on the original multimodal data stream to obtain quality-controlled data; the data quality control includes missing value detection, drift correction, and outlier removal; performing spatiotemporal heterogeneity preprocessing on the quality control data to obtain preprocessed multi-source features; The multi-source features are input into a soil moisture estimation model based on ensemble learning to generate a soil moisture estimation result for a target area.

2. The method for automatically measuring soil moisture based on data processing according to claim 1, characterized in that: Also includes: Comparing the soil moisture estimation result with the preset crop water requirement threshold, generating an irrigation decision signal, and pushing it to the irrigation execution terminal through the edge-cloud collaborative mechanism; The real-time soil moisture data transmitted back after irrigation is recorded, and the soil moisture estimation model is adaptively trained online based on the Bayesian update algorithm and the real-time soil moisture data to achieve self-evolution of the soil moisture estimation model.

3. The method for automatically measuring soil moisture based on data processing according to claim 1, characterized in that: Deploy nodes of soil-environment multi-parameter sensors with unique identifiers, and complete self-identification and topology construction of the nodes through a self-organizing network protocol, including: During the deployment phase, a unique identifier based on EUI-64 format is written to each node, and the unique identifier is encrypted and stored in the non-volatile memory of the node; After the node is powered on, it periodically sends a broadcast message carrying the unique identifier within a preset handshake window and synchronously monitors the broadcast messages of neighboring nodes; Generate a neighbor list using the unique identifiers of the monitored neighboring nodes and the corresponding received signal strength indicator RSSI; Determine the parent node based on the neighbor list according to a preset parent node selection algorithm, generate a local routing table, and realize node self-identification; The neighbor list and the local routing table are reported to the edge gateway through the self-organizing network protocol, and the edge gateway aggregates and forms the network topology structure of the target monitoring area, and sends synchronization confirmation messages to all nodes to complete the topology construction.

4. The method for automatically measuring soil moisture based on data processing according to claim 3, characterized in that: Performing real-time data quality control on the original multimodal data stream to obtain quality control data includes: The original multimodal data stream is time-series merged according to the unique identifier of the node within the preset monitoring time window to generate the data sequence to be detected; Perform sampling interval threshold determination on the data sequence to be detected, mark missing sampling points, and interpolate and fill in the missing data based on the historical mean of similar time periods-KNN regression fusion algorithm to obtain missing value completion data; Calculating a sliding baseline for the missing value completion data, detecting sensor drift using a point-by-point recursive slope significance test, and if drift is detected, using a recursive least squares method to estimate the drift amount in real time and perform dynamic correction, outputting drift correction data; The drift correction data is subjected to anomaly discrimination using the three-sigma principle combined with the isolation forest model, and data points exceeding the confidence interval are removed or replaced with the local weighted median to output the quality control data.

5. The method for automatically measuring soil moisture based on data processing according to claim 4, characterized in that: The calculation formula of the similar period historical mean-KNN regression fusion algorithm is: Among them, α i,t is the mixing coefficient, ω ij,t is the weight, the mixing coefficient and weight are adaptively determined by and Joint decision-making; is the missing soil moisture interpolation result of node i at time t; X i,s is the measured soil moisture of node i in the historical similar period s; is the set of historical moments that are in the same intraday phase as time t; For collection The number of elements; K is the number of neighbor nodes used for KNN regression; ω ij,t is the similarity weight of node j to node i at time t; d ij is the spatial distance between node i and node j; ΔT ij,t is the soil temperature difference between node i and node j at time t; d ,λ T are the attenuation scale parameters of spatial distance and temperature difference respectively; α i,t is the adaptive fusion coefficient of the historical mean and KNN estimation, with a value of 0-1; For node i in the set sample variance of soil moisture within the soil; is the sample variance of the K nearest neighbor soil moisture values ​​of node i at time t; is the sample variance of the K nearest neighbor soil moisture values ​​of node i at time t; is the sample variance of the K nearest neighbor soil moisture values ​​of node i at time t; is the sample variance of the K nearest neighbor soil moisture values ​​of node i at time t; is the sample variance of the K nearest neighbor soil moisture values ​​of node i at time t; β is the positive adjustment parameter that controls the sensitivity of the fusion coefficient.

6. The method for automatically measuring soil moisture based on data processing according to claim 4, characterized in that: Calculate the sliding baseline for the missing value completion data, and use a point-by-point recursive slope significance test to detect sensor drift. If drift is detected, use the recursive least squares method to estimate the drift amount in real time and perform dynamic correction, and output drift correction data, including: Set the length to L b The time sliding window is used to fill in the missing data X t Perform sliding median operation to obtain the baseline sequence in, For tL b +1 to t continuous L b A set of data points; Based on the recursive least squares formula k t =k t-1 +g t ((X t -B t )-(k t-1 )), Update the current slope k in real time t ; where λ is the forgetting factor; P t is the error covariance; P t-1 is the error covariance of the previous moment; B t is the median result, i.e. the sliding baseline at time t; k t is the current slope obtained by recursive estimation, that is, the drift trend coefficient; k t-1 is the estimated slope value of the previous moment; g t is the recursive least squares gain coefficient; k t Perform a one-point t-test, if It is determined that the node sensor has drifted and triggers correction; where t α is the preset significance threshold; After the drift is triggered, the recursive least squares method δ t =δ t-1 +g t ((X t -B t )-δ t-1 ) for the drift offset δ t Perform online estimation and update P synchronously t With g t ; Among them, δ t-1 is the drift offset of the previous moment; Generate drift correction data in real time and in The calibration mode is automatically released after M sampling cycles have passed. M is the number of sampling cycles that must continuously meet the steady-state condition to determine drift elimination.

7. The method for automatically measuring soil moisture based on data processing according to claim 1, characterized in that: The quality control data is preprocessed for temporal and spatial heterogeneity to obtain preprocessed multi-source features, including: Aligning the quality control data corresponding to the nodes of each soil-environment multi-parameter sensor with timestamps according to a unified reference sampling interval, and performing linear interpolation or nearest neighbor extension on missing time slices to obtain a continuous aligned data sequence; Based on the latitude and longitude information of the nodes, the geographic coordinates corresponding to the aligned data sequences are projected into a unified coordinate system, and the elevation and soil depth labels are marked in the same survey area to form a data block with spatial attributes; For the data blocks of the same time section, a weighted interpolation method is used to generate complete grid data on a regular grid of a set resolution; Different physical quantities in the grid data are normalized using interval scaling rules, and then organized according to node-time-parameter three-dimensional indexes to obtain preprocessed multi-source features.

8. The method for automatically measuring soil moisture based on data processing according to claim 7, characterized in that: Different physical quantities in the grid data are normalized using interval scaling rules, and then organized according to the node-time-parameter three-dimensional index to obtain preprocessed multi-source features, including: For each physical quantity in the raster data, extracting a minimum value and a maximum value within the data range of the most recent complete crop growing season to establish a scaling interval for the physical quantity; Converting the values ​​of the physical quantities of each of the grid data to a unified standard interval according to the scaling interval, retaining the original resolution and spatial position; Allocating a three-dimensional index key consisting of a node unique identifier, an observation timestamp, and a physical quantity code to each of the normalized raster data to form a key-value pairing record; The key-value pairing records of all physical quantities are aggregated in parallel according to the three-dimensional index key to generate a node-time-parameter three-dimensional feature tensor, and the node-time-parameter three-dimensional feature tensor is stored in a feature database to obtain the multi-source feature.

9. The method for automatically measuring soil moisture based on data processing according to claim 1, characterized in that: The soil moisture estimation model includes an input layer, a spatial feature integration layer, a time-dependent recurrent layer, an attention fusion layer and a regression output layer; the input layer establishes parallel connections with the spatial feature integration layer and the time-dependent recurrent layer respectively, the outputs of the spatial feature integration layer and the time-dependent recurrent layer are jointly connected to the attention fusion layer, and the output of the attention fusion layer is independently connected to the regression output layer; the input layer is used to receive the preprocessed multi-source features and split the multi-source features in the spatial dimension and the time dimension respectively to form a parallel input stream; the spatial feature integration layer is composed of multiple gradient boosting decision tree sub-models in parallel, and the spatial feature integration layer is used to use the spatial dimension data in the parallel input stream to generate a node-scale spatial representation vector; the time-dependent recurrent layer is composed of a bidirectional long short-term memory network; the time-dependent recurrent layer is used to receive the time series data in the parallel input stream and extract the time-dependent features of continuous observations; The attention fusion layer is used to align the spatial representation vector and the time-dependent feature according to the node index and dynamically assign weights to the two to generate a fused feature vector; the regression output layer is used to perform a full connection operation on the fused feature vector and output the soil moisture estimation result of the target area.

10. An automatic soil moisture measurement and analysis system based on data processing, characterized in that: include: A node deployment unit is used to deploy nodes of soil-environment multi-parameter sensors with unique identifiers and complete self-identification and topology construction of the nodes through a self-organizing network protocol; a data acquisition unit, configured to utilize the nodes to synchronously acquire volumetric water content data, temperature data, dielectric constant data, and meteorological data within a preset sampling period to generate an original multimodal data stream; A quality control unit, configured to perform real-time data quality control on the original multimodal data stream to obtain quality control data; the data quality control includes missing value detection, drift correction, and outlier removal; A data preprocessing unit, configured to perform spatiotemporal heterogeneity preprocessing on the quality control data to obtain preprocessed multi-source features; The integrated learning unit is used to input the multi-source features into a soil moisture estimation model based on integrated learning to generate a soil moisture estimation result of the target area.

Images