Big data preprocessing system for smart agriculture

Abstract

The invention relates to the technical field of big data preprocessing, in particular to a smart agriculture-oriented big data preprocessing system, which comprises a smart agriculture data fingerprint construction module for calculating a sliding window standard deviation and a frequency exceeding a preset crop canopy temperature threshold value for a data stream of each smart agriculture sensor, and jointly calculating a short-term correlation coefficient with the reading of the adjacent sensor. According to the method, the standard deviation and the over-limit frequency of the sliding window of the data stream of the sensor are extracted, and the short-term correlation coefficient between adjacent sensors is introduced to construct the multi-dimensional quality feature vector, so that the high-resolution recognition of the abnormal structure of the data is realized, and the accuracy of the quality evaluation of the original data is enhanced; and comparing the standard deviation and the short-term correlation coefficient with a preset threshold value, performing intersection screening, accurately defining a suspected noise data segment, dynamically calculating a predicted value of each time point in a local sequence context, realizing point-by-point repair of an abnormal segment, and retaining natural continuity of a time structure.

Description

Technical Field

[0001] The present invention relates to the field of big data preprocessing technology, and in particular to a big data preprocessing system for smart agriculture. Background Art

[0002] The field of big data preprocessing technology is a key basic link in data engineering and information processing. It mainly studies how to clean, regularize, transform and structure the original data after data collection is completed to meet the quality requirements of downstream analysis, modeling and decision support.

[0003] Existing technologies generally rely on field-level processing of static raw data, with cleaning, regularization, and format conversion as their primary goals. However, these methods lack the ability to identify the fluctuation characteristics of time series data and the dynamic consistency across nodes. They often use global rules or mean-based substitution methods to repair missing or outliers, which can easily lead to the breakdown of key trends or residual noise. For example, in multi-sensor collaborative measurement, reading delays or abnormal fluctuations between adjacent points may not be captured, causing the data input source to deviate from the actual scenario. Existing methods often rely on a fixed frequency for data sampling, ignoring the structural complexity and rate of change of the data itself. This results in redundant storage in static areas and sampling omissions in dynamically fluctuating areas. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of the present invention is to solve the shortcomings of the existing technology and propose a big data preprocessing system for smart agriculture.

[0005] In order to achieve the above objectives, the present invention adopts the following technical solutions: A big data preprocessing system for smart agriculture includes: The smart agriculture data fingerprint construction module calculates the sliding window standard deviation and the frequency of exceeding the preset crop canopy temperature threshold for each smart agriculture sensor data stream, and jointly calculates the short-term correlation coefficient with the adjacent sensor readings to generate a data quality fingerprint vector; A data stream noise calibration module compares the sliding window standard deviation and the short-term correlation coefficient with a preset threshold based on the data quality fingerprint vector, obtains suspected noise data segments, generates predicted values, and replaces the original values in the suspected noise data segments with the predicted values to establish a calibration data record; A data information entropy measurement module, based on the calibration data records, discretizes the soil moisture data values into intervals within a sliding time window, counts the frequency of occurrence of data points in each interval, calculates a data stream uncertainty measurement value, compares the data stream uncertainty measurement value with a preset sampling rate level threshold, selects a matching frequency level, and generates a dynamic sampling frequency identifier; The adaptive data storage module extracts data points from the calibration data record at the frequency specified by the dynamic sampling frequency identifier to form a subset of data to be archived, configures a timestamp, a sensor ID and a link to the data quality fingerprint vector for each data point in the subset of data to be archived, and establishes an optimized storage data index.

[0006] Preferably, the steps of obtaining the data quality fingerprint vector are: Based on the crop canopy temperature data streams from the main sensor and adjacent sensors, an equal-length sliding time window is set. The main sensor temperature sequence and the adjacent sensor temperature sequence within each time window are intercepted. The two sets of temperature values are aligned and matched at the corresponding time points, and the mean and standard deviation are calculated respectively to obtain the mean sequence and standard deviation sequence within each sliding window. Calculate the short-term correlation coefficient based on the mean sequence and standard deviation sequence in each sliding window; According to the short-term correlation coefficient, the response consistency degree of the crop canopy temperature data between the main sensor and the adjacent sensors in the current sliding time window is determined, and the short-term correlation coefficient is used as a data quality fingerprint vector.

[0007] Preferably, the step of acquiring the suspected noise data segment is: Based on the data quality fingerprint vector, extract the sliding window standard deviation and short-term correlation coefficient corresponding to the current sliding time window, call the sliding window standard deviation threshold and the short-term correlation coefficient threshold, and establish a corresponding parameter combination to be determined; According to the parameter combination to be determined, the sliding window standard deviation value is compared with the corresponding sliding window standard deviation threshold one by one, and all time windows whose sliding window standard deviation value exceeds the sliding window standard deviation threshold are marked; then, the short-term correlation coefficient is compared with the corresponding short-term correlation coefficient threshold one by one, and all time windows whose short-term correlation coefficient value does not reach the short-term correlation coefficient threshold are marked, and the intersection of the two marking results is obtained to generate an abnormal data time window; Based on the abnormal data time window, the original data stream record corresponding to the abnormal data time window is traced back, and the corresponding data segment is intercepted from the original data record with the start and end timestamps of the abnormal data time window as the boundary, and marked as a suspected noise data segment to generate a suspected noise data segment.

[0008] Preferably, the steps of obtaining the calibration data record are: Based on the start and end timestamps of the suspected noise data segment, a plurality of consecutive valid crop canopy temperature values of the noise segment in the preceding sliding time window are selected forward, and a plurality of consecutive valid crop canopy temperature values of the noise segment in the following sliding time window are selected backward, and a sequence of preceding valid data points and a sequence of following valid data points are extracted respectively to form a time-temperature reference sequence; Calculating a predicted value at a time point in each noise data segment based on the time-temperature reference sequence; Based on the predicted value at each time point, the original values of crop canopy temperature in the suspected noise data segments are replaced point by point in turn. All replaced data segments are marked as calibrated to generate calibration data records.

[0009] Preferably, the step of obtaining the data stream uncertainty metric value is: Based on the soil moisture data stream in the calibration data record, the soil moisture is divided into intervals within a fixed-length sliding time window, each soil moisture is mapped to its corresponding discrete interval, and all interval sequences are arranged in chronological order. The discrete interval transition path formed by every two consecutive data points is counted to obtain a discrete interval state transition frequency matrix; A data stream uncertainty metric value is calculated based on the discrete interval state transition frequency matrix.

[0010] Preferably, the steps of obtaining the dynamic sampling frequency identifier are: Based on the data stream uncertainty metric, extract each sampling rate level threshold from a predefined sampling rate level threshold set, and perform difference calculations between the data stream uncertainty metric and each sampling rate level threshold in turn, record each difference, and generate a set of numerical differences between the data stream uncertainty metric and the sampling rate level threshold; According to the set of numerical differences between the data stream uncertainty metric and the sampling rate level threshold, the absolute value of each difference is determined one by one, and the difference with the smallest value is selected from all the absolute values. The sampling rate level threshold corresponding to the smallest difference is traced back as the sampling rate level threshold closest to the data stream uncertainty metric, and a matching sampling rate level is generated; Based on the matched sampling rate level, according to the one-to-one mapping relationship between the predefined sampling rate level and the dynamic sampling frequency identifier, the dynamic sampling frequency identifier corresponding to the sampling rate level is determined, and the dynamic sampling frequency identifier is used as the basis for adjusting the sampling frequency within the current sliding time window to generate a dynamic sampling frequency identifier.

[0011] Preferably, the steps of obtaining the subset of data to be archived are: Based on the timestamp corresponding to each data point in the calibration data record, extract the time interval specified by the current dynamic sampling frequency identifier one by one, starting from the start timestamp of the calibration data record, and gradually moving backward according to the time interval to determine the target timestamp of each extraction point in sequence to form a target timestamp sequence; According to the target timestamp sequence, the data points closest to the target timestamp in the calibration data record are matched one by one, the absolute value of the difference between the target timestamp and the timestamp of the data point in the calibration data record is calculated, and the data point with the smallest absolute value of the difference under each target timestamp is selected as the extracted data point to generate a set of matched data points; Based on the matched data point set, all extracted data points are arranged in chronological order according to the target timestamp sequence, and all soil moisture data values corresponding to the data points are completely extracted from the calibration data records to form a data subset to be archived.

[0012] Preferably, the steps of obtaining the optimized storage data index are: Based on each data point in the subset of data to be archived, extract the original timestamp of each data point one by one, and normalize the original timestamp format of each data point one by one according to a unified timestamp format standard to generate a standardized data point timestamp set; According to the standardized data point timestamp set, the smart agricultural sensor corresponding to each data point in the subset of data to be archived is matched one by one, and the unique identification code of each sensor is obtained by querying the device registration information library of the smart agricultural sensor one by one, and each unique identification code is assigned to the corresponding data point to generate a sensor ID set for each data point; Based on the sensor ID set, the data quality fingerprint vectors associated with each sensor ID are retrieved one by one, the unique access link of the data quality fingerprint vector is extracted, and the data quality fingerprint vectors are associated and bound with the standardized data point timestamps of the corresponding data points one by one to form an optimized storage data index.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are: The present invention extracts the standard deviation and frequency of overshoots from a sliding window of sensor data streams and introduces the short-term correlation coefficient between adjacent sensors to construct a multidimensional quality feature vector, achieving high-resolution identification of data anomaly structures and enhancing the accuracy of raw data quality assessment. The standard deviation and short-term correlation coefficient are compared with preset thresholds and then subjected to intersection screening to accurately define suspected noise data segments. The predicted value for each time point is dynamically calculated within the local sequence context, achieving point-by-point repair of anomaly segments while preserving the natural continuity of the temporal structure. By performing interval discretization, frequency calculation, and transition pattern analysis on the corrected soil moisture data, a refined uncertainty index is generated based on the joint entropy measure of data distribution and transition behavior, shifting the sampling strategy's responsiveness from static level switching to a mapping expression of fluid complexity. Time-series extraction and matching are used instead of fixed-point sampling in sampling frequency control to ensure that data point selection is more consistent with actual dynamic distribution patterns. The timestamps, sensor IDs, and fingerprint links corresponding to the sampled data points are configured, and cross-parameter reference relationships are established at the data structure level, providing a highly readable foundation for subsequent index extraction and dynamic analysis. This achieves continuous gains from improving data integrity to optimizing storage structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] See also Figure 1 The present invention provides a technical solution: a big data preprocessing system for smart agriculture, comprising: The smart agriculture data fingerprint construction module calculates the sliding window standard deviation and the frequency of exceeding the preset crop canopy temperature threshold for each smart agriculture sensor data stream, and jointly calculates the short-term correlation coefficient with the adjacent sensor readings to generate a data quality fingerprint vector; The data stream noise calibration module compares the sliding window standard deviation and short-term correlation coefficient with the preset threshold based on the data quality fingerprint vector, obtains the suspected noise data segment, generates the predicted value, and replaces the original value in the suspected noise data segment with the predicted value to establish the calibration data record; The data information entropy measurement module discretizes the soil moisture data values into intervals within the sliding time window based on the calibration data records, counts the frequency of occurrence of data points in each interval, calculates the data stream uncertainty measurement value, compares the data stream uncertainty measurement value with the preset sampling rate level threshold, selects the matching frequency level, and generates a dynamic sampling frequency identifier; The adaptive data storage module extracts data points from the calibration data records at the frequency specified by the dynamic sampling frequency identifier to form a subset of data to be archived. It configures a timestamp, sensor ID, and a link to the data quality fingerprint vector for each data point in the subset of data to be archived, and establishes an optimized storage data index.

[0017] The steps to obtain the data quality fingerprint vector are: Based on the crop canopy temperature data streams from the main sensor and adjacent sensors, an equal-length sliding time window is set. The main sensor temperature sequence and the adjacent sensor temperature sequence within each time window are intercepted. The two sets of temperature values are aligned and matched at the corresponding time points, and the mean and standard deviation are calculated respectively to obtain the mean sequence and standard deviation sequence within each sliding window. According to the mean sequence and standard deviation sequence in each sliding window, the short-term correlation coefficient is calculated. The calculation formula is: ; in, is the short-term correlation coefficient between the temperature readings of the main sensor and the adjacent sensors in the current sliding window, The main sensor is The crop canopy temperature value at a time point, is the mean value of the temperature series of the main sensor in the current sliding window, For adjacent sensors The crop canopy temperature value at a time point, is the mean of the temperature series of adjacent sensors in the current sliding window, , is the dynamic event weight, where and are the standard deviations of the temperature series of the main sensor and the adjacent sensor in the current sliding window, is the total number of time points in the sliding time window, is a positive adjustment factor that controls the sensitivity of the weight function to temperature offset; According to the short-term correlation coefficient, the response consistency of the crop canopy temperature data between the main sensor and the adjacent sensors in the current sliding time window is determined, and the short-term correlation coefficient is used as the data quality fingerprint vector.

[0018] Specifically, from the crop canopy temperature data stream of the main sensor and the adjacent sensor, a fixed length of 10 minutes of equal-length sliding time window is first set, and the sliding is performed in steps of 1 minute. The window length is determined based on the analysis of historical crop canopy temperature data, which can effectively capture the typical temperature fluctuation cycle caused by agricultural activities such as irrigation and light changes or environmental factors, while avoiding excessively high-frequency computing overhead. In each time window, the system synchronously intercepts the temperature reading sequence of the main sensor and the adjacent sensor, and aligns them through the timestamps of each data point. If a timestamp is missing in a sensor sequence, the reading of the corresponding time point of the other sensor will be ignored during the calculation, thereby ensuring that the two sequences have the same number of data points in the window and the time points correspond one to one. For example, for a time window from 10:00 am to 10:09 am The temperature sequence obtained by the main sensor A is [25.1, 25.2, 25.3, 25.2, 25.4, 25.5, 25.3, 25.6, 25.7, 25.5]°C, and the temperature sequence obtained by the adjacent sensor B is [25.0, 25.1, 25.2, 25.1, 25.3, 25.4, 25.2, 25.5, 25.6, 25.4]°C. Next, the arithmetic mean and sample standard deviation of these two aligned temperature sequences are calculated independently. For sensor A, the average temperature is 25.38°C and the standard deviation is 0.193°C. For sensor B, the average temperature is 25.28°C and the standard deviation is 0.199°C. As the time window continues to slide, the system will repeat this process to generate a set of corresponding means and standard deviations for each window, and obtain the mean sequence and standard deviation sequence within each sliding window.

[0019] formula: The benefit of the formula is that by introducing dynamic event weights , which can amplify the influence of data points that deviate from the mean value within the window. In smart agriculture scenarios, when two adjacent sensors experience drastic temperature fluctuations at the same time, it usually indicates a real physical environment change, such as the start of an irrigation system or strong light changes, rather than random noise from a single sensor. Through dynamic event weighting, Assigning higher weight to such synchronous "events" makes More sensitive to the synchronous response behavior of sensors, allowing for more accurate differentiation between real environmental changes and sensor failures or noise; The acquisition steps are as follows: periodically collect data through temperature sensors (e.g., platinum resistance temperature sensors with the model of PT100) deployed in the crop canopy area, with the acquisition frequency set to once per minute, and obtain the data of the main sensor at the first The crop canopy temperature value at a time point. For example, at time point , the collected temperature was 25.1℃.

[0020] The acquisition step is to calculate the arithmetic mean of all 10 temperature data points in the current 10-minute sliding time window based on the main sensor temperature sequence obtained in the previous step. For example, if the temperature sequence in the window is [25.1, 25.3, 25.8, 25.6, 25.2, 25.1, 25.3, 25.8, 25.6, 25.2]℃, then its mean is , the mean value of the temperature series of the main sensor in the current sliding window is 25.4℃.

[0021] The acquisition steps are as follows: use a sensor with the same model and acquisition frequency as the main sensor, place it near the main sensor (for example, 50 cm apart), and synchronously acquire the adjacent sensor’s The crop canopy temperature value at a time point. For example, at time point ,The temperature collected from the adjacent sensor is 25.2℃.

[0022] The steps to obtain are as follows: The calculation method is the same as that of the temperature series of adjacent sensors in the same sliding time window. For example, if the temperature series in the window is [25.2, 25.4, 25.9, 25.7, 25.3, 25.2, 25.4, 25.9, 25.7, 25.3]℃, then its mean is , the average value of the temperature series of adjacent sensors in the current sliding window is 25.5℃.

[0023] and The acquisition step is to calculate the sample standard deviation of the temperature sequence of the main sensor and the adjacent sensor in the current sliding window. For the above main sensor temperature sequence, its standard deviation is For the above adjacent sensor temperature series, the standard deviation .

[0024] The acquisition steps are as follows: This parameter is the total number of time points in the sliding time window, which is determined by the window length and sampling frequency. The window length is set to 10 minutes and the sampling frequency is 1 time per minute, so .

[0025] The steps for obtaining are as follows: this positive adjustment factor is used to control the sensitivity of the weight function to temperature offset. It is determined by optimizing and analyzing a one-month historical data set containing known real environmental events (such as constant switching) and sensor noise. The specific method is to use a grid search to traverse the range of [0.5, 5.0] with a step size of 0.1. Value, for each group Value calculation for all historical data windows Values and evaluate normal time periods and event time periods The discrimination of the value distribution is selected so that the difference between the two distributions is maximized (for example, the statistic of the Kolmogorov-Smirnov test is maximized). After testing the tomato greenhouse dataset, it was determined that Time zone works best.

[0026] The steps to obtain the dynamic event weight are as follows: It is calculated that it will be two sensors at the time point The normalized combined deviation is mapped to the interval [1, 2) through the hyperbolic tangent function.

[0027] Calculation process: In a sliding time window ( ), the temperature series of the main sensor and adjacent sensors are collected, and the relevant statistical parameters have been calculated: ℃, ℃, ℃, ℃, .

[0028] Take the third time point in the window as an example for calculation, ℃, ℃.

[0029] First calculate the dynamic event weight of the point : ; ; Repeat this process for all 10 points in the window to get the weight sequence .

[0030] Next, calculate The numerator of : For example, after calculating all 10 points, the sum of the weighted covariances is 0.536.

[0031] Then, calculate the two square roots in the denominator, first the square root of the weighted sum of the variances of the primary sensor : For example, after calculating all 10 points, the weighted variance sum of the main sensor is 0.537, and its square root is .

[0032] Then calculate the square root of the weighted sum of variances of adjacent sensors : For example, after calculating all 10 points, the weighted sum of variances of adjacent sensors is 0.537, and its square root is .

[0033] Finally, calculate the short-term correlation coefficient : ; The results show that within the current sliding time window, the short-term correlation coefficient between the crop canopy temperature readings of the main sensor and the adjacent sensor is 0.9976, which is very close to 1, indicating that the readings of the two sensors show extremely high consistency and synchronization, especially after considering the sharp temperature fluctuations. This strong correlation indicates that the data stream reflects real environmental changes and has a high degree of credibility.

[0034] According to the calculated short-term correlation coefficient , combined with other data quality metrics within the same sliding window, to jointly determine the consistency of crop canopy temperature data responses between the main sensor and adjacent sensors. Specifically, rather than using only the short-term correlation coefficient, a multi-dimensional vector is constructed to quantify data quality. This vector consists of three core indicators. The first indicator is the previously calculated temperature standard deviation of the main sensor in the current window. , which directly reflects the fluctuation amplitude of the data stream. For example, the value is 0.292℃. The second indicator is the frequency of exceeding the preset crop canopy temperature threshold. This threshold is set according to the agronomic knowledge of the crop type and growth stage. For example, for tomatoes in the growth period, the upper temperature threshold of physiological stress is set to 32℃. The system will traverse all temperature readings of the main sensor in the window, count the number of data points greater than 32℃, and then divide it by the total number of data points in the window. , get a frequency value. If no reading in the window exceeds 32℃, the frequency is 0. The third indicator is the short-term correlation coefficient just calculated. , for example, the value is 0.9976. Finally, the three metrics, i.e., sliding window standard deviation, frequency exceeding threshold, and short-term correlation coefficient, are combined in a predetermined order [0.292, 0, 0.9976] to generate a data quality fingerprint vector.

[0035] The steps for obtaining the suspected noise data segment are: Based on the data quality fingerprint vector, the sliding window standard deviation and short-term correlation coefficient corresponding to the current sliding time window are extracted, the sliding window standard deviation threshold and the short-term correlation coefficient threshold are retrieved, and the corresponding parameter combination to be determined is established; According to the parameter combination to be determined, the sliding window standard deviation value is compared with the corresponding sliding window standard deviation threshold one by one, and all time windows whose sliding window standard deviation value exceeds the sliding window standard deviation threshold are marked; then the short-term correlation coefficient is compared with the corresponding short-term correlation coefficient threshold in turn, and all time windows whose short-term correlation coefficient value does not reach the short-term correlation coefficient threshold are marked. The intersection of the two marking results is calculated to generate the abnormal data time window; Based on the abnormal data time window, the original data stream record corresponding to the abnormal data time window is traced back, and the corresponding data segment is intercepted from the original data record with the start and end timestamps of the abnormal data time window as the boundary. It is marked as a suspected noise data segment to generate a suspected noise data segment.

[0036] Specifically, based on the data quality fingerprint vector, the system extracts the first element from each received fingerprint vector, such as [0.45, 0.0, 0.55], as the current sliding window standard deviation, that is, 0.45, and extracts the third element as the short-term correlation coefficient, that is, 0.55. Subsequently, the system retrieves the corresponding sliding window standard deviation threshold and short-term correlation coefficient threshold from a pre-built threshold configuration library, where the sliding window standard deviation threshold is dynamic, and its setting process is as follows: first, collect historical temperature data of the target crop under similar growth environment for at least one complete growth cycle, and divide the data into 24 hours a day, calculate the temperature standard deviation of all 10-minute windows in each hour, and form 24 groups of standard deviation data sets, and then calculate the mean and standard deviation of each data set. Finally, the standard deviation threshold for a specific hour is set to its mean plus 1.5 times the standard deviation. For example, for 10 a.m. to At 11:00, if the mean of the historical standard deviation is 0.2°C and the standard deviation is 0.1°C, the threshold for this period is 0.2 + 1.5 * 0.1 = 0.35°C. The short-term correlation coefficient threshold is set based on a statistical analysis of historical data sets containing known sensor faults (such as signal drift and poor contact). By calculating the short-term correlation coefficient values for normal and faulty periods and plotting their probability density distribution, the intersection point that maximizes the distinction between the two distributions is selected as the threshold. For example, if the analysis finds that 95% of normal data has a coefficient value above 0.7, while 90% of faulty data has a coefficient value below 0.7, the short-term correlation coefficient threshold is fixed at 0.7. After obtaining the standard deviation threshold (for example, 0.35°C) and correlation coefficient threshold (0.7) corresponding to the current window, the system pairs the current value with the threshold to establish the corresponding parameter combination to be determined {(0.45, 0.35), (0.55, 0.7)}.

[0037] According to the parameter combination to be determined, the system starts a double marking and intersection processing flow. First, the system establishes two temporary Boolean mark lists with the same length as the number of time windows currently being processed, and the initial values are both false. Then, the system performs the first traversal. For each parameter combination to be determined in each time window, for example, {(0.45, 0.35), (0.55, 0.7)}, only the sliding window standard deviation is judged, and the standard deviation value 0.45 is compared with the corresponding standard deviation threshold 0.35. Since 0.45 is greater than 0.35, the standard deviation of the window is judged to be abnormal, and the system marks the corresponding position of the window as true in the first mark list. If the standard deviation of another window is 0.2, which is less than the threshold 0.35, its corresponding position remains false, completing the traversal of all windows. After the traversal, the first list records all time windows in which the standard deviation exceeds the standard. Then, the system performs a second traversal, this time judging the short-term correlation coefficient and comparing the coefficient value of 0.55 with the corresponding threshold of 0.7. Since 0.55 does not reach 0.7, the correlation of the window is judged to be abnormal. The system marks the position corresponding to the window as true in the second mark list. If the correlation coefficient of another window is 0.9, which is higher than the threshold of 0.7, its corresponding position remains false. After the second traversal, the second list records all time windows in which the correlation does not meet the standard. Finally, the system performs a bitwise logical AND operation on the two Boolean mark lists. Only windows with true values at the same position in both lists will be identified in the final result. The intersection of the two marking results is calculated to generate an abnormal data time window.

[0038] Based on the abnormal data time window, the system first integrates the identifiers of these windows (for example, the sequence number or starting timestamp of the time window). If there are continuous or partially overlapping abnormal data time windows, for example, window A covers 10:00 to 10:09, and window B covers 10:08 to 10:17, the system will merge them into a continuous abnormal period, i.e. 10:00 to 10:17, to form a set of non-overlapping abnormal time intervals. Then, the system uses the start and end timestamps of these determined abnormal time intervals as query boundaries, and directly performs precise searches in the database or log file that stores the original sensor readings, and traces back the original data stream records corresponding to the identifiers. For each In an abnormal time interval, such as 10:00 to 10:17, the system will extract all data points recorded by the main sensor during this period, including timestamps and temperature values, to form a raw data slice. The system then marks the entire data slice by adding a status field to the metadata of each data point and setting its value to "suspected noise" or moving these data points to a temporary memory structure dedicated to storing data to be processed and attaching an identifier of the source interval, thereby logically isolating them from normal data. After completing the data extraction and marking of all abnormal time intervals, all marked data segments are collected together to generate a suspected noise data segment.

[0039] The steps to obtain the calibration data record are: Based on the start and end timestamps of the suspected noise data segment, multiple consecutive valid crop canopy temperature values of the noise segment in the preceding sliding time window are selected forward, and multiple consecutive valid crop canopy temperature values of the noise segment in the subsequent sliding time window are selected backward. The preceding and subsequent valid data point sequences are extracted respectively to form a time-temperature reference sequence. According to the time-temperature reference sequence, the predicted value of each time point in the noise data segment is calculated using the following formula: ; in, For time point The predicted value of is the last valid time point in the preceding sliding time window, For time point The corresponding crop canopy temperature value, is the first valid time point in the subsequent sliding time window, For time point The corresponding crop canopy temperature value, is the temperature change slope obtained by linear fitting the terminal temperature point in the previous sliding time window, is the temperature change slope obtained by linear fitting of the front temperature point in the subsequent sliding time window, is the target time point to be interpolated in the noise data segment; Based on the predicted value at each time point, the original values of crop canopy temperature in the suspected noise data segments are replaced point by point in turn. All replaced data segments are marked as calibrated to generate calibration data records.

[0040] Specifically, based on the start and end timestamps of the suspected noise data segment, for example, a noise segment from timestamp 1678846200 (corresponding to 10:10:00) to 1678846500 (corresponding to 10:15:00), the system first traces back from the start timestamp 1678846200 of this noise segment, retrieves the adjacent data points that are not marked as "suspected noise" in the original data record, and selects 5 such data points in succession. This number is determined based on experience. It can ensure the stability of linear fitting without introducing interference from historical data that is too far away. For example, five valid values of crop canopy temperature from 10:05:00 to 10:09:00 are selected. Similarly, the system searches backward from the end timestamp 1678846500 of the noise segment and selects 5 adjacent and consecutive valid data points. For example, from 10: The system then extracts the five valid values of crop canopy temperature from 16:00 to 10:20:00. The system then extracts the two sets of data points completely, including their respective timestamps and temperature values, to form a sequence of valid preceding data points, such as [(10:05:00, 26.2°C), (10:06:00, 26.3°C), (10:07:00, 26.4°C), (10:08:00, 26.4°C] ), (10:09:00, 26.5℃)], and subsequent valid data point sequences, such as [(10:16:00, 27.0℃), (10:17:00, 27.1℃), (10:18:00, 27.2℃), (10:19:00, 27.1℃), (10:20:00, 27.3℃)], and finally merge these two sequences to form a time-temperature reference sequence.

[0041] formula: The benefit of the formula is that it proposes a bidirectional trend-weighted interpolation method instead of a simple linear interpolation. The first part of the formula uses the trend before the noise segment (given by the slope represents) to predict a value, and the second part uses the trend after the noise segment (represented by the slope Represents) to reversely infer a value, the final predicted value is the weighted average of these two values, with weight and According to the point to be interpolated The distance from the two endpoints is dynamically adjusted, which allows the interpolation result to smoothly connect the dynamic changes of the previous and next data streams. This is especially suitable for environmental data such as temperature, which has inertia and trends, and avoids abrupt inflection points at data breakpoints. The acquisition step is to extract the timestamp of the last data point of the previous valid data point sequence from the time-temperature reference sequence formed in the previous step. This timestamp represents the last known valid time point before the start of the noise segment. For example, if the last point of the previous sequence is (10:09:00, 26.5℃), then The value is 10:09:00, which is expressed as 1678846140 in Unix timestamp with the second level.

[0042] The steps for obtaining , find and extract the corresponding crop canopy temperature value in the time-temperature reference sequence. For example, the temperature value at the time point 10:09:00 is 26.5℃, so .

[0043] The acquisition step is to extract the timestamp of the first data point of the subsequent valid data point sequence from the time-temperature reference sequence. This timestamp represents the first known valid time point after the end of the noise period. For example, if the first point of the subsequent sequence is (10:16:00, 27.0℃), then The value is 10:16:00, which is expressed as 1678846560 in Unix timestamp with the second level.

[0044] The steps for obtaining , find and extract the corresponding crop canopy temperature value in the time-temperature reference sequence. For example, the temperature value at the time point 10:16:00 is 27.0℃, so .

[0045] The steps to obtain are to perform a simple linear regression analysis using the previous valid data point sequence (5 points) to calculate the slope of the temperature change over time, where time is the independent variable and temperature is the dependent variable. The calculation formula is: ,in , is the timestamp, For example, for the sequence [(10:05, 26.2), (10:06, 26.3), (10:07, 26.4), (10:08, 26.4), (10:09, 26.5)], we get , indicating that the temperature rises by approximately 0.0015°C per second.

[0046] The steps to obtain are: The same method is used to perform linear regression analysis on the subsequent valid data point sequence (5 points) to calculate its temperature change slope. For example, for the sequence [(10:16, 27.0), (10:17, 27.1), (10:18, 27.2), (10:19, 27.1), (10:20, 27.3)], the calculation is , indicating that the temperature rises by approximately 0.0018°C per second.

[0047] The acquisition step is to traverse each time point in the suspected noise data segment and use its timestamp as the target time point to be interpolated. For example, if the first point in the noise segment needs to be interpolated and its timestamp is 10:10:00, then .

[0048] Calculation process: Taking the above parameters as an example, calculate the time point 10:10:00 in the noise data segment ( ) .

[0049] Known parameters: (10:09:00); ℃; (10:16:00); ℃; °C / s; °C / s; (10:10:00); First, calculate the time difference and weight factor: Second; Second; Second; Weight 1: ; Weight 2: ; Next, two trend extrapolation values are calculated: Forward extrapolation value: ; Backward extrapolation value: ; Finally, perform a weighted average: ; ; The results show that at the time point 10:10:00, the predicted crop canopy temperature value obtained by bidirectional trend weighted interpolation is 26.556℃. This value will be used as the calibration value to replace the original noise value at this time point. By repeating this calculation for all time points in the noise segment, a whole section of smooth and trend-reasonable calibration data can be obtained.

[0050] Based on the predicted value at each time point, for example, the predicted value calculated at 10:10:00 is 26.556℃, the system starts a replacement and marking process. The process first locks the original data point in the suspected noise data segment that completely matches the timestamp of the current predicted value. For example, the system finds an entry with a timestamp of 10:10:00 in the original data record, and its original value may be an abnormal 3.5℃. The system then performs a direct overwrite operation, updating the temperature value in the entry from 3.5 to 26.556. At the same time, in order to track the traces of data processing, the system will modify a status identifier in the metadata of the data point, updating it from "suspected noise" to "interpolation calibration". This process is performed on each time point in the suspected noise data segment in turn. For example, the predicted value at 10:11:00 is replaced with its original value, and then the predicted value at 10:12:00 is replaced, until the last point in the data segment (10:15:00) is also replaced and marked. When all the original values in the entire suspected noise data segment are replaced point-by-point with their corresponding predicted values, the system will uniformly mark this entire segment of data (from 10:10:00 to 10:15:00) as calibrated, for example, by adding a "calibration completed" flag in the metadata of the data segment. Finally, these processed data segments are merged with the original valid data stream to generate a calibration data record.

[0051] The steps to obtain the uncertainty measure of data stream are: Based on the soil moisture data stream in the calibration data record, the soil moisture is divided into intervals within a fixed-length sliding time window. Each soil moisture condition is mapped to its corresponding discrete interval. All interval sequences are arranged in chronological order. The discrete interval transition path formed by every two consecutive data points is counted to obtain the discrete interval state transition frequency matrix. According to the discrete interval state transition frequency matrix, the uncertainty measure of the data stream is calculated. The calculation formula is: ; in, is the uncertainty measure of the data stream, Indicates that the time window starts from the The discrete interval is transferred to the The frequency of data points in a discrete interval, Indicates that the time window falls into the The total frequency of data points in discrete intervals, represents the total number of soil moisture data points within the sliding time window, is the total number of discrete intervals of soil moisture.

[0052] Specifically, based on the soil moisture data stream in the calibration data record, a fixed-length sliding time window of 30 minutes is used, and it slides in steps of 5 minutes. The selection of this window length is based on the analysis of the duration of the impact of typical irrigation events, which can effectively capture the significant changes in the soil moisture state. Within this time window, the system first discretizes the soil moisture (measured in percentage of volumetric water content) into intervals. This division standard is set according to the water requirement characteristics of specific crops (for example, tomatoes) at different growth stages, and is not an arbitrary division. For example, the soil moisture is divided into four discrete intervals: interval 1 (dryness stress) is 0% to 15%, interval 2 (suitable) is 15% to 30%, interval 3 (moist) is 30% to 45%, and interval 4 (saturation) is greater than 45%. Then, the system traverses each soil moisture condition in the window. Data points are compared with the above interval boundaries, so that each data point is mapped to the discrete interval number to which it belongs. For example, a soil moisture data point with a value of 28% is mapped to interval 2, and a data point with a value of 12% is mapped to interval 1. In this way, the original continuous numerical sequence is converted into a discrete interval sequence arranged in chronological order, such as [2, 2, 3, 3, 2, 1, 1]. Finally, the system initializes a 4x4 all-zero matrix as the transfer frequency matrix, and traverses the discrete interval sequence to count the discrete interval transfer paths formed by every two consecutive data points. If there is a transfer from interval 2 to interval 3 in the sequence, the element value of the 2nd row and 3rd column in the matrix is increased by 1. If there is a hold from interval 3 to interval 3, the element value of the 3rd row and 3rd column is increased by 1. After the traversal is completed, the discrete interval state transfer frequency matrix is obtained.

[0053] formula: The benefit of the formula is that it uses the form of transfer entropy to measure the uncertainty of the data stream, which goes beyond the limitation of traditional Shannon entropy that only measures the complexity of state distribution. It focuses on the "transition" behavior between states, that is, the dynamic process of changing from one soil moisture range to another. By calculating the conditional uncertainty of the next state given the current state, it can more accurately capture the inherent dynamic pattern of the data stream. A data stream with drastic but regular changes (such as periodic irrigation) has a low transfer entropy, while a disordered, randomly fluctuating noisy data stream has a high transfer entropy. The sampling rate can be adjusted based on the "predictability" of data changes rather than just the "volatility"; The acquisition steps are as follows: This parameter is the total number of discrete intervals of soil moisture. Its value is directly determined by the interval division strategy of soil moisture in the previous step. This division is based on the agronomic characteristics of crops and divides the soil moisture conditions into several levels with clear physiological significance. For example, according to the water demand of tomato growth, four states are set: dry stress, suitable, moist, and saturated. Therefore, the total number of discrete intervals is is 4.

[0054] The acquisition steps are as follows: This parameter is the total number of soil moisture data points contained in the current fixed-length sliding time window, which is determined by the window length and data acquisition frequency. In this embodiment, the sliding time window length is set to 30 minutes, and the soil moisture data acquisition frequency in the calibration data record is 1 data point per minute. Therefore, the total number of data points in each window is .

[0055] The acquisition steps are as follows: This parameter represents the number of The discrete interval is transferred to the The frequency of the discrete interval is directly derived from the discrete interval state transition frequency matrix generated in the previous step. Rank The element value of the column is For example, within the aforementioned 30-minute window, a total of 5 shifts from interval 2 (suitable) to interval 3 (humid) were observed. .

[0056] The acquisition steps are: This parameter indicates that within the sliding time window, all The total frequency of the data points in the discrete interval is calculated by traversing the current window by The data points are converted into a discrete interval sequence, and the number of occurrences of each interval number is counted directly. For example, among 30 data points, 18 data points are mapped to interval 2 (suitable), then .

[0057] Calculation process: The calculation is performed within a 30-minute sliding window, where , From the previous step, we get the discrete interval state transition frequency matrix and the total frequency of data points in each interval .

[0058] Setting a specific scenario, we get the following frequencies: , , , .

[0059] and a transfer frequency matrix : .

[0060] , .

[0061] , .

[0062] The total number of transfers is .

[0063] Calculate each non-zero transition term : For transfer (1, 2): ; For transfer (2, 2): ; For transfer (2, 3): ; For transfer (3, 2): ; This sums all the terms: ; Finally, the uncertainty measure of the data stream is calculated : ; The results show that the uncertainty measure of the data stream within the current sliding window is 0.427, which is a relatively low value. It means that the changes in soil moisture conditions are highly regular and predictable. For example, the data mainly transfers between a few states in a fixed pattern, rather than jumping randomly and disorderly.

[0064] The steps to obtain the dynamic sampling frequency identifier are: Based on the data stream uncertainty metric, each sampling rate level threshold is extracted from a predefined sampling rate level threshold set, and the data stream uncertainty metric is sequentially subjected to difference calculations with each sampling rate level threshold, each difference is recorded, and a set of numerical differences between the data stream uncertainty metric and the sampling rate level threshold is generated; According to the set of numerical differences between the data stream uncertainty metric and the sampling rate level threshold, the absolute value of each difference is determined one by one, and the difference with the smallest value is selected from all the absolute values. The sampling rate level threshold corresponding to the smallest difference is traced back as the sampling rate level threshold closest to the data stream uncertainty metric, and a matching sampling rate level is generated; Based on the matching sampling rate level, according to the one-to-one mapping relationship between the predefined sampling rate level and the dynamic sampling frequency identifier, the dynamic sampling frequency identifier corresponding to the sampling rate level is determined, and the dynamic sampling frequency identifier is used as the basis for adjusting the sampling frequency within the current sliding time window to generate a dynamic sampling frequency identifier.

[0065] Specifically, based on the data stream uncertainty metric value, such as 0.427 calculated in the previous step, the system first reads from a predefined sampling rate level threshold set stored in the configuration file. The set is obtained by clustering analysis of a large amount of historical data. Specifically, soil moisture data under different weather conditions and different irrigation strategies are collected, and the data stream uncertainty metric value of each 30-minute window is calculated. Then, the K-means clustering algorithm is used to divide these metrics into 4 clusters, and the centroid of each cluster is used as the sampling rate level threshold. For example, the threshold set obtained is {level 1: 0.2, level 2: 0.5, etc. Level 3: 1.0, Level 4: 1.8}, then the system extracts each sampling rate level threshold in the set one by one, and subtracts the current data stream uncertainty measure value 0.427 from them respectively. The calculation process is: 0.427-0.2=0.227, 0.427-0.5=-0.073, 0.427-1.0=-0.573, 0.427-1.8=-1.373. The system records these calculated differences and generates a set of numerical differences between the data stream uncertainty measure values and the sampling rate level thresholds, namely {0.227, -0.073, -0.573, -1.373}.

[0066] According to the set of numerical differences between the uncertainty measure of the data stream and the sampling rate level threshold, that is, {0.227, -0.073, -0.573, -1.373}, the system enters a nearest neighbor matching process. The process first calculates the absolute value of each difference in the set one by one to obtain a new set. The calculation process is: |0.227|=0.227, |-0.073|=0.073, |-0.573|=0.573, |-1.373|=1.373. The absolute value set generated is {0.227, 0.073, 0.573, 1.373}. Next, the system performs a linear search in this absolute value set. To find and determine the element with the smallest value, by comparison, it can be found that 0.073 is the minimum value in this set. After finding the minimum absolute value, the system immediately traces back to the original difference set and finds the original difference -0.073 corresponding to 0.073. Then, the system further traces which sampling rate level threshold the difference is calculated by. By consulting the calculation record, it can be determined that -0.073 is obtained by subtracting the data stream uncertainty measure value 0.427 from the sampling rate level threshold value 0.5. Therefore, the system determines 0.5 as the sampling rate level threshold closest to the current data stream uncertainty measure value, and based on this threshold, generates a matching sampling rate level, namely "Level 2".

[0067] Based on the matching sampling rate level, i.e., "Level 2," the system queries a pre-established one-to-one mapping table stored in system memory. This table explicitly defines the correspondence between each sampling rate level and a specific dynamic sampling frequency identifier. This mapping relationship is established based on a trade-off analysis between resource consumption and information value. That is, for each uncertainty level, simulation experiments are used to determine the minimum sampling frequency that can both basically capture the data variation characteristics at that level and maximize the savings in storage and transmission resources. For example, the mapping table is defined as follows: Level 1 corresponds to "low frequency" (sampling every 30 minutes), Level 2 corresponds to "medium frequency" (sampling every 10 minutes), Level 3 corresponds to "high frequency" (sampling every 5 minutes), and Level 4 corresponds to "real-time" (sampling every 1 minute). Based on the currently matched "Level 2," the system searches the table and finds its corresponding dynamic sampling frequency identifier, "medium frequency." This identifier is internally encoded as a specific sampling interval, i.e., 600 seconds. The system then uses this dynamic sampling frequency identifier ("medium frequency" or its corresponding 600 seconds) as a direct instruction for subsequent data extraction operations within the current sliding time window to generate a dynamic sampling frequency identifier.

[0068] The steps to obtain the subset of data to be archived are: Based on the timestamp corresponding to each data point in the calibration data record, the time interval specified by the current dynamic sampling frequency identifier is extracted one by one, starting from the start timestamp of the calibration data record, and moving backward step by step according to the time interval to determine the target timestamp of each extraction point in turn to form a target timestamp sequence; According to the target timestamp sequence, the data points closest to the target timestamp in the calibration data record are matched one by one, the absolute value of the difference between the target timestamp and the timestamp of the data point in the calibration data record is calculated, and the data point with the smallest absolute value of the difference under each target timestamp is selected as the extracted data point to generate a set of matched data points; Based on the matched data point set, all extracted data points are arranged in the time order of the target timestamp sequence, and all soil moisture data values corresponding to the data points are completely extracted from the calibration data records to form a subset of data to be archived.

[0069] Specifically, based on the timestamp corresponding to each data point in the calibration data record, the system first retrieves the dynamic sampling frequency identifier generated by the previous step from the memory. For example, the identifier obtained is "medium frequency", and the time interval corresponding to its internal code is 600 seconds. Then, the system determines the time range of the calibration data record to be processed and obtains its starting timestamp. For example, the starting timestamp of a data block is 1678845600 (corresponding to 10:00:00 am). The system uses this starting timestamp as the target timestamp of the first extraction point, and uses this as a benchmark, according to the 600-second time interval, Through cumulative operations, a series of target timestamps are generated backward step by step. The specific calculation is: the first target timestamp is 1678845600, the second is 1678845600+600=1678846200 (10:10:00), and the third is 1678846200+600=1678846800 (10:20:00). This process will continue until the generated target timestamp exceeds the end timestamp of the calibration data record to be processed. All target timestamps generated within this range are collected to form a target timestamp sequence.

[0070] According to the target timestamp sequence, for example [1678845600, 1678846200, 1678846800], the system traverses each target timestamp in the sequence to find the best matching point in the calibration data record. For the first target timestamp 1678845600, the system performs a small range search in the calibration data record with this timestamp as the center, and calculates the absolute value of the difference between it and the timestamps of the adjacent data points. Since the calibration data record is at intervals of 1 minute, there is a data point with a timestamp of 1678845600, and its absolute value of the difference is 0, so this data point is directly selected. For the second target timestamp 1678846200, the system repeats the same matching process and selects the data point with a timestamp of 1678846200 in the calibration data record. If, during the matching process, the calibration data record is missing data at a certain target timestamp, for example, the target timestamp is 1678847100 (10:25:00), but the record only contains data points at 1678847040 (10:24:00) and 1678847160 (10:26:00), the system will calculate the absolute value of the difference, that is, |1678847100-1678847040|=60 seconds and |1678847100-1678847160|=60 seconds respectively. If the differences are the same, the system follows the preset rules and selects the data point with the earlier time, that is, the data point with the timestamp 1678847040 as the extracted data point, and gathers all the data points selected under the target timestamp to generate a set of matching data points.

[0071] Based on the set of matched data points, a sorting operation is first performed to ensure that all extracted data points in the set are arranged strictly in the order of their own timestamps. The sorting basis is consistent with the order of the target timestamp sequence. After the sorting is completed, the system traverses this ordered set of matching data points. For each data point in the set, the system accurately extracts its core business data, that is, the soil moisture data value, from the complete entry of the calibration data record. For example, for the first sorted data point, its timestamp is 1678845600, and the system extracts the core business data from the calibration data record. The soil moisture value corresponding to the timestamp read from the record is 25.5%. For the second data point, its timestamp is 1678846200, and the system reads its corresponding soil moisture value as 28.1%. The system adds these extracted soil moisture values to a new data list in turn. The order of the data in the list completely corresponds to the time order of the extracted data points. When the soil moisture data values of all matching data points are extracted and stored in the new list, this list, which only contains ordered soil moisture values extracted at a dynamic frequency, forms a subset of the data to be archived.

[0072] The steps to obtain the optimized storage data index are: Based on each data point in the subset of data to be archived, the original timestamp of each data point is extracted one by one, and the original timestamp format of each data point is normalized one by one according to the unified timestamp format standard to generate a standardized data point timestamp set; According to the standardized data point timestamp set, the smart agricultural sensor corresponding to each data point in the subset of data to be archived is matched one by one. The unique identification code of each sensor is obtained by querying the device registration information database of the smart agricultural sensor one by one. Each unique identification code is assigned to the corresponding data point to generate the sensor ID set for each data point; Based on the sensor ID set, the data quality fingerprint vector associated with each sensor ID is retrieved one by one, the unique access link of the data quality fingerprint vector is extracted, and it is associated and bound with the standardized data point timestamp of the corresponding data point one by one to form an optimized storage data index.

[0073] Specifically, for each data point in the subset of data to be archived, the system initiates a timestamp normalization process. This process iterates through the data points in the subset one by one and extracts the original timestamp embedded in each data point. These original timestamps exist in various formats, such as Unix timestamps (such as 1678845600), human-readable strings (such as "2023-03-15 10:00:00"), or other custom formats. The system uses a unified timestamp format standard, namely the ISO8601 extended format (YYYY-MM-DDTHH:MM:SS.sssZ). , each extracted original timestamp is parsed and converted. For example, 1678845600 will be converted to "2023-03-15T10:00:00.000Z", and "2023-03-1510:00:00" will be converted to "2023-03-15T10:00:00.000Z". The conversion process is completed through the built-in date and time processing library, which can automatically recognize multiple common formats and output standard formats. All standardized timestamps after conversion are collected into a new list to generate a standardized data point timestamp set.

[0074] Based on the standardized data point timestamp set, the system performs a sensor identification and assignment process, which processes the data points in the subset of data to be archived one by one. Since each data point has its own temporary identifier or internal name of the source sensor when it is collected, the system first extracts this identifier, such as "soil sensor-area A-01". Then, the system accesses a centrally managed smart agriculture sensor device registration information database stored in a relational database. The database records the detailed information of all sensors, including their internal names and unique global identification codes (for example, a UUID format string "f81d4fae-7dec-11d0-a765-00a 0c91e6bf6"), installation location, model, etc. The system uses the extracted internal name "Soil Sensor - Region A-01" as the query primary key and performs an exact match search in the device registration information database. After a successful query, the system obtains the corresponding unique identification code "f81d4fae-7dec-11d0-a765-00a0c91e6bf6". Finally, the system assigns this obtained unique identification code as a new attribute to the data point currently being processed. After repeating this operation for all data points in the subset, all data points obtain their unique sensor IDs. These IDs are collected to generate a sensor ID set for each data point.

[0075] Based on the sensor ID set, the system begins to build the final index structure. The process processes each data point in the subset of data to be archived one by one. First, the system extracts the sensor ID of the data point, such as "f81d4fae-7dec-11d0-a765-00a0c91e6bf6", and its corresponding standardized data point timestamp, such as "2023-03-15T10:10:00.000Z". Then, the system needs to find the data quality fingerprint vector corresponding to the acquisition time of the data point. These fingerprint vectors are generated in the preprocessing stage, associated with each sliding time window, and stored in a dedicated time series database. Its access key consists of the sensor ID and the start timestamp of the window. Based on the timestamp of the current data point, the system calculates the sliding time window to which it belongs (for example, The data point at 10:10:00 belongs to the window from 10:00:00 to 10:19:00). Then, a query request is constructed, using the sensor ID and the starting timestamp of the window to retrieve the corresponding data quality fingerprint vector from the database, for example, [0.45, 0.0, 0.55]. The system generates a unique access link or internal reference address for the fingerprint vector. Finally, the system associates this access link with the standardized timestamp of the data point to form an index record, for example, {"timestamp":"2023-03-15T10:10:00.000Z","fingerprint_ref":"fp_ref_f81d4fae_1678845600"}. After completing this operation for all data points, an optimized storage data index is formed.

Claims

1. A big data preprocessing system for smart agriculture, characterized by: The system comprises: The smart agriculture data fingerprint construction module calculates the sliding window standard deviation and the frequency of exceeding the preset crop canopy temperature threshold for each smart agriculture sensor data stream, and jointly calculates the short-term correlation coefficient with the adjacent sensor readings to generate a data quality fingerprint vector; A data stream noise calibration module compares the sliding window standard deviation and the short-term correlation coefficient with a preset threshold based on the data quality fingerprint vector, obtains suspected noise data segments, generates predicted values, and replaces the original values in the suspected noise data segments with the predicted values to establish a calibration data record; A data information entropy measurement module, based on the calibration data records, discretizes the soil moisture data values into intervals within a sliding time window, counts the frequency of occurrence of data points in each interval, calculates a data stream uncertainty measurement value, compares the data stream uncertainty measurement value with a preset sampling rate level threshold, selects a matching frequency level, and generates a dynamic sampling frequency identifier; The adaptive data storage module extracts data points from the calibration data record at the frequency specified by the dynamic sampling frequency identifier to form a subset of data to be archived, configures a timestamp, a sensor ID and a link to the data quality fingerprint vector for each data point in the subset of data to be archived, and establishes an optimized storage data index.

2. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the data quality fingerprint vector are: Based on the crop canopy temperature data streams from the main sensor and adjacent sensors, an equal-length sliding time window is set. The main sensor temperature sequence and the adjacent sensor temperature sequence within each time window are intercepted. The two sets of temperature values are aligned and matched at the corresponding time points, and the mean and standard deviation are calculated respectively to obtain the mean sequence and standard deviation sequence within each sliding window. Calculate the short-term correlation coefficient based on the mean sequence and standard deviation sequence in each sliding window; According to the short-term correlation coefficient, the response consistency degree of the crop canopy temperature data between the main sensor and the adjacent sensors in the current sliding time window is determined, and the short-term correlation coefficient is used as a data quality fingerprint vector.

3. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the suspected noise data segment are: Based on the data quality fingerprint vector, extract the sliding window standard deviation and short-term correlation coefficient corresponding to the current sliding time window, call the sliding window standard deviation threshold and the short-term correlation coefficient threshold, and establish a corresponding parameter combination to be determined; According to the parameter combination to be determined, the sliding window standard deviation value is compared with the corresponding sliding window standard deviation threshold value one by one, and all time windows whose sliding window standard deviation value exceeds the sliding window standard deviation threshold value are marked; Then, the short-term correlation coefficient is compared with the corresponding short-term correlation coefficient threshold in turn, and all time windows whose short-term correlation coefficient value does not reach the short-term correlation coefficient threshold are marked. The intersection of the two marking results is obtained to generate the abnormal data time window; Based on the abnormal data time window, the original data stream record corresponding to the abnormal data time window is traced back, and the corresponding data segment is intercepted from the original data record with the start and end timestamps of the abnormal data time window as the boundary, and marked as a suspected noise data segment to generate a suspected noise data segment.

4. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the calibration data record are: Based on the start and end timestamps of the suspected noise data segment, a plurality of consecutive valid crop canopy temperature values of the noise segment in the preceding sliding time window are selected forward, and a plurality of consecutive valid crop canopy temperature values of the noise segment in the following sliding time window are selected backward, and a sequence of preceding valid data points and a sequence of following valid data points are extracted respectively to form a time-temperature reference sequence; Calculating a predicted value at a time point in each noise data segment based on the time-temperature reference sequence; Based on the predicted value at each time point, the original values of crop canopy temperature in the suspected noise data segments are replaced point by point in turn. All replaced data segments are marked as calibrated to generate calibration data records.

5. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the data stream uncertainty measurement value are: Based on the soil moisture data stream in the calibration data record, the soil moisture is divided into intervals within a fixed-length sliding time window, each soil moisture is mapped to its corresponding discrete interval, and all interval sequences are arranged in chronological order. The discrete interval transition path formed by every two consecutive data points is counted to obtain a discrete interval state transition frequency matrix; A data stream uncertainty metric value is calculated based on the discrete interval state transition frequency matrix.

6. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the dynamic sampling frequency identifier are: Based on the data stream uncertainty metric, extract each sampling rate level threshold from a predefined sampling rate level threshold set, and perform difference calculations between the data stream uncertainty metric and each sampling rate level threshold in turn, record each difference, and generate a set of numerical differences between the data stream uncertainty metric and the sampling rate level threshold; According to the set of numerical differences between the data stream uncertainty metric and the sampling rate level threshold, the absolute value of each difference is determined one by one, and the difference with the smallest value is selected from all the absolute values. The sampling rate level threshold corresponding to the smallest difference is traced back as the sampling rate level threshold closest to the data stream uncertainty metric, and a matching sampling rate level is generated; Based on the matched sampling rate level, according to the one-to-one mapping relationship between the predefined sampling rate level and the dynamic sampling frequency identifier, the dynamic sampling frequency identifier corresponding to the sampling rate level is determined, and the dynamic sampling frequency identifier is used as the basis for adjusting the sampling frequency within the current sliding time window to generate a dynamic sampling frequency identifier.

7. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the subset of data to be archived are: Based on the timestamp corresponding to each data point in the calibration data record, extract the time interval specified by the current dynamic sampling frequency identifier one by one, starting from the start timestamp of the calibration data record, and gradually moving backward according to the time interval to determine the target timestamp of each extraction point in sequence to form a target timestamp sequence; According to the target timestamp sequence, the data points closest to the target timestamp in the calibration data record are matched one by one, the absolute value of the difference between the target timestamp and the timestamp of the data point in the calibration data record is calculated, and the data point with the smallest absolute value of the difference under each target timestamp is selected as the extracted data point to generate a set of matched data points; Based on the matched data point set, all extracted data points are arranged in chronological order according to the target timestamp sequence, and all soil moisture data values corresponding to the data points are completely extracted from the calibration data records to form a data subset to be archived.

8. The big data preprocessing system for smart agriculture according to claim 1, characterized in that: The steps for obtaining the optimized storage data index are: Based on each data point in the subset of data to be archived, extract the original timestamp of each data point one by one, and normalize the original timestamp format of each data point one by one according to a unified timestamp format standard to generate a standardized data point timestamp set; According to the standardized data point timestamp set, the smart agricultural sensor corresponding to each data point in the subset of data to be archived is matched one by one, and the unique identification code of each sensor is obtained by querying the device registration information library of the smart agricultural sensor one by one, and each unique identification code is assigned to the corresponding data point to generate a sensor ID set for each data point; Based on the sensor ID set, the data quality fingerprint vectors associated with each sensor ID are retrieved one by one, the unique access link of the data quality fingerprint vector is extracted, and the data quality fingerprint vectors are associated and bound with the standardized data point timestamps of the corresponding data points one by one to form an optimized storage data index.

Images