Agrometeorological monitoring data processing method and system
By collecting, identifying, and removing abnormal data and performing segmented compression, the problems of data quality and transmission efficiency in traditional agricultural meteorological monitoring have been solved, achieving efficient data processing and transmission.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-04-07
AI Technical Summary
In traditional agricultural meteorological monitoring data processing, data quality control struggles to identify complex anomalies, leading to data noise contamination, which affects decision-making accuracy. Furthermore, the high-frequency data collection results in low compression efficiency, high communication power consumption, and significant storage pressure.
Data from farmland areas was collected through an internet-based meteorological data platform. Quality control and radial characteristic data were determined, anomalies were identified and marked, and abnormal data were removed. KD-trees were used for nearest neighbor search and feature data point set partitioning. The data was then segmented and compressed to obtain a clean dataset, which was then transmitted via internet protocols.
While ensuring data quality, efficient compression of agricultural meteorological data was achieved, reducing communication power consumption and storage pressure, and improving the accuracy and efficiency of data processing.
Smart Images
Figure CN121808601A_ABST
Abstract
Description
Technical Field
[0001] This application relates to data processing, and more specifically, to a method and system for processing agricultural meteorological monitoring data. Background Technology
[0002] Agricultural meteorological monitoring is a crucial component of modern agricultural management. By collecting and processing meteorological data in real time, it provides critical support for irrigation decisions, disaster early warning, and yield forecasting. With the widespread application of IoT technology in farmland environments, massive amounts of meteorological data are being collected at high frequency. However, ensuring data quality and efficiently transmitting it to processing platforms has become a key challenge in improving the level of agricultural intelligence.
[0003] Traditional agricultural meteorological monitoring data quality control often relies on simple threshold judgment methods, which cannot effectively identify complex and hidden anomalies caused by sensor malfunctions, environmental interference, or abnormal weather. This "data noise" gets mixed into the operational system, directly affecting the accuracy of subsequent decision-making models. Meanwhile, farmland IoT devices are widely distributed and rely on wireless communication, resulting in massive amounts of data collected at high frequencies. Traditional compression algorithms have low compression ratios, while lossy compression with high compression ratios loses key fluctuation characteristics, leading to high communication power consumption and heavy storage pressure. Therefore, how to achieve efficient compression of agricultural meteorological data while ensuring data quality has become a challenge for the industry. Summary of the Invention
[0004] This application provides a method and system for processing agricultural meteorological monitoring data, which can achieve efficient compression of agricultural meteorological data while ensuring data quality.
[0005] In a first aspect, this application provides a method for processing agricultural meteorological monitoring data based on Internet big data, comprising the following steps: Meteorological datasets for the target farmland area were collected through an internet-based meteorological data platform. Determine multiple quality control feature data and radial feature data of the meteorological dataset; Anomalies are identified in all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probabilities. Then, all data in the meteorological dataset are marked as anomalies based on all comprehensive anomaly probabilities. Remove all data marked as abnormal from the meteorological dataset, divide the data space of the remaining data in the meteorological dataset, and then perform a nearest neighbor search on the divided data space to obtain the feature data point set of the remaining data at different state boundaries when performing data compression; Multiple data fluctuation features of the feature data point set are determined, and the compression error of the feature data point set is allocated according to all the data fluctuation features. Then, the feature data point set is segmented and compressed to obtain a clean dataset. The purified dataset was transmitted to the agricultural meteorological monitoring data processing platform via Internet protocol.
[0006] In some embodiments, determining multiple quality control feature data and radial feature data of the meteorological dataset specifically includes: Determine multiple abrupt change parameters, fluctuation parameters, and consistency parameters of the surface data in the meteorological dataset; The set of mutation parameters, fluctuation parameters, and consistency parameters is used as multiple quality control feature data of the meteorological dataset; Radial feature extraction is performed on the weather radar data in the meteorological dataset to obtain multiple radial feature data of the meteorological dataset.
[0007] In some embodiments, anomaly identification is performed on all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probability, specifically including: Identify multiple outliers in all quality control characteristic data and radial characteristic data; Based on all abnormal states, determine the comprehensive anomaly probability corresponding to all quality control feature data and radial feature data.
[0008] In some embodiments, marking all data in the meteorological dataset as anomalies based on all comprehensive anomaly probabilities specifically includes: Obtain the preset anomaly probability threshold; Compare all the combined anomaly probabilities with the anomaly probability threshold; If the overall anomaly probability is greater than or equal to the anomaly probability threshold, then the data corresponding to the overall anomaly probability is marked as an anomaly; If the overall anomaly probability is less than the anomaly probability threshold, then no marking is performed.
[0009] In some embodiments, partitioning the data space of the remaining data in the meteorological dataset specifically includes: Construct the data space for the remaining data in the meteorological dataset and determine the initial segmentation dimension; The data space is divided into two subspaces based on the initial segmentation dimension; Obtain the preset values of data points during segmentation; Repeat the above partitioning process for all subspaces until the number of data points in all subspaces is less than the preset value, thus obtaining the partitioned data space.
[0010] In some embodiments, performing a nearest neighbor search on the partitioned data space to obtain feature data points of the remaining data at different state boundaries during data compression specifically includes: Determine the nearest neighbor search dimension; Select one data point from each data point in the divided data space as the selected data; Determine multiple state analogy differences between selected data and data within the range of the nearest neighbor search dimension; Determine the state boundary feature points of the selected data within the nearest neighbor search dimension based on the differences in all state categories; Continue to determine the state boundary feature points of the remaining data points; The set of all state boundary feature points is used as the feature data point set of the remaining data at different state boundaries during data compression.
[0011] In some embodiments, determining multiple data fluctuation features of the feature data point set specifically includes: Determine multiple data segments in the set of feature data points; Determine the fluctuation range and stability corresponding to all data segments; Determine the extreme fluctuation probability and spatial continuity for all data segments; All fluctuation amplitudes, stability, extreme fluctuation probabilities, and spatial continuity are used as data fluctuation characteristics of the feature data point set.
[0012] Secondly, this application provides an agricultural meteorological monitoring data processing system, comprising: The data acquisition module is used to collect meteorological datasets of the target farmland area through an internet-based meteorological data platform. The processing module is used to determine multiple quality control feature data and radial feature data of the meteorological dataset; The processing module is also used to identify anomalies in all quality control feature data and radial feature data, obtain the corresponding comprehensive anomaly probability, and then mark all data in the meteorological dataset as anomalies based on all comprehensive anomaly probabilities. The processing module is also used to remove all data marked as abnormal from the meteorological dataset, divide the data space of the remaining data in the meteorological dataset, and then perform a nearest neighbor search on the divided data space to obtain the feature data point set of the remaining data at different state boundaries when performing data compression. The processing module is further configured to determine multiple data fluctuation characteristics of the feature data point set, allocate the compression error of the feature data point set according to all the data fluctuation characteristics, and then perform segmented compression on the feature data point set to obtain a clean dataset. The execution module is used to transmit the clean dataset to the agricultural meteorological monitoring data processing platform via Internet protocol.
[0013] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described method for processing agricultural meteorological monitoring data based on Internet big data.
[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for processing agricultural meteorological monitoring data based on Internet big data.
[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The agricultural meteorological monitoring data processing method based on Internet big data provided in this application first collects a meteorological dataset of a target farmland area through an Internet meteorological data platform; determines multiple quality control feature data and radial feature data of the meteorological dataset; performs anomaly identification on all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probability, and then marks all data in the meteorological dataset as anomalies based on all comprehensive anomaly probabilities; removes all data marked as anomalies from the meteorological dataset, divides the data space of the remaining data in the meteorological dataset, and then performs nearest neighbor search on the divided data space to obtain feature data point sets of the remaining data at different state boundaries when performing data compression; determines multiple data fluctuation characteristics of the feature data point set, allocates the compression error of the feature data point set according to all data fluctuation characteristics, and then performs segmented compression on the feature data point set to obtain a clean dataset; and transmits the clean dataset to the agricultural meteorological monitoring data processing platform via Internet protocol.
[0016] Therefore, in the agricultural meteorological monitoring data processing method based on Internet big data in this application, firstly, meteorological datasets of the target farmland area are collected through an Internet meteorological data platform; multiple quality control feature data and radial feature data of the meteorological dataset are determined; anomaly identification is performed on all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probability; then, all data in the meteorological dataset are marked as anomalies based on all comprehensive anomaly probabilities; wherein, the comprehensive anomaly probability is a continuous numerical index describing the overall degree of anomaly of a single data record; it is used for the final binary judgment decision from multi-dimensional feature anomaly state to normal or abnormal, which facilitates subsequent data cleaning and avoids misjudgment of a single feature; secondly, from the meteorological dataset... All data marked as abnormal are removed from the meteorological dataset. The data space of the remaining data is then divided, and a nearest neighbor search is performed on the divided data space to obtain the feature data point set of the remaining data at different state boundaries during data compression. Here, the state boundary refers to the critical region or data point in agricultural meteorological monitoring data where key transitions occur between different dynamic modes, such as stable, rising, falling, and drastic changes. Multiple data fluctuation characteristics of the feature data point set are determined, and the compression error of the feature data point set is allocated according to all data fluctuation characteristics. The feature data point set is then segmented and compressed to obtain a clean dataset. The clean dataset is then transmitted to the agricultural meteorological monitoring data processing platform via an internet protocol. This scheme can achieve efficient compression of agricultural meteorological data while ensuring data quality. Attached Figure Description
[0017] Figure 1 This is an exemplary flowchart of an agricultural meteorological monitoring data processing method based on Internet big data, according to some embodiments of this application; Figure 2 This is an exemplary flowchart illustrating the determination of quality control feature data and radial feature data according to some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating the determination of the overall anomaly probability according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an agricultural meteorological monitoring data processing system according to some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a computer device for implementing an agricultural meteorological monitoring data processing system according to some embodiments of this application. Detailed Implementation
[0018] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0019] refer to Figure 1 The figure is an exemplary flowchart of an agricultural meteorological monitoring data processing method based on Internet big data, according to some embodiments of this application. This method mainly includes the following steps: In step 101, meteorological datasets for the target farmland area are collected through an internet-based meteorological data platform.
[0020] In specific implementation, the collection of meteorological datasets for the target farmland area through the Internet meteorological data platform can be achieved in the following way: Based on the latitude and longitude coordinates of the farmland area, a data collection request is constructed. The meteorological data is collected by calling the service of the meteorological data platform of the China Meteorological Data Network through the Internet meteorological data platform interface, thereby obtaining a multi-source meteorological dataset including ground observation data and weather radar data. The returned formatted data is then converted into a structured data table through a data parser to obtain the meteorological dataset for the target farmland area. The meteorological dataset for the target farmland area refers to a standardized, structured collection that integrates multi-source meteorological observation data. This dataset includes at least time-series observation data from ground automatic weather stations, such as temperature, humidity, wind speed, and precipitation, and basic data from weather radar, such as reflectivity and radial velocity. All data are aligned and integrated according to a unified time and space reference to form basic data units that can be used for subsequent quality control and intelligent compression processing. Other methods can also be used in other embodiments, which are not limited here.
[0021] In step 102, multiple quality control feature data and radial feature data of the meteorological dataset are determined.
[0022] In some embodiments, reference Figure 2 As shown, this figure is an exemplary flowchart for determining quality control feature data and radial feature data in some embodiments of this application. In this embodiment, determining multiple quality control feature data and radial feature data of the meteorological dataset can be achieved by the following steps: First, in step 1021, multiple mutation parameters, fluctuation parameters, and consistency parameters of the ground data in the meteorological dataset are determined; Secondly, in step 1022, the set of mutation parameters, fluctuation parameters and consistency parameters is used as multiple quality control feature data of the meteorological dataset; Finally, in step 1023, radial feature extraction is performed on the weather radar data in the meteorological dataset to obtain multiple radial feature data of the meteorological dataset.
[0023] In specific implementation, the meteorological dataset of the target farmland area collected through the Internet meteorological data platform can be implemented in the following way: Time series analysis methods can be used to detect anomalies in the ground observation data. For example, the Z-Score algorithm can be used to calculate the abrupt change threshold of each data point in the ground observation data within adjacent time steps as the abrupt change parameter, and the standard deviation of each data point in the ground observation data within each sliding window can be calculated as the fluctuation parameter. Then, by comparing the correlation of data from adjacent meteorological stations in the same time period, the Pearson correlation coefficient can be calculated as the consistency parameter. Other methods can also be used in other embodiments, which are not limited here.
[0024] In specific implementation, radial feature extraction is performed on the weather radar data in the meteorological dataset to obtain multiple radial feature data of the meteorological dataset. This can be achieved in the following way: Radar feature extraction is performed on the weather radar data in the meteorological dataset based on radar digital signal processing technology: First, the number of effective echoes on each radial data point is calculated, and the number of distances with reflectivity greater than a threshold is counted. The threshold can be set as an empirical value based on historical statistics, and is set to 0.7 in this application. Next, the maximum radial echo distance is determined, and the location of the farthest effective echo point is identified. Then, the average intensity of the radial echo is calculated, and the reflectivity values of the effective echo points are arithmetically averaged. Finally, the correlation of radial reflectivity factors between adjacent volume scans is calculated, and the temporal continuity is analyzed using the Pearson correlation coefficient. The number of effective echoes, the average intensity of radial echoes, the correlation of radial reflectivity factors, and the temporal continuity on each radial data point are all used as radial feature data of the meteorological dataset. Other methods can also be used in other embodiments, which are not limited here.
[0025] It should be noted that the radial feature data in this application is a multi-dimensional feature set describing the spatial distribution characteristics and temporal stability of radar echoes, specifically including four core indicators: the number of radial echoes, the maximum distance of radial echoes, the average intensity of radial echoes, and the correlation of radial reflectivity factors. Different types of data anomalies will exhibit different distribution patterns in the radial feature space. For example, hardware failures usually manifest as abrupt changes in radial features, while ground clutter has specific spatial distribution characteristics. Therefore, by extracting these four dimensions of radial features, a complete data quality assessment index can be constructed.
[0026] In step 103, anomaly identification is performed on all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probability, and then all data in the meteorological dataset are marked as anomalies based on all comprehensive anomaly probabilities.
[0027] In some embodiments, reference Figure 3As shown, this figure is an exemplary flowchart for determining the overall anomaly probability in some embodiments of this application. In this embodiment, anomaly identification is performed on all quality control feature data and radial feature data to obtain the corresponding overall anomaly probability, which can be achieved by the following steps: First, in step 1031, multiple abnormal states of all quality control feature data and radial feature data are identified; Secondly, in step 1032, the comprehensive anomaly probability corresponding to all quality control feature data and radial feature data is determined based on all the abnormal states.
[0028] In specific implementation, determining multiple abnormal states of all quality control feature data and radial feature data can be achieved in the following way: A multi-dimensional anomaly identification model is constructed based on fuzzy logic theory. First, a corresponding fuzzy membership function is established for each type of feature parameter in the quality control feature data: For mutation parameters, an ascending half-trapezoidal membership function is used, with a membership degree of 1 when the rate of change exceeds 3.5 times the historical standard deviation; for fluctuation parameters, a descending half-trapezoidal membership function is used, with a membership degree of 1 when the data variance is consistently lower than the sensor accuracy threshold; for consistency parameters, a triangular membership function is used, with a membership degree of 1 when the correlation coefficient is lower than 0.65; for radial feature data, the output value of each feature parameter in the corresponding membership function is calculated in real time to obtain its abnormal state quantification index within the range of [0,1]; thus, the membership degree in the preset fuzzy membership function is used as the abnormal state of each corresponding feature parameter; other methods can also be used in other embodiments, which are not limited here.
[0029] It should be noted that traditional methods cannot effectively identify problems at the data acquisition level, such as sensor malfunctions and environmental interference. However, quality control characteristic data reflects the working status of the monitoring equipment itself and the reliability of the data acquisition link. Therefore, quality control characteristic data can ensure that the raw data collected is authentic and reliable. Secondly, traditional methods have difficulty distinguishing between abnormal weather echoes and non-meteorological interference. However, radial characteristic data reflects the actual atmospheric conditions and the reliability of radar detection performance. Therefore, radial characteristic data can ensure that the identified signals such as rainstorms and hail are real meteorological phenomena, rather than equipment malfunctions or false signals.
[0030] In specific implementation, determining the comprehensive anomaly probability corresponding to all quality control feature data and radial feature data based on all abnormal states can be achieved in the following way: using a weighted fusion algorithm for multi-feature collaborative analysis: establishing a weighted comprehensive identification equation through the anomaly membership degree of mutation parameters, fluctuation parameters, consistency parameters, number of radial echoes, maximum distance of radial echoes, and average intensity of radial echoes, and then calculating the comprehensive anomaly probability of each data point in real time through this weighted equation; wherein, the weight coefficients of the weighted comprehensive identification equation are determined by the analytic hierarchy process and satisfy the constraint that the sum of the total weight coefficients is 1; other methods can also be used in other embodiments, which are not limited here.
[0031] It should be noted that the comprehensive anomaly probability in this application is a continuous numerical index describing the overall degree of anomaly of a single data record; it is used for the final binary judgment decision from multi-dimensional feature anomaly state to normal or abnormal, which facilitates subsequent data cleaning and avoids misjudgment based on a single feature; among them, fuzzy logic describes the probability of a certain anomaly state occurring through the membership concept of values between 0 and 1, rather than simply yes or no, which is particularly suitable for describing common anomalies in agricultural meteorological data that are difficult to define with precise thresholds. Secondly, the use of an ascending half-trapezoidal membership function for abrupt change parameters can effectively capture sudden anomalies such as a sudden drop in temperature or a sudden change in wind speed; the use of a descending half-trapezoidal membership function for fluctuation parameters helps to identify persistent anomalies such as long-term data stagnation caused by sensor failure or data transmission interruption; and the use of a triangular membership function for consistency parameters makes it easier to discover spatial inconsistency anomalies that differ too much from neighboring sensor data.
[0032] In some embodiments, anomaly labeling of all data in the meteorological dataset based on all comprehensive anomaly probabilities can be achieved using the following steps: Obtain the preset anomaly probability threshold; Compare all the combined anomaly probabilities with the anomaly probability threshold; If the overall anomaly probability is greater than or equal to the anomaly probability threshold, then the data corresponding to the overall anomaly probability is marked as an anomaly; If the overall anomaly probability is less than the anomaly probability threshold, then no marking is performed.
[0033] In step 104, all data marked as abnormal are removed from the meteorological dataset, the data space of the remaining data in the meteorological dataset is divided, and then a nearest neighbor search is performed on the divided data space to obtain the feature data point set of the remaining data at different state boundaries when data compression is performed.
[0034] In some embodiments, partitioning the data space of the remaining data in the meteorological dataset can be achieved using the following steps: Construct the data space for the remaining data in the meteorological dataset and determine the initial segmentation dimension; The data space is divided into two subspaces based on the initial segmentation dimension; Obtain the preset values of data points during segmentation; Repeat the above partitioning process for all subspaces until the number of data points in all subspaces is less than the preset value, thus obtaining the partitioned data space.
[0035] In specific implementation, the data space of the remaining data in the meteorological dataset is constructed, and the initial segmentation dimension is determined in the following way: First, the meteorological elements of each data point, such as temperature, humidity, and light intensity, are all used as dimensions to construct a multi-dimensional feature space, and this space is used as the input to construct a KD tree; Second, the construction process starts from the root node: the dimension with the largest variance is selected as the initial segmentation dimension, and the median segmentation method is used in combination with the initial segmentation dimension to determine the segmentation node, dividing the data space represented by the current segmentation node into two subspaces and generating two new child nodes. Then, the above segmentation process is repeated recursively for the subspace represented by each newly generated child node, so that the KD tree continues to grow until the number of data points contained in all child nodes is less than a preset threshold, where the threshold is usually set to 50 data points; Finally, a highly structured KD tree space index is formed; In other embodiments, other methods can also be used, which are not limited here.
[0036] It's important to note that in traditional agricultural meteorological monitoring data processing, the time complexity of a global search in the multidimensional feature space is O(n²), which is extremely computationally expensive when dealing with massive datasets. KD-trees, through recursive spatial binary partitioning, organize high-dimensional data into a balanced tree structure, reducing the time complexity of nearest neighbor search to O(log n), thus enabling real-time data processing on resource-constrained edge devices. Furthermore, KD-trees consistently select the dimension with the highest variance for partitioning during construction, ensuring that each partition effectively distinguishes data points with different characteristics. In agricultural meteorological scenarios, this means the system automatically identifies the key element dimensions that best differentiate different meteorological states, thereby most effectively separating different data state categories in the feature space.
[0037] In some embodiments, performing a nearest neighbor search on the partitioned data space to obtain the feature data point set of the remaining data at different state boundaries during data compression can be achieved using the following steps: Determine the nearest neighbor search dimension; Select one data point from each data point in the divided data space as the selected data; Determine multiple state analogy differences between selected data and data within the range of the nearest neighbor search dimension; Determine the state boundary feature points of the selected data within the nearest neighbor search dimension based on the differences in all state categories; Continue to determine the state boundary feature points of the remaining data points; The set of all state boundary feature points is used as the feature data point set of the remaining data at different state boundaries during data compression.
[0038] In specific implementation, the nearest neighbor search dimension can be determined in the following way: the system determines the key dimensions of the nearest neighbor search based on feature importance analysis. By calculating the variance contribution rate of each meteorological element in historical data, the top k dimensions with the highest contribution rates, such as temperature, humidity, and light intensity, are selected as search dimensions to ensure that the search process focuses on the most discriminative feature space; in this application, k is set to 5; other methods can also be used in other embodiments, which are not limited here.
[0039] In specific implementation, determining the state analogy difference between selected data and data within the nearest neighbor search dimension range can be achieved in the following way: Perform a k-nearest neighbor search for each selected data point in the selected data, where k is set to 5 in this application, thereby calculating the state category difference between each selected data point and its k nearest neighbors; wherein, the difference calculation adopts a distance metric method based on probability distribution, obtaining a difference value within the range [0,1] by comparing the state distribution differences between a single data point and its neighbors in various meteorological element dimensions; other methods can also be used in other embodiments, which are not limited here.
[0040] In specific implementation, determining the state boundary feature points of selected data within the nearest neighbor search dimension based on the differences in all state categories can be achieved as follows: A difference threshold δ = 0.8 is set. When the maximum difference between a selected data point and its k nearest neighbors exceeds this threshold, the point is determined to be a state boundary feature point. The boundary type, such as a temperature abrupt change boundary or a humidity transition boundary, and the confidence score are recorded simultaneously. Other methods can be used in other embodiments, which are not limited here.
[0041] It should be noted that the state boundary in this application refers to the critical region or data point in agricultural meteorological monitoring data where key transitions occur between different dynamic modes, such as stable, rising, falling, and drastic changes. These boundary points are usually the turning points where data characteristics change significantly. The feature data point set refers to the set of key feature data points located on these state boundaries, selected from massive clean data through techniques such as nearest neighbor search. Identifying state boundaries and constructing feature data point sets provides highly refined input for subsequent data compression. By retaining a small number of key points that best represent the dynamic changes in the data, the amount of data that needs to be processed is significantly reduced, while ensuring that the key morphological features and trends of the data are preserved. Secondly, the spatial partitioning capability of KD trees is combined with state boundary identification. By calculating the state category difference between each data point and its neighbors through nearest neighbor search, key feature points located at the decision boundaries of different state categories can be accurately identified. At critical nodes of state transition in agricultural meteorological data, obvious boundary regions will form in the multidimensional feature space. The data characteristics of these boundary points are significantly different from those of their neighbors. By constructing an efficient spatial index structure, these key data points carrying important information can be quickly located, and state boundary identification ensures the preservation of key change features of the data.
[0042] In step 105, multiple data fluctuation features of the feature data point set are determined, and the compression error of the feature data point set is allocated according to all the data fluctuation features. Then, the feature data point set is segmented and compressed to obtain a clean dataset.
[0043] In some embodiments, determining multiple data fluctuation characteristics of the feature data point set can be achieved using the following steps: Determine multiple data segments in the set of feature data points; Determine the fluctuation range and stability corresponding to all data segments; Determine the extreme fluctuation probability and spatial continuity for all data segments; All fluctuation amplitudes, stability, extreme fluctuation probabilities, and spatial continuity are used as data fluctuation characteristics of the feature data point set.
[0044] In specific implementation, determining multiple data segments in the feature data point set can be achieved in the following way: First, the entire dataset is divided into multiple continuous data segments with adjacent feature data points as endpoints; determining the fluctuation amplitude and stability corresponding to all data segments can be achieved in the following way: Calculate the fluctuation amplitude of each data segment, i.e., the standard deviation between all data points in the segment and the segment mean; then analyze the stability of the data segments by calculating the average of the change rates of adjacent data points to assess trend stability; determining the extreme fluctuation probability and bias continuity corresponding to all data segments can be achieved in the following way: Use statistical analysis methods to calculate the kurtosis coefficient of the data segments to assess the extreme fluctuation probability, and calculate the correlation coefficient with adjacent data segments to assess spatial continuity; other methods can also be used in other embodiments, which are not limited here.
[0045] It should be noted that the data fluctuation characteristics in this application refer to a set of statistical indicators extracted from the feature data point set to quantify the dynamic behavior of the data, including four core dimensions: fluctuation amplitude, stability, probability of extreme fluctuations, and spatial continuity. These characteristics together constitute a complete mathematical description of the dynamic characteristics of the data segment, providing a decision-making basis for subsequent adaptive compression. Among them, the core role of the data fluctuation characteristics is to provide a precise quantitative basis for the dynamic allocation of compression errors. Different types of data fluctuation characteristics have significantly different impacts on reconstruction quality. Data segments with drastic fluctuations require stricter error control, while stable data segments can tolerate larger compression errors, thus establishing an adaptive correlation between data characteristics and compression parameters.
[0046] In some embodiments, allocating the compression error of the feature data point set according to all data fluctuation characteristics can be achieved by the following steps: Select a data segment from the set of feature data points as the selected data segment; The error value of the selected data segment during data compression is determined based on the fluctuation amplitude and stability of the selected data segment among all data fluctuation characteristics. Continue to determine the error value of the remaining data segments during data compression.
[0047] In specific implementation, determining the error value of a selected data segment during data compression based on the fluctuation amplitude and stability of the selected data segment among all data fluctuation characteristics can be achieved in the following way: dynamically calculate the compression error threshold based on the fluctuation characteristics of the data segment; for data segments with large fluctuation amplitudes, use a dynamic error allocation formula based on standard deviation scaling to allocate a smaller error threshold; for data segments with poor stability, introduce a weighted error correction formula based on extreme fluctuation probabilities in the error calculation to further tighten the error limit; finally, combine spatial continuity for context-aware adjustment, and appropriately relax the error limit for spatially isolated data segments to avoid over-compression, and bind the calculated error threshold to the data segment; other methods can also be used in other embodiments, which are not limited here.
[0048] It should be noted that the error value for data compression in this application is an adaptive threshold parameter dynamically calculated based on the local fluctuation characteristics of the data segment. It defines the maximum allowable reconstruction deviation of a single point of data during the compression process, so that data segments with drastic fluctuations are assigned smaller error values to retain detailed features, while data segments with stable fluctuations are assigned larger error values to improve compression efficiency. In particular, the error value during data compression can optimize the overall compression efficiency by accurately adapting to data segments with different characteristics, even when stable data segments are under-compressed or when details are lost in volatile data segments. Secondly, it overcomes the problem of poor adaptability of general compression algorithms in agricultural meteorological monitoring scenarios.
[0049] In some embodiments, segmenting and compressing the feature data point set to obtain a clean dataset can be achieved using the following steps: Select a data segment from the set of feature data points as the selected data segment; Determine the boundary region based on all feature data points of the selected data segment; All data points in the selected data segment that exceed the boundary region are treated as individual segmentation points; Compress the selected data segment and all corresponding segmentation points of the selected data segment; Continue compressing the remaining data segments to obtain a clean dataset.
[0050] In specific implementation, the boundary region can be determined based on all feature data points of the selected data segment in the following way: take the starting feature point of the selected data segment as the compression base point, initialize the sector boundary region based on the error threshold allocated to the segment; then traverse all data points in the segment, dynamically adjust the slope of the sector boundary, and when a data point that exceeds the current sector region is encountered, immediately mark the point as a segmentation point and open a new sector region; other methods can also be used in other embodiments, which are not limited here.
[0051] In specific implementation, the compression of the selected data segment and all corresponding segment points can be achieved in the following way: perform linear approximation on the data within all fan-shaped regions of the selected data segment and all corresponding segment points, retaining only the key point coordinates and slope parameters of the region boundaries; finally, integrate the segment points, slope parameters, and error information obtained after compressing all data segments to form a clean dataset with a high compression ratio; other methods can also be used in other embodiments, which are not limited here.
[0052] It should be noted that the "pure dataset" in this application refers to a standardized dataset that retains the key dynamic features and statistical characteristics of the original data after outlier removal, feature point screening, and adaptive segmented compression. The integrated processing avoids the cumulative effect of errors and overcomes the risk of feature loss in agricultural meteorological data scenarios using general compression algorithms. An adaptive compression mechanism based on fluctuation characteristics ensures the complete preservation of key agricultural meteorological event features. Segmented compression dynamically assigns different compression errors to data segments with varying volatility, avoiding the problems of insufficient compression in stable segments and undersampling in fluctuating segments inherent in traditional fixed error threshold methods. This achieves a higher compression ratio for stable data while ensuring no loss of waveform features of key meteorological events. After assigning error thresholds, a sector compression algorithm is used to independently compress each data segment, thus representing a large amount of original data with a small number of line segments and key parameters, achieving effective compression while preserving the overall trend and morphological characteristics of the data, resulting in better fitting of non-stationary meteorological data sequences.
[0053] In step 106, the purified dataset is transmitted to the agricultural meteorological monitoring data processing platform via Internet protocol.
[0054] In some embodiments, the transmission of the clean dataset to the agricultural meteorological monitoring data processing platform via Internet protocol can be achieved through the following steps: The clean dataset is packaged into a standardized data package; The Internet transmission protocol is dynamically selected based on the size of the clean dataset and the real-time requirements. The packaged clean dataset is transmitted to the agricultural meteorological monitoring data processing platform via the Internet transmission protocol.
[0055] In specific implementation, packaging the clean dataset into a standardized data packet can be achieved as follows: The dataset is structured and encapsulated using TLV (Type-Length-Value) encoding format. Version number, data source identifier, timestamp, and data type fields are set in the data packet header. Compressed feature point coordinates, slope parameters, and metadata information are stored sequentially in the data body. Finally, a CRC32 checksum is added to form the complete data packet. Dynamically selecting the internet transmission protocol based on the size of the clean dataset and real-time requirements can be achieved as follows: A protocol selection decision tree is established. When the data packet size is less than 1KB and real-time transmission is required, the MQTT protocol is selected. When the data packet size is between 1KB and 10MB, the HTTP / 2 protocol is selected. When transmitting batches of historical data larger than 10MB, FTP over [the-database] is enabled. The TLS protocol is used, and the transmission strategy is dynamically adjusted according to network quality. The packaged clean dataset can be transmitted to the agricultural meteorological monitoring data processing platform via the Internet transmission protocol in the following way: a layered transmission guarantee mechanism is adopted, a TLS 1.3 encrypted channel is established at the transport layer, a breakpoint resumption and retransmission mechanism is implemented at the application layer, and a transmission quality monitoring module is set up to monitor bandwidth utilization and packet loss rate in real time. When the packet loss rate exceeds 5%, the transmission path is automatically switched to ensure data integrity and timeliness. Other methods can also be used in other embodiments, which are not limited here.
[0056] In another aspect, in some embodiments, this application provides an agricultural meteorological monitoring data processing system, with reference to... Figure 4 The figure is a schematic diagram of the structure of an agricultural meteorological monitoring data processing system according to some embodiments of this application. The agricultural meteorological monitoring data processing system includes: an acquisition module 401, a processing module 402, and an execution module 403, which are described below: Acquisition module 401, in this application, is mainly used to acquire meteorological datasets of the target farmland area through an Internet meteorological data platform; Processing module 402, in this application, is used to determine multiple quality control feature data and radial feature data of the meteorological dataset; It should be noted that the processing module 402 in this application is also used to identify anomalies in all quality control feature data and radial feature data, obtain the corresponding comprehensive anomaly probability, and then mark all data in the meteorological dataset as anomalies based on all comprehensive anomaly probabilities. Additionally, it should be noted that the processing module 402 in this application is also used to remove all data marked as abnormal from the meteorological dataset, divide the data space of the remaining data in the meteorological dataset, and then perform a nearest neighbor search on the divided data space to obtain the feature data point set of the remaining data at different state boundaries when performing data compression. Additionally, it should be noted that the processing module 402 in this application is also used to determine multiple data fluctuation characteristics of the feature data point set, allocate the compression error of the feature data point set according to all the data fluctuation characteristics, and then perform segmented compression on the feature data point set to obtain a clean dataset. The execution module 403 in this application is mainly used to transmit the clean dataset to the agricultural meteorological monitoring data processing platform via Internet protocol.
[0057] In addition, this application also provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described method for processing agricultural meteorological monitoring data based on Internet big data.
[0058] In some embodiments, reference Figure 5 The figure is a schematic diagram of the structure of a computer device for implementing an agricultural meteorological monitoring data processing method based on Internet big data, according to some embodiments of this application. The agricultural meteorological monitoring data processing method based on Internet big data in the above embodiments can be implemented through... Figure 5 The computer device shown is used to implement this, and the computer device includes at least one processor 501, a communication bus 502, a memory 503, and at least one communication interface 504.
[0059] Processor 501 can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).
[0060] The communication bus 502 can be used to transmit information between the aforementioned components.
[0061] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CDROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 503 may exist independently and be connected to processor 501 via communication bus 502. Memory 503 may also be integrated with processor 501.
[0062] The memory 503 stores program code for executing the scheme of this application, and its execution is controlled by the processor 501. The processor 501 executes the program code stored in the memory 503. The program code may include one or more software modules. The method used in the above embodiments can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.
[0063] Communication interface 504 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0064] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single CPU) processor or a multi-core (multi CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).
[0065] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.
[0066] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for processing agricultural meteorological monitoring data based on Internet big data.
[0067] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0068] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for processing agricultural meteorological monitoring data based on Internet big data, characterized in that, The method includes the following steps: Meteorological datasets for the target farmland area were collected through an internet-based meteorological data platform. Determine multiple quality control feature data and radial feature data of the meteorological dataset; Anomalies are identified in all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probabilities. Then, all data in the meteorological dataset are marked as anomalies based on all comprehensive anomaly probabilities. Remove all data marked as abnormal from the meteorological dataset, divide the data space of the remaining data in the meteorological dataset, and then perform a nearest neighbor search on the divided data space to obtain the feature data point set of the remaining data at different state boundaries when performing data compression; Multiple data fluctuation features of the feature data point set are determined, and the compression error of the feature data point set is allocated according to all the data fluctuation features. Then, the feature data point set is segmented and compressed to obtain a clean dataset. The purified dataset was transmitted to the agricultural meteorological monitoring data processing platform via Internet protocol.
2. The method as described in claim 1, characterized in that, The determination of multiple quality control feature data and radial feature data of the meteorological dataset specifically includes: Determine multiple abrupt change parameters, fluctuation parameters, and consistency parameters of the surface data in the meteorological dataset; The set of mutation parameters, fluctuation parameters, and consistency parameters is used as multiple quality control feature data of the meteorological dataset; Radial feature extraction is performed on the weather radar data in the meteorological dataset to obtain multiple radial feature data of the meteorological dataset.
3. The method as described in claim 1, characterized in that, Anomaly identification is performed on all quality control feature data and radial feature data to obtain the corresponding comprehensive anomaly probability, specifically including: Identify multiple outliers in all quality control characteristic data and radial characteristic data; Based on all abnormal states, determine the comprehensive anomaly probability corresponding to all quality control feature data and radial feature data.
4. The method as described in claim 1, characterized in that, Anomaly labeling is performed on all data in the meteorological dataset based on all comprehensive anomaly probabilities, specifically including: Obtain the preset anomaly probability threshold; Compare all the combined anomaly probabilities with the anomaly probability threshold; If the overall anomaly probability is greater than or equal to the anomaly probability threshold, then the data corresponding to the overall anomaly probability is marked as an anomaly; If the overall anomaly probability is less than the anomaly probability threshold, then no marking is performed.
5. The method as described in claim 1, characterized in that, The specific division of the data space for the remaining data in the meteorological dataset includes: Construct the data space for the remaining data in the meteorological dataset and determine the initial segmentation dimension; The data space is divided into two subspaces based on the initial segmentation dimension; Obtain the preset values of data points during segmentation; Repeat the above partitioning process for all subspaces until the number of data points in all subspaces is less than the preset value, thus obtaining the partitioned data space.
6. The method as described in claim 1, characterized in that, Performing a nearest neighbor search on the partitioned data space to obtain the feature data point set of the remaining data at different state boundaries during data compression specifically includes: Determine the nearest neighbor search dimension; Select one data point from each data point in the divided data space as the selected data; Determine multiple state analogy differences between selected data and data within the range of the nearest neighbor search dimension; Determine the state boundary feature points of the selected data within the nearest neighbor search dimension based on the differences in all state categories; Continue to determine the state boundary feature points of the remaining data points; The set of all state boundary feature points is used as the feature data point set of the remaining data at different state boundaries during data compression.
7. The method as described in claim 1, characterized in that, Determining multiple data fluctuation features of the feature data point set specifically includes: Determine multiple data segments in the set of feature data points; Determine the fluctuation range and stability corresponding to all data segments; Determine the extreme fluctuation probability and spatial continuity for all data segments; All fluctuation amplitudes, stability, extreme fluctuation probabilities, and spatial continuity are used as data fluctuation characteristics of the feature data point set.
8. An agricultural meteorological monitoring data processing system, characterized in that, The agricultural meteorological monitoring data processing system includes: The data acquisition module is used to collect meteorological datasets of the target farmland area through an internet-based meteorological data platform. The processing module is used to determine multiple quality control feature data and radial feature data of the meteorological dataset; The processing module is also used to identify anomalies in all quality control feature data and radial feature data, obtain the corresponding comprehensive anomaly probability, and then mark all data in the meteorological dataset as anomalies based on all comprehensive anomaly probabilities. The processing module is also used to remove all data marked as abnormal from the meteorological dataset, divide the data space of the remaining data in the meteorological dataset, and then perform a nearest neighbor search on the divided data space to obtain the feature data point set of the remaining data at different state boundaries when performing data compression. The processing module is further configured to determine multiple data fluctuation characteristics of the feature data point set, allocate the compression error of the feature data point set according to all the data fluctuation characteristics, and then perform segmented compression on the feature data point set to obtain a clean dataset. The execution module is used to transmit the clean dataset to the agricultural meteorological monitoring data processing platform via Internet protocol.
9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the agricultural meteorological monitoring data processing method based on Internet big data as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the agricultural meteorological monitoring data processing method based on Internet big data as described in any one of claims 1 to 7.