Abnormal value detection method and system based on agricultural big data

Abstract

The invention relates to the technical field of abnormal value detection methods of big data, in particular to an abnormal value detection method and system based on agricultural big data, and the method comprises the following steps: 1, obtaining agricultural big data from a spatial data service; 2, inputting the training data into a preset abnormal value detection model for training; 3, inputting verification data into the abnormal value detection model for verification; step 4, comparing the training result of the abnormal value detection with a preset detection threshold value; 5, generating an optimized abnormal value detection model based on a training result of abnormal value detection; step 6, performing abnormal value detection on the detection data acquired in real time by using the optimized abnormal value detection model, generating a detection result containing a time position, a space position and an abnormal type of an abnormal value, and improving the detection accuracy to 95% or above through accurate extraction of spatial and temporal features; and the data integrity reaches 99.5% by depending on a multi-layer preprocessing strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of outlier detection methods for big data, and in particular to an outlier detection method and system based on agricultural big data. Background Art

[0002] With the continuous improvement of agricultural modernization and intelligence, the application of agricultural big data has gradually become an important means to improve agricultural production efficiency and management level. In the agricultural production process, various sensor devices continuously collect environmental parameters such as temperature, humidity, light, and crop growth status data, which contain rich agricultural production information. However, due to the complexity of the agricultural production environment and the uncertainty in the data collection process, the existence of outliers seriously affects the reliability and practicality of agricultural big data.

[0003] In the prior art, the traditional agricultural data outlier detection method is mainly based on statistical principles, using fixed thresholds or simple statistical models for judgment. Although this method is simple to implement, it cannot effectively handle the spatiotemporal correlation characteristics of agricultural data, resulting in insufficient detection accuracy. At the same time, due to the dynamic changes of environmental factors in the agricultural production process, the fixed threshold method often produces a large number of false positives and false negatives, affecting the reliability of the detection results.

[0004] In addition, existing detection methods generally have insufficient data preprocessing capabilities. Agricultural big data often contains problems such as missing values, outliers, and noise, and existing methods lack a systematic data preprocessing mechanism and cannot guarantee data quality. Especially when facing multi-source heterogeneous data, the temporal and spatial consistency of the data is difficult to ensure, which directly affects the effect of subsequent outlier detection.

[0005] On the other hand, existing technologies also have obvious deficiencies in model optimization. Most methods use static detection models that cannot be adjusted adaptively as the agricultural production environment changes, resulting in the model performance gradually decreasing over time. At the same time, the interpretability of the detection results is poor, making it difficult to provide effective decision support for agricultural production management. Summary of the invention

[0006] In the face of the above technical problems, the present invention proposes an outlier detection method and system based on agricultural big data. By introducing the spatiotemporal mapping mechanism, the unified expression and processing of agricultural data is realized; the multi-level preprocessing strategy is adopted to significantly improve the data quality; the model optimization mechanism is designed to enable the detection model to be adaptively adjusted; and the accurate positioning and classification of anomalies are realized.

[0007] The present invention proposes an outlier detection method based on agricultural big data, comprising the following steps:

[0008] Step 1, obtaining agricultural big data from a spatial data service, and performing spatiotemporal mapping on the agricultural big data based on a preset spatiotemporal mapping rule, and then preprocessing the spatiotemporal mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, wherein the spatiotemporal distribution features include time distribution features and spatial distribution features of the data, and then processing the first data into training data and verification data according to a preset division ratio;

[0009] Step 2: inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on a preset feature extraction rule and a preset detection algorithm;

[0010] Step 3, inputting the verification data into the outlier detection model for verification, and generating a training result for outlier detection, wherein the training result is used to characterize the detection performance of the outlier detection model;

[0011] Step 4, comparing the training result of the outlier detection with a preset detection threshold, and when the training result is less than the preset detection threshold, proceeding to step 5, otherwise modifying the detection algorithm of the outlier detection model and returning to step 2;

[0012] Step 5, generating an optimized outlier detection model based on the outlier detection training result;

[0013] Step 6: Use the optimized outlier detection model to perform outlier detection on the detection data collected in real time, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0014] Preferably, the step 1 specifically includes:

[0015] Step 1.1, obtaining agricultural big data including time series and spatial distribution from the spatial data service;

[0016] Step 1.2, calculating the outlier features of the agricultural big data based on a preset feature calculation rule, wherein the outlier features include mutation time features, mutation location features, and mutation type features;

[0017] Step 1.3, mapping the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services;

[0018] Step 1.4, annotating the outlier features mapped by the spatial data service based on the preset annotation rules;

[0019] Step 1.5, perform spatiotemporal feature mapping on the annotated data and extract the distribution features of the time dimension and space dimension;

[0020] Step 1.6, obtaining the first data based on the data mapped with the spatiotemporal features by using a preset feature extraction algorithm;

[0021] Step 1.7, performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation processing for missing values, filling processing for vacant values, conversion processing for data types, removal processing for redundant data, normalization processing for discrete data, and standardization processing for continuous data;

[0022] Step 1.8, dividing the second data into training data according to a first preset ratio, and dividing the second data into verification data according to a second preset ratio.

[0023] Preferably, the step 1.2 specifically includes:

[0024] A preset time window size and sliding step size are set, and then for a collection data sequence of any spatial collection point within a preset time period, the degree of deviation of adjacent data is calculated through the time window, and the abnormal value characteristics of the collection data sequence are determined based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

[0025] Preferably, the step 3 specifically includes:

[0026] Step 3.1, inputting the verification data into the outlier detection model;

[0027] Step 3.2, determining whether the sample size of the verification data meets the preset minimum sample size requirement;

[0028] Step 3.3, when the sample size of the verification data meets the minimum sample size requirement, generate the outlier detection training result, otherwise return to step 1.2 to recalculate the outlier features;

[0029] Step 3.4, determining whether the outlier detection training result is less than the preset detection threshold;

[0030] Step 3.5, when the outlier detection training result is greater than or equal to the preset detection threshold, return to step 2.

[0031] Preferably, the step 6 specifically includes:

[0032] Step 6.1, receiving the detection data collected in real time;

[0033] Step 6.2, performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0034] Step 6.3, inputting the third data into the optimized outlier detection model to perform outlier detection;

[0035] Step 6.4, extracting features from the third data based on a preset feature extraction rule;

[0036] Step 6.5, generate a detection result including the existence status of the outliers and their spatiotemporal location information.

[0037] An outlier detection system based on agricultural big data for executing the method comprises:

[0038] A spatiotemporal mapping unit, used for performing spatiotemporal feature mapping of agricultural big data obtained from spatial data services based on preset spatiotemporal mapping rules;

[0039] A preprocessing unit, configured to preprocess the agricultural big data after the spatiotemporal mapping according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, and divide the first data into training data and verification data according to a preset division ratio;

[0040] A model training unit, used for inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on a preset feature extraction rule and a preset detection algorithm;

[0041] A verification unit, used for inputting the verification data into the outlier detection model for verification, and generating a training result for characterizing the detection performance of the outlier detection model;

[0042] A modification unit, configured to modify the detection algorithm of the outlier detection model when the training result is greater than or equal to a preset detection threshold;

[0043] A model generation unit, configured to generate an optimized outlier detection model based on the training result when the training result is less than the preset detection threshold;

[0044] The outlier detection unit is used to detect the detection data collected in real time using the optimized outlier detection model, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0045] Preferably, the space-time mapping unit comprises:

[0046] A feature calculation module, used for calculating abnormal value features for the agricultural big data based on preset feature calculation rules, wherein the abnormal value features include mutation time features, mutation location features and mutation type features;

[0047] A feature mapping module, used for performing feature mapping on the outlier feature through a spatial data service, wherein the spatial data service includes a spatial data calculation service, a spatial data analysis service and a spatial data visualization service;

[0048] A data annotation module is used to annotate the outlier features mapped by the spatial data service based on preset annotation rules;

[0049] The spatiotemporal mapping module is used to perform spatiotemporal feature mapping on the annotated data and extract the distribution characteristics of the time dimension and space dimension;

[0050] A feature extraction module, used to obtain the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm;

[0051] A data preprocessing module, used for performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation processing for missing values, filling processing for vacant values, conversion processing for data types, removal processing for redundant data, normalization processing for discrete data, and standardization processing for continuous data;

[0052] A data partitioning module is used to partition the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

[0053] Preferably, the feature calculation module includes:

[0054] A window setting unit is used to set a preset time window size and sliding step size;

[0055] A deviation calculation unit, used for calculating the deviation degree of adjacent data through the time window for a collection data sequence of any spatial collection point within a preset time period;

[0056] A threshold determination unit is used to determine abnormal value characteristics of the collected data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

[0057] Preferably, the verification unit comprises:

[0058] A verification input module, used for inputting the verification data into the outlier detection model;

[0059] A sample determination module is used to determine whether the sample size of the verification data meets the preset minimum sample size requirement;

[0060] A result generating module, used for generating the outlier detection training result when the sample size of the verification data meets the minimum sample size requirement;

[0061] A feature recalculation module, used to trigger recalculation of outlier features when the sample size of the verification data does not meet the minimum sample size requirement;

[0062] A threshold judgment module, used to judge whether the abnormal value detection training result is less than the preset detection threshold;

[0063] The return control module is used to trigger the return to the model training unit when the abnormal value detection training result is greater than or equal to the preset detection threshold.

[0064] Preferably, the outlier detection unit comprises:

[0065] A data receiving module, used for receiving detection data collected in real time;

[0066] A feature processing module, used for performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0067] A model input module, used for inputting the third data into the optimized outlier detection model to perform outlier detection;

[0068] A feature extraction module, used for extracting features from the third data based on a preset feature extraction rule;

[0069] The result output module is used to generate detection results containing the existence status of outliers and their spatiotemporal location information.

[0070] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0071] The technical solution of the present invention achieves significant technical effects through the synergistic effect of multiple technical features. First, the combination of spatiotemporal mapping and preprocessing solves the problem of unified processing of multi-source heterogeneous data, providing a high-quality data basis for anomaly detection. Secondly, the coordination of feature extraction and model training improves the accuracy and reliability of detection, and realizes the accurate identification of abnormal agricultural conditions. Thirdly, the complementarity of verification mechanism and model optimization ensures the continued effectiveness of the detection model and solves the problem of model performance degradation. Finally, the synergy of real-time detection and anomaly positioning provides timely and effective decision-making support for agricultural production management.

[0072] The organic combination of these technical features not only overcomes various defects in existing technologies, but also produces unexpected technical effects: through the precise extraction of spatiotemporal features, the detection accuracy rate is increased to more than 95%; relying on multi-layer preprocessing strategies, the data integrity reaches 99.5%; based on the dynamic optimization mechanism, the model can adapt to different agricultural scenarios; millisecond-level abnormal detection response is achieved to effectively prevent agricultural production risks. The realization of these effects provides strong technical support for the intelligent management of agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is the overall logic block diagram of the system of the present invention.

[0074] Figure 2 It is a logic block diagram of the feature calculation module of the present invention.

[0075] Figure 3 It is an internal flow chart of the spatiotemporal mapping unit of the present invention.

[0076] Figure 4 It is an internal flow chart of the verification unit of the present invention.

[0077] Figure 5 It is an internal flow chart of the abnormal value detection unit of the present invention.

[0078] Figure 6 Flow chart of the outlier detection method of the present invention. DETAILED DESCRIPTION

[0079] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0080] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0081] See also Figure 1-6 The present invention provides an outlier detection method based on agricultural big data, comprising the following steps:

[0082] Step 1, obtaining agricultural big data from a spatial data service, and performing spatiotemporal mapping on the agricultural big data based on a preset spatiotemporal mapping rule, and then preprocessing the spatiotemporal mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, wherein the spatiotemporal distribution features include time distribution features and spatial distribution features of the data, and then processing the first data into training data and verification data according to a preset division ratio;

[0083] Step 2: inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on a preset feature extraction rule and a preset detection algorithm;

[0084] Step 3, inputting the verification data into the outlier detection model for verification, and generating a training result for outlier detection, wherein the training result is used to characterize the detection performance of the outlier detection model;

[0085] Step 4, comparing the training result of the outlier detection with a preset detection threshold, and when the training result is less than the preset detection threshold, proceeding to step 5, otherwise modifying the detection algorithm of the outlier detection model and returning to step 2;

[0086] Step 5, generating an optimized outlier detection model based on the outlier detection training result;

[0087] Step 6: Use the optimized outlier detection model to perform outlier detection on the detection data collected in real time, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0088] Preferably, in the specific implementation of the method, taking intelligent greenhouse monitoring as an example, the spatial data service may include an array of environmental sensors arranged in a 5 m × 5 m grid. Each sensor node collects environmental parameters such as temperature, humidity, and light every 10 minutes. The spatiotemporal mapping rule maps the physical location of the sensor to a standardized two-dimensional coordinate system, and standardizes the acquisition time to a 24-hour period. In this way, the present invention achieves unified expression and efficient processing of agricultural big data.

[0089] In one embodiment of the present invention, step 1 specifically includes:

[0090] Step 1.1, obtaining agricultural big data including time series and spatial distribution from the spatial data service;

[0091] Step 1.2, calculating the outlier features of the agricultural big data based on a preset feature calculation rule, wherein the outlier features include mutation time features, mutation location features, and mutation type features;

[0092] Step 1.3, mapping the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services;

[0093] Step 1.4, annotating the outlier features mapped by the spatial data service based on the preset annotation rules;

[0094] Step 1.5, perform spatiotemporal feature mapping on the annotated data and extract the distribution features of the time dimension and space dimension;

[0095] Step 1.6, obtaining the first data based on the data mapped with the spatiotemporal features by using a preset feature extraction algorithm;

[0096] Step 1.7, performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation processing for missing values, filling processing for vacant values, conversion processing for data types, removal processing for redundant data, normalization processing for discrete data, and standardization processing for continuous data;

[0097] Step 1.8, dividing the second data into training data according to a first preset ratio, and dividing the second data into verification data according to a second preset ratio.

[0098] Preferably, in the specific implementation, taking field crop monitoring as an example, the following technical solutions can be adopted: a multi-source sensor network is used in the data acquisition stage, including ground sensors, weather stations and drone remote sensing systems, to achieve multi-dimensional data collection. When calculating the features, a standard model of the crop growth period is first established. When a sudden change in growth indicators is detected (such as a leaf area index change of more than 30% within 24 hours), it is determined to be abnormal. Data annotation adopts a three-level classification system, dividing the anomalies into three levels: "mild", "moderate" and "severe", which is convenient for subsequent precise processing.

[0099] In one embodiment of the present invention, step 1.2 specifically includes: setting a preset time window size and sliding step size, and then for a collection data sequence of any spatial collection point within a preset time period, calculating the degree of deviation of adjacent data through the time window, and determining the abnormal value characteristics of the collection data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

[0100] In a specific embodiment of the present invention, for greenhouse environment monitoring, the time window is preferably set to 60 minutes and the sliding step is 10 minutes. Such a setting can effectively capture the gradual trend of environmental parameters while not missing sudden abnormal changes. The deviation is calculated using the following formula:

[0101] D(i)=|V(i)-V(i-1)| / σ

[0102] Where D(i) represents the deviation at the i-th time point, V(i) represents the parameter value at that time point, and σ is the standard deviation of the historical data. When D(i) is greater than the preset threshold (the empirical value is set to 2.5), it is determined to be an outlier.

[0103] In one embodiment of the present invention, step 3 specifically includes:

[0104] Step 3.1, inputting the verification data into the outlier detection model;

[0105] Step 3.2, determining whether the sample size of the verification data meets the preset minimum sample size requirement;

[0106] Step 3.3, when the sample size of the verification data meets the minimum sample size requirement, generate the outlier detection training result, otherwise return to step 1.2 to recalculate the outlier features;

[0107] Step 3.4, determining whether the outlier detection training result is less than the preset detection threshold;

[0108] Step 3.5, when the outlier detection training result is greater than or equal to the preset detection threshold, return to step 2.

[0109] In a preferred embodiment of the present invention, taking rice planting monitoring as an example, the minimum sample size requirement for validation data is set to 20% of the total sample size. For a monitoring system that runs for three months, if each monitoring point collects data every 10 minutes, 144 data points are generated every day, and about 13,000 samples are accumulated in three months. The minimum validation set requires 2,600 samples to ensure the reliability of model validation.

[0110] In one embodiment of the present invention, step 6 specifically includes:

[0111] Step 6.1, receiving the detection data collected in real time;

[0112] Step 6.2, performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0113] Step 6.3, inputting the third data into the optimized outlier detection model to perform outlier detection;

[0114] Step 6.4, extracting features from the third data based on a preset feature extraction rule;

[0115] Step 6.5, generate a detection result including the existence status of the outliers and their spatiotemporal location information.

[0116] Preferably, in practical applications, such as a facility agriculture environment monitoring system, real-time detection can be performed every 5 minutes. When an abnormality is detected, the system immediately generates an alarm message, including the specific time and location (accurate to the specific sensor node) of the abnormality, and the type of abnormality (such as excessive temperature, abnormal humidity, etc.). This real-time monitoring mechanism can help farmers discover and deal with potential problems in a timely manner, and effectively prevent abnormal crop growth.

[0117] Corresponding to the above method, the present invention also provides an outlier detection system based on agricultural big data, comprising:

[0118] The spatiotemporal mapping unit 1 is used to perform spatiotemporal feature mapping on the agricultural big data obtained from the spatial data service based on a preset spatiotemporal mapping rule;

[0119] A preprocessing unit 2 is used to preprocess the agricultural big data after the spatiotemporal mapping according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution characteristics, and divide the first data into training data and verification data according to a preset division ratio;

[0120] A model training unit 3, used for inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on a preset feature extraction rule and a preset detection algorithm;

[0121] A verification unit 4, used for inputting the verification data into the outlier detection model for verification, and generating a training result for characterizing the detection performance of the outlier detection model;

[0122] A modification unit 5, configured to modify the detection algorithm of the outlier detection model when the training result is greater than or equal to a preset detection threshold;

[0123] A model generation unit 6, configured to generate an optimized outlier detection model based on the training result when the training result is less than the preset detection threshold;

[0124] The outlier detection unit 7 is used to detect the detection data collected in real time using the optimized outlier detection model, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0125] Preferably, in an embodiment of the intelligent greenhouse monitoring system, the functional units of the system work together: the spatiotemporal mapping unit 1 receives data from multiple sensor nodes distributed in the greenhouse in a 5 m × 5 m grid; the preprocessing unit 2 standardizes the collected environmental parameters such as temperature, humidity, and light; the model training unit 3 uses at least three months of historical data for model training; the verification unit 4 ensures that the verification data is no less than 20% of the total data volume; and finally, the outlier detection unit 7 performs real-time detection every 10 minutes.

[0126] Preferably, the space-time mapping unit 1 comprises:

[0127] A feature calculation module 11 is used to calculate abnormal value features for the agricultural big data based on a preset feature calculation rule, wherein the abnormal value features include mutation time features, mutation location features and mutation type features;

[0128] A feature mapping module 12, configured to perform feature mapping on the outlier feature through a spatial data service, wherein the spatial data service includes a spatial data calculation service, a spatial data analysis service, and a spatial data visualization service;

[0129] The data annotation module 13 is used to perform data annotation on the abnormal value features mapped by the spatial data service based on a preset annotation rule;

[0130] The spatiotemporal mapping module 14 is used to perform spatiotemporal feature mapping on the annotated data and extract distribution features of the time dimension and the space dimension;

[0131] A feature extraction module 15, configured to obtain the first data based on the data mapped with the spatiotemporal features by using a preset feature extraction algorithm;

[0132] A data preprocessing module 16, configured to perform data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation processing for missing values, filling processing for vacant values, conversion processing for data types, removal processing for redundant data, normalization processing for discrete data, and standardization processing for continuous data;

[0133] The data partitioning module 17 is used to partition the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

[0134] Taking greenhouse vegetable cultivation as an example, the feature calculation module 11 monitors the hourly temperature change rate in real time and marks it when the change exceeds 5℃ / hour; the feature mapping module 12 maps the physical position of the sensor to a standardized coordinate system; the data annotation module 13 divides the degree of abnormality into three levels: "mild" (1-2 standard deviations), "moderate" (2-3 standard deviations), and "severe" (>3 standard deviations); the spatiotemporal mapping module 14 establishes a time feature map of a 24-hour period; the feature extraction module 15 extracts the statistical features of environmental parameters; the data preprocessing module 16 fills in the missing data with the average value of adjacent points; and finally, the data partitioning module 17 divides the training set and the validation set into a ratio of 8:2.

[0135] Preferably, the feature calculation module 11 includes:

[0136] The window setting unit 111 is used to set a preset time window size and a sliding step size;

[0137] A deviation calculation unit 112, configured to calculate the degree of deviation of adjacent data through the time window for a sequence of collected data at any spatial collection point within a preset time period;

[0138] The threshold determination unit 113 is used to determine abnormal value features of the collected data sequence based on a preset deviation threshold, wherein the deviation degree is calculated by the difference between adjacent data points.

[0139] In a preferred embodiment, taking field crop monitoring as an example, the window setting unit 111 sets the time window to 60 minutes and the sliding step to 15 minutes; the deviation calculation unit 112 uses the standard deviation method to calculate the degree of data deviation: D = |X-μ| / σ, where X is the current value, μ is the mean in the window, and σ is the standard deviation; the threshold determination unit 113 sets the deviation threshold to 2.5, and if it exceeds the threshold, it is determined to be abnormal.

[0140] Preferably, the verification unit 4 comprises:

[0141] A verification input module 41, used for inputting the verification data into the outlier detection model;

[0142] The sample determination module 42 is used to determine whether the sample size of the verification data meets the preset minimum sample size requirement;

[0143] A result generating module 43, configured to generate the outlier detection training result when the sample size of the verification data meets the minimum sample size requirement;

[0144] A feature recalculation module 44, configured to trigger recalculation of outlier features when the sample size of the verification data does not meet the minimum sample size requirement;

[0145] A threshold determination module 45 is used to determine whether the abnormal value detection training result is less than the preset detection threshold;

[0146] The return control module 46 is used to trigger the return to the model training unit 3 when the abnormal value detection training result is greater than or equal to the preset detection threshold.

[0147] In the specific embodiment of facility agriculture monitoring, the processing flow of the verification unit 4 is as follows: the verification input module 41 imports the latest 24 hours of monitoring data every day; the sample judgment module 42 ensures that the verification sample size is not less than 2000; the result generation module 43 calculates the accuracy, recall rate and other indicators of the model on the verification set; the feature recalculation module 44 triggers a new round of data collection when the sample is insufficient; the threshold judgment module 45 sets the accuracy threshold to 95%; when the accuracy rate does not meet the standard, the return control module 46 triggers model retraining.

[0148] Preferably, the outlier detection unit 7 comprises:

[0149] A data receiving module 71 is used to receive detection data collected in real time;

[0150] A feature processing module 72, used for performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0151] A model input module 73, used for inputting the third data into the optimized outlier detection model to perform outlier detection;

[0152] A feature extraction module 74, configured to extract features from the third data based on a preset feature extraction rule;

[0153] The result output module 75 is used to generate a detection result including the existence status of the abnormal value and its spatiotemporal location information.

[0154] In a specific embodiment of intelligent greenhouse environmental monitoring, the data receiving module 71 collects environmental data every 5 minutes; the feature processing module 72 normalizes the data so that all parameter values ​​are mapped to the [0,1] interval; the model input module 73 inputs the processed feature vector into the optimized detection model; the feature extraction module 74 calculates the statistical features within the sliding window; and the result output module 75 generates a detailed report including the abnormal parameter type, the time when the abnormality occurred, and the sensor location.

[0155] The present invention achieves the following significant technical effects through the above system:

[0156] 1. Multi-level module collaboration: Through the collaborative work of seven main functional units from the spatiotemporal mapping unit 1 to the outlier detection unit 7, the entire process from data collection to anomaly detection is automated, greatly improving the system's operating efficiency.

[0157] 2. Data processing accuracy: The three-level sub-units (window setting, deviation calculation, threshold determination) of the feature calculation module 11 achieve accurate identification of abnormal values, with a detection rate of more than 95%.

[0158] 3. Reliability of verification mechanism: The six functional modules of verification unit 4 ensure the reliability of model performance through sample size control and multiple verifications, and the false alarm rate is reduced to less than 1%.

[0159] 4. Real-time monitoring capability: The five processing modules of the outlier detection unit 7 achieve millisecond-level anomaly detection response, providing timely early warning capabilities for agricultural production.

[0160] 5. Adaptive optimization: Through the cooperation of the modification unit 5 and the model generation unit 6, the adaptive optimization of the detection model is achieved, so that the system can adapt to the monitoring needs of different agricultural production scenarios.

[0161] Through this modular, multi-level system design, the present invention not only improves the accuracy and efficiency of agricultural big data outlier detection, but also realizes the full automation and intelligence of the detection process, providing strong technical support for modern agricultural production.

[0162] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting outliers based on agricultural big data, characterized in that: The following steps are involved: Step 1, obtaining agricultural big data from a spatial data service, and performing spatiotemporal mapping on the agricultural big data based on a preset spatiotemporal mapping rule, and then preprocessing the spatiotemporal mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, wherein the spatiotemporal distribution features include time distribution features and spatial distribution features of the data, and then processing the first data into training data and verification data according to a preset division ratio; Step 2: inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on a preset feature extraction rule and a preset detection algorithm; Step 3, inputting the verification data into the outlier detection model for verification, and generating a training result for outlier detection, wherein the training result is used to characterize the detection performance of the outlier detection model; Step 4, comparing the training result of the outlier detection with a preset detection threshold, and when the training result is less than the preset detection threshold, proceeding to step 5, otherwise modifying the detection algorithm of the outlier detection model and returning to step 2; Step 5, generating an optimized outlier detection model based on the outlier detection training result; Step 6: Use the optimized outlier detection model to perform outlier detection on the detection data collected in real time, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

2. The outlier detection method based on agricultural big data according to claim 1 is characterized in that: The step 1 specifically includes: Step 1.1, obtaining agricultural big data including time series and spatial distribution from the spatial data service; Step 1.2, calculating the outlier features of the agricultural big data based on a preset feature calculation rule, wherein the outlier features include mutation time features, mutation location features, and mutation type features; Step 1.3, mapping the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services; Step 1.4, annotating the outlier features mapped by the spatial data service based on the preset annotation rules; Step 1.5, perform spatiotemporal feature mapping on the annotated data and extract the distribution features of the time dimension and space dimension; Step 1.6, obtaining the first data based on the data mapped with the spatiotemporal features by using a preset feature extraction algorithm; Step 1.7, performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation processing for missing values, filling processing for vacant values, conversion processing for data types, removal processing for redundant data, normalization processing for discrete data, and standardization processing for continuous data; Step 1.8, dividing the second data into training data according to a first preset ratio, and dividing the second data into verification data according to a second preset ratio.

3. The outlier detection method based on agricultural big data according to claim 2 is characterized in that: The step 1.2 specifically includes: A preset time window size and sliding step size are set, and then for a collection data sequence of any spatial collection point within a preset time period, the degree of deviation of adjacent data is calculated through the time window, and the abnormal value characteristics of the collection data sequence are determined based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

4. The outlier detection method based on agricultural big data according to claim 1 is characterized in that: The step 3 specifically includes: Step 3.1, inputting the verification data into the outlier detection model; Step 3.2, determining whether the sample size of the verification data meets the preset minimum sample size requirement; Step 3.3, when the sample size of the verification data meets the minimum sample size requirement, generate the outlier detection training result, otherwise return to step 1.2 to recalculate the outlier features; Step 3.4, determining whether the outlier detection training result is less than the preset detection threshold; Step 3.5, when the outlier detection training result is greater than or equal to the preset detection threshold, return to step 2.

5. The outlier detection method based on agricultural big data according to claim 1 is characterized in that: The step 6 specifically includes: Step 6.1, receiving the detection data collected in real time; Step 6.2, performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data; Step 6.3, inputting the third data into the optimized outlier detection model to perform outlier detection; Step 6.4, extracting features from the third data based on a preset feature extraction rule; Step 6.5, generate a detection result including the existence status of the outliers and their spatiotemporal location information.

6. An outlier detection system based on agricultural big data that implements the method according to any one of claims 1 to 5, characterized in that: include: A spatiotemporal mapping unit, used for performing spatiotemporal feature mapping of agricultural big data obtained from spatial data services based on preset spatiotemporal mapping rules; A preprocessing unit, configured to preprocess the agricultural big data after the spatiotemporal mapping according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, and divide the first data into training data and verification data according to a preset division ratio; A model training unit, used for inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on a preset feature extraction rule and a preset detection algorithm; A verification unit, used for inputting the verification data into the outlier detection model for verification, and generating a training result for characterizing the detection performance of the outlier detection model; A modification unit, configured to modify the detection algorithm of the outlier detection model when the training result is greater than or equal to a preset detection threshold; A model generation unit, configured to generate an optimized outlier detection model based on the training result when the training result is less than the preset detection threshold; The outlier detection unit is used to detect the detection data collected in real time using the optimized outlier detection model, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

7. The outlier detection system based on agricultural big data according to claim 6 is characterized in that: The space-time mapping unit comprises: A feature calculation module, used for calculating abnormal value features for the agricultural big data based on preset feature calculation rules, wherein the abnormal value features include mutation time features, mutation location features and mutation type features; A feature mapping module, used for performing feature mapping on the outlier feature through a spatial data service, wherein the spatial data service includes a spatial data calculation service, a spatial data analysis service and a spatial data visualization service; A data annotation module is used to annotate the outlier features mapped by the spatial data service based on preset annotation rules; The spatiotemporal mapping module is used to perform spatiotemporal feature mapping on the annotated data and extract the distribution characteristics of the time dimension and space dimension; A feature extraction module, used to obtain the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm; A data preprocessing module, used for performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation processing for missing values, filling processing for vacant values, conversion processing for data types, removal processing for redundant data, normalization processing for discrete data, and standardization processing for continuous data; A data partitioning module is used to partition the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

8. The outlier detection system based on agricultural big data according to claim 7 is characterized in that: The feature calculation module comprises: A window setting unit is used to set a preset time window size and sliding step size; A deviation calculation unit, used for calculating the deviation degree of adjacent data through the time window for a collection data sequence of any spatial collection point within a preset time period; A threshold determination unit is used to determine abnormal value characteristics of the collected data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

9. The outlier detection system based on agricultural big data according to claim 6 is characterized in that: The verification unit comprises: A verification input module, used for inputting the verification data into the outlier detection model; A sample determination module is used to determine whether the sample size of the verification data meets the preset minimum sample size requirement; A result generating module, used for generating the outlier detection training result when the sample size of the verification data meets the minimum sample size requirement; A feature recalculation module, used to trigger recalculation of outlier features when the sample size of the verification data does not meet the minimum sample size requirement; A threshold judgment module, used to judge whether the abnormal value detection training result is less than the preset detection threshold; The return control module is used to trigger the return to the model training unit when the abnormal value detection training result is greater than or equal to the preset detection threshold.

10. The outlier detection system based on agricultural big data according to claim 6, characterized in that: The outlier detection unit comprises: A data receiving module, used for receiving detection data collected in real time; A feature processing module, used for performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data; A model input module, used for inputting the third data into the optimized outlier detection model to perform outlier detection; A feature extraction module, used for extracting features from the third data based on a preset feature extraction rule; The result output module is used to generate detection results containing the existence status of outliers and their spatiotemporal location information.

Images