An outlier detection method and system based on agricultural big data

Abstract

The present invention relates to the technical field of outlier detection methods for big data, and in particular to an outlier detection method and system based on agricultural big data. The method and system comprise the following steps: step 1, obtaining agricultural big data from a spatial data service; step 2, inputting training data into a preset outlier detection model for training; step 3, inputting verification data into the outlier detection model for verification; step 4, comparing the training results of the outlier detection with a preset detection threshold; step 5, generating an optimized outlier detection model based on the training results of the outlier detection; and step 6, performing outlier detection on the detection data collected in real time using the optimized outlier detection model, and generating a detection result including the time position, spatial position and anomaly type of the outlier. By accurately extracting spatiotemporal features, the detection accuracy is improved to over 95%. By relying on a multi-layer preprocessing strategy, the data integrity reaches 99.5%.

Description

Technical Field

[0001] The present invention relates to the technical field of outlier detection methods for big data, and in particular to an outlier detection method and system based on agricultural big data. Background Art

[0002] With the continuous advancement of agricultural modernization and intelligentization, the application of agricultural big data has become an important means of improving agricultural production efficiency and management. During agricultural production, various sensor devices continuously collect data on environmental parameters such as temperature, humidity, and light, as well as crop growth status. This data contains a wealth of agricultural production information. However, due to the complexity of the agricultural production environment and the uncertainty in the data collection process, the presence of outliers seriously affects the reliability and practicality of agricultural big data.

[0003] Conventional methods for detecting outliers in agricultural data are primarily based on statistical principles, employing fixed thresholds or simple statistical models. While simple to implement, these methods fail to effectively address the spatiotemporal correlations inherent in agricultural data, resulting in insufficient detection accuracy. Furthermore, due to the dynamic nature of environmental factors during agricultural production, fixed threshold methods often produce a high number of false positives and false negatives, compromising the reliability of detection results.

[0004] Furthermore, existing detection methods generally suffer from insufficient data preprocessing capabilities. Agricultural big data often contains missing values, outliers, and noise, and existing methods lack systematic data preprocessing mechanisms, making it difficult to guarantee data quality. This is especially true when dealing with multi-source, heterogeneous data, making it difficult to ensure spatial and temporal consistency, which directly impacts the effectiveness of subsequent outlier detection.

[0005] On the other hand, existing technologies also have significant shortcomings in model optimization. Most methods use static detection models that are unable to adapt to changes in the agricultural production environment, resulting in a gradual decline in model performance over time. Furthermore, the interpretability of detection results is poor, making it difficult to provide effective decision support for agricultural production management. Summary of the Invention

[0006] To address these technical challenges, this paper proposes a method and system for detecting outliers based on agricultural big data. By introducing a spatiotemporal mapping mechanism, this approach achieves unified representation and processing of agricultural data. A multi-level preprocessing strategy significantly improves data quality. A model optimization mechanism enables adaptive adjustment of the detection model, achieving precise location and classification of anomalies.

[0007] The present invention proposes an outlier detection method based on agricultural big data, comprising the following steps:

[0008] Step 1: Acquire agricultural big data from a spatial data service, perform spatiotemporal mapping on the agricultural big data based on a preset spatiotemporal mapping rule, and then preprocess the spatiotemporally mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, wherein the spatiotemporal distribution features include temporal distribution features and spatial distribution features of the data. Then, the first data is processed into training data and validation data according to a preset division ratio.

[0009] Step 2: inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on preset feature extraction rules and a preset detection algorithm;

[0010] Step 3: inputting the verification data into the outlier detection model for verification and generating a training result for outlier detection, wherein the training result is used to characterize the detection performance of the outlier detection model;

[0011] Step 4: Compare the training result of the outlier detection with a preset detection threshold. If the training result is less than the preset detection threshold, proceed to step 5; otherwise, modify the detection algorithm of the outlier detection model and return to step 2.

[0012] Step 5: generating an optimized outlier detection model based on the outlier detection training results;

[0013] Step 6: Use the optimized outlier detection model to perform outlier detection on the detection data collected in real time, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0014] Preferably, the step 1 specifically includes:

[0015] Step 1.1, obtaining agricultural big data including time series and spatial distribution from the spatial data service;

[0016] Step 1.2, calculating outlier features for the agricultural big data based on preset feature calculation rules, wherein the outlier features include mutation time features, mutation location features, and mutation type features;

[0017] Step 1.3, feature mapping the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services;

[0018] Step 1.4: annotate the outlier features mapped by the spatial data service based on the preset annotation rules;

[0019] Step 1.5: Perform spatiotemporal feature mapping on the labeled data and extract the distribution features of the time and space dimensions;

[0020] Step 1.6, obtaining the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm;

[0021] Step 1.7, performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensating for missing values, filling in vacancies, converting data types, removing redundant data, normalizing discrete data, and standardizing continuous data;

[0022] Step 1.8: Divide the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

[0023] Preferably, the step 1.2 specifically includes:

[0024] A preset time window size and sliding step size are set, and then for the collected data sequence of any spatial collection point within the preset time period, the degree of deviation of adjacent data is calculated through the time window, and the outlier characteristics of the collected data sequence are determined based on the preset deviation threshold, where the degree of deviation is calculated by the difference between adjacent data points.

[0025] Preferably, the step 3 specifically includes:

[0026] Step 3.1, inputting the verification data into the outlier detection model;

[0027] Step 3.2: Determine whether the sample size of the verification data meets the preset minimum sample size requirement;

[0028] Step 3.3: When the sample size of the verification data meets the minimum sample size requirement, generate the outlier detection training result; otherwise, return to step 1.2 to recalculate the outlier features;

[0029] Step 3.4, determining whether the outlier detection training result is less than the preset detection threshold;

[0030] Step 3.5: When the outlier detection training result is greater than or equal to the preset detection threshold, return to step 2.

[0031] Preferably, step 6 specifically includes:

[0032] Step 6.1, receiving the detection data collected in real time;

[0033] Step 6.2, performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0034] Step 6.3, inputting the third data into the optimized outlier detection model to perform outlier detection;

[0035] Step 6.4, performing feature extraction on the third data based on a preset feature extraction rule;

[0036] Step 6.5: Generate a detection result including the presence status of the outliers and their spatiotemporal location information.

[0037] An outlier detection system based on agricultural big data for executing the method includes:

[0038] A spatiotemporal mapping unit is used to perform spatiotemporal feature mapping on the agricultural big data obtained from the spatial data service based on preset spatiotemporal mapping rules;

[0039] a preprocessing unit, configured to preprocess the spatiotemporal mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, and divide the first data into training data and verification data according to a preset division ratio;

[0040] A model training unit, configured to input the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on preset feature extraction rules and a preset detection algorithm;

[0041] a verification unit, configured to input the verification data into the outlier detection model for verification, and generate a training result for characterizing the detection performance of the outlier detection model;

[0042] a modification unit, configured to modify the detection algorithm of the outlier detection model when the training result is greater than or equal to a preset detection threshold;

[0043] A model generation unit, configured to generate an optimized outlier detection model based on the training result when the training result is less than the preset detection threshold;

[0044] The outlier detection unit is used to detect the detection data collected in real time using the optimized outlier detection model and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0045] Preferably, the spatiotemporal mapping unit includes:

[0046] A feature calculation module is used to calculate outlier features for the agricultural big data based on preset feature calculation rules, wherein the outlier features include mutation time features, mutation location features, and mutation type features;

[0047] A feature mapping module, configured to perform feature mapping on the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services;

[0048] The data annotation module is used to annotate the outlier features mapped by the spatial data service based on preset annotation rules;

[0049] The spatiotemporal mapping module is used to perform spatiotemporal feature mapping on the labeled data and extract the distribution characteristics of the time and space dimensions;

[0050] A feature extraction module, configured to obtain the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm;

[0051] a data preprocessing module, configured to perform data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation for missing values, filling of vacant values, conversion of data types, removal of redundant data, normalization of discrete data, and standardization of continuous data;

[0052] The data partitioning module is used to partition the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

[0053] Preferably, the feature calculation module includes:

[0054] Window setting unit, used to set the preset time window size and sliding step;

[0055] a deviation calculation unit, configured to calculate the degree of deviation of adjacent data from a sequence of collected data at any spatial collection point within a preset time period through the time window;

[0056] A threshold determination unit is used to determine abnormal value characteristics of the collected data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

[0057] Preferably, the verification unit includes:

[0058] a verification input module, configured to input the verification data into the outlier detection model;

[0059] A sample determination module is used to determine whether the sample size of the verification data meets the preset minimum sample size requirement;

[0060] A result generating module, configured to generate the outlier detection training result when the sample size of the verification data meets the minimum sample size requirement;

[0061] A feature recalculation module, configured to trigger recalculation of outlier features when the sample size of the verification data does not meet the minimum sample size requirement;

[0062] A threshold judgment module is used to judge whether the outlier detection training result is less than the preset detection threshold;

[0063] The return control module is used to trigger the return to the model training unit when the abnormal value detection training result is greater than or equal to the preset detection threshold.

[0064] Preferably, the outlier detection unit includes:

[0065] A data receiving module is used to receive detection data collected in real time;

[0066] a feature processing module, configured to perform spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0067] A model input module, configured to input the third data into the optimized outlier detection model for outlier detection;

[0068] A feature extraction module, configured to extract features from the third data based on a preset feature extraction rule;

[0069] The result output module is used to generate detection results containing the existence status of outliers and their spatiotemporal location information.

[0070] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0071] The technical solution of the present invention achieves significant technical effects through the synergistic effect of multiple technical features. First, the combination of spatiotemporal mapping and preprocessing solves the problem of unified processing of multi-source heterogeneous data, providing a high-quality data foundation for anomaly detection. Second, the coordination of feature extraction and model training improves the accuracy and reliability of detection, enabling accurate identification of agricultural anomalies. Third, the complementary verification mechanism and model optimization ensure the continued effectiveness of the detection model and solve the problem of model performance degradation. Finally, the synergy of real-time detection and anomaly localization provides timely and effective decision-making support for agricultural production management.

[0072] The organic combination of these technical features not only overcomes various shortcomings of existing technologies but also produces unexpected technical results: through the precise extraction of spatiotemporal features, detection accuracy is increased to over 95%; relying on a multi-layer preprocessing strategy, data integrity reaches 99.5%; based on a dynamic optimization mechanism, the model can adapt to different agricultural scenarios; and millisecond-level anomaly detection response is achieved, effectively preventing agricultural production risks. The realization of these results provides strong technical support for the intelligent management of agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is the overall logic block diagram of the system of the present invention.

[0074] Figure 2 This is a logic block diagram of the feature calculation module of the present invention.

[0075] Figure 3 This is an internal flow chart of the spatiotemporal mapping unit of the present invention.

[0076] Figure 4 This is an internal flow chart of the verification unit of the present invention.

[0077] Figure 5 This is an internal flow chart of the abnormal value detection unit of the present invention.

[0078] Figure 6 Flowchart of the outlier detection method of the present invention. DETAILED DESCRIPTION

[0079] To further illustrate the technical means and effects employed by the present invention to achieve its intended objectives, the following detailed description, along with the accompanying drawings and preferred embodiments, includes a detailed description of the specific implementations, structures, features, and effects thereof. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0080] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0081] See Figure 1-6 The present invention provides an outlier detection method based on agricultural big data, comprising the following steps:

[0082] Step 1: Acquire agricultural big data from a spatial data service, perform spatiotemporal mapping on the agricultural big data based on a preset spatiotemporal mapping rule, and then preprocess the spatiotemporally mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, wherein the spatiotemporal distribution features include temporal distribution features and spatial distribution features of the data. Then, the first data is processed into training data and validation data according to a preset division ratio.

[0083] Step 2: inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on preset feature extraction rules and a preset detection algorithm;

[0084] Step 3: inputting the verification data into the outlier detection model for verification and generating a training result for outlier detection, wherein the training result is used to characterize the detection performance of the outlier detection model;

[0085] Step 4: Compare the training result of the outlier detection with a preset detection threshold. If the training result is less than the preset detection threshold, proceed to step 5; otherwise, modify the detection algorithm of the outlier detection model and return to step 2.

[0086] Step 5: generating an optimized outlier detection model based on the outlier detection training results;

[0087] Step 6: Use the optimized outlier detection model to perform outlier detection on the detection data collected in real time, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0088] In a specific implementation of this method, using smart greenhouse monitoring as an example, the spatial data service can include an array of environmental sensors arranged in a 5m x 5m grid. Each sensor node collects environmental parameters such as temperature, humidity, and light every 10 minutes. A spatiotemporal mapping rule maps the sensor's physical location to a standardized two-dimensional coordinate system, while also normalizing the collection time to a 24-hour period. In this way, the present invention achieves unified representation and efficient processing of agricultural big data.

[0089] In one embodiment of the present invention, step 1 specifically includes:

[0090] Step 1.1, obtaining agricultural big data including time series and spatial distribution from the spatial data service;

[0091] Step 1.2, calculating outlier features for the agricultural big data based on preset feature calculation rules, wherein the outlier features include mutation time features, mutation location features, and mutation type features;

[0092] Step 1.3, feature mapping the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services;

[0093] Step 1.4: annotate the outlier features mapped by the spatial data service based on the preset annotation rules;

[0094] Step 1.5: Perform spatiotemporal feature mapping on the labeled data and extract the distribution features of the time and space dimensions;

[0095] Step 1.6, obtaining the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm;

[0096] Step 1.7, performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensating for missing values, filling in vacancies, converting data types, removing redundant data, normalizing discrete data, and standardizing continuous data;

[0097] Step 1.8: Divide the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

[0098] Optimally, in specific implementation, using field crop monitoring as an example, the following technical solution can be adopted: During the data acquisition phase, a multi-source sensor network, including ground sensors, weather stations, and drone remote sensing systems, is employed to achieve multi-dimensional data collection. During feature calculation, a standard model for the crop growth period is first established. When a sudden change in a growth indicator is detected (e.g., a change in leaf area index exceeding 30% within 24 hours), it is identified as an anomaly. Data annotation utilizes a three-level classification system, categorizing anomalies into "mild," "moderate," and "severe," facilitating subsequent precise processing.

[0099] In one embodiment of the present invention, step 1.2 specifically includes: setting a preset time window size and sliding step size, and then for a collection data sequence of any spatial collection point within a preset time period, calculating the degree of deviation of adjacent data through the time window, and determining the outlier characteristics of the collection data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

[0100] In a specific embodiment of the present invention, for greenhouse environment monitoring, the time window is preferably set to 60 minutes and the sliding step is 10 minutes. This setting can effectively capture the gradual change trend of environmental parameters while not missing sudden abnormal changes. The deviation is calculated using the following formula:

[0101] D(i)=|V(i)-V(i-1)| / σ

[0102] Where D(i) represents the deviation at the i-th time point, V(i) represents the parameter value at that time point, and σ is the standard deviation of the historical data. When D(i) is greater than the preset threshold (the empirical value is set to 2.5), it is determined to be an outlier.

[0103] In one embodiment of the present invention, step 3 specifically includes:

[0104] Step 3.1, inputting the verification data into the outlier detection model;

[0105] Step 3.2: Determine whether the sample size of the verification data meets the preset minimum sample size requirement;

[0106] Step 3.3: When the sample size of the verification data meets the minimum sample size requirement, generate the outlier detection training result; otherwise, return to step 1.2 to recalculate the outlier features;

[0107] Step 3.4, determining whether the outlier detection training result is less than the preset detection threshold;

[0108] Step 3.5: When the outlier detection training result is greater than or equal to the preset detection threshold, return to step 2.

[0109] In a preferred embodiment of the present invention, using rice cultivation monitoring as an example, the minimum sample size requirement for validation data is set at 20% of the total sample size. For a monitoring system operating for three months, if each monitoring point collects data every 10 minutes, this generates 144 data points per day, accumulating approximately 13,000 samples over three months. Therefore, a minimum validation set of 2,600 samples is required to ensure the reliability of model validation.

[0110] In one embodiment of the present invention, step 6 specifically includes:

[0111] Step 6.1, receiving the detection data collected in real time;

[0112] Step 6.2, performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0113] Step 6.3, inputting the third data into the optimized outlier detection model to perform outlier detection;

[0114] Step 6.4, performing feature extraction on the third data based on a preset feature extraction rule;

[0115] Step 6.5: Generate a detection result including the presence status of the outliers and their spatiotemporal location information.

[0116] Ideally, in practical applications, such as a facility agriculture environmental monitoring system, real-time monitoring can be performed every five minutes. When an anomaly is detected, the system immediately generates an alert, including the specific time and location (accurate to the specific sensor node) of the anomaly, and the type of anomaly (e.g., excessive temperature, abnormal humidity, etc.). This real-time monitoring mechanism can help farmers promptly identify and address potential problems, effectively preventing abnormal crop growth.

[0117] Corresponding to the above method, the present invention also provides an outlier detection system based on agricultural big data, comprising:

[0118] The spatiotemporal mapping unit 1 is used to perform spatiotemporal feature mapping on the agricultural big data obtained from the spatial data service based on preset spatiotemporal mapping rules;

[0119] A preprocessing unit 2 is configured to preprocess the agricultural big data after the spatiotemporal mapping according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, and divide the first data into training data and verification data according to a preset division ratio;

[0120] A model training unit 3 is configured to input the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on preset feature extraction rules and a preset detection algorithm;

[0121] A verification unit 4 is configured to input the verification data into the outlier detection model for verification and generate a training result for characterizing the detection performance of the outlier detection model;

[0122] A modification unit 5 is configured to modify the detection algorithm of the outlier detection model when the training result is greater than or equal to a preset detection threshold;

[0123] A model generating unit 6 is configured to generate an optimized outlier detection model based on the training result when the training result is less than the preset detection threshold;

[0124] The outlier detection unit 7 is used to detect the detection data collected in real time using the optimized outlier detection model, and generate a detection result including the time position, spatial position and anomaly type of the outlier.

[0125] Preferably, in an embodiment of the intelligent greenhouse monitoring system, the functional units of the system work together: the spatiotemporal mapping unit 1 receives data from multiple sensor nodes distributed in the greenhouse in a 5 m × 5 m grid; the preprocessing unit 2 standardizes the collected environmental parameters such as temperature, humidity, and light; the model training unit 3 uses at least three months of historical data for model training; the verification unit 4 ensures that the verification data is no less than 20% of the total data volume; and finally, the outlier detection unit 7 performs real-time detection every 10 minutes.

[0126] Preferably, the spatiotemporal mapping unit 1 includes:

[0127] A feature calculation module 11 is used to calculate outlier features for the agricultural big data based on preset feature calculation rules, wherein the outlier features include mutation time features, mutation location features, and mutation type features;

[0128] A feature mapping module 12 is configured to perform feature mapping on the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services;

[0129] The data annotation module 13 is used to perform data annotation on the outlier features mapped by the spatial data service based on preset annotation rules;

[0130] The spatiotemporal mapping module 14 is used to perform spatiotemporal feature mapping on the annotated data and extract the distribution features of the time dimension and the space dimension;

[0131] A feature extraction module 15 is configured to obtain the first data based on the data mapped with the spatiotemporal features using a preset feature extraction algorithm;

[0132] a data preprocessing module 16 configured to perform data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensating for missing values, filling in vacant values, converting data types, removing redundant data, normalizing discrete data, and standardizing continuous data;

[0133] The data partitioning module 17 is configured to partition the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

[0134] Taking greenhouse vegetable cultivation as an example, the feature calculation module 11 monitors the hourly temperature change rate in real time and marks it when the change exceeds 5°C / hour; the feature mapping module 12 maps the physical location of the sensor to a standardized coordinate system; the data annotation module 13 classifies the degree of anomaly into three levels: "mild" (1-2 standard deviations), "moderate" (2-3 standard deviations), and "severe" (>3 standard deviations); the spatiotemporal mapping module 14 establishes a time feature map for a 24-hour period; the feature extraction module 15 extracts the statistical features of environmental parameters; the data preprocessing module 16 fills in missing data with the average value of nearby points; and finally, the data partitioning module 17 divides the training set and the validation set into a ratio of 8:2.

[0135] Preferably, the feature calculation module 11 includes:

[0136] The window setting unit 111 is used to set a preset time window size and sliding step size;

[0137] The deviation calculation unit 112 is used to calculate the deviation degree of adjacent data through the time window for the data sequence collected at any spatial collection point within a preset time period;

[0138] The threshold determination unit 113 is configured to determine abnormal value features of the collected data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by calculating the difference between adjacent data points.

[0139] In a preferred embodiment, taking field crop monitoring as an example, the window setting unit 111 sets the time window to 60 minutes and the sliding step to 15 minutes; the deviation calculation unit 112 uses the standard deviation method to calculate the degree of data deviation: D = |X-μ| / σ, where X is the current value, μ is the mean in the window, and σ is the standard deviation; the threshold judgment unit 113 sets the deviation threshold to 2.5, and if it exceeds the threshold, it will be judged as abnormal.

[0140] Preferably, the verification unit 4 includes:

[0141] A verification input module 41, configured to input the verification data into the outlier detection model;

[0142] The sample determination module 42 is used to determine whether the sample size of the verification data meets the preset minimum sample size requirement;

[0143] A result generating module 43 is configured to generate the outlier detection training result when the sample size of the verification data meets the minimum sample size requirement;

[0144] A feature recalculation module 44 is configured to trigger recalculation of outlier features when the sample size of the verification data does not meet the minimum sample size requirement;

[0145] A threshold judgment module 45 is used to judge whether the abnormal value detection training result is less than the preset detection threshold;

[0146] The return control module 46 is used to trigger a return to the model training unit 3 when the abnormal value detection training result is greater than or equal to the preset detection threshold.

[0147] In the specific embodiment of facility agriculture monitoring, the processing flow of the verification unit 4 is as follows: the verification input module 41 imports the latest 24 hours of monitoring data every day; the sample judgment module 42 ensures that the verification sample size is not less than 2000; the result generation module 43 calculates the accuracy, recall rate and other indicators of the model on the verification set; the feature recalculation module 44 triggers a new round of data collection when the sample is insufficient; the threshold judgment module 45 sets the accuracy threshold to 95%; when the accuracy rate does not meet the standard, the return control module 46 triggers model retraining.

[0148] Preferably, the outlier detection unit 7 includes:

[0149] The data receiving module 71 is used to receive the detection data collected in real time;

[0150] A feature processing module 72 is configured to perform spatiotemporal feature mapping and preprocessing on the detection data to obtain third data;

[0151] A model input module 73, configured to input the third data into the optimized outlier detection model for outlier detection;

[0152] A feature extraction module 74 is configured to extract features from the third data based on a preset feature extraction rule;

[0153] The result output module 75 is used to generate a detection result including the presence status of the abnormal value and its spatiotemporal location information.

[0154] In a specific embodiment of intelligent greenhouse environmental monitoring, the data receiving module 71 collects environmental data every 5 minutes; the feature processing module 72 normalizes the data so that all parameter values ​​are mapped to the interval [0, 1]; the model input module 73 inputs the processed feature vector into the optimized detection model; the feature extraction module 74 calculates the statistical features within the sliding window; and the result output module 75 generates a detailed report including the abnormal parameter type, the time when the abnormality occurred, and the sensor location.

[0155] The present invention achieves the following significant technical effects through the above system:

[0156] 1. Multi-level module collaboration: Through the collaborative work of seven main functional units, from spatiotemporal mapping unit 1 to outlier detection unit 7, the entire process from data collection to anomaly detection is automated, significantly improving system operating efficiency.

[0157] 2. Data processing accuracy: The three-level sub-units of the feature calculation module 11 (window setting, deviation calculation, and threshold determination) achieve accurate identification of outliers, with a detection rate of over 95%.

[0158] 3. Reliability of the verification mechanism: The six functional modules of the verification unit 4 ensure the reliability of the model performance through sample size control and multiple verifications, and the false alarm rate is reduced to below 1%.

[0159] 4. Real-time monitoring capability: The five processing modules of the outlier detection unit 7 achieve millisecond-level anomaly detection response, providing timely early warning capabilities for agricultural production.

[0160] 5. Adaptive optimization: Through the cooperation of the modification unit 5 and the model generation unit 6, the adaptive optimization of the detection model is achieved, so that the system can adapt to the monitoring needs of different agricultural production scenarios.

[0161] Through this modular, multi-level system design, the present invention not only improves the accuracy and efficiency of agricultural big data outlier detection, but also realizes the full automation and intelligence of the detection process, providing strong technical support for modern agricultural production.

[0162] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting outliers based on agricultural big data, characterized in that: The following steps are involved: Step 1: Acquire agricultural big data from a spatial data service, perform spatiotemporal mapping on the agricultural big data based on a preset spatiotemporal mapping rule, and then preprocess the spatiotemporally mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, wherein the spatiotemporal distribution features include temporal distribution features and spatial distribution features of the data. Then, the first data is processed into training data and validation data according to a preset division ratio. Step 2: inputting the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on preset feature extraction rules and a preset detection algorithm; Step 3: inputting the verification data into the outlier detection model for verification and generating a training result for outlier detection, wherein the training result is used to characterize the detection performance of the outlier detection model; Step 4: Compare the training result of the outlier detection with a preset detection threshold. If the training result is less than the preset detection threshold, proceed to step 5; otherwise, modify the detection algorithm of the outlier detection model and return to step 2. Step 5: generating an optimized outlier detection model based on the outlier detection training results; Step 6: Utilize the optimized outlier detection model to perform outlier detection on the detection data collected in real time, and generate a detection result including the time position, spatial position and anomaly type of the outlier; The step 1 specifically includes: Step 1.1, obtaining agricultural big data including time series and spatial distribution from the spatial data service; Step 1.2, calculating outlier features for the agricultural big data based on preset feature calculation rules, wherein the outlier features include mutation time features, mutation location features, and mutation type features; Step 1.3, feature mapping the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services; Step 1.4: annotate the outlier features mapped by the spatial data service based on the preset annotation rules; Step 1.5: Perform spatiotemporal feature mapping on the labeled data and extract the distribution features of the time and space dimensions; Step 1.6, obtaining the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm; Step 1.7, performing data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensating for missing values, filling in vacancies, converting data types, removing redundant data, normalizing discrete data, and standardizing continuous data; Step 1.8: Divide the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

2. The outlier detection method based on agricultural big data according to claim 1, characterized in that: The step 1.2 specifically includes: A preset time window size and sliding step size are set, and then for the collected data sequence of any spatial collection point within the preset time period, the degree of deviation of adjacent data is calculated through the time window, and the outlier characteristics of the collected data sequence are determined based on the preset deviation threshold, where the degree of deviation is calculated by the difference between adjacent data points.

3. The outlier detection method based on agricultural big data according to claim 1, characterized in that: The step 3 specifically includes: Step 3.1, inputting the verification data into the outlier detection model; Step 3.2: Determine whether the sample size of the verification data meets the preset minimum sample size requirement; Step 3.3: When the sample size of the verification data meets the minimum sample size requirement, generate the outlier detection training result; otherwise, return to step 1.2 to recalculate the outlier features; Step 3.4, determining whether the outlier detection training result is less than the preset detection threshold; Step 3.5: When the outlier detection training result is greater than or equal to the preset detection threshold, return to step 2.

4. The outlier detection method based on agricultural big data according to claim 1, characterized in that: The step 6 specifically includes: Step 6.1, receiving the detection data collected in real time; Step 6.2, performing spatiotemporal feature mapping and preprocessing on the detection data to obtain third data; Step 6.3, inputting the third data into the optimized outlier detection model to perform outlier detection; Step 6.4, performing feature extraction on the third data based on a preset feature extraction rule; Step 6.5: Generate a detection result including the presence status of the outliers and their spatiotemporal location information.

5. An outlier detection system based on agricultural big data that implements the method according to any one of claims 1 to 4, characterized in that: include: A spatiotemporal mapping unit is used to perform spatiotemporal feature mapping on the agricultural big data obtained from the spatial data service based on preset spatiotemporal mapping rules; a preprocessing unit, configured to preprocess the spatiotemporal mapped agricultural big data according to a preset feature extraction algorithm to obtain first data containing spatiotemporal distribution features, and divide the first data into training data and verification data according to a preset division ratio; A model training unit, configured to input the training data into a preset outlier detection model for training, wherein the outlier detection model is constructed based on preset feature extraction rules and a preset detection algorithm; a verification unit, configured to input the verification data into the outlier detection model for verification, and generate a training result for characterizing the detection performance of the outlier detection model; a modification unit, configured to modify the detection algorithm of the outlier detection model when the training result is greater than or equal to a preset detection threshold; A model generation unit, configured to generate an optimized outlier detection model based on the training result when the training result is less than the preset detection threshold; The outlier detection unit is used to detect the detection data collected in real time using the optimized outlier detection model and generate a detection result including the time position, spatial position and anomaly type of the outlier.

6. The outlier detection system based on agricultural big data according to claim 5 is characterized in that: The spatiotemporal mapping unit includes: A feature calculation module is used to calculate outlier features for the agricultural big data based on preset feature calculation rules, wherein the outlier features include mutation time features, mutation location features, and mutation type features; A feature mapping module, configured to perform feature mapping on the outlier features through spatial data services, wherein the spatial data services include spatial data calculation services, spatial data analysis services, and spatial data visualization services; The data annotation module is used to annotate the outlier features mapped by the spatial data service based on preset annotation rules; The spatiotemporal mapping module is used to perform spatiotemporal feature mapping on the labeled data and extract the distribution characteristics of the time and space dimensions; A feature extraction module, configured to obtain the first data based on the data after the spatiotemporal feature mapping by using a preset feature extraction algorithm; a data preprocessing module, configured to perform data preprocessing on the first data to obtain second data, wherein the data preprocessing includes compensation for missing values, filling of vacant values, conversion of data types, removal of redundant data, normalization of discrete data, and standardization of continuous data; The data partitioning module is used to partition the second data into training data according to a first preset ratio and into verification data according to a second preset ratio.

7. The outlier detection system based on agricultural big data according to claim 6 is characterized in that: The feature calculation module includes: Window setting unit, used to set the preset time window size and sliding step; a deviation calculation unit, configured to calculate the degree of deviation of adjacent data from a sequence of collected data at any spatial collection point within a preset time period through the time window; A threshold determination unit is used to determine abnormal value characteristics of the collected data sequence based on a preset deviation threshold, wherein the degree of deviation is calculated by the difference between adjacent data points.

8. The outlier detection system based on agricultural big data according to claim 5 is characterized in that: The verification unit includes: a verification input module, configured to input the verification data into the outlier detection model; A sample determination module is used to determine whether the sample size of the verification data meets the preset minimum sample size requirement; A result generating module, configured to generate the outlier detection training result when the sample size of the verification data meets the minimum sample size requirement; A feature recalculation module, configured to trigger recalculation of outlier features when the sample size of the verification data does not meet the minimum sample size requirement; A threshold judgment module is used to judge whether the outlier detection training result is less than the preset detection threshold; The return control module is used to trigger the return to the model training unit when the abnormal value detection training result is greater than or equal to the preset detection threshold.

9. The outlier detection system based on agricultural big data according to claim 5, characterized in that: The outlier detection unit includes: A data receiving module is used to receive detection data collected in real time; a feature processing module, configured to perform spatiotemporal feature mapping and preprocessing on the detection data to obtain third data; A model input module, configured to input the third data into the optimized outlier detection model for outlier detection; A feature extraction module, configured to extract features from the third data based on a preset feature extraction rule; The result output module is used to generate detection results containing the existence status of outliers and their spatiotemporal location information.

Images