Agricultural model data assimilation and automatic tuning method and system based on machine learning, electronic equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence and agricultural data processing, and provides an agricultural model data assimilation and automatic tuning method and system based on machine learning, electronic equipment and a storage medium. The method comprises the steps of original data receiving, data assimilation, original agent model and small auxiliary model construction, optimal parameter dynamic search and model prediction. According to the method, the target machine learning model approaches to the output of a traditional physical model, so that the calculation complexity is greatly reduced, and the calculation efficiency is improved; through comprehensive extraction and optimization of time, space and management features, the precision of agricultural carbon emission simulation, crop growth prediction and management measure evaluation can be effectively improved, and the model can accurately capture dynamic changes of various factors in the agricultural production process; through the small auxiliary model, the prediction result of the original agent model can be corrected and supplemented, the false alarm rate is reduced, the response capability of the model to the abnormal state is enhanced, and the overall prediction efficiency is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and agricultural data processing technology, and in particular to a method, system, electronic device and storage medium for agricultural model data assimilation and automatic optimization based on machine learning. Background Technology

[0002] Dynamic simulation and optimization of current agricultural systems have important applications in areas such as carbon emission prediction, crop growth monitoring, and farmland management optimization.

[0003] However, traditional physical models involve solving high-dimensional variables and nonlinear equations, resulting in high computational complexity that makes them difficult to meet real-time requirements. Furthermore, parameter tuning for these models often relies on human experience or exhaustive search, leading to inefficiency and difficulty adapting to dynamic changes in complex systems. Existing data assimilation methods have limited modeling capabilities when dealing with multi-source heterogeneous data and time-series data, failing to effectively capture dynamic dependencies. While deep learning technology excels in time-series modeling and capturing nonlinear relationships, current machine learning solutions have failed to combine the advantages of deep learning with physical models. The computational efficiency and generalization ability of surrogate models need improvement, and parameter tuning methods are mostly static searches, which are insufficient for real-time optimization. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method, system, electronic device, and storage medium for agricultural model data assimilation and automatic optimization based on machine learning, thereby solving the problems of high computational complexity, low accuracy of multi-factor prediction, high false alarm rate, and low overall prediction efficiency in existing technologies.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A machine learning-based method for data assimilation and automatic tuning of agricultural models, comprising:

[0007] Receive historical data and real-time input data to obtain raw data;

[0008] The original data is assimilated to obtain assimilated data; the data assimilation includes: feature classification, key feature extraction, and data normalization or standardization.

[0009] By approximating the output of a traditional physical model using a target machine learning model, an original surrogate model and a small auxiliary model are obtained; the traditional physical model includes any one of DNDC, CFD, and meteorological models.

[0010] By combining Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques, the original surrogate model is dynamically searched for optimal parameters to obtain an optimized surrogate model.

[0011] The newly collected agricultural data is input into the optimized surrogate model and the small auxiliary model for prediction, and the model prediction results are obtained.

[0012] Preferably, the original data is assimilated to obtain assimilated data, including:

[0013] The original data is processed to a uniform time scale, time series alignment is performed, and missing data is filled by interpolation to obtain preprocessed data.

[0014] The preprocessed data is then classified into discrete features and continuous features.

[0015] Extract the date features from the preprocessed data;

[0016] The discrete features, continuous features, and date features are extracted using time features, spatial features, and management measures features to obtain the assimilated data.

[0017] Preferably, the assimilated data is obtained by extracting time features, spatial features, and management measure features from the discrete features, the continuous features, and the date features, including:

[0018] The date feature is smoothed using a sliding window technique;

[0019] Extract the relative time difference between time nodes in the date feature;

[0020] The relative time difference is converted into numerical features to obtain time features.

[0021] Preferably, the assimilated data is obtained by extracting time features, spatial features, and management measure features from the discrete features, the continuous features, and the date features, including:

[0022] By combining GIS data, image feature calculations are performed on the continuous features to obtain the full-area features; the full-area features include: vegetation index, surface temperature, and humidity.

[0023] The target farmland plots are classified and coded to obtain plot codes;

[0024] Spatial features are obtained by matching the land parcel codes with the full-area features;

[0025] The discrete management measures in the discrete features are categorically One-hot encoded, and the fertilizer amount, irrigation amount, and pesticide amount in the discrete features are normalized or standardized. The agricultural operations in the discrete features are converted into time series to obtain management measure features.

[0026] The assimilated data is obtained by performing feature dimensionality reduction on the time features, the spatial features, and the management measures features.

[0027] Preferably, the output of the traditional physical model is approximated by the target machine learning model to obtain the original proxy model and a small auxiliary model, including:

[0028] The pre-collected assimilation data is input into the conventional physics model to obtain the conventional model output;

[0029] The traditional model output and the assimilated data are input into the target machine learning model for backpropagation training to obtain the original proxy model with model convergence.

[0030] Preferably, the original surrogate model is dynamically searched for optimal parameters by combining Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques to obtain an optimized surrogate model, including:

[0031] Construct a method to minimize the mean squared error; the expression for minimizing the mean squared error is:

[0032]

[0033] in, The minimum mean square error is defined as N; N is the total number of input features. The result of the target machine learning model processing the i-th input feature; This represents the processing result of the traditional physical model on the i-th input feature;

[0034] Set constraints based on the law of conservation of matter;

[0035] By combining the minimization of mean square error and the constraint of the law of conservation of mass, a loss function is obtained; the expression of the loss function is: in, The calculated value of the loss function; λ is the calculated value constrained by the law of conservation of matter; λ is the fusion coefficient.

[0036] The target machine learning model is iteratively trained using the loss function combined with Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques to obtain the optimized surrogate model.

[0037] Preferably, the process of approximating the output of a traditional physical model using a target machine learning model to obtain the original proxy model and a small auxiliary model further includes:

[0038] Construct the small auxiliary model; the small auxiliary model includes: a lightweight neural network;

[0039] Based on the classification modeling method, complex regression tasks are transformed into binary or multi-class classification problems to obtain the target classification problem; the complex regression tasks include: carbon emission prediction and crop growth prediction.

[0040] The assimilated data is input into the small auxiliary model, and the small auxiliary model is iteratively trained using the result of the target classification problem as the target output of the small auxiliary model to obtain the trained small auxiliary model.

[0041] Preferably, a machine learning-based agricultural model data assimilation and automatic tuning system includes:

[0042] The data input module is used to receive historical data and real-time input data to obtain raw data;

[0043] The data processing module is used to assimilate the original data to obtain assimilated data; the data assimilation includes: feature classification, key feature extraction, and data normalization or standardization;

[0044] The deep learning modeling module is used to approximate the output of a traditional physical model through a target machine learning model, thereby obtaining an original surrogate model and a small auxiliary model; the traditional physical model includes any one of DNDC, CFD, and meteorological models.

[0045] The parameter optimization module is used to combine Bayesian optimization, Gaussian process regression and automatic differential optimization techniques to dynamically search for the optimal parameters of the original surrogate model and obtain an optimized surrogate model.

[0046] The prediction output module is used to input newly collected agricultural data into the optimized surrogate model and the small auxiliary model for prediction, and obtain the model prediction results.

[0047] Preferably, an electronic device includes: at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the aforementioned machine learning-based agricultural model data assimilation and automatic tuning method.

[0048] Preferably, a non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the aforementioned machine learning-based agricultural model data assimilation and automatic tuning method.

[0049] The present invention discloses the following technical effects:

[0050] This invention provides a method, system, electronic device, and storage medium for data assimilation and automatic optimization of agricultural models based on machine learning. By approximating the output of traditional physical models through a target machine learning model, it solves the problem of high computational complexity in existing physical models and reduces the computational complexity of the models. Through comprehensive extraction and optimization of temporal, spatial, and management features, it solves the problem of low accuracy in multi-factor prediction in existing technologies and achieves the capture of dynamic changes in various factors during agricultural production. Through a small auxiliary model, it solves the problems of high false alarm rate and low overall prediction efficiency in existing technologies and achieves correction and supplementation of the prediction results of the original proxy model. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A schematic diagram of the data assimilation and automatic optimization process for agricultural models based on machine learning provided in an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a machine learning-based agricultural model data assimilation and automatic optimization system provided in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram illustrating the working principle of the proxy model provided in this embodiment of the invention;

[0055] Figure 4 This is a flowchart of automatic parameter tuning for agricultural models based on machine learning, provided in an embodiment of the present invention. Detailed Implementation

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The purpose of this invention is to provide a method, system, electronic device, and storage medium for data assimilation and automatic optimization of agricultural models based on machine learning, which solves the problems of high computational complexity, low accuracy of multi-factor prediction, high false alarm rate, and low overall prediction efficiency in existing technologies.

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Figure 1 This is a schematic diagram of the data assimilation and automatic tuning process for agricultural models based on machine learning, provided in an embodiment of the present invention. Figure 1 As shown, this invention provides a method for data assimilation and automatic optimization of agricultural models based on machine learning, including:

[0060] Step 100: Receive historical data and real-time input data to obtain raw data;

[0061] Step 200: Assimilate the original data to obtain assimilated data; the data assimilation includes: feature classification, key feature extraction, and data normalization or standardization;

[0062] Step 300: Approximate the output of the traditional physical model using the target machine learning model to obtain the original surrogate model and a small auxiliary model; the traditional physical model includes any one of DNDC, CFD, and meteorological models;

[0063] Step 400: Combine Bayesian optimization, Gaussian process regression and automatic differential optimization techniques to perform dynamic search for optimal parameters on the original surrogate model to obtain an optimized surrogate model;

[0064] Step 500: Input the newly collected agricultural data into the optimized surrogate model and the small auxiliary model for prediction, and obtain the model prediction results.

[0065] Further, the original data is assimilated to obtain assimilated data, including:

[0066] The original data is processed to a uniform time scale, time series alignment is performed, and missing data is filled by interpolation to obtain preprocessed data.

[0067] The preprocessed data is then classified into discrete features and continuous features.

[0068] Extract the date features from the preprocessed data;

[0069] The discrete features, continuous features, and date features are extracted using time features, spatial features, and management measures features to obtain the assimilated data.

[0070] Specifically, the discrete features, continuous features, and date features are extracted using time features, spatial features, and management measure features to obtain the assimilated data, including:

[0071] The date feature is smoothed using a sliding window technique;

[0072] Extract the relative time difference between time nodes in the date feature;

[0073] The relative time difference is converted into numerical features to obtain time features.

[0074] Further, time features, spatial features, and management measure features are extracted from the discrete features, continuous features, and date features to obtain the assimilated data, including:

[0075] By combining GIS data, image feature calculations are performed on the continuous features to obtain the full-area features; the full-area features include: vegetation index, surface temperature, and humidity.

[0076] The target farmland plots are classified and coded to obtain plot codes;

[0077] Spatial features are obtained by matching the land parcel codes with the full-area features;

[0078] The discrete management measures in the discrete features are categorically One-hot encoded, and the fertilizer amount, irrigation amount, and pesticide amount in the discrete features are normalized or standardized. The agricultural operations in the discrete features are converted into time series to obtain management measure features.

[0079] The assimilated data is obtained by performing feature dimensionality reduction on the time features, the spatial features, and the management measures features.

[0080] Specifically, by approximating the output of a traditional physics model using a target machine learning model, an original surrogate model and a small auxiliary model are obtained, including:

[0081] The pre-collected assimilation data is input into the conventional physics model to obtain the conventional model output;

[0082] The traditional model output and the assimilated data are input into the target machine learning model for backpropagation training to obtain the original proxy model with model convergence.

[0083] Furthermore, by combining Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques, the original surrogate model is dynamically searched for optimal parameters to obtain an optimized surrogate model, including:

[0084] Construct a method to minimize the mean squared error; the expression for minimizing the mean squared error is:

[0085]

[0086] in, The minimum mean square error is defined as N; N is the total number of input features. The result of the target machine learning model processing the i-th input feature; This represents the processing result of the traditional physical model on the i-th input feature;

[0087] Set constraints based on the law of conservation of matter;

[0088] By combining the minimization of mean square error and the constraint of the law of conservation of mass, a loss function is obtained; the expression of the loss function is: in, The calculated value of the loss function; λ is the calculated value constrained by the law of conservation of matter; λ is the fusion coefficient.

[0089] The target machine learning model is iteratively trained using the loss function combined with Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques to obtain the optimized surrogate model.

[0090] Specifically, by approximating the output of a traditional physics model using a target machine learning model, the original proxy model and a small auxiliary model are obtained, which also include:

[0091] Construct the small auxiliary model; the small auxiliary model includes: a lightweight neural network;

[0092] Based on the classification modeling method, complex regression tasks are transformed into binary or multi-class classification problems to obtain the target classification problem; the complex regression tasks include: carbon emission prediction and crop growth prediction.

[0093] The assimilated data is input into the small auxiliary model, and the small auxiliary model is iteratively trained using the result of the target classification problem as the target output of the small auxiliary model to obtain the trained small auxiliary model.

[0094] refer to Figure 2 A machine learning-based agricultural model data assimilation and automatic tuning system includes:

[0095] The data input module is used to receive historical data and real-time input data to obtain raw data;

[0096] The data processing module is used to assimilate the original data to obtain assimilated data; the data assimilation includes: feature classification, key feature extraction, and data normalization or standardization;

[0097] The deep learning modeling module is used to approximate the output of a traditional physical model through a target machine learning model, thereby obtaining an original surrogate model and a small auxiliary model; the traditional physical model includes any one of DNDC, CFD, and meteorological models.

[0098] The parameter optimization module is used to combine Bayesian optimization, Gaussian process regression and automatic differential optimization techniques to dynamically search for the optimal parameters of the original surrogate model and obtain an optimized surrogate model.

[0099] The prediction output module is used to input newly collected agricultural data into the optimized surrogate model and the small auxiliary model for prediction, and obtain the model prediction results.

[0100] refer to Figure 3 The efficient agricultural model data assimilation and model tuning method in this embodiment includes a data assimilation stage, a surrogate model construction stage, and a parameter tuning stage. The data assimilation stage uses a deep learning model to dynamically model historical data and real-time input, capturing dependencies in the time series and fusing multi-source heterogeneous data, including but not limited to time series, spatial data, management measure data, environmental data, text data, and image data. Deep learning models include LSTM, xLSTM, Phi-3, and PINN. Time series data includes precipitation data, temperature data, and humidity data. Spatial data includes arable land grade and soil fertility data. Management measure data includes fertilization data. Text data includes agricultural knowledge bases. Image data includes remote sensing images.

[0101] Further, the data assimilation stage includes: processing multi-source data to a unified time scale, aligning time series using sliding window technology, and interpolating to fill missing data; subsequently, feature classification and engineering optimization are performed, including one-hot encoding of crop types and management measures for discrete data, normalizing or standardizing temperature and humidity for continuous data, extracting key time node features including sowing, fertilization, and harvest dates, and enhancing environmental simulation of spatial data through GIS information. For the different data types mentioned above, continuous data is processed using feature enhancement tools such as differential coding and sliding window variance coding to obtain time-stable series data; for discrete data, after one-hot encoding, regularization based on the mean is performed according to historical distribution to obtain continuous data. After processing data from different sources in terms of time resolution and numerical type, for the remaining dimensionality misalignment, a linear projection layer is used to increase the dimensionality of categories with insufficient dimensions, finally obtaining data with unified dimensions and time resolution, which is then added element-wise to form high-quality assimilated data.

[0102] Preferably, in the surrogate model construction phase, machine learning surrogate models, including LSTM, xLSTM, Phi-3, PINN, etc., are used to approximate traditional process models, reducing computational complexity while maintaining high prediction accuracy. Small auxiliary models are introduced to classify and model specific prediction tasks, improving the model's prediction stability and efficiency for key scenarios. The small model architecture used is one or more of the following models combined with expert models: fully connected feedforward neural networks, random forest models, support vector machine models, etc. The small model uses the combined hidden layers of the final time step after receiving time-series data input from the large model as input, and is optimized using the learning algorithm corresponding to the selected architecture.

[0103] refer to Figure 4 The automatic parameter tuning stage combines Bayesian optimization, Gaussian process regression, and model optimization techniques based on automatic differentiation to dynamically search for the optimal parameter combination in a high-dimensional parameter space. It also optimizes the objective function through cross-validation, comprehensively considering prediction error and computational complexity. With continuous data input, this forms a closed loop that automatically improves model performance and adaptability. Specifically, the overall process can be divided into the following steps:

[0104] 1) The aforementioned feature engineering techniques are used to align and preprocess multimodal agricultural data to obtain unified time-series data.

[0105] 2) Select hyperparameters suitable for the real environment, input time series data into the agricultural physics model, and generate forward simulation results data.

[0106] 3) Input the forward simulation results and time series data into the surrogate model, and train it using backpropagation until the surrogate model converges.

[0107] 4) Select the target result data that needs to be optimized, and use the optimization algorithm based on automatic differentiation to optimize the surrogate model to obtain the undetermined modification values ​​of the input hyperparameters.

[0108] 5) Perform Bayesian importance sampling on the hyperparameters to be modified, and repeat the above steps until the optimization target value meets the requirements or exceeds the maximum number of iterations.

[0109] The above process can be widely applied to fields such as real-time monitoring of agricultural carbon emissions, dynamic prediction of crop growth, and simulation and optimization of agricultural production management, supporting real-time decision-making and significantly improving model performance, work efficiency, and system robustness.

[0110] Specifically, in the proxy model construction phase of the above process, this embodiment uses a hybrid machine learning proxy model to approximate the traditional process model. Its core objective is to reduce computational complexity while maintaining high prediction accuracy. This process can be formally described by the following mathematical expression:

[0111] Mathematical expression of the surrogate model approximation process model

[0112] The computational complexity of traditional process models stems from the fact that the evolution of the system state is typically described by partial differential equations (PDEs) or empirical statistical models.

[0113]

[0114] in: For the traditional physical model, X is the input variable, θ is the model parameter, and Y is the output variable.

[0115] The computational complexity of a physical model is typically:

[0116]

[0117] Where: N t The number of time steps; the physical model requires numerous iterative calculations. N S The number of spatial grid points affects computational accuracy. d represents the dimension. Due to the high-dimensional solution and nonlinear characteristics, the computational cost of this model is extremely high, making it difficult to meet real-time requirements.

[0118] A surrogate model replaces the physical model. To reduce computational complexity, a deep learning surrogate model is used to approximate the physical model.

[0119]

[0120] in: As a surrogate model, it is used to learn and approximate traditional physical models. The output response is defined as follows: φ represents the trainable parameters in the surrogate model; γ represents the set of control input parameters, which have physical meaning in the physical model and are used as inputs in the surrogate model to learn the mapping relationship between "input parameters and output response". The surrogate model does not train γ, but instead uses optimization algorithms to select or sample it to improve system performance. In this embodiment, the surrogate model uses the same input variable γ as the traditional physical model to ensure that the surrogate model can learn to approximate the physical response under the same input conditions. It should be noted that γ is only used as input in the surrogate model and is not a hyperparameter or trainable weight in the neural network structure; its role is completely different from that of φ. The trainable part of the surrogate model only includes φ, and its training objective is to minimize the error between the model's prediction and the physical model's result under the same γ and X conditions.

[0121] The computational complexity of the proxy model is:

[0122]

[0123] Where: N pThe number of parameters in the surrogate model is much smaller than the number of computational grids in the physical model. Since the surrogate model does not rely on iterative solutions to PDEs, the computational complexity is significantly reduced.

[0124] Furthermore, in order to ensure the prediction accuracy of the surrogate model, this embodiment introduces an error metric:

[0125]

[0126] In the formula, ε represents the proxy model. The error measure between the prediction results of the physical model and the actual prediction results under the same input conditions X is modeled using the Euclidean norm. This error term is not only used for subsequent model prediction performance evaluation but also serves as a key component of the loss function during the training phase, supervising the iterative update of the trainable parameters φ of the surrogate model. Combined with constraints adhering to physical laws such as the conservation of matter, a comprehensive objective function of the following form can be constructed:

[0127]

[0128] in, To comply with the physical laws such as the conservation of matter, α is a weighting adjustment coefficient.

[0129] Minimize the mean squared error (MSE):

[0130]

[0131] This embodiment optimizes the proxy model so that its output approximates the results of the physical model.

[0132] Physics-Informed Loss, within the PINN (Physics-Informed Neural Networks) framework, introduces the law of conservation of matter as a constraint in this embodiment:

[0133]

[0134] in: This is a data-driven error. The model is constrained to follow physical laws, such as carbon conservation, water conservation, and energy conservation. In addition, other constraints can be added, such as gradient 1 norm and weight 1 norm. These constraints are common and may help the model converge and prevent overfitting.

[0135] Preferably, the surrogate model has computational advantages compared to the traditional physics model: the computational complexity is reduced from... Reduce to Significantly improves computational efficiency. By capturing nonlinear relationships through deep learning, it enables rapid inference on large-scale data, meeting the needs of real-time simulation. Maintains high prediction accuracy by optimizing data assimilation and physical constraints, ensuring the model conforms to both historical observations and physical laws. The parameter tuning stage combines Bayesian optimization and Gaussian process regression with a goal-oriented optimization method based on backpropagation, dynamically searching for the optimal parameter combination in a high-dimensional parameter space and optimizing the objective function through cross-validation. Therefore, this invention automatically trains the model based on the optimization results, eliminating the need to completely traverse the large-scale input space, significantly reducing tuning time and computational resource overhead. It is particularly suitable for applications requiring rapid response or high control precision, enhancing the model's input controllability and sensitivity analysis capabilities while maintaining prediction accuracy, providing an efficient means for the refined control of complex agricultural systems.

[0136] Specifically, the data assimilation stage includes: classifying the input data into discrete and continuous features; performing one-hot encoding on discrete features and normalizing (e.g., MinMaxScaler) or standardizing (e.g., LayerNorm) on continuous features; extracting key date features such as planting date, fertilization date, and irrigation date to enhance the model's ability to capture time nodes. A sliding window technique is used to process irregular time series data, improving the model's ability to capture temporal changes.

[0137] Furthermore, the feature extraction method in this embodiment mainly covers key steps such as temporal features, spatial features, management measure features, and data preprocessing and coding, aiming to support agricultural carbon emission prediction and optimization of agricultural management measures. The specific methods are as follows: 1) Temporal feature extraction: Sliding window technique: Processes irregular time series data to ensure the model can capture dynamic changes in the agricultural production process; Time offset features: Calculates the relative time difference of key time nodes to enhance the modeling ability of the impact of specific agricultural events; Periodic features: Extracts seasonal changes, suitable for long-term trend analysis of agricultural planting cycles and environmental monitoring; Time series coding: Converts temporal information into numerical features; 2) Spatial feature extraction: GIS data fusion: Combines Geographic Information System (GIS) data, such as soil type and terrain slope, to enhance the accuracy of agricultural production environment simulation; Remote sensing data processing: Extracts features such as vegetation index (NDVI), surface temperature, and humidity to monitor the growth status of agricultural crops in real time; Regional classification features: Classifies and codes different farmland plots. This method supports regionally differentiated agricultural management and precision crop management; 3) Management measure feature processing: One-hot encoding: Classifying discrete management measures to improve the model's distinguishing ability; Continuous variable normalization: Normalizing or standardizing numerical features such as fertilizer application, irrigation, and pesticide application to avoid scale differences affecting model training; Management event serialization: Converting agricultural operations into time series inputs to capture the impact of management behavior on system state; 4) Data preprocessing and encoding: Missing value imputation: Using interpolation or machine learning methods to imput missing sensor data to ensure data integrity; Outlier detection: Using statistical methods or deep learning to identify abnormal data to improve data reliability; High-dimensional feature dimensionality reduction: For multi-source heterogeneous data, using dimensionality reduction methods such as PCA and t-SNE to reduce feature redundancy and improve computational efficiency. This feature extraction method focuses on the agricultural field and can effectively improve the accuracy of agricultural carbon emission simulation, crop growth prediction, and management measure evaluation. Through comprehensive extraction and optimization of time, space, and management features, it ensures that the model can accurately capture the dynamic changes of various factors in the agricultural production process, thereby achieving precise management of intelligent agricultural production.

[0138] Optionally, a small auxiliary model can be introduced to optimize the prediction task, including:

[0139] A lightweight auxiliary model is constructed using a small LSTM model or a lightweight neural network, taking the multi-source heterogeneous data output from the data assimilation stage as input. The input data includes historical carbon emission data, real-time meteorological data, sensor monitoring data, and management measure data.

[0140] The auxiliary model uses a classification modeling approach to transform some complex regression prediction tasks into binary or multi-class problems, including but not limited to predicting whether it is a high carbon emission day, whether crops have reached the harvest period, and whether equipment has potential malfunctions.

[0141] The classification results of the auxiliary model are used to correct and supplement the prediction results of the main agent model, reduce the false alarm rate, enhance the model's ability to respond to abnormal states, and improve the overall prediction efficiency, stability and system robustness.

[0142] Specifically, this embodiment is applicable to the simulation and optimization tasks of the following complex systems:

[0143] Agricultural carbon emission simulation and prediction: 1) Dynamic monitoring of carbon emissions: Real-time capture of daily carbon dioxide and methane emissions during agricultural production using multi-source heterogeneous data (such as meteorological data, soil type, crop type, etc.). 2) Optimization of carbon reduction measures: By simulating the impact of different management measures, help farmers and agricultural managers develop low-carbon production strategies to achieve carbon neutrality goals.

[0144] Crop growth dynamic simulation: 1) Crop growth status prediction: Combining meteorological data, soil properties, and management measures, dynamically simulate the crop growth process from sowing to maturity to provide accurate yield prediction. 2) Farmland environment monitoring: Utilize remote sensing data to monitor crop growth in real time, identify pest and disease risks, and optimize management measures.

[0145] Agricultural management measures assessment and optimization: 1) Determine the optimal fertilization amount and timing by simulating the impact of different fertilization schemes on crop yield and carbon emissions. 2) Irrigation strategy optimization: Dynamically adjust irrigation frequency and amount based on crop water requirements and meteorological conditions to improve water resource utilization efficiency. 3) Planting structure optimization: Optimize planting structure based on climate conditions and soil type to improve land use efficiency and agricultural production returns.

[0146] Furthermore, the data input module supports receiving data streams in various formats via a standardized API interface, including CSV files, JSON data streams, and real-time data from IoT sensors; the data processing module includes a feature classification module, a key feature extraction module, and a normalization module; the deep learning modeling module integrates LSTM, xLSTM, and Phi-3 models, and can dynamically switch or combine models according to different tasks; the parameter optimization module combines Bayesian optimization and Gaussian process regression techniques to adjust the weights of the objective function in real time to optimize computational performance.

[0147] Optionally, the system features a modular architecture, supporting the following functional extensions: This embodiment supports adding new deep learning models via plugins to adapt to the dynamic characteristics and multi-source data fusion in the agricultural field. The model library is expandable, allowing users to flexibly select or integrate multiple models according to specific agricultural production tasks, improving the accuracy of carbon emission prediction, crop growth monitoring, and agricultural management optimization. Feature extraction algorithms can be adapted to agricultural production needs, supporting crop type identification, soil type classification, meteorological data fusion, and remote sensing image processing. Through the integration of GIS data, remote sensing data, and field sensor data, the system can dynamically capture the spatiotemporal changes in agricultural production, ensuring high accuracy and robustness of the model in complex agricultural scenarios. This embodiment also supports a multi-task learning mode, enabling joint modeling for different agricultural objectives. The multi-task learning framework facilitates information sharing between tasks, enhancing the model's generalization ability.

[0148] Optionally, the system in this embodiment supports automated training and deployment, including: automated data preprocessing and feature extraction; automated training and dynamic updating of deep learning models; and deployment of models through a microservice architecture to support real-time prediction and dynamic optimization.

[0149] Furthermore, the automated training and deployment process can be broken down into the following steps:

[0150] Automated data processing workflow: The system presets data cleaning rules and feature extraction strategies, automatically identifies the discrete and continuous characteristics of agricultural data, performs corresponding encoding, normalization and missing value filling operations, and automatically synchronizes key agricultural time nodes and spatial information;

[0151] Model training and update mechanism: This system has an automatic model retraining function, which can automatically trigger the model retraining task when data changes, the deviation between the surrogate model's optimized prediction and the physical model's simulation results is large, or the scene is adjusted. During the iteration process of the surrogate model, a model update strategy based on GIT-LSF version management is used to ensure that the historical optimized model is saved. Before the new model is launched, the historical model and the candidate new model are tested to ensure the model's traceability and performance improvement.

[0152] Model Deployment and Service-Oriented Architecture: Models are deployed as independently callable service interfaces using MLOps services such as MLflow. The model version number is used as the service version number, supporting containerized deployment and horizontal scaling, thus improving system deployment flexibility and operational stability. The model service supports high-concurrency request processing and dynamic parameter injection to meet the needs of multi-task parallel prediction, and also supports version-based online model testing.

[0153] Online optimization support: The system can combine Bayesian optimization and backpropagation optimization mechanisms to receive business feedback information in real time, dynamically adjust model input or parameter settings, realize online adaptive optimization, and improve prediction response capability and application flexibility;

[0154] Visualization and monitoring capabilities: The system provides a visualization interface for the model training process and prediction results, displaying model performance indicators, input and output trends, and model stability in real time. It also integrates model anomaly detection and logging mechanisms to ensure system maintainability and operational security.

[0155] Specifically, feature engineering comprises two core parts: classification and encoding, and temporal feature extraction. First, discrete features of the input data are encoded using one-hot encoding to enhance their expressive power, enabling the model to more accurately distinguish between different categories of features. Continuous features are normalized or standardized to eliminate dimensional differences between features, thereby improving the model's training efficiency and convergence speed. Second, to further improve the accuracy of time series modeling, key time node features are extracted. These features reflect the impact of important events on system behavior, capture the dynamic changes of data over time, and provide the model with more targeted input features.

[0156] Furthermore, time series modeling employs deep learning models to dynamically model time series data and capture the dependencies between features. LSTM is suitable for modeling short-term dependencies, while xLSTM, with its enhanced memory capabilities, is better suited for capturing long-term dependencies. The Phi-3 model further improves modeling efficiency by adjusting the embedding layers to handle large-scale input data. The model input consists of historical features and management measures data, and the output is the target variable for the next time step.

[0157] Preferably, the assimilated data has broad application value, laying the foundation for the dynamic simulation and optimization of complex agricultural systems. In the agricultural field, assimilated data can provide accurate daily-scale dynamic predictions of carbon emissions, supporting the optimization of low-carbon management measures in agricultural production, including dynamic adjustments to strategies such as fertilization and irrigation. Through accurate carbon emission predictions, agricultural producers can flexibly adjust management measures according to climate and soil conditions to achieve efficient and low-carbon agricultural production. Furthermore, the data assimilation results can also be used for the dynamic monitoring and prediction of crop growth status. Based on multi-source data assimilation, the model can capture key parameters of crop growth in real time, providing scientific planting decision support for agricultural managers. By dynamically simulating the crop growth process, the system can predict crop yield, identify growth anomalies, and provide management suggestions. In irrigation management, the data assimilation results can support the optimization of dynamic irrigation strategies based on crop water requirements and meteorological conditions, ensuring that crops receive sufficient water at each stage of growth while avoiding water waste caused by over-irrigation. In fertilization management, the model can dynamically adjust the amount and timing of fertilizer application according to crop growth status and soil nutrient levels, achieving precision fertilization. Through the aforementioned functions, the data assimilation-based agricultural model not only enhances the level of intelligence in agricultural production but also effectively supports the development of green agriculture and the achievement of carbon emission reduction targets.

[0158] Specifically, depending on the needs of the specific scenario, a single model or a combination of models can be selected for optimization and application. For example, an LSTM model can be used to capture time series features; an improved xLSTM model can be used to enhance long-term memory capabilities and avoid the forgetting problem of traditional LSTM; and the Phi-3 model can be used to adjust the embedding layers to adapt to large-scale continuous input data and improve the model's computational efficiency. Specifically, to enhance the robustness and generalization ability of the model in complex environments, this invention can adopt a multi-model fusion strategy to construct a combined model architecture according to actual application needs. This includes, but is not limited to, the following methods: Model ensemble: Multiple models such as LSTM, xLSTM, and Phi-3 are trained independently in parallel. The prediction results of multiple models are fused through an ensemble strategy. Options include simple averaging, weighted averaging, or dynamic weighting based on task relevance to improve overall prediction stability. Expert hybrid mechanism (MoE): The system can construct multiple dedicated sub-models. A gating network dynamically assigns weights to different sub-models based on input features, enabling automatic adaptation of the model to different data regions or tasks, strengthening the model's ability to express heterogeneous data structures. Uncertainty-driven fusion strategy: Weighted fusion is performed based on the uncertainty of the same input and output results from each sub-model. Higher weights are assigned to the outputs of models with lower uncertainty, improving prediction reliability. This strategy can be combined with Bayesian models or Monte Carlo Dropout methods to achieve uncertainty estimation. The above combined strategies can be flexibly configured according to the system deployment scenario, and the final prediction results are fused through a unified output interface. With the integration of an automated training and deployment framework, each sub-model can be updated independently, supporting modular management and iterative optimization.

[0159] Preferably, in resource-constrained scenarios, to improve prediction efficiency and reduce false alarm rates, this embodiment optimizes the prediction task by introducing a small auxiliary model. Specifically, this involves constructing a lightweight auxiliary model that takes multi-source heterogeneous data output from the data assimilation stage as input. The input data includes historical carbon emission data, real-time meteorological data, sensor monitoring data, and management measure data. For specific prediction targets, the auxiliary model employs a classification modeling approach, transforming some complex regression tasks into binary or multi-class classification problems, such as predicting whether it is a high-emission day for carbon emissions, whether crops have reached harvest time, or whether equipment has potential malfunctions.

[0160] As an optional implementation, this embodiment also provides an electronic device, including: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the aforementioned machine learning-based agricultural model data assimilation and automatic tuning method.

[0161] As an optional implementation, this embodiment also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the aforementioned machine learning-based agricultural model data assimilation and automatic tuning method.

[0162] The beneficial effects of the present invention are as follows:

[0163] This invention approximates the output of traditional physical models using a target machine learning model, significantly reducing computational complexity and improving computational efficiency. Through comprehensive extraction and optimization of temporal, spatial, and management features, it effectively improves the accuracy of agricultural carbon emission simulation, crop growth prediction, and management measure evaluation. The model can accurately capture the dynamic changes of various factors in agricultural production. Through a small auxiliary model, it can correct and supplement the prediction results of the original proxy model, reducing the false alarm rate, enhancing the model's ability to respond to abnormal states, and improving the overall prediction efficiency.

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0165] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for data assimilation and automatic optimization of agricultural models based on machine learning, characterized in that, include: Receive historical data and real-time input data to obtain raw data; The original data is assimilated to obtain assimilated data; The data assimilation includes: feature classification, key feature extraction, and data normalization or standardization; By approximating the output of a traditional physical model using a target machine learning model, an original surrogate model and a small auxiliary model are obtained; the traditional physical model includes any one of DNDC, CFD, and meteorological models. By combining Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques, the original surrogate model is dynamically searched for optimal parameters to obtain an optimized surrogate model. The newly collected agricultural data is input into the optimized surrogate model and the small auxiliary model for prediction, and the model prediction results are obtained.

2. The method for data assimilation and automatic optimization of agricultural models based on machine learning according to claim 1, characterized in that, The original data is assimilated to obtain assimilated data, including: The original data is processed to a uniform time scale, time series alignment is performed, and missing data is filled by interpolation to obtain preprocessed data. The preprocessed data is then classified into discrete features and continuous features. Extract the date features from the preprocessed data; The discrete features, continuous features, and date features are extracted using time features, spatial features, and management measures features to obtain the assimilated data.

3. The method for data assimilation and automatic optimization of agricultural models based on machine learning according to claim 2, characterized in that, The discrete features, continuous features, and date features are subjected to time feature extraction, spatial feature extraction, and management measure feature extraction to obtain the assimilated data, including: The date feature is smoothed using a sliding window technique; Extract the relative time difference between time nodes in the date feature; The relative time difference is converted into numerical features to obtain time features.

4. The method for data assimilation and automatic optimization of agricultural models based on machine learning according to claim 3, characterized in that, The discrete features, continuous features, and date features are subjected to time feature extraction, spatial feature extraction, and management measure feature extraction to obtain the assimilated data, including: By combining GIS data, image feature calculations are performed on the continuous features to obtain the full-area features; the full-area features include: vegetation index, surface temperature, and humidity. The target farmland plots are classified and coded to obtain plot codes; Spatial features are obtained by matching the land parcel codes with the full-area features; The discrete management measures in the discrete features are categorically One-hot encoded, and the fertilizer amount, irrigation amount, and pesticide amount in the discrete features are normalized or standardized. The agricultural operations in the discrete features are converted into time series to obtain management measure features. The assimilated data is obtained by performing feature dimensionality reduction on the time features, the spatial features, and the management measures features.

5. The method for data assimilation and automatic optimization of agricultural models based on machine learning according to claim 4, characterized in that, By approximating the output of a traditional physics model using a target machine learning model, we obtain the original proxy model and a small auxiliary model, including: The pre-collected assimilation data is input into the conventional physics model to obtain the conventional model output; The traditional model output and the assimilated data are input into the target machine learning model for backpropagation training to obtain the original proxy model with model convergence.

6. The method for data assimilation and automatic optimization of agricultural models based on machine learning according to claim 5, characterized in that, By combining Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques, the original surrogate model is dynamically searched for optimal parameters to obtain an optimized surrogate model, including: Construct a method to minimize the mean squared error; the expression for minimizing the mean squared error is: in, The minimum mean square error is defined as N; N is the total number of input features. The result of the target machine learning model processing the i-th input feature; This represents the processing result of the traditional physical model on the i-th input feature; Set constraints based on the law of conservation of matter; By combining the minimization of mean square error and the constraint of the law of conservation of mass, a loss function is obtained; the expression of the loss function is: in, The calculated value of the loss function; λ is the calculated value constrained by the law of conservation of matter; λ is the fusion coefficient. The target machine learning model is iteratively trained using the loss function combined with Bayesian optimization, Gaussian process regression, and automatic differential optimization techniques to obtain the optimized surrogate model.

7. The method for data assimilation and automatic optimization of agricultural models based on machine learning according to claim 6, characterized in that, By approximating the output of a traditional physics model using a target machine learning model, we obtain the original proxy model and a small auxiliary model, which also include: Construct the small auxiliary model; the small auxiliary model includes: a lightweight neural network; Based on the classification modeling method, complex regression tasks are transformed into binary or multi-class classification problems to obtain the target classification problem; the complex regression tasks include: carbon emission prediction and crop growth prediction. The assimilated data is input into the small auxiliary model, and the small auxiliary model is iteratively trained using the result of the target classification problem as the target output of the small auxiliary model to obtain the trained small auxiliary model.

8. A machine learning-based agricultural model data assimilation and automatic optimization system, characterized in that, include: The data input module is used to receive historical data and real-time input data to obtain raw data; The data processing module is used to assimilate the original data to obtain assimilated data; the data assimilation includes: feature classification, key feature extraction, and data normalization or standardization; The deep learning modeling module is used to approximate the output of a traditional physical model through a target machine learning model, thereby obtaining an original surrogate model and a small auxiliary model; the traditional physical model includes any one of DNDC, CFD, and meteorological models. The parameter optimization module is used to combine Bayesian optimization, Gaussian process regression and automatic differential optimization techniques to dynamically search for the optimal parameters of the original surrogate model and obtain an optimized surrogate model. The prediction output module is used to input newly collected agricultural data into the optimized surrogate model and the small auxiliary model for prediction, and obtain the model prediction results.

9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform a machine learning-based agricultural model data assimilation and automatic tuning method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method for data assimilation and automatic optimization of agricultural models based on machine learning, as described in any one of claims 1 to 7.

Images