Crop yield prediction method based on multi-source real-time data and model fusion
By constructing a real-time processing and model fusion system for multi-source heterogeneous data, the problems of data integration and nonlinear relationship capture in crop yield forecasting were solved, achieving high-precision and stable yield forecasting and providing real-time decision support.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-03-13
AI Technical Summary
Existing crop yield prediction models lack systematic integration of multi-source heterogeneous data, are unable to capture complex nonlinear relationships and interaction effects among multiple factors, and lack real-time data processing and complete agricultural decision support capabilities.
By capturing multi-source heterogeneous data in real time using Flink CDC, a cluster of linear regression models and nonlinear models are constructed. K-fold cross-validation is used to train the base learner, and a stacking strategy is combined for model fusion. A meta-learner is used for secondary learning to achieve real-time processing and dynamic prediction of multi-source heterogeneous data.
It significantly improves the accuracy and stability of crop yield forecasting, enables timely response to environmental changes, and provides precise decision support for agricultural production.
Smart Images
Figure CN121660183A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of smart agriculture, and in particular to a method for predicting crop yield based on the fusion of multi-source real-time data and models. Background Technology
[0002] With the intensification of global climate change and the increasing complexity of agricultural production, accurate prediction of crop yield is of great significance for ensuring food security and optimizing the allocation of agricultural resources. The final yield of crops is a comprehensive reflection of the combined effects of multiple factors, including genetic characteristics, environmental factors, and field management measures. These factors include climate conditions, soil characteristics, vegetation growth status, geographical location, and sowing and harvesting time. Therefore, establishing a quantitative correlation model between multi-dimensional characteristics and crop yield is a core technological challenge for achieving precision agriculture and intelligent decision-making.
[0003] In existing technologies, the prediction and analysis of crop yields typically suffers from the following technical limitations: First, traditional yield prediction models tend to focus on single-type data analysis, such as only meteorological or soil factors, and lack systematic integration of multi-source heterogeneous data. These models are difficult to effectively integrate multi-dimensional features such as climate data, soil data, vegetation indices, geographic information, pesticide usage, and sowing and harvesting times, resulting in incomplete model information and limited prediction accuracy.
[0004] Secondly, in terms of model construction, existing studies mostly adopt a single machine learning algorithm or a simple model combination strategy; linear regression models cannot capture the complex nonlinear relationships and interaction effects among multiple factors; although a single tree model can handle nonlinear relationships, it is prone to overfitting and has poor stability; and simple model averaging or fixed weight fusion strategies cannot adaptively optimize the contribution of different models, making it difficult to give full play to the complementary advantages of each algorithm.
[0005] Finally, existing technical solutions generally lack comprehensive agricultural decision support capabilities; most systems only provide yield forecasts, lacking functions such as comprehensive evaluation of model performance, visualization and comparison of forecast results, feature importance analysis, and residual diagnosis, making it difficult to provide agricultural producers with intuitive and reliable decision-making basis. Furthermore, existing solutions are mostly based on static analysis of historical data, lacking support for real-time data stream processing and dynamic updates. Summary of the Invention
[0006] This invention provides a crop yield prediction method based on the fusion of multi-source real-time data and models, in order to overcome the problems of lack of systematic integration of multi-source heterogeneous data, inability to capture complex nonlinear relationships and interaction effects among multiple factors, and lack of complete agricultural decision support capabilities.
[0007] To achieve the above objectives, the technical solution of this invention is: a method for predicting crop yield based on the fusion of multi-source real-time data and models, comprising: S1. Capture the changed data of the source database in real time through Flink CDC and synchronize it to the Kafka message group to obtain the real-time data stream; based on the real-time data stream, obtain multi-source heterogeneous data and perform preprocessing to obtain preprocessed multi-source heterogeneous data. S2. Based on the preprocessed multi-source heterogeneous data, a structured dataset is obtained, and then the features of the structured dataset are obtained, and the feature matrix of the structured dataset is constructed. S3. Construct a cluster of linear regression models and obtain the optimal linear regression model from the cluster of linear regression models; S4. Both the optimal linear regression model and the nonlinear model are used as base learners; based on the feature matrix of the structured dataset, the base learners are trained using K-fold cross-validation to obtain the trained base learners, and the combined prediction results of the trained base learners are obtained; the nonlinear model includes random forest and gradient boosting tree models. S5. Use the prediction results of the trained base learner as the meta-feature matrix; introduce the meta-learner, and obtain the crop yield prediction results based on the meta-feature matrix and the meta-learner.
[0008] Furthermore, the steps to obtain the optimal linear regression model include: S41. Based on the linear regression model, ridge regression model, Lasso regression model, and linear grid regression model, construct the following cluster of linear regression models:
[0009] In the formula, This is a cluster of linear regression models; It is a linear regression model; Ridge regression model; For Lasso regression model; It is a linear grid regression model; S42. Based on the linear regression model family, the optimal linear regression model is obtained through cross-validation; the expression for the cross-validation method is:
[0010] In the formula, It is the optimal linear model; For models in the linear regression model family The average obtained through cross-validation ; The coefficient of determination.
[0011] Furthermore, based on the feature matrix of the structured dataset, the base learner is trained using K-fold cross-validation to obtain the trained base learner and its prediction results. The method used is as follows: S21. Using K-fold cross-validation, the feature matrix of the structured dataset is divided into a subset of K folds. S22. Sequentially obtain the subsets of the kth, k=1, ... Kth folds as the validation set, and the subsets of the remaining K-1 folds as the training set; train the base learner using the training set to obtain the trained base learner; based on the validation set and the trained base learner, obtain the predicted value of the subset of the kth fold. S23. Sequentially concatenate the predicted values corresponding to the subsets of the kth, k=1, ..., Kth folds to obtain the prediction results of the trained base learner, namely the prediction results of the trained optimal linear regression model, the prediction results of the trained random forest model, and the prediction results of the trained gradient boosting tree model; combine the prediction results of the trained optimal linear regression model, the prediction results of the trained random forest model, and the prediction results of the trained gradient boosting tree model into a prediction matrix to obtain the combined prediction results of the trained base learner.
[0012] Furthermore, the expression for the prediction result of the trained optimal linear regression model is as follows:
[0013] In the formula, The optimal linear regression model is used to predict crop yield. All of these are the optimal linear regression model weights obtained through training. All of these are independent variable features in the feature matrix of a structured dataset; n This represents the total number of independent variable features in the feature matrix of the structured dataset. The prediction result expression of the trained random forest model is as follows:
[0014] In the formula, To predict crop yields using a random forest model; This represents the number of trees in the random forest model. For the index of the tree; For the first Trees for input samples The prediction results; The input sample is the feature matrix of the structured dataset; The expression for the prediction result of the trained gradient boosting tree model is:
[0015] In the formula, For the gradient boosting tree model Forecasted crop yields; For the gradient boosting tree model Forecasted crop yields; The learning rate; For the first Predicted values for the tree rings; For the iteration round index, and ; When the maximum number of iterations is reached At that time, the first The predicted yield of each crop cycle is used as the final predicted yield, that is... .
[0016] Furthermore, the meta-learner is a ridge regression model, and the expression for the ridge regression model is:
[0017] In the formula, The final predicted yield for Stacking fusion; For ridge regression function; The prediction results of the trained gradient boosting tree model.
[0018] Furthermore, in step S1, multi-source heterogeneous data is obtained and preprocessed, the preprocessing including: IQR method is used to identify and remove outliers in numerical features of rainfall, temperature, and soil moisture. Extract the temporal features of the sowing date and harvest date to obtain the sowing month, sowing year-days, harvest date, and harvest year-days. Label and encode crop type and regional classification variables; The interaction characteristics between temperature and rainfall are obtained by multiplying temperature and rainfall. Fill in missing values for numerical variables using the median; Based on the correlation between crop yield and characteristics, q characteristics with the highest correlation to crop yield are selected; the correlation between crop yield and characteristics is calculated using the Pearson correlation coefficient.
[0019] Beneficial Effects: This invention provides a crop yield prediction method based on multi-source real-time data and model fusion. By constructing a base learner combination of "optimized linear model clusters + supplementary nonlinear models," it comprehensively utilizes the advantages of linear models in capturing clear patterns and nonlinear models (such as random forests and gradient boosting trees) in mining complex nonlinear relationships. Furthermore, by employing a stacking strategy, the prediction results of each base learner are used as meta-features. Through secondary learning and fusion using meta-learners, the prediction capabilities of multiple models can be effectively integrated, reducing the bias and variance of a single model, thereby significantly improving the accuracy and stability of the final yield prediction. By integrating Flink CDC and Kafka message queues, real-time capture and streaming processing of changes in multi-source heterogeneous data (such as meteorological, soil, remote sensing, and agricultural records) are achieved. This overcomes the prediction lag problem of traditional data based on static or periodic updates, and can respond promptly to changes in the environment and crop growth status, providing a solid data foundation for dynamic and accurate yield prediction. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart of the crop yield prediction method of the present invention; Figure 2 This is a performance comparison chart of each model in the embodiments of the present invention; Figure 3 This is a comparison between the actual output and the predicted output of the basic model in this embodiment of the invention; Figure 4 This is a comparison chart of the prediction performance of the Stacking fusion model and the optimal base model in an embodiment of the present invention; Figure 5 This is a residual analysis diagram for all models in the embodiments of the present invention; Figure 6 This is a heatmap showing the relationship between features and yield in an embodiment of the present invention; Figure 7 This is a comparison chart of the output of each model in the embodiments of the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] This embodiment provides a method for predicting crop yield based on the fusion of multi-source real-time data and models, such as... Figure 1 As shown, it includes: S1. Capture the changed data of the source database in real time through Flink CDC and synchronize it to the Kafka message group to obtain the real-time data stream; based on the real-time data stream, obtain multi-source heterogeneous data and perform preprocessing to obtain preprocessed multi-source heterogeneous data. S2. Based on the preprocessed multi-source heterogeneous data, a structured dataset is obtained, and then the features of the structured dataset are obtained, and the feature matrix of the structured dataset is constructed. S3. Construct a cluster of linear regression models and obtain the optimal linear regression model from the cluster of linear regression models; S4. Both the optimal linear regression model and the nonlinear model are used as base learners; based on the feature matrix of the structured dataset, the base learners are trained using K-fold cross-validation to obtain the trained base learners, and the combined prediction results of the trained base learners are obtained; the nonlinear model includes random forest and gradient boosting tree models. S5. Use the prediction results of the trained base learner as the meta-feature matrix; introduce the meta-learner, and obtain the crop yield prediction results based on the meta-feature matrix and the meta-learner.
[0024] Specifically, the source database is a relational database that stores crop growth environment data, agricultural records, and historical yield data; The multi-source heterogeneous data includes: meteorological data, soil data, remote sensing data, and agricultural records data.
[0025] Specifically, the steps for constructing the feature matrix of a structured dataset are as follows: S21. Align and fuse the preprocessed multi-source heterogeneous data according to a unified timestamp and spatial representation to obtain fused data; S22. Based on the fused data, construct a structured dataset; wherein the structured dataset is a two-dimensional structure, where rows of the two-dimensional structure represent samples and columns of the two-dimensional structure represent feature variables; S23. Extract the feature variables of the structured dataset and stack them according to the samples to obtain the feature matrix of the structured dataset; The features of the structured dataset include: rainfall (mm), temperature (°C), humidity (%), sunshine hours, temperature-rainfall interaction term, soil moisture (%), soil pH, NDVI index, latitude, longitude, pesticide usage (ml), sowing month, sowing year-days, harvest day, and harvest year-days.
[0026] Preferably, the steps to obtain the optimal linear regression model include: S41. Based on the linear regression model, ridge regression model, Lasso regression model, and linear grid regression model, construct the following cluster of linear regression models:
[0027] In the formula, This is a cluster of linear regression models; It is a linear regression model; Ridge regression model; For Lasso regression model; It is a linear grid regression model; S42. Based on the linear regression model family, the optimal linear regression model is obtained through cross-validation; the expression for the cross-validation method is:
[0028] In the formula, It is the optimal linear model; For models in the linear regression model family The average obtained through cross-validation ; The coefficient of determination.
[0029] Preferably, based on the feature matrix of the structured dataset, the base learner is trained using K-fold cross-validation to obtain the trained base learner and its prediction results. The method used is as follows: S21. Using K-fold cross-validation, the feature matrix of the structured dataset is divided into a subset of K folds. S22. Sequentially obtain the subsets of the kth, k=1, ... Kth folds as the validation set, and the subsets of the remaining K-1 folds as the training set; train the base learner using the training set to obtain the trained base learner; based on the validation set and the trained base learner, obtain the predicted value of the subset of the kth fold. S23. Sequentially concatenate the predicted values corresponding to the subsets of the kth, k=1, ..., Kth folds to obtain the prediction results of the trained base learner, namely the prediction results of the trained optimal linear regression model, the prediction results of the trained random forest model, and the prediction results of the trained gradient boosting tree model; combine the prediction results of the trained optimal linear regression model, the prediction results of the trained random forest model, and the prediction results of the trained gradient boosting tree model into a prediction matrix to obtain the combined prediction results of the trained base learner.
[0030] 4. The crop yield prediction method based on multi-source real-time data and model fusion according to claim 3, characterized in that the expression for the prediction result of the trained optimal linear regression model is:
[0031] In the formula, The optimal linear regression model is used to predict crop yield. All of these are the optimal linear regression model weights obtained through training. All of these are independent variable features in the feature matrix of a structured dataset; n This represents the total number of independent variable features in the feature matrix of the structured dataset. The prediction result expression of the trained random forest model is as follows:
[0032] In the formula, To predict crop yields using a random forest model; This represents the number of trees in the random forest model. For the index of the tree; For the first Trees for input samples The prediction results; The input sample is the feature matrix of the structured dataset; The expression for the prediction result of the trained gradient boosting tree model is:
[0033] In the formula, For the gradient boosting tree model Forecasted crop yields; For the gradient boosting tree model Forecasted crop yields; The learning rate; For the first Predicted values for the tree rings; For the iteration round index, and ; When the maximum number of iterations is reached At that time, the first The predicted yield of each crop cycle is used as the final predicted yield, that is... .
[0034] Preferably, the meta-learner is a ridge regression model, and the expression of the ridge regression model is:
[0035] In the formula, The final predicted yield for Stacking fusion; For ridge regression function; The prediction results of the trained gradient boosting tree model.
[0036] Preferably, in step S1, multi-source heterogeneous data is obtained and preprocessed, the preprocessing including: IQR method is used to identify and remove outliers in numerical features of rainfall, temperature, and soil moisture. Extract the temporal features of the sowing date and harvest date to obtain the sowing month, sowing year-days, harvest date, and harvest year-days. Label and encode crop type and regional classification variables; The interaction characteristics between temperature and rainfall are obtained by multiplying temperature and rainfall. Fill in missing values for numerical variables using the median; Based on the correlation between crop yield and features, q features with the highest correlation to crop yield are selected; the correlation between crop yield and features is calculated using the Pearson correlation coefficient, wherein the features are those in the structured dataset that can directly affect crop yield; in this embodiment, q is set to 10.
[0037] In this embodiment, the final yield prediction result, the intermediate prediction results of each base learner, and the performance indicators of each model are output to a visual decision support interface. This interface uses model performance comparison charts, prediction effect comparison charts, multi-dimensional feature correlation heatmaps, and residual analysis charts to comprehensively analyze the impact of climate, soil, vegetation, geography, management, and time factors on crop yield, providing a basis for decision-making in precision agriculture. The images displayed on the visualization interface are as follows: Figures 2 to 6 As shown.
[0038] In a specific embodiment, the crop yield prediction method of this embodiment is compared and verified with the average fusion model and the weighted fusion model, and the results are as follows: Figure 7As shown, the crop yield prediction method in this embodiment demonstrates the best practical application value, with a prediction result matching the actual yield value by as much as 98.9%, which is significantly better than the average voting fusion method (82.5%) and weighted voting fusion (92.2%).
[0039] The present invention has the following beneficial effects: This invention discloses a crop yield prediction method based on multi-source real-time data and model fusion. By constructing a base learner combination of "optimized linear model clusters + supplementary nonlinear models", it comprehensively utilizes the advantages of linear models in capturing clear patterns and nonlinear models (such as random forests and gradient boosting trees) in mining complex nonlinear relationships. Furthermore, by adopting a stacking strategy, the prediction results of each base learner are used as meta-features, and secondary learning and fusion are performed through meta-learners. This effectively integrates the prediction capabilities of multiple models, reduces the bias and variance of a single model, and thus significantly improves the accuracy and stability of the final yield prediction. By integrating Flink CDC and Kafka message queues, real-time capture and streaming processing of changes in multi-source heterogeneous data (such as meteorological, soil, remote sensing, and agricultural records) are achieved. This overcomes the prediction lag problem of traditional data based on static or periodic updates, and can respond promptly to changes in the environment and crop growth status, providing a solid data foundation for dynamic and accurate yield prediction.
[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for predicting crop yield based on the fusion of multi-source real-time data and models, characterized in that, include: S1. Capture changes in the source database in real time using Flink CDC and synchronize them to the Kafka message group to obtain a real-time data stream; Based on the real-time data stream, multi-source heterogeneous data is obtained and preprocessed to obtain preprocessed multi-source heterogeneous data. S2. Based on the preprocessed multi-source heterogeneous data, a structured dataset is obtained, and then the features of the structured dataset are obtained, and the feature matrix of the structured dataset is constructed. S3. Construct a cluster of linear regression models and obtain the optimal linear regression model from the cluster of linear regression models; S4. Both the optimal linear regression model and the nonlinear model are used as base learners; based on the feature matrix of the structured dataset, the base learners are trained using K-fold cross-validation to obtain the trained base learners, and the combined prediction results of the trained base learners are obtained; the nonlinear model includes random forest and gradient boosting tree models. S5. Use the prediction results of the trained base learner as the meta-feature matrix; introduce the meta-learner, and obtain the crop yield prediction results based on the meta-feature matrix and the meta-learner.
2. The crop yield prediction method based on multi-source real-time data and model fusion according to claim 1, characterized in that, The steps to obtain the optimal linear regression model include: S41. Based on the linear regression model, ridge regression model, Lasso regression model, and linear grid regression model, construct the following cluster of linear regression models: In the formula, This is a cluster of linear regression models; It is a linear regression model; Ridge regression model; For Lasso regression model; It is a linear grid regression model; S42. Based on the linear regression model family, the optimal linear regression model is obtained through cross-validation; the expression for the cross-validation method is: In the formula, It is the optimal linear model; For models in the linear regression model family The average obtained through cross-validation ; The coefficient of determination.
3. The crop yield prediction method based on multi-source real-time data and model fusion according to claim 1, characterized in that, Based on the feature matrix of the structured dataset, the base learner is trained using K-fold cross-validation to obtain the trained base learner and its prediction results. The method used is as follows: S21. Using K-fold cross-validation, the feature matrix of the structured dataset is divided into a subset of K folds. S22. Sequentially obtain the subsets of the kth, k=1, ... Kth folds as the validation set, and the subsets of the remaining K-1 folds as the training set; train the base learner using the training set to obtain the trained base learner; based on the validation set and the trained base learner, obtain the predicted value of the subset of the kth fold. S23. Sequentially concatenate the predicted values corresponding to the subsets of the kth, k=1, ..., Kth folds to obtain the prediction results of the trained base learner, namely the prediction results of the trained optimal linear regression model, the prediction results of the trained random forest model, and the prediction results of the trained gradient boosting tree model; combine the prediction results of the trained optimal linear regression model, the prediction results of the trained random forest model, and the prediction results of the trained gradient boosting tree model into a prediction matrix to obtain the combined prediction results of the trained base learner.
4. The crop yield prediction method based on multi-source real-time data and model fusion according to claim 3, characterized in that, The expression for the prediction result of the trained optimal linear regression model is as follows: In the formula, The optimal linear regression model is used to predict crop yield. All of these are the optimal linear regression model weights obtained through training. All of these are independent variable features in the feature matrix of a structured dataset; n This represents the total number of independent variable features in the feature matrix of the structured dataset. The prediction result expression of the trained random forest model is as follows: In the formula, To predict crop yields using a random forest model; This represents the number of trees in the random forest model. For the index of the tree; For the first Trees for input samples The prediction results; The input sample is the feature matrix of the structured dataset; The expression for the prediction result of the trained gradient boosting tree model is: In the formula, For the gradient boosting tree model Forecasted crop yields; For the gradient boosting tree model Forecasted crop yields; The learning rate; For the first Predicted values for the tree rings; For the iteration round index, and ; When the maximum number of iterations is reached At that time, the first The predicted yield of each crop cycle is used as the final predicted yield, that is... .
5. The crop yield prediction method based on multi-source real-time data and model fusion according to claim 3, characterized in that, The meta-learner is a ridge regression model, and the expression for the ridge regression model is: In the formula, The final predicted yield for Stacking fusion; For ridge regression function; The prediction results of the trained gradient boosting tree model.
6. The crop yield prediction method based on multi-source real-time data and model fusion according to claim 1, characterized in that, Step S1 involves obtaining multi-source heterogeneous data and performing preprocessing, the preprocessing including: IQR method is used to identify and remove outliers in numerical features of rainfall, temperature, and soil moisture. Extract the temporal features of the sowing date and harvest date to obtain the sowing month, sowing year-days, harvest date, and harvest year-days. Label and encode the crop type and regional classification variables; The interaction characteristics between temperature and rainfall are obtained by multiplying temperature and rainfall. Fill in missing values for numerical variables using the median; Based on the correlation between crop yield and characteristics, q characteristics with the highest correlation to crop yield are selected; the correlation between crop yield and characteristics is calculated using the Pearson correlation coefficient.