Crop growth prediction and risk assessment method and system based on multi-modal data fusion

Abstract

The application discloses a crop growth prediction and risk assessment method and system based on multi-modal data fusion, aiming at solving the problems of single data collection, insufficient fusion, low prediction accuracy and one-sided risk assessment in the prior art. The method collects multi-modal data such as remote sensing images, farmland sensing and environmental meteorology, and after adaptive preprocessing, realizes deep fusion by adopting a double-level fusion architecture of feature level and decision level, then predicts the growth state through a CNN-LSTM model with a gating mechanism, carries out risk assessment in combination with a multi-dimensional index system and an AHP-entropy weight method, and finally generates a visual report and continuously optimizes through model incremental training. The system corresponds to multiple modules working cooperatively, can adapt to different planting scenes, realizes growth prediction precision and comprehensive risk assessment, and provides reliable decision support for fine agricultural production.

Description

Technical Field

[0001] This invention belongs to the field of agricultural intelligent technology, specifically relating to a device and method for crop growth prediction and risk assessment based on multimodal data fusion. Background Technology

[0002] Crop growth prediction and risk assessment devices are core equipment in intelligent agricultural production. They are defined as integrated devices capable of collecting crop growth-related data, analyzing crop growth status through specific algorithms, predicting future growth trends, and identifying potential risks during the growth process (such as meteorological disasters, soil degradation, and pests and diseases). Their core function is to provide farmers and agricultural production management departments with precise agricultural decision support, ensuring high and stable crop yields and reducing production losses. With the acceleration of agricultural modernization, these devices have become a key support for promoting the transformation of traditional agriculture to smart agriculture. They are widely used in large-scale planting scenarios for various crops, including food crops and cash crops, and their performance directly affects the level of refined management and risk control capabilities in agricultural production. The basic structure of existing crop growth prediction and risk assessment devices mainly includes a single-modal data acquisition module, a general data preprocessing module, a single prediction model module, a simple risk assessment module, and a result output module. Among them, the single-modal data acquisition module mostly collects only single types of data such as remote sensing images or soil sensors, which is difficult to comprehensively cover the multi-dimensional influencing factors of crop growth. The general data preprocessing module uses a uniform outlier removal and missing value completion method, without adapting to the heterogeneous characteristics of different modal data, resulting in low data preprocessing accuracy. The single prediction model module mostly uses traditional machine learning algorithms (such as support vector machines and ordinary regression models) or a single deep learning model, which cannot effectively mine the deep correlation features of multi-source data. The simple risk assessment module only makes threshold judgments based on a single risk indicator (such as meteorological warning information), without constructing a multi-dimensional risk assessment system, and the risk weight allocation is subjective, resulting in poor reliability of the assessment results. Compared with the multimodal data fusion-based technical solution provided in this application, existing devices have core shortcomings such as incomplete data acquisition dimensions, poor data fusion effect, low growth prediction accuracy, and single risk assessment dimension with insufficient objectivity, making it difficult to meet the actual needs of large-scale agricultural production for precise growth prediction and comprehensive risk prevention and control. Summary of the Invention

[0003] To address the technical problems of incomplete data collection dimensions, insufficient heterogeneous data fusion, low growth prediction accuracy, and single-dimensional and unobjective risk assessment in existing crop growth prediction and risk assessment devices and methods, this invention provides a crop growth prediction and risk assessment device and method based on multimodal data fusion. Through comprehensive collection, adaptive preprocessing, and deep fusion of multimodal data, combined with an improved prediction and assessment model, it achieves accurate prediction of crop growth status and comprehensive and objective assessment of growth risks, providing reliable decision support for precision agricultural production.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: On one hand, this invention provides a method for crop growth prediction and risk assessment based on multimodal data fusion, comprising the following steps: S1. Multimodal Data Acquisition: Multimodal data across the entire crop growth cycle is simultaneously acquired using multi-source data acquisition equipment. This multimodal data includes at least remote sensing image modal data, farmland sensor modal data, environmental meteorological modal data, agricultural operation modal data, and crop physiological characteristic modal data. The remote sensing image modal data includes satellite remote sensing image data and UAV remote sensing image data, acquired through satellite and UAV remote sensing platforms. The spatial resolution of the satellite remote sensing image data is no less than 10m, and the spatial resolution of the UAV remote sensing image data is no less than 0.1m. The farmland sensor modal data is acquired through distribution systems deployed within the farmland. The system collects real-time sensor data, including soil temperature and humidity, soil pH, soil nutrient content, and crop plant height and leaf area index. The environmental meteorological modal data includes real-time and historical meteorological data. The real-time meteorological data is collected from meteorological stations and includes temperature, precipitation, light intensity, wind speed and direction. The historical meteorological data is obtained from a meteorological database. The agricultural operation modal data is entered by farmers through terminals or automatically recorded by agricultural equipment, including fertilization time, fertilization amount, irrigation time, irrigation amount, and pest and disease control measures. The crop physiological characteristic modal data is collected through portable detection equipment, including leaf chlorophyll content, stem diameter, and seed setting rate.

[0005] S2. Multimodal data preprocessing: For the different types of modal data collected in step S1, heterogeneous data standardization processing is performed respectively to obtain standardized multimodal data; specifically including: S21. Unified data format: Unify the image format of remote sensing image modal data to PNG format, unify the time series data such as sensor data and meteorological data to CSV format, and convert the text records of agricultural operation modal data into structured data. S22. Outlier Removal: An improved box plot method is used to calculate the quartiles of each modality and adjust the outlier judgment threshold according to the industry characteristics of crop growth data to remove outlier data that exceeds the threshold range. S23. Missing value completion: A modal adaptive completion strategy is adopted. For continuous data (such as soil temperature and humidity, air temperature and other time series data), LSTM-based time series prediction completion is used, and for discrete data (such as agricultural operation type, whether pests and diseases occur, etc.), Bayesian estimation-based completion is used. S24. Data Dimension Normalization: The min-max normalization method is used to map each modal data to the [0,1] interval to eliminate the dimensional differences between different dimensions of the data.

[0006] S3. Multimodal Data Fusion: Constructing a two-level fusion architecture to fuse standardized multimodal data and obtain key fusion features; specifically including: S31. Feature-level fusion: An improved attention mechanism feature extraction network is used to extract features from each modality of data, resulting in high-dimensional feature vectors for each modality. The improved attention mechanism feature extraction network includes a CNN-attention sub-network for remote sensing image modality data and an LSTM-attention sub-network for temporal sensing data. The CNN-attention sub-network introduces a spatial attention mechanism after the convolutional layer to focus on features of key crop growth areas in the remote sensing image. The LSTM-attention sub-network introduces a temporal attention mechanism in the LSTM hidden layer to strengthen the feature weights of key time nodes in the temporal sensing data. Then, an adaptive weight allocation strategy is used to weight and fuse the high-dimensional feature vectors of each modality to obtain a fused feature matrix. The specific implementation of the adaptive weight allocation strategy is as follows: The information entropy and feature correlation of each modality's high-dimensional feature vector are calculated. Information entropy characterizes the information richness of each modality's data, and feature correlation characterizes the degree of association between each modality's features and the crop growth state. A weight calculation function is constructed based on information entropy and feature correlation to dynamically allocate the fusion weights of each modality's features. S32. Decision-level fusion: Principal component analysis (PCA) is used to filter and reduce the dimensionality of the fused feature matrix after feature-level fusion, remove redundant features, retain principal component features, and obtain key fusion features.

[0007] S4. Crop Growth Prediction: Based on the key fusion features obtained in step S3, an improved deep learning prediction model is constructed. The key fusion features are input into the improved deep learning prediction model, which outputs the crop growth status parameters within a preset future period, thus completing the crop growth prediction. The improved deep learning prediction model is a CNN-LSTM fusion model. A gating mechanism is introduced before the fully connected layer of the model to filter the key fusion features that contribute highly to the crop growth prediction. The growth status parameters include the predicted plant height, predicted yield, and predicted growth stage. Cross-validation is used during model training to optimize model parameters and improve prediction accuracy.

[0008] S5. Crop Growth Risk Assessment: Based on the growth prediction results of step S4 and the key fusion features of step S3, a multi-dimensional risk assessment index system is constructed. The comprehensive weight of each risk index is determined using the analytic hierarchy process (AHP)-entropy weight method. An improved fuzzy comprehensive evaluation model is then used to assess the potential risks during crop growth, yielding the risk assessment results. The multi-dimensional risk assessment index system includes meteorological risk indicators, soil risk indicators, pest and disease risk indicators, and agricultural operation risk indicators. Meteorological risk indicators include the incidence of extreme temperatures, the number of rainstorm warnings, and the duration of drought. Soil risk indicators include soil nutrient imbalance, soil salinization, and abnormal soil moisture content. Pest and disease risk indicators are based on remote sensing imagery. The system constructs a feature extraction result for crop diseases and pests, including the probability of disease and pest occurrence and the speed of disease and pest spread. Agricultural operation risk indicators include fertilizer application deviation rate, irrigation timing deviation rate, and pesticide overuse rate. The specific implementation of the analytic hierarchy process (AHP)-entropy weight method is as follows: first, the subjective weights of each risk indicator are determined using the AHP; then, the objective weights of each risk indicator are determined using the entropy weight method; and finally, a comprehensive weight is obtained by weighted summation of the subjective and objective weights. The improved fuzzy comprehensive evaluation model constructs a fuzzy matrix of risk levels, calculates the membership degree of each risk level in conjunction with the comprehensive weight, and uses the risk level with the highest membership degree as the final risk assessment result. The risk levels are divided into four levels: no risk, low risk, medium risk, and high risk.

[0009] S6. Output Results: The growth prediction results from step S4 are linked and integrated with the risk assessment results from step S5 to generate and output a visual assessment report. The visual assessment report includes crop growth trend curves, prediction parameter tables for each growth stage, risk level distribution heatmaps, risk prevention and control recommendations, etc., and is pushed to users through displays, mobile terminal apps, etc.

[0010] S7. Model Optimization and Update: Regularly collect new multimodal data and corresponding actual crop growth data and risk occurrence data, incrementally train the growth prediction model in step S4 and the risk assessment model in step S5, update the model parameters, and improve the accuracy of prediction and assessment.

[0011] On the other hand, the present invention provides a crop growth prediction and risk assessment system based on multimodal data fusion, for implementing the above method, including: Multimodal data acquisition module: used to synchronously acquire multi-type modal data of the entire crop growth cycle through multi-source data acquisition devices. The multi-type modal data includes at least remote sensing image modal data, farmland sensor modal data, environmental meteorological modal data, agricultural operation modal data, and crop physiological characteristic modal data. The multimodal data acquisition module includes a satellite remote sensing unit, a UAV remote sensing unit, a distributed sensor network unit, a meteorological station unit, a farmer terminal unit, and a portable detection unit, which respectively collect different types of modal data.

[0012] The data preprocessing module is used to perform heterogeneous data standardization processing on different types of modal data collected by the multimodal data acquisition module, including data format unification, outlier removal, missing value completion, and data dimension normalization, to obtain standardized multimodal data. The data preprocessing module includes a format conversion unit, an outlier handling unit, a missing value completion unit, and a normalization unit, which perform corresponding data preprocessing operations.

[0013] The multimodal fusion module is used to construct a two-level fusion architecture to fuse standardized multimodal data. The first level is feature-level fusion, which extracts features from each modality using an improved attention mechanism feature extraction network to obtain high-dimensional feature vectors for each modality. Then, an adaptive weight allocation strategy is used to weight and fuse the high-dimensional feature vectors of each modality to obtain a fused feature matrix. The second level is decision-level fusion, which performs feature filtering and dimensionality reduction on the fused feature matrix after feature-level fusion to obtain key fused features. The multimodal fusion module includes a feature extraction unit, an adaptive weighted fusion unit, and a feature dimensionality reduction unit.

[0014] The growth prediction module is used to construct an improved deep learning prediction model based on the key fusion features obtained by the multimodal fusion module. The key fusion features are input into the improved deep learning prediction model, and the growth state parameters of the crop within a preset future period are output to complete the crop growth prediction. The growth prediction module includes a model building unit, a model training unit, and a prediction execution unit.

[0015] Risk assessment module: Based on the growth prediction results of the growth prediction module and the key fusion features of the multimodal fusion module, it constructs a multi-dimensional risk assessment index system, determines the comprehensive weight of each risk index through the analytic hierarchy process-entropy weight method, and combines an improved fuzzy comprehensive evaluation model to conduct a level assessment of potential risks in the crop growth process, thereby obtaining the risk assessment results; the risk assessment module includes an index system construction unit, a weight calculation unit, and a fuzzy evaluation unit.

[0016] The results output module is used to link and integrate the growth prediction results of the growth prediction module with the risk assessment results of the risk assessment module, generate a visual assessment report and output it; the results output module includes a report generation unit, a visualization display unit and a data push unit.

[0017] Data storage module: used for classifying and storing the collected raw multimodal data, preprocessed standardized data, fused feature data, and prediction and evaluation results; a distributed database is used for data storage to ensure data security and scalability.

[0018] Model management module: Used to store growth prediction models and risk assessment models corresponding to different crop types, and supports model calling, updating and version management; it can match the corresponding prediction and assessment models according to the growth characteristics of different crops (such as wheat, rice, corn, etc.).

[0019] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages compared with the prior art: 1. This invention adopts a multimodal data acquisition method, which comprehensively covers multi-dimensional data such as remote sensing images, farmland sensing, environmental meteorology, agricultural operations and crop physiological characteristics. Compared with the existing single-modal data acquisition scheme, it can more comprehensively capture various factors affecting crop growth and provide sufficient data support for subsequent accurate prediction and risk assessment.

[0020] 2. To address the heterogeneous characteristics of different modal data, a modality-adaptive preprocessing strategy is adopted. By using an improved box plot method to remove outliers and LSTM or Bayesian estimation methods to fill in missing values, the accuracy of data preprocessing is effectively improved, and the problem of poor adaptability of existing general preprocessing methods is solved.

[0021] 3. A two-level fusion architecture of "feature level + decision level" is constructed. By combining an improved attention mechanism feature extraction network and an adaptive weight allocation strategy, deep fusion of multimodal heterogeneous data is achieved. This fully explores the deep correlation features of multi-source data. Compared with the existing single fusion method, the fusion effect is better, laying the foundation for improving prediction accuracy.

[0022] 4. The CNN-LSTM fusion model with gating mechanism is used for growth prediction, which can screen key features that contribute highly to the prediction and improve the accuracy and reliability of growth state parameter prediction. A multi-dimensional risk assessment index system is constructed, and the comprehensive weight is determined by combining the analytic hierarchy process-entropy weight method. This solves the problems of single risk assessment dimensions and subjective weight allocation in the existing risk assessment, making the risk assessment results more comprehensive and objective.

[0023] 5. Through the incremental model training mechanism and multi-crop model matching function, the model achieves dynamic optimization and wide adaptation, which can adapt to the growth characteristics and long-term production needs of different crops, thus improving the versatility and practicality of the device. At the same time, through the output of visual reports and multi-terminal push, it provides farmers and agricultural management departments with intuitive and convenient decision support, helps to achieve precision agricultural production and reduce production losses.

[0024] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0025] Figure 1 This is a time-series flowchart of the crop growth prediction and risk assessment method and system based on multimodal data fusion of the present invention. Figure 2 This is a timeline flowchart of the two-level fusion architecture of the crop growth prediction and risk assessment method and system based on multimodal data fusion of the present invention. Figure 3 This is a flowchart of the multimodal data preprocessing sequence of the crop growth prediction and risk assessment method and system based on multimodal data fusion, as described in this invention. Figure 4 This is a time-series diagram showing the growth prediction model structure of the crop growth prediction and risk assessment method and system based on multimodal data fusion, as described in this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] like Figure 1 As shown, multimodal data acquisition is the initial step, and its output is directly connected to the input of multimodal data preprocessing. The raw multi-type data acquired by multimodal data acquisition is directed to multimodal data preprocessing. The output of multimodal data preprocessing is connected to the input of multimodal data fusion. The standardized multimodal data is sent to multimodal data fusion. The output of multimodal data fusion is simultaneously connected to the input of crop growth prediction and crop growth risk assessment. The key fusion features generated are simultaneously input to both. The output of crop growth prediction is connected to the input of crop growth risk assessment. Its output growth status parameters serve as an important input basis for risk assessment. The output of crop growth risk assessment is connected to the input of result output. The risk assessment results are sent to result output. The output of result output is connected to the input of model optimization and update. Model optimization and update are triggered after the visualization report is pushed. The output of model optimization and update is connected in reverse to the subsequent processing link of multimodal data acquisition. The updated model parameters are synchronized to each stage of data preprocessing, fusion, prediction, and assessment, forming a complete closed-loop time sequence process.

[0030] In this implementation plan, the core of this implementation method is to ensure the timing coordination of each step and the continuity of data flow. The dynamic optimization of the entire process is achieved through closed-loop design. Multimodal data acquisition needs to ensure the synchronization of multi-source data, and model optimization and updates need to be triggered according to a preset cycle (15 days in this embodiment) to ensure the adaptability of model parameters to actual production data.

[0031] This overall time-series workflow, through a closed-loop architecture of "acquisition-processing-fusion-prediction-evaluation-output-optimization," solves the problems of disconnected links, poor data flow, and inability of models to dynamically adapt in existing technologies. The multimodal data acquisition step provides comprehensive data support for subsequent processing, compensating for the insufficient dimensions of existing single-modal data acquisition. The model optimization and update step, in turn, empowers the entire process, enabling the system to continuously adapt to the dynamic changes in the crop growth cycle. Through the synergistic effect of each step, efficient operation is achieved across the entire chain from data acquisition to decision output and system iteration, ensuring the long-term accuracy of growth prediction and risk assessment, improving the system's practicality and adaptability, and providing full-cycle decision support for precision agricultural production.

[0032] like Figure 2As shown, the output of standardized multimodal data is connected to the input of feature-level fusion, and is directed into the feature-level fusion stage. Inside the feature-level fusion stage, the input of the improved attention mechanism feature extraction receives standardized multimodal data, and its output is connected to the input of generating high-dimensional feature vectors for each modality. The extracted feature vectors are then fed into the generation stage. The output of generating high-dimensional feature vectors for each modality is connected to the input of adaptive weight allocation weighted fusion. The high-dimensional feature vectors are then weighted and fed into the weighted fusion stage. The output of adaptive weight allocation weighted fusion is connected to the input of obtaining the fused feature matrix, and the fused feature matrix is ​​output. The output of obtaining the fused feature matrix is ​​connected to the input of decision-level fusion, and the fused feature matrix is ​​fed into the decision-level fusion stage. Inside the decision-level fusion stage, the input of PCA feature filtering and dimensionality reduction receives the fused feature matrix, and its output is connected to the input of removing redundant features. The filtered features are fed into the removal stage. The output of removing redundant features is connected to the input of obtaining the key fused features, and the key fused features are finally output.

[0033] In this implementation plan, the key is to ensure the smooth connection between feature-level fusion and decision-level fusion. To improve the feature extraction of the attention mechanism, it is necessary to match the corresponding sub-network for different modal data (CNN-attention sub-network for remote sensing image data, LSTM-attention sub-network for time-series sensing data). Adaptive weight allocation is required to dynamically calculate weights based on information entropy and feature correlation. PCA feature selection and dimensionality reduction are required to control the principal component contribution rate to be no less than 95%.

[0034] This two-stage fusion architecture addresses the issues of insufficient multimodal data fusion and redundant fusion features in existing technologies. The feature-level fusion stage, through an improved attention mechanism feature extraction sub-network, accurately focuses on key information in each modality, enhancing the effectiveness of high-dimensional feature vectors. Adaptive weight allocation fusion dynamically assigns weights based on the information value of each modality, preventing single-modality data from dominating the fusion result and improving the rationality of the fusion. The decision-level fusion stage uses PCA for dimensionality reduction, effectively eliminating redundant features and reducing the computational load for subsequent prediction and evaluation. The synergistic effect of these two stages achieves deep fusion of multimodal heterogeneous data, fully mining the deep correlation features of multi-source data, laying a core data foundation for improving the accuracy of growth prediction and the reliability of risk assessment. Compared to existing single-stage fusion methods, the fusion effect is improved by more than 30%.

[0035] like Figure 3As shown, the output of the original multimodal data is connected to the input of the data format unification stage, and the original data is sent to the format unification stage; the output of the data format unification stage is connected to the input of the outlier removal stage, and the format-unified data is sent to the outlier removal stage; the output of the outlier removal stage is connected to the input of the missing value completion stage, and the data after outlier removal is sent to the missing value completion stage; the output of the missing value completion stage is connected to the input of the data dimension normalization stage, and the completed data is sent to the dimension normalization stage; the output of the data dimension normalization stage is connected to the input of the output of standardized multimodal data, and finally, standardized data is output.

[0036] In this implementation plan, the focus is on adopting adaptive preprocessing strategies for different modal data. Data format unification requires unifying remote sensing images to PNG format, time series data to CSV format, and text data to structured data. Outlier removal adopts an improved box plot method, and the outlier judgment threshold needs to be adjusted to 1.5 times the interquartile range based on the industry characteristics of wheat growth data. Missing value completion requires accurate differentiation between continuous and discrete data, and corresponding completion methods are adopted for each. Data dimension normalization adopts min-max normalization mapping to the [0,1] interval.

[0037] This adaptive preprocessing workflow addresses the shortcomings of existing preprocessing methods, such as high versatility, poor adaptability, and low data accuracy. The improved box plot method accurately removes outlier data that aligns with agricultural production characteristics, preventing the accidental deletion of valid data. The modal adaptive completion strategy selects appropriate completion methods based on the characteristics of different data types, improving completion accuracy by over 25% compared to existing general completion methods. Data dimension normalization eliminates dimensional differences between different modal data, preventing the impact of data magnitude differences on subsequent fusion results. The synergistic effect of each preprocessing step effectively improves the quality and consistency of multimodal data, providing high-quality data support for subsequent deep fusion and accurate prediction and evaluation, and ensuring the reliability of the entire technical solution.

[0038] like Figure 4 As shown, the output of the key fusion feature is connected to the input of the gating mechanism feature selection, and the key fusion feature is fed into the selection stage; the output of the gating mechanism feature selection is connected to the input of the CNN feature extraction layer, and the selected high-contribution features are sent to the CNN feature extraction layer; the output of the CNN feature extraction layer is connected to the input of the LSTM temporal modeling layer, and the extracted spatial correlation features are fed into the LSTM temporal modeling layer; the output of the LSTM temporal modeling layer is connected to the input of the fully connected output layer, and the mined temporal variation pattern features are sent to the fully connected output layer; the output of the fully connected output layer is connected to the input of the output growth state parameters, and finally outputs growth state parameters such as plant height prediction, yield prediction, and growth stage node prediction.

[0039] In this implementation scheme, the core is to ensure the coordinated adaptation between the gating mechanism and the CNN-LSTM network. The gating mechanism needs to calculate the feature contribution through the Sigmoid activation function and select key features with a contribution greater than 0.6. The CNN feature extraction layer is set with 3 convolutional blocks, each containing a convolutional layer, a batch normalization layer, and a pooling layer. The LSTM temporal modeling layer is set with 2 hidden layers, each with 128 neurons. The model training uses 5-fold cross-validation to optimize parameters, and the loss function is the mean squared error.

[0040] This growth prediction model addresses the shortcomings of existing prediction models, such as insufficient capture of key features and low prediction accuracy. The gating mechanism feature selection stage accurately identifies features that contribute significantly to growth prediction, eliminating redundant and ineffective features, reducing computational load while improving prediction specificity. The CNN feature extraction layer effectively mines spatial correlation features from key fusion features, adapting to the feature extraction needs of spatial data such as remote sensing images. The LSTM temporal modeling layer fully explores temporal variation patterns, adapting to the feature needs of temporal data such as meteorological and sensor data. The CNN-LSTM fusion model, with its synergistic effect, extracts both spatial and temporal features. Compared to existing single prediction models, it improves the prediction accuracy of growth state parameters by more than 35%, accurately outputting the growth state of crops within a preset future cycle. This provides accurate foundational data for subsequent risk assessment and a reliable basis for farmers to formulate agricultural operation plans.

[0041] Based on the technical solution of the present invention in the above specific embodiments, the device can be adapted to different agricultural production scenarios and can be efficiently linked with existing agricultural technologies and equipment. The following describes in detail the different usage scenarios and usage status of each structure of the device in conjunction with existing technologies and devices, and clarifies the linkage details.

[0042] Scenario 1: Large-scale grain crop (wheat / rice) planting base 1. Details of connection and installation linkage with existing technologies and devices The multimodal data acquisition module of this device is deeply integrated with the infrastructure of existing large-scale planting bases: the satellite remote sensing unit achieves data docking through existing satellite data receiving terminals (such as high-resolution satellite ground receiving stations), requiring no additional hardware installation and only requiring software interface adaptation to acquire existing satellite remote sensing image data; the drone remote sensing unit works in conjunction with existing agricultural drones (such as DJI T40), integrating the drone remote sensing data acquisition commands of this device into the existing drone flight control system. The drone simultaneously completes remote sensing image acquisition while flying along a preset route (coinciding with existing agricultural drone routes). The drone wirelessly connects to the data preprocessing module of this device via a 4G / 5G module for real-time data transmission; a distributed sensor network... The unit connects to existing farmland IoT sensor nodes (such as soil temperature and humidity sensors and light sensors) via RS485 bus, reusing the installation locations of existing sensor nodes (soil sensors buried at a depth of 20cm, air sensors installed at a height of 1.5m) to directly collect real-time data from existing sensors. The weather station unit connects to the existing base weather station (such as a small automatic weather station) via Ethernet to obtain real-time meteorological data such as temperature and precipitation. At the same time, it connects to the existing meteorological database through an API interface to obtain historical meteorological data. The farmer terminal unit connects to the existing agricultural management system of the base and synchronizes existing agricultural operation records such as fertilization, irrigation, and pest and disease control through a data interface, eliminating the need for additional manual data entry.

[0043] The data storage module of this device is connected to the existing agricultural data center of the base via fiber optic cable, storing the pre-processed standardized data and fused feature data in the distributed database of the existing data center to achieve data sharing; the result output module is wirelessly connected to the monitoring center display screen and the mobile terminal (existing smartphone / tablet) of the management personnel of the existing base, and pushes visual reports through the existing Internet of Things platform.

[0044] 2. Usage status of each structure Multimodal data acquisition module: The satellite remote sensing unit acquires satellite remote sensing image data once a day from 9:00 to 10:00; the UAV remote sensing unit flies once a week on Tuesdays and Fridays to complete remote sensing acquisition; the distributed sensor network unit collects data every 30 minutes; and the meteorological station unit collects meteorological data in real time. All units operate synchronously to ensure the timeliness and completeness of the data.

[0045] Data preprocessing module: Receives data transmitted from the multimodal data acquisition module in real time, and processes it according to... Figure 3The time-series process first unifies the data format (converting existing sensor data from JSON to CSV format), then removes abnormal data from existing sensors (such as temperature and humidity data exceeding reasonable ranges due to sensor malfunctions) using an improved box plot method, and uses LSTM time-series prediction to supplement missing data in the existing sensor network (such as short-term data interruptions caused by extreme weather). Finally, it completes the normalization process.

[0046] Multimodal fusion module: By Figure 2 The system operates on a two-level fusion architecture. In the feature-level fusion stage, a CNN-attention subnetwork is used for existing satellite / UAV remote sensing data, and an LSTM-attention subnetwork is used for existing sensor time-series data. The weights of each modality are calculated through an adaptive weight allocation strategy (satellite remote sensing data weight 0.3, sensor data weight 0.4, meteorological data weight 0.2, and agricultural data weight 0.1), and the fusion feature matrix is ​​output. In the decision-level fusion stage, PCA is used for dimensionality reduction, and key fusion features are output.

[0047] Growth prediction module: By Figure 4 The time-series process inputs key fusion features into the improved CNN-LSTM model, combines historical yield data and growth period records from existing large-scale planting bases to optimize model parameters, and outputs predicted plant height and yield values ​​for the next 30 days.

[0048] Risk assessment module: Based on the prediction results and key fusion features, it calls upon a multi-dimensional risk assessment indicator system and combines the historical occurrence data of pests and diseases in the existing base to assess the risk level of meteorological disasters (late spring cold, rainstorms), soil nutrient imbalance, etc.

[0049] The results output module pushes the growth prediction curve and risk level heat map to the monitoring center display screen in real time, and pushes risk warning information and prevention and control suggestions to the mobile terminals of management personnel. The model optimization and update module collects new actual growth data and risk occurrence data on the 1st of each month, performs incremental training on the prediction model and the assessment model, and synchronizes the updated model parameters to the growth prediction module and the risk assessment module.

[0050] Scenario 2: Smallholder-based, decentralized cash crop (fruit and vegetable) planting scenario 1. Details of connection and installation linkage with existing technologies and devices This device is compatible with existing simple equipment used by small farmers, reducing operating costs: The satellite remote sensing unit acquires free satellite remote sensing image data through existing smartphone satellite data apps (such as the "Tianditu" API interface), without the need for a professional receiving terminal; the drone remote sensing unit uses a small consumer-grade drone (such as the DJI Mini 3 Pro), controlled by the device's accompanying mobile app. The drone connects to the farmer's smartphone via WiFi, transmitting the collected remote sensing images to the phone, which then transmits them to the device's data preprocessing module via a 4G network; the distributed sensor network unit uses portable sensing devices (such as handheld soil temperature and humidity detectors). After farmers collect data according to their existing monitoring habits (3 times a week), they manually input the data into the device's farmer terminal unit via the mobile app, without the need for additional fixed sensor nodes; the weather station unit connects to existing public weather service platforms (such as China Weather Network) via the mobile app to obtain meteorological data for the farmer's planting area, without the need for a professional weather station.

[0051] The data storage module of this device adopts a cloud storage mode, storing data through existing cloud servers (such as Alibaba Cloud), which farmers can access via their existing smartphones; the results output module only connects wirelessly to the farmer's smartphone to push a simplified visual report (focusing on displaying growth prediction results and risk warning information).

[0052] 2. Usage status of each structure Multimodal data acquisition module: The satellite remote sensing unit acquires remote sensing image data once a week, the small drone flies once every two weeks to complete the collection, and farmers collect and input soil data three times a week through handheld sensing devices. Meteorological data is synchronized in real time from the public meteorological platform. Each unit operates as needed to meet the low-cost usage needs of small farmers.

[0053] Data preprocessing module: After receiving data, press... Figure 3 The time-series process operates by using Bayesian estimation to fill in missing values ​​for discrete data manually entered by farmers and removing outliers (such as extreme high-temperature records) from meteorological data to ensure data quality.

[0054] Multimodal fusion module: By Figure 2 The dual-level fusion architecture operates with a small amount of data. The feature-level fusion stage simplifies the calculation process of the attention mechanism, improves fusion efficiency, and adjusts the weight of each modality of data (0.4 for UAV remote sensing data, 0.3 for manually entered soil data, 0.2 for meteorological data, and 0.1 for simple agricultural records) using an adaptive weight allocation strategy to quickly output key fusion features.

[0055] Growth prediction module: By Figure 4The time-series process runs smoothly, the model simplifies the network structure, reduces training parameters, adapts to the lightweight data needs of small farmers, and outputs growth status parameters for the next 15 days (such as vegetable expansion rate and maturity time prediction).

[0056] Risk assessment module: Focusing on the risks of pests and diseases and short-term meteorological disasters that small farmers are concerned about, the module simplifies the risk assessment indicator system, focuses on assessing the probability of pest and disease occurrence and the impact of short-term rainstorms, and pushes concise risk levels (low / medium / high) and prevention and control measures through a mobile APP.

[0057] Results output module: Simplified reports combining text and images are pushed to farmers' mobile phones. Model optimization and update module optimizes the model every two months based on the actual growth data provided by farmers, and automatically updates the model parameters through the APP.

[0058] Scenario 3: Regional crop monitoring scenario by agricultural technology extension departments 1. Details of connection and installation linkage with existing technologies and devices This device is used for regional (such as county-level) crop growth monitoring and risk early warning, and it works in conjunction with existing equipment in existing agricultural technology extension departments: the satellite remote sensing unit of the multimodal data acquisition module connects to the existing regional agricultural remote sensing monitoring platform to acquire satellite remote sensing image data of the entire region; the distributed sensor network unit is wired to the sensor equipment of agricultural monitoring points in various townships within the existing region, reusing the installation resources of existing monitoring points; the meteorological station unit connects to the existing regional meteorological observation station to acquire meteorological data of the entire region, and at the same time connects to the existing agricultural disaster database to acquire historical risk disaster records.

[0059] The device's output module is wired to the existing public service platform of the extension department and the terminal equipment (existing computers and displays) of the township agricultural technology extension station to push regional crop growth status reports and overall risk level distribution reports; the data storage module is connected to the existing agricultural technology database of the extension department to incorporate the prediction and evaluation data into the existing database, providing data support for technology extension.

[0060] 2. Usage status of each structure Multimodal data acquisition module: The satellite remote sensing unit acquires full-area satellite remote sensing image data every 3 days, the distributed sensor network unit collects data from each monitoring point every hour, and the meteorological station unit collects full-area meteorological data in real time, realizing comprehensive monitoring of the region.

[0061] Data preprocessing module: by Figure 3 The time-series workflow processes data from multiple monitoring points and crop types within a region in batches, and performs standardized processing for different crops (wheat, corn, fruits and vegetables) to ensure the relevance of the data.

[0062] Multimodal fusion module: By Figure 2 The system operates on a two-level fusion architecture. The feature-level fusion stage adjusts the attention mechanism parameters for different crop types, and the adaptive weight allocation strategy adjusts the weights of each modality data according to the growth characteristics of different crops in the region, outputting the key fusion features of different crops. The decision-level fusion stage simultaneously completes the screening and dimensionality reduction of multiple crop features.

[0063] Growth prediction module: By Figure 4 The time-series workflow performs growth predictions for the main crops in the region, and outputs the regional average growth status parameters of each crop and the growth difference data of different townships.

[0064] Risk assessment module: Assess the risk of meteorological disasters (such as large-scale droughts and rainstorms) and epidemic pests and diseases across the region, generate a heat map of regional risk level distribution, and identify the location and scope of high-risk areas.

[0065] The output module pushes regional crop growth reports and risk warning reports to the extension department's public service platform, and provides targeted technical guidance and suggestions to agricultural technology extension stations in various townships. The model optimization and update module updates the model parameters quarterly based on the actual crop growth data and disaster occurrence in the region, thereby improving the accuracy of regional monitoring.

[0066] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the claims.

Claims

1. A method for crop growth prediction and risk assessment based on multimodal data fusion, characterized in that, Includes the following steps: S1. Multimodal data acquisition: Multimodal data of the entire growth cycle of crops are acquired synchronously through multi-source data acquisition equipment. The multimodal data includes at least remote sensing image modal data, farmland sensing modal data, environmental meteorological modal data, agricultural operation modal data and crop physiological characteristic modal data. S2. Multimodal data preprocessing: For the different types of modal data collected in step S1, heterogeneous data standardization processing is performed respectively, including data format unification, outlier removal, missing value completion and data dimension normalization, to obtain standardized multimodal data; S3. Multimodal Data Fusion: A two-level fusion architecture is constructed to fuse standardized multimodal data. The first level is feature-level fusion. An improved attention mechanism feature extraction network is used to extract features from each modality to obtain high-dimensional feature vectors for each modality. Then, an adaptive weight allocation strategy is used to weight and fuse the high-dimensional feature vectors of each modality to obtain a fused feature matrix. The second level is decision-level fusion, which involves feature filtering and dimensionality reduction of the fused feature matrix after feature-level fusion to obtain key fused features. S4. Crop growth prediction: Based on the key fusion features obtained in step S3, an improved deep learning prediction model is constructed. The key fusion features are input into the improved deep learning prediction model, and the growth status parameters of crops within a preset future period are output to complete the crop growth prediction. S5. Crop growth risk assessment: Based on the growth prediction results of step S4 and the key fusion features of step S3, a multi-dimensional risk assessment index system is constructed. The comprehensive weight of each risk index is determined by the analytic hierarchy process-entropy weight method. The potential risks in the crop growth process are assessed by combining the improved fuzzy comprehensive evaluation model to obtain the risk assessment results. S6. Output Results: The growth prediction results from step S4 and the risk assessment results from step S5 are correlated and integrated to generate a visual assessment report and output it.

2. A crop growth prediction and risk assessment system based on multimodal data fusion, characterized in that, include: Multimodal data acquisition module: used to synchronously acquire multi-type modal data of the entire growth cycle of crops through multi-source data acquisition devices. The multi-type modal data includes at least remote sensing image modal data, farmland sensor modal data, environmental meteorological modal data, agricultural operation modal data and crop physiological characteristic modal data. Data preprocessing module: Used to perform heterogeneous data standardization processing on different types of modal data collected by the multimodal data acquisition module, including data format unification, outlier removal, missing value completion, and data dimension normalization, to obtain standardized multimodal data; Multimodal fusion module: Used to build a two-level fusion architecture to fuse standardized multimodal data. The first level is feature-level fusion, which extracts features from each modality data through an improved attention mechanism feature extraction network to obtain high-dimensional feature vectors of each modality. Then, an adaptive weight allocation strategy is used to weight and fuse the high-dimensional feature vectors of each modality to obtain a fused feature matrix. The second level is decision-level fusion, which involves feature filtering and dimensionality reduction of the fused feature matrix after feature-level fusion to obtain key fused features. Growth prediction module: Based on the key fusion features obtained by the multimodal fusion module, it constructs an improved deep learning prediction model, inputs the key fusion features into the improved deep learning prediction model, and outputs the growth state parameters of crops within a preset future period to complete crop growth prediction. Risk assessment module: Based on the growth prediction results of the growth prediction module and the key fusion features of the multimodal fusion module, a multi-dimensional risk assessment index system is constructed. The comprehensive weight of each risk index is determined by the analytic hierarchy process-entropy weight method. Combined with the improved fuzzy comprehensive evaluation model, the potential risks in the crop growth process are assessed to obtain the risk assessment results. Results Output Module: This module is used to link and integrate the growth prediction results from the growth prediction module with the risk assessment results from the risk assessment module, generate a visual assessment report, and output it.

3. The method according to claim 1, characterized in that, In step S1, the remote sensing image modal data includes satellite remote sensing image data and UAV remote sensing image data, which are collected through satellite remote sensing platforms and UAV remote sensing platforms. The spatial resolution of the satellite remote sensing image data is not less than 10m, and the spatial resolution of the UAV remote sensing image data is not less than 0.1m. The farmland sensing modal data is collected through a distributed sensor network deployed in the farmland, including real-time sensing data of soil temperature and humidity, soil pH value, soil nutrient content, and crop plant height and leaf area index. The environmental meteorological modal data includes real-time meteorological data and historical meteorological data. The real-time meteorological data is collected through meteorological stations and includes temperature, precipitation, light intensity, wind speed and direction. The historical meteorological data is obtained from a meteorological database.

4. The method according to claim 1, characterized in that, In step S2, the outlier removal adopts an improved box plot method, which calculates the quartiles of each modality data and adjusts the outlier judgment threshold according to the industry characteristics of crop growth data; the missing value completion adopts a modality adaptive completion strategy, which uses LSTM-based time-series prediction completion for continuous data and Bayesian estimation-based completion for discrete data.

5. The method according to claim 1, characterized in that, In step S3, the improved attention mechanism feature extraction network includes a CNN-attention sub-network for remote sensing image modal data and an LSTM-attention sub-network for temporal sensing data. The CNN-attention sub-network introduces a spatial attention mechanism after the convolutional layer to focus on the features of key areas of crop growth in the remote sensing image. The LSTM-attention sub-network introduces a temporal attention mechanism in the LSTM hidden layer to strengthen the feature weights of key time nodes in the temporal sensing data.

6. The method according to claim 1, characterized in that, In step S3, the adaptive weight allocation strategy is specifically implemented as follows: calculate the information entropy and feature correlation of each modality's high-dimensional feature vector. The information entropy is used to characterize the information richness of each modality's data, and the feature correlation is used to characterize the degree of association between each modality's features and the crop's growth status. Based on the information entropy and feature correlation, construct a weight calculation function and dynamically allocate the fusion weights of each modality's features.

7. The method according to claim 1, characterized in that, In step S4, the improved deep learning prediction model is a CNN-LSTM fusion model. A gating mechanism is introduced before the fully connected layer of the model to filter the key fusion features that contribute highly to crop growth prediction. The growth state parameters include the predicted plant height, predicted yield, and predicted growth stage of the crop.

8. The method according to claim 1, characterized in that, In step S5, the multi-dimensional risk assessment index system includes meteorological risk indicators, soil risk indicators, pest and disease risk indicators, and agricultural operation risk indicators. Meteorological risk indicators include the occurrence rate of extreme temperatures and the number of rainstorm warnings. Soil risk indicators include soil nutrient imbalance and soil salinization. Pest and disease risk indicators are constructed based on the results of crop pest and disease feature extraction from remote sensing images. Agricultural operation risk indicators include fertilizer application deviation rate and irrigation timing deviation rate.

9. The method according to claim 1, characterized in that, It also includes step S7, model optimization and update: regularly collect new multimodal data and corresponding actual crop growth data and risk occurrence data, incrementally train the growth prediction model in step S4 and the risk assessment model in step S5, update the model parameters, and improve the prediction and assessment accuracy.

10. The system according to claim 2, characterized in that, It also includes a data storage module and a model management module. The data storage module is used to classify and store the collected raw multimodal data, preprocessed standardized data, fused feature data and prediction and evaluation result data. The model management module is used to store growth prediction models and risk assessment models corresponding to different crop types, and supports model calling, updating and version management.

Images