Crop disease diffusion prediction method and system based on multi-modal fusion

Abstract

The invention discloses a crop disease diffusion prediction method and system based on multi-modal fusion, and the method comprises the following steps: S1, collecting and preprocessing an RGB image sequence and a sensor data sequence of a crop growth environment, and generating an RGB image time sequence difference result and a sensor difference result through time difference processing; s2, mapping the RGB image time sequence difference result and the sensor difference result to a shared time sequence space through a time alignment algorithm, and generating a sensor alignment result and an RGB alignment result; s3, an FD-ViT prediction model is constructed; inputting the sensor alignment result and the RGB alignment result into an FD-ViT prediction model for prediction, and generating a prediction result; and S4, generating a disease diffusion thermodynamic diagram and early warning information according to a prediction result. According to the method, RGB image data and sensor network data are fused, a Transform-based time sequence prediction model is constructed, and early recognition and diffusion trend prediction of crop diseases are realized.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture technology, and in particular to a method and system for predicting the spread of crop diseases based on multimodal fusion. Background Technology

[0002] With the rapid development of smart agriculture, the use of artificial intelligence technology for crop disease identification and management has become a research hotspot.

[0003] Current agricultural disease management systems mainly rely on two technical approaches: one is computer vision methods based on RGB images, which use deep learning models to identify disease symptoms on the crop surface; the other is methods based on environmental sensor networks, which predict the risk of disease occurrence by monitoring changes in environmental parameters.

[0004] However, these methods have significant limitations in practical applications: single-modal data has limitations; studies show that in complex agricultural environments, the average recognition accuracy of single-modal systems is only 65-75%, which is insufficient to meet the needs of precision agriculture. Disease spread prediction capabilities are inadequate; existing technologies mainly focus on static disease identification and lack the ability to dynamically predict disease spread trends. Traditional methods perform poorly when processing high-dimensional, nonlinear agricultural data, resulting in low prediction accuracy and short warning times. Multimodal fusion technology is immature; most studies employ simple feature splicing or weighted fusion strategies, failing to effectively address the differences in time scale, feature dimension, and semantic expression among different modalities. Model adaptability is poor; traditional convolutional neural networks (CNNs) have inherent limitations when processing long-sequence data, making it difficult to capture long-distance dependencies. Due to these problems, agricultural disease prediction systems urgently need an innovative technological solution capable of providing early warnings and accurately predicting disease spread trends. Summary of the Invention

[0005] To address the limitations of existing technologies, such as single-modal limitations, insufficient disease spread prediction capabilities, immature multimodal fusion, and poor model adaptability, this invention provides a method and system for predicting the spread of crop diseases based on multimodal fusion. By innovatively fusing RGB image data and sensor network data, a Transformer-based time-series prediction model is constructed to achieve early identification and spread trend prediction of crop diseases.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] Firstly, a method for predicting the spread of crop diseases based on multimodal fusion includes the following steps:

[0008] S1. Collect and preprocess RGB image sequences and sensor data sequences of crop growth environment, and generate RGB image time-series difference results and sensor difference results through time difference processing;

[0009] S2. The RGB image temporal difference result and the sensor difference result are mapped to a shared time series space using a time alignment algorithm to generate sensor alignment result and RGB alignment result;

[0010] S3. Construct the FD-ViT prediction model; input the sensor alignment result and the RGB alignment result into the FD-ViT prediction model to make a prediction and generate a prediction result;

[0011] S4. Generate a disease spread heat map and early warning information based on the prediction results.

[0012] Preferably, the preprocessing includes: performing illumination correction and noise reduction on the RGB image sequence, and using the CLAHE algorithm to enhance contrast; and identifying and correcting outliers in the sensor data sequence using the 3σ principle and isolated forest.

[0013] Preferably, S2 includes:

[0014] Based on the device timestamp synchronized using the NTP protocol, a time mapping matrix between the RGB image sequence and the sensor data is established.

[0015] The sensor differential results are upsampled to 30Hz using cubic spline interpolation to obtain the sensor upsampling results; the RGB image temporal differential results are downsampled using average pooling to obtain the RGB downsampling results.

[0016] A learnable linear projection layer is used to map the sensor upsampling results and the RGB downsampling results to the same dimension, resulting in the sensor alignment result and the RGB alignment result.

[0017] As a preferred option, S2 also includes a fault tolerance strategy. When a missing segment is detected on the timeline, cubic spline interpolation is used to complete it, and a maximum allowable misalignment window Δt = 5 frames is set. If the misalignment exceeds this window, an anomaly flag is triggered.

[0018] Preferably, the FD-ViT prediction model described in S3 includes:

[0019] An RGB branch encoder is used to extract the spatiotemporal features of the RGB image sequence and generate RGB encoding results;

[0020] A sensor branch encoder is used to extract the temporal characteristics of the sensor data and generate sensor encoding results.

[0021] The cross-modal fusion layer fuses the RGB encoding results and the sensor encoding results through a dynamic cross-modal attention mechanism to generate a fusion result;

[0022] The time series prediction head outputs the predicted probability of disease spread based on the fusion result using a Transformer decoder.

[0023] Preferably, S1 further includes segmenting the diseased region using a target segmentation network with an exponential moving average strategy to generate a diseased region mask M. t In S3, the specific implementation of the RGB branch encoder includes:

[0024] The RGB alignment result is divided into 8×8 patches to form a three-dimensional patch structure, resulting in an RGB image patch segmentation result. Each RGB image patch segmentation result is flattened and mapped to a 768-dimensional embedding space through linear projection to obtain the first RGB linear patch result.

[0025] Add time position encoding to the first RGB linear patch result to obtain the second RGB linear patch result;

[0026] The second RGB linear patch result is input into a 12-layer Transformer encoder to extract the spatiotemporal features of the RGB image sequence, thus obtaining the RGB attention result;

[0027] The RGB attention result is compared with the disease area mask M. t Multiply them to obtain the RGB encoding result.

[0028] Preferably, in S3, the dynamic cross-modal attention mechanism includes:

[0029] A bidirectional cross-modal attention calculation module is used to calculate the attention from RGB features to sensor features and from sensor features to RGB features;

[0030] The dynamic fusion gating module evaluates the reliability of each mode based on environmental conditions and generates fusion weights, as shown in the following formula:

[0031] ω Rgb =σ(W env x env +b env )

[0032] ω Sensor =1-ω Rgb

[0033] Where σ is the sigmoid function, W env ∈R 1*128 ,benv ∈R,x env ω represents the feature vector of environmental conditions. Rgb ω represents the RGB feature weights. Sensor For sensor feature weights;

[0034] The weighted fusion module performs weighted fusion on the RGB encoding result and the sensor encoding according to the fusion weight to generate the fusion result.

[0035] Preferably, in S3, the time series prediction head is specifically implemented as follows:

[0036] The time series of the fusion result is input into the Transformer decoder to capture the dependencies within the sequence; the output of the Transformer decoder is used to perform attention calculation with the fusion result to obtain contextual information.

[0037] After the Transformer decoder outputs the prediction result for the first time step, it adds the prediction result of the first time step to the input sequence as the input for the second time step; this step is repeated continuously to predict the results for multiple future time steps.

[0038] The prediction result is generated by fusing the results of time steps generated at different time scales.

[0039] Preferably, S4 includes:

[0040] The result of the time step is input into a convolutional layer with 128 dimensions as input, 64 channels as output, and a kernel size of 3*3 to obtain the first convolutional result; the first convolutional result is input into a convolutional layer with 64 channels as input, 1 channel as output, and a kernel size of 3*3 to obtain the second convolutional result; the second convolutional result is used with the Sigmoid function to generate a normalized output result as the probability of each pixel having a defect;

[0041] The normalized output is converted into a heatmap, with red representing high-probability areas and blue representing low-probability areas, and then superimposed onto the original image to generate the disease spread heatmap and the early warning information.

[0042] Secondly, a crop disease spread prediction system based on multimodal fusion includes:

[0043] The data acquisition module is used to acquire RGB image sequences and sensor data sequences of the crop growth environment;

[0044] The data processing module is used to perform temporal difference processing on the RGB image sequence to extract dynamic features of the diseased area; and to perform first-order difference processing and standardization on the sensor data sequence.

[0045] The time alignment module is used to map RGB image features and sensor features to a shared time series space using a time alignment algorithm.

[0046] The FD-ViT prediction model includes an RGB branch encoder, a sensor branch encoder, a cross-modal fusion layer, and a time-series prediction head.

[0047] The decision output module is used to generate disease spread heat maps, early warning information, and control instructions based on the prediction results;

[0048] The aforementioned multimodal fusion-based crop disease spread prediction system is used to implement the multimodal fusion-based crop disease spread prediction method and its steps as described in the first aspect.

[0049] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0050] (1) Advantages of ViT Time Series Prediction: The FD-ViT model of this invention integrates RGB + multi-dimensional sensors (soil, air), enabling multi-modal fusion. While ViT models in agriculture are primarily used for static identification and classification of disease images, this invention extends ViT to disease spread dynamics modeling for the first time. By constructing a three-dimensional spatiotemporal patch (H×W×T) and combining it with a Transformer decoder, it achieves pixel-level prediction of disease spread probability within the next 72 hours, outputting a dynamic heatmap. This task involves spatiotemporal sequence generation, fundamentally different from existing ViT applications. Unlike traditional ViT technologies that focus on disease location and classification rather than spread prediction, this invention applies the Vision Transformer architecture to the task of "spatiotemporal sequence prediction of crop disease spread," generating a specific form of a future disease spread heatmap.

[0051] This invention enables the application of the ViT architecture to time-series prediction of agricultural disease spread. Through spatiotemporal patch segmentation and a self-attention mechanism, ViT can capture long-range dependencies in disease spread, overcoming the limitations of traditional CNNs in long-sequence modeling. Comparative experiments show that the FD-ViT model achieves a disease spread prediction accuracy of 87.6% (72-hour prediction), a 15.2% improvement over the traditional LSTM model.

[0052] (2) Multimodal temporal feature alignment: This invention innovatively projects RGB images and sensor data onto a shared time series space, eliminating global motion interference through temporal difference processing and highlighting disease-related features. This solves the temporal misalignment problem caused by differences in sampling rates across different modalities, enabling the system to capture the dynamic evolution of disease spread. In actual testing, this method improved the temporal feature alignment accuracy by 35%, providing more accurate input for subsequent predictions.

[0053] (3) Dynamic Cross-Modal Attention Mechanism: This invention proposes a conditional attention mechanism that adaptively adjusts the contribution of each mode according to environmental conditions, achieving "modality-independent prediction": even if some modes fail, the system can still maintain its predictive ability. For example, under insufficient lighting conditions, the system automatically increases the weight of sensor data to ensure stable prediction performance. Experiments show that this mechanism improves the prediction stability of the system by 28% under various environmental conditions.

[0054] (4) Disease Spread Dynamics Modeling: This invention combines crop growth models with ViT prediction to construct a coupled dynamics model of environment-disease-crop growth. This achieves a crucial leap from "disease identification" to "disease prediction," increasing the early warning time from 12.5 hours in traditional methods to 22.4 hours, a 79% improvement. In practical application at the Shuangpu Yuanxiang Strawberry Picking Garden in Xihu District, the system successfully provided an early warning of gray mold spread 21.8 hours in advance, reducing pesticide use by 38.5%.

[0055] (5) Task-aware edge-cloud collaborative deployment: The present invention adopts an edge-cloud collaborative architecture that is deeply customized for the task of disease spread prediction. Its collaborative logic is fundamentally different from the general computing offloading architecture: short-term dynamic changes can be captured by RGB, so the edge only processes RGB and predicts the next 6 hours; long-term spread is driven by the environment and requires multimodal fusion, so the cloud completes the 72-hour prediction; the edge only uploads differential features filtered by the disease area mask (reducing the amount of data by an average of 80%), realizing "semantic compression" for the prediction task; when the sensor fails or the time misalignment exceeds the limit, the edge automatically switches to single-modal mode and uploads the abnormality mark. The cloud adjusts the fusion weight accordingly to ensure the robustness of the system; the measured end-to-end latency of the system is controlled within 200ms, which not only meets the needs of real-time agricultural regulation, but also greatly reduces the computing and communication burden of the edge devices.

[0056] (6) Significant practical application value: Practical application of this invention in strawberry cultivation shows that it can reduce yield loss caused by diseases by 42.7%, reduce manual inspection costs by 50%, and improve strawberry quality and commercialization rate. At the same time, by reducing the use of chemical pesticides through precise application, it promotes green agricultural production, provides data support for organic agriculture and green certification, and has significant socio-economic and environmental benefits. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention;

[0058] Figure 2 This is a schematic diagram of an example of the "Berry facial features" sensor network in Embodiment 1 of the present invention;

[0059] Figure 3 This is a schematic diagram of the FD-ViT model structure in Embodiment 1 of the present invention;

[0060] Figure 4 This is a schematic diagram of the time alignment algorithm in Embodiment 1 of the present invention;

[0061] Figure 5 This is a schematic diagram of the dynamic cross-modal attention mechanism in Embodiment 1 of the present invention;

[0062] Figure 6 This is a schematic diagram of the time series prediction head process in Embodiment 1 of the present invention;

[0063] Figure 7 This is a visualization example of the disease spread prediction results in Embodiment 1 of the present invention;

[0064] Figure 8 This is a schematic diagram of the integrated water and fertilizer machine according to Embodiment 1 of the present invention;

[0065] Figure 9 This is a schematic diagram of the system framework of Embodiment 2 of the present invention. Detailed Implementation

[0066] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0067] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0068] Example 1: Taking strawberries as an example

[0069] like Figure 1 The method for predicting the spread of crop diseases based on multimodal fusion, as shown, includes the following steps:

[0070] S1. Collect RGB image sequences and sensor data sequences of the strawberry growing environment;

[0071] like Figure 2 As shown, a network of Berry's "five senses" sensors was deployed in a strawberry greenhouse, including:

[0072] RGB Vision Subsystem: High-resolution camera array, capturing images of strawberry plants at a frequency of 30fps (daytime) and 5fps (nighttime), with a resolution of 1920×1080 and a dual-view perspective of 45° overhead and 0° horizontal.

[0073] Sensor network: Real-time monitoring of 16 environmental parameters, including soil moisture, temperature, pH value, nitrogen, phosphorus and potassium content, air temperature and humidity, CO2 concentration, light intensity, air pressure, etc. The sampling frequency is 1Hz (key parameters) / 10 minutes (stable parameters), and the data accuracy is ±0.1℃ for temperature, ±1% for humidity, and ±5ppm for nitrogen, phosphorus and potassium.

[0074] During data acquisition, the time synchronization between RGB images and sensor data is ensured. The NTP protocol is used to achieve time synchronization between devices, and the timestamp accuracy reaches the millisecond level.

[0075] like Figure 3 As shown, preprocessing of the RGB image sequence and sensor data sequence includes:

[0076] RGB image sequence preprocessing: Illumination correction and noise reduction were performed on the strawberry RGB image sequence, and the contrast was enhanced using the CLAHE algorithm to obtain the RGB preprocessing results;

[0077] Sensor data sequence preprocessing: The sensor data of the strawberry growing environment are obtained and outliers in the sensor data are identified and corrected by the 3σ principle and isolated forest, and the sensor correction output results are obtained.

[0078] Diseased regions are segmented using a target segmentation network with an exponential moving average strategy, generating a diseased region mask M. t ; Calculate the difference image between consecutive frames from the RGB preprocessing results: D t =|I t -I t-1 |⊙M t This highlights the dynamic changes in the diseased area, where ⊙ represents element-wise multiplication to obtain the RGB dynamic disease result; the RGB dynamic disease result is then binarized with a threshold of T = 0.1max(D t The RGB binarization result is obtained; the morphological opening and closing operation is performed on the obtained binarized image to remove noise, and the temporal difference result of the RGB image is obtained.

[0079] The sensor correction output is subjected to first-order difference to eliminate the trend term, resulting in the sensor first-order difference output; the sensor first-order difference output is then standardized using Z-score to obtain the sensor standardized result; weights are assigned to different standardized results based on the correlation between each parameter and disease development to obtain the sensor difference result.

[0080] S2. The temporal difference results of the RGB image and the sensor difference results are mapped to a shared time series space using a time alignment algorithm;

[0081] The time alignment of this invention is a signal-level time sequence alignment, specifically designed for agricultural multimodal asynchronous acquisition scenarios. Through cubic spline interpolation and learnable projection, it maps the heterogeneous sampling rate RGB (30Hz) and sensor (1Hz / 10min) to a unified time series space, and introduces a fault-tolerant mechanism to handle timestamp misalignment.

[0082] like Figure 4 As shown, it specifically includes:

[0083] Based on the device timestamp synchronized by the NTP protocol, a time mapping matrix T between RGB images and sensor data is established, where T[i,j]=1 indicates that the i-th RGB frame is time-aligned with the j-th sensor data point;

[0084] The sensor differential results are upsampled to 30Hz using cubic spline interpolation to obtain the sensor upsampling results. The interpolation formula is as follows:

[0085] S i (x)=a i +b i (xx i )+c i (xx i ) 2 +d i (xx i ) 3

[0086] Where the coefficient a i b i c i d i Determined by four adjacent data points;

[0087] The temporal difference results of the RGB image are downsampled using average pooling to obtain the RGB downsampled results;

[0088] A learnable linear projection layer is used to map the sensor upsampling results and RGB downsampling results to the same dimension D=256, resulting in the sensor alignment result and RGB alignment result, as shown in the following formula:

[0089] F aligned=W*F raw +b

[0090] Where W∈R 256*d Let b be the weight matrix, and b ∈ R. 256 F is the bias vector. raw F is the original feature vector. aligned The aligned feature vectors;

[0091] The time alignment algorithm also includes a fault tolerance strategy: when a missing segment of the timeline is detected, cubic spline interpolation is used to complete it, and a maximum allowable misalignment window Δt = 5 frames is set. If the misalignment exceeds this window, an anomaly flag is triggered.

[0092] S3. Input the aligned multimodal features into the FD-ViT prediction model (Fusion-Disease VisionTransformer);

[0093] The FD-ViT model of this invention integrates RGB+multidimensional sensors (soil, air), enabling multimodal fusion.

[0094] In agriculture, the ViT (Vision Transformer) model is primarily used for static identification and classification of disease images. This invention, however, extends ViT to disease spread dynamics modeling for the first time. By constructing a 3D spatiotemporal patch (H×W×T) and combining it with a Transformer decoder, it achieves pixel-level prediction of disease spread probability within the next 72 hours, outputting a dynamic heatmap. This task falls under spatiotemporal sequence generation, fundamentally different from existing ViT applications.

[0095] Since the dynamic cross-modal attention mechanism is not a simple weighting, this invention introduces an environmentally sensitive gating module to dynamically evaluate the reliability of each modality based on real-time conditions such as light intensity and humidity, thereby achieving adaptive modal fusion. For example, when there is insufficient light at night, the RGB weights are automatically reduced and the sensor weights are increased to ensure prediction robustness—a mechanism that has not yet been reported in the field of agricultural disease prediction.

[0096] The FD-ViT prediction model includes an RGB branch encoder, a sensor branch encoder, a cross-modal fusion layer, and a time-series prediction head.

[0097] (1) RGB branch encoder, used to extract the spatiotemporal features of RGB image sequences. The specific implementation steps are as follows:

[0098] The RGB alignment result is divided into 8×8 patches, forming [T patch =10,W patch =8,H patchA 3D patch structure with C=8, C=3] is used to obtain RGB image patch segmentation results; each RGB image patch segmentation result is flattened and mapped to a 768-dimensional embedding space through linear projection to obtain the first RGB linear patch result; a temporal position code is added to the obtained first RGB linear patch result to obtain the second RGB linear patch result, as shown in the following formula:

[0099] PE(pos,2i)=sin(pos / 10000 2i / d )

[0100] PE(pos,2i+1)=cos(pos / 10000 2i / d )

[0101] Where pos is the position index, i is the dimension index, d is the embedding dimension, and 10000 2i / d This is the scaling factor;

[0102] The second RGB linear patch result is input into a 12-layer Transformer encoder to extract the spatiotemporal features of the RGB image sequence. Each layer contains an 8-head self-attention mechanism and a feedforward network, with a hidden layer dimension of 3072, to obtain the RGB attention result. The RGB attention result obtained by the Transformer is then compared with the disease area mask M. t Multiply to obtain the RGB encoding result, as shown in the formula below:

[0103] A'=A⊙M t

[0104] Where ⊙ represents element-wise multiplication.

[0105] (2) Sensor branch encoder, used to extract the time features of sensor data, is implemented as follows:

[0106] The sensor alignment results are input into a 3-layer 1D-CNN to extract the local temporal patterns of the sensor data. The kernel sizes are 3, 5, and 7, the number of channels are 64, 128, and 256, the stride is 1, and the padding is the same, to obtain the sensor convolution output results.

[0107] The sensor convolution output is input into a two-layer LSTM with 256 hidden units to capture the long-term dependencies of the sensor data, thus obtaining the sensor LSTM output. The obtained sensor LSTM output is then input into an environmentally sensitive gating module to obtain the sensor encoding result.

[0108] (3) Cross-modal fusion layer: The dynamic cross-modal attention mechanism is used to fuse the RGB encoding results and the sensor encoding results to obtain the fusion result.

[0109] like Figure 5 As shown, the dynamic cross-modal attention mechanism of the cross-modal fusion layer includes:

[0110] The bidirectional cross-modal attention calculation module is used to calculate the attention from RGB features to sensor features and from sensor features to RGB features, as shown in the following formula:

[0111] RGB to Sensor Attention:

[0112] Sensor->RGB Attention:

[0113] Where Q, K, and V are the query, key vector, and value, respectively, and d k For feature dimensions.

[0114] After bidirectional attention, RGB enhanced features and sensor enhanced features are obtained. These two semantically aligned and complementary enhanced features are then fed into the subsequent dynamic fusion gating module for weighted fusion.

[0115] The dynamic fusion gating module evaluates the reliability of each mode based on environmental conditions and generates fusion weights, as shown in the following formula:

[0116] ω Rgb =σ(W env x env +b env )

[0117] ω Sensor =1-ω Rgb

[0118] Where σ is the sigmoid function, W env ∈R 1*128 ,b env ∈R,x env The environmental condition feature vector (containing parameters such as light intensity, humidity, and temperature), ω Rgb ω represents the RGB feature weights. Sensor For sensor feature weights;

[0119] The weighted fusion module performs weighted fusion of RGB enhanced features and sensor enhanced features according to fusion weights, as shown in the following formula:

[0120] F=ω Rgb *F Rgb +ω Sensor *F Sensor

[0121] Among them, F Rgb and F Sensor These are the RGB encoding result and the sensor encoding, respectively, with F representing the fused feature.

[0122] (4) Time series prediction head: Based on the Transformer decoder, the probability of strawberry disease spread within a preset future time period is output, such as... Figure 6 As shown, the specific steps are as follows:

[0123] The time series of RGB and sensor fusion results is input into the Transformer decoder to capture the dependencies within the sequence;

[0124] Attention is calculated on the decoder output and the fusion result obtained after processing by the cross-modal fusion layer to obtain contextual information;

[0125] The decoder outputs the prediction result for the first time step;

[0126] The prediction result of the first time step is added to the input sequence as the input of the second time step;

[0127] By repeatedly performing the above steps, the results of multiple future time steps can be predicted step by step.

[0128] The results of time steps generated at different time scales are fused together using the following formula:

[0129]

[0130] Among them, y t For the prediction result at time step t, β t Weights based on time scale;

[0131] The FD-ViT prediction model is deployed in an edge-cloud collaborative architecture. The edge performs lightweight RGB processing and short-term prediction, while the cloud performs multimodal fusion and long-term prediction. Data transmission is optimized to reduce data transmission volume by 80%.

[0132] S4. Generate a disease spread heatmap and early warning information based on the strawberry disease spread prediction results, including:

[0133] The time step prediction results output from the Transformer decoder are input into a convolutional layer with 128 dimensions as input, 64 channels as output, and a kernel size of 3*3 to obtain the first convolution result.

[0134] The first convolution result is input into a convolutional layer with 64 input channels, 1 output channel, and a kernel size of 3*3 to obtain the second convolution result;

[0135] The second convolution result is then processed using the Sigmod function to obtain the normalized output, which represents the probability of each pixel being affected by a defect.

[0136] The normalized output is converted into a heatmap, with high-probability areas represented in red and low-probability areas in blue. This heatmap is then overlaid onto the original image to obtain the final output, as shown below. Figure 7 As shown;

[0137] The step of generating control instructions based on the prediction results includes adjusting irrigation volume, optimizing fertilization plans, and providing pest and disease control suggestions. These instructions are used to coordinate with integrated water and fertilizer equipment to perform precise control. Figure 8 As shown.

[0138] This invention proposes a novel solution to the data quality and feature extraction problems in existing agricultural disease prediction technologies by combining data optimization, multimodal feature fusion, and self-supervised learning techniques. The core innovations of this invention are as follows:

[0139] (1) Multimodal Temporal Feature Alignment: A feature alignment method based on temporal difference domain is proposed. By using the temporal difference of RGB images and the first-order difference of sensor data, different modal data are mapped to a shared time series space. This method effectively solves the temporal misalignment problem caused by the difference in sampling rates of different modalities, enabling the system to capture the dynamic evolution process of disease spread. Experiments show that this method improves the temporal feature alignment accuracy by 35%, providing more accurate input for subsequent prediction.

[0140] (2) Innovative Dynamic Cross-Modal Attention Mechanism: A conditional attention mechanism was designed to adaptively adjust the contribution of RGB and sensor modes based on environmental conditions (such as illumination and humidity). This achieves "modality-independent prediction": even if some modes fail (such as insufficient light at night or sensor malfunction), the system can still maintain its predictive ability. This mechanism improves the prediction stability of the system by 28% under various environmental conditions, achieving superior performance compared to traditional fusion methods.

[0141] (3) Innovation in ViT Time Series Prediction: The ViT architecture can be applied to the time series prediction task of agricultural disease spread. Through spatiotemporal patch segmentation and self-attention mechanism, ViT can capture the long-distance dependencies of disease spread, overcoming the limitations of traditional CNN in long sequence modeling. Comparative experiments show that the FD-ViT model achieves a disease spread prediction accuracy of 87.6% (72-hour prediction).

[0142] (4) Practical Application Value Innovation: The practical application of this invention in strawberry cultivation shows that it can reduce yield loss caused by diseases by 42.7%, reduce manual inspection costs by 50%, and improve strawberry quality and commercialization rate. At the same time, by reducing the use of chemical pesticides through precise application, it promotes green agricultural production, provides data support for organic agriculture and green certification, and has significant socio-economic and environmental benefits.

[0143] Example 2:

[0144] like Figure 9 The system shown is a crop disease spread prediction system based on multimodal fusion, comprising:

[0145] The data acquisition module is used to acquire RGB image sequences and sensor data sequences of the crop growth environment;

[0146] The data processing module is used to perform temporal difference processing on RGB image sequences to extract dynamic features of diseased areas; and to perform first-order difference processing and standardization on sensor data sequences.

[0147] The time alignment module is used to map RGB image features and sensor features to a shared time series space using a time alignment algorithm.

[0148] The FD-ViT prediction model includes an RGB branch encoder, a sensor branch encoder, a cross-modal fusion layer, and a time-series prediction head.

[0149] The decision output module is used to generate disease spread heat maps, early warning information, and control instructions based on the prediction results.

[0150] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A method for predicting crop disease spread based on multi-modal fusion, characterized in that, The method comprises the following steps: S1, collecting and preprocessing an RGB image sequence and a sensor data sequence of a crop growth environment, and generating an RGB image time difference result and a sensor difference result through time difference processing; S2, mapping the RGB image time difference result and the sensor difference result to a shared time sequence space through a time alignment algorithm to generate a sensor alignment result and an RGB alignment result; S3, constructing an FD-ViT prediction model; inputting the sensor alignment result and the RGB alignment result into the FD-ViT prediction model for prediction to generate a prediction result; S4, generating a disease spread thermodynamic map and early warning information according to the prediction result.

2. The crop disease spread prediction method based on multi-modal fusion according to claim 1, characterized in that, The FD-ViT prediction model in S3 comprises: an RGB branch encoder configured to extract spatio-temporal features of the RGB image sequence to generate an RGB encoding result; a sensor branch encoder configured to extract time features of the sensor data to generate a sensor encoding result; a cross-modal fusion layer configured to fuse the RGB encoding result and the sensor encoding result through a dynamic cross-modal attention mechanism to generate a fusion result; a time sequence prediction head configured to output the prediction result of a disease spread probability by using a Transformer decoder according to the fusion result.

3. The crop disease spread prediction method based on multi-modal fusion according to claim 2, characterized in that, S1 further comprises performing disease area segmentation by a target segmentation network with an exponential moving average strategy to generate a disease area mask M t In S3, the RGB branch encoder specifically realizes: The RGB alignment result is divided into 8x8 patches to form a three-dimensional patch structure to obtain an RGB image patch segmentation result; each RGB image patch segmentation result is flattened and mapped to a 768-dimensional embedding space through linear projection to obtain a first RGB linear patch result; time position encoding is added to the first RGB linear patch result to obtain a second RGB linear patch result; the second RGB linear patch result is input into a 12-layer Transformer encoder to extract spatio-temporal features of the RGB image sequence to obtain an RGB attention result; multiplying the RGB attention result with the disease area mask M t to obtain the RGB encoding result.

4. The crop disease spread prediction method based on multi-modal fusion according to claim 2, characterized in that, In S3, the dynamic cross-modal attention mechanism comprises: a bidirectional cross-modal attention calculation module configured to calculate the attention of RGB features to sensor features and sensor features to RGB features; a dynamic fusion gating module configured to evaluate the reliability of each modality based on environmental conditions and generate a fusion weight, the formula being as follows: ω Rgb = σ(W env x env + b env ) ω Sensor = 1 - ω Rgb wherein σ is a sigmod function, W env ∈R 1*128 ,b env ∈R,x env is an environmental condition feature vector; ω Rgb is an RGB feature weight, ω Sensor is a sensor feature weight; a weighted fusion module configured to perform weighted fusion on the RGB encoding result and the sensor encoding according to the fusion weight to generate the fusion result.

5. The crop disease spread prediction method based on multi-modal fusion according to claim 2, characterized in that, In S3, the time sequence prediction head is specifically implemented as follows: input the time sequence of the fusion result into a Transformer decoder to capture the dependency within the sequence; perform attention calculation on the output of the Transformer decoder and the fusion result to obtain context information; after the Transformer decoder outputs the prediction result of the first time step, the prediction result of the first time step is added to the input sequence as the input of the second time step; this step is repeated to gradually predict the results of multiple future time steps; fuse the results of the time steps generated at different time scales to generate the prediction result.

6. The crop disease spread prediction method based on multi-modal fusion according to claim 1, characterized in that, The preprocessing includes: performing illumination correction and denoising processing on the RGB image sequence, and enhancing contrast using a CLAHE algorithm; identifying and correcting abnormal points in the sensor data sequence through a 3σ principle and an isolation forest.

7. The crop disease spread prediction method based on multi-modal fusion according to claim 1, characterized in that S2 The method comprises the following steps: Synchronizing device timestamps based on an NTP protocol to establish a time mapping matrix of the RGB image sequence and the sensor data; Upsampling the sensor difference result to 30Hz through a cubic spline interpolation method to obtain a sensor upsampled result; and downsampling the RGB image time series difference result using average pooling to obtain an RGB downsampled result. Mapping the sensor upsampled result and the RGB downsampled result to the same dimension using a learnable linear projection layer to obtain the sensor aligned result and the RGB aligned result.

8. The crop disease spread prediction method based on multi-modal fusion according to claim 7, characterized in that, S2 further comprises a fault tolerance strategy, when a time axis missing segment is detected, a cubic spline interpolation is used for completion, and a maximum allowed misalignment window Δt = 5 frames is set, if the misalignment exceeds, an abnormality flag is triggered.

9. The crop disease spread prediction method based on multi-modal fusion of claim 5, wherein S4 The method comprises the following steps: Inputting the time step result into a convolution layer with an input of 128 dimensions, an output of 64 channels, and a kernel size of 3*3 to obtain a first convolution result; inputting the first convolution result into a convolution layer with an input of 64 channels, an output of 1 channel, and a kernel size of 3*3 to obtain a second convolution result; using a Sigmod function to generate a normalized output result as the probability of each pixel being diseased; Converting the normalized output result into a heat map, using red to represent high probability areas and blue to represent low probability areas, and superimposing it onto the original image to generate the disease spread heat map and the warning information.

10. A crop disease spread prediction system based on multi-modal fusion, characterized in that, The method comprises the following steps: A data acquisition module is configured to acquire an RGB image sequence and a sensor data sequence of a crop growing environment; A data processing module is configured to perform time difference processing on the RGB image sequence to extract dynamic features of disease areas; and perform first-order difference processing and standardization on the sensor data sequence; A time alignment module is configured to map RGB image features and sensor features to a shared time series space through a time alignment algorithm; An FD-ViT prediction model comprises an RGB branch encoder, a sensor branch encoder, a cross-modal fusion layer, and a time series prediction head; A decision output module is configured to generate a disease spread heat map, warning information, and control instructions based on the prediction result. The crop disease spread prediction system based on multi-modal fusion is used to implement the crop disease spread prediction method based on multi-modal fusion and the steps thereof.

Images