A crop disease identification method based on multi-modal fusion and knowledge distillation

Abstract

This invention provides a crop disease identification method based on multimodal fusion and knowledge distillation in the field of crop disease identification technology, comprising: Step S1, collecting a large amount of historical multimodal crop monitoring data to construct a dataset, and performing adaptive matching of the dataset in the time dimension; Step S2, creating a crop disease identification model based on a feature extraction module, a cross-modal interaction module, and a prediction output module, training the crop disease identification model with the dataset, and using the trained crop disease identification model as a teacher model; Step S3, performing knowledge distillation on the teacher model to transfer the knowledge learned by the teacher model to the created student model, and deploying the student model to an edge computing device; Step S4, the edge computing device performing crop disease identification based on the deployed student model. The advantages of this invention are: it greatly improves the accuracy of disease identification and enhances the generalization ability, while taking into account the feasibility and efficiency of practical deployment.

Description

Technical Field

[0001] This invention relates to the field of crop disease identification technology, and in particular to a crop disease identification method based on multimodal fusion and knowledge distillation. Background Technology

[0002] Against the backdrop of escalating global climate change, frequent meteorological disasters such as high temperatures, droughts, and extreme precipitation have led to a significant increase in the frequency and severity of crop diseases and pests, posing a serious threat to global food security. Crop diseases not only directly cause yield reductions and quality declines but can also trigger large-scale agricultural economic losses and even affect the stability of regional food supplies. Therefore, achieving accurate identification and real-time monitoring of crop diseases has become a critical technological challenge that urgently needs to be addressed in the field of smart agriculture.

[0003] Traditional disease monitoring methods often rely on a single data source. For example, visible light images can be used to identify apparent lesions, hyperspectral data can retrieve internal biochemical parameters of leaves, and meteorological sensors can provide information related to environmental stress. However, these single-modal data often only capture one aspect of disease occurrence and cannot comprehensively reflect the complex physiological and ecological changes in crops during disease occurrence and development. This is especially true in the early or latent stages of disease, when external symptoms are not obvious but internal metabolism has already become abnormal, making the monitoring capability of a single data source clearly insufficient.

[0004] Multimodal data fusion technology, by integrating multiple information sources such as remote sensing, imagery, and environmental data, theoretically possesses more comprehensive potential for disease monitoring. However, existing fusion methods still face several technical bottlenecks in practical applications, hindering their effective deployment in real agricultural scenarios. Specifically, existing technologies face the following main problems:

[0005] First, regarding spatiotemporal alignment, different modal data (such as UAV RGB images, multispectral data, and meteorological sensor data) exhibit differences in resolution and sampling frequency. Existing methods often employ simple resampling for alignment, neglecting the nonlinear temporal dynamics of crop physiological processes. This leads to spatiotemporal biases in the fused features, making it difficult to accurately depict the continuous development of diseases. Second, regarding feature interaction, most models employ late-stage decision fusion or shallow feature stitching, failing to fully explore the potential biophysical relationships between multimodal data and limiting the model's ability to discriminate disease stress responses. Finally, regarding model generalization and practical deployment, existing high-precision multimodal models often rely on expensive sensing equipment and have high computational complexity, making them unsuitable for large-scale agricultural production scenarios. Although some research has attempted to leverage general pre-trained models such as the Visual Transformer for transfer learning, the lack of prior knowledge in the agricultural field leads to spectral-environment feature distortion and overfitting under small sample conditions.

[0006] Therefore, how to provide a crop disease identification method based on multimodal fusion and knowledge distillation, which can improve the accuracy of disease identification and enhance generalization ability, while taking into account the feasibility and efficiency of practical deployment, has become an urgent technical problem to be solved. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a crop disease identification method based on multimodal fusion and knowledge distillation, which can improve the accuracy of disease identification and enhance the generalization ability, while taking into account the feasibility and efficiency of practical deployment.

[0008] This invention is implemented as follows: a crop disease identification method based on multimodal fusion and knowledge distillation, comprising the following steps:

[0009] Step S1: Collect a large amount of historical multimodal crop monitoring data from crop planting sites, including spectral data, RGB image data, and environmental meteorological data;

[0010] Step S2: Preprocess the historical multimodal crop monitoring data to construct a dataset;

[0011] Step S3: Perform adaptive matching of the dataset along the time dimension;

[0012] Step S4: Create a crop disease identification model based on the feature extraction module, cross-modal interaction module, and prediction output module. Train the crop disease identification model using the dataset and use the trained crop disease identification model as the teacher model.

[0013] The feature extraction module is used to extract spectral features, visual features, and environmental features from spectral data, RGB image data, and environmental meteorological data, respectively.

[0014] The cross-modal interaction module is used to modulate and fuse spectral features, visual features, and environmental features through a multi-head attention mechanism to obtain a fused feature vector;

[0015] The prediction output module is used to infer the fused feature vector and output the crop disease identification result. When the prediction output module infers the fused feature vector, it first flattens the fused feature vector into a one-dimensional vector, then maps it to the disease category space through a fully connected layer, and applies the Softmax function to output the probability of each disease category. Finally, the maximum probability is taken as the disease identification result.

[0016] The loss function of the crop disease identification model is the InfoNCE function;

[0017] Step S5: Create a student model, perform knowledge distillation on the teacher model to transfer the knowledge learned by the teacher model to the student model, and deploy the student model to an edge computing device located in a crop planting area.

[0018] Step S6: The edge computing device identifies crop diseases based on the deployed student model.

[0019] Furthermore, step S1 specifically includes:

[0020] From crop planting sites, a large amount of historical multimodal crop monitoring data, including spectral data, RGB image data, and environmental meteorological data, is collected using a sensor array including drones, cameras, temperature sensors, humidity sensors, light sensors, and rainfall sensors. The historical multimodal crop monitoring data are then correlated and integrated based on the timestamps and geographic coordinates of the collected data.

[0021] Furthermore, in step S1, the environmental meteorological data includes at least ambient temperature, ambient humidity, light intensity, and rainfall.

[0022] Furthermore, step S2 specifically includes:

[0023] Radiometric calibration, image mosaicking, and spectral information extraction are performed on the spectral data in the historical multimodal crop monitoring data. Timestamp alignment, spatial resolution alignment, noise suppression, and outlier removal are performed on the historical multimodal crop monitoring data to complete the preprocessing of the historical multimodal crop monitoring data. A dataset is constructed based on the preprocessed historical multimodal crop monitoring data.

[0024] Furthermore, the timestamp alignment and spatial resolution alignment are based on interpolation.

[0025] Furthermore, step S3 specifically includes:

[0026] The alignment time is calculated using a time alignment function, and the dataset is then adaptively matched in the time dimension based on the alignment time using a dynamic time warping algorithm.

[0027] Furthermore, step S5 specifically includes:

[0028] A student model is created, and soft labels are generated using the teacher model. Knowledge distillation is performed on the teacher model to transfer the knowledge learned by the teacher model to the student model. During the transfer process, the soft labels are used as supervision signals. The performance of the student model after the transfer is completed is verified, and the student model that passes the performance verification is deployed to an edge computing device located in a crop planting area.

[0029] Furthermore, step S6 specifically includes:

[0030] Edge computing devices collect real-time multimodal crop monitoring data from crop planting areas through sensor arrays. After preprocessing the real-time multimodal crop monitoring data, the data is input into a deployed student model to identify crop diseases.

[0031] The advantages of this invention are:

[0032] 1. By collecting a large amount of historical multimodal crop monitoring data, including spectral data, RGB image data, and environmental meteorological data, the data is preprocessed to construct a dataset. Adaptive matching of the dataset's time dimension is then performed. Next, a crop disease identification model is created based on a feature extraction module, a cross-modal interaction module, and a prediction output module. This model is trained on the adaptively matched dataset. The trained model is then used as a teacher model, and knowledge distillation is performed to transfer the learned knowledge to a student model. The student model is then deployed to edge computing devices located in crop planting areas. The edge computing devices then perform operations based on the deployed student models. Crop disease identification involves using Dynamic Time Warping (DTW) and interpolation to refine the spatiotemporal alignment of multimodal crop monitoring data, eliminating biases caused by sampling differences and laying a data foundation for accurate identification. Furthermore, a multi-head attention mechanism is employed for deep cross-modal feature interaction and fusion, fully exploring the biophysical correlations between spectral, visual, and environmental data, significantly improving the accuracy and early judgment capabilities of disease identification. Finally, knowledge distillation technology compresses the knowledge of a complex teacher model into a lightweight student model, using soft-label supervision to ensure its generalization performance. The efficient student model is then deployed on edge computing devices, ultimately greatly improving disease identification accuracy and enhancing generalization capabilities while ensuring practical deployment feasibility and efficiency.

[0033] 2. By using a multimodal data fusion and knowledge distillation learning framework, the accuracy, robustness, and practical deployment efficiency of crop disease identification are significantly improved. Specifically, the invention introduces a dynamic spatiotemporal alignment mechanism, which adaptively fuses multi-source and multi-resolution data, overcoming the information alignment challenges caused by inconsistent sampling frequencies and spatial scales in traditional methods, thus providing a solid foundation for multimodal collaborative modeling. Furthermore, by leveraging a cross-modal interaction module, feature interactions are established between different modalities, effectively capturing the physiological and morphological changes of crops under disease stress and enhancing the model's ability to discriminate multiple types of diseases in complex farmland environments. In addition, a knowledge distillation mechanism is introduced, distilling the rich discriminative knowledge contained in the fully trained teacher model into a lightweight student model that relies solely on RGB image data. This enables knowledge transfer between different model structures, significantly reducing computational complexity and memory consumption, and giving the student model excellent edge deployment capabilities, supporting real-time disease diagnosis in low-power field devices. This invention is not only applicable to economic crops such as citrus but can also be extended to various scenarios in field crops and facility agriculture, demonstrating good versatility and promotional value, providing reliable and easy-to-use technical support for smart agriculture.

[0034] 3. By using a spatiotemporal alignment mechanism, the problem of inconsistent temporal sequence and resolution of spectral data, RGB image data and environmental meteorological data caused by differences in sampling frequency is solved. It can adaptively fuse multi-resolution inputs to generate soft-aligned feature representations that are more consistent with the dynamic process of disease development.

[0035] 4. By using attention mechanisms and cross-modal feature mapping, the synergistic relationship between spectral features, visual features, and environmental features can be effectively captured, thereby enhancing the efficiency and reliability of multi-source information fusion.

[0036] 5. The disease response patterns and environmental information learned in the multimodal teacher model can be efficiently transferred to the student model that only relies on RGB image data. While maintaining the accuracy of disease monitoring, the system cost and computational overhead are significantly reduced, which greatly improves the applicability of the model in real field environments. Attached Figure Description

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] Figure 1 This is a flowchart of a crop disease identification method based on multimodal fusion and knowledge distillation according to the present invention. Detailed Implementation

[0039] The overall approach of the technical solution in this application is as follows: Dynamic time warping algorithm and interpolation method are used to perform refined spatiotemporal alignment of multimodal crop monitoring data, eliminating biases caused by sampling differences and laying a data foundation for accurate identification. Then, a multi-head attention mechanism is used for deep cross-modal feature interaction and fusion, fully exploring the biophysical correlations between spectral, visual, and environmental data, significantly improving the accuracy of disease identification and early judgment capabilities. Finally, knowledge distillation technology is used to compress the knowledge of the complex teacher model into a lightweight student model, using soft-label supervision to ensure its generalization performance. The efficient student model is deployed on edge computing devices to improve disease identification accuracy and enhance generalization capabilities while considering the feasibility and efficiency of practical deployment.

[0040] Please refer to Figure 1 As shown, a preferred embodiment of the crop disease identification method based on multimodal fusion and knowledge distillation of the present invention includes the following steps:

[0041] Step S1: Collect a large amount of historical multimodal crop monitoring data from crop planting sites, including spectral data, RGB image data, and environmental meteorological data;

[0042] Step S2: Preprocess the historical multimodal crop monitoring data to construct a dataset;

[0043] Step S3: Perform adaptive matching of the dataset along the time dimension;

[0044] Step S4: Create a crop disease identification model based on the feature extraction module, the cross-modal interaction unit (CMIU) module, and the prediction output module. Train the crop disease identification model using the dataset and use the trained crop disease identification model as the teacher model (Multi-head Attention for the Fusion of Multi-source Information, MAFMI). The crop disease identification model uses cross-modal contrastive learning for self-supervised pre-training. Specifically, positive samples are constructed using multimodal data (such as UAV images and environmental meteorological text) from the same plot at the same time. Negative samples are constructed by randomly replacing at least one modality of the data (such as replacing it with images from other plots). By maximizing the mutual information (similarity) of positive sample pairs in the feature space and minimizing the similarity of negative sample pairs, a strong joint feature representation between different modalities is learned.

[0045] Step S5: Create a student model, perform knowledge distillation on the teacher model to transfer the knowledge learned by the teacher model to the student model, and deploy the student model to an edge computing device located in a crop planting area.

[0046] Step S6: The edge computing device identifies crop diseases based on the deployed student model.

[0047] Step S1 specifically involves:

[0048] From crop planting sites, a large amount of historical multimodal crop monitoring data, including spectral data, RGB image data, and environmental meteorological data, is collected through a sensor group including a drone (a Mavic 3M drone equipped with a multispectral camera), a camera (RGB camera), a temperature sensor, a humidity sensor, a light sensor, and a rainfall sensor. The historical multimodal crop monitoring data are then correlated and integrated based on the timestamps and geographic coordinates of the collected data.

[0049] In step S1, the environmental meteorological data includes at least ambient temperature (Temp, °C), ambient humidity (Humidity, %PH), light intensity (Candela, CD), and rainfall (mm).

[0050] Step S2 specifically involves:

[0051] Radiometric calibration, image mosaicking, and spectral information extraction are performed on the spectral data in the historical multimodal crop monitoring data. Timestamp alignment, spatial resolution alignment, noise suppression, and outlier removal are performed on the historical multimodal crop monitoring data to complete the preprocessing of the historical multimodal crop monitoring data. A dataset is constructed based on the preprocessed historical multimodal crop monitoring data.

[0052] The radiation calibration process is as follows:

[0053] The original image in the spectral data was converted into a relative reflectance image using the intensity correction method. Then, using ENVI software, the pixel mean of each calibration plate region in the relative reflectance image was extracted. Finally, the extracted pixel mean was linearly regressed with its true reflectance value to obtain the following equation for radiometric calibration:

[0054] Ref = Gain × DN + Offset;

[0055] Where Ref represents reflectivity; Gain represents the gain coefficient of the general calibration equation, with a value of 0.0003; DN represents the original pixel value recorded by the sensor; and Offset represents the offset coefficient of the general calibration equation, with a value of 0.03.

[0056] To improve the sensitivity of spectral characteristics to diseases, the interpretation results of the spectral moisture absorption band (970nm) were corrected using environmental meteorological data. The correction formula is as follows:

[0057] ;

[0058] in, Indicates the corrected spectral characteristics; γ represents the original i-band reflectivity; β represents the correction factor, with a value of 0.05; RH represents the relative humidity; γ represents the correction factor, with a value of 0.02; Temp represents the temperature.

[0059] The noise suppression process is as follows:

[0060] Discrete Wavelet Transform (DWT) and threshold denoising are employed to eliminate transient outliers. First, the sensor time-series data is zero-mean processed, decomposing the time-domain signal into two frequency sub-bands. The low-frequency approximation coefficients (AC) retain the trend characteristics of the original signal, while the high-frequency detail coefficients (DC) contain noise, edge, and abrupt change information. Second, the standard deviation of the high-frequency detail components is calculated, with a threshold set at three times the standard deviation.

[0061] ;

[0062] in, This indicates the high-frequency detail components after noise reduction; This indicates the high-frequency detail components before noise reduction; Indicates the threshold;

[0063] pass Perform inverse wavelet transform with AC to reconstruct the denoised sensor time series data:

[0064] .

[0065] To effectively remove random noise from RGB image data caused by sensor noise, signal interference, or complex environmental conditions, an adaptive median filter (AMF) is used for targeted denoising of the RGB image data.

[0066] ;

[0067] in, This represents the pixel value after noise reduction at pixel (x, y); This represents the pixel value of pixel (x, y) before noise reduction; This represents the minimum grayscale value; Indicates the maximum grayscale value; Indicates the median gray level;

[0068] The formula for calculating spatial resolution is: d = S * H ​​ / f;

[0069] Where d represents spatial resolution; S represents pixels; H represents flight altitude; and f represents focal length.

[0070] When crops suffer from disease stress, their internal metabolic processes become disordered, leading to significant deviations in physiological parameters such as chlorophyll content, water status, and biomass accumulation from the normal range of healthy plants. Against this physiological backdrop, different spectral bands exhibit high sensitivity to specific physiological parameters. The green band (550 nm) effectively reflects changes in carotenoids; the red band (660 nm) is highly indicative of the chlorophyll a / b ratio and its content; the red-edge band (735 nm) is sensitive to nutrient stress symptoms such as nutrient deficiencies; and the near-infrared band (790 nm) is closely related to biomass accumulation and water status. Therefore, this invention selects the green, red, red-edge, and near-infrared bands as characteristic spectral data, supplemented by spatial auxiliary data in KML format, to support accurate monitoring and diagnostic analysis of crop diseases.

[0071] The timestamp alignment and spatial resolution alignment are based on interpolation.

[0072] Step S3 specifically involves:

[0073] The alignment time is calculated using a time alignment function, and the dataset is then adaptively matched in the time dimension based on the alignment time using a Dynamic Time Warping (DTW) algorithm. In other words, the DTW algorithm enables flexible alignment of multimodal data under non-uniform sampling.

[0074] The adaptive matching process is as follows:

[0075] The phenological distance function is introduced to quantify the differences in phenological periods between different time points. The phenological distance function is constructed based on historical phenological observation data and can reflect the transition patterns of key growth stages of crops. Secondly, a time alignment function is defined to align data from different sensors or different time points in time, and to find an optimal alignment time that minimizes the time alignment function.

[0076] ;

[0077] ;

[0078] in, The phenological distance function is represented by t, which represents a point in time. Indicates alignment time; The “growth status value” (represented by NVDI) indicates the vegetation health on day t. Indicates the first The "growth status value" of vegetation health on a given day; This represents the maximum value of NVDI within one period; This represents the minimum value of NVDI within a period; Indicates the time alignment function; This represents the argument of the minimum. This represents the feature vector of meteorological data at time t; Indicates spectral data in time The reflectance data feature vector; λ represents the weighting coefficient;

[0079] To overcome the limitations of traditional linear interpolation methods in dealing with dynamic changes in farmland environments, a dynamic time warping algorithm is used to solve the above optimization problem. Specifically, by constructing a cost matrix and finding the optimal curved path, elastic alignment between two sequences is achieved.

[0080] ;

[0081] ;

[0082] in, This is an environmental meteorological data sequence; It is a multispectral sequence; Align the path; This is a distance metric defined based on the above comprehensive objective function.

[0083] In step S4, the feature extraction module is used to extract spectral features, visual features, and environmental features from spectral data, RGB image data, and environmental meteorological data, respectively.

[0084] The cross-modal interaction module is used to modulate and fuse spectral features (red edge, near-infrared band, etc.), visual features (RGB image), and environmental features (temperature, humidity, and light intensity, etc.) through a multi-head attention mechanism to obtain a fused feature vector (avoiding information redundancy caused by traditional stitching and achieving efficient fusion of multimodal data).

[0085] The prediction output module is used to infer the fused feature vector and output the crop disease identification result. When the prediction output module infers the fused feature vector, it first flattens the fused feature vector into a one-dimensional vector, then maps it to the disease category space through a fully connected layer, and applies the Softmax function to output the probability of each disease category. Finally, the maximum probability is taken as the disease identification result.

[0086] The loss function of the crop disease identification model uses the InfoNCE function to narrow the embedding distance of positive samples. The formula is as follows:

[0087] ;

[0088] in, This represents the similarity score of positive samples; express Similarity to all negative samples in the batch; τ represents the temperature parameter (default 0.07); N represents the batch size.

[0089] To calculate the attention scores among different features, a multi-head attention mechanism is employed. First, a linear transformation is performed on each feature to generate a query vector Q, a key vector K, and a value vector V. For the spectral feature S, a learnable weight matrix W is used. Q W K W V Perform a linear transformation:

[0090] Q S =W Q *S;

[0091] K S =W K *S;

[0092] V S =W V *S;

[0093] The same processing is applied to environmental feature E and visual feature I to obtain Q. E K E V E and Q I K I V I Calculate the attention scores between different feature vectors. Taking spectral feature S as an example, the multimodal attention score matrix is ​​as follows:

[0094] ;

[0095] ;

[0096] Where, d k The dimension of the key vector is represented by T; T represents the transpose.

[0097] The fused features are calculated by weighted summation of the value vectors to obtain the fused feature representation. Taking spectral feature S as an example, its fused features are as follows:

[0098] ;

[0099] in, This represents the attention score matrix of the spectral feature S and itself. In actual calculation, when calculating the fused features of S, its own attention matrix will also participate in the weighting to retain some original feature information.

[0100] Perform the same operation on E and I to obtain and Finally, the three fused feature vectors are concatenated to obtain the final fused feature vector:

[0101] ;

[0102] ReLU is the activation function to enhance feature representation. It is trained with the Adam optimizer (learning rate 0.001, dropout rate 0.1) to generate a three-dimensional tensor, which is then flattened and reduced in dimensionality by fully connected layers.

[0103] Step S5 specifically involves:

[0104] A student model (ViT-Tiny) that relies solely on RGB image data is created. Soft labels are generated using the teacher model. Knowledge distillation is performed on the teacher model to transfer the knowledge learned by the teacher model to the student model. During the transfer process, the soft labels are used as supervision signals. The performance of the student model after the transfer is completed is verified. The student model that passes the performance verification is deployed to an edge computing device located in a crop planting area.

[0105] In the knowledge distillation stage, a relatively high temperature parameter (valued at 7) is set, and the teacher model is trained using soft labels and a constructed positive-negative sample contrast loss, enabling it to learn a smoother and more biologically plausible feature distribution. Finally, the knowledge contained in the teacher model is transferred to a lightweight student model that only requires RGB image data input through knowledge distillation. During inference, the student model uses a standard temperature parameter of value 1 to generate hard predictions, thereby achieving high-precision, low-latency field disease diagnosis on edge computing devices.

[0106] The student model uses the lightweight unimodal network ViT-Tiny, taking only RGB image data as input. Its goal is to mimic the "thought process" of the teacher model. The intermediate layer feature output of the student model with a temperature parameter of value 7 is named Y. S1 The feature output of the student model with a temperature parameter of 1 is named Y. S2 ;

[0107] Due to the feature output Y of the teacher model and the student model t Y S1Since the dimensions and dimensions are inconsistent, a "feature adapter" needs to be established between the feature layers of the teacher and student models. This involves adding a projection network with two fully connected layers after the intermediate layer of the student model to project the Y... S1 Projected onto Y t The same dimensions form Y S And Y S The weights are learnable;

[0108] The loss function for distillation is as follows:

[0109] ;

[0110] in, Output the standard cross-entropy loss between the student model and the real hard labels; For feature distillation loss, the projected student model features Y S Teacher model feature Y t The mean square error between them; For the student model, output Y for the temperature parameter with a value of 1. S2 , and the teacher model output Y t The KL divergence; ε=0.5, α=0.3, σ=0.2 are the equilibrium coefficients.

[0111] In practical implementation, for low-computing-power agricultural environments, a lightweight inference framework optimized for edge computing, such as PyTorch Mobile, is selected. PyTorch Mobile features a streamlined kernel structure and ARM NEON acceleration support, making it suitable for resource-constrained embedded terminals. The student model is encapsulated and operator fused using TorchScript format, improving computational efficiency and eliminating reliance on the Python runtime environment. Inference hardware adapted to edge devices is deployed, using high-performance ARM architecture single-board computers to ensure stable neural network inference capabilities under low power consumption. The optimized model and inference engine are embedded into the agricultural equipment terminal system to achieve real-time disease identification. A data access and inference pipeline is constructed, and a unified data processing interface based on Python is designed to integrate real-time information such as RGB images, ensuring low-latency inference capabilities. The system is tested in real farmland scenarios to evaluate key indicators including model accuracy and inference speed.

[0112] Step S6 specifically involves:

[0113] Edge computing devices collect real-time multimodal crop monitoring data from crop planting areas through sensor arrays. After preprocessing the real-time multimodal crop monitoring data, the data is input into a deployed student model to identify crop diseases.

[0114] In practice, a deep learning inference framework suitable for low-computing-power embedded environments can be selected, and the student model can be converted into a format supported by the target inference framework. Then, edge-side inference hardware (edge ​​computing devices) can be adapted for agricultural equipment to ensure that real-time requirements are met under power consumption constraints. Finally, the optimized student model can be deployed in the embedded system of the edge computing device to realize front-end intelligent inference capabilities.

[0115] In summary, the advantages of this invention are as follows:

[0116] 1. By collecting a large amount of historical multimodal crop monitoring data, including spectral data, RGB image data, and environmental meteorological data, the data is preprocessed to construct a dataset. Adaptive matching of the dataset's time dimension is then performed. Next, a crop disease identification model is created based on a feature extraction module, a cross-modal interaction module, and a prediction output module. This model is trained on the adaptively matched dataset. The trained model is then used as a teacher model, and knowledge distillation is performed to transfer the learned knowledge to a student model. The student model is then deployed to edge computing devices located in crop planting areas. The edge computing devices then perform operations based on the deployed student models. Crop disease identification involves using Dynamic Time Warping (DTW) and interpolation to refine the spatiotemporal alignment of multimodal crop monitoring data, eliminating biases caused by sampling differences and laying a data foundation for accurate identification. Furthermore, a multi-head attention mechanism is employed for deep cross-modal feature interaction and fusion, fully exploring the biophysical correlations between spectral, visual, and environmental data, significantly improving the accuracy and early judgment capabilities of disease identification. Finally, knowledge distillation technology compresses the knowledge of a complex teacher model into a lightweight student model, using soft-label supervision to ensure its generalization performance. The efficient student model is then deployed on edge computing devices, ultimately greatly improving disease identification accuracy and enhancing generalization capabilities while ensuring practical deployment feasibility and efficiency.

[0117] 2. By using a multimodal data fusion and knowledge distillation learning framework, the accuracy, robustness, and practical deployment efficiency of crop disease identification are significantly improved. Specifically, the invention introduces a dynamic spatiotemporal alignment mechanism, which adaptively fuses multi-source and multi-resolution data, overcoming the information alignment challenges caused by inconsistent sampling frequencies and spatial scales in traditional methods, thus providing a solid foundation for multimodal collaborative modeling. Furthermore, by leveraging a cross-modal interaction module, feature interactions are established between different modalities, effectively capturing the physiological and morphological changes of crops under disease stress and enhancing the model's ability to discriminate multiple types of diseases in complex farmland environments. In addition, a knowledge distillation mechanism is introduced, distilling the rich discriminative knowledge contained in the fully trained teacher model into a lightweight student model that relies solely on RGB image data. This enables knowledge transfer between different model structures, significantly reducing computational complexity and memory consumption, and giving the student model excellent edge deployment capabilities, supporting real-time disease diagnosis in low-power field devices. This invention is not only applicable to economic crops such as citrus but can also be extended to various scenarios in field crops and facility agriculture, demonstrating good versatility and promotional value, providing reliable and easy-to-use technical support for smart agriculture.

[0118] 3. By using a spatiotemporal alignment mechanism, the problem of inconsistent temporal sequence and resolution of spectral data, RGB image data and environmental meteorological data caused by differences in sampling frequency is solved. It can adaptively fuse multi-resolution inputs to generate soft-aligned feature representations that are more consistent with the dynamic process of disease development.

[0119] 4. By using attention mechanisms and cross-modal feature mapping, the synergistic relationship between spectral features, visual features, and environmental features can be effectively captured, thereby enhancing the efficiency and reliability of multi-source information fusion.

[0120] 5. The disease response patterns and environmental information learned in the multimodal teacher model can be efficiently transferred to the student model that only relies on RGB image data. While maintaining the accuracy of disease monitoring, the system cost and computational overhead are significantly reduced, which greatly improves the applicability of the model in real field environments.

[0121] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A crop disease identification method based on multimodal fusion and knowledge distillation, characterized in that: Includes the following steps: Step S1: Collect a large amount of historical multimodal crop monitoring data from crop planting sites, including spectral data, RGB image data, and environmental meteorological data; Step S2: Preprocess the historical multimodal crop monitoring data to construct a dataset; Step S3: Perform adaptive matching of the dataset in the time dimension; calculate the alignment time using the time alignment function, and perform adaptive matching of the dataset in the time dimension based on the alignment time using the dynamic time warping algorithm; Specifically, based on historical phenological observation data, a phenological distance function is constructed to quantify the differences in phenological periods between different time points and reflect the transition patterns of key growth stages of crops; a time alignment function is defined to time-align data from different sensors or different time points, find the optimal alignment time, and minimize the time alignment function. ； ； in, The phenological distance function is represented by t, which represents a point in time. Indicates alignment time; The NVDI value represents the vegetation health status on day t. Indicates the first NVDI of vegetation health; This represents the maximum value of NVDI within one period; This represents the minimum value of NVDI within a period; Indicates the time alignment function; Indicates the parameter to be minimized; This represents the feature vector of meteorological data at time t; Indicates spectral data in time The reflectance data feature vector; λ represents the weighting coefficient; The dynamic time warping algorithm solves this problem by constructing a cost matrix and finding the optimal curved path to achieve elastic alignment between two sequences. ； ； in, This is an environmental meteorological data sequence; It is a multispectral sequence; Align the path; For distance measurement; Step S4: A crop disease identification model is created based on the feature extraction module, cross-modal interaction module, and prediction output module. The crop disease identification model is trained using the dataset, and the trained model is used as the teacher model. The feature extraction module extracts spectral features, visual features, and environmental features from spectral data, RGB image data, and environmental meteorological data, respectively. The cross-modal interaction module modulates and fuses the spectral features, visual features, and environmental features using a multi-head attention mechanism to obtain a fused feature vector. The prediction output module infers the fused feature vector and outputs the crop disease identification result. When inferring the fused feature vector, the prediction output module first flattens the fused feature vector into a one-dimensional vector, then maps it to the disease category space through a fully connected layer, and applies the Softmax function to output the probability of each disease category. Finally, the maximum probability is taken as the disease identification result. The loss function of the crop disease identification model is the InfoNCE function. Step S5: Create a student model that relies solely on RGB image data; perform knowledge distillation on the teacher model to transfer the knowledge learned by the teacher model to the student model; and deploy the student model to an edge computing device located in a crop planting area. Step S6: The edge computing device identifies crop diseases based on the deployed student model.

2. The crop disease identification method based on multimodal fusion and knowledge distillation as described in claim 1, characterized in that: Specifically, step S1 involves collecting a large amount of historical multimodal crop monitoring data, including spectral data, RGB image data, and environmental meteorological data, from the crop planting area using a sensor group including drones, cameras, temperature sensors, humidity sensors, light sensors, and rainfall sensors. The historical multimodal crop monitoring data is then correlated and integrated based on the collected timestamps and geographic coordinate information.

3. The crop disease identification method based on multimodal fusion and knowledge distillation as described in claim 1, characterized in that: In step S1, the environmental meteorological data includes at least ambient temperature, ambient humidity, light intensity, and rainfall.

4. The crop disease identification method based on multimodal fusion and knowledge distillation as described in claim 1, characterized in that: Specifically, step S2 involves performing radiometric calibration, image mosaicking, and spectral information extraction on the spectral data in each of the historical multimodal crop monitoring data. It also involves performing timestamp alignment, spatial resolution alignment, noise suppression, and outlier removal on each of the historical multimodal crop monitoring data to complete the preprocessing of each of the historical multimodal crop monitoring data. Finally, a dataset is constructed based on the preprocessed historical multimodal crop monitoring data.

5. The crop disease identification method based on multimodal fusion and knowledge distillation as described in claim 1, characterized in that: The timestamp alignment and spatial resolution alignment are based on interpolation.

6. The crop disease identification method based on multimodal fusion and knowledge distillation as described in claim 1, characterized in that: Specifically, step S5 involves creating a student model, generating soft labels using the teacher model, performing knowledge distillation on the teacher model to transfer the knowledge learned by the teacher model to the student model, using the soft labels as supervision signals during the transfer process, performing performance verification on the student model after the transfer is completed, and deploying the student model that has passed the performance verification to an edge computing device located in a crop planting area.

7. The crop disease identification method based on multimodal fusion and knowledge distillation as described in claim 1, characterized in that: Specifically, step S6 involves the edge computing device collecting real-time multimodal crop monitoring data from the crop planting area through a sensor group, preprocessing the real-time multimodal crop monitoring data, and then inputting it into the deployed student model to identify crop diseases using the student model.

Images