Multimodal based early crop disease detection method

Abstract

The present application is suitable for the field of modern agriculture and computer technology, and provides a crop early disease detection method based on multi-modal, which comprises the following steps: based on a convolutional neural network, a gated recurrent unit and a multi-head attention mechanism layer, a crop disease detection model is constructed; based on established experimental conditions, leaf reflectance spectrum measurement and leaf physiological variable measurement are performed to realize data collection; collected data are used as sample data for model training and testing, and the sample data are screened and pretreated to obtain a sample data set; the crop disease detection model is trained and tested through the sample data set; crop detection data are input into the trained crop disease detection model, and an output result is obtained; the present application integrates GRU and CNN, can effectively capture spatial and temporal features from data, and combines spectrum and physiological data and data conversion, so that ShB infection can be accurately detected early.

Description

Technical Field

[0001] This invention belongs to the fields of modern agriculture and computer technology, and particularly relates to the field of deep learning technology. Specifically, it provides a method for early detection of crop diseases based on multimodal approaches. Background Technology

[0002] Plant diseases pose a significant threat to food security, and rice sheath blight (ShB) is a destructive pathogen. Sheath blight is a major disease in rice, severely impacting yield and productivity, especially in intensive cultivation, where it can lead to yield reductions of 20% to 50%, depending on the severity of infection. The surge in its prevalence is primarily due to the widespread planting of high-yielding semi-dwarf rice varieties, excessive nitrogen fertilizer use, and the favorable microenvironment created by high crop density. Sheath blight mainly targets the leaves and sheaths of rice plants. Depending on environmental factors, infected plants show symptoms within 24-144 hours, with the greatest damage occurring during the tillering stage.

[0003] Routine diagnosis of diseases such as sheath blight relies heavily on phenotypic observation and manual examination, methods that are often subjective and prone to error. Furthermore, these methods typically identify the disease at a late stage, after substantial damage has already occurred. This delay in detection severely hinders effective disease management and control measures. Recent advances in detection technologies such as molecular diagnostics have improved the accuracy and timeliness of disease detection, but they still face challenges in terms of cost, complexity, and scalability, making them unsuitable for high-throughput, field-based disease diagnosis. These challenges underscore the need for precise, efficient, and non-invasive technologies capable of detecting early, asymptomatic infections.

[0004] Recent advances in spectroscopic techniques and machine learning have opened new avenues for plant disease detection, enabling non-destructive, rapid, and accurate monitoring. Near-infrared spectroscopy (NIR), in particular, has become a powerful tool in this field due to its ability to capture a wide range of spectral information beyond the capabilities of conventional imaging techniques. NIR utilizes the unique spectral characteristics of plants to detect subtle physiological and biochemical changes caused by diseases. This non-destructive technique captures reflectance data across a continuous spectrum, enabling the identification of specific wavelengths associated with disease symptoms. Furthermore, handheld instruments, similar to those used in current research, require minimal operator training and are relatively cost-effective. Recent studies have demonstrated the potential of hyperspectral imaging in diagnosing various plant diseases, including rice blast, bacterial blight, and the bacterial disease rice leaf streak, through the analysis of changes in spectral reflectance.

[0005] While some studies have investigated the estimation of ShB disease severity and the identification of leaf samples infected in the late stages of infection, few studies have focused on spectroscopic detection of ShB in the early stages of infection and examination of leaf biochemical properties. Some studies have demonstrated that early responses to plant biotic stress, manifested as changes in plant physiological parameters or photosensitivity mechanisms, can be detected using sensitive spectroscopic indicators. Subtle spectral and physiological changes can pose a significant challenge to the early and definitive identification of infected samples. Reflectance spectroscopy is beneficial for revealing subtle spectral signals induced by complex plant pathogen infections. Characteristic band extraction, as a simple and effective tool for characterizing spectral changes, is also used to highlight factors of interest and suppress the influence of other factors, making it a popular tool for plant disease detection.

[0006] Previous research has primarily focused on the independent use of hyperspectral or physiological measurements. However, the integration of these two modalities, leveraging their synergistic potential, remains largely unexplored. Therefore, it is possible to explore using advanced mathematical computational methods to combine these data sources to improve the accuracy and efficiency of ShB detection. In this context, the potential of advanced data analysis techniques such as deep learning remains largely untapped.

[0007] Recent studies have demonstrated the significant advantages of machine learning in plant disease detection when using different sensitive indicators as inputs. While these studies have provided all spectral features or extracted biochemical / biophysical features to construct disease models, they have not attempted to optimize the combination of input variables for more effective plant disease detection. Furthermore, a deep understanding of the contribution of relevant spectral features to disease detection is lacking. Moreover, the studies have not adequately explored the effective combination of feature bands with key physiological parameters to improve the accuracy of disease diagnosis. Previous research has shown that carefully selecting the combination of feature subsets can significantly improve the efficiency of classification models without significantly reducing classification accuracy. In addition, many studies have found that the number of input variables or spectral features has a significant impact on the performance of machine learning algorithms. However, research on the application of multimodal strategies that fuse spectral information with physiological data in disease diagnosis remains limited. Therefore, methods that integrate spectral and physiological features and optimize the combination of input variables warrant further exploration in future research.

[0008] Recent advances in machine learning and deep learning have further enhanced the potential of these diagnostic methods. Convolutional neural networks (CNNs) and gated recurrent units (GRUs) have been successfully applied to process spectral data and extract meaningful features for disease classification. However, a review reveals that conventional research methods largely focus on the arrangement and analysis of one-dimensional data, failing to fully leverage the advantages of image recognition, exemplified by CNNs, and warrant further exploration. Summary of the Invention

[0009] The purpose of this invention is to provide a multimodal method for early crop disease detection. By building a multimodal prediction model, namely a crop disease detection model, it effectively integrates disease-specific spectral features and key co-physiological parameters to capture subtle pre-symptom changes and achieve high accuracy in early ShB infection detection; and to provide an exploratory scheme for the application of deep learning in early crop disease detection.

[0010] This invention is implemented as follows: a multimodal method for detecting early crop diseases, the method comprising the following steps:

[0011] A crop disease detection model is constructed based on a dual-branch architecture that integrates convolutional neural networks, gated recurrent units, and multi-head attention mechanisms.

[0012] Based on the established experimental conditions, leaf reflectance spectrum and leaf physiological variables were measured to collect data.

[0013] The collected data is used as sample data for model training and testing, and the sample data is filtered and preprocessed to obtain a sample dataset.

[0014] The crop disease detection model is trained and tested using the sample dataset.

[0015] The crop detection data is input into the trained crop disease detection model, and the early crop disease detection results are output.

[0016] Furthermore, the experimental conditions were constructed based on an RXZ-1000 artificial climate chamber, and the crop used in the experiment was Wuyou Rice No. 4.

[0017] The RXZ-1000 artificial climate chamber is equipped with:

[0018] High-precision lighting: 0-8000 lux;

[0019] Temperature control: 0-50℃ ± 0.1℃ fluctuation;

[0020] Automatic humidity control: 50-95%, ±3% fluctuation.

[0021] Automated day / night cycle control system: 0-99 hours;

[0022] Crops were divided into a control group, a simulated infection group, and an actual infection group, and inoculated based on a specified infection technique;

[0023] The specified infection techniques are one of the following: detached leaf method, microchamber method, mist chamber method, mycelial suspension spraying method, and mycelial embedding method.

[0024] Furthermore, the leaf reflectance spectrum is measured using a spectrometer, and the photosynthetic parameters of the leaves are collected in situ using a portable photosynthesis meter in a non-invasive manner. The chlorophyll fluorescence in the leaves is non-destructively evaluated using a portable modulated chlorophyll fluorescence meter under natural and dark conditions to obtain fluorescence parameters. The physiological variables of the leaves are measured by combining the photosynthetic parameters and fluorescence parameters.

[0025] The spectrometer used was a handheld ASD FieldSpec2 (HH2) spectrometer, which accurately captured spectra between 325 and 1075 nm with a sampling interval of 1.5 nm. The acquired spectral data could be processed using ViewSpec Pro software V6.20, which simplified the collection, visualization and analysis of spectral data.

[0026] The portable photosynthesis system (model LI-6800) features a circular leaf chamber (circular, 2 square centimeters in area, emitting wavelengths of 660 nm and 453 nm) with a 6800-02 red and blue light source, enabling robust control of environmental conditions. This device calculates key physiological parameters based on the mass balance of carbon dioxide and water vapor within the leaf chamber; these parameters include the net photosynthetic rate (A, μmol·m⁻¹). -2 ·s -1 ), transpiration rate (E, mmol·m -2 ·s -1 ), porosity (Cond, mol·m) -2 ·s -1 ), Total water conductivity (CondTotal, molH2O·m -2 ·s -1 ), CO2 total conductivity (CondCO2, molH2O·m -2 ·s -1 ), intercellular carbon dioxide concentration (Ci, μmol·mol) -1 Leaf surface temperature (Tleaf, °C), leaf-to-air temperature difference (Ti-Ta, °C), relative humidity and CO2 concentration on leaf surface (RHsfc, % and C2sfc, μmol·mol⁻¹) -1 )wait.

[0027] To ensure accuracy, the portable photosynthesis system is calibrated in a climate chamber with the same settings as the growth chamber, allowing for rapid and accurate control of environmental variables such as carbon dioxide concentration, light intensity, humidity, and temperature. This setup minimizes the impact of external factors on measurements. The measurement points of the portable photosynthesis system are consistent with the previously mentioned spectral measurement points, and the average data from three points on each leaf is used to represent its photosynthetic parameters. Data can be collected, visualized, and output using FlashAnalysis-v1.0 and Microsoft Excel 2019 software.

[0028] The portable modulated chlorophyll fluorometer (model OS1P) integrates a comprehensive light source system, including adjustable LED white light, modulated red light, and far-red light. It also features a photosynthetically active radiation (PAR) clip with a sensor and thermistor for accurate environmental readings.

[0029] Under illumination, maximum fluorescence (Fms), steady-state fluorescence (Fs), photosynthetically active radiation (PAR), actual photochemical quantum yield (Y(II)), and electron transport rate (ETR) were measured to capture light-induced excitation. Measurements were carefully configured to ensure accuracy: modulation intensity was adjusted to 50%, saturation intensity to 75%, flash duration was fixed at 0.8 seconds, and automatic gain control was activated.

[0030] After 30 minutes of dark adaptation, the initial fluorescence (F0), maximum fluorescence (Fm), variable fluorescence (Fv), maximum photochemical quantum yield (Fv / Fm), and potential PSII activity (Fv / Fo) were precisely measured in situ using a dark-adapted leaf clamp. Measurement settings included: modulation intensity of 25%, saturation intensity of 75%, flash duration of 0.8 seconds, and automatic gain control enabled. Additionally, the far-red light intensity was set to 50% for 5 seconds, a key adjustment for activating photosystem I (PSI) and promoting rapid re-oxidation of photosystem II (PSII); other settings were default. Fluorescence parameters were efficiently collected, analyzed, and presented using Microsoft Excel 2019 and Origin 2022 software.

[0031] Furthermore, the steps for constructing a crop disease detection model based on a dual-branch architecture integrating convolutional neural networks, gated recurrent units, and multi-head attention mechanisms specifically include:

[0032] In the convolutional neural network branch, an image input layer is set up, and the image input layer is connected to the convolutional layer. The convolutional kernel and stride of the convolutional layer are set. The convolutional layer, batch normalization layer, pooling layer and fully connected layer are connected in sequence. The activation function ReLU is introduced between the batch normalization layer and the pooling layer as a corrected linear unit. A flat layer is connected after the fully connected layer.

[0033] In the gated loop unit branch, the image input layer and the gated loop unit are connected, and a fully connected layer and a flat layer are also set between the image input layer and the gated loop unit;

[0034] In the multi-head attention mechanism layer, a feature fusion layer is used as the input layer, which is connected to a flat layer of convolutional neural network and gated recurrent unit. The output of the feature fusion layer is fed into a fully connected layer, and then connected to a normalized exponential layer and a sheath blight classification layer.

[0035] Furthermore, the steps of filtering and preprocessing the sample data specifically include:

[0036] Data screening: Monte Carlo partial least squares and interquartile range methods were used to screen outliers in spectral data and physiological parameters;

[0037] Data preprocessing: Combining two or more data processing techniques; the two or more data processing techniques are: continuous wavelet transform, multivariate scatter correction, Savitzky-Golay convolutional smoothing, and minimum-maximum normalization.

[0038] Furthermore, in the step of using the collected data as sample data for model training and testing, the one-dimensional sequence is converted into a two-dimensional image by converting Cartesian coordinates to polar coordinates and calculating the angle between data point vectors based on Gram angles and fields (GASF).

[0039] Furthermore, the method also includes:

[0040] The performance of the crop disease detection model was analyzed; the performance evaluation indicators included average accuracy, recall, precision, and F1 score.

[0041] This invention provides a multimodal method for early crop disease detection. The crop disease detection model is a deep learning model (MMCG-MHA) integrating GRU and CNN in a dual-branch multimodal manner and combined with a multi-head self-attention mechanism. It can utilize Gram angle sum and field (GASF) to transform single-modal data into two-dimensional images, achieving data layer or feature layer fusion, and can effectively capture spatial and temporal features from multimodal data. When detecting ShB infection in rice leaves, it can combine leaf spectral and physiological data with data transformation, thereby enabling early and accurate detection of ShB infection. In addition, the crop disease detection model can be used not only for rice sheath blight detection, but also for various other diseases caused by fungi, making it widely applicable. Attached Figure Description

[0042] Figure 1 This is a flowchart of a multimodal-based method for detecting early crop diseases in one embodiment;

[0043] Figure 2 Images of rice leaves at various disease severity levels as defined in one embodiment;

[0044] Figure 3 This is a schematic diagram of the MMCG-MHA model used for early detection of rice sheath blight in one embodiment;

[0045] Figure 4This is a waterfall plot showing the changes in spectral reflectance over time (days 0-6) for the control group (CK), simulated disease group (SDG), and infected group (IDG) in one embodiment.

[0046] Figure 5 This is a heatmap showing the correlation between photosynthetic parameters (a), fluorescence parameters (b), and ShB in one embodiment;

[0047] Figure 6 This is a linear graph showing the changes of various physiological indicators (A, Ci, E, Ti-Ta, RHsfc, C2sfc, Y(II), ETR) during the duration of the disease in one embodiment.

[0048] Figure 7 and Figure 8 Here is an ANOVA analysis of key cophysiological parameters and a Tukey HSD dual Y-axis horizontal bar chart for comparison in one embodiment. Figure 7 It shows the average values ​​of different parameters across individual treatment groups. Figure 8 The average difference between the dual-treatment groups was shown;

[0049] Figure 9 This is a schematic diagram of GASF transformation for 27 combinations of variables in one embodiment;

[0050] Figure 10 Confusion matrix diagram of the classification accuracy of optimal treatment CARS-IVISSA in distinguishing rice health status and ShB disease duration in an MMCG-MHA model of one embodiment;

[0051] Figure 11 This is a schematic diagram of the sensitive bands selected for different feature band extraction methods in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script.

[0054] In one embodiment, such as Figures 1 to 11 As shown, a multimodal method for detecting early crop diseases may include the following steps:

[0055] A crop disease detection model is constructed based on a dual-branch architecture that integrates convolutional neural networks, gated recurrent units, and multi-head attention mechanisms.

[0056] Based on the established experimental conditions, leaf reflectance spectrum and leaf physiological variables were measured to collect data.

[0057] The collected data is used as sample data for model training and testing, and the sample data is filtered and preprocessed to obtain a sample dataset.

[0058] The crop disease detection model is trained and tested using the sample dataset.

[0059] The crop detection data is input into the trained crop disease detection model, and the early crop disease detection results are output.

[0060] In one example of this embodiment, the experimental conditions have been established:

[0061] Built on the RXZ-1000 artificial climate chamber, the rice variety used in the experiment was Wuyou Rice No. 4, also known as Daohuaxiang No. 2.

[0062] The RXZ-1000 artificial climate chamber is equipped with:

[0063] High-precision lighting: 0-8000 lux;

[0064] Temperature control: 0-50℃ ± 0.1℃ fluctuation;

[0065] Automatic humidity control: 50-95%, ±3% fluctuation.

[0066] Automated day / night cycle control system: 0-99 hours;

[0067] Crops were divided into a control group, a simulated infection group, and an actual infection group, and inoculated based on a specified infection technique;

[0068] The specified infection techniques are one of the following: detached leaf method, microchamber method, mist chamber method, mycelial suspension spraying method, and mycelial embedding method.

[0069] For example, the experiment included rice cultivation and artificial inoculation.

[0070] The experiment was conducted from June to August 2022 in a glass greenhouse at the Nanling Campus of Jilin University, Changchun, China (44°50′N, 125°18′E), using a dedicated RXZ-1000 artificial climate chamber manufactured by Jiangnan Instrument Factory. It comprised two aspects: two distinct phases of infection experiments, beginning on June 25th and July 15th respectively. Initially, 100 tillering-stage rice plants were used to determine the most effective infection technique from five different methods. Subsequently, the second phase involved setting up 80 barrels in three distinct groups: a control group, a simulated infection group, and an actual infection group, with 20, 20, and 40 barrels allocated to each group, respectively. To reduce the risk of cross-infection, each group was isolated separately in an RXZ-1000 artificial climate chamber, with 20 barrels placed in each chamber.

[0071] The RXZ-1000 artificial climate chamber described above allows for meticulous control of the entire rice growth process and Rhizoctonia solani infection. For example:

[0072] Throughout the nursery stage, the experiment followed these steps: selecting high-quality seeds, drying, soaking, promoting germination, sowing, managing seedlings, transplanting, regular watering, fertilizing, and applying insecticides, but not fungicides. A crucial adjustment was made before sowing to enhance seed health: the seeds were first soaked in clean water for 12 hours to ensure full hydration. Subsequently, the seeds were soaked in a 25% cypermethrin solution, diluted 2000 to 4000 times, for 10–12 hours. This disinfection process aimed to effectively eradicate surface pathogens before germination and sowing, ensuring a healthier start for the seedlings.

[0073] Five artificial inoculation techniques for rice sheath blight were evaluated. These methods included detached leaf inoculation, microchamber inoculation, mist chamber inoculation, mycelial suspension spraying, and mycelial embedding. For example, after preliminary experiments, mycelial embedding was chosen due to its moderate disease progression rate and ease of operation. The experiment was set up with three groups: a blank control group (CK), a simulated inoculation group (SDG) using PDA (potato dextrose agar) blocks without pathogens, and an actual inoculation group (IDG) using PDA blocks cultured with Rhizoctonia solani. The inoculation method and location could also be flexibly adjusted according to the actual situation. Two 0.5 cm diameter circular blocks of Rhizoctonia solani mycelium cultured on PDA blocks for 72 hours were used, with the mycelium facing the stem, and inoculated or simulated inoculated at the base of the stem of seedlings approximately four weeks old. As a control, the plants remained untreated. After inoculation, specific environmental conditions were maintained in an artificial climate chamber as described above.

[0074] In this experiment, Daohuaxiang No. 2 rice variety was chosen because its specific characteristics were suitable for the research objective. After four weeks of rice cultivation, only healthy seedlings with similar growth conditions were carefully selected. These selected seedlings were then transplanted into plastic cultivation containers (25.5 cm long, 16.7 cm wide, and 26.5 cm high), with each container holding four seedlings, for the infection experiment. The Rhizoctonia solani strain used was a second-generation standard strain provided by the Department of Microbiology, Nankai University, and produced by Beijing Baozang Biotechnology Co., Ltd., ensuring high standards and consistency in the infection process.

[0075] Before the infection process began, the conditions in the artificial climate chamber were carefully adjusted to reflect the conditions of the natural field environment. During infection, in order to closely mimic the environmental conditions in which sheath blight is highly prevalent, especially in humid and rainy weather, the environment of the artificial climate chamber was adjusted to maintain a temperature of 28±0.2℃, a relative humidity of 95±3%, a light intensity of 4000-4500LUX, and a day-night cycle of 12 hours of light followed by 12 hours of darkness.

[0076] In addition, in one example, the severity of diseases in rice is defined, such as... Figure 2 As shown, leaves without any lesions are considered healthy. Figure 2 (a) The early, asymptomatic stage refers to the period before visible lesions appear on the leaves. During this stage, although the lesions are not obvious, they are characterized by a few water-soaked lesions, usually one or two, which are grayish-green in color, while the rest of the leaf surface remains unaffected. Figure 2 (b). In contrast, the severe infection stage can be identified by the appearance of multiple irregularly shaped marbled lesions on the leaf surface, characterized by a whitish center and brown edges. Figure 2 (c). To avoid misclassification, continuous visual inspections were conducted throughout the study. Representative rice samples were selected prior to inoculation and subjected to in-depth spectroscopic and biochemical analyses from pre-inoculation (Heal), early asymptomatic (days post-inoculation (DAI), DAI 1–5), to severe infection (ShB, DAI 6–7).

[0077] In one example of this embodiment, the collected data is used as sample data for model training and testing, and the sample data is filtered and preprocessed to obtain a sample dataset;

[0078] This step includes data screening: using Monte Carlo partial least squares and interquartile range methods to screen outliers in spectral data and physiological parameters;

[0079] Data preprocessing: Combining two or more data processing techniques; the two or more data processing techniques are: continuous wavelet transform, multivariate scatter correction, Savitzky-Golay convolutional smoothing, and minimum-maximum normalization.

[0080] Eliminating outliers can prevent data distortion and its impact on subsequent analyses caused by measurement errors, instrument malfunctions, or sample characteristics. By employing appropriate outlier detection methods, such as statistical or machine learning techniques, spectral and physiological parameters that do not conform to the data distribution can be effectively identified and removed, thus providing an accurate and robust foundation for subsequent model building and data interpretation.

[0081] For example, Monte Carlo partial least squares (MC-PLS) and interquartile range (IQR) methods are used to screen outliers in spectral data and physiological parameters.

[0082] Preprocessing of spectral data: Continuous wavelet transform (CWT), multivariate scatter correction (MSC), Savitzky-Golay convolutional smoothing (SG), and minimum-maximum normalization (MMN) can be any or any combination of these schemes;

[0083] The selection was based on their respective unique advantages. CWT is effective for baseline correction, while MSC precisely targets scattering problems while minimizing the effects of baseline drift. SG plays a crucial role in reducing noise in spectral data, thus significantly improving the signal-to-noise ratio. Finally, MMN normalizes the scale of the dataset, making it easier to identify overlapping spectral features and ensuring that all data points are scaled to the same numerical range.

[0084] In one example of this embodiment, the preprocessed data is randomly divided into a training set (66.7%) and a test set (33.3%) for subsequent processing, model training, and testing, etc.

[0085] In one example of this embodiment, feature selection techniques such as Monte Carlo Partial Least Squares (MC-PLS) and Interquartile Range (IQR) separate disease-specific spectral features, which may have different effects on the model results.

[0086] This example allows for the evaluation of such effects; namely, the effectiveness of the two feature selection techniques and the impact of interval variables on model validity, and the determination of the optimal combination of interval and individual variable selection, as well as the optimal combination of features and model, to improve model robustness.

[0087] Furthermore, five interval variable feature extraction methods can be used: Interval Combination Optimization (ICO), Interval Partial Least Squares (IPLS), Backward Interval Partial Least Squares (BIPLS), Collaborative Interval Partial Least Squares (SIPLS), and Interval Variable Iterative Space Shrinkage (IVISSA), to compare the importance of interval variables in each method; and two individual variable selection techniques can be used: Competitive Adaptive Reweighted Sampling (CARS) and Continuous Projection Algorithm (SPA).

[0088] In one example of this embodiment, such as Figure 3 As shown, the steps for constructing a crop disease detection model based on a dual-branch architecture integrating convolutional neural networks, gated recurrent units, and multi-head attention mechanisms specifically include:

[0089] In the convolutional neural network branch, an image input layer is set up, and the image input layer is connected to the convolutional layer. The convolutional kernel and stride of the convolutional layer are set. The convolutional layer, batch normalization layer, pooling layer and fully connected layer are connected in sequence. The activation function ReLU is introduced between the batch normalization layer and the pooling layer as a corrected linear unit. A flat layer is connected after the fully connected layer.

[0090] The image input layer can be defined using the `imageInputLayer` function, receiving a color image of size [227, 227, 3]. In the convolutional layers, 3x3 kernels are used with padding set to 3 to preserve the spatial dimension of the feature map. The stride is set to 2, effectively reducing the spatial dimension of the feature map while preserving important edge information through the padding strategy. This convolutional operation captures local spatial features, providing foundational features for subsequent deeper network layers. For the ReLU activation function, negative values ​​are set to 0 to alleviate gradient vanishing and enhance the representation of complex functions. A fully connected layer (FC) and a flattened layer (flat1) map the high-dimensional features to a 128-dimensional fully connected layer FC1000; then flattening is performed to provide input for classification or other tasks.

[0091] In the gated loop unit (GRU) branch, the image input layer and the gated loop unit are connected, and a fully connected layer and a flat layer are also set between the image input layer and the gated loop unit;

[0092] The Gated Recurrent Unit (GRU) enables the model to remember relevant information in long sequences while ignoring irrelevant parts. Initially, data is input through an image input layer, allowing the model to receive one-dimensional sequence data with variable features. Next, the data is normalized to ensure standardization, optimizing the model's convergence and generalization capabilities. Then, a flat layer (flat2) maps multi-dimensional features to one-dimensional vectors, providing the necessary data structure for deep processing within the model. The GRU is configured with 128 output dimensions to further capture dynamic features in the sequence data, efficiently processing the information flow of long sequences through its internal update and reset gate mechanism.

[0093] In the multi-head attention mechanism layer, a feature fusion layer is used as the input layer and connected to a flat layer of convolutional neural network and gated recurrent unit. The output of the feature fusion layer is fed into a fully connected layer, and then connected to a normalized exponential layer and a sheath blight classification layer. The feature fusion layer uses the addLayers function.

[0094] Specifically, the `connectLayers` function combines the outputs of the GRU and CNN with a fusion layer to integrate spatiotemporal features. The feature map is planarized into a vector and fed into a fully connected layer (FC1000) for feature extraction, adapting to the number of classes. Finally, a normalized exponential layer (softmax(Prob)) transforms the feature vector into a probability distribution, which is then used by a sheath blight classification layer (ShBclassification) to generate classification predictions. The normalized exponential layer ensures that the sum of probabilities is 1, making it suitable for multi-class tasks.

[0095] Such a crop disease detection model can process raw one-dimensional numerical data as well as two-dimensional Gram angle and field (GASF) images, which are mapped through specific transformations to adapt to the complexity of multimodal data; and the multi-head attention mechanism layer allows the model to process attention in parallel in multiple subspaces, enhancing its ability to build rich feature relationships and identify feature contributions between different locations.

[0096] The multi-head attention mechanism layer is configured with multiple attention heads and 50 value channels, with an output size of 100.

[0097] In one example of this embodiment, a 10x cross-validation method was used to evaluate the model's generalization ability on different subsets of data. Evaluation metrics included mean accuracy, recall, precision, and F1 score, providing a comprehensive understanding of the model's performance across various scenarios and assessing its potential and effectiveness in real-world applications.

[0098] In one example of this embodiment, such as Figure 4As shown, analysis of the raw spectral data revealed that the average spectral reflectance of the infected group samples gradually decreased with the progression of the disease, reaching its lowest point at the onset of lesions. In contrast, the reflectance of the control group and the simulated disease group increased slightly over time. Therefore, the spectral data of the control group and the simulated disease group were classified as the healthy (Heal) group. Although there were no significant differences in the overall shape of the near-infrared (NIR) spectra between the different treatments, differences in the average NIR reflectance intensity were observed in certain spectral regions. These differences can be effectively used to diagnose the duration of the disease.

[0099] For example, to analyze the relationship between photosynthetic and fluorescence parameters, Pearson correlation analysis is used to select key indicators, avoid redundancy, and improve analytical efficiency. Figure 5 As shown, each cell in the correlation matrix represents a parameter, and color intensity and cell size indicate the strength of the correlation. Darker colors indicate stronger correlations; red indicates a positive correlation, and blue indicates a negative correlation. ShB shows a high positive correlation with RHsfc, Ti-Ta, Ci, and C2sfc, and a high negative correlation with A, E, Y(II), and ETR. Therefore, these eight highly significant indicators were selected as cophysiological parameters. These indicators, along with spectral features, were input into the model to verify the improved accuracy.

[0100] In the study of rice sheath blight, the changes of various physiological indicators at different time points and between different groups (control group (CK), simulated inoculation group (SDG), and actual inoculation group (IDG)) provide a basis for a deeper understanding of the impact of the disease. Figure 6 The changes of various physiological indicators (A, Ci, E, Ti-Ta, RHsfc, C2sfc, Y(II), ETR) during the disease were shown, and the results between different groups were compared.

[0101] Specifically, CK group A remained relatively stable, ranging from 5.74 to 6.12 μmol·m⁻¹. -2 ·s -1 The values ​​fluctuated slightly within the range. The SDG and IDG groups were similar to the CK group in the early stages of the disease, but the A value in the IDG group decreased significantly in the later stages, reaching its lowest point of 11.65 and 11.34 μmol·m⁻² at DAI 2 and 3, respectively. -2 ·s -1 It further decreased to 4.88 μmol·m at ShB (sixth day post-vaccination). -2 ·s -1This indicates that the disease has a significant inhibitory effect on photosynthesis. This trend suggests that the impact of the disease on the photosynthetic rate becomes more pronounced with increasing infection time. The Ci values ​​in the CK and SDG groups generally showed relatively stable small fluctuations, but the Ci value in the IDG group continued to rise, significantly increasing to 201.98 μmol·mol⁻¹ at ShB (six days after inoculation). -1 This indicates that the early stages of the disease did not significantly affect intercellular CO2 concentration, but gas exchange was significantly disrupted as the disease progressed. The E values ​​of the CK and SDG groups showed consistent performance, fluctuating within a relatively stable range. The E value of the IDG group was not significantly different from that of the CK group in the early stages, but decreased significantly at DAI 2 and 3, and then gradually recovered to a higher level. This is related to water use and stomatal regulation; the disease may affect leaf water balance and photosynthetic efficiency. Ti-Ta showed small differences among the groups; the differences between the CK and SDG groups were not significant, while the temperature difference in the IDG group increased slightly in the later stages of the disease, indicating that the disease reduced the transpiration rate, decreased leaf cooling effect, and led to increased leaf temperature. The RHsfc value of the CK group remained stable, while the relative humidity increased in the SDG and IDG groups in the later stages of the disease, especially with a significant increase in DAI 5. This may be related to disease-induced stomatal closure and transpiration inhibition, affecting leaf surface water dynamics and altering the leaf surface microenvironment. C2sfc remained relatively stable, with slight increases in the CK and SDG groups. In the IDG group, C2sfc levels significantly increased during the course of the disease, reaching 295.87 μmol·mol⁻¹ at ShB (six days post-vaccination). -1 This indicates that the disease affected gas exchange and CO2 accumulation in the leaves, possibly due to CO2 accumulation caused by stomatal closure or increased CO2 release induced by the pathogen. Y(II) performance remained relatively stable in the CK and SDG groups. However, the actual photometric quantum yield of Y(II) in the IDG group decreased slightly in the later stages of the disease, indicating a negative impact of the disease on the photosystem II electron transport chain, further confirming the decrease in net photosynthetic rate. The ETR remained stable in the CK and SDG groups, but the IDG group showed significant fluctuations during the disease period, reaching 38.83 μmol·m⁻³ at DAI 3. -2 ·s -1 The peak value was reached, and then gradually decreased. This indicates that the disease is increasingly interfering with electron transport involved in photosynthesis, further affecting photosynthetic efficiency.

[0102] In summary, significant differences were observed in the changes of various physiological indicators among the different treatment groups during the disease process. The CK and SDG groups remained stable in most physiological indicators, indicating that the simulated disease treatment had no significant impact on rice physiology. In contrast, the IDG group showed significant changes in all indicators, suggesting the combined effects of the disease on rice photosynthesis, water regulation, and overall physiological state.

[0103] In one example of this embodiment, ANOVA (Analysis of Variance) was used to compare the differences in eight physiological indicators among the three treatment groups. If the ANOVA showed a significant difference, a statistical model was used to perform a post-hoc multiple comparison (Tukey HSD) test to further determine the significant differences between the different treatment groups. Figure 7 As shown, except for Y(II), RHsfc, and ETR, 5 out of the 8 physiological indicators showed significant differences (p < 0.05), therefore, the Tukey HSD test can be performed. Figure 8 The results showed that there were no significant differences in any physiological parameters between CK and SDG at different stages of infection. Figure 6 The observed changes in various physiological indicators were consistent. Therefore, future studies will consider these stages of infection as healthy.

[0104] In one example of this embodiment, such as Figure 11 As shown, a schematic diagram of the sensitive bands selected by different feature band extraction methods is presented. These selected bands have a common region, mainly concentrated in the range of 500-585nm and 737-855nm, indicating that these key sensitive bands are crucial for identifying ShB.

[0105] In this way, the selection mainly focuses on the troughs (500-530 nm), peaks (550-585 nm), and near-infrared region (830-855 nm) of the original spectrum. These wavelengths reflect the impact of diseases on the spectral characteristics of plant leaves to varying degrees. Analyzing these wavelengths can further help us understand the mechanisms by which diseases affect plant physiological and biochemical processes, providing a theoretical basis for subsequent disease diagnosis and prediction.

[0106] In one example of this embodiment, reference Figure 9 The study demonstrates the sensitive bands selected by different feature band extraction methods and the GASF (Gram angle and field) transform incorporating co-physiological parameters. Whether using a single sensitive band or combining it with co-physiological parameters, the data transformed by GASF exhibit different structural characteristics. These differences not only reveal the diversity and complexity of the feature band extraction method selection process but also highlight the important role of co-physiological parameters in improving and optimizing the GASF transformation process.

[0107] In one example of this embodiment, the impact of different feature extraction methods and multimodal data fusion strategies on model performance was compared. Experimental results show that the multimodal fusion model combining physiological parameters and specific sensitive bands exhibits superior performance across all test sets. Among them, the IVISSA-CARS (Interval Variable Iterative Space Shrinkage Method-Competitive Adaptive Reweighted Sampling) sensitive feature band, incorporating physiological parameters, achieved the highest accuracy of 94.1667% in MMCG-MHA. Specific classification results are as follows: Figure 10 As shown, this effectively verifies the effectiveness of the strategy in improving the accuracy of disease diagnosis.

[0108] Furthermore, by comparing the performance of individual models on the same dataset (Table 1), it can be found that the MMCG-MHA model has significant advantages in qualitative analysis tasks. It not only improves the model's efficiency but also significantly improves its accuracy. Compared with the single-modal one-dimensional data feature classification model GRU, MMCG-MHA achieves an improvement of up to 18.958%, and compared with the classic single-modal CNN image classification model, it achieves an improvement of up to 11.6525%. This indicates that MMCG-MHA can not only process and integrate key information from different modalities but also effectively merge this information and capture their inherent relationships, thereby obtaining more comprehensive and accurate qualitative analysis results. In contrast, classic models often only analyze a single modality, failing to fully utilize the complementarity and synergy between multimodal data.

[0109] Table 1 shows the accuracy of different feature band extraction methods in determining rice health status and sheath blight duration in various models.

[0110]

[0111] Table 1 clearly shows that for the original full spectrum, both single-modal and multimodal models perform poorly, with accuracy below 50%. After applying CWT (Continuous Wavelet Transform) + MSC (Multivariate Scattering Correction) + SG (SG-Smoothing) + MMN (Maximum-Minimum Normalization) spectral preprocessing, the model performance improves slightly, but not significantly, due to data redundancy and low efficiency. Furthermore, using only spectral sensitive feature bands or physiological parameters generally results in low recognition accuracy, indicating room for improvement. However, combining both methods leads to a significant performance boost. The recognition effects of different processing methods vary considerably. Compared to the full spectrum, selecting ShB sensitive feature bands not only improves model accuracy but also significantly improves efficiency by reducing input parameters. In addition, combining physiological parameters as auxiliary information with specific sensitive bands effectively captures biological changes under disease conditions, thereby enhancing the model's discriminative ability and accuracy. This finding further validates the crucial role of physiological parameters in improving the performance of disease recognition models.

[0112] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal method for detecting early crop diseases, characterized in that, Includes the following steps: A crop disease detection model is constructed based on a dual-branch architecture that integrates a convolutional neural network, a gated recurrent unit, and a multi-head attention mechanism layer. Based on the established experimental conditions, leaf reflectance spectrum and leaf physiological variables were measured to collect data. The collected data is used as sample data for model training and testing, and the sample data is filtered and preprocessed to obtain a sample dataset. The crop disease detection model is trained and tested using the sample dataset. Input the crop detection data into the trained crop disease detection model, and output the early crop disease detection results. The steps for constructing a crop disease detection model based on a dual-branch architecture integrating convolutional neural networks, gated recurrent units, and multi-head attention mechanisms specifically include: In the convolutional neural network branch, an image input layer is set up, and the image input layer is connected to the convolutional layer. The convolutional kernel and stride of the convolutional layer are set. The convolutional layer, batch normalization layer, pooling layer and fully connected layer are connected in sequence. The activation function ReLU is introduced between the batch normalization layer and the pooling layer as a corrected linear unit. A flat layer is connected after the fully connected layer. In the gated loop unit branch, the image input layer and the gated loop unit are connected, and a fully connected layer and a flat layer are also set between the image input layer and the gated loop unit; In the multi-head attention mechanism layer, the feature fusion layer is used as the input layer and connected to the flat layer of the convolutional neural network and gated recurrent unit. The output of the feature fusion layer is fed into a fully connected layer, and then connected to a normalized exponential layer and a sheath blight classification layer. The steps of filtering and preprocessing the sample data specifically include: Data screening: Monte Carlo partial least squares and interquartile range methods were used to screen outliers in spectral data and physiological parameters; Data preprocessing: Combining two or more data processing techniques; where the two or more data processing techniques are: continuous wavelet transform, multivariate scatter correction, Savitzky-Golay convolutional smoothing, and minimum-maximum normalization; In the step of using the collected data as sample data for model training and testing, the one-dimensional sequence is converted into a two-dimensional image by converting Cartesian coordinates to polar coordinates based on Gram angles and fields, and calculating the angle between data point vectors.

2. The method for detecting early crop diseases based on multimodality according to claim 1, characterized in that, The experimental conditions described were based on an RXZ-1000 artificial climate chamber, and the crop used in the experiment was Wuyou Rice No.

4. The RXZ-1000 artificial climate chamber is equipped with: High-precision lighting: 0-8000 lux; Temperature control: 0-50℃ ± 0.1℃ fluctuation; Automatic humidity control: 50-95%, ±3% fluctuation. Automated day / night cycle control system: 0-99 hours; Crops were divided into a control group, a simulated infection group, and an actual infection group, and inoculated based on a specified infection technique; The specified infection techniques are one of the following: detached leaf method, microchamber method, mist chamber method, mycelial suspension spraying method, and mycelial embedding method.

3. The method for detecting early crop diseases based on multimodality according to claim 1, characterized in that, in, Leaf reflectance spectra were measured using a spectrometer. Non-invasive in-situ synchronous acquisition of photosynthetic parameters of leaves was performed using a portable photosynthesis meter. Chlorophyll fluorescence in leaves was non-destructively assessed using a portable modulated chlorophyll fluorescence meter under natural and dark conditions to obtain fluorescence parameters. The physiological variables of leaves were measured by combining photosynthetic parameters and fluorescence parameters.

4. The method for detecting early crop diseases based on multimodality according to claim 1, characterized in that, The method further includes: The performance of the crop disease detection model was analyzed; the performance evaluation indicators included average accuracy, recall, precision, and F1 score.

Images