A field diagnosis method and device for crop diseases based on multimodal data fusion

Abstract

This application discloses a method and apparatus for field diagnosis of crop diseases based on multimodal data fusion. The method includes constructing a sunlight-induced chlorophyll fluorescence feature vector and a hyperspectral reflectance feature vector; calculating a fusion operator; constructing a first training set based on the sunlight-induced chlorophyll fluorescence feature vector, the hyperspectral reflectance feature vector, the fusion operator, and the disease level; training a dual-branch hybrid CNN-Transformer network based on the first training set; and outputting the disease level corresponding to the target crop from the trained dual-branch hybrid CNN-Transformer network. In this embodiment, it can better meet the combined requirements of deep fusion of heterogeneous data, high early diagnostic sensitivity, and strong universality in field diagnosis, resulting in better field disease diagnosis performance.

Description

Technical Field

[0001] The embodiments in this specification pertain to the field of agricultural disease monitoring, and specifically relate to a method and device for field diagnosis of crop diseases based on multimodal data fusion. Background Technology

[0002] With the development of precision agriculture, rapid, non-destructive, and early diagnosis of crop diseases is of great significance for reducing yield losses and guiding scientific pesticide application. Traditional field disease diagnosis relies on visual observation by plant protection personnel or laboratory biochemical analysis, which has limitations such as strong subjectivity, poor timeliness, and difficulty in obtaining large-area spatial distribution information.

[0003] In recent years, hyperspectral remote sensing technology has been widely used for plant disease monitoring due to its ability to acquire continuous and detailed spectral information of crops. However, traditional hyperspectral disease monitoring mainly relies on reflectance spectra, which is not sensitive enough to changes in plant photosynthetic function. Furthermore, solar-induced chlorophyll fluorescence (SIF) is a direct probe of plant photosynthesis, reflecting the functional state of photosynthetic organs and exhibiting unique response advantages in the early stages of disease stress. Therefore, hyperspectral imaging and SIF can be used synergistically for disease diagnosis. However, existing methods for synergistically using these two technologies for disease diagnosis often involve simple data splicing, with insufficient handling of data heterogeneity, poor multi-scale matching and universality, making it difficult to meet the field diagnostic needs of deeply integrating heterogeneous data and achieving both high early diagnostic sensitivity and strong universality. Summary of the Invention

[0004] The embodiments of this disclosure provide a method and apparatus for field diagnosis of crop diseases based on multimodal data fusion, which aims to solve one or more of the above-mentioned problems and other potential problems.

[0005] According to a first aspect of this disclosure, a field diagnosis method for crop diseases based on multimodal data fusion is provided. The method includes collecting solar-induced chlorophyll fluorescence data and ground hyperspectral reflectance data corresponding to the crop in a standard sample field to construct a solar-induced chlorophyll fluorescence feature vector and a hyperspectral reflectance feature vector; performing nonlinear coupling on the matched solar-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector to calculate a fusion operator. The fusion operator includes a physiological-structural coupling operator to reflect the actual photosynthetic efficiency per unit canopy greenness, a pigment-fluorescence coupling operator to simultaneously capture changes in chlorophyll absorption characteristics and light energy utilization efficiency, and a red-edge-fluorescence linkage operator to reflect the synergistic relationship between impaired photosynthetic function and blue shift of chlorophyll absorption edge; constructing a first training set based on the solar-induced chlorophyll fluorescence feature vector, hyperspectral reflectance feature vector, fusion operator, and disease level; and training a dual-branch hybrid CNN-Transformer network based on the first training set. The dual-branch hybrid CNN-Transformer network includes a hyperspectral branch, a sunlight-induced chlorophyll fluorescence branch, a cross-modal attention interaction module, a gated fusion layer, and a classification layer. The cross-modal attention interaction module uses the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors and performs attention weighting on them respectively to obtain a weighted fluorescence feature vector and a weighted hyperspectral feature vector. The hyperspectral branch outputs the first branch feature based on the weighted hyperspectral feature vector and the fusion operator, the sunlight-induced chlorophyll fluorescence branch outputs the second branch feature based on the weighted fluorescence feature vector, the gated fusion layer dynamically weights the first branch feature and the second branch feature based on the gating coefficient to obtain the fused feature, and the classification layer outputs the disease level based on the fused feature. Based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed, the trained dual-branch hybrid CNN-Transformer network outputs the disease level corresponding to the target crop.

[0006] According to a second aspect of this disclosure, a field diagnostic device for crop diseases based on multimodal data fusion is provided. The device includes a data acquisition module configured in a standard sample field to collect data on solar-induced chlorophyll fluorescence (SCR) and ground hyperspectral reflectance (HIR) to construct SCR feature vectors and HIR feature vectors; a fusion operator calculation module configured to nonlinearly couple the matched SCR and HIR feature vectors to calculate a fusion operator, which includes a physiological-structural coupling operator reflecting the actual photosynthetic efficiency per unit canopy greenness, a pigment-fluorescence coupling operator simultaneously capturing changes in chlorophyll absorption characteristics and light energy utilization efficiency, and a red-edge-fluorescence linkage operator reflecting the synergistic relationship between photosynthetic function impairment and blue shift of chlorophyll absorption edges; and a model training module configured to construct a first training set based on the SCR feature vector, HIR feature vector, fusion operator, and disease severity level, and to train a dual-branch hybrid CNN-T model based on the first training set. The dual-branch hybrid CNN-Transformer network includes a hyperspectral branch, a sunlight-induced chlorophyll fluorescence branch, a cross-modal attention interaction module, a gated fusion layer, and a classification layer. The cross-modal attention interaction module uses the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors and performs attention weighting on them respectively to obtain a weighted fluorescence feature vector and a weighted hyperspectral feature vector. The hyperspectral branch outputs the first branch feature based on the weighted hyperspectral feature vector and the fusion operator, the sunlight-induced chlorophyll fluorescence branch outputs the second branch feature based on the weighted fluorescence feature vector, the gated fusion layer dynamically weights the first branch feature and the second branch feature based on the gating coefficient to obtain the fused feature, and the classification layer outputs the disease level based on the fused feature. The disease diagnosis module is configured based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed, and the trained dual-branch hybrid CNN-Transformer network outputs the disease level corresponding to the target crop.

[0007] According to a third aspect of this disclosure, an electronic device is provided, including one or more processors and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform a method provided according to a first scheme.

[0008] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to the first aspect.

[0009] The solution provided in the embodiments of this specification can adaptively and collaboratively integrate two modal data, sunlight-induced chlorophyll fluorescence data and hyperspectral reflectance data, through a cross-modal attention mechanism and a fusion operator based on physical mechanisms. This is achieved by using a dual-branch hybrid CNN-Transformer network adapted to heterogeneous data. This solution can better meet the combined requirements of deep integration of heterogeneous data, high early diagnostic sensitivity, and strong universality in field diagnosis, resulting in better field diagnosis of diseases. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A flowchart illustrating a field diagnosis method for crop diseases based on multimodal data fusion, according to some embodiments of this disclosure, is shown.

[0012] Figure 2 The following are schematic diagrams illustrating the design of standard sample plots according to some embodiments of this disclosure;

[0013] Figure 3 A schematic diagram of the architecture of a dual-branch hybrid CNN-Transformer network according to some embodiments of this disclosure is shown;

[0014] Figure 4 A schematic diagram illustrating the principle of a stacked ensemble learning decision-making process according to some embodiments of this disclosure is shown;

[0015] Figure 5 The diagram shows a schematic representation of a field diagnostic device for crop diseases based on multimodal data fusion, representing some embodiments of this disclosure.

[0016] Figure 6 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “in response to determination”, or “in response to detection”.

[0019] As mentioned earlier, there is currently no way to combine SIF and hyperspectral imaging for disease diagnosis, and existing methods that combine SIF and hyperspectral imaging have several shortcomings: (1) they rely on a single data source and have limited ability to characterize the spatiotemporal heterogeneity of diseases; (2) most methods simply combine SIF and reflectance without utilizing the physical coupling between the two; (3) they lack time-series modeling of the dynamic process of disease development; (4) the models lack physical mechanism constraints and have poor generalization ability across varieties, years, and regions; (5) the scale conversion methods are limited and it is difficult to achieve accurate transfer from leaves to the canopy; and (6) they lack systematic construction and interpretability analysis of disease-specific spectral characteristics. Therefore, there is an urgent need for a solution that can meet the field diagnosis needs of deeply integrating heterogeneous data, having high early diagnostic sensitivity, and strong universality.

[0020] Figure 1 The diagram illustrates a flowchart of a crop disease field diagnosis method 100 based on multimodal data fusion, representing some embodiments of this disclosure. Method 100 can be executed by a terminal, which may include, but is not limited to, mobile phones, tablets, desktop computers, servers, etc. Figure 1 As shown, in method 100, step 102 can collect the corresponding solar-induced chlorophyll fluorescence data and ground hyperspectral reflectance data of crops in standard sample fields to construct solar-induced chlorophyll fluorescence feature vector and hyperspectral reflectance feature vector.

[0021] In this embodiment, as Figure 2As shown, taking the diagnosis of black spot disease in broccoli as an example, standard sample plots with multiple varieties, growth stages (i.e., multiple time phases), and disease levels can be pre-established in the experimental field. The configuration of these standard sample plots could be, for example, planting at least three broccoli varieties, with at least four replicates for each variety. At least three plots for each variety are randomly selected for black spot inoculation at the seedling stage, pre-heading stage, and heading stage. Isolation strips with a width of at least 2 meters are established to prevent disease spread. Such field trials can be repeated at least three times. Disease levels can be classified, for example, as healthy (level 0, no lesions), mildly infected (level 1, lesion area less than 10%), moderately infected (level 2, lesion area between 10% and 30%), and severely infected (level 3, lesion area greater than 30%). The specific disease level can be determined by obtaining crop samples at data collection points and bringing them back to the laboratory for testing. In standard sample plots, ground data acquisition units can be set up, including tower-based monitoring equipment, an ASD FieldSpec 4 spectrometer, and a QE PRO spectrometer, to acquire hyperspectral reflectance data and sunlight-induced chlorophyll fluorescence data. A three-point sampling method is used, selecting three average points diagonally within each experimental plot to set up tower-based monitoring equipment at a height of 1-1.5m and a field of view of 25°. Ground hyperspectral reflectance data can be directly acquired by the spectrometer. Furthermore, based on the Fraunhofer dark line principle, the 3FLD method can be used to extract sunlight-induced chlorophyll fluorescence data under different disease severity levels, with the extracted bands being... The absorption band is calculated using the following formula:

[0022]

[0023] in, The solar irradiance spectral intensity within the Fraunhofer line, The spectral intensity of the vegetation canopy reflectance within the Fraunhofer line. , This indicates the weights assigned to the two reference bands to the left and right of the absorption line. , , This indicates the spectral intensity of solar irradiance to the left and right of the absorption line. , This indicates the spectral intensity of the vegetation canopy reflectance to the left and right of the absorption line.

[0024] By performing preprocessing such as normalization and encoding on sunlight-induced chlorophyll fluorescence data and ground hyperspectral reflectance data, feature vectors of sunlight-induced chlorophyll fluorescence and hyperspectral reflectance can be constructed.

[0025] In method 100, step 104 can nonlinearly couple the matched daylight-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector to calculate a fusion operator. The fusion operator includes a physiological-structural coupling operator to reflect the actual photosynthetic efficiency per unit canopy greenness, a pigment-fluorescence coupling operator to simultaneously capture changes in chlorophyll absorption characteristics and light energy utilization efficiency, and a red-edge-fluorescence linkage operator to reflect the synergistic relationship between impaired photosynthetic function and blue shift of chlorophyll absorption edge.

[0026] In this embodiment, the time- and space-matched solar-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector are treated as the same set of samples and nonlinearly coupled to generate a set of fusion operators with clear physiological and physical meanings, replacing the traditional simple feature splicing method. This fusion operator is not simply a feature arrangement, but rather achieves multi-dimensional enhancement of disease stress information by characterizing the intrinsic mechanistic relationships between different data. Specifically, the physiological-structural coupling operator quantifies the actual photosynthetic efficiency per unit canopy greenness level. For example, it can be obtained by correlating fluorescence intensity, which reflects the photosynthetic functional state, with vegetation indices reflecting canopy structure or greenness, thereby identifying early stress signals where canopy structure has not yet shown changes but photosynthetic function has already declined. The pigment-fluorescence coupling operator is used to simultaneously capture the dynamic relationship between chlorophyll absorption characteristics and light energy utilization efficiency. For example, it can be obtained by correlating the band reflectance difference, which reflects chlorophyll absorption capacity, with a composite index reflecting actual photosynthetic efficiency, thereby sensitively reflecting the degree of imbalance between pigment content and photosynthetic function under disease stress. The red-edge-fluorescence linkage operator is used to reflect the synergistic relationship between impaired photosynthetic function and the blue shift of chlorophyll absorption edge. For example, it can be obtained by synergistically enhancing the red-edge parameter, which characterizes the boundary position of the chlorophyll absorption spectrum, and the fluorescence intensity, which characterizes photosynthetic yield. This results in a more significant composite response signal than a single indicator when disease-induced pigment degradation and decreased photosynthetic function occur simultaneously. The variables used in the calculation process of the aforementioned fusion operator can be directly obtained from the sunlight-induced chlorophyll fluorescence vector and the hyperspectral reflectance feature vector, or derived through known formulas. The specific calculation method can be adjusted according to different actual needs. After obtaining the fusion operator, it will be used as an independent and physically interpretable derived feature to supplement the model in subsequent training, thereby enhancing the model's sensitivity to disease stress and decision transparency.

[0027] In method 100, step 106 can construct a first training set based on the sunlight-induced chlorophyll fluorescence feature vector, the hyperspectral reflectance feature vector, the fusion operator, and the disease level. A dual-branch hybrid CNN-Transformer network is then trained based on this first training set. The dual-branch hybrid CNN-Transformer network includes a hyperspectral branch, a sunlight-induced chlorophyll fluorescence branch, a cross-modal attention interaction module, a gated fusion layer, and a classification layer. The cross-modal attention interaction module uses the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors and performs attention weighting on each, resulting in a weighted fluorescence feature vector and a weighted hyperspectral feature vector. The hyperspectral branch outputs the first branch feature based on the weighted hyperspectral feature vector and the fusion operator. The sunlight-induced chlorophyll fluorescence branch outputs the second branch feature based on the weighted fluorescence feature vector. The gated fusion layer dynamically weights the first and second branch features based on the gating coefficient to obtain the fused feature. The classification layer outputs the disease level based on the fused feature.

[0028] In this embodiment, a first training set will be constructed based on the sunlight-induced chlorophyll fluorescence feature vector, the hyperspectral reflectance feature vector, the fusion operator, and the disease level. The dual-branch hybrid CNN-Transformer network will be trained using the first training set. The sunlight-induced chlorophyll fluorescence feature vector, the hyperspectral reflectance feature vector, and the fusion operator will be used as input data, and the disease level will be used as output data. The disease level in the training set is real data obtained by sampling from standard sample fields and laboratory testing.

[0029] The structure of a dual-branch hybrid CNN-Transformer network can be as follows: Figure 3 As shown, the hyperspectral branch and the solar-induced chlorophyll fluorescence branch can cover the model's input layer and feature extraction branch. The hyperspectral reflectance feature vector and fusion operator are input to the hyperspectral branch, and the solar-induced chlorophyll fluorescence feature vector is input to the solar-induced chlorophyll fluorescence branch. Next, the hyperspectral reflectance feature vector and the solar-induced chlorophyll fluorescence feature vector are first input to the cross-modal attention interaction module. The cross-modal attention interaction module uses the solar-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors for attention weighting. Specifically, the calculation process is as follows: assuming the hyperspectral reflectance feature vector is... The sunlight-induced chlorophyll fluorescence feature vector is (k is the dimension of fluorescence parameters, including) (Intensity, F687 / F760, F740 / F760, normalized SIF, etc.) First, H and S are mapped to the same embedding space of dimension d through a linear transformation, and a learnable cross-modal attention weight matrix is ​​set. , , For hyperspectral band filtering guided by the solar-induced chlorophyll fluorescence feature vector as the query vector, with S as the query and H as the key and value, then:

[0030]

[0031]

[0032]

[0033] in, , , .

[0034] Next, calculate the attention score:

[0035]

[0036] Based on the attention score, a weighted hyperspectral feature vector can be calculated:

[0037]

[0038] For fluorescence parameter filtering guided by hyperspectral reflectance feature vectors as query vectors, with H as the query and S as the key and value, then:

[0039]

[0040]

[0041]

[0042] in, , , .

[0043] Next, calculate the attention score:

[0044]

[0045] Based on the attention score, a weighted fluorescence feature vector can be calculated:

[0046]

[0047] The hyperspectral branch can use a 1D-CNN local feature extractor to concatenate the weighted hyperspectral feature vector with the fusion operator to output the first branch features. Its branch structure can be 3 layers of one-dimensional convolution (kernel sizes of 7, 5, and 3, stride of 1, padding method 'SAME'), each layer followed by normalization and ReLU activation, and finally a global max pooling layer.

[0048] The sunlight-induced chlorophyll fluorescence branch can use a Transformer temporal encoder to output second-branch features based on weighted fluorescence feature vectors. Its branch structure can employ standard Transformer encoder layers, including a multi-head self-attention network (4 heads) and a feedforward network, stacked in two layers. The output is calculated using the features corresponding to the [CLS] label or sequence mean pooling. This branch can learn the temporal decay pattern of sunlight-induced chlorophyll fluorescence as the disease progresses, compensating for the dynamic information of photosynthetic function that static hyperspectral imaging cannot capture.

[0049] The gating fusion layer is configured with learnable gating coefficients, which can be calculated using the following formula:

[0050]

[0051] in, For the sigmoid function, and These are trainable parameters.

[0052] The final formula for calculating the fused features is:

[0053]

[0054] in, This is the first branch feature. This is the second branch feature. This gating mechanism enables the network to dynamically and softly weight the features based on the modal reliability of the current sample, for example, automatically favoring the hyperspectral branch when the SIF vector is incomplete.

[0055] Finally, the fused features pass through a classification layer, where the fully connected layer outputs the probabilities of four disease categories: healthy, mild, moderate, and severe. This results in the model outputting the disease level. The final output disease level can be the level corresponding to the highest predicted probability among the four disease categories: healthy, mild, moderate, and severe.

[0056] In method 100, step 108 can output the disease level of the target crop based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed by a trained dual-branch hybrid CNN-Transformer network.

[0057] In this embodiment, after training the dual-branch hybrid CNN-Transformer network, the target hyperspectral reflectance feature vector corresponding to the target crop for which disease diagnosis is needed can be obtained. The dual-branch hybrid CNN-Transformer network then outputs the corresponding disease level to achieve disease diagnosis of the target crop. Due to the setting of the gating fusion layer, the network learns to use the SIF feature vector to improve accuracy during training. However, during inference, only the hyperspectral reflectance feature vector can be input, and the SIF branch position can be filled with the default value, allowing the gating coefficients to automatically converge to 1. This completely disables the weights of the SIF branch, relying solely on the hyperspectral branch for forward computation, reducing the difficulty and complexity of data acquisition when actually using the network for prediction.

[0058] In one possible implementation, the standard sample plots include crops of different varieties, different growth stages, and different disease levels;

[0059] Constructing the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector, including:

[0060] The crop growth area is segmented based on the growth status, and the average reflectance spectrum of each growth area is calculated based on the ground hyperspectral reflectance data. Hyperspectral reflectance feature vectors are then constructed based on each average reflectance spectrum.

[0061] The sunlight-induced chlorophyll fluorescence data of each growth area were continuously recorded in chronological order to construct a sunlight-induced chlorophyll fluorescence feature vector.

[0062] In this embodiment, the sunlight-induced chlorophyll fluorescence feature data and hyperspectral reflectance feature data will be further processed before being used to construct the sunlight-induced chlorophyll fluorescence feature vector and hyperspectral reflectance feature vector. Specifically, a pre-trained U-Net model can be used to segment the crop growth region (e.g., flower head region, leaf canopy region, etc.) according to its growth state. The U-Net model structure includes an encoder (4 convolutional blocks, each containing 2 3×3 convolutions and 1 2×2 max pooling) and a decoder (4 upsampling blocks, each containing 1 2×2 upconvolution, 2 3×3 convolutions, and skip connections). The input image size is 256×256×3, and the output is a binary segmentation map (flower head / leaf). For each growth region, the spectrum of at least 50 pixels is extracted from the growth region and averaged to obtain the average reflectance spectrum of that growth region. Then, a hyperspectral reflectance feature vector is constructed based on the average reflectance spectrum of each growth region. In addition, the sunlight-induced chlorophyll fluorescence (SIF) data for each growth region will be recorded chronologically (e.g., at the same time each day) for N consecutive days, forming a time-series vector of length N, which will serve as the SIF feature vector. This time-series sequence can capture the dynamic decay trend of the SIF signal under disease stress, providing a long-range dependency learning basis for the Transformer encoder.

[0063] In one possible implementation, the matched daylight-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector are nonlinearly coupled to calculate a fusion operator, including:

[0064] The normalized vegetation index, fluorescence intensity emitted by leaves during actual photosynthesis after absorbing light energy, reflectance difference between near-infrared and green light bands, red edge position index, and photochemical reflectance index are determined from the matched sunlight-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector.

[0065] The physiological-structural coupling operator is calculated based on the ratio of fluorescence intensity to normalized vegetation index.

[0066] Calculate the product of fluorescence intensity and photochemical reflectance index, and calculate the pigment-fluorescence coupling operator based on the ratio of reflectance difference to the product value;

[0067] The red-edge-fluorescence linkage operator is calculated based on the product of the natural logarithm of fluorescence intensity and the red-edge position index.

[0068] In this embodiment, the calculation formula for the physiological-structural coupling operator is:

[0069]

[0070] in, The fluorescence intensity extracted in the 760 nm oxygen absorption band is a direct component determined from the hyperspectral reflectance eigenvector. The normalized vegetation index is used. To avoid small constants with a denominator of zero.

[0071] The formula for calculating the normalized vegetation index can be:

[0072]

[0073] in, and These are all reflectance values ​​for the corresponding bands (usually 780nm for near-infrared and 670nm for red) in the hyperspectral reflectance feature vector, and can also be directly extracted and determined from the hyperspectral reflectance feature vector.

[0074] The calculation formula for the pigment-fluorescence coupling operator is as follows:

[0075]

[0076] in, This represents the reflectance value at 780 nm (corresponding to near-infrared) in the hyperspectral reflectance vector. This represents the reflectance value at 550 nm (corresponding to green light) in the hyperspectral reflectance vector. It is the photochemical reflectance index.

[0077] The formula for calculating the photochemical reflectance index is:

[0078]

[0079] in, and These are the reflectance values ​​at 531 nm and 570 nm in the hyperspectral reflectance vector, respectively.

[0080] The calculation formula for the red-edge-fluorescence linkage operator is:

[0081]

[0082] in, The red-edge position index is defined as the wavelength corresponding to the maximum value of the first derivative of reflectance in the red-edge region of the spectrum from 680 nm to 780 nm. It can be determined by differentiating the spectral curve of the hyperspectral reflectance eigenvector in this range and finding the peaks.

[0083] In one possible implementation, the method further includes:

[0084] In standard sample fields, aerial hyperspectral reflectance data of crops collected by drones are obtained, and the aerial hyperspectral reflectance data is marked as ground hyperspectral reflectance data.

[0085] Based on the target crop's hyperspectral reflectance feature vector, a trained dual-branch hybrid CNN-Transformer network outputs the disease level of the target crop, including:

[0086] Based on the target hyperspectral reflectance feature vector collected by the UAV for the target crop to be diagnosed, the disease level of the target crop is output by the trained dual-branch hybrid CNN-Transformer network.

[0087] In this embodiment, in addition to collecting ground hyperspectral reflectance data via a ground-based spectrometer, aerial hyperspectral reflectance data collected at low altitudes can also be obtained via a spectrometer mounted on a drone. When constructing the training dataset, the aerial hyperspectral reflectance data is labeled as ground hyperspectral reflectance data, meaning the model treats aerial hyperspectral reflectance data as equivalent to ground hyperspectral reflectance data, generating hyperspectral reflectance feature vectors for training. When using both types of data, scale alignment is crucial; that is, coordinates and resolution must match. Thus, after validating and training a sufficiently reliable dual-branch hybrid CNN-Transformer network using ground data (i.e., ground hyperspectral reflectance data and sunlight-induced chlorophyll fluorescence data) collected from fixed equipment in standard sample fields, the aerial hyperspectral reflectance data collected by the drone can be directly used as input for prediction in practical applications. Because drones are more mobile and can cover more actual fields, reliable disease diagnosis can be achieved without ground equipment.

[0088] In one possible implementation, the method further includes:

[0089] A second training set is constructed based on the sunlight-induced chlorophyll fluorescence feature vector and the disease level. A sunlight-induced chlorophyll fluorescence-specific model is trained based on the second training set to obtain the first probability vector corresponding to each disease level output by the sunlight-induced chlorophyll fluorescence-specific model.

[0090] A third training set is constructed based on hyperspectral reflectance feature vectors, fusion operators, and disease severity levels. A hyperspectral reflectance-specific model is trained based on the third training set to obtain the second probability vector corresponding to each disease severity level output by the hyperspectral reflectance-specific model.

[0091] Obtain the third probability vector corresponding to each disease level output by the dual-branch hybrid CNN-Transformer network;

[0092] Using the concatenated vector of the first probability vector, the second probability vector, and the third probability vector as input features and the true disease level as the supervision label, a meta-learner is trained so that the trained meta-learner can be used to output the final diagnosis result of the disease level.

[0093] In this embodiment, as Figure 4 As shown, to further improve the robustness of the model, a two-layer stacked ensemble architecture can be set up. In addition to the dual-branch hybrid CNN-Transformer network, a dedicated model for sunlight-induced chlorophyll fluorescence is trained solely based on the sunlight-induced chlorophyll fluorescence feature vector and disease severity level. A dedicated model for hyperspectral reflectance is trained solely based on the hyperspectral reflectance feature vector, fusion operator, and disease severity level. The sunlight-induced chlorophyll fluorescence model can use a lightweight LSTM network, and the hyperspectral reflectance model can use an XGBoost classifier. This allows the models trained in three ways to output a first probability vector, a second probability vector, and a third probability vector, respectively. These three probability vectors are concatenated into a 12-dimensional concatenated vector, which can then be input into a meta-learner for training. The meta-learner can use a lightweight random forest or a three-layer fully connected neural network, trained with the actual disease severity level as the supervision label. This allows the trained meta-learner to automatically fit the implicit weight distribution of the prediction contributions of each model under different disease stages and variety data. Finally, the prediction results of the three models are combined to output the final diagnostic result, making the disease severity level determined by the final diagnosis more accurate.

[0094] In one possible implementation, the method further includes:

[0095] A disease feature library was constructed by integrating the feature vectors of sunlight-induced chlorophyll fluorescence, hyperspectral reflectance, weighted fluorescence, weighted hyperspectral, and fusion operators.

[0096] The disease feature database is reduced in dimensionality using sparse principal component analysis to obtain sparse principal components, and a feature mapping table is constructed based on the correlation coefficient between the sparse principal components and the disease severity.

[0097] In this embodiment, directly acquired sunlight-induced chlorophyll fluorescence feature vectors and hyperspectral reflectance feature vectors, as well as interpretable derived features such as weighted fluorescence feature vectors, weighted hyperspectral feature vectors, and fusion operators generated during the processing of the dual-branch hybrid CNN-Transformer network, can be integrated to construct a disease feature library. The feature library is then dimensionality-reduced through sparse principal component analysis to construct a feature mapping table. This feature mapping table can serve as an interpretation basis for model diagnostic results, enhancing the physical interpretability of features and providing transparent decision-making support for subsequent model diagnoses. Secondly, when the model needs to be applied across years and regions, it will not require retraining the network from scratch. Sparse principal components (i.e., high-weight key features selected by sparse representation) can be directly extracted from the disease feature library for rapid fine-tuning or transfer learning at a lower dimension. Furthermore, with continuous field trials or actual diagnoses, new features can continuously be added to the disease feature library. By periodically re-representing the sparse representation, the disease feature library can continuously adapt to new varieties and environmental changes, maintaining the timeliness and accuracy of diagnosis.

[0098] Figure 5 This document illustrates a schematic diagram of the structure of a crop disease field diagnostic device 500 based on multimodal data fusion, representing some embodiments of this disclosure. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments. Figure 5As shown, the device 500 includes a data acquisition module 501, configured in a standard sample field, which collects sunlight-induced chlorophyll fluorescence data and ground hyperspectral reflectance data corresponding to the crop, respectively, to construct sunlight-induced chlorophyll fluorescence feature vectors and hyperspectral reflectance feature vectors; a fusion operator calculation module 502, configured to nonlinearly couple the matched sunlight-induced chlorophyll fluorescence vectors and hyperspectral reflectance feature vectors, and calculate a fusion operator, which includes a physiological-structural coupling operator to reflect the actual photosynthetic efficiency under unit canopy greenness, a pigment-fluorescence coupling operator to simultaneously capture changes in chlorophyll absorption characteristics and light energy utilization efficiency, and a red-edge-fluorescence linkage operator to reflect the synergistic relationship between photosynthetic function impairment and blue shift of chlorophyll absorption edge; and a model training module 503, configured to construct a first training set based on the sunlight-induced chlorophyll fluorescence feature vector, hyperspectral reflectance feature vector, fusion operator, and disease level, and train a dual-branch hybrid CNN-Transformer network based on the first training set; The dual-branch hybrid CNN-Transformer network includes a hyperspectral branch, a sunlight-induced chlorophyll fluorescence branch, a cross-modal attention interaction module, a gated fusion layer, and a classification layer. The cross-modal attention interaction module uses the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors and performs attention weighting on each, resulting in a weighted fluorescence feature vector and a weighted hyperspectral feature vector. The hyperspectral branch outputs the first branch feature based on the weighted hyperspectral feature vector and the fusion operator, while the sunlight-induced chlorophyll fluorescence branch outputs the second branch feature based on the weighted fluorescence feature vector. The gated fusion layer dynamically weights the first and second branch features based on the gating coefficient to obtain the fused feature. The classification layer outputs the disease level based on the fused feature. The disease diagnosis module 504 is configured to output the disease level of the target crop based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed, and the trained dual-branch hybrid CNN-Transformer network outputs the disease level corresponding to the target crop.

[0099] In one possible implementation, the standard sample plots include crops of different varieties, different growth stages, and different disease levels;

[0100] The data acquisition module 501 is also configured to segment the crop growth area based on the growth status, calculate the average reflectance spectrum of each growth area based on the ground hyperspectral reflectance data, and construct a hyperspectral reflectance feature vector based on each average reflectance spectrum; and continuously record the sunlight-induced chlorophyll fluorescence data of each growth area in chronological order to construct a sunlight-induced chlorophyll fluorescence feature vector.

[0101] In one possible implementation, the fusion operator calculation module 502 is further configured to determine the normalized vegetation index, the fluorescence intensity emitted by the leaf during actual photosynthesis after absorbing light energy, the reflectance difference between the near-infrared band and the green band, the red edge position index, and the photochemical reflectance index from the matched solar-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector; calculate the physiological-structural coupling operator based on the ratio of fluorescence intensity to the normalized vegetation index; calculate the product of fluorescence intensity and the photochemical reflectance index; calculate the pigment-fluorescence coupling operator based on the ratio of the reflectance difference to the product; and calculate the red edge-fluorescence linkage operator based on the product of the natural logarithm of fluorescence intensity and the red edge position index.

[0102] In one possible implementation, the data acquisition module 501 is also configured in a standard sample field to acquire aerial hyperspectral reflectance data of crops collected by the UAV and mark the aerial hyperspectral reflectance data as ground hyperspectral reflectance data.

[0103] The disease diagnosis module 504 is also configured to output the disease level of the target crop based on the target hyperspectral reflectance feature vector collected by the UAV for the target crop to be diagnosed, and the disease level of the target crop is output by the trained dual-branch hybrid CNN-Transformer network.

[0104] In one possible implementation, the device further includes a stacked integration module configured to construct a second training set based on sunlight-induced chlorophyll fluorescence feature vectors and disease severity levels, to train a sunlight-induced chlorophyll fluorescence-specific model based on the second training set, and obtain a first probability vector corresponding to each disease severity level output by the sunlight-induced chlorophyll fluorescence-specific model; construct a third training set based on hyperspectral reflectance feature vectors, fusion operators, and disease severity levels, to train a hyperspectral reflectance-specific model based on the third training set, and obtain a second probability vector corresponding to each disease severity level output by the hyperspectral reflectance-specific model; obtain a third probability vector corresponding to each disease severity level output by a dual-branch hybrid CNN-Transformer network; and train a meta-learner using the concatenated vector of the first, second, and third probability vectors as input features and the actual disease severity level as a supervision label, so that the trained meta-learner is used to output the final diagnostic result of the disease severity level.

[0105] In one possible implementation, the device further includes a data integration module configured to integrate sunlight-induced chlorophyll fluorescence feature vectors, hyperspectral reflectance feature vectors, weighted fluorescence feature vectors, weighted hyperspectral feature vectors, and fusion operators to construct a disease feature library; the disease feature library is then dimensionality-reduced based on sparse principal component analysis to obtain sparse principal components, and a feature mapping table is constructed based on the correlation coefficient between the sparse principal components and the disease severity.

[0106] In one possible implementation, the gating coefficient is used to increase the weight of the other branch feature when either the first branch feature or the second branch feature is incomplete.

[0107] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0108] Figure 6 A block diagram of an electronic device 600 that can implement various embodiments of the present disclosure is shown. For example... Figure 6 As shown, the electronic device 600 includes a processor 610, a disk drive 620, an input / output interface 630, a network interface 640, and a memory 650. The processor 610, disk drive 620, input / output interface 630, network interface 640, and memory 650 can communicate with each other via a communication bus 660.

[0109] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0110] The memory 650 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage devices, etc. The memory 650 can store the operating system 651 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 652 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 653, a data storage management system 654, etc. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 650 and is called and executed by the processor 610.

[0111] The input / output interface 630 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, vibrators, indicator lights, etc.

[0112] Network interface 640 is used to connect a communication module (not shown in the figure) to enable communication and interaction between the device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0113] Bus 660 includes a pathway for transmitting information between various components of the device, such as processor 610, disk drive 620, input / output interface 630, network interface 640, and memory 650.

[0114] It should be noted that although the above-described device only shows the processor 610, disk drive 620, input / output interface 630, network interface 640, memory 650, bus 660, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the method of this application, and does not necessarily include all the components shown in the figures.

[0115] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A field diagnosis method for crop diseases based on multimodal data fusion, characterized in that, The method includes: In standard sample fields, solar-induced chlorophyll fluorescence data and ground hyperspectral reflectance data corresponding to crops were collected to construct solar-induced chlorophyll fluorescence feature vectors and hyperspectral reflectance feature vectors. The matched sunlight-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector are nonlinearly coupled to calculate a fusion operator. The fusion operator includes a physiological-structural coupling operator to reflect the actual photosynthetic efficiency per unit canopy greenness, a pigment-fluorescence coupling operator to simultaneously capture changes in chlorophyll absorption characteristics and light energy utilization efficiency, and a red-edge-fluorescence linkage operator to reflect the synergistic relationship between impaired photosynthetic function and blue shift of chlorophyll absorption edge. A first training set is constructed based on the sunlight-induced chlorophyll fluorescence feature vector, hyperspectral reflectance feature vector, fusion operator, and disease severity level. A dual-branch hybrid CNN-Transformer network is trained based on the first training set. The dual-branch hybrid CNN-Transformer network includes a hyperspectral branch, a sunlight-induced chlorophyll fluorescence branch, a cross-modal attention interaction module, a gated fusion layer, and a classification layer. The cross-modal attention interaction module is used to treat the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors and perform attention weighting on them respectively to obtain a weighted fluorescence feature vector and a weighted hyperspectral feature vector. The hyperspectral branch is used to output a first branch feature based on the weighted hyperspectral feature vector and the fusion operator. The sunlight-induced chlorophyll fluorescence branch is used to output a second branch feature based on the weighted fluorescence feature vector. The gated fusion layer is used to dynamically weight the first branch feature and the second branch feature based on the gating coefficient to obtain a fused feature. The classification layer is used to output the disease severity level based on the fused feature. Based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed, the disease level corresponding to the target crop is output by the trained dual-branch hybrid CNN-Transformer network.

2. The field diagnosis method for crop diseases based on multimodal data fusion according to claim 1, characterized in that, The standard sample plots include crops of different varieties, different growth stages, and different disease levels; The construction of the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector includes: The crop growth area is segmented based on the growth status, and the average reflectance spectrum of each growth area is calculated based on the ground hyperspectral reflectance data, so as to construct a hyperspectral reflectance feature vector based on each average reflectance spectrum. The sunlight-induced chlorophyll fluorescence data of each growth region are continuously recorded in chronological order to construct a sunlight-induced chlorophyll fluorescence feature vector.

3. The field diagnosis method for crop diseases based on multimodal data fusion according to claim 1, characterized in that, The nonlinear coupling of the matched solar-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector, and the calculation of the fusion operator, includes: The normalized vegetation index, fluorescence intensity emitted by leaves during actual photosynthesis after absorbing light energy, reflectance difference between near-infrared and green light bands, red edge position index, and photochemical reflectance index are determined from the matched sunlight-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector. The physiological-structural coupling operator is calculated based on the ratio of fluorescence intensity to normalized vegetation index. Calculate the product of the fluorescence intensity and the photochemical reflectance index, and calculate the pigment-fluorescence coupling operator based on the ratio of the reflectance difference to the product value; The red-edge-fluorescence linkage operator is calculated based on the product of the natural logarithm of the fluorescence intensity and the red-edge position index.

4. The field diagnosis method for crop diseases based on multimodal data fusion according to claim 1, characterized in that, The method further includes: In a standard sample field, aerial hyperspectral reflectance data of crops collected by drones are obtained, and the aerial hyperspectral reflectance data is marked as ground hyperspectral reflectance data. The disease level of the target crop, based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed, is output by a trained dual-branch hybrid CNN-Transformer network, including: Based on the target hyperspectral reflectance feature vector collected by the UAV for the target crop to be diagnosed, the disease level corresponding to the target crop is output by the trained dual-branch hybrid CNN-Transformer network.

5. The field diagnosis method for crop diseases based on multimodal data fusion according to claim 1, characterized in that, The method further includes: A second training set is constructed based on the sunlight-induced chlorophyll fluorescence feature vector and the disease level, and a sunlight-induced chlorophyll fluorescence-specific model is trained according to the second training set to obtain the first probability vector corresponding to each disease level output by the sunlight-induced chlorophyll fluorescence-specific model. A third training set is constructed based on the hyperspectral reflectance feature vector, fusion operator, and disease level, and a hyperspectral reflectance-specific model is trained based on the third training set to obtain the second probability vector corresponding to each disease level output by the hyperspectral reflectance-specific model. Obtain the third probability vector corresponding to each disease level output by the dual-branch hybrid CNN-Transformer network; Using the concatenated vector of the first probability vector, the second probability vector, and the third probability vector as input features and the true disease level as the supervision label, a meta-learner is trained so that the trained meta-learner can be used to output the final diagnosis result of the disease level.

6. The field diagnosis method for crop diseases based on multimodal data fusion according to claim 1, characterized in that, The method further includes: By integrating the aforementioned sunlight-induced chlorophyll fluorescence feature vector, hyperspectral reflectance feature vector, weighted fluorescence feature vector, weighted hyperspectral feature vector, and fusion operator, a disease feature library is constructed. The disease feature library is dimensionality reduced based on sparse principal component analysis to obtain sparse principal components, and a feature mapping table is constructed based on the correlation coefficient between the sparse principal components and the disease severity.

7. The field diagnosis method for crop diseases based on multimodal data fusion according to claim 1, characterized in that, The gating coefficient is used to increase the weight of the other branch feature when either the first branch feature or the second branch feature is incomplete.

8. A field diagnostic device for crop diseases based on multimodal data fusion, characterized in that, The device includes: The data acquisition module is configured in a standard sample field to collect data on the solar-induced chlorophyll fluorescence and ground hyperspectral reflectance of the crops, respectively, in order to construct solar-induced chlorophyll fluorescence feature vectors and hyperspectral reflectance feature vectors. The fusion operator calculation module is configured to perform nonlinear coupling on the matched sunlight-induced chlorophyll fluorescence vector and hyperspectral reflectance feature vector to calculate the fusion operator. The fusion operator includes a physiological-structural coupling operator to reflect the actual photosynthetic efficiency under unit canopy greenness, a pigment-fluorescence coupling operator to simultaneously capture changes in chlorophyll absorption characteristics and light energy utilization efficiency, and a red-edge-fluorescence linkage operator to reflect the synergistic relationship between impaired photosynthetic function and blue shift of chlorophyll absorption edge. The model training module is configured to construct a first training set based on the sunlight-induced chlorophyll fluorescence feature vector, hyperspectral reflectance feature vector, fusion operator, and disease severity level. A dual-branch hybrid CNN-Transformer network is then trained based on this first training set. The dual-branch hybrid CNN-Transformer network includes a hyperspectral branch, a sunlight-induced chlorophyll fluorescence branch, a cross-modal attention interaction module, a gated fusion layer, and a classification layer. The cross-modal attention interaction module uses the sunlight-induced chlorophyll fluorescence feature vector and the hyperspectral reflectance feature vector as query vectors and performs attention weighting on each, resulting in a weighted fluorescence feature vector and a weighted hyperspectral feature vector. The hyperspectral branch outputs a first branch feature based on the weighted hyperspectral feature vector and the fusion operator. The sunlight-induced chlorophyll fluorescence branch outputs a second branch feature based on the weighted fluorescence feature vector. The gated fusion layer dynamically weights the first and second branch features according to a gating coefficient to obtain a fused feature. The classification layer outputs the disease severity level based on the fused feature. The disease diagnosis module is configured to output the disease level of the target crop based on the target hyperspectral reflectance feature vector corresponding to the target crop to be diagnosed by a trained dual-branch hybrid CNN-Transformer network.

9. An electronic device, characterized in that, include: One or more processors, and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of a field diagnosis method for crop diseases based on multimodal data fusion as described in any one of claims 1-7.

10. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements a field diagnosis method for crop diseases based on multimodal data fusion according to any one of claims 1-7.

Images