Crop disease and pest anomaly detection multi-modal deep learning framework and method oriented to cross-crop and cross-modal crop disease and pest anomaly detection

Abstract

The invention relates to the technical field of agricultural information, and discloses a cross-crop and cross-modal oriented multi-modal deep learning framework and method for crop disease and insect pest anomaly detection. The data preprocessing module is used for receiving and preprocessing data of at least three modes: unmanned aerial vehicle remote sensing image data, time sequence meteorological data and crop growth stage information; a visual feature extraction module which is connected with the multi-modal data input layer, employs an improved residual network ResNet as a basic framework, carries out the feature extraction of the remote sensing image data of the unmanned plane, and obtains a visual feature vector; and the time sequence meteorological feature extraction module is connected with the multi-modal data input layer. Through an innovative cross-modal Transform fusion mechanism, visual form information of the unmanned aerial vehicle, dynamic change information of time sequence weather and priori knowledge of a crop growth stage can be effectively integrated, limitation of single-modal information is overcome, a complex mechanism of occurrence of diseases and insect pests is described more comprehensively, and thus the detection accuracy is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of agricultural information technology, specifically to a multimodal deep learning framework and method for detecting crop pests and diseases across crops and modalities. Background Technology

[0002] Crop diseases and pests are among the major biological disasters that restrict agricultural production and threaten global food security. According to the Food and Agriculture Organization of the United Nations (FAO), annual crop yield losses due to diseases and pests can reach 20%-40%. Therefore, achieving early, accurate, and rapid detection of crop diseases and pests is of great significance for guiding precision pesticide application, reducing pesticide overuse, and ensuring the quality and safety of agricultural products and the ecological environment.

[0003] Traditional pest and disease detection methods mainly rely on manual field surveys by plant protection experts. This method is not only time-consuming, labor-intensive, subjective, and inefficient, but also fails to meet the needs of real-time monitoring of large areas of farmland. With the development of sensor technology, drone platforms, and artificial intelligence, automatic pest and disease detection technology based on computer vision and deep learning has become a research hotspot.

[0004] Currently, pest and disease detection methods based on a single modality (mainly RGB images captured by drones or ground-based cameras) have made some progress, such as using convolutional neural networks (CNNs) for image classification or object detection. However, the occurrence and development of crop pests and diseases are closely related to multiple factors, exhibiting the following significant characteristics:

[0005] Multimodality: The manifestations of pests and diseases are not only reflected in the visual morphology of crops (such as leaf spots, curling, wilting, etc.), but are also closely related to meteorological conditions (such as temperature, humidity, and rainfall, which are key triggers for the occurrence and spread of many pests and diseases) and the crop's own growth stage (different growth stages have different susceptibility to the same pests and diseases). Relying solely on a single image modality is often insufficient for accurate judgment and is easily affected by factors such as changes in lighting and background interference, leading to misjudgments.

[0006] Strong temporal characteristics: The occurrence and development of pests and diseases is a dynamic process, and the temporal changes of meteorological factors have both cumulative and lag effects on the outbreak of pests and diseases. For example, prolonged periods of overcast, rainy, and humid weather may indicate a high risk of downy mildew. Most existing methods fail to effectively utilize the temporal characteristics of meteorological data.

[0007] Imbalanced sample distribution: In real-world field scenarios, there are far more healthy crop samples than pest and disease samples, especially for rare or newly emerging pests and diseases, where labeled samples are even scarcer. This severe imbalanced sample distribution can cause model training to favor the majority class, resulting in poor detection performance for minority pest and disease classes.

[0008] Insufficient cross-crop universality: Different crops have their own unique types and manifestations of diseases and pests. Many existing models are designed for specific crops and lack effective cross-crop knowledge transfer capabilities, making them difficult to apply directly to new crop varieties or disease and pest detection tasks.

[0009] To address this, we propose a multimodal deep learning framework and method for detecting crop pests and diseases anomalies across crops and modalities. Summary of the Invention

[0010] To achieve the above objectives, this invention provides the following technical solution: a multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities, comprising:

[0011] A multimodal data input layer is used to receive and preprocess data in at least three modalities: UAV remote sensing image data, time-series meteorological data, and crop growth stage information;

[0012] The visual feature extraction module is connected to the multimodal data input layer. It uses an improved ResNet residual network as the basic architecture to extract features from the UAV remote sensing image data and obtain visual feature vectors.

[0013] The time-series meteorological feature extraction module is connected to the multimodal data input layer and uses a long short-term memory network (LSTM) to extract time-series features from the time-series meteorological data to obtain a time-series meteorological feature vector.

[0014] The growth stage embedding module is connected to the multimodal data input layer and encodes the crop growth stage information into a fixed-dimensional growth stage embedding vector.

[0015] The cross-modal fusion module is connected to the visual feature extraction module, the temporal meteorological feature extraction module, and the growth stage embedding module, respectively. It uses an attention-based fusion mechanism to deeply fuse the visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector to generate a joint feature representation.

[0016] The anomaly detection and classification module is connected to the cross-modal fusion module. It performs anomaly detection of crop diseases and pests based on the joint feature representation, and further classifies the disease and pest type or growth abnormality type when anomalies are detected.

[0017] Preferably, the improved ResNet is based on the original ResNet with the following improvements:

[0018] A channel attention mechanism is introduced to adaptively recalibrate the channel feature responses;

[0019] Introduce spatial attention mechanisms to enhance focus on key spatial locations in pest and disease-affected areas;

[0020] A multi-scale feature fusion strategy is adopted to fuse feature maps of different levels to improve the detection capability of pest and disease areas of different sizes.

[0021] For small sample crops or pest / disease categories, transfer learning or meta-learning initialization strategies are introduced.

[0022] Preferably, the time-series meteorological data includes at least one or more sequence data of temperature, humidity, rainfall, and light intensity over a historical period; the LSTM network is used to capture the dynamic changes of meteorological factors over time and their potential correlation with the occurrence of pests and diseases.

[0023] Preferably, the crop growth stage information is a discrete category label, and the growth stage embedding module uses a learnable embedding layer to map the discrete growth stage labels into a high-dimensional dense vector; or, the crop growth stage information contains continuous growth cycle parameters, which are then encoded into a growth stage embedding vector through a fully connected layer.

[0024] Preferably, the attention-based fusion mechanism used in the cross-modal fusion module is a Transformer encoder structure or a variant thereof, specifically including:

[0025] The visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector are adjusted in dimension to form corresponding feature sequences;

[0026] The feature sequences of each modality are concatenated or mapped to a unified embedding dimension through linear projection to form the initial fusion sequence;

[0027] The initial fusion sequence is processed using a multi-head self-attention mechanism and a feedforward neural network to model the dependencies between features within a modality and the interactions between features of different modalities, and to output an enhanced joint feature representation.

[0028] Preferably, the cross-modal fusion module also introduces modality-specific positional encoding to preserve the original temporal or spatial positional information of each modality feature; or, it introduces a cross-modal cross-attention mechanism to explicitly model the attention weight of a certain modality feature to other modality features.

[0029] Preferably, the anomaly detection and classification module includes:

[0030] An anomaly detection branch is used to determine whether the current crop has pests, diseases, or abnormal growth.

[0031] A classification and discrimination branch, when the anomaly discrimination branch determines that it is abnormal, further distinguishes the specific type of pest or disease or the type of abnormal growth;

[0032] The anomaly detection branch and the classification detection branch share some fusion features, or each uses an independent classifier.

[0033] Preferably, the system also includes a sample imbalance handling module, integrated into the training process of the framework, which employs at least one of the following methods to alleviate the sample imbalance problem:

[0034] Weighted loss functions for minority class samples, such as Focal Loss or Class-Balanced Loss;

[0035] Data augmentation strategies, especially random cropping, rotation, color jittering, MixUp or CutMix operations on drone images of a few types of pests and diseases;

[0036] Synthesis of small-sample pest and disease images based on Generative Adversarial Network (GAN).

[0037] A method for detecting crop pest and disease anomalies across crops and modalities includes the following steps:

[0038] S1: Collect and preprocess UAV remote sensing image data, time-series meteorological data, and crop growth stage information through the multimodal data input layer;

[0039] S2: The visual feature extraction module extracts features from the preprocessed UAV remote sensing image data to obtain visual feature vectors;

[0040] S3: The time series meteorological feature extraction module extracts time series features from the preprocessed time series meteorological data to obtain the time series meteorological feature vector;

[0041] S4: Encode crop growth stage information through the growth stage embedding module to obtain the growth stage embedding vector;

[0042] S5: Through the cross-modal fusion module, an attention-based fusion mechanism is used to deeply fuse the visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector to generate a joint feature representation;

[0043] S6: The anomaly detection and classification module performs anomaly detection of crop diseases and pests based on the joint feature representation. If an anomaly is detected, the disease or pest type or growth abnormality type is further classified, and the detection result is output.

[0044] Preferably, in step S2, the improved ResNet enhances the effectiveness of visual feature extraction by introducing channel attention mechanism, spatial attention mechanism, multi-scale feature fusion strategy and / or few-shot learning initialization strategy; in step S3, the LSTM network is used to learn the long-term dependencies and dynamic patterns of meteorological data; in step S5, the Transformer encoder structure captures the complex interactions between multimodal features through its multi-head self-attention mechanism; in step S6, the anomaly detection and classification module completes anomaly discrimination and fine-grained classification through independent or shared parameter classifiers.

[0045] Compared with existing technologies, this invention provides a multimodal deep learning framework and method for anomaly detection of crop diseases and pests across crops and modalities, which has the following beneficial effects:

[0046] 1. This multimodal deep learning framework and method for anomaly detection of crop diseases and pests across crops and modalities effectively integrates visual morphological information from UAVs, dynamic changes in temporal weather data, and prior knowledge of crop growth stages through an innovative cross-modal Transformer fusion mechanism. It overcomes the limitations of single-modal information, more comprehensively depicts the complex mechanisms of disease and pest occurrence, and thus significantly improves detection accuracy. Especially in complex field environments, the LSTM module is specifically designed to process temporal weather data, which can capture the long-term temporal dependence and dynamic patterns of meteorological factors, helping to reveal early signs and trends of disease and pest occurrence, and enabling earlier warnings.

[0047] 2. This multimodal deep learning framework and method for anomaly detection of crop diseases and pests across crops and modalities alleviates the learning difficulties for minority disease and pest samples by integrating sample imbalance handling modules (such as improved loss functions, data augmentation, GAN synthesis, etc.) into the framework, thereby improving the detection capability for rare diseases and pests. The improved ResNet can be better generalized to different crops through transfer learning and attention mechanisms. The fusion of multimodal information enables the model to learn not only the visual features of diseases and pests, but also their association with the environment and the crop's own state. This association can be transferred to new crops to a certain extent, and good detection performance can be achieved by fine-tuning with a small number of new crop samples.

[0048] 3. This multimodal deep learning framework and method for anomaly detection of crop diseases and pests across crops and modalities is presented. The modules of the framework are relatively independent and can be adjusted and replaced according to actual application scenarios and data availability (such as changing different visual backbones, adjusting the number of LSTM layers, modifying the Transformer structure, etc.), which facilitates subsequent research and engineering application expansion. Detailed Implementation

[0049] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example

[0051] Examples of multimodal deep learning frameworks and methods for anomaly detection of crop diseases and pests across crops and modalities.

[0052] A multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities, including:

[0053] A multimodal data input layer is used to receive and preprocess data in at least three modalities: UAV remote sensing image data, time-series meteorological data, and crop growth stage information;

[0054] The visual feature extraction module is connected to the multimodal data input layer. It uses an improved ResNet residual network as the basic architecture to extract features from the UAV remote sensing image data and obtain visual feature vectors.

[0055] The time-series meteorological feature extraction module is connected to the multimodal data input layer and uses a long short-term memory network (LSTM) to extract time-series features from the time-series meteorological data to obtain a time-series meteorological feature vector.

[0056] The growth stage embedding module is connected to the multimodal data input layer and encodes the crop growth stage information into a fixed-dimensional growth stage embedding vector.

[0057] The cross-modal fusion module is connected to the visual feature extraction module, the temporal meteorological feature extraction module, and the growth stage embedding module, respectively. It uses an attention-based fusion mechanism to deeply fuse the visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector to generate a joint feature representation.

[0058] The anomaly detection and classification module is connected to the cross-modal fusion module. It performs anomaly detection of crop diseases and pests based on the joint feature representation, and further classifies the disease and pest type or growth abnormality type when anomalies are detected.

[0059] Specifically, the improved ResNet incorporates the following improvements over the original ResNet:

[0060] A channel attention mechanism is introduced to adaptively recalibrate the channel feature responses;

[0061] Introduce spatial attention mechanisms to enhance focus on key spatial locations in pest and disease-affected areas;

[0062] A multi-scale feature fusion strategy is adopted to fuse feature maps of different levels to improve the detection capability of pest and disease areas of different sizes.

[0063] For small sample crops or pest / disease categories, transfer learning or meta-learning initialization strategies are introduced.

[0064] Specifically, time-series meteorological data includes sequence data of at least one or more of temperature, humidity, rainfall, and light intensity over a historical period; the LSTM network is used to capture the dynamic changes of meteorological factors over time and their potential correlation with the occurrence of pests and diseases.

[0065] Specifically, if the crop growth stage information is a discrete category label, the growth stage embedding module uses a learnable embedding layer to map the discrete growth stage labels into a high-dimensional dense vector; or, if the crop growth stage information contains continuous growth cycle parameters, then it is encoded into a growth stage embedding vector through a fully connected layer.

[0066] Specifically, the cross-modal fusion module employs an attention-based fusion mechanism, which is a Transformer encoder structure or a variant thereof, including:

[0067] The visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector are adjusted in dimension to form corresponding feature sequences;

[0068] The feature sequences of each modality are concatenated or mapped to a unified embedding dimension through linear projection to form the initial fusion sequence;

[0069] The initial fusion sequence is processed using a multi-head self-attention mechanism and a feedforward neural network to model the dependencies between features within a modality and the interactions between features of different modalities, and to output an enhanced joint feature representation.

[0070] Specifically, the cross-modal fusion module also introduces modality-specific positional encoding to preserve the original temporal or spatial positional information of each modality feature; or, it introduces a cross-modal attention mechanism to explicitly model the attention weight of a certain modality feature to other modality features.

[0071] Specifically, the anomaly detection and classification module includes:

[0072] An anomaly detection branch is used to determine whether the current crop has pests, diseases, or abnormal growth.

[0073] A classification and discrimination branch, when the anomaly discrimination branch determines that it is abnormal, further distinguishes the specific type of pest or disease or the type of abnormal growth;

[0074] The anomaly detection branch and the classification detection branch share some fusion features, or each uses an independent classifier.

[0075] Specifically, it also includes a sample imbalance handling module, integrated into the training process of the framework, which employs at least one of the following methods to alleviate the sample imbalance problem:

[0076] Weighted loss functions for minority class samples, such as Focal Loss or Class-Balanced Loss;

[0077] Data augmentation strategies, especially random cropping, rotation, color jittering, MixUp or CutMix operations on drone images of a few types of pests and diseases;

[0078] Synthesis of small-sample pest and disease images based on Generative Adversarial Network (GAN).

[0079] A method for detecting crop pest and disease anomalies across crops and modalities includes the following steps:

[0080] S1: Collect and preprocess UAV remote sensing image data, time-series meteorological data, and crop growth stage information through the multimodal data input layer;

[0081] S2: The visual feature extraction module extracts features from the preprocessed UAV remote sensing image data to obtain visual feature vectors;

[0082] S3: The time series meteorological feature extraction module extracts time series features from the preprocessed time series meteorological data to obtain the time series meteorological feature vector;

[0083] S4: Encode crop growth stage information through the growth stage embedding module to obtain the growth stage embedding vector;

[0084] S5: Through the cross-modal fusion module, an attention-based fusion mechanism is used to deeply fuse the visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector to generate a joint feature representation;

[0085] S6: The anomaly detection and classification module performs anomaly detection of crop diseases and pests based on the joint feature representation. If an anomaly is detected, the disease or pest type or growth abnormality type is further classified, and the detection result is output.

[0086] Specifically, in step S2, the improved ResNet enhances the effectiveness of visual feature extraction by introducing channel attention mechanism, spatial attention mechanism, multi-scale feature fusion strategy and / or few-shot learning initialization strategy; in step S3, the LSTM network is used to learn the long-term dependencies and dynamic patterns of meteorological data; in step S5, the Transformer encoder structure captures the complex interactions between multimodal features through its multi-head self-attention mechanism; in step S6, the anomaly detection and classification module completes anomaly discrimination and fine-grained classification through independent or shared parameter classifiers.

[0087] Through the above technical solution, this invention, via an innovative cross-modal Transformer fusion mechanism, effectively integrates the visual morphological information of UAVs, the dynamic changes in temporal meteorological data, and prior knowledge of crop growth stages. This overcomes the limitations of single-modal information, more comprehensively depicting the complex mechanisms of pest and disease occurrence, thus significantly improving detection accuracy. Especially in complex field environments, the LSTM module, specifically designed for processing temporal meteorological data, can capture the long-term temporal dependence and dynamic patterns of meteorological factors, helping to reveal early signs and trends of pest and disease occurrence, enabling earlier warnings. Furthermore, by integrating a sample imbalance processing module (such as an improved loss function, data augmentation, and GAN synthesis) into the framework, it alleviates the limitations of single-modal information. Learning from a few types of pest and disease samples is difficult, which improves the detection capability for rare pests and diseases. The improved ResNet can better generalize to different crops through transfer learning and attention mechanisms. The fusion of multimodal information enables the model to learn not only the visual features of pests and diseases, but also their association with the environment and the crop's own state. This association can be transferred to new crops to a certain extent. With fine-tuning using a small number of new crop samples, good detection performance can be achieved. The modules of the framework are relatively independent and can be adjusted and replaced according to the actual application scenario and data availability (such as changing different visual backbones, adjusting the number of LSTM layers, modifying the Transformer structure, etc.), which facilitates subsequent research and engineering application expansion.

[0088] Example 1

[0089] Multimodal data input layer 110: Receives RGB image I (e.g., resolution 512x512) from the UAV platform, time-series meteorological data M (e.g., daily averages of [temp, humidity, rainfall, sunlight] over the past 14 days, totaling 14x4 = 56 feature points), and crop growth stage label G (e.g., "jointing stage", encoded as an integer 2). Image I is resized to 224x224 and normalized to [0,1]; meteorological data M is Z-score normalized; and the growth stage label G is embedded into a d_g dimensional vector through the embedding layer E_embed.

[0090] Visual feature extraction module 120: An improved ResNet50 is used as the backbone. Specifically, an SE (Squeeze-and-Excitation) channel attention module is introduced after each Bottleneck module of ResNet50, and after the final global average pooling, the shallow feature map (such as the output of layer 2) is further fused with the deep feature map (the output of layer 4) in an FPN manner after upsampling to obtain a 2048-dimensional visual feature vector V_vis.

[0091] Time-series meteorological feature extraction module 130: Input the standardized meteorological data sequence M (14x4) into a single-layer LSTM network, set the LSTM hidden layer dimension to 128, and take the output of the last time step as a 256-dimensional time-series meteorological feature vector V_met.

[0092] Growth stage embedding module 140: The growth stage label G (assuming a total of 5 stages, index 0-4) is mapped to a 64-dimensional growth stage embedding vector V_stage through a learnable embedding layer.

[0093] Cross-modal fusion module 150: This is the core module.

[0094] Feature Projection and Serialization: V_vis (2048 dimensions) is projected to 512 dimensions through a linear layer; V_met (256 dimensions) is projected to 512 dimensions through a linear layer; V_stage remains 64 dimensions (or is also projected to 512 dimensions to unify the dimensions; in this embodiment, it is kept for simplicity and unified later after concatenation). Considering that V_vis is a global feature, it can be regarded as a sequence of length 1; V_met is a sequence of length 14 (each time step corresponds to a meteorological feature vector, so the 256-dimensional V_met needs to be split into 14 16-dimensional vectors or V_met can be regarded as a whole feature vector, and a length of 1 is more reasonable. Since the LSTM outputs the representation of the entire sequence, V_met is a single vector, a sequence of length 1. V_stage is also a single vector, a sequence of length 1).

[0095] Constructing the fusion sequence: Concatenate the feature vectors of the three modalities into a sequence S_in = [V_vis_proj; V_met_proj; V_stage]. Assuming V_vis_proj and V_met_proj are both 512-dimensional and V_stage is 64-dimensional, then the total dimension of S_in is 512 + 512 + 64 = 1088 dimensions, and the sequence length is 3 (each modality feature is treated as a token).

[0096] Add position encoding: Since the sequence length is short (3), a learnable position encoding P_pos can be added to each token to obtain S_in_pos=S_in+P_pos.

[0097] Transformer encoding: S_in_pos is input into a structure containing two Transformer Encoder Layers. Each Encoder Layer includes a multi-head self-attention mechanism (number of heads h = 8, dimension of each head 512 / h≈64) and a feedforward neural network (hidden layer dimension 2048, activation function ReLU). After Transformer encoding, an enhanced joint feature sequence S_out is obtained.

[0098] Feature aggregation: Perform global average pooling on S_out (or take the output of the first token, or perform attention pooling) to obtain the final 1024-dimensional joint feature representation F_fused.

[0099] Anomaly Detection and Classification Module 160:

[0100] Anomaly detection branch: F_fused is input to a binary classifier (such as a two-layer fully connected network, with 512 dimensions in the middle layer and 2 dimensions in the output layer), activated by Softmax, and outputs the probability of normal / abnormal.

[0101] Classification and discrimination branch: If the anomaly detection branch outputs an anomaly, then F_fused is input to another multi-classifier (such as a three-layer fully connected network with intermediate layers of 512-dimensional and 256-dimensional dimensions, and an output layer of C-dimensional dimensions, where C is the number of pest and disease types + 1 (the subdivision under the major anomaly category) or directly the number of all possible anomaly types), and Softmax activation is used to output the probability of the specific category.

[0102] Loss function: Cross-Entropy Loss is used in the anomaly detection branch. Focal Loss is used in the classification branch to address the imbalance problem, where the α parameter is dynamically adjusted according to the number of samples in each class, and γ = 2.

[0103] Imbalanced sample handling: In the data preprocessing stage, the drone images of a few types of pests and diseases were randomly rotated (±15°), horizontally flipped, color jittered (brightness ±0.2, contrast ±0.2), randomly cropped (retaining 80% of the area), and enhanced with MixUp (α=0.2).

[0104] Example 2

[0105] Improved ResNet: An SE module is inserted after each Bottleneck residual block of ResNet50's conv2_x, conv3_x, conv4_x, and conv5_x. The SE module first performs global average pooling on the feature map to obtain channel statistics, then obtains channel weights through two fully connected layers (dimensionality reduction ratio of 16) and Sigmoid activation, and finally multiplies them with the original feature map. For FPN fusion, the output of conv3_x (C3, 256-dimensional, 56x56) is upsampled by 2 times, and then combined with the output of conv5_x (C5, 2048-dimensional, 7x7) through a 1x1 convolution to reduce the dimensionality to 256 dimensions. After element-wise addition, it undergoes a 3x3 convolution and upsampling, and finally merges with C5 to obtain a feature map that combines details and semantics. Then, global average pooling is performed to obtain the visual feature V_vis.

[0106] Another variation of cross-modal Transformer fusion: to explicitly model intermodal interactions, cross-modal cross-attention can be introduced after standard self-attention. For example, a visual feature token can be used as the query, and meteorological and growth stage features can be concatenated as the key and value to calculate the attention of the visual token to other modalities; and vice versa. This helps the model learn "which visual features are more important under specific growth stages and meteorological conditions".

[0107] Growth stage embedding: If the crop growth stage information is more refined, such as including continuous variables such as accumulated temperature and number of days in the growing season, these continuous variables can be directly concatenated and encoded into V_stage through a fully connected layer (e.g., input dimension 2, output dimension 64), and then fused with the results of discrete label embedding (e.g., added or concatenated and then projected).

[0108] Example 3: Model Training and Deployment

[0109] Training data: Images, corresponding meteorological data, and growth stage information from multiple datasets were collected, including those for wheat rust, powdery mildew, aphid damage, and rice blast, sheath blight, and rice stem borer damage. These datasets were divided into training, validation, and test sets in a 7:2:1 ratio. Oversampling and data augmentation were performed on minority class samples.

[0110] Training process: The AdamW optimizer was used with an initial learning rate of 1e-4, weight decay of 1e-5, and a batch size of 32. Cosine annealing was used for learning rate scheduling. Training was conducted for 200 epochs, and early stopping was initiated when the accuracy on the validation set no longer improved.

[0111] Deployment: The trained model can be deployed on edge computing devices (such as drone-borne computing units, field smart monitoring stations) or cloud servers. After the drone inspection captures images, the model can be input with real-time / recent meteorological data and estimated growth stage to obtain detection results, which are then transmitted to the user terminal via wireless network.

[0112] Experimental verification: Comparative experiments were conducted in wheat and rice experimental fields in a certain region. The framework of this invention significantly outperformed the single-modal CNN model using only images, the simple feature splicing and fusion model, and the multimodal model that does not consider time-series meteorological information in terms of accuracy, recall (especially for minority classes) and F1 score for pest and disease detection. This verifies the effectiveness and superiority of the invention.

[0113] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities, characterized by: include: A multimodal data input layer is used to receive and preprocess data in at least three modalities: UAV remote sensing image data, time-series meteorological data, and crop growth stage information; The visual feature extraction module is connected to the multimodal data input layer and uses an improved ResNet residual network as the basic architecture to extract features from the UAV remote sensing image data to obtain visual feature vectors. The time-series meteorological feature extraction module is connected to the multimodal data input layer and uses a long short-term memory network (LSTM) to extract time-series features from the time-series meteorological data to obtain a time-series meteorological feature vector. The growth stage embedding module is connected to the multimodal data input layer and encodes the crop growth stage information into a fixed-dimensional growth stage embedding vector. The cross-modal fusion module is connected to the visual feature extraction module, the temporal meteorological feature extraction module, and the growth stage embedding module, respectively. It uses an attention-based fusion mechanism to deeply fuse the visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector to generate a joint feature representation. The anomaly detection and classification module is connected to the cross-modal fusion module. It performs anomaly detection of crop diseases and pests based on the joint feature representation, and further classifies the disease and pest type or growth abnormality type when anomalies are detected.

2. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 1, characterized in that: The improved ResNet incorporates the following improvements over the original ResNet: A channel attention mechanism is introduced to adaptively recalibrate the channel feature responses; Introduce spatial attention mechanisms to enhance focus on key spatial locations in pest and disease-affected areas; A multi-scale feature fusion strategy is adopted to fuse feature maps of different levels to improve the detection capability of pest and disease areas of different sizes. For small sample crops or pest / disease categories, transfer learning or meta-learning initialization strategies are introduced.

3. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 1, characterized in that: The time-series meteorological data includes at least one or more sequence data of temperature, humidity, rainfall, and light intensity over a historical period; the LSTM network is used to capture the dynamic changes of meteorological factors over time and their potential correlation with the occurrence of pests and diseases.

4. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 1, characterized in that: The crop growth stage information is a discrete category label, and the growth stage embedding module uses a learnable embedding layer to map the discrete growth stage labels into a high-dimensional dense vector; or, the crop growth stage information contains continuous growth cycle parameters, which are then encoded into a growth stage embedding vector through a fully connected layer.

5. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 1, characterized in that: The cross-modal fusion module employs an attention-based fusion mechanism, specifically a Transformer encoder structure or a variant thereof, including: The visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector are adjusted in dimension to form corresponding feature sequences; The feature sequences of each modality are concatenated or mapped to a unified embedding dimension through linear projection to form the initial fusion sequence; The initial fusion sequence is processed using a multi-head self-attention mechanism and a feedforward neural network to model the dependencies between features within a modality and the interactions between features of different modalities, and to output an enhanced joint feature representation.

6. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 5, characterized in that: The cross-modal fusion module also introduces modality-specific positional encoding to preserve the original temporal or spatial positional information of each modality feature; or, it introduces a cross-modal cross-attention mechanism to explicitly model the attention weight of a certain modality feature to other modality features.

7. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 1, characterized in that: The anomaly detection and classification module includes: An anomaly detection branch is used to determine whether the current crop has pests, diseases, or abnormal growth. A classification and discrimination branch, when the anomaly discrimination branch determines that it is abnormal, further distinguishes the specific type of pest or disease or the type of abnormal growth; The anomaly detection branch and the classification detection branch share some fusion features, or each uses an independent classifier.

8. The multimodal deep learning framework for anomaly detection of crop diseases and pests across crops and modalities as described in claim 1, characterized in that: It also includes a sample imbalance handling module, integrated into the training process of the framework, which uses at least one of the following methods to alleviate the sample imbalance problem: Weighted loss functions for minority class samples, such as Focal Loss or Class-Balanced Loss; Data augmentation strategies, especially random cropping, rotation, color jittering, MixUp or CutMix operations on drone images of a few types of pests and diseases; Synthesis of small-sample pest and disease images based on Generative Adversarial Network (GAN).

9. A method for detecting crop pest and disease anomalies across crops and modalities, characterized by: Includes the following steps: S1: Collect and preprocess UAV remote sensing image data, time-series meteorological data, and crop growth stage information through the multimodal data input layer; S2: The visual feature extraction module extracts features from the preprocessed UAV remote sensing image data to obtain visual feature vectors; S3: The time series meteorological feature extraction module extracts time series features from the preprocessed time series meteorological data to obtain the time series meteorological feature vector; S4: Encode crop growth stage information through the growth stage embedding module to obtain the growth stage embedding vector; S5: Through the cross-modal fusion module, an attention-based fusion mechanism is used to deeply fuse the visual feature vector, the temporal meteorological feature vector, and the growth stage embedding vector to generate a joint feature representation; S6: The anomaly detection and classification module performs anomaly detection of crop diseases and pests based on the joint feature representation. If an anomaly is detected, the disease or pest type or growth abnormality type is further classified, and the detection result is output.

10. The multimodal deep learning framework and method for anomaly detection of crop diseases and pests across crops and modalities as described in claim 9, characterized in that: In step S2, the improved ResNet enhances the effectiveness of visual feature extraction by introducing channel attention mechanism, spatial attention mechanism, multi-scale feature fusion strategy and / or few-shot learning initialization strategy; in step S3, the LSTM network is used to learn the long-term dependencies and dynamic patterns of meteorological data; in step S5, the Transformer encoder structure captures the complex interactions between multimodal features through its multi-head self-attention mechanism; in step S6, the anomaly detection and classification module completes anomaly discrimination and fine-grained classification through independent or shared parameter classifiers.