Small sample plant disease identification method and system based on CLIP model lightweight adaptation

Abstract

The invention discloses a small sample plant disease identification method and system based on CLIP model lightweight adaptation, and the method comprises the steps: collecting plant disease and insect pest images, carrying out the disease category labeling, and constructing a plant disease and insect pest image data set; generating multi-source text descriptions in one-to-one correspondence with the plant disease and insect pest images in the plant disease and insect pest image data set; a small sample plant disease recognition model is constructed based on the CLIP model, and a lightweight adapter module is embedded into the output end of an image encoder of the CLIP model to output adaptive image features; taking the plant disease and insect pest image in the training set and the matched multi-source text description as input, and extracting image features and text features in the plant disease and insect pest image; training the small sample plant disease recognition model by adopting a mixed loss function; verifying the model performance through the verification set to determine an optimal model parameter; and inputting a to-be-detected plant image into the trained small sample plant disease identification model to obtain a category identification result of the plant disease.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to a small-sample plant disease identification method and system based on the CLIP model with lightweight adaptation. Background Technology

[0002] Timely and accurate identification of plant diseases is crucial for ensuring global food security and promoting sustainable agricultural practices. While deep learning technology has been widely explored in the field of plant pathology classification, its deployment in practical agricultural scenarios still faces numerous technical bottlenecks, as follows:

[0003] Traditional deep learning methods, especially convolutional neural networks (CNNs), have demonstrated high performance in plant disease classification tasks, but their performance is highly dependent on large-scale, high-quality labeled datasets. However, in real-world agricultural scenarios, obtaining such datasets faces significant limitations: on the one hand, labeling plant diseases requires specialized personnel, which is labor-intensive and costly; on the other hand, data on rare or newly emerging diseases exhibit a long-tailed distribution, making it difficult to generate large-scale labeled data. Consequently, these issues severely limit the scalability of standard CNN models in practical deployments, failing to meet the diverse and complex needs of agricultural disease identification.

[0004] To alleviate the problem of data scarcity, Few-Shot Learning (FSL) technology has emerged. Its core objective is to achieve generalized recognition of new disease categories using limited support samples, providing a solution to the data shortage problem. However, traditional FSL strategies still face bottlenecks in implementation: First, meta-learning algorithms generally suffer from high computational overhead and unstable training processes, making them difficult to adapt to the low-cost deployment requirements of agricultural scenarios. Second, metric-based methods (such as ProtoNet) are sensitive to data noise and cannot effectively distinguish disease subclasses with similar visual features, resulting in difficulty in guaranteeing recognition accuracy.

[0005] In recent years, visual-language models (VLMs), such as the contrastive language-image pre-trained (CLIP) model, have demonstrated strong zero-shot recognition capabilities by aligning visual and textual modalities within a unified semantic space, offering a new potential solution to the challenging task of plant disease identification. However, when these models are directly applied to plant disease identification tasks, a significant domain gap exists. Specifically, the pre-training data for CLIP models primarily consists of object-centric natural images (e.g., distinguishing between "cats" and "cars"), focusing on the recognition of global object geometry. In contrast, plant disease identification is a fine-grained classification task, relying on subtle local texture changes such as leaf discoloration spots, fungal patterns, and vein discoloration, rather than global object features. Therefore, this difference between the pre-training task and the target task means that the frozen visual representation of the original CLIP model cannot effectively capture the discriminative pathological features required for plant disease identification.

[0006] To bridge the aforementioned domain bias, related studies have attempted to optimize models through fine-tuning. However, existing fine-tuning schemes still have significant drawbacks: First, full-parameter fine-tuning is computationally expensive and difficult to adapt to deployment conditions in agricultural scenarios. Second, full-parameter fine-tuning can easily lead to catastrophic forgetting of the model's general feature representations, resulting in the loss of its original generalization ability. Third, existing fine-tuning methods mostly focus on text-based cue learning, limiting their ability to model fine-grained visual changes in plant diseases. Fourth, some adaptation methods based on external caching or memory mechanisms experience parameter scaling that increases with the number of disease categories, leading to high storage pressure and low operating efficiency in large-scale agricultural disease identification applications, thus failing to meet practical application needs.

[0007] In summary, current plant disease identification technologies all have insurmountable shortcomings and cannot efficiently and stably meet the application needs of actual agricultural scenarios. There is an urgent need to propose a plant disease identification technology that can adapt to data-scarce scenarios, effectively capture fine-grained features of diseases, and has low computational and storage costs. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides a small-sample plant disease identification method based on the CLIP model with lightweight adaptation. This small-sample plant disease identification method can adapt to data-scarce scenarios, effectively capture fine-grained features of diseases, thereby significantly reducing training computation costs and improving identification accuracy.

[0009] The second objective of this invention is to provide a small-sample plant disease identification system based on a lightweight adaptation of the CLIP model.

[0010] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0011] A lightweight adaptation method for identifying plant diseases in small samples based on the CLIP model includes the following steps:

[0012] Step S1: Collect images of plant diseases and pests and label them with disease categories to construct a plant disease and pest image dataset. Divide the plant disease and pest image dataset into a training set, a validation set, and a test set.

[0013] Step S2: Generate multi-source text descriptions that correspond one-to-one with each plant disease and pest image in the plant disease and pest image dataset, forming text description data that matches the plant disease and pest image dataset;

[0014] Step S3: Construct a small-sample plant disease identification model based on the CLIP model, and freeze all parameters of the image encoder and text encoder in the CLIP model; embed a lightweight adapter module at the output of the image encoder of the CLIP model, and dynamically adjust the original features of the fixed dimension output by the image encoder through the lightweight adapter module to obtain adapted image features;

[0015] Step S4: Construct a hybrid loss function, taking the plant disease and pest images and their accompanying multi-source text descriptions in the training set as input, extracting image features and text features from the plant disease and pest images, and training the small sample plant disease recognition model based on the hybrid loss function; during the training process, keep the parameters of the image encoder and text encoder of the CLIP model unchanged, and verify the model performance through the validation set to determine the optimal model parameters;

[0016] Step S5: Input the plant image to be tested into the trained small sample plant disease recognition model to obtain the plant disease category recognition result.

[0017] Preferably, in step S1, after collecting plant disease and pest images, the plant disease and pest images are first pre-processed before annotation, and the pre-processing before annotation includes size scaling; after the plant disease and pest images are annotated, the annotated plant disease and pest images are then normalized and data augmented, wherein the data augmentation specifically includes random cropping, random flipping, rotation and color jitter.

[0018] Preferably, in step S2, the specific steps for generating the multi-source text description are as follows:

[0019] Step S201: Generate a basic description using a predefined template, wherein the format of the predefined template is "a leaf photo with [category name]";

[0020] Step S202: Generate a visual description and a domain knowledge description of the plant disease, wherein the visual description includes visual features such as lesion color, lesion shape, lesion texture, and disease location; and the domain knowledge description includes the causes of the disease.

[0021] Step S203: Generate multiple text descriptions with different expressions for each disease category, extract the features of each text description and perform averaging to form robust text prototype features for that disease category.

[0022] Preferably, in step S2, the multi-source text description is generated using an automated text generation model, which is a large language model.

[0023] Preferably, in step S3, the lightweight adapter module includes a main adapter branch and a gated branch; the main adapter branch and the gated branch receive the raw features output by the image encoder in the CLIP model in parallel, wherein,

[0024] The main adaptation branch adopts a bottleneck structure, which compresses and restores the feature dimension by means of learnable linear transformation, and then activates it by means of the GELU activation function to generate fine-grained features of diseases specific to the task.

[0025] The gated branch first performs a global feature aggregation operation on the original features, and then generates a channel gated vector through multi-layer learnable transformation. The channel gated vector is used to perform channel-weighted modulation on the fine-grained disease features output by the main adaptation branch to obtain the main adaptation features. The main adaptation features are fused with the original features by using a residual connection method. The fused result is then subjected to layer normalization to finally obtain the adapted image features.

[0026] Preferably, in step S4, the hybrid loss function is a weighted combination of contrast loss and center loss, wherein the contrast loss is used to constrain the alignment relationship between image features and corresponding text features in the feature space; the center loss is used to constrain the features of samples of the same disease category to cluster towards the center of their corresponding category features, thereby enhancing the compactness of intra-class features; the parameters of the lightweight adapter module are optimized by jointly minimizing the contrast loss and the center loss.

[0027] Preferably, in step S5, the small-sample plant disease identification model determines the disease category of the plant image to be tested by calculating the similarity between the adapted image features and the text features of each disease category, and outputs the corresponding identification result.

[0028] A lightweight, CLIP-based system for identifying small-sample plant diseases includes:

[0029] Image and text data construction module: used to collect plant disease and pest images and label disease categories, construct plant disease and pest image dataset, divide the plant disease and pest image dataset into training set, validation set and test set, and generate multi-source text descriptions that correspond one-to-one with each plant disease and pest image in the plant disease and pest image dataset, forming text description data that matches the plant disease and pest image dataset.

[0030] The model building and feature adaptation module includes a lightweight adapter module. The model building and feature adaptation module is used to build a small sample plant disease identification model based on the CLIP model, freeze all parameters of the image encoder and text encoder in the CLIP model, and embed the lightweight adapter module at the output of the image encoder in the CLIP model to dynamically adjust the fixed-dimensional original features output by the image encoder and output adapted disease image features.

[0031] Model training module: This module is used to construct a hybrid loss function. It receives plant disease and pest images and their accompanying multi-source text descriptions from the training set output by the image and text data construction module as joint input. It extracts the corresponding image features and text features through the small sample plant disease identification model. Based on the hybrid loss function, it iteratively trains the small sample plant disease identification model, ensuring that the parameters of the image encoder and text encoder of the CLIP model remain unchanged during the training process. It also verifies the disease identification performance of the model through the validation set, determines and stores the optimal model parameters, and obtains the optimal small sample plant disease identification model.

[0032] Recognition and reasoning module: It is used to receive the plant image to be tested, call the small sample plant disease recognition model corresponding to the optimal parameters stored in the model training module, perform feature extraction and matching discrimination on the plant image to be tested, and output the plant disease category recognition result.

[0033] An electronic device includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the small-sample plant disease identification method based on CLIP model lightweight adaptation.

[0034] A computer-readable storage medium stores, in the form of computer-readable instructions, a computer program implemented according to the described CLIP model-based lightweight adaptation method for identifying small-sample plant diseases, which, when called by a computer, executes the steps included in the corresponding method.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] 1. The small-sample plant disease identification method based on CLIP model lightweight adaptation of the present invention introduces a lightweight adapter module containing a main adaptation branch and a gated branch under the premise of freezing the parameters of the pre-trained CLIP model, so as to realize the dynamic calibration of the fine-grained features of plant diseases, thereby effectively enhancing the ability to express key information related to diseases, and thus improving the accuracy and reliability of disease identification.

[0037] 2. The small-sample plant disease identification method based on CLIP model lightweight adaptation of the present invention trains only the lightweight adapter module embedded in the output end of the image encoder. It achieves efficient feature adaptation through bottleneck structure and channel gating mechanism. Compared with the method of fine-tuning all parameters of the pre-trained model, the small-sample plant disease identification method of the present invention significantly reduces the scale of parameter update and computational overhead in the training stage. In addition, it combines multi-source text description and hybrid loss constraint, thereby improving the generalization ability of the model under small sample conditions, and is suitable for deployment in resource-constrained agricultural application scenarios.

[0038] 3. The small-sample plant disease identification method based on CLIP model lightweight adaptation of the present invention effectively solves the problems of domain bias in fine-grained features of plant diseases and difficulty in small-sample training of general vision models through dual-path adapter structure and hybrid loss function. Experimental results show that the small-sample plant disease identification method of the present invention can achieve high-precision disease identification with a very low number of parameters (0.59M) in the case of very few samples, which is significantly better than the existing fine-tuning methods and is suitable for agricultural resource-constrained scenarios. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the small-sample plant disease identification method based on the CLIP model lightweight adaptation of the present invention.

[0040] Figure 2 This is a flowchart illustrating the overall process architecture of the small-sample plant disease identification method based on the CLIP model lightweight adaptation of the present invention.

[0041] Figure 3 This is a schematic diagram of the internal structure of the lightweight adapter module.

[0042] Figure 4 This is a visualization comparison of the t-SNE features of the small-sample plant disease identification method based on the CLIP model and its lightweight adaptation in the present invention on the PlantVillage dataset. The left side shows the distribution of the original features output by the image encoder in the traditional CLIP model on the PlantVillage dataset; the right side shows the distribution of the image features adapted by the lightweight adapter module on the PlantVillage dataset.

[0043] Figure 5 This is a visualization comparison of t-SNE features of the small-sample plant disease identification method based on the CLIP model with lightweight adaptation according to the present invention on a field joint dataset. The left side shows the distribution of the original features output by the image encoder in the traditional CLIP model on the field joint dataset; the right side shows the distribution of the image features adapted by the lightweight adapter module on the field joint dataset. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0045] Example 1

[0046] See Figures 1-5 The present invention provides a small-sample plant disease identification method based on CLIP model lightweight adaptation, comprising the following steps:

[0047] Step S1: Collect plant disease and pest images and label them with disease categories to construct a plant disease and pest image dataset. Divide the plant disease and pest image dataset into a training set, a validation set, and a test set. After collecting the plant disease and pest images, perform preprocessing on the images before labeling, including size scaling. After labeling the plant disease and pest images, normalize and perform data augmentation on the labeled images. The data augmentation specifically uses methods such as random cropping, random flipping, rotation, and color jittering.

[0048] To verify the effectiveness and generalization ability of the small-sample plant disease identification method of the present invention in different scenarios, this embodiment uses laboratory environment datasets and real field environment datasets for experimental evaluation. The laboratory environment dataset uses the PlantVillage dataset, which contains plant leaf images collected under controlled laboratory conditions. The background of these plant leaf images is relatively simple and the lighting conditions are stable. It mainly includes images of healthy and diseased leaves of various common crops. The PlantVillage dataset covers a variety of plant species and corresponding disease types, and can be used to verify the model's ability to identify plant diseases under standard conditions.

[0049] To evaluate the robustness of the model in real and complex environments, this embodiment further constructs and adopts a field joint dataset. The field joint dataset consists of multiple data sources collected from real fields, including disease images of crops such as beans, coffee, and pumpkins. Since the images in this field joint dataset are collected in natural planting environments, there are complex backgrounds, changes in lighting, occlusion, and diverse shooting angles, which can more realistically reflect the characteristics of plant disease images in actual agricultural application scenarios.

[0050] The dataset is divided into training, validation, and test sets according to a preset ratio (e.g., 7:2:1) for model training, parameter tuning, and performance evaluation. During training, uniform preprocessing operations are performed on the input images, including size scaling and normalization. In addition, to improve the generalization ability of the model, data augmentation strategies such as random cropping, random flipping, rotation, and color jitter can be adopted.

[0051] Step S2: To fully utilize the image-text alignment capability of the CLIP model, in this embodiment, a multi-source text description is constructed for each plant disease and pest category, thereby generating a multi-source text description that corresponds one-to-one with each plant disease and pest image in the plant disease and pest image dataset, forming text description data that matches the plant disease and pest image dataset; wherein, the specific steps for generating the multi-source text description are as follows:

[0052] Step S201: Generate a basic description using a predefined template, wherein the format of the predefined template is "a leaf photo with [category name]";

[0053] Step S202: Generate a visual description and a domain knowledge description of the plant disease, wherein the visual description includes visual features such as lesion color, lesion shape, lesion texture, and disease location; the domain knowledge description includes the causes of the disease.

[0054] In this embodiment, an automated text generation model is introduced to generate visual description text and domain knowledge description text; the automated text generation model can be a large language model.

[0055] Step S203: Generate multiple text descriptions with different expressions for each disease category, extract the features of each text description and perform averaging to form robust text prototype features for that disease category;

[0056] In this embodiment, the text description is feature extracted using the CLIP text encoder; multiple text features corresponding to the same disease category are aggregated to form the text prototype features of that disease category, thereby improving the robustness of the text representation.

[0057] Step S3: Construct a small-sample plant disease identification model based on the CLIP model, and freeze all parameters of the image encoder and text encoder in the CLIP model; embed a lightweight adapter module at the output of the image encoder of the CLIP model, and dynamically adjust the original features of the fixed dimension output by the image encoder through the lightweight adapter module to obtain adapted image features;

[0058] In this embodiment, a pre-trained CLIP model is used as the feature extraction infrastructure. The image encoder in the CLIP model adopts a Vision Transformer-based encoding structure, preferably the ViT-L / 14 architecture, and the text encoder in the CLIP model adopts a Transformer-based text encoding structure. During the model training and inference phases, the pre-trained parameters of both the image encoder and the text encoder are frozen and do not participate in gradient updates to preserve their general visual-semantic representation capabilities and reduce training computational overhead. In addition, to compensate for the domain bias between the pre-trained features of the CLIP model and the fine-grained features of plant diseases, a lightweight adapter (PD-Adapter) module is embedded at the output of the image encoder to dynamically adapt the visual features output by the image encoder.

[0059] like Figure 2 As shown, the lightweight adapter (PD-Adapter) module includes a main adapter branch and a gated branch; the main adapter branch and the gated branch receive the raw features output by the image encoder in the CLIP model in parallel;

[0060] Assume the original features output by the image encoder are ,in, For batch size, For feature dimensions; when the image encoder uses the ViT-L / 14 architecture, ;

[0061] The final output characteristic Y of the lightweight adapter (PD-Adapter) module can be expressed as:

[0062] ;

[0063] In the formula: Presentation layer normalization operation, This represents element-wise multiplication. Indicates the output of the main adapter branch. This represents the gate vector output by the gated branch.

[0064] The main adaptation branch is used to learn task-specific fine-grained features of diseases (such as leaf spot texture). This main adaptation branch adopts a bottleneck structure, and its implementation is as follows:

[0065] ;

[0066] In the formula: and The weight matrix is ​​a learnable matrix. The bottleneck dimension is set to 256 in this embodiment. It is a non-linear activation function.

[0067] The bottleneck structure first compresses and then restores the feature dimension by using a learnable linear transformation, which reduces the number of trainable parameters while enhancing the ability to model fine-grained features of diseases and helps to alleviate the overfitting problem under small sample conditions. Then, it is activated by the GELU activation function to generate task-specific fine-grained features of diseases.

[0068] The gated branch is used to generate channel-level attention weights to dynamically filter key feature channels. In this embodiment, the gated branch first performs a global feature aggregation operation on the original features, and then generates channel gated vectors through multiple learnable transformations. The implementation method is as follows:

[0069] ;

[0070] ;

[0071] In the formula: Represents the global average eigenvector; and Let be a learnable weight matrix; where, weight matrix This is used to project the global feature vector from the original feature dimension to a lower dimension, such as a 64-dimensional space, to reduce the parameter size and computational cost of the gated branch; subsequently, through the weight matrix... Restore the features to the channel dimension consistent with the output features of the main adaptation branch; for Activation function; generated channel gating vector The output features (i.e. disease fine-grained features) applied only to the main adaptation branch are used to perform channel-weighted modulation on the output features of the main adaptation branch to obtain the main adaptation features, thereby adaptively enhancing the channels that are highly correlated with disease discrimination while suppressing background noise channels.

[0072] Finally, the master adaptation features and the original features are fused using a residual connection method. The fused result is then subjected to layer normalization, thereby maintaining the stability of the pre-trained feature distribution while introducing plant disease-related discrimination information, ultimately obtaining the adapted image features.

[0073] Step S4: Construct a hybrid loss function, taking the plant disease and pest images and their accompanying multi-source text descriptions in the training set as input, extracting image features and text features from the plant disease and pest images, and training the small sample plant disease recognition model based on the hybrid loss function; during the training process, keep the parameters of the image encoder and text encoder of the CLIP model unchanged, and verify the model performance through the validation set to determine the optimal model parameters;

[0074] In this embodiment, to improve the separability of different plant disease categories in the feature space, the hybrid loss function is a weighted combination of contrast loss and center loss, wherein,

[0075] The contrast loss is used to constrain the alignment relationship between image features and corresponding text features in the feature space, making the matched image-text feature pairs closer in the feature space, while widening the distance between mismatched feature pairs.

[0076] The contrast loss Contrast loss from image to text Contrast loss compared to text-to-image The weighted average is calculated using the following formula:

[0077] ;

[0078] Among them, the image-to-text contrast loss The expression is:

[0079] ;

[0080] Contrast loss from text to image The expression is:

[0081] ;

[0082] In the formula: Indicates batch size, Indicates the first The adapted image features corresponding to each sample This represents the corresponding text features. This represents the similarity measurement function, preferably cosine similarity. This is a temperature coefficient used to adjust the similarity distribution, preferably 0.07.

[0083] The central loss This central loss is used to constrain features of samples from the same disease category to cluster towards the feature center of their corresponding category, thereby enhancing the compactness of intra-class features. The calculation formula is:

[0084] ;

[0085] Combining the two types of losses mentioned above, the hybrid loss function is defined as follows:

[0086] ;

[0087] In the formula: The hyperparameter for balancing the weights is set to 0.7 in this embodiment.

[0088] During training, the parameters of both the image encoder and text encoder in the CLIP model are frozen. Only the parameters of the lightweight adapter (PD-Adapter) module embedded in the output of the image encoder are updated to maintain the general representation ability of the pre-trained model while avoiding overfitting under small sample conditions. In this embodiment, the parameters of the lightweight adapter (PD-Adapter) module are optimized by jointly minimizing the contrast loss and the center loss.

[0089] In the experiments, the few-shot plant disease identification method of this invention was evaluated using the PlantVillage dataset and a combined field dataset. The experiments employed a K-shot few-shot learning setting, where K was set to 1, 2, 4, 8, and 16. For both the PlantVillage and field datasets, mutually exclusive validation and test sets were used for evaluation, with 100 images per class. During training, the parameters of the image encoder and text encoder of the CLIP model were frozen, and only the parameters of the lightweight adapter (PD-Adapter) module were updated. The model was trained using the AdamW optimizer, with 300 epochs and a batch size of 128. The experiments were conducted using the PyTorch framework and trained on a single NVIDIA GeForce RTX4090 GPU. To enhance the model's generalization ability, data augmentation strategies such as randomized cropping, flipping, rotation, and color jittering were employed during training.

[0090] Table 1: Accuracy Comparison Table for PlantVillage Dataset

[0091]

[0092] As shown in Table 1, the small-sample plant disease identification model of the present invention demonstrates superior classification accuracy. Under all K-shot settings of the PlantVillage dataset, the identification accuracy of the small-sample plant disease identification model of the present invention is better than the other six identification models. Its accuracy in the 1-shot setting reaches 71.05%, which is significantly better than CoOp's 34.73% (difference of 36.32%) and Tip-Adapter's 42.59% (difference of 28.46%). This indicates that the small-sample plant disease identification model of the present invention can more effectively utilize pre-trained knowledge under extremely scarce data conditions, thereby overcoming the overfitting problem of traditional fine-tuning methods when samples are insufficient.

[0093] For the 16-shot setting, the accuracy of the small-shot plant disease identification model of this invention reaches 94.26%, which surpasses the accuracy of lightweight CNN models with full parameter fine-tuning (such as 93.57% for EfficientNet-B0 and 93.30% for ResNet18). This result highlights that the small-shot plant disease identification model of this invention achieves classification performance comparable to or even higher than that of deep networks trained with full parameters while maintaining lightweight characteristics (requiring only 0.59M parameters for training). Compared with the CoOp method, which only focuses on text modality adjustment, the small-shot plant disease identification model of this invention captures the subtle features of plant diseases more accurately through dynamic adjustment of visual features.

[0094] To further verify the robustness of the small-sample plant disease identification model of the present invention in complex environments, it was evaluated on a joint field dataset, and the results are shown in Table 2.

[0095] Table 2: Comparison of accuracy rates for joint field datasets

[0096]

[0097] As shown in Table 2, on the joint field dataset with higher visual variability (such as complex lighting and cluttered backgrounds), although the accuracy of all recognition models decreased compared to the laboratory environment, the small-sample plant disease recognition model of this invention still maintained a significant leading position. Specifically, in the 16-shot scenario, the small-sample plant disease recognition model of this invention achieved an accuracy of 76.45%, which is 11.36% higher than CoOp and 14.16% higher than Tip-Adapter. Tip-Adapter showed performance degradation (overfitting signs) from 8-shot to 16-shot, while the small-sample plant disease recognition model of this invention maintained steady growth, which proves its training stability under different data scales. This trend highlights the effectiveness of the small-sample plant disease recognition model of this invention in cross-domain generalization and handling complex field backgrounds.

[0098] Table 3: Model Efficiency Analysis and Comparison

[0099]

[0100] As can be seen from Table 3, the small-sample plant disease identification model of the present invention achieves the best balance between identification accuracy and adaptation efficiency. Although CoOp and LP++ have very few parameters, their accuracy is much lower than that of the present invention (13.21% and 17.29% lower, respectively), which indicates that it is difficult to capture fine-grained plant disease features by only performing text-side optimization or simple linear probing.

[0101] Furthermore, compared to Tip-Adapter based on feature caching mechanisms, this invention exhibits significant technical advantages. The small-sample plant disease identification model of this invention requires only 0.59M trainable parameters, comparable to Tip-Adapter's 0.60M. However, in terms of training computation, the small-sample plant disease identification model of this invention requires only 1.18M FLOPs, approximately 56% of Tip-Adapter's (2.10M FLOPs). While significantly reducing computational overhead, the accuracy of the small-sample plant disease identification model of this invention is actually improved by 4.88% (reaching 94.26%). Therefore, this invention, through the efficient design of channel-level attention mechanisms and residual connections, achieves in-depth mining of disease features while maintaining lightweight characteristics.

[0102] In summary, the small-sample plant disease identification model of this invention not only outperforms existing methods in detection accuracy, but its low storage requirements (0.59M parameters) and low computational burden make it very suitable for integration into agricultural mobile devices or embedded systems with limited computing resources, providing an efficient and feasible solution for real-time monitoring of plant diseases in practical scenarios.

[0103] To visually demonstrate the effectiveness of the small-sample plant disease identification model of this invention in optimizing feature space distribution, t-SNE technology was used to perform dimensionality reduction and visualization analysis on the high-dimensional features extracted by the model; for example... Figure 4 and Figure 5 As shown, the distribution of the original features output by the image encoder in the pre-trained CLIP model and the features adapted by the lightweight adapter (PD-Adapter) module on the PlantVillage dataset and the field joint dataset are compared.

[0104] like Figure 4As shown in the figure (corresponding to the PlantVillage dataset), the original features extracted by the pre-trained CLIP model exhibit significant class overlap, indicating that although the original CLIP model has general visual recognition capabilities, it is difficult to effectively distinguish fine-grained plant disease features (such as similar leaf spot textures) without targeted fine-tuning. In contrast, the feature distribution after adaptation by the lightweight adapter (PD-Adapter) module presents a clear cluster structure, with distinct boundaries between feature clusters of different disease categories, and highly clustered sample points of the same category. This significant improvement proves that the present invention can extract more discriminative disease features.

[0105] like Figure 5 As shown in the diagram (corresponding to the joint field dataset), in field environments with complex backgrounds and varying light levels, the confusion level of the original CLIP features is further aggravated, making it almost impossible to distinguish between different disease categories. However, after applying the small-sample plant disease identification model of this invention, despite the influence of environmental noise, the feature distribution still exhibits good intra-class compactness and inter-class separability. This optimization of the feature space is mainly due to the hybrid loss function introduced in this invention, where the contrastive loss separates different categories, while the center loss forces samples of the same category to cluster towards the class center.

[0106] Furthermore, the visualization results also verified the role of the gated branch; by dynamically suppressing background-related noise channels and enhancing lesion-related texture channels, the lightweight adapter (PD-Adapter) module successfully extracted robust lesion features in complex backgrounds, thereby achieving... Figure 4 and Figure 5 The high-quality feature clustering shown in the figure; these visual evidences are consistent with the quantitative experimental results in Tables 1 and 2, further confirming the technical advantages of the present invention in solving the problems of fine-grained identification and cross-domain generalization of plant diseases.

[0107] Step S5: Input the plant image to be tested into the trained small sample plant disease recognition model to obtain the plant disease category recognition result;

[0108] In this embodiment, the small-sample plant disease identification model determines the disease category of the plant image to be tested by calculating the similarity between the adapted image features and the text features of each disease category, and outputs the corresponding identification result.

[0109] In addition to the methods mentioned above, a classification head can also be used to identify the disease category to which the image of the plant to be tested belongs.

[0110] Example 2

[0111] The present invention provides a lightweight, CLIP-based small-sample plant disease identification system, comprising:

[0112] Image and text data construction module: used to collect plant disease and pest images and label disease categories, construct plant disease and pest image dataset, divide the plant disease and pest image dataset into training set, validation set and test set, and generate multi-source text descriptions that correspond one-to-one with each plant disease and pest image in the plant disease and pest image dataset, forming text description data that matches the plant disease and pest image dataset.

[0113] The model building and feature adaptation module includes a lightweight adapter module. The model building and feature adaptation module is used to build a small sample plant disease identification model based on the CLIP model, freeze all parameters of the image encoder and text encoder in the CLIP model, and embed the lightweight adapter module at the output of the image encoder in the CLIP model to dynamically adjust the fixed-dimensional original features output by the image encoder and output adapted disease image features.

[0114] Model training module: This module is used to construct a hybrid loss function. It receives plant disease and pest images and their accompanying multi-source text descriptions from the training set output by the image and text data construction module as joint input. It extracts the corresponding image features and text features through the small sample plant disease identification model. Based on the hybrid loss function, it iteratively trains the small sample plant disease identification model. During the training process, the parameters of the image encoder and text encoder of the CLIP model remain unchanged. The model's disease identification performance is verified through the validation set. The optimal model parameters are determined and stored to obtain the optimal small sample plant disease identification model.

[0115] Recognition and reasoning module: It is used to receive the plant image to be tested, call the small sample plant disease recognition model corresponding to the optimal parameters stored in the model training module, perform feature extraction and matching discrimination on the plant image to be tested, and output the plant disease category recognition result.

[0116] Example 3

[0117] The electronic device of the present invention includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to perform the steps of the small sample plant disease identification method based on CLIP model lightweight adaptation described in Embodiment 1.

[0118] Example 4

[0119] The computer-readable storage medium of the present invention stores, in the form of computer-readable instructions, a computer program implemented according to the small sample plant disease identification method based on CLIP model lightweight adaptation as described in Embodiment 1. When the computer program is called and executed by a computer, it performs the steps included in the corresponding method.

[0120] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for identifying small-sample plant diseases based on a lightweight adaptation of the CLIP model, characterized in that, Includes the following steps: Step S1: Collect images of plant diseases and pests and label them with disease categories to construct a plant disease and pest image dataset. Divide the plant disease and pest image dataset into a training set, a validation set, and a test set. Step S2: Generate multi-source text descriptions that correspond one-to-one with each plant disease and pest image in the plant disease and pest image dataset, forming text description data that matches the plant disease and pest image dataset; Step S3: Construct a small-sample plant disease identification model based on the CLIP model, and freeze all parameters of the image encoder and text encoder in the CLIP model; embed a lightweight adapter module at the output of the image encoder of the CLIP model, and dynamically adjust the original features of the fixed dimension output by the image encoder through the lightweight adapter module to obtain adapted image features; Step S4: Construct a hybrid loss function, taking the plant disease and pest images and their accompanying multi-source text descriptions in the training set as input, extracting image features and text features from the plant disease and pest images, and training the small sample plant disease recognition model based on the hybrid loss function; during the training process, keep the parameters of the image encoder and text encoder of the CLIP model unchanged, and verify the model performance through the validation set to determine the optimal model parameters; Step S5: Input the plant image to be tested into the trained small sample plant disease recognition model to obtain the plant disease category recognition result.

2. The method for identifying small-sample plant diseases based on the CLIP model with lightweight adaptation according to claim 1, characterized in that, In step S1, after collecting plant disease and pest images, the plant disease and pest images are first pre-processed before annotation, which includes size scaling; after the plant disease and pest images are annotated, the annotated plant disease and pest images are then normalized and data augmented, wherein the data augmentation operations include random cropping, random flipping, rotation, and color jittering.

3. The method for identifying small-sample plant diseases based on the CLIP model with lightweight adaptation according to claim 2, characterized in that, In step S2, the specific steps for generating the multi-source text description are as follows: Step S201: Generate a basic description using a predefined template, wherein the format of the predefined template is "a leaf photo with [category name]"; Step S202: Generate a visual description and a domain knowledge description of the plant disease, wherein the visual description includes visual features such as lesion color, lesion shape, lesion texture, and disease location; and the domain knowledge description includes the causes of the disease. Step S203: Generate multiple text descriptions with different expressions for each disease category, extract the features of each text description and perform averaging to form robust text prototype features for that disease category.

4. The method for identifying small-sample plant diseases based on the CLIP model with lightweight adaptation according to claim 1, characterized in that, In step S2, the multi-source text description is generated using an automated text generation model, which is a large language model.

5. The method for identifying small-sample plant diseases based on the CLIP model with lightweight adaptation according to claim 3, characterized in that, In step S3, the lightweight adapter module includes a main adaptation branch and a gated branch; the main adaptation branch and the gated branch receive the raw features output by the image encoder in the CLIP model in parallel, wherein, The main adaptation branch adopts a bottleneck structure, which compresses and restores the feature dimension by means of learnable linear transformation, and then activates it by means of the GELU activation function to generate fine-grained features of diseases specific to the task. The gated branch first performs a global feature aggregation operation on the original features, and then generates a channel gated vector through multi-layer learnable transformation. The channel gated vector is used to perform channel-weighted modulation on the fine-grained disease features output by the main adaptation branch to obtain the main adaptation features. The main adaptation features are fused with the original features by using a residual connection method. The fused result is then subjected to layer normalization to finally obtain the adapted image features.

6. The method for identifying small-sample plant diseases based on the CLIP model with lightweight adaptation according to claim 5, characterized in that, In step S4, the hybrid loss function is composed of a weighted combination of contrast loss and center loss. The contrast loss is used to constrain the alignment relationship between image features and corresponding text features in the feature space. The center loss is used to constrain the features of samples of the same disease category to cluster towards the center of their corresponding category features, thereby enhancing the compactness of intra-class features. The parameters of the lightweight adapter module are optimized by jointly minimizing the contrast loss and the center loss.

7. The method for identifying small-sample plant diseases based on the CLIP model with lightweight adaptation according to claim 1, characterized in that, In step S5, the small-sample plant disease identification model determines the disease category of the plant image to be tested by calculating the similarity between the adapted image features and the text features of each disease category, and outputs the corresponding identification result.

8. A lightweight, adaptable small-sample plant disease identification system based on the CLIP model, characterized in that, include: Image and text data construction module: used to collect plant disease and pest images and label disease categories, construct plant disease and pest image dataset, divide the plant disease and pest image dataset into training set, validation set and test set, and generate multi-source text descriptions that correspond one-to-one with each plant disease and pest image in the plant disease and pest image dataset, forming text description data that matches the plant disease and pest image dataset. The model building and feature adaptation module includes a lightweight adapter module. The model building and feature adaptation module is used to build a small sample plant disease identification model based on the CLIP model, freeze all parameters of the image encoder and text encoder in the CLIP model, and embed the lightweight adapter module at the output of the image encoder in the CLIP model to dynamically adjust the fixed-dimensional original features output by the image encoder and output adapted disease image features. Model training module: used to construct a hybrid loss function, receiving plant disease and pest images and their accompanying multi-source text descriptions from the training set output by the image and text data construction module as joint input, and extracting the corresponding image features and text features respectively through the small sample plant disease identification model; The small sample plant disease identification model is iteratively trained based on the hybrid loss function, and the parameters of the image encoder and text encoder of the CLIP model are kept constant during the training process. The disease identification performance of the model is verified through the validation set, and the optimal model parameters are determined and stored to obtain the optimal small sample plant disease identification model. Recognition and reasoning module: It is used to receive the plant image to be tested, call the small sample plant disease recognition model corresponding to the optimal parameters stored in the model training module, perform feature extraction and matching discrimination on the plant image to be tested, and output the plant disease category recognition result.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to call and run a computer program stored in the memory to perform the steps of the small sample plant disease identification method based on the CLIP model lightweight adaptation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implementing the small-sample plant disease identification method based on the CLIP model lightweight adaptation as described in any one of claims 1 to 7. When the computer program is called and executed by a computer, it performs the steps included in the corresponding method.

Images