Crop disease identification method based on prompt learning

Abstract

The invention relates to a crop disease identification method based on prompt learning. The method comprises the steps of obtaining a crop disease image data set; constructing a prompt learning model; inputting the training set into a prompt learning model for training; carrying out dynamic calibration by adopting a dynamic calibration strategy; the calibrated prompt vector is updated through a momentum buffering mechanism, the adjusted prompt vector is replaced, and an updated prompt learning model is obtained; and inputting a to-be-identified crop disease image into the updated prompt learning model to obtain an identification result. According to the method, prompt learning is introduced into a crop disease identification task for the first time, feature information of different levels is effectively fused, and the base class identification precision and the new class generalization performance of the model are remarkably improved; the domain generalization ability of the model is improved through the dynamic calibration strategy, extra training data is not needed, cross-modal knowledge in the CLIP model is used for directly adjusting and optimizing the result during testing, and the problems of feature offset and performance degradation existing in a traditional entropy minimization method are effectively solved.

Description

Technical Field

[0001] The invention relates to the technical field of prompt learning and crop disease recognition, in particular to a crop disease recognition method based on prompt learning. Background Art

[0002] Timely and accurate identification and prevention of agricultural diseases are of great significance to ensuring food security and sustainable agricultural development. In the traditional practice of crop disease identification, agricultural experts mainly rely on visual inspection to diagnose diseases, which is not only time-consuming and labor-intensive, but also inefficient when facing large-scale farmland, and it is difficult to meet the needs of modern agricultural production. With the development of computer vision technology, crop disease identification methods based on deep learning have gradually become a research hotspot. However, traditional deep learning models often show poor generalization performance and are difficult to maintain stable recognition effects when facing new disease types or different environmental conditions. This lack of generalization ability has seriously affected the widespread application of deep learning methods in agricultural production practices.

[0003] In recent years, the rise of visual language pre-training models has provided new ideas for solving the above problems. Among them, CLIP (Contrastive Language Image Pre-training) has demonstrated strong zero-sample migration capabilities by performing contrastive learning pre-training on large-scale image-text pair data. Cued learning can improve the performance of the model on specific tasks while keeping the parameters of the pre-trained model unchanged, providing new ideas for optimizing the performance of CLIP in downstream tasks. However, there are currently relatively few methods for applying cued learning to agriculture.

[0004] The practicality of the model is often more demanding in the agricultural field, and the generalization ability of the model is crucial to the actual deployment of the model. Traditional practices mainly focus on building training data sets that cover a larger distribution, and enhance the generalization ability of the model by enabling the model to learn a large number of features during the training phase. However, there are many types of crop diseases, with various manifestations, and different crops may have the same disease. Therefore, for agricultural disease images, building a large and reliable data set is a difficult task. The core idea of ​​test-time adaptation is to directly use the data in the test phase to adjust the model and adaptively optimize the test results, which provides a new idea for solving the domain generalization problem in the agricultural field. Therefore, a crop recognition method based on prompt learning is needed, and a method that can directly tune the test results during testing is needed. Summary of the invention

[0005] In order to solve the problems of data scarcity and class imbalance unique to agricultural scenarios, the purpose of the present invention is to provide a crop disease recognition method based on prompt learning that introduces prompt learning into the task of crop disease recognition, which can better utilize the rich knowledge contained in the pre-trained model, mix features according to the hierarchical depth of the visual backbone network, effectively fuse feature information at different levels, and significantly improve the model's recognition accuracy for base classes and the generalization performance of new classes.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for identifying crop diseases based on prompt learning, the method comprising the following steps in order:

[0007] (1) Acquire crop disease image data and form a crop disease image dataset, and divide the crop disease image dataset into a training set, a validation set, and a test set;

[0008] (2) constructing a prompt learning model: adding a progressive feature mixing module and a learnable context vector to the CLIP model to obtain a prompt learning model, and obtaining a prompt vector through the prompt learning model; the CLIP model includes an image encoder for extracting feature information of the disease image, and a text encoder for processing a text description of the disease category;

[0009] (3) inputting the training set into the prompt learning model for training: the progressive feature mixing module mixes the multi-level feature information of the extracted disease to obtain mixed features, and uses the mixed features to adjust the prompt vector to obtain an adjusted prompt vector;

[0010] (4) Dynamic calibration strategy is used for dynamic calibration: the test set and the adjusted prompt vector are input into the pre-trained CLIP model to obtain the calibrated prompt vector and test results;

[0011] (5) The calibrated prompt vector is updated through a momentum buffer mechanism to obtain an updated prompt vector, and the updated prompt vector is used to replace the prompt vector adjusted in step (3) to obtain an updated prompt learning model;

[0012] (6) The crop disease image to be identified is input into the updated prompt learning model to obtain the identification result.

[0013] The step (1) specifically refers to: the crop disease image dataset includes corn leaf disease dataset, rice leaf disease dataset, sugarcane leaf disease dataset, citrus leaf disease dataset, apple leaf disease dataset, PlantVillage, PlantDoc and LWDCD2020.

[0014] The step (2) specifically refers to: adding a progressive feature mixing module to the backbone network of the image encoder, the backbone network of the image encoder adopts ViT-B / 16, the learnable context vector is located in the prompt text processed by the text encoder, and the format of the prompt text is: [V]1[V]2...[V] M [CLASS], where [V]1[V]2...[V] M is the learnable context vector, [CLASS] is the disease name of the i-th category, and M represents the length of the learnable context vector.

[0015] In step (3), the progressive feature mixing module mixes the multi-level feature information of the extracted diseases, specifically referring to: the progressive feature mixing module selects shallow features, middle features and deep features of the image encoder backbone network for feature mixing, wherein the shallow features select the output of the 4th convolutional layer of the image encoder backbone network, which is used to extract the basic visual features of crop diseases; the middle features select the output of the 8th convolutional layer of the image encoder backbone network, which is used to extract the middle-level structural features of the diseases; the deep features select the output of the 11th convolutional layer of the image encoder backbone network, which is used to extract the high-level semantic features of the diseases; the feature mixing of each layer follows the following steps:

[0016] For each layer, the mixing coefficient λ l Sampling from a Beta distribution:

[0017] λ l ~Beta(α l ,α l )

[0018] Among them, λ l Represents the parameters of the Beta distribution of the lth layer, Beta represents the Beta distribution; α l is a constant, α of shallow features, middle features, and deep features l The one-to-one correspondence is constant 0.4, 0.2, 0.1, and the larger λ l The value makes the shallow feature mixing more balanced, and the smaller λ l The value makes the deep feature mixing more focused; at each layer, the mixed features passed from the previous layer are mixed with the newly selected sample features of the current layer:

[0019]

[0020] Among them, f l is the mixed feature passed from the previous layer, f l ′ is the newly selected sample feature of the current layer; is the mixed feature; the mixed feature It will continue to spread deeper;

[0021] For the mixed features The label is inherited from the label y′ of the current layer sample. The mixed feature The loss function calculation formula is:

[0022]

[0023] Among them, L CE represents the cross entropy loss function.

[0024] In step (3), the hint vector is adjusted using the mixed features. The adjusted hint vector specifically refers to: first, the cosine similarity is calculated using the hint vector and the image features obtained by the image encoder, that is, the probability that the image belongs to each category is calculated:

[0025]

[0026] Where p(y=i|g(v)) represents the probability that the image belongs to category i, sim(·,·) represents the cosine similarity, h(t) and g(v) represent the text features extracted by the text encoder and the image features extracted by the image encoder of the prompt learning model, respectively, τ is the temperature parameter used to adjust the smoothness of the probability distribution, and K is the total number of disease categories;

[0027] Then, the loss function is calculated by the predicted probability of belonging to each category and the category to which the image actually belongs, the gradient is calculated using the loss function, and the gradient is back-propagated to adjust the prompt vector; the calculation formula of the loss function L is:

[0028]

[0029] The prompt vector is then adjusted by back-propagating the gradient through the text encoder to improve the recognition accuracy of the prompt learning model.

[0030] The step (4) specifically refers to: for a test image v, first use n different data enhancers to process it to obtain multiple images of different viewing angles, only retain samples with low test entropy H(P(t|v)), select the top 10% of the lowest entropy values ​​as calibration target samples, for each selected low entropy view, take the top K predicted category names to calculate the calibration score, and finally optimize the learnable parameters by gradient descent to maximize the expected calibration; P(t|v) is the test result, where P is the probability, t is the prompt text, and P(t|v) is the probability that the image belongs to various categories;

[0031] Cosine similarity is defined based on the CLIP model:

[0032] CLIP(t,v)=sim(h(t) clip,g(v) clip )

[0033] Among them, h(t) clip and g(v) clip They represent the text features extracted by the text encoder and the image features extracted by the image encoder of the CLIP model respectively;

[0034] Calculate CLIP score:

[0035] CLIP S (t,v)=w×max(CLIP(t,v),0)

[0036] Among them, w = 2.5 is a constant coefficient;

[0037] Define the calibration function:

[0038]

[0039] in, is the average CLIP score of the sampled image-text pairs;

[0040] To calculate the gradient of the non-differentiable dynamic calibration function, the REINFORCE algorithm is used to calculate the gradient of the non-differentiable dynamic calibration function:

[0041]

[0042] Among them, θ is the prompt vector to be tuned; is the gradient of the expected value of the calibration function R(t,v) with respect to θ, R(t,v) is the calibration function, and P(t|v;θ) is the probability distribution;

[0043] After each iteration, θ is updated to The corresponding momentum buffer ψ is updated according to the momentum buffer mechanism:

[0044]

[0045] Among them, γ∈[0,1) is the momentum factor.

[0046] It can be seen from the above technical scheme that the beneficial effects of the present invention are: first, the present invention introduces prompt learning into the crop disease recognition task for the first time. Compared with the traditional deep learning method, prompt learning can better utilize the rich knowledge contained in the pre-trained model, and provides a new idea for solving the problems of data scarcity and category imbalance in agricultural scenarios; second, the prompt learning model in the present invention mixes features according to the hierarchical depth of the visual backbone network, effectively integrates feature information at different levels, and significantly improves the model's recognition accuracy for the base class and the generalization performance of the new class; third, a dynamic calibration strategy based on CLIP knowledge, namely a dynamic calibration strategy, is designed to improve the domain generalization ability of the model. No additional training data is required, and the cross-modal knowledge in the CLIP model is used to directly tune the results during testing, which effectively solves the problem of feature offset and performance degradation in the traditional entropy minimization method. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 A workflow diagram of the prompt learning model in the present invention;

[0049] Figure 3 This is a flow chart of the method of the dynamic calibration strategy in the present invention. DETAILED DESCRIPTION

[0050] like Figure 1 As shown, a method for identifying crop diseases based on prompt learning comprises the following steps in order:

[0051] (1) Acquire crop disease image data and form a crop disease image dataset, and divide the crop disease image dataset into a training set, a validation set, and a test set;

[0052] (2) constructing a prompt learning model: adding a progressive feature mixing module and a learnable context vector to the CLIP model to obtain a prompt learning model, and obtaining a prompt vector through the prompt learning model; the CLIP model includes an image encoder for extracting feature information of the disease image, and a text encoder for processing a text description of the disease category;

[0053] (3) inputting the training set into the prompt learning model for training: the progressive feature mixing module mixes the multi-level feature information of the extracted disease to obtain mixed features, and uses the mixed features to adjust the prompt vector to obtain an adjusted prompt vector;

[0054] (4) Dynamic calibration strategy is used for dynamic calibration: the test set and the adjusted prompt vector are input into the pre-trained CLIP model to obtain the calibrated prompt vector and test results;

[0055] (5) The calibrated prompt vector is updated through a momentum buffer mechanism to obtain an updated prompt vector, and the updated prompt vector is used to replace the prompt vector adjusted in step (3) to obtain an updated prompt learning model;

[0056] (6) Input the crop disease image to be identified into the updated prompt learning model to obtain the identification result. Output the disease category and the corresponding confidence score, and support the simultaneous identification of multiple diseases in the same image; this method has strong generalization and can adapt to the identification needs of different crops, different disease types, and different environmental conditions, providing technical support for disease prevention and control in agricultural production.

[0057] The step (1) specifically refers to: the crop disease image dataset includes a corn leaf disease dataset, a rice leaf disease dataset, a sugarcane leaf disease dataset, a citrus leaf disease dataset, an apple leaf disease dataset, PlantVillage, PlantDoc and LWDCD2020. Referring to the data of China Statistical Yearbook 2023, eight datasets, including the top six crops in terms of yield and PlantVillage and PlantDoc, are selected as crop disease image data. Among them, PlantVillage, PlantDoc and LWDCD2020 are open source datasets, and the datasets of the other five crops are all from the Kaggle platform.

[0058] like Figure 2 As shown, the step (2) specifically refers to: adding the progressive feature mixing module to the backbone network of the image encoder, the backbone network of the image encoder adopts ViT-B / 16, the learnable context vector is located in the prompt text processed by the text encoder, and the format of the prompt text is: [V]1[V]2...[V] M [CLASS], where [V]1[V]2...[V] M is the learnable context vector, [CLASS] is the disease name of the i-th category, and M represents the length of the learnable context vector.

[0059] In step (3), the progressive feature mixing module mixes the multi-level feature information of the extracted diseases, specifically referring to: the progressive feature mixing module selects shallow features, middle features and deep features of the image encoder backbone network for feature mixing, wherein the shallow features select the output of the 4th convolutional layer of the image encoder backbone network, which is used to extract the basic visual features of crop diseases; the middle features select the output of the 8th convolutional layer of the image encoder backbone network, which is used to extract the middle-level structural features of the diseases; the deep features select the output of the 11th convolutional layer of the image encoder backbone network, which is used to extract the high-level semantic features of the diseases; the feature mixing of each layer follows the following steps:

[0060] For each layer, the mixing coefficient λ l Sampling from a Beta distribution:

[0061] λ l ~Beta(α l ,α l )

[0062] Among them, λ l Represents the parameters of the Beta distribution of the lth layer, Beta represents the Beta distribution; α l is a constant, α of shallow features, middle features, and deep features l The one-to-one correspondence is constant 0.4, 0.2, 0.1, and the larger λ l The value makes the shallow feature mixing more balanced, and the smaller λ l The value makes the deep feature mixing more focused; at each layer, the mixed features passed from the previous layer are mixed with the newly selected sample features of the current layer:

[0063]

[0064] Among them, f l is the mixed feature passed from the previous layer, f l ′ is the newly selected sample feature of the current layer; is the mixed feature; the mixed feature It will continue to spread deeper;

[0065] For the mixed features The label is inherited from the label y′ of the current layer sample. The mixed feature The loss function calculation formula is:

[0066]

[0067] Among them, L CE represents the cross entropy loss function.

[0068] In step (3), the hint vector is adjusted using the mixed features. The adjusted hint vector specifically refers to: first, the cosine similarity is calculated using the hint vector and the image features obtained by the image encoder, that is, the probability that the image belongs to each category is calculated:

[0069]

[0070] Among them, p(y=i|g(v)) represents the probability that the image belongs to category i, sim(·,·) represents the cosine similarity, h(t) and g(v) represent the text features extracted by the text encoder and the image features extracted by the image encoder of the prompt learning model, respectively, τ is the temperature parameter used to adjust the smoothness of the probability distribution, and K is the total number of disease categories. The learnable prompt word optimization mechanism can adaptively adjust the model's feature expression for different disease categories and improve the model's recognition accuracy.

[0071] Then, the loss function is calculated by the predicted probability of belonging to each category and the category to which the image actually belongs, the gradient is calculated using the loss function, and the gradient is back-propagated to adjust the prompt vector; the calculation formula of the loss function L is:

[0072]

[0073] Then, the prompt vector is adjusted by back-propagating the gradient through the text encoder to improve the recognition accuracy of the prompt learning model. Through this progressive feature mixing method, the model can integrate disease feature information at different levels and improve the expression and recognition capabilities of complex crop disease features; this feature mixing strategy does not rely on external data enhancement, but only performs feature-level mixing within the model, which has good efficiency and stability.

[0074] like Figure 3 As shown, step (4) specifically refers to: for a test image v, first use n different data enhancers to process it, obtain multiple images of different viewing angles, only retain samples with low test entropy H(P(t|v)), select the top 10% of the lowest entropy values ​​as calibration target samples, for each selected low entropy view, take the top K predicted category names to calculate the calibration score, and finally optimize the learnable parameters by gradient descent to maximize the expected calibration; P(t|v) is the test result, where P is the probability, t is the prompt text, and P(t|v) is the probability that the image belongs to various categories;

[0075] Only samples with low test H(P(t|v)) are retained, because high entropy values ​​indicate that the model lacks confidence in the prediction results, and such predictions are considered unreliable. In practice, 64 enhanced views are generated, and the top 10% with the lowest entropy values ​​are selected as calibration target samples. For each selected low-entropy view, the calibration score is calculated by taking the category names of the top K predictions, and finally the learnable parameters are optimized by gradient descent to maximize the expected calibration.

[0076] Cosine similarity is defined based on the CLIP model:

[0077] CLIP(t,v)=sim(h(t) clip ,g(v) clip )

[0078] Among them, h(t) clip and g(v) clip They represent the text features extracted by the text encoder and the image features extracted by the image encoder of the CLIP model respectively;

[0079] Calculate CLIP score:

[0080] CLIP S (t,v)=w×max(CLIP(t,v),0)

[0081] Among them, w = 2.5 is a constant coefficient;

[0082] Define the calibration function:

[0083]

[0084] in, is the average CLIP score of the sampled image-text pairs;

[0085] To calculate the gradient of the non-differentiable dynamic calibration function, the REINFORCE algorithm is used to calculate the gradient of the non-differentiable dynamic calibration function:

[0086]

[0087] Among them, θ is the prompt vector to be tuned; is the gradient of the expected value of the calibration function R(t,v) with respect to θ, R(t,v) is the calibration function, and P(t|v;θ) is the probability distribution;

[0088] After each iteration, θ is updated to The corresponding momentum buffer ψ is updated according to the momentum buffer mechanism:

[0089]

[0090] Among them, γ∈[0,1) is the momentum factor. The dynamic calibration strategy optimizes the model prediction results in the test phase by utilizing the cross-modal knowledge of the CLIP model, which effectively improves the generalization performance of the model under different crop varieties and different environmental conditions.

[0091] The image encoder of the CLIP model uses ViT-B / 16. During the training phase, 16 images are used for each disease category. The optimizer uses the AdamW optimizer with a basic learning rate of 7e. -3 ; The dynamic calibration strategy uses CLIP-ViT-L / 14 as the calibration model; During the calibration process, the momentum factor γ = 0.99, and the benchmark parameters are updated every 64 samples.

[0092] In summary, the present invention introduces prompt learning into the crop disease recognition task for the first time. Compared with traditional deep learning methods, prompt learning can better utilize the rich knowledge contained in the pre-trained model, and provides a new idea for solving the problems of data scarcity and category imbalance in agricultural scenarios; the prompt learning model in the present invention mixes features according to the hierarchical depth of the visual backbone network, effectively integrates feature information at different levels, and significantly improves the model's recognition accuracy for base classes and generalization performance for new classes; a dynamic calibration strategy based on CLIP knowledge, namely a dynamic calibration strategy, is designed to improve the domain generalization ability of the model without the need for additional training data. The cross-modal knowledge in the CLIP model is used to directly tune the results during testing, which effectively solves the problem of feature offset and performance degradation in the traditional entropy minimization method.

Claims

1. A method for identifying crop diseases based on prompt learning, characterized in that: The method comprises the following steps in order: (1) Acquire crop disease image data and form a crop disease image dataset, and divide the crop disease image dataset into a training set, a validation set, and a test set; (2) constructing a prompt learning model: adding a progressive feature mixing module and a learnable context vector to the CLIP model to obtain a prompt learning model, and obtaining a prompt vector through the prompt learning model; the CLIP model includes an image encoder for extracting feature information of the disease image, and a text encoder for processing a text description of the disease category; (3) inputting the training set into the prompt learning model for training: the progressive feature mixing module mixes the multi-level feature information of the extracted disease to obtain mixed features, and uses the mixed features to adjust the prompt vector to obtain an adjusted prompt vector; (4) Dynamic calibration strategy is used for dynamic calibration: the test set and the adjusted prompt vector are input into the pre-trained CLIP model to obtain the calibrated prompt vector and test results; (5) The calibrated prompt vector is updated through a momentum buffer mechanism to obtain an updated prompt vector, and the updated prompt vector is used to replace the prompt vector adjusted in step (3) to obtain an updated prompt learning model; (6) The crop disease image to be identified is input into the updated prompt learning model to obtain the identification result.

2. The crop disease identification method based on prompt learning according to claim 1, characterized in that: The step (1) specifically refers to: the crop disease image dataset includes corn leaf disease dataset, rice leaf disease dataset, sugarcane leaf disease dataset, citrus leaf disease dataset, apple leaf disease dataset, PlantVillage, PlantDoc and LWDCD2020.

3. The crop disease identification method based on prompt learning according to claim 1, characterized in that: The step (2) specifically refers to: adding a progressive feature mixing module to the backbone network of the image encoder, the backbone network of the image encoder adopts ViT-B / 16, the learnable context vector is located in the prompt text processed by the text encoder, and the format of the prompt text is: [V]1[V]2...[V] M [CLASS], where [V]1[V]2...[V] M is the learnable context vector, [CLASS] is the disease name of the i-th category, and M represents the length of the learnable context vector.

4. The crop disease identification method based on prompt learning according to claim 1, characterized in that: In step (3), the progressive feature mixing module mixes the multi-level feature information of the extracted diseases, specifically referring to: the progressive feature mixing module selects shallow features, middle features and deep features of the image encoder backbone network for feature mixing, wherein the shallow features select the output of the 4th convolutional layer of the image encoder backbone network, which is used to extract the basic visual features of crop diseases; the middle features select the output of the 8th convolutional layer of the image encoder backbone network, which is used to extract the middle-level structural features of the diseases; the deep features select the output of the 11th convolutional layer of the image encoder backbone network, which is used to extract the high-level semantic features of the diseases; the feature mixing of each layer follows the following steps: For each layer, the mixing coefficient λ l Sampling from a Beta distribution: l l ~Beta(α l ,a l ) Among them, λ l Represents the parameters of the Beta distribution of the lth layer, Beta represents the Beta distribution; α l is a constant, α of shallow features, middle features, and deep features l The one-to-one correspondence is constant 0.4, 0.2, 0.1, and the larger λ l The value makes the shallow feature mixing more balanced, and the smaller λ l The value makes the deep feature mixing more focused; at each layer, the mixed features passed from the previous layer are mixed with the newly selected sample features of the current layer: Among them, f l is the mixed feature passed from the previous layer, f l ′ is the newly selected sample feature of the current layer; is the mixed feature; the mixed feature It will continue to spread deeper; For the mixed features The label is inherited from the label y′ of the current layer sample. The mixed feature The loss function calculation formula is: Among them, L CE represents the cross entropy loss function.

5. The crop disease identification method based on prompt learning according to claim 1, characterized in that: In step (3), the hint vector is adjusted using the mixed features. The adjusted hint vector specifically refers to: first, the cosine similarity is calculated using the hint vector and the image features obtained by the image encoder, that is, the probability that the image belongs to each category is calculated: Where p(y=i|g(v)) represents the probability that the image belongs to category i, sim(·,·) represents the cosine similarity, h(t) and g(v) represent the text features extracted by the text encoder and the image features extracted by the image encoder of the prompt learning model, respectively, τ is the temperature parameter used to adjust the smoothness of the probability distribution, and K is the total number of disease categories; Then, the loss function is calculated by the predicted probability of belonging to each category and the category to which the image actually belongs, the gradient is calculated using the loss function, and the gradient is back-propagated to adjust the prompt vector; the calculation formula of the loss function L is: The prompt vector is then adjusted by back-propagating the gradient through the text encoder to improve the recognition accuracy of the prompt learning model.

6. The crop disease identification method based on prompt learning according to claim 1, characterized in that: The step (4) specifically refers to: for a test image v, first use n different data enhancers to process it to obtain multiple images of different viewing angles, only retain samples with low test entropy H(P(t|v)), select the top 10% of the lowest entropy values ​​as calibration target samples, for each selected low entropy view, take the top K predicted category names to calculate the calibration score, and finally optimize the learnable parameters by gradient descent to maximize the expected calibration; P(t|v) is the test result, where P is the probability, t is the prompt text, and P(t|v) is the probability that the image belongs to various categories; Cosine similarity is defined based on the CLIP model: CLIP(t,v)=sim(h(t) clip ,g(v) clip ) Among them, h(t) clip and g(v) clip They represent the text features extracted by the text encoder and the image features extracted by the image encoder of the CLIP model respectively; Calculate CLIP score: CLIP S (t,v)=w×max(CLIP(t,v),0) Among them, w = 2.5 is a constant coefficient; Define the calibration function: in, is the average CLIP score of the sampled image-text pairs; To calculate the gradient of the non-differentiable dynamic calibration function, the REINFORCE algorithm is used to calculate the gradient of the non-differentiable dynamic calibration function: Among them, θ is the prompt vector to be tuned; is the gradient of the expected value of the calibration function R(t,v) with respect to θ, R(t,v) is the calibration function, and P(t|v;θ) is the probability distribution; After each iteration, θ is updated to The corresponding momentum buffer ψ is updated according to the momentum buffer mechanism: Among them, γ∈[0,1) is the momentum factor.

Images