Fine-grained crop disease image classification method based on image text fusion

Abstract

The invention discloses a fine-grained crop disease image classification method based on image text fusion, and belongs to the technical field of image data processing, and the method comprises the following steps: 1, carrying out the image data collection in an agricultural website through a web crawler method, carrying out the image data selection through an agricultural field expert, and carrying out the image data collection; taking the selected image as a crop disease image; 2, extracting an image high-dimensional feature map by using a pre-trained image feature extractor; 3, weakening interference information in the image by using a scab extraction algorithm, and highlighting scab features; step 4, a pre-trained BERT model is utilized to embed text description for each image, visual and text information is fused into a unified feature space, and the weight of each piece of modal information is dynamically adjusted; and step 5, performing accurate classification of crop diseases by using the fused features. According to the method, various disease types can be effectively distinguished under various conditions.

Description

Technical Field

[0001] The invention belongs to the technical field of image data processing, and in particular relates to a fine-grained crop disease image classification method based on image-text fusion. Background Art

[0002] The invasion of crop diseases often brings serious losses to agricultural production. Quickly and accurately identifying the types of crop diseases and taking effective measures before the disease spreads can reduce many unnecessary losses. However, crop disease images taken in real scenes often have complex interference information. In order to solve this problem, a solution is to use a model that can accurately extract crop lesions to classify crop disease images, that is, to train a convolutional neural network that can weaken interference information and enhance lesion and crop information to complete the crop disease image classification task.

[0003] Different types of diseases on some crops look similar and difficult to distinguish. Relying solely on the feature extraction capabilities of deep visual neural networks has performance bottlenecks. In the process of extracting features from images, the present invention adds text clues describing the image to supplement the visual features, thereby obtaining a more representative feature representation. This fusion method is expected to improve the accuracy and robustness of the crop disease recognition system, enabling it to effectively distinguish various disease types under various circumstances. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a fine-grained crop disease image classification method based on image-text fusion. First, a pre-trained image feature extractor is used to obtain the high-dimensional features of crop diseases, and then a lesion extraction algorithm is applied to weaken the interference information in the image and highlight the lesion features. Next, a text description is embedded for each image through a pre-trained BERT model. In the multimodal feature fusion stage, the visual and textual information are fused into a unified feature space, and the weight of each modal information is dynamically adjusted to optimize the classification performance. Finally, the fused features are used to accurately classify crop diseases.

[0005] The technical solution of the present invention is as follows:

[0006] A fine-grained crop disease image classification method based on image-text fusion includes the following steps:

[0007] Step 1: Use a web crawler to collect image data on agricultural websites, and have agricultural experts select the image data, and use the selected images as crop disease images;

[0008] Step 2: Use a pre-trained image feature extractor to extract a high-dimensional feature map of the image;

[0009] Step 3: Use the lesion extraction algorithm to weaken the interference information in the image and highlight the lesion features;

[0010] Step 4: Use the pre-trained BERT model to embed text descriptions for each image, fuse visual and text information into a unified feature space, and dynamically adjust the weight of each modality information;

[0011] Step 5: Use the fused features to accurately classify crop diseases.

[0012] Furthermore, the specific process of step 2 is as follows:

[0013] Step 2.1: Pre-train the Resnet50 visual feature extraction model on the ImageNet dataset to obtain the image encoder f c ; Image encoder f c Fine-tuning optimization is performed using the cross entropy loss function;

[0014] Step 2.2: Input the crop disease image x into the image encoder f c , after the last convolutional layer, the high-dimensional visual feature map x is output c .

[0015] Furthermore, the specific process of step 3 is as follows:

[0016] Step 3.1: x c The dimension reduction operation is performed through a convolution block to obtain the reduced-dimensional visual feature map x′ c , expressed as:

[0017] x′ c =σ(BN(Conv 3*3 (x c )));

[0018] Among them, BN(·) represents the normalization layer; σ represents the relu activation function; Conv 3*3 (·) is a convolutional block of size 3*3;

[0019] Step 3.2: x′ c Extract the peak response coordinates from each feature map in ; cluster the coordinates through a clustering algorithm and finally cluster them into five clusters;

[0020] Step 3.3: Add the feature maps belonging to the same cluster element by element to obtain five feature maps, which are represented by v c ; Use global average pooling operation on each feature map to obtain the global representation v of each feature map w ;

[0021] Step 3.4: Set the global representation v wThe input fully connected layer learns the weight representation w1 of each cluster and converts it into a probability distribution between 0 and 1 through the Softmax operation; the fully connected layer is represented as:

[0022] w1=W T v w +b;

[0023] Where W and b represent the weight and bias of the fully connected layer respectively; T is the transposed symbol;

[0024] Step 3.5: Replace w1 with v c Multiply to obtain the weighted feature map v c ′; The weights of the weighted feature graphs are adaptively changed during the learning process according to the clustering results;

[0025] Step 3.6: v c ′ and high-dimensional visual feature map x c Multiply element by element to obtain the processed visual feature map v e .

[0026] Furthermore, the specific process of step 4 is as follows:

[0027] Step 4.1: Represent the image-text pair data as {(i1,t1),…,(i n ,t n )}, where n represents the number of image-text pairs; i n is the nth image; t n is the text corresponding to the nth image;

[0028] Step 4.2: Given a pre-trained text encoder represented as T c , and fine-tuned and optimized by the cross entropy loss function; the text t is passed through T c Get the text encoding vector t c ;

[0029] Step 4.3: Process the visual feature map v e After the deformation operation, it is v e ′, and transform the image feature dimension to 768 dimensions, the same as the text, through the fully connected layer; it can be expressed as:

[0030] v′ e =FC(flatten(v e ));

[0031] Among them, flatten(·) means flattening the input vector by reshaping the first two dimensions into one dimension; FC(·) stands for the fully connected layer, which is used for dimensionality reduction;

[0032] Step 4.4: Combine the two modal features v e′ and t c To form a multimodal feature sequence j; then use an attention layer to measure the contribution of each modality to the classification performance; finally, use a fully connected layer to learn a new multimodal embedding space; expressed as:

[0033] j=FC(w2×concat(v′ e ,t c ));

[0034] Among them, w2 represents the attention layer weight; concat(·) represents the concatenation operation;

[0035] Step 4.5. Finally, use the summation operation on dimension 1 to obtain the multimodal feature representation j′.

[0036] Furthermore, the specific process of step 5 is as follows: after obtaining the multimodal feature representation vector of each image-text pair, the data set is represented as D = {(j1′,l1),…,(j n ′,l n )};j n ′ is the image-text fusion feature vector corresponding to the nth image; l n is the category label corresponding to the nth image; the cross entropy loss function is used to calculate the classification loss Loss ce ; expressed as:

[0037]

[0038] Among them, j i ′ is the image-text fusion feature vector corresponding to the i-th image; l i is the category label corresponding to the i-th image; E θ Represents the parameters before model optimization; E′ θ represents the parameters after model optimization; v is the serial number of the crop disease type; m represents the crop disease type in the data set; p i Represents a model that belongs to category l i The probability of ; y′ is a binary indicator.

[0039] Beneficial technical effects of the present invention: The present invention can effectively reduce the interference information in the image and highlight the characteristics of the diseased spots. The present invention designs a new image-text fusion space and dynamically adjusts the weight of each modal information to optimize the classification performance. Finally, the fused features are used to achieve accurate classification of crop diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the fine-grained crop disease image classification method based on image-text fusion of the present invention;

[0041] Figure 2 The present invention is a flow chart of the crop disease spot extraction algorithm. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] The present invention implements a fine-grained crop disease image classification method based on image-text fusion. It can effectively reduce the interference information in the image and highlight the characteristics of the disease spots. A new image-text fusion space is designed, and the weight of each modal information is dynamically adjusted to optimize the classification performance. Finally, the fused features are used to achieve accurate classification of crop diseases.

[0044] like Figure 1 and Figure 2 As shown, the method of the embodiment of the present invention mainly includes the following steps:

[0045] Step 1: Use a web crawler to collect image data on agricultural websites, and have agricultural experts select the image data, and use the selected images as crop disease images;

[0046] Step 2: Use the pre-trained image feature extractor to extract the high-dimensional feature map of the image; the specific process is as follows:

[0047] Step 2.1: Pre-train the Resnet50 visual feature extraction model on the ImageNet dataset to obtain the image encoder f c . Image encoder f c Fine-tuning optimization is performed using the cross entropy loss function.

[0048] Step 2.2: Input the crop disease image x into the image encoder f c , after the last convolutional layer, the high-dimensional visual feature map x is output c .

[0049] Step 3: Use the lesion extraction algorithm to weaken the interference information in the image and highlight the lesion features; the specific process is as follows:

[0050] Step 3.1, firstly, c The dimension reduction operation is performed through a convolution block to obtain the reduced-dimensional visual feature map x′ c . It can be expressed as:

[0051] x′ c =σ(BN(Conv 3*3 (x c )));

[0052] The convolution block Conv has 2048 input channels, 64 output channels, a size of 3*3, a stride of 1, and a padding of 1. BN(·) represents the normalization layer. σ represents the relu activation function. Conv 3*3 (·) is a convolutional block of size 3*3.

[0053] Step 3.2: x′ c The peak response coordinates are extracted from each feature map in . The coordinates are clustered by a clustering algorithm and finally clustered into five clusters.

[0054] Step 3.3: Add the feature maps belonging to the same cluster element by element to obtain five feature maps, which are represented by v c . Use the GAP global average pooling operation on each feature map to obtain the global representation v of each feature map w .

[0055] Step 3.4: Set the global representation v w The input fully connected layer learns the weight representation w1 of each cluster. And through the Softmax operation, it is converted into a probability distribution between 0 and 1. The fully connected layer is represented as:

[0056] w1=W T v w +b;

[0057] Where W and b represent the weight and bias of the fully connected layer respectively; T is the transposed symbol.

[0058] Step 3.5: Replace w1 with v c Multiply to obtain the weighted feature map v c The weights of the weighted feature graph can be adaptively changed during the learning process according to the clustering results to improve the accuracy of the classification results.

[0059] Step 3.6, finally v c ′ and high-dimensional visual feature map x c Multiply element by element to obtain the processed visual feature map v e .

[0060] The above process weakens the interference information and enhances the information of lesions and crops.

[0061] Step 4: Use the pre-trained BERT model to embed text descriptions for each image, fuse visual and text information into a unified feature space, and dynamically adjust the weight of each modality information to optimize classification performance; the specific process is as follows:

[0062] Step 4.1: The image-text pair data used in the experiment is represented as {(i1, t1),…, (i n ,t n)}, where n represents the number of image-text pairs. n is the nth image; t n is the text corresponding to the nth image.

[0063] Step 4.2: Given a pre-trained text encoder represented as T c , and fine-tuned and optimized by the cross entropy loss function. c Get the text encoding vector t c .

[0064] Step 4.3: Process the visual feature map v e After the deformation operation, it is v e ′, and transform the image feature dimension to 768 dimensions, the same as the text, through the fully connected layer. It can be expressed as:

[0065] v′ e =FC(flatten(v e ));

[0066] Here, flatten(·) means flattening the input vector by reshaping the first two dimensions into one dimension. FC(·) stands for the fully connected layer, which is used for dimensionality reduction.

[0067] Step 4.4: Combine the two modal features v e ′ and t c To form a multimodal feature sequence j. Then use an attention layer to measure the contribution of each modality to the classification performance. Finally, use the fully connected layer to learn a new multimodal embedding space. It can be expressed as:

[0068] j=FC(w2×concat(v′ e ,t c ));

[0069] Wherein, w2 represents the weight of the attention layer. concat(·) represents a concatenation operation, and the present invention concatenates on dimension 1.

[0070] Step 4.5, but obviously the joint sequence feature cannot be used for classification tasks. So finally use the summation operation on dimension 1 to obtain the multimodal feature representation j′.

[0071] Step 5: Use the fused features to accurately classify crop diseases. The specific process is as follows:

[0072] After obtaining the multimodal feature representation vector of each image-text pair, the dataset can be represented as D = {(j1′,l1),…,(j n ′,l n )}. n′ is the image-text fusion feature vector corresponding to the nth image; l n is the category label corresponding to the nth image. The cross entropy loss function is used to calculate the classification loss Loss ce . It can be expressed as:

[0073]

[0074] Among them, j′ i is the image-text fusion feature vector corresponding to the i-th image; l i is the category label corresponding to the i-th image; E θ Represents the parameters before model optimization. E′ θ represents the parameters after model optimization. v is the serial number of the crop disease type; m represents the crop disease type in the data set. p i Represents a model that belongs to category l i The probability of y′ is a binary indicator (0 or 1), and the prediction vector p of the sample is calculated i belongs to category l i It takes 1 when there is a loss, otherwise it takes 0.

[0075] In order to prove the feasibility and superiority of the model trained by the method of the present invention, a comparative experiment was conducted with the existing VGG16 image classification model, Resnet50 image classification model, Xception image classification model, CondenseNet image classification model, InceptionV4 image classification model, EfficientNet image classification model, MobileNetV3 image classification model, Res2Net image classification model, Vit image classification model, CrisisViT image classification model and SFRMet image classification model. The above models are all the best models in the field of deep learning image classification in previous years. The experimental results are shown in Table 1:

[0076] Table 1 Performance comparison results of each model

[0077]

[0078]

[0079] Accuracy measures the ratio of the number of samples correctly classified by the classifier to the total number of samples. Precision measures the proportion of samples predicted by the classifier to be positive that are actually positive. It measures the accuracy of the classifier's judgment of positive samples. Recall measures the proportion of all samples that are actually positive that are correctly predicted to be positive. It focuses on how many of all actual positive samples are correctly identified as positive. F1-Score combines Precision and Recall, and is their harmonic mean, which is used to comprehensively evaluate the performance of the model. The larger the values ​​of Accuracy, Precision, Recall, and F1-Score, the stronger the performance of the model in the task of fine-grained crop disease image classification. It can be seen from the experimental results that the method of the present invention achieved the best prediction results, and the values ​​of Accuracy, Precision, Recall, and F1-Score were all maximum values.

[0080] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A fine-grained crop disease image classification method based on image-text fusion, characterized in that: The steps include: Step 1: Use a web crawler to collect image data on agricultural websites, and have agricultural experts select the image data, and use the selected images as crop disease images; Step 2: Use a pre-trained image feature extractor to extract a high-dimensional feature map of the image; Step 3: Use the lesion extraction algorithm to weaken the interference information in the image and highlight the lesion features; Step 4: Use the pre-trained BERT model to embed text descriptions for each image, fuse visual and text information into a unified feature space, and dynamically adjust the weight of each modality information; Step 5: Use the fused features to accurately classify crop diseases.

2. According to claim 1, the fine-grained crop disease image classification method based on image-text fusion is characterized in that: The specific process of step 2 is as follows: Step 2.1: Pre-train the Resnet50 visual feature extraction model on the ImageNet dataset to obtain the image encoder f c ; Image encoder f c Fine-tuning optimization is performed using the cross entropy loss function; Step 2.2: Input the crop disease image x into the image encoder f c , after the last convolutional layer, the high-dimensional visual feature map x is output c .

3. The fine-grained crop disease image classification method based on image-text fusion according to claim 2 is characterized in that: The specific process of step 3 is as follows: Step 3.1: x c The dimension reduction operation is performed through a convolution block to obtain the reduced-dimensional visual feature map x′ c , expressed as: x′ c =σ(BN(Conv 3*3 (x c ))); Among them, BN(·) represents the normalization layer; σ represents the relu activation function; Conv 3*3 (·) is a convolutional block of size 3*3; Step 3.2: x′ c Extract the peak response coordinates from each feature map in ; cluster the coordinates through a clustering algorithm and finally cluster them into five clusters; Step 3.3: Add the feature maps belonging to the same cluster element by element to obtain five feature maps, which are represented by v c ; Use global average pooling operation on each feature map to obtain the global representation v of each feature map w ; Step 3.4: Set the global representation v w The input fully connected layer learns the weight representation w1 of each cluster and converts it into a probability distribution between 0 and 1 through the Softmax operation; the fully connected layer is represented as: w1=W T v w +b; Where W and b represent the weight and bias of the fully connected layer respectively; T is the transposed symbol; Step 3.5: Replace w1 with v c Multiply to obtain the weighted feature map v′ c ;The weights of the weighted feature graphs are adaptively changed during the learning process according to the clustering results; Step 3.6: v′ c With the high-dimensional visual feature map x c Multiply element by element to obtain the processed visual feature map v e .

4. The fine-grained crop disease image classification method based on image-text fusion according to claim 3 is characterized in that: The specific process of step 4 is as follows: Step 4.1: Represent the image-text pair data as {(i1,t1),…,(i n ,t n )}, where n represents the number of image-text pairs; i n is the nth image; t n is the text corresponding to the nth image; Step 4.2: Given a pre-trained text encoder represented as T c , and fine-tuned and optimized by the cross entropy loss function; the text t is passed through T c Get the text encoding vector t c ; Step 4.3: Process the visual feature map v e After the deformation operation, it is v′ e , and transform the image feature dimension to 768 dimensions, the same as the text, through the fully connected layer; it can be expressed as: in e =FC(flatten(v e )); Among them, flatten(·) means flattening the input vector by reshaping the first two dimensions into one dimension; FC(·) stands for the fully connected layer, which is used for dimensionality reduction; Step 4.4: Combine the two modal features v′ by concatenation e and t c To form a multimodal feature sequence j; then use an attention layer to measure the contribution of each modality to the classification performance; finally, learn a new multimodal embedding space through a fully connected layer; expressed as: j=FC(w2×concat(v′ e ,t c )); Among them, w2 represents the attention layer weight; concat(·) represents the concatenation operation; Step 4.

5. Finally, use the summation operation on dimension 1 to obtain the multimodal feature representation j′.

5. The fine-grained crop disease image classification method based on image-text fusion according to claim 4 is characterized in that: The specific process of step 5 is as follows: after obtaining the multimodal feature representation vector of each image-text pair, the data set is represented as D = {(j′1,l1),…,(j′ n ,l n )};j′ n is the image-text fusion feature vector corresponding to the nth image; l n is the category label corresponding to the nth image; the cross entropy loss function is used to calculate the classification loss Loss ce ; expressed as: Among them, j′ i is the image-text fusion feature vector corresponding to the i-th image; l i is the category label corresponding to the i-th image; E θ Represents the parameters before model optimization; E′ θ represents the parameters after model optimization; v is the serial number of crop disease types; m represents the types of crop diseases in the data set; p i Represents a model that belongs to category l i The probability of ; y′ is a binary indicator.

Images