Crop disease image classification method based on knowledge distillation network and metric learning

Abstract

The invention discloses a crop disease image classification method based on a knowledge distillation network and metric learning, and belongs to the technical field of image data processing. The knowledge distillation network comprises a teacher network and a student network; a crop disease image is converted into feature vectors through an image feature extractor, and the vectors are mapped to a measurement space through a mapper for similarity measurement. In a metric space, a clustering algorithm is adopted to generate hard pseudo labels for images, and the labels are main sources of image labels. Soft pseudo labels are generated through a teacher network classifier, and the soft pseudo labels assist model training. And a smoothing factor is introduced to filter knowledge learned by the student network, so that negative effects caused by noise tags are reduced. The method combines the advantages of deep metric learning and the knowledge distillation network, and can be suitable for different types of crop disease image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a crop disease image classification method based on knowledge distillation network and metric learning. Background Art

[0002] The invasion of crop diseases often causes serious losses to agricultural production. Rapidly and accurately identifying the types of crop diseases and taking effective measures before the disease spreads can reduce many unnecessary losses. However, training crop disease image classification models in a supervised learning manner requires a large number of datasets with labeled images. However, such datasets are currently scarce. In order to solve this problem, a knowledge distillation network model and unsupervised deep metric learning methods are used to classify a large number of unlabeled crop disease images, that is, training a convolutional neural network to assign pseudo labels to a large number of unlabeled images.

[0003] There is an inherent problem in the deep metric learning-based method, which is that the clustering algorithm uses the similarity between image features to assign pseudo labels to images, and the pseudo labels contain noisy labels. The presence of noisy labels will cause the model to fit in the wrong direction during training. This problem is called the pseudo-label noise diffusion problem. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a crop disease image classification method based on knowledge distillation network and metric learning. First, the image features are extracted by an image feature extractor, and then the image is mapped to a metric space by a mapper. In the metric space, a clustering algorithm is used to generate hard pseudo labels for the image. At the same time, a teacher network classifier is used to generate soft pseudo labels to supervise the class prediction vector of the student network. Finally, the knowledge learned by the student network is filtered by a smoothing factor and passed to the teacher network, thereby solving the problem of pseudo label noise diffusion.

[0005] The technical solution of the present invention is as follows:

[0006] A crop disease image classification method based on knowledge distillation network and metric learning, wherein the knowledge distillation network includes a teacher network and a student network; the method specifically includes the following steps:

[0007] Step 1: Use a web crawler to collect image data on agricultural websites, and have agricultural experts select the image data, and use the selected images as original images of crop diseases;

[0008] Step 2: Use the image feature extractor of the knowledge distillation network to extract the image feature vector;

[0009] Step 3: Use the mapper of the knowledge distillation network to embed the image features into the metric space of each network;

[0010] Step 4: Use a clustering algorithm to add hard pseudo labels to the image in the metric space of the teacher network, and use the hard pseudo labels and the embedding vectors of the metric space in the metric space of the student network for comparative learning;

[0011] Step 5: Use the teacher network classifier to generate soft pseudo labels, and use the soft pseudo labels to supervise the class prediction vector of the student network classifier to assist model training;

[0012] Step 6: Use the exponentially weighted moving average method to smoothly update the parameters of the teacher network;

[0013] Step 7: Acquire the crop image to be identified in real time, and obtain the classification result of the image based on the trained model.

[0014] Furthermore, the specific process of step 2 is as follows: given two image feature extractors, namely teacher network feature extractor f t and the student network feature extractor f s , and is optimized by two different losses; through the extractor f t and f s Encode the crop disease image into a feature vector; pass through the extractor f t (x) and f s The output of (x) is represented as the teacher network visual feature vector x t and the student network visual feature vector x s .

[0015] Furthermore, the specific process of step 3 is as follows: given two mappers, namely teacher network mapper v t and Student Network Mapper v s , the mapper is represented by a linear layer; x t and x s After the mapper, it is embedded into the metric space; x t and x s Represented as a 512-dimensional feature embedding vector x in the metric space t ′ and x s '; The process is expressed as:

[0016] X′ t =W T x t +b;

[0017] x′ s =W T x s +b;

[0018] Where W and b represent the weight and bias of the fully connected layer of the mapper, respectively; T is the transposed symbol.

[0019] Furthermore, the specific process of step 4 is as follows:

[0020] Step 4.1: Compare the feature embedding vector x of each image in the metric space using a hierarchical clustering algorithm t ′ and cluster them, and generate hard pseudo labels through clustering algorithm;

[0021] According to the hard pseudo-labels, we construct a dataset with labeled data D = {(x1, y1), (x2, y2), …, (x i ,y i ),…,(x I ,y I )}; where I represents the number of all original crop disease images in the dataset; x i is the i-th image; y i is the hard pseudo label corresponding to the i-th image;

[0022] Step 4.2: Use the difficult triplet loss function for contrastive learning. First, use the hard pseudo label of each image to perform sampling operations and construct difficult triplet for each image. The difficult triplet constructed after the i-th image sampling operation is in is the anchor sample of the i-th image, is the difficult positive sample of the i-th image, is the difficult negative sample of the i-th image; and The acquisition process is expressed by the formula:

[0023]

[0024]

[0025] Among them, x p is the anchor sample; x n is a difficult negative sample; the argmax(·) function finds the positive sample farthest from the anchor sample; d(·,·) represents the Euclidean distance;

[0026] Hard triplet loss function It is expressed by the formula:

[0027]

[0028] where σ is the distance hyperparameter in the triplet loss; [·] + Indicates that the calculation result is a positive value;

[0029] Step 4.3: In the process of calculating the Euclidean distance, the embedding vector generated by the student network mapper is used; the model directly updates the parameters of the student network when updating parameters.

[0030] Furthermore, the specific process of step 5 is as follows:

[0031] Given two classifiers, namely teacher network classifier g t and the student network classifier g s , the classifier is represented by a linear layer; x t The teacher network classifier generates a soft pseudo label p t ;x s The student network classifier generates the category prediction vector p s ; The process is expressed as:

[0032]

[0033]

[0034] in, and Respectively represent the weight and bias of the fully connected layer of the classifier;

[0035] Use soft pseudo label p t To supervise the category prediction vector p s , use the soft cross entropy loss function to calculate the loss; soft cross entropy loss function It is expressed by the formula:

[0036]

[0037] in, is the soft pseudo label of the i-th image; is the category prediction vector of the i-th image; Softmax(·) is the softmax function, which is expressed by the formula:

[0038]

[0039] Among them, q represents the category prediction vector p s is an item in ; j is the category index; J represents the number of all categories; exp(·) is an exponential function with e as the base.

[0040] Furthermore, in step 6, the parameters of the teacher network are updated using the exponentially weighted moving average method; at the current iteration time t, the parameters of the teacher network are expressed as E(θ t ), the parameters of the student network are E(θ t ′); The updating process is expressed by the formula:

[0041]

[0042] Among them, α is the smoothing factor.

[0043] Beneficial technical effects brought by the present invention:

[0044] Through unsupervised learning and knowledge distillation networks, a large amount of unlabeled crop disease image data is effectively utilized, avoiding the large amount of labeling costs required by traditional supervised methods, and improving data utilization and training efficiency;

[0045] The combination of hard pseudo-label generation through clustering and soft pseudo-label generation through the teacher network can reduce the noise in pseudo-labels, effectively solving the problem that the model may choose the wrong direction to converge in unsupervised metric learning training, and improving the generalization ability and robustness of the model;

[0046] This method combines the advantages of deep metric learning and knowledge distillation networks, and can be applied to different types of crop disease image classification problems. It has certain versatility and scalability, which helps to promote and apply it in actual production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the crop disease image classification method based on knowledge distillation network and metric learning of the present invention;

[0048] Figure 2 The present invention is a flowchart of using soft pseudo-label and hard pseudo-label supervision model training. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0050] The present invention proposes a crop disease image classification method based on knowledge distillation network and metric learning. The crop disease images are converted into feature vectors by an image feature extractor, and these vectors are mapped to the metric space by a mapper for similarity measurement. In the metric space, a clustering algorithm is used to generate hard pseudo labels for images, which are the main source of image labels. Soft pseudo labels are generated by the teacher network classifier, and the soft pseudo labels assist model training. A smoothing factor is introduced to filter the knowledge learned by the student network to reduce the negative impact of noise labels.

[0051] like Figure 1 and Figure 2 As shown, the present invention can solve the problem of lack of labels in the image data of crop diseases, and the method effectively solves the problem of pseudo-label noise diffusion in unsupervised deep metric learning methods. The knowledge distillation network mainly includes a teacher network and a student network. The main process of the method of the embodiment of the present invention is as follows:

[0052] Step 1: Use a web crawler to collect image data on agricultural websites, and have agricultural experts select the image data, and use the selected images as original images of crop diseases;

[0053] Step 2: Use the image feature extractor of the knowledge distillation network to extract the image feature vector; the specific process is:

[0054] Given two image feature extractors, namely teacher network feature extractor f t and the student network feature extractor f s , and is optimized by two different losses. Through the extractor f t and f s Encode the crop disease image into a feature vector; pass through the extractor f t (x) and f s The output of (x) is represented as the teacher network visual feature vector x t and the student network visual feature vector x s .

[0055] Step 3: Use the mapper of the knowledge distillation network to embed the image features into the metric space of each network; the specific process is as follows:

[0056] Given two mappers, teacher network mapper v t and Student Network Mapper v s , the mapper is represented by a linear layer. x t and x s After the mapper, it is embedded into the metric space. t and x s Represented as a 512-dimensional feature embedding vector x in the metric space t ′ and x s ′. The process can be expressed as:

[0057] X′ t =W T x t +b;

[0058] x′ s =W T x s +b;

[0059] Where W and b represent the weight and bias of the fully connected layer of the mapper, respectively; T is the transposed symbol.

[0060] Step 4: Use a clustering algorithm in the metric space of the teacher network to add hard pseudo labels to the image. Use the hard pseudo labels and the embedding vectors of the metric space in the metric space of the student network for comparative learning.

[0061] Step 4.1: Compare the feature embedding vector x of each image in the metric space using a hierarchical clustering algorithm t ′ and cluster them. In hierarchical clustering, the similarity between clusters is usually measured by Euclidean distance. The hard pseudo labels generated by clustering algorithms are also the main source of image labels.

[0062] After obtaining the hard pseudo labels, we can obtain a dataset with labeled data. The dataset is represented as D = {(x1, y1), (x2, y2), …, (x i ,y i ),…,(x I ,y I )}. Where I represents the number of all original crop disease images in the dataset; x i is the i-th image; y i is the hard pseudo label corresponding to the i-th image.

[0063] Step 4.2: Use the difficult triplet loss function for contrastive learning. First, use the hard pseudo label y of each image i Perform sampling operations to construct difficult triples for each image; the difficult triples constructed after the i-th image sampling operation are in is the anchor sample of the ith image, is the difficult positive sample of the i-th image, x i n is the difficult negative sample of the i-th image. and The acquisition process can be expressed by the formula:

[0064]

[0065]

[0066] Among them, x p is the anchor sample; x n is a difficult negative sample; the argmax(·) function finds the positive sample farthest from the anchor sample; d(·,·) represents the Euclidean distance.

[0067] Hard triplet loss function It can be expressed by the formula:

[0068]

[0069] Among them, σ is the distance hyperparameter in the triplet loss, which means that the distance between the anchor sample and the positive sample and the distance between the anchor sample and the negative sample are σ. [·] + Indicates that the calculation result is a positive value.

[0070] Step 4.3: In the process of calculating the Euclidean distance, the embedding vector of the image generated by the student network mapper is used. The model directly updates the parameters of the student network when updating parameters.

[0071] Step 5: Use the teacher network classifier to generate soft pseudo labels, and use the soft pseudo labels to supervise the class prediction vector of the student network classifier to assist model training; the specific process is:

[0072] Given two classifiers, namely teacher network classifier g t and the student network classifier g s , the classifier is represented by a linear layer. x t The vector generated by the teacher network classifier becomes the soft pseudo label p t That is, the probability of belonging to each category is less than 1. s The student network classifier generates the category prediction vector p s . The process can be expressed as:

[0073]

[0074]

[0075] in, and They represent the weight and bias of the fully connected layer of the classifier respectively.

[0076] Use soft pseudo label p t To supervise the category prediction vector p s , use the soft cross entropy loss function to calculate the loss. Soft cross entropy loss function It can be expressed by the formula:

[0077]

[0078] in, is the soft pseudo label of the i-th image; is the category prediction vector of the i-th image; Softmax(·) is the softmax function, which can be expressed by the formula:

[0079]

[0080] Among them, q represents the category prediction vector p s is an item in ; j is the category index; J represents the number of all categories. exp(·) is an exponential function with e as the base.

[0081] Step 6: Use the exponentially weighted moving average method to smoothly update the parameters of the teacher network to reduce the negative impact of noisy pseudo labels on model training. The specific process is:

[0082] The exponentially weighted moving average method is used to update the parameters of the teacher network. At the current iteration time t, the parameters of the teacher network are expressed as E(θ t ), the parameters of the student network are E(θ t ′). The updating process can be expressed by the formula:

[0083]

[0084] Among them, α is the smoothing factor;

[0085] At the beginning of network training, a small smoothing coefficient is used to enable the teacher network to obtain the distribution of training data more quickly. This will achieve more stable performance improvement in the early stage. As training progresses, the smoothing coefficient gradually increases, allowing the teacher network to adjust its decision boundary more finely. Obtaining more accurate hard and soft pseudo labels helps the student network better learn data feature representation and different image category information. This allows the model to avoid being misled by noisy pseudo labels in the process of learning the optimal decision boundary.

[0086] Step 7: Acquire the crop image to be identified in real time, and obtain the classification result of the image based on the trained model.

[0087] In order to prove the feasibility and superiority of the model trained by the method of the present invention, comparative experiments were conducted with the existing technologies Proxy-NCA (metric learning model based on proxy loss), Circle (metric learning model based on circle loss), Proxy-Anchor (metric learning model based on proxy-anchor loss), Contrastive (metric learning model based on contrastive loss), DAS (metric learning model based on dense anchor sampling), Probabilistic DML (metric learning model based on non-orthogonal probabilistic proxy) and MmPI (metric learning model based on hierarchical clustering). The experimental results are shown in Table 1:

[0088] Table 1 Performance comparison of each model

[0089]

[0090]

[0091] NMI measures the normalized similarity of two clustering results. Recall@K calculates the percentage of all positive images in the top K images returned. The larger their values, the stronger the performance of the model in the unsupervised deep metric learning image classification task. It can be seen from the experimental results that the method of the present invention achieves the best prediction results, and the values ​​of NMI, Recall@1, Recall@2, Recall@4 and Recall@8 are all maximum values.

[0092] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A crop disease image classification method based on knowledge distillation network and metric learning, characterized in that: The knowledge distillation network includes a teacher network and a student network; the method specifically includes the following steps: Step 1: Use a web crawler to collect image data on agricultural websites, and have agricultural experts select the image data, and use the selected images as original images of crop diseases; Step 2: Use the image feature extractor of the knowledge distillation network to extract the image feature vector; Step 3: Use the mapper of the knowledge distillation network to embed the image features into the metric space of each network; Step 4: Use a clustering algorithm to add hard pseudo labels to the image in the metric space of the teacher network, and use the hard pseudo labels and the embedding vectors of the metric space in the metric space of the student network for comparative learning; Step 5: Use the teacher network classifier to generate soft pseudo labels, and use the soft pseudo labels to supervise the class prediction vector of the student network classifier to assist model training; Step 6: Use the exponentially weighted moving average method to smoothly update the parameters of the teacher network; Step 7: Acquire the crop image to be identified in real time, and obtain the classification result of the image based on the trained model.

2. The crop disease image classification method based on knowledge distillation network and metric learning according to claim 1, characterized in that: The specific process of step 2 is: given two image feature extractors, namely teacher network feature extractor f t and the student network feature extractor f s , and is optimized by two different losses; through the extractor f t and f s Encode the crop disease image into a feature vector; pass through the extractor f t (x) and f s The output of (x) is represented as the teacher network visual feature vector x t and the student network visual feature vector x s .

3. The crop disease image classification method based on knowledge distillation network and metric learning according to claim 2 is characterized in that: The specific process of step 3 is as follows: given two mappers, namely teacher network mapper v t and Student Network Mapper v s , the mapper is represented by a linear layer; x t and x s After the mapper, it is embedded into the metric space; x t and x s Represented as a 512-dimensional feature embedding vector x in the metric space t ′ and x s ′ ; The process is expressed as: x t ′ =W T x t +b; x s ′ =W T x s +b; Where W and b represent the weight and bias of the fully connected layer of the mapper, respectively; T is the transposed symbol.

4. The crop disease image classification method based on knowledge distillation network and metric learning according to claim 3 is characterized in that: The specific process of step 4 is as follows: Step 4.1: Compare the feature embedding vector x of each image in the metric space using a hierarchical clustering algorithm t ′ The similarity is calculated and clustered, and hard pseudo labels are generated through clustering algorithm; According to the hard pseudo-labels, we construct a dataset with labeled data D = {(x1, y1), (x2, y2), …, (x i ,y i ),…,(x I ,y I )}; where I represents the number of all original crop disease images in the dataset; x i is the i-th image; y i is the hard pseudo label corresponding to the i-th image; Step 4.2: Use the difficult triplet loss function for contrastive learning. First, use the hard pseudo label of each image to perform sampling operations and construct difficult triplet for each image. The difficult triplet constructed after the i-th image sampling operation is in is the anchor sample of the i-th image, is the difficult positive sample of the i-th image, is the difficult negative sample of the i-th image; and The acquisition process is expressed by the formula: Among them, x p is the anchor sample; x n is a difficult negative sample; the argmax(·) function finds the positive sample farthest from the anchor sample; d(·,·) represents the Euclidean distance; Hard triplet loss function It is expressed by the formula: where σ is the distance hyperparameter in the triplet loss; [·] + Indicates that the calculation result is a positive value; Step 4.3: In the process of calculating the Euclidean distance, the embedding vector generated by the student network mapper is used; the model directly updates the parameters of the student network when updating parameters.

5. The crop disease image classification method based on knowledge distillation network and metric learning according to claim 4, characterized in that: The specific process of step 5 is as follows: Given two classifiers, namely teacher network classifier g t and the student network classifier g s , the classifier is represented by a linear layer; x t The teacher network classifier generates a soft pseudo label p t ;x s The student network classifier generates the category prediction vector p s ; The process is expressed as: in, and Respectively represent the weight and bias of the fully connected layer of the classifier; Use soft pseudo label p t To supervise the category prediction vector p s , use the soft cross entropy loss function to calculate the loss; soft cross entropy loss function It is expressed by the formula: in, is the soft pseudo label of the i-th image; is the category prediction vector of the i-th image; Softmax(·) is the softmax function, which is expressed by the formula: Among them, q represents the category prediction vector p s is an item in ; j is the category index; J represents the number of all categories; exp(·) is an exponential function with e as the base.

6. The crop disease image classification method based on knowledge distillation network and metric learning according to claim 5, characterized in that: In step 6, the parameters of the teacher network are updated using the exponentially weighted moving average method; at the current iteration time t, the parameters of the teacher network are expressed as E(θ t ), the parameters of the student network are E(θ t ′ ); The update process is expressed by the formula: Among them, α is the smoothing factor.

Images