A small sample learning method for apple tree leaf disease image classification

Abstract

The present application relates to a kind of small sample learning method for apple tree leaf disease image classification, with common feature similar apple rust, black spot and mixed disease as research object, based on small sample, using convolutional neural network learns the nonlinear mapping of input image in feature space, obtains the feature vector of each image.According to the feature vector, the center point, the average distance of all points from the center point and the radius of several clusters are calculated.Then, according to the relationship between the distance of the point to the cluster center and the radius, the outlier candidate set is found, then the local reachable density of the outlier factor in the outlier candidate set is calculated, and the outlier factor is confirmed according to the density value, and is removed.Finally, the remaining feature points are used as the support set, and the average value of the support set in the embedding space is calculated, and the nearest class prototype is found according to the search, so that the embedded query point can be classified, which reduces the cost under the premise of improving the classification accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to an image classification method, in particular to a small sample learning method for apple tree leaf disease image classification, which utilizes deep convolutional neural network to classify apple tree leaf diseases under small sample condition, finds out outliers in samples according to feature vector distance and sample point density between images, calculates the average value of support set in embedding space after removing outliers, and classifies according to the relationship between the sample and the value, effectively improving the class accuracy and robustness. BACKGROUND

[0002] Pathogens and insects are the main factors that threaten the safety of apple orchards. The timely deployment of orchard disease management depends on the early detection of diseases, and false and delayed diagnosis can lead to overuse or insufficient use of chemical drugs, thereby increasing production costs and having a huge impact on the environment and health. Currently, the detection of diseases and pests in apple orchards mainly relies on manual inspection by crop consultants. A large number of experts and farmer groups can determine plant diseases according to the symptoms on the leaves. However, using manpower to observe is both time-consuming and expensive. In addition, it is inefficient to continuously monitor all plants in the field area. Therefore, it is necessary to automatically detect plant diseases.

[0003] The development of computer imaging technology and machine learning shows great potential in accelerating disease diagnosis. Digital cameras can capture high-quality images of disease symptoms, and computer vision methods can use symptomatic digital images to classify diseases. With the rapid development of deep learning in computer vision, image-based plant disease detection has attracted particular attention. Recently, deep convolutional neural networks have demonstrated strong performance on image classification and detection problems. Using deep convolutional neural networks to extract image features, crop images taken under uniform settings are classified, especially using computer vision to identify crop diseases under more challenging photographic conditions, including complex backgrounds, different resolutions, and various lighting conditions. For example, using a convolutional neural network as the basic architecture to detect and distinguish banana spot disease from healthy leaves, and using a deep convolutional neural network to accurately identify apple leaf diseases, using a K-means clustering algorithm to detect infected parts, and using a support vector machine to classify healthy and infected apples based on color, texture, and shape. In addition, convolutional neural networks combined with transfer learning achieve good performance, which uses pre-trained models such as AlexNet, VGG, ResNet, etc., and then updates the parameters. In the past few years, a lot of research has been done on disease detection based on transfer learning. However, there is little difference between transfer learning methods, and the design of these methods, especially the development of methods using deep convolutional neural networks, largely depends on rich labeled data, but for apple tree leaf disease identification, there are many types of diseases and different morphologies, and the acquisition and labeling of the data set require a large amount of manpower and financial cost. SUMMARY

[0004] The technical problem to be solved by the present application is that the existing method using deep convolutional neural network for apple tree leaf disease requires a large amount of labeled data, resulting in high cost. The present application uses common apple rust, black spot and mixed diseases as research objects, based on small sample, uses convolutional neural network to learn the nonlinear mapping of input image in feature space, and obtains the feature vector of each image. According to the feature vector, the center point, the average distance of all points from the center point and the radius of several clusters are calculated. Then, according to the relationship between the distance of the point to the cluster center and the radius, the outlying candidate set is found out, and then the local reachable density of the outlying candidate set is calculated, and the outlying factors are confirmed according to the density value and removed. Finally, the remaining feature points are used as the support set, and the average value of the support set in the embedding space is calculated, and the nearest class prototype is found out, so as to classify the embedded query point. Under the premise of improving the classification accuracy and robustness, the cost is reduced.

[0005] The application discloses a small sample learning method for apple tree leaf disease image classification, which is based on small samples and adopts a convolutional neural network to accurately classify apple tree leaf diseases. n , y n )}, wherein each represents a D-dimensional feature vector of each sample, y i ∈(1,..., K) represents a corresponding label, and S k represents a set of samples of a class k. A deep convolutional neural network calculates an M-dimensional representation of each class through an embedding function, and calculates a mean vector of each class center point according to feature representations of all samples in the class. In each class, a distance d(c k , x i ) between each sample and a center point is calculated, and d(c k , x i ) is compared with R i . If d(c k , x i ) > R i , the sample is put into a "outlier candidate set". For convenience of representation, the sample and the center of each class are regarded as points. A center point is defined as O, and other sample points are defined as x. d w (O) is a wth distance of the point O, d w (O) = d(O, x), which represents that the point x is a wth nearest point to the point O. Therefore, an average reachable distance of all points in a wth neighborhood of the point O to the point O, i.e., a local reachable density, can be calculated. If the point O and the neighborhood points belong to the same class, the reachable distance is a smaller d w (O), and the local reachable density is larger, and vice versa. Then, according to the local reachable density, a local outlier factor can be calculated, which is represented as an average of ratios of local reachable densities of other points in a neighborhood N w (O) of the point O to the local reachable density of the point O. If the value is close to 1, it is indicated that densities of the neighborhood points of O are similar, and O can belong to the same class as the neighborhood; if the ratio is smaller than 1, it is indicated that the density of O is higher than that of the neighborhood points, and O is a dense point; and if the ratio is larger than 1, it is indicated that the density of O is smaller than that of the neighborhood points, and O can be an abnormal point. After the abnormal point is found, the abnormal point is removed. In each class, a mean vector of the center point is calculated again for other feature vectors after the abnormal point is removed. For each sample point x to be classified, a probability that the sample point x belongs to a class k is calculated, so that the leaf disease image classification under small samples is realized.

[0006] The following table shows an analysis table of influences of outlier removal on performance:

[0007]

[0008] The classification accuracy analysis table is as follows:

[0009] BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The first w distance diagram of the present application DETAILED DESCRIPTION

[0011] The present application is further described in detail below with reference to the accompanying drawings. A small sample learning method for apple tree leaf disease image classification is used to study common apple rust disease, black spot disease and mixed disease with similar features. Based on small samples, a convolutional neural network is used to learn the nonlinear mapping of input images in the feature space to obtain the feature vector of each image. The center point, the average distance of all points from the center point and the radius of several clusters are calculated according to the feature vector. Then, according to the distance of the point to the cluster center and the radius, the outlying candidate set is found, and the local reachable density of the outlying candidate set is calculated, and the outlying factors confirmed according to the density value are removed. Finally, the remaining feature points are used as the support set, and the average value of the support set in the embedding space is calculated. According to the search for the nearest class prototype, the embedded query point can be classified. The classification accuracy and robustness are improved, and the cost is reduced. In small sample classification, N classes of samples are defined as s = {(x1, y1),..., (xN, yN)}, where each x n , y n} represents the D-dimensional feature vector of each sample, and y i ∈(1,..., K) represents the corresponding label, and S k represents a set of samples of class k.

[0012] Deep convolutional neural network through embedding function Calculate the M-dimensional representation of each class where is the learnable parameter in the embedding function. According to the feature representation of all samples in the class, the mean vector c k of each class center point is calculated:

[0013]

[0014] where |S k | represents the number of samples in the set of samples of class k, represents the feature representation of sample x i . With c k as the class center point, the average distance of each other point from the center point is calculated as:​

[0015]

[0016] Where d(·, ·) represents the Euclidean distance between the feature vectors of two samples. Within each class, the distance d(c) between each sample and the median is calculated. k x i ), and d(c k x i ) and R i Compare them. If d(c) k x i )>R i The sample is then placed into the "outlier candidate set". For ease of representation, the sample and the center of each class are considered as points. The center point is defined as O, and the other sample points are defined as x. w (O) is the w-th distance from point O, d w (O) = d(O, x), which means that point x is the w-th point closest to O. For example... Figure 1 As shown, point x6 is the 6th point closest to the center O, and the value of w is 6. Define N. w (O) is the w-th distance neighborhood of point O, N w (O)={x′∈D\{O}|d(O,x′)≤d w (O)}, that is, N w (O) includes all points within the w-th neighborhood of point O. It is easy to see that N w (O)≥w, Figure 1 In the equation, the 6th distance neighborhood of point O is: N6(O) = {x1, x2, x3, x4, x5, x6, x7}. Based on the above definition, calculate the average reachability distance from all points in the w-th neighborhood of point O to O, i.e., the local reachability density:

[0017]

[0018] Where d w (O, x) is the w-th reachable distance from point x to point O, which is at least the w-th distance from point O. If point O and its neighboring points belong to the same category, then the reachable distance is the smaller d. w (O) indicates a high local density, while the opposite indicates a low local density.

[0019] The local outlier factor can be calculated based on the local reachability density, and the calculation method is as follows:

[0020]

[0021] The local outlier factor is represented by the neighborhood N of point O. w(O) The average of the ratio of the local reachable density of other points to the local reachable density of point O. If the value is close to 1, it means that the density of O's neighborhood is similar, O can belong to the same class as its neighborhood, if this ratio is less than 1, it means that the density of O is higher than its neighborhood, O is a dense point; if this ratio is greater than 1, it means that the density of O is less than its neighborhood, O can be an outlier. After finding the outliers, the outliers are removed.

[0022] In each class, the mean vector of the center point is calculated again according to formula (1) for other feature vectors after removing outliers. For each sample point x that needs to be classified, the probability that it belongs to class k is calculated as

[0023]

[0024] Here, the reciprocal of the distance is taken because the optimal distance is the smallest possible distance of the sample to the class. This distribution is based on the softmax over the distance between the embedding of the query sample and the reconstructed features of the class. The loss function for the small sample learning stage is

[0025]

[0026] In order to verify the performance of the small sample learning method for apple tree leaf disease image classification, the MiniImageNet dataset is used as a benchmark for evaluating the performance of the small sample learning method. The dataset is a randomly selected subset from ImageNet. It contains only 100 classes of 60,000 images, with 600 images per class. These images were taken by Canon Rebel T5i DSLR and smartphones under various lighting, angle, surface and noise conditions, and the dataset has high complexity. 3651 high-quality labeled RGB images are selected, including cedar apple rust, apple black star disease, complex disease (leaves with more than one disease on the same leaf) and healthy apple leaves. Among them, there are 1200 cedar apple rust, 1399 black star disease, 187 complex disease and 865 healthy leaves, and the imbalance of samples increases the complexity of the dataset. In order to evaluate the classification, 80% of the dataset is used for training and the remaining 20% is used for testing.

[0027] The experimental environment is Xeon(R) CPU E5-2620 V3, graphics card NVIDIA(R) Titan X, GPU memory is 12GB, driver is 418.67, CUDA is 10.1 version, operating system is Ubuntu 18.04LTS, and pytorch version is v1.0.0. The main network is AMDIM (ndf = 192, ndepth = 8, nrkhs = 1536), the dimension is set to 1536, and the learning rate is 0.0002.

[0028] First, in order to verify the influence of removing outliers on the classification effect, comparative experiments were carried out during training, and the network was trained when outliers existed and were removed respectively. The experimental results prove that after removing outliers, the network is easier to converge and has high classification accuracy.

[0029] Then, using support vector machine (SVM), AlexNet, GoogLeNet, VGGNet-16 and ResNet learning models, the apple disease data set was trained, the learning rate was set to 0.001, and SGD was selected as the optimization algorithm. The experimental results show that the proposed method can make full use of the large amount of information samples, and the accuracy on the test set reaches 97.62%, which is higher than other models. AlexNet model has good classification ability, and the average accuracy is 92.30%. GoogLeNet has multiple Inceptions and the ability of multi-dimensional feature extraction, but its network is not adjusted by apple pathological image features, and the final classification accuracy reaches 94.17%. ResNet-20 as a residual neural network, its accuracy is 94.42%. VGGNet-16 achieves an accuracy of 96.50% through transfer learning. In addition, the accuracy of the SVM model with SGD optimizer is 55.23%.

[0030] The experimental results show that the traditional method relies on expert-designed classification features to a great extent to improve recognition accuracy, and the level of expert experience has a great influence on the selection of classification features. Compared with the traditional method, the proposed small sample deep convolutional neural network can automatically extract the best classification features from multiple dimensions, and can also learn hierarchical features from low-level features such as edges, corners and colors to high-level semantics, effectively improving the classification accuracy and robustness.

Claims

1. A small sample learning method for apple tree leaf disease image classification, characterized in that: The convolutional neural network is used to learn the nonlinear mapping of the input image in the feature space, and the feature vector of each image is obtained, the center point, the average distance of all points from the center point and the radius of several clusters are calculated according to the feature vector, then the outlying candidate set is found according to the distance of the point to the cluster center and the radius, then the local reachable density of the outlying candidate set is calculated, and the outlying factor is confirmed according to the density value, and the outlying factor is removed, finally the remaining feature points are taken as the support set, and the average value of the support set in the embedding space is calculated, and the nearest class prototype is found according to the search, so that the embedded query point can be classified, according to the process of object recognition of human visual nervous system, a fine-grained classification model is designed based on saliency detection for agricultural pest images with various categories and large differences between classes and within classes, the model mainly consists of an object positioning module (OPM) and a multi-feature fusion module (MFFM), the OPM integrates image shallow detail information and deep spatial information through a U-shaped network structure, preliminarily delimits the salient region and outputs spatial semantic features, the MFFM obtains fusion features through bilinear pooling operation on spatial semantic features and abstract semantic features, increases detail features and weakens background information, after the image is processed by the OPM and the MFFM, auxiliary training is performed through target region cropping and mask, and finally the classification is performed according to the obtained features, for pest images with complex background and similar features, the cost is reduced under the premise of improving the classification accuracy; the OPM module is based on ResNet-50 and symmetrically designed as two parts of feature extraction and image generation, the input image size is 448*448, the original image and the target pixel-level label are input into the training model, after the image is processed by the feature extraction part, the output feature map is input into the transition bridge module Bridge for feature buffering, then similar to the U-shaped network, the feature map is inversely convoluted and up-sampled to the upper layer through the feature splicing convolution mode, in this process, each layer of the image generation part outputs a saliency detection image, and the loss function is calculated with the target pixel-level label, during the model construction process, due to the complex and diverse background of pest images, it is easy to confuse with the target, in order to improve the accuracy of target region segmentation, a more accurate image loss function L MBMS : In the formula, i represents the i-th convolutional layer; (a) a cross-entropy function L bce Loss for computing pixel-level annotation: wherein: G(w i ,h i ) represents the label value of the pixel point (w i ,h i ); s(w i ,h i ) represents the predicted value of the model; (b) a structural loss function L ms-ssim For the loss of computing region-level annotation, for any pixel point p in the region P with size N x N: Then, L can be obtained according to the above equations (3) to (6) ms-ssim : where x = {x j : j = 1,..., N 2} and y = {y j : j = 1,..., N 2} are the pixel values in the region P on the ground truth image G and the prediction result S, respectively; L(x, y) is the luminance contrast factor; C(x, y) is the contrast factor; S(x, y) is the structure contrast factor; μ x , μ y , and σ x , σ y are the mean and variance of x and y, respectively; σ xy is the covariance of x and y; M is the window shrinkage factor; is the center pixel of the region P; In the multi-feature fusion stage, a multi-feature fusion module MFFM is designed in combination with the idea of fine-grained image classification. Unlike simply using the shallow features of the input image for mask processing to guide the image to focus on the target region, the module adds the deep spatial semantic features obtained by the target positioning module, and performs multi-modal bilinear pooling processing on the deep abstract semantic features obtained by the residual network, realizes feature fusion, and obtains features that contain not only shallow detail information and deep semantic information, but also spatial position information. After the two feature matrices are fused, the data dimension is converted from [M x N] to [MN x 1], , where R denotes the real space, and the sign(x) function denotes the sign function, which returns 1 if x is greater than 0, returns 0 if x is equal to 0, and returns -1 if x is less than 0. Then, data processing is performed according to formula (8). The fused features are spliced with the spatial semantic features and the abstract semantic features, and input to a linear classifier, so that the disease and pest image classification can be performed.

Images