High definition identification method of plant leaf, system, apparatus, storage medium, and device

Abstract

To provide a high definition identification method of a plant leaf, a system, an apparatus, a storage medium, and a device.SOLUTION: A method initially masks an image of a plant leaf at random to obtain an emphasized image, inputs the emphasized image as a pair with an original image to a feature extraction network model, and obtains a feature vector. The method inputs the feature vector of the original image and the feature vector of a mask image to a classification network layer to perform variety identification, inputs the feature vector of the original image and the feature vector of the mask image to an adversarial network layer to perform binary classification identification, and inputs the feature vector of the mask image to a self-coding network module to use a loss function of three tasks, for self-supervised learning of image restoration. In a self-supervised task, the mask image makes a feature extraction network notice a local feature (texture, leaf vein) of a leaf, by learning of pixel restoration. The original image makes the feature extraction network notice original shape information of the leaf and the overall feature, in a variety identification task.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to the technical field of computer vision identification, and in particular to a method, system, apparatus, storage medium and device for high-resolution identification of plant leaves. [Background technology]

[0002] Plants not only provide us with oxygen and food, but also have high medicinal properties. However, many plant species are in danger of extinction due to human intervention. Reliable and efficient identification of plant varieties can promote plant breeders' property rights protection and innovation for genetic resources. Currently, many researchers have begun to actively research automatic plant identification methods. This research is not only useful for botanists, but also has great value for the general public. In most cases, people know the value of various plants, but are unable to identify them, and as a result, new and rare species cannot be protected in a timely manner. Research has been conducted on identifying plants from their leaves, flowers, and other parts. Among them, leaves are widely considered to be the most reliable source of information because they are available all year round and are one of the most stable main organs of a plant. For this reason, leaves are also called the "fingerprint" of a plant. However, it requires specialized knowledge, and analyzing plant leaves in a botanical laboratory is an expensive and time-consuming task. To solve this problem, many computer vision researchers use leaf images as a tool to identify plant types.

[0003] In the past decades, many methods have been used to identify plant species by extracting features from leaf images. These methods can be mainly categorized into traditional manual methods and deep learning methods. In traditional manual methods, leaf visual features such as leaf shape, texture, veins, and color are manually extracted as clues to identify plant species. Meanwhile, deep learning methods use leaf images as training samples and learn discriminant features directly from the original representation of leaf images. As more and more researchers and scholars pay attention to the rich information contained in plant leaves, artificial intelligence has been applied to intelligent classification of plant leaves. Deep learning methods have achieved many successes in automatically extracting leaf features in end-to-end learning for plant species identification. Compared with traditional manual methods, deep learning methods are fully data-driven, so they are generally more likely to capture discriminant features when sufficient training data is available.

[0004] Although methods based on leaf images have been widely used for plant species identification, plant species identification still poses great challenges due to the high leaf similarity between lineages. Because plant leaves have small inter-lineage differences and large intra-lineage differences, the use of plant species classification models is often ineffective, and there is a need to improve the performance of high-resolution classification. Summary of the Invention

[0005] In order to solve the shortcomings of the prior art and improve the performance of high-definition identification under the premise of guaranteeing inference speed, the present invention proposes a method, system, apparatus, storage medium and device for high-definition identification of plant leaves. The specific technical solutions are as follows:

[0006] According to a first aspect of the present invention, S1: constructing a high-resolution leaf classification dataset and dividing the high-resolution leaf classification dataset into a training set and a test set according to 8:2; Step S2: for each leaf image in the training set, scale the leaf image to a size of 224×224 to obtain an original image, divide the leaf image into N×N small blocks of the same size, and randomly select (N×N) / 2 small blocks for masking to generate a mask image corresponding to the original image; Step S3: inputting the original image and the mask image into a feature extraction network model in pairs to obtain feature maps {M,φ(M)} of the original image and the mask image, where M represents the feature map of the original image and φ(M) represents the feature map of the mask image; Step S4 of average pooling the feature maps {M, φ(M)} of the original image and the mask image and inputting them into a classification layer to obtain classification results for the feature maps {M, φ(M)} of the original image and the mask image, and calculating a classification loss function using the true label; Step S5 of average pooling the feature maps {M, φ(M)} of the original image and the mask image and inputting them into an adversarial network to obtain a binary classification result for the feature maps {M, φ(M)} of the original image and the mask image, and calculating an adversarial loss function using the true label; Step S6 of inputting the feature map φ(μ) of the mask image into an auto-encoding network module to perform pixel restoration of the image, obtain a restored image, and calculate the similarity of the pixels at the mask position between the restored image and the original image as a self-supervised loss function; Step S7: obtaining a weighted sum of the classification loss function, the adversarial loss function, and the self-supervised loss function to obtain an overall loss function, and training an entire network with the goal of minimizing the overall loss function, the entire network including at least the feature extraction network model, the classification layer, the adversarial network, and the auto-encoding network module; and step S8 of inputting the test set images into a trained feature extraction network model and a classification layer to predict classification, and determining a category of the image based on the predicted label distribution, Here, step S6 is A step of inputting the feature map φ(μ) of the mask image into an autoencoding network module composed of a four-layer transposed convolutional network to perform pixel restoration of the image to obtain a restored image Z', the size of the restored image Z' and the size of the original image Z are both 224×224; A step of calculating a similarity between pixels at a mask position between the restored image Z′ and the original image Z as a self-supervised loss function, the calculation of the self-supervised loss function being

number

number

[0007] According to a second aspect of the present invention, a high-resolution leaf classification dataset construction module configured to construct a high-resolution leaf classification dataset and split the high-resolution leaf classification dataset into a training set and a test set according to an 8:2 ratio; a mask image generation module configured to, for each leaf image in the training set, scale the size of the leaf image to 224×224 to obtain an original image, divide the leaf image into N×N small blocks of the same size, and randomly select and mask (N×N) / 2 small blocks to generate a mask image corresponding to the original image; a feature map acquisition module configured to input the original image and the mask image in pairs into a feature extraction network model to obtain feature maps {M, φ(M)} of the original image and the mask image, where M represents the feature map of the original image and φ(M) represents the feature map of the mask image; A classification module configured to average pool the feature maps {M, φ(M)} of the original image and the mask image and input them to a classification layer to obtain a classification result for the feature maps {M, φ(M)} of the original image and the mask image, and calculate a classification loss function using a true label; A binary classification module configured to average pool the feature maps {M, φ(M)} of the original image and the mask image, input them to an adversarial network, obtain a binary classification result for the feature maps {M, φ(M)} of the original image and the mask image, and calculate an adversarial loss function using a true label; A similarity calculation module configured to input the feature map φ(M) of the mask image to an auto-encoding network module to perform pixel restoration of the image to obtain a restored image, and calculate the similarity of pixels at the mask position between the restored image and the original image as a self-supervised loss function; a weighting fusion module configured to perform a weighted sum of the classification loss function, the adversarial loss function, and the self-supervised loss function to obtain an overall loss function, and to train an overall network with the goal of minimizing the overall loss function, the overall network including at least the feature extraction network model, the classification layer, the adversarial network, and the auto-encoding network module; an inference module configured to input the test set images into a trained feature extraction network model and a classification layer to make a classification prediction, and determine a category of the image based on a predicted label distribution, Here, the similarity calculation module specifically: The feature map φ(μ) of the mask image is input to an autoencoding network module consisting of a four-layer transposed convolutional network to perform pixel restoration of the image, thereby obtaining a restored image Z'; A similarity between the pixels at the mask position of the restored image Z′ and the original image Z is calculated as a self-supervised loss function; The size of the restored image Z' and the size of the original image Z are both 224 x 224, The self-supervised loss function is calculated as follows:

number

number

[0008] According to a third aspect of the present invention, there is provided a high-resolution plant leaf identification device comprising a memory and at least one processor, wherein the memory stores machine-executable instructions, and wherein the at least one processor, when executing the machine-executable instructions, performs the high-resolution plant leaf identification method described in the first aspect.

[0009] According to a fourth aspect of the present invention, there is provided a computer readable storage medium storing a program which, when executed by a processor, performs the method for high-resolution identification of plant leaves described in the first aspect.

[0010] According to a fifth aspect of the present invention there is provided a computing device including a memory and a processor, the memory storing machine executable instructions which, when executed by the processor, performs the method for high resolution plant leaf identification described in the first aspect.

[0011] The advantages and beneficial effects of the present invention are as follows:

[0012] The autoencoding network module allows the network to learn how to restore the masked leaf image to the original image. In the process of learning the restoration, the network must learn to pay attention to the relevant information of the leaf between small blocks, such as texture, veins, color and other continuity features. Therefore, this module allows the feature extraction network to pay attention to the micro-level leaf features and capture richer local information of the leaf.

[0013] The enhanced image (i.e., the mask image below) allows the model to pay more attention to local details such as leaf texture and veins, while adding some noise interference to the model, i.e., different from the actual data distribution. We design an adversarial loss function to distinguish between the original image and the enhanced image, and form an adversarial mutual coexistence with the classification loss to prevent the classification model from overfitting to the enhanced data.

[0014] The original image preserves the leaf shape information and global features of the leaf, while the enhanced image captures the local details of the leaf, and by integrating global and local features, it fully compensates for the shortcomings of current classification networks in extracting high-resolution features of leaves. This method achieved a classification accuracy of 90.6% on an existing high-resolution classification dataset of soybean leaves, demonstrating its advanced nature and practicality. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a network architecture diagram of the present invention. [Diagram 2] 2 is a flow chart of the training phase of the present invention. [Figure 3a] 1 is a sample example of an original image, a mask image, and a restored image in the algorithm of the present invention. [Figure 3b] 1 is a sample example of an original image, a mask image, and a restored image in the algorithm of the present invention. [Figure 3c] 1 is a sample example of an original image, a mask image, and a restored image in the algorithm of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which: It will be understood that the specific embodiments described herein are intended only to illustrate and explain the present invention and are not intended to limit the present invention.

[0017] As shown in FIGS. 1 and 2, the present invention provides a fine-grained identification method for plant leaves, including the following steps 1 to 8.

[0018] Step 1: Construct a high-resolution leaf classification dataset, and divide the high-resolution leaf classification dataset into a training set and a test set according to the 8:2 ratio.

[0019] For example, images of soybean leaves were collected in northeastern China. Specifically, 107 varieties of soybean were collected, with 30 images for each variety, for a total of 3210 images.

[0020] Step 2: For each leaf image in the training set, scale the size of the leaf image to 224×224 to be the original image, divide it into N×N small blocks of the same size, and randomly select (N×N) / 2 small blocks for masking to generate a mask image corresponding to the original image. Illustratively, N=14.

[0021] Download the convolutional network model ResNet50 pre-trained on the ImageNet dataset and use it as the feature extraction network model. The original image and the mask image are input into the feature extraction network model in pairs to obtain the feature maps {M,φ(M)} of the original image and the mask image output by the convolutional network of the final layer of the feature extraction network model. M represents the feature map of the original image, φ(M) represents the feature map of the mask image, and the size of the feature maps is 14 × 14. The feature maps {M,φ(M)} of the original image and the mask image are average pooled to obtain the feature vectors {Ι,φ(Ι)} of the original image and the mask image. I represents the feature vector of the original image, φ(Ι) represents the feature vector of the mask image, and the dimension of the feature vectors is 2048.

[0022] In one embodiment, the operation of average pooling the feature maps {M, φ(M)} of the original image and the mask image may be performed in steps 4 and 5 below.

[0023] Step 4: The feature vectors {Ι,φ(Ι)} of the original image and the mask image are input to a classification layer C(·) consisting of a 2048×107 fully connected network, and the classification result output by the classification layer C(·) is obtained, and the classification loss function is calculated using the true label l. The classification loss function is calculated as follows:

number

[0024] Step 5: Input the feature vectors {Ι,φ(Ι)} of the original image and the mask image to the adversarial network D(·) consisting of a 2048×2 fully connected network, obtain the binary classification result output by the adversarial network D(·), and encode it as the true label d∈{0,1} 2Here, the adversarial network is similar to what remains after removing the generator from a generative adversarial network. The adversarial network outputs a binary classification result for the feature vectors {Ι,φ(Ι)} of the original image and the mask image, and is used to calculate the adversarial loss function. The adversarial loss function is calculated as follows:

number

[0025] Step 6: The feature map φ(Μ) of the mask image is input to an auto-encoding network module composed of a four-layer transposed convolution network to perform pixel restoration of the image, and the restored image Z' is output. The size of the restored image Z' and the size of the original image Z are both 224 × 224. The similarity of the pixels at the mask position between the restored image Z' and the original image Z is calculated as the self-supervised loss function. In this embodiment, the mean square loss function is used, and the calculation of the self-supervised loss function is as follows:

number

number

[0026] Step 7: Obtain a weighted sum of the classification loss function, the adversarial loss function, and the self-supervised loss function to obtain a total loss function, and train the entire network, including at least the feature extraction network model, the classification layer, the adversarial network, and the autoencoding network module, with the goal of minimizing the total loss function. The calculation of the total loss function is as follows:

number

[0027] Step 8: Input the test set images into the trained feature extraction network model and classification layer to make classification predictions, and determine the category of the images based on the predicted label distribution.

[0028] In this embodiment, training is performed using the Adam Optimizer, the batch size is set to 32, the initial learning rate is 0.02, the current learning rate is multiplied by 0.1 when the training epoch is in [80,120], and training is stopped when the training epoch reaches 150. This can be achieved by using a single GPU server A100.

[0029] In this invention, when the difference between high-resolution image classes is small, the self-supervised task focuses the model on the local continuous features of the leaf (texture, veins), and the supervised task focuses the model on the original shape information and global features of the leaf. By integrating global and local features, the shortcomings of current classification networks in high-resolution feature extraction of leaves are fully compensated for. Furthermore, to bias the classification model to the actual test data, an adversarial loss function is used to distinguish between the original image and the enhanced image. The method can achieve a classification accuracy of 90.6% on the existing high-resolution classification dataset of soybean leaves, demonstrating its advanced nature and practicality.

[0030] The above-mentioned embodiments are only for illustrating the technical solutions of the present invention, but are not limited thereto. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can be modified, or part or all of the technical features therein can be replaced with equivalents, and such modifications or replacements do not take the essence of the corresponding technical solutions out of the scope of the technical solutions of the embodiments of the present invention.

Claims

1. S1: constructing a high-resolution leaf classification dataset and dividing the high-resolution leaf classification dataset into a training set and a test set according to 8:2; S2: for each leaf image in the training set, scale the leaf image to a size of 224×224 to obtain an original image, divide the leaf image into N×N small blocks of the same size, and randomly select (N×N) / 2 small blocks for masking to generate a mask image corresponding to the original image; Step S3: inputting the original image and the mask image into a feature extraction network model in pairs to obtain feature maps {M, φ(M)} of the original image and the mask image, where M represents the feature map of the original image and φ(M) represents the feature map of the mask image; Step S4 of average pooling the feature maps {M, φ(M)} of the original image and the mask image and inputting them into a classification layer to obtain classification results for the feature maps {M, φ(M)} of the original image and the mask image, and calculating a classification loss function using the true label; Step S5 of average pooling the feature maps {M, φ(M)} of the original image and the mask image and inputting them into an adversarial network to obtain a binary classification result for the feature maps {M, φ(M)} of the original image and the mask image, and calculating an adversarial loss function using the true label; Step S6 of inputting the feature map φ(M) of the mask image into an auto-encoding network module to perform pixel restoration of the image, obtain a restored image, and calculate the similarity of the pixels at the mask position between the restored image and the original image as a self-supervised loss function; Step S7: obtaining a weighted sum of the classification loss function, the adversarial loss function, and the self-supervised loss function to obtain an overall loss function, and training an entire network with the goal of minimizing the overall loss function, the entire network including at least the feature extraction network model, the classification layer, the adversarial network, and the auto-encoding network module; and a step S8 of inputting the test set images into a trained feature extraction network model and a classification layer to predict classification, and determining a category of the images based on the predicted label distribution, Here, step S6 is A step of inputting the feature map φ(M) of the mask image into an auto-encoding network module composed of a four-layer transposed convolutional network to perform pixel restoration of the image to obtain a restored image Z', the size of the restored image Z' and the size of the original image Z are both 224 x 224; A step of calculating a similarity between pixels at a mask position between the restored image Z′ and the original image Z as a self-supervised loss function, the calculation of the self-supervised loss function being [0010] And L ｓｓｌ represents the self-supervised loss function, n represents the total number of pixels in the mask position, and Z ｉ represents the pixel value of the i-th mask position of the original image Z, [0025] represents the pixel value of the i-th mask position of the reconstruction map Z′; A method for identifying plant leaves in high resolution, comprising:

2. In step S2, N=14.

2. The method for identifying plant leaves at high resolution according to claim 1.

3. Step S3 is downloading a convolutional network model ResNet50 pre-trained on the ImageNet dataset as the feature extraction network model, inputting the original image and the mask image into the feature extraction network model in pairs, and obtaining feature maps {M, φ(M)} of the original image and the mask image output by the final layer convolutional network, each of which has a size of 14×14; 2. The method for identifying plant leaves at high resolution according to claim 1.

4. Step S4 is average pooling the feature maps {M, φ(M)} of the original image and the mask image to obtain feature vectors {I, φ(I)} of the original image and the mask image, where I represents the feature vector of the original image, and φ(I) represents the feature vector of the mask image, and the dimensions of the feature vectors are both 2048; A step of inputting the feature vectors {I, φ(I)} of the original image and the mask image into a classification layer C(·) consisting of a 2048×107 fully connected network, obtaining a classification result output by the classification layer C(·), and calculating a classification loss function using a true label l, the calculation of the classification loss function being as follows: [0030] And L ｃｌｓ is the classification loss function, 2. The method for identifying plant leaves at high resolution according to claim 1.

5. Step S5 is average pooling the feature maps {M, φ(M)} of the original image and the mask image to obtain feature vectors {I, φ(I)} of the original image and the mask image, where I represents the feature vector of the original image, and φ(I) represents the feature vector of the mask image, and the dimensions of the feature vectors are both 2048; The feature vectors {I, φ(I)} of the original image and the mask image are input to an adversarial network D(·) consisting of a 2048×2 fully connected network, and the binary classification result output by the adversarial network D(·) is obtained, and the one-hot encoded true label d∈{0, 1} ２ A step of calculating an adversarial loss function using [0045] And L ａｄｖ is the adversarial loss function, 1 is a vector of (1,1), the true label of the original image d is (1,0), and the true label of the mask image d is (0,1).

2. The method for identifying plant leaves at high resolution according to claim 1.

6. Step S7 a weighted sum of the classification loss function, the adversarial loss function, and the self-supervised loss function to obtain an overall loss function, and training the entire network with the goal of minimizing the overall loss function, the calculation of the overall loss function comprising: [0050] where L is the total loss function, and L ｃｌｓ is the classification loss function, L ａｄｖ is the adversarial loss function, and α, β, and γ are the classification loss functions L ｃｌｓ , adversarial loss function L ａｄｖ , self-supervised loss function L ｓｓｌ are weights corresponding to α=1, β=1, and γ=0.

8.

2. The method for identifying plant leaves at high resolution according to claim 1.

7. The entire network is trained using the Adam optimizer with a batch size of 32, an initial learning rate of 0.02, a current learning rate multiplied by 0.1 when the training epoch is in [80, 120], and training is stopped when the training epoch reaches 150.

7. The method for identifying plant leaves at high resolution according to claim 6.

8. a high-resolution leaf classification dataset construction module configured to construct a high-resolution leaf classification dataset and split the high-resolution leaf classification dataset into a training set and a test set according to an 8:2 ratio; a mask image generation module configured to, for each leaf image in the training set, scale the size of the leaf image to 224×224 to obtain an original image, divide the leaf image into N×N small blocks of the same size, and randomly select and mask (N×N) / 2 small blocks to generate a mask image corresponding to the original image; a feature map acquisition module configured to input the original image and the mask image in pairs into a feature extraction network model to obtain feature maps {M, φ(M)} of the original image and the mask image, where M represents the feature map of the original image and φ(M) represents the feature map of the mask image; A classification module configured to average pool the feature maps {M, φ(M)} of the original image and the mask image and input them to a classification layer to obtain a classification result for the feature maps {M, φ(M)} of the original image and the mask image, and calculate a classification loss function using a true label; A binary classification module configured to average pool the feature maps {M, φ(M)} of the original image and the mask image, input them to an adversarial network, obtain a binary classification result for the feature maps {M, φ(M)} of the original image and the mask image, and calculate an adversarial loss function using a true label; A similarity calculation module configured to input the feature map φ(M) of the mask image to an auto-encoding network module to perform pixel restoration of the image to obtain a restored image, and calculate a similarity of pixels at the mask position between the restored image and the original image as a self-supervised loss function; a weighting fusion module configured to perform a weighted sum of the classification loss function, the adversarial loss function, and the self-supervised loss function to obtain an overall loss function, and to train an overall network with the goal of minimizing the overall loss function, the overall network including at least the feature extraction network model, the classification layer, the adversarial network, and the auto-encoding network module; an inference module configured to input the test set images into a trained feature extraction network model and a classification layer to make a classification prediction, and determine a category of the image based on a predicted label distribution, Here, the similarity calculation module specifically: The feature map φ(M) of the mask image is input to an autoencoding network module composed of a four-layer transposed convolutional network to perform pixel restoration of the image, thereby obtaining a restored image Z'; The similarity between the pixels at the mask position of the restored image Z′ and the original image Z is calculated as a self-supervised loss function; The size of the restored image Z' and the size of the original image Z are both 224 x 224, The self-supervised loss function is calculated as follows: [006] And L ｓｓｌ represents the self-supervised loss function, n represents the total number of pixels in the mask position, and Z ｉ represents the pixel value of the i-th mask position of the original image Z, [0070] represents the pixel value of the i-th mask position of the restoration map Z′. A high-definition plant leaf identification system.

9. A high-resolution plant leaf identification device comprising a memory and at least one processor, the memory storing machine-executable instructions, the at least one processor executing the machine-executable instructions to perform the high-resolution plant leaf identification method according to any one of claims 1 to 7. A high-definition plant leaf identification device characterized by the above.

10. A computer-readable storage medium storing a program, the program being executed by a processor to perform the high-resolution plant leaf identification method according to any one of claims 1 to 7. A computer-readable storage medium comprising:

11. A computing device including a memory and a processor, the memory storing machine-executable instructions, the processor executing the machine-executable instructions to perform the method for identifying high-resolution plant leaves according to any one of claims 1 to 7. A computing device comprising:

Images