Fine-grained plant species recognition method based on pair-wise feature learning

Abstract

The application discloses a fine-grained plant variety identification method based on pair feature learning, constructs a plant variety identification model, and comprises a feature extraction module and a classification module. A mutual attention learning module is arranged between the feature extraction module and the classification module, so that a pair feature learning network is constructed. The pair feature learning network is trained by using a training sample set, and then the feature extraction module and the classification module are extracted from the trained pair feature learning network to form a trained plant variety identification model. The leaf image to be identified is input into the trained plant variety identification model, and an identification result is obtained. The pair feature learning network is used for learning the similar features between the same type of leaves and the different features of different types of leaves, so that the subtle differences between the leaves are compared, and the accuracy of plant variety identification is improved.

Description

Technical Field

[0001] This invention belongs to the field of plant variety technology, and more specifically, relates to a fine-grained plant variety identification method based on pairwise feature learning. Background Technology

[0002] Plant variety identification is a field of great interest in both variety intellectual property protection and new variety breeding research. Traditional methods for identifying plant varieties require breeders to possess specialized breeding knowledge and extensive experience, and suffer from drawbacks such as high barriers to entry, low efficiency, and inconsistent accuracy. Over the decades, various molecular marker methods have emerged in the field of plant variety identification. These methods often involve complex experimental procedures, require a certain level of professional skill, and are time-consuming, making them virtually unusable for ordinary agricultural workers such as farmers. Therefore, we urgently need a simple, rapid, and accurate method for plant variety identification.

[0003] Leaves play a crucial role in botanists' identification of plant species, and using computer vision methods to extract leaf features from leaf images for classification is an ideal approach. On one hand, the shape, texture, and veins of plant leaves are more durable and easily distinguishable than other parts of the plant, and are widely used in plant taxonomy and species identification. On the other hand, trained end-to-end computer vision models can quickly and accurately classify target images, allowing users to easily and quickly obtain classification results without needing to understand the internal mechanisms.

[0004] Numerous computer vision software and algorithms for plant variety classification based on leaf classification have been reported to date. Algorithms invented by Price, CA et al., De Vylder, J et al., and Zhou, J et al. identify leaves by constructing morphological phenotypes. Other methods utilize leaf shape, vein, and texture features individually; for example, Wang, B et al. classified leaves of different species by extracting shape features, achieving good results. Naresh, Y et al. classified leaves by extracting texture features. Furthermore, many researchers have attempted to classify leaves using deep learning. Lee et al. used convolutional neural networks to learn from raw leaf images for leaf image recognition. Tan et al. proposed an effective plant leaf image recognition model by combining multiple pre-trained CNN networks as feature extractors. However, despite significant progress in plant species classification, the software and algorithms they proposed struggle to capture subtle differences in leaf features between different varieties of the same species, resulting in unsatisfactory performance.

[0005] Recently, Zhang, Y, et al. proposed a deep learning leaf classification method called MFCIS, which extracts and concatenates the shape, veins, and texture features of leaves, then inputs the concatenated features into a fully connected layer to complete the classification. However, this model has insufficient classification accuracy and poor generalization ability. Furthermore, when dealing with plant leaves whose shape changes throughout the growth cycle, the network needs to be trained separately for each growth cycle. H. Tavakoli et al. proposed a feature extraction method that extracts pixel intensity features from grayscale images of leaves, inputs these features into a neural network for classification, and conducted experiments on a soybean leaf dataset. Although the paper presents a novel feature extraction method, the classification accuracy is unsatisfactory. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fine-grained plant variety identification method based on pairwise feature learning. By learning the similar features between leaves of the same type and the differences between leaves of different types through a pairwise feature learning network, the method can compare subtle differences between leaves and thus improve the accuracy of plant variety identification.

[0007] To achieve the above-mentioned objectives, the fine-grained plant variety identification method based on pairwise feature learning of the present invention includes the following steps:

[0008] S1: Collect leaf image samples of several plants according to the actual situation, and label the plant species of each leaf image sample to obtain the training sample set.

[0009] S2: Construct a plant variety identification model based on the actual situation, including a feature extraction module and a classification module. The feature extraction module is used to extract features from leaf images to obtain leaf features, and the classification module is used to obtain classification results based on leaf features.

[0010] S3: A mutual attention learning module is set between the feature extraction module and the classification module of the plant variety identification model to construct a pairwise feature learning network; for training sample I in the training sample set, positive samples I are selected from its class of samples. s Negative samples I are selected from samples of different classes. d Then, the corresponding training sample features X and positive sample features X are input into the feature extraction module respectively to obtain the corresponding training sample features X and positive sample features X. s Negative sample features X d The training sample features X and the positive sample features X s Negative sample features X d Input the mutual attention learning module to extract training sample features X and positive sample features X. s Mutual attention between training sample features X and negative sample features X dThe mutual attention between the three features is used to fuse the three features, resulting in feature X′, which is then input into the classification module to obtain the classification result.

[0011] S4: The pairwise feature learning network is trained using the training sample set, where the training samples for each batch are selected using the following method:

[0012] N categories are randomly selected from all categories in the training sample set, and K training samples are randomly selected from each category. These N×K training samples are then input into the current feature extraction module to obtain the features of each training sample. For each training sample, the sample with the highest similarity to the feature is selected from the samples of the same class as the positive sample, and the sample with the lowest similarity to the feature is selected from the samples of different classes as the negative sample.

[0013] Then, feature extraction and classification modules are extracted from the trained pairwise feature learning network to form a trained plant variety recognition model.

[0014] S5: Input the image of the leaf to be identified into the trained plant variety recognition model to obtain the recognition result.

[0015] This invention relates to a fine-grained plant variety identification method based on convolutional neural networks. A plant variety identification model is constructed, comprising a feature extraction module and a classification module. A mutual attention learning module is established between the feature extraction module and the classification module to construct a pairwise feature learning network. The pairwise feature learning network is trained using a training sample set. Then, the feature extraction module and the classification module are extracted from the trained pairwise feature learning network to form a trained plant variety identification model. The leaf image to be identified is input into the trained plant variety identification model to obtain the identification result.

[0016] The present invention has the following beneficial effects:

[0017] 1) This invention improves the accuracy of plant variety identification by fully extracting the common features of leaf images of the same type and the difference features between leaf images of different types through pairwise feature comparison learning;

[0018] 2) This invention achieves good results on leaf datasets of different varieties and has good generalization ability;

[0019] 3) This invention is an end-to-end model, without complicated preprocessing steps and training processes, making it easy to use. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a specific implementation of the fine-grained plant variety identification method based on pairwise feature learning of the present invention.

[0021] Figure 2This is a structural diagram of the plant variety identification model in this invention;

[0022] Figure 3 This is a structural diagram of the mutual attention learning module in this embodiment. Detailed Implementation

[0023] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0024] Example

[0025] Figure 1 This is a flowchart illustrating a specific implementation of the fine-grained plant variety identification method based on pairwise feature learning of the present invention. Figure 1 As shown, the specific steps of the fine-grained plant variety identification method based on pairwise feature learning of the present invention include:

[0026] S101: Obtain plant leaf image samples:

[0027] Based on the actual situation, collect leaf image samples of several plants, and label each leaf image sample with its plant species to obtain a training sample set.

[0028] S102: Constructing a plant variety identification model:

[0029] Develop a plant variety identification model based on the actual situation. Figure 2 This is a structural diagram of the plant variety identification model in this invention. (Example) Figure 2 As shown, the plant variety identification model in this invention includes a feature extraction module and a classification module. The feature extraction module is used to extract features from leaf images to obtain leaf features, and the classification module is used to obtain classification results based on the leaf features.

[0030] Effectively extracting various features from leaf images is crucial for this invention. With advancements in image capture technology, the clarity of obtained leaf images has increased significantly, encompassing a wealth of texture and vein features. However, due to the high resolution of the input leaf images, directly using the original size as network input would result in an excessively large model size, making it impractical. A method is needed to reduce the original image size while preserving its original texture and vein features. To address this issue, this embodiment proposes a feature extraction module based on multi-scale feature fusion. Figure 3 This is a structural diagram of the feature extraction module in this embodiment. For example... Figure 3As shown, the feature extraction module in this embodiment includes a downsampling module, a downsampled image convolutional neural network, a center cropping module, a center image convolutional neural network, and a feature stitching module. Each module will be described in detail below.

[0031] The downsampling module is used to downsample the leaf image to obtain a downsampled image of size H×W, and then send it to the downsampled image convolutional neural network.

[0032] A downsampled image convolutional neural network is used to extract features x1 from the downsampled image and send them to the feature concatenation module. The contour features of the blade can be effectively extracted based on the downsampled image.

[0033] The center cropping module is used to crop the leaf image to obtain a center image of size H×W, and then send it to the center image convolutional neural network.

[0034] A convolutional neural network based on the center image is used to extract features x2 from the center image and send them to the feature concatenation module. Based on the center image, texture and vein features of the leaves can be effectively extracted.

[0035] The feature splicing module is used to splice feature x1 and feature x2 to obtain blade feature X.

[0036] As described above, the feature extraction module in this embodiment scales the image to a size acceptable for training the convolutional neural network while preserving the contour, vein, and texture features of the leaf image, playing a significant role in leaf classification. In this embodiment, the downsampled image convolutional neural network and the center image convolutional neural network are implemented based on the ResNet-50 network, removing the last two fully connected layers of the ResNet-50 network and using their output as features.

[0037] The classification module can be implemented using common classifiers. In this embodiment, a fully connected layer plus a softmax layer is used.

[0038] S103: Constructing a pairwise feature learning network:

[0039] A mutual attention learning module is set between the feature extraction module and the classification module of the plant variety identification model to construct a pairwise feature learning network. For training sample I in the training sample set, positive samples I are selected from its class of samples. s Negative samples I are selected from samples of different classes. d Then, the corresponding training sample features X and positive sample features X are input into the feature extraction module respectively to obtain the corresponding training sample features X and positive sample features X. s Negative sample features X d The training sample features X and the positive sample features X s Negative sample features Xd Input the mutual attention learning module to extract training sample features X and positive sample features X. s Mutual attention between training sample features X and negative sample features X d The mutual attention between the three features is used to fuse the three features, resulting in feature X′, which is then input into the classification module to obtain the classification result.

[0040] Figure 3 This is a structural diagram of the mutual attention learning module in this embodiment. For example... Figure 3 As shown, the mutual attention learning module in this invention includes a positive sample feature enhancement module, a positive sample multilayer perceptron, a positive sample sigmoid module, a negative sample feature enhancement module, a negative sample multilayer perceptron, a negative sample sigmoid module, and a feature fusion module, wherein:

[0041] The positive sample feature enhancement module is used to calculate the training sample features X and the positive sample features X. s The Hadamard product yields positive sample enhancement features. And send it to the positive sample multilayer perceptron.

[0042] Positive sample multilayer perceptron is used to enhance the features of positive samples. The feature vector S is obtained through processing and then sent to the positive sample Sigmoid module.

[0043] The positive sample Sigmoid module is used to process the feature vector S using the Sigmoid function to obtain the common attention vector g. s And send it to the feature fusion module.

[0044] The negative sample feature enhancement module is used to calculate the training sample features X and the negative sample features X. d The Hadamard product yields positive sample enhancement features. And send it to the negative sample multilayer perceptron.

[0045] Negative sample multilayer perceptron is used to enhance features of negative samples. The feature vector D is obtained through processing and then sent to the positive sample Sigmoid module.

[0046] The negative sample Sigmoid module is used to process the feature vector D using the Sigmoid function to obtain the differential attention vector g. d It is then sent to the feature fusion module.

[0047] The feature fusion module is used to employ the common attention vector g s and difference attention vector g d The feature X′ is obtained by processing the feature X of the training sample. The calculation formula is as follows:

[0048] X′=X+(X⊙g s )+(X⊙g d )

[0049] As described above, the mutual attention learning module in this embodiment has two branches: positive sample feature processing and negative sample feature processing. First, it obtains a vector S emphasizing intra-class commonalities and a vector D emphasizing inter-class differences. Then, it obtains the common attention vector g. s and difference attention vector g d Common attention vector g s This is used to represent common attention among different images of the same class. It highlights common semantics within the class by focusing attention on common regions of different images within the same class, while the difference attention vector g d By focusing attention on the regions of difference between different classes, semantic differences between classes are highlighted. The resulting feature X′ contains features of commonalities among the same varieties, as well as features of differences between images of different varieties. By distinguishing such features, classification errors in fine-grained classification can be reduced.

[0050] S104: The plant variety recognition model obtained through training:

[0051] The pairwise feature learning network is trained using a training sample set. To ensure that each batch learns features effectively, the training samples for each batch are selected during training using the following method:

[0052] N categories are randomly selected from all categories in the training sample set, and K training samples are randomly selected from each category. These N×K training samples are then input into the current feature extraction module to obtain the features of each training sample. For each training sample, the sample with the highest feature similarity from samples of the same category is selected as a positive sample, and the sample with the lowest feature similarity from samples of different categories is selected as a negative sample. In this embodiment, Euclidean distance is used for feature similarity.

[0053] Then, feature extraction and classification modules are extracted from the trained pairwise feature learning network to form a trained plant variety recognition model.

[0054] S105: Plant Variety Identification

[0055] The image of the leaf to be identified is input into the trained plant variety recognition model to obtain the recognition result.

[0056] To better illustrate the technical effects of this invention, a specific example is used for experimental verification. In this experimental verification, the soybean variety leaf dataset released in 2017 by the Soybean Experimental Station of the Heilongjiang Academy of Agricultural Sciences in Mudanjiang City, Heilongjiang Province, and the sweet cherry dataset released in May 2020 by the Shandong Fruit Research Institute in Tai'an City, Shandong Province, are used. In this embodiment, the training sample image size is 3000×3000, and the downsampled image and center image size are 448×448, resulting in a feature vector dimension of 2048. During the training of the pairwise feature learning network, 10 categories are randomly selected in each batch, and 4 images are randomly extracted from each category. The SGD algorithm is used for optimization, with a momentum of 0.9, a learning rate of 0.001, and a weight decay of 5e. -4 In this embodiment, five cross-validations were performed to obtain more reliable results.

[0057] In this embodiment, six commonly used image classification techniques are selected as comparison methods, namely:

[0058] VGG16, see the paper "Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.15562014."

[0059] ResNet50, see the literature "He, K.; Zhang,

[0060] HSC, see "Wang, B.; Gao, Y. Hierarchical string cuts: a translation, rotation, scale, and mirror invariant descriptor for fast shape retrieval. IEEE Transactions on Image Processing 2014, 23, 4101–4111."

[0061] PH, see the document "Reininghaus, J. et al. A stable multi-scale kernel for topological machine learning, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 4741–4748 (IEEE, Boston, MA, USA, 2015)."

[0062] Xception, see the literature "Chollet, F.

[0063] MFCIS, see the literature "Zhang, Y.; Peng, J.; Yuan,

[0064] In this embodiment, the classification accuracy, F1-Score, and AUC (Area Under Curve) of the embodiments of the present invention and six comparison methods are statistically analyzed. Table 1 is a statistical comparison table of the experimental results of the embodiments of the present invention and the other six comparison methods on the sweet cherry dataset. Table 2 is a statistical comparison table of the experimental results of the embodiments of the present invention and the other six comparison methods on the soybean variety leaf dataset.

[0065]

[0066] Table 1

[0067]

[0068] Table 2

[0069] The results of the leaf count rejection test for sweet cherry are shown in Table 1. Our invention achieved an accuracy 0.1203 higher, an F1-Score 0.1267 higher, and an AUC 0.0393 higher than the best-performing model. The results of the leaf count rejection test for soybean varieties are shown in Table 2. Our invention achieved an accuracy 0.2396 higher, an F1-Score 0.1762 higher, and an AUC 0.0865 higher than the best-performing model. In conclusion, compared to previous leaf-based variety classification methods, our method demonstrates a significant advantage in fine-grained leaf classification tasks, accurately classifying the leaves of various plant varieties after training.

[0070] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A fine-grained plant variety identification method based on pairwise feature learning, characterized in that, Includes the following steps: S1: Collect leaf image samples of several plants according to the actual situation, and label the plant species of each leaf image sample to obtain the training sample set. S2: Construct a plant variety identification model based on the actual situation, including a feature extraction module and a classification module. The feature extraction module is used to extract features from leaf images to obtain leaf features, and the classification module is used to obtain classification results based on leaf features. S3: A mutual attention learning module is set between the feature extraction module and the classification module of the plant variety identification model to construct a pairwise feature learning network. The mutual attention learning module includes a positive sample feature enhancement module, a positive sample multilayer perceptron, a positive sample sigmoid module, a negative sample feature enhancement module, a negative sample multilayer perceptron, a negative sample sigmoid module, and a feature fusion module. The feature fusion module is used to apply the common attention vector output by the positive sample sigmoid module. The difference attention vector output by the negative sample Sigmoid module Features of training samples Features are obtained through processing The calculation formula is: ； For the training samples in the training sample set Select positive samples from its similar samples. negative samples are selected from samples of different classes. Then, the corresponding training sample features are input into the feature extraction module. Positive sample characteristics Negative sample features , train sample features Positive sample characteristics Negative sample features Input mutual attention learning module to extract features from training samples and positive sample features Mutual attention between them, training sample features and negative sample features Mutual attention between the three features and fusion of the three features based on mutual attention to obtain the feature. And input the data into the classification module to obtain the classification results; S4: The pairwise feature learning network is trained using the training sample set, where the training samples for each batch are selected using the following method: Randomly select from all categories in the training sample set There are 10 categories, and each category is randomly selected. 1 training sample, then this Each training sample is input into the current feature extraction module to obtain the features of each training sample; for each training sample, the sample with the highest similarity to the feature is selected from the samples of the same class as the training sample as the positive sample, and the sample with the lowest similarity to the feature is selected from the samples of different classes as the negative sample. Then, feature extraction and classification modules are extracted from the trained pairwise feature learning network to form a trained plant variety recognition model. S5: Input the image of the leaf to be identified into the trained plant variety recognition model to obtain the recognition result.

2. The fine-grained plant variety identification method according to claim 1, characterized in that, The feature extraction module in step S2 includes a downsampling module, a downsampling image convolutional neural network, a center cropping module, a center image convolutional neural network, and a feature concatenation module, wherein: The downsampling module is used to downsample the leaf image to obtain a size of The downsampled image is then sent to the downsampled image convolutional neural network; Downsampled image convolutional neural networks are used to extract features from downsampled images. And send it to the feature splicing module; The center cropping module is used to center crop the leaf image to obtain a size of [size missing]. The center image is sent to the center image convolutional neural network; A convolutional neural network for the center image is used to extract features from the center image. And send it to the feature splicing module; The feature concatenation module is used to concatenate features. and characteristics By splicing the parts together, the leaf features are obtained. .

3. The fine-grained plant variety identification method according to claim 1, characterized in that, In the mutual attention learning module of step S3: The positive sample feature enhancement module is used to calculate the features of training samples. and positive sample features The Hadamard product yields positive sample enhancement features. And send it to the positive sample multilayer perceptron; Positive sample multilayer perceptron is used to enhance the features of positive samples. The feature vector is obtained through processing. And send it to the positive sample Sigmoid module; The positive sample Sigmoid module is used to apply the Sigmoid function to the feature vector. After processing, a common attention vector is obtained. And send it to the feature fusion module; The negative sample feature enhancement module is used to calculate the features of training samples. and negative sample features The Hadamard product yields the negative sample enhancement features. And send it to the negative sample multilayer perceptron; Negative sample multilayer perceptron is used to enhance features of negative samples. The feature vector is obtained through processing. And send it to the negative sample Sigmoid module; The negative sample Sigmoid module is used to apply the Sigmoid function to the feature vector. The process is performed to obtain the differential attention vector. It is then sent to the feature fusion module.

Images