An image leaf recognition method and device based on a hierarchical attention mechanism

Abstract

The application discloses a kind of image leaf identification method and device based on hierarchical attention mechanism, including generating the different levels of category activation map of input image;Different levels of category activation map is fused into an attention map, and using the attention map, obtain several different sampling images;The feature vectors of each sampling image are spliced with the feature vectors of input image, and based on the spliced feature vectors obtained, obtain image leaf identification result.The present application is based on the classification strategy of human botanist to construct hierarchical attention mechanism, and combines sampling transformation and deep learning method, amplifies the key area in picture, and can achieve very high accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an image leaf recognition method and device based on a hierarchical attention mechanism, which can accurately identify leaf categories in natural scenes. Background Technology

[0002] Plants are widely distributed in natural habitats and human societies, providing essential nutrients for human survival and playing a vital role in regulating carbon-oxygen balance and environmental temperature and humidity. Plant classification is of paramount importance in species conservation, agriculture, and the horticulture industry. However, due to the high cost of manual classification, the need for highly specialized botanical knowledge among classification personnel, and the reliance on extensive training, automated plant classification holds greater promise and is therefore essential for research. Automated plant classification includes the automatic classification of flowers, fruits, leaves, and tissues. Compared to flowers and fruits, which vary significantly across different growth stages, plant leaves exhibit more stable characteristics, making leaf classification one of the primary methods for achieving automated plant classification.

[0003] In computer vision, despite numerous attempts (such as using complex computational vision algorithms), plant recognition remains a challenging and unsolved problem due to the vast diversity of plant species in nature and the small shape differences between similar species. The main difficulty in plant leaf recognition lies in the small inter-class differences and large intra-class variance. On the one hand, compared to general image recognition, leaves of similar plant categories often have only a few minor differences, placing high demands on the model's recognition capabilities. The model needs to be able to locate the distinguishable regions on the leaf in order to correctly classify it. On the other hand, images of leaves from the same plant species also exhibit significant variance. For example, the shape, color, and vein pattern of leaves change considerably at different growth stages, further increasing the difficulty of automated plant leaf recognition.

[0004] Generally, semi-supervised methods in machine learning can effectively identify most plants when processing plant leaf images, but this method is time-consuming and labor-intensive, hindering its widespread application. Currently, with the development of image technology, automatic classification of leaf images after feature extraction using computer technology has become the mainstream approach. In related research, leaf features are usually extracted manually; for example, leaf classification often utilizes shape differences between different leaves, with leaf edges being a key feature and frequently used for extraction. Furthermore, techniques such as leaf vein texture detection, combining singular value decomposition (SVD) and sparse representation (SR) to process dimensionality-reduced plant images, as well as moment-invariant methods for multi-component shapes and artificial neural networks using support vector machines, have also achieved some success. However, these methods rely on feature selection and manual processing. Processing high-level and multi-scale features significantly increases complexity and workload, leading to a decline in the versatility of these methods as the number and types of plants to be identified increase.

[0005] In recent years, due to the superior performance of deep learning convolutional neural networks in computer vision, they have become a primary means of solving problems such as image classification, image recognition, and semantic segmentation. Deep learning methods have achieved good results in plant classification, with overall performance superior to most manual feature extraction and classification methods, especially their excellent generalization ability. Most recent leaf recognition methods are based on end-to-end training of deep convolutional neural networks and use leaf features at a single scale, which limits the accuracy of leaf recognition. Furthermore, data-driven deep learning methods require a large amount of labeled training data, but due to the difficulty of fine-grained annotation, such annotations are often hard to obtain in practical applications, limiting the application of deep learning-based leaf recognition methods in real-world scenarios. Therefore, it is essential to research a method that can accurately identify leaves using multi-scale features of leaf images. Summary of the Invention

[0006] This invention proposes a method and apparatus for leaf recognition in natural scene images based on a hierarchical attention mechanism. Inspired by the classification strategies of botanists, this invention utilizes the inherent hierarchical structure of deep neural networks to design a hierarchical attention mechanism that extracts leaf features from coarse to fine. Then, selective sampling is performed based on the attention map for image enhancement, and finally, an S-CNN is trained for leaf image classification. Specifically, the hierarchical attention mechanism uses a class activation map to construct the attention map, which can locate the region most relevant to leaf recognition. Furthermore, a trainable Gaussian kernel function is used to transform the original attention map, enhancing key regions while suppressing background regions.

[0007] The technical solution adopted in this invention is as follows:

[0008] A leaf recognition method based on a hierarchical attention mechanism includes the following steps:

[0009] 1) Generate activation maps of different levels of the input image;

[0010] 2) Merge activation maps of different levels into an attention map, and use this attention map to obtain several different sampled images;

[0011] 3) Concatenate the feature vectors of each sampled image with the feature vector of the input image, and obtain the image leaf recognition result based on the obtained concatenated feature vector.

[0012] Furthermore, activation maps of different hierarchical categories are obtained through the following steps:

[0013] 1) Input the input image into the convolutional neural network ResNet-50 to obtain the first stage output features of size 8n*8n, the second stage output features of size 4n*4n, the third stage output features of size 2n*2n, and the fourth stage output features of size n*n, where n≥1;

[0014] 2) Using the output features of the second stage, the third stage, and the fourth stage, generate class activation maps of size 4n*4n, 2n*2n, and n*n, respectively.

[0015] Furthermore, the attention map is synthesized through the following steps:

[0016] 1) Using bilinear interpolation, class activation maps of size 2n*2n and n*n are enlarged to size 4n*4n respectively;

[0017] 2) The two magnified results are weighted and fused with a category activation map of size 4n*4n to obtain an attention map.

[0018] Furthermore, several different sampled images are obtained through the following steps:

[0019] 1) Apply a sliding window to the attention map to obtain local maxima in different windows, which can be used as candidate extreme points;

[0020] 2) Group these candidate extreme points using a set threshold;

[0021] 3) Smooth each group of candidate points using a learnable Gaussian kernel to obtain several smoothed sampling images;

[0022] 4) Selectively sample the input image according to the sampling map to obtain the corresponding sampled image.

[0023] Furthermore, based on formula Td ={(x, y)|(x, y)∈T if A x，y >δ} and formula T c ={(x, y)|(x, y)∈T if A x，y >δ} group these candidate points, where (x,y) are the coordinates of the candidate extreme points, A x，y Let be the value of the candidate extreme point (x, y) on the attention map, δ be the set threshold, and T be... d For the discriminative group, T c This is a supplementary group.

[0024] Furthermore, the smooth attention map A corresponding to the candidate extreme point (x,y) i，x，y for:

[0025]

[0026] Where β is the parameter of the Gaussian kernel, (x,y) are the coordinates of the candidate extremum point, and A x，y The value of the candidate extreme point (x,y) on the attention map.

[0027] Furthermore, the feature vectors of each sampled image and the feature vector of the input image are obtained through the following steps:

[0028] 1) Use the conjoined convolutional neural network S-CNN to extract features from each sampled image and the input image respectively;

[0029] 2) Based on the features of each sampled image and the input image, obtain the feature vectors of each sampled image and the feature vector of the input image.

[0030] Furthermore, the conjoined convolutional neural network S-CNN is trained end-to-end, with the cross-entropy loss function as the loss function.

[0031] Further, the image leaf recognition results are obtained through the following steps:

[0032] 1) Input the concatenated feature vectors into a fully connected layer to obtain the probability distribution of each category;

[0033] 2) Use the softmax function to normalize each probability distribution;

[0034] 3) The category corresponding to the result after maximum normalization is used as the image leaf recognition result.

[0035] Furthermore, by improving the accuracy of identification As an evaluation metric, n represents the number of correctly identified samples, and N represents the total number of test samples.

[0036] A storage medium storing a computer program, wherein the computer program is configured to execute the method described above at runtime.

[0037] An electronic device includes a memory and a processor, wherein the memory stores a program for performing the methods described above.

[0038] Compared with existing technologies, the advantages of this invention are:

[0039] 1. A hierarchical attention mechanism is proposed, which aligns with the classification strategies of human botanists.

[0040] 2. By combining sampling transformation, the key areas in the image are magnified, which is beneficial for leaf identification.

[0041] 3. By combining deep learning methods, a high accuracy rate can be achieved in leaf recognition. Attached Figure Description

[0042] Figure 1 This is a network framework diagram for image leaf recognition based on a hierarchical attention mechanism. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0044] The process of the deep learning-based shadow detection method in natural scenes of this invention is as follows: Figure 1 As shown, it is mainly divided into two stages: the training stage and the testing stage.

[0045] The steps in the training phase are as follows:

[0046] 1) The training images are processed through a convolutional neural network, and multiple attention maps are obtained through a hierarchical attention mechanism for use in the subsequent sampling process;

[0047] The processing steps in step 1) are as follows: First, the training images are processed through a convolutional neural network to obtain multi-level features of the training images. Utilizing the hierarchical structure of the convolutional neural network, multiple classifiers are constructed to generate activation maps of different levels. In the ResNet-50 convolutional neural network used in this invention, for a leaf image with an input size of 224*224, the output feature size of the first stage of ResNet-50 is 56*56, the second stage is reduced to 28*28, the third stage is 14*14, and the output feature size of the last stage is 7*7. The resulting three activation maps are 28*28, 14*14, and 7*7, respectively.

[0048] Since shallow features are not robust and are not suitable for image recognition, the hierarchical attention mechanism does not use the image features of the first stage. Instead, it utilizes the feature outputs of the second, third, and fourth stages of the ResNet-50 convolutional neural network to train three classifiers with different granularities. Using the classifier weights and features of the corresponding stages, three class activation maps with different granularities are obtained.

[0049] This step is inspired by the classification strategies of human botanists, fully simulating their classification process. The botanist's classification strategy involves three steps: first, observing the leaf's external outline; second, observing the leaf's shape and color; and finally, observing the leaf's texture. The hierarchical attention mechanism proposed in this invention focuses on the edge and outline information of the input leaf image in the shallow layers of the network, on the shape and color information in the middle layers, and on the texture information in the high layers, aligning with the strategy of human plant leaf classification. Compared to the black-box model of traditional convolutional neural networks, this invention is more consistent with human cognition and has greater interpretability.

[0050] 2) Based on the category activation map obtained in step 1), use bilinear interpolation to merge it into an attention map of the same size. Use this attention map to obtain two different sampled images through a sampling transformation network to increase the diversity of the images.

[0051] Step 2) enlarges the 7*7 and 14*14 class activation maps to 28*28 size using bilinear interpolation, and then weights and fuses them with the class activation map output from the second stage of the network to obtain a 28*28 attention map. Next, a sliding window method is used to obtain local maxima on the attention map, identifying candidate extrema. These candidate extrema indicate that the region has a high response value and is more likely to be a key region for classification. Subsequently, these candidate extrema are grouped using a set threshold to obtain more diverse image features. After grouping, each group of candidate points is smoothed using a learnable Gaussian kernel, resulting in two smoothed sampled images. Based on these sampled images, selective sampling is performed on the original image, enlarging important foreground regions and reducing unimportant background and classification-irrelevant regions, resulting in two sampled images. Specifically:

[0052] Construct a sampling map Q using a smooth attention map. d and Q c Q d and Q c The attention regions that converge on the two sets of candidate extrema produce a summation effect, which allows each sampling map to focus on multiple regions that are helpful for classification, without being limited by a manually defined number:

[0053]

[0054]

[0055] The image sampling formula is X new =X(f(v), g(v)), where f and g are sampling functions, indicating the position from which to sample in the original image, and the sampling coordinate functions f and g satisfy:

[0056]

[0057] Its solution satisfies:

[0058]

[0059]

[0060] Where k is the Gaussian kernel that weights different coordinate positions. For coordinate values ​​far from the center point, the weight is smaller, and for values ​​close to the center point, the weight is larger. This allows for more sampling of the image region near the candidate extreme point and less sampling of the image region near the candidate extreme point, thereby amplifying the effect of achieving the key foreground region and suppressing the background and irrelevant regions.

[0061] In acquiring sampled image I d and I c Then, it is fed into two branches of the S-CNN to extract features and obtain the feature vector v through global mean pooling. d and v c Then, it is concatenated with the feature vector v of the original input image and fed into the classifier to obtain the final classification result y.

[0062] 3) The sampled images obtained in step 2) are fed into the discriminative group of the conjoined convolutional neural network to extract features for final recognition.

[0063] Step 3) first extracts features from the two sampled images and obtains their respective feature vectors through global mean pooling. These feature vectors are then concatenated with the feature vectors from the original image to obtain the feature vector used for classification. A fully connected layer is then used to obtain the probability distribution of each category, and finally, the softmax function is used to normalize it. The category with the largest output value is then selected as the image recognition result. Figure 1In ResNet-50, conv_1-1, conv1_2, conv1_3, conv1_4, conv_2-1, conv2_2, conv2_3, and conv2_4 represent different stages of the convolutional layers. GAP refers to Global Average Pooling, which transforms feature tensors into feature vectors, facilitating input into a linear classifier. The two networks share parameters, similar to S-CNN, ensuring identical parameters for both. The networks are trained end-to-end, using cross-entropy loss as the objective function.

[0064] The steps in the testing phase are as follows:

[0065] 1) Input the test image into the trained network, and the output category is the network recognition result.

[0066] 2) The test results are judged using the accuracy (acc).

[0067] The leaf recognition network of this invention is further explained below. The image leaf recognition method based on a hierarchical attention mechanism proposed in this invention is mainly derived from an improvement on S-CNN. The hierarchical attention mechanism utilizes the inherent hierarchical structure of convolutional neural networks, first calculating the category activation map using features from different levels:

[0068]

[0069] Where CAM stands for Class Activation Map, i represents the i-th level, f is the image feature at that level, w is the classification weight of the correct class in the corresponding classifier, c represents the c-th channel of the image feature, and C is the total number of channels. By weighting the correct class according to its classification weight, an attention map for that correct class can be constructed to show the importance of different regions in the image for classification.

[0070] Then, the multi-level category activation maps are weighted and fused to obtain an attention map that can represent the importance of image regions:

[0071]

[0072] Where α i The fusion weights for different levels are determined manually. A represents the final attention map. This invention uses three levels of category activation maps.

[0073] For the sampling network, a Gaussian kernel was used for smoothing, and the smoothing method was as follows:

[0074]

[0075] Where β is the parameter of the Gaussian kernel, obtained through network training. Ai，x，y This is a smoothed attention map corresponding to the candidate extremum point at (x,y). The smoothing effect of the Gaussian kernel both amplifies the image of the candidate extremum point region and suppresses interference from background and irrelevant information.

[0076] The sampled image obtained after sampling is X new =X(f(v), g(v)), where f and g are sampling functions, indicating the position from which to sample in the original image, and the sampling coordinate functions f and g satisfy:

[0077]

[0078] Solving this equation will yield two different sampled images.

[0079] Furthermore, features are extracted from the sampled images using S-CNN, and global mean pooling is used to pool the image features into feature vectors. Then, the feature vectors from the two sampled images are concatenated with the feature vector from the original image to obtain aggregated image features, which are then fed into a linear classifier to obtain the classification result. Specifically, the linear classifier in this invention is implemented with a single fully connected layer, and a softmax transformation is used to ensure that its output follows a probability distribution. The dimension of the final output probability vector is equal to the number of leaf categories. The network's recognition result is determined by the dimension of the vector containing the highest probability value.

[0080] The network is trained end-to-end, using cross-entropy loss. The total loss function includes the classifier loss functions for the three different stages of the hierarchical attention mechanism, the classifier loss functions for the two branches, and the classifier loss function for aggregating image features. The total loss function is:

[0081]

[0082] Where j represents the index of the j-th classifier, and there are a total of 6 classifiers.

[0083] The image leaf recognition method based on hierarchical attention mechanism proposed in this invention has the following test environment and experimental results:

[0084] (1) Test environment:

[0085] System environment: Ubuntu 16.04;

[0086] Hardware environment: Memory: 16GB, GPU: TITIA NXP, Hard drive: 1TB;

[0087] (2) Experimental data:

[0088] Training data:

[0089] Using the Leaf-Snap dataset, which includes 3300 training images representing 180 different leaf classes, the model was trained until the objective function converged and the model performance reached a stable value.

[0090] Data preprocessing: Both training and testing leaf images were resized to 224x224 using bilinear interpolation. During training, random cropping and random flipping were used for data augmentation.

[0091] Training optimization method: ADAM, with an initial learning rate of 0.01, which is reduced to 0.96 times the original learning rate after every 4 epochs of training.

[0092] Test data: Leaf-Snap test set (6600 images)

[0093] Evaluation method: Calculate the recognition accuracy on the test set.

[0094] (3) Experimental results:

[0095] To illustrate the effectiveness of this invention, the same dataset was used to test whether or not the hierarchical attention mechanism proposed in this invention was used, and the Leaf-Snap test set was used.

[0096] The test results of the present invention are shown in Table 1 below:

[0097] Table 1. Comparison of test results between existing methods and the present invention

[0098] 1 This invention (does not use a hierarchical attention mechanism) 0.964 2 This invention (using a hierarchical attention mechanism) 0.993

[0099] Here, Acc refers to accuracy. As can be clearly seen from the table, the leaf recognition method based on a hierarchical attention mechanism involved in this invention shows a significant improvement in results and exhibits a high overall performance.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A leaf recognition method for images based on a hierarchical attention mechanism, comprising the following steps: 1) Generate activation maps of different levels of the input image; wherein, the activation maps of different levels of the input image are obtained through the following steps: The input image is fed into a ResNet-50 convolutional neural network, and the resulting images are of size [size missing]. The first stage output features and their size are The second-stage output feature has a size of The third stage output features and size are The fourth stage output features, where ; Using the output features of the second stage, the third stage, and the fourth stage, respectively generate output features of size . , and Category activation graph; 2) Merge activation maps of different levels into an attention map, and use this attention map to obtain several different sampled images; the attention map is synthesized through the following steps: Using bilinear interpolation, the size is... and The category activation maps are magnified to respectively size; Compare the two magnified results with a size of The category activation maps are weighted and fused to obtain the attention map; 3) Concatenate the feature vectors of each sampled image with the feature vector of the input image, and obtain the image leaf recognition result based on the obtained concatenated feature vector.

2. The method as described in claim 1, characterized in that, Several different sampled images are obtained through the following steps: 1) Apply a sliding window to the attention map to obtain local maxima in different windows, which can be used as candidate extreme points; 2) Group these candidate extreme points using a set threshold; 3) Smooth each group of candidate extrema points using a learnable Gaussian kernel to obtain several smoothed sampling maps; among them, the smoothed attention map corresponding to the candidate extrema point (x,y) is... , Let (x, y) be the parameters of the Gaussian kernel, and (x, y) be the coordinates of the candidate extrema. Let (x, y) be the value of the candidate extreme point (x, y) on the attention map. For the discriminative group, For supplementary group; 4) Selectively sample the input image according to the sampling map to obtain the corresponding sampled image.

3. The method as described in claim 1, characterized in that, The feature vectors of each sampled image and the feature vector of the input image are obtained through the following steps: 1) Use the conjoined convolutional neural network S-CNN to extract features from each sampled image and the input image respectively; 2) Based on the features of each sampled image and the input image, obtain the feature vectors of each sampled image and the feature vector of the input image.

4. The method as described in claim 3, characterized in that, The conjoined convolutional neural network S-CNN is trained end-to-end, and the loss function is the cross-entropy loss function.

5. The method as described in claim 1, characterized in that, The following steps are used to obtain the image leaf recognition results: 1) Input the concatenated feature vectors into a fully connected layer to obtain the probability distribution of each category; 2) Use the softmax function to normalize each probability distribution; 3) The category corresponding to the result after maximum normalization is used as the image leaf recognition result.

6. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the method as claimed in any one of claims 1-5.

Images