Intelligent identification method and device for plant leaves and pest categories in agricultural field

Abstract

The application discloses a kind of classification method and device for different plant leaves and different pests in agricultural field, method includes: obtaining data set and pre-processing;Data set is carried out data enhancement;The Vision Transformer backbone network based on self-attention mechanism is constructed, and the features of image are extracted;The most discriminative area corresponding token in the coding features of backbone network is selected by saliency feature screening engine for classification;The overall structure of object in image is extracted by visual structure perceiver to selected token to assist classification;The feature combination mode of random partial exchange is enriched to the features generated by saliency feature screening engine and visual structure perceiver, and the features of two parts are fused using learnable parameters for network training, and the trained network is used to classify different plant leaves and different pests.The application can automatically classify different plant leaves and pest images with high precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an intelligent identification method and device for plant leaves and pest categories in the agricultural field. Background Art

[0002] In modern agricultural production, accurately identifying and classifying different types of plant leaves and pests on them is crucial, as it directly impacts the effectiveness of pest control and the healthy growth of crops. It is crucial for precision agriculture, pest management, and crop health monitoring. Traditionally, this task relies primarily on the experience of agricultural experts and plant protection personnel, who visually identify leaf lesions and pest types through direct observation or the use of simple magnifying devices. However, this approach suffers from several issues: 1. Inefficiency: Manual identification is time-consuming and labor-intensive, making it difficult to adapt to the needs of large-scale farmland. 2. Limited accuracy: Identification accuracy is highly dependent on individual experience, and results can vary between operators. 3. Poor timeliness: Manual identification is slow, preventing real-time monitoring and rapid response, and can easily miss optimal control opportunities. Therefore, the use of artificial intelligence (AI) to assist in the classification of plant leaf and pest images is crucial. This not only improves classification efficiency but also accuracy. Furthermore, computer systems can process image data in real time to promptly detect and report potential disease and pest problems, facilitating timely preventive measures.

[0003] In modern agriculture, research on fine-grained image classification of plant leaves and pests is gaining momentum. In recent years, several studies have demonstrated the effectiveness of deep learning techniques for this problem. Xu et al., "Deep Learning-Based Image Recognition of Agricultural Pests," developed an object detection framework based on a cascaded RCNN, employing techniques such as data augmentation and sliding window cropping to improve the model's ability to recognize a variety of pest types and image sizes. This approach has shown some success in improving pest classification accuracy. WasswaShafik et al., "Using transfer learning-based plant disease classification and detection for sustainable agriculture," used a pretrained VGG model with two Inception layers to identify and classify agricultural diseases such as corn and rice. This combined transfer learning approach resulted in a model with high accuracy in disease recognition across different crops (80.38% for corn and 92% for rice). Although these deep learning-based classification methods are faster and more accurate than manual classification, most of them directly apply existing neural network models to the automatic classification of plant leaves and pests, and do not consider the optimization of the model based on the characteristics of these two types of images in actual scenes. Therefore, there are certain limitations and room for improvement.

[0004] In practical production, automatic classification of different plant leaves and pests faces multiple challenges. The first is the difficulty of large intra-class variance. In the agricultural sector, even images of the same plant or pest species can exhibit significant differences at different growth stages, under different lighting conditions, or due to varying environmental factors. For example, the color and shape of leaves can vary depending on water content, lighting, or nutritional status; and the size, morphology, and color of the same pest species can also differ at different stages of growth. These factors increase intra-class variance, complicating image classification. The second challenge is small inter-class variance. Many pests and plant leaves are visually highly similar, making it difficult to distinguish between them. For example, different types of leaf spot may differ only slightly in the shape or color of the spot's edge. This requires the model to be able to recognize and accurately classify these subtle differences. Finally, classification is hindered by complex background interference. Agricultural image backgrounds are often complex and varied, including other plants, soil, and shadows, all of which can interfere with the recognition of plant leaves and pests. For example, other plants or weeds behind a leaf may have a similar color to the target leaf, or pests may hide on the underside of the leaf, greatly increasing the difficulty of the classification task. Based on a comprehensive analysis of these difficulties, this embodiment innovatively proposes a fine-grained image classification model that simultaneously focuses on both fine-grained discriminative features and the overall structural features of the object in the image. This model not only highlights the foreground area of ​​the image to be classified, but also focuses on the most important foreground areas that are most helpful for classification, achieving high-precision image classification results. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for intelligent identification of plant leaves and pest categories in the agricultural field, while focusing on a fine-grained image classification model that pays attention to both fine discriminative features and overall structural features of the object in the image. While highlighting the foreground area of ​​the image to be classified, it also focuses on the important areas in the foreground that are most helpful for classification, thereby achieving high-precision image classification effects.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for classifying different plant leaves and different pests in the agricultural field, comprising the following steps:

[0008] Obtain plant leaf datasets and agricultural pest datasets and perform preprocessing;

[0009] Perform data augmentation on the dataset;

[0010] A Vision Transformer backbone network based on the self-attention mechanism is constructed, and the enhanced dataset is input into the Vision Transformer backbone network to extract image features. The Vision Transformer backbone network includes a Patch Embedding layer and multiple Transformer Encoder layers. Each Transformer Encoder layer includes a multi-head self-attention layer (MHA) and a multi-layer perceptron (MLP). The MHA uses multiple attention heads to process the input in parallel to capture information from different subspaces. The MLP consists of linear layers and nonlinear activation functions to further process the output of the attention mechanism. In the Patch Embedding layer, each image patch is encoded into a D-dimensional token. In all subsequent Transformer Encoder layers, each token maintains this dimension. After the Encoder is encoded, the additional D-dimensional classification token is used as a vector representing the features of the entire image for classification.

[0011] The salient feature filtering engine is used to select the tokens corresponding to the most discriminative regions in the backbone network coding features for classification;

[0012] The visual structure perceptron performs visual structure perception on the selected tokens to extract the overall structure of the object in the image to assist classification;

[0013] The features generated by the salient feature screening engine and the visual structure perceptron are randomly partially exchanged to enrich the feature combination pattern, and the features of the two parts are fused with learnable parameters for model training. The trained model is used to classify different plant leaves and different pests.

[0014] As a preferred technical solution, the plant leaf dataset and the agricultural pest dataset are obtained and preprocessed, specifically:

[0015] Collect plant leaf datasets and agricultural pest datasets;

[0016] Filter the dataset, remove poor quality images and corresponding labels in the dataset, and ensure that the number of images of each category in the dataset is uniform and the images are complete;

[0017] The images in the filtered dataset are uniformly processed into the set size.

[0018] As a preferred technical solution, the data enhancement of the dataset is specifically as follows:

[0019] The images in the dataset are randomly cropped, horizontally flipped, vertically flipped, brightness and contrast are randomly adjusted, and the images are randomly erased, and the data is normalized at the end.

[0020] As a preferred technical solution, the Vision Transformer backbone network based on the self-attention mechanism is constructed, and the enhanced data set is input into the Vision Transformer backbone network to extract image features, specifically:

[0021] First, a linear projection layer is used to project the K blocks divided into the image in the dataset into a D-dimensional potential embedding τ k , and add positional encoding to the generated tokens, and add an additional classification token for classification;

[0022] Then the pre-trained ViT-B_16 backbone network is used as the feature extractor to extract the features of the image;

[0023] Take the first token of the feature encoded by the last layer of the backbone network, which is also the final classification token τ cls Used for classification.

[0024] As a preferred technical solution, the tokens corresponding to the most discriminative regions in the backbone network coding features are selected for classification by the significant feature screening engine, specifically:

[0025] Get the attention weight between any two tokens in each layer of the backbone network;

[0026] At layer l, the cumulative dot product of the attention weights from layer 1 to layer l is performed to obtain the cumulative attention weight of all tokens. The size of this attention weight is directly used to calculate the first part of the significance score of the i-th token, thereby obtaining the first part of the summary of the scores of all tokens in layer l;

[0027] The second part of calculating the score of the i-th token: Initialize a sum A uniformly distributed vector U with the same dimension is then measured using the Jensen-Shannon divergence The similarity with the U data distribution; the second part of the significance score of each token is obtained by combining the corresponding Jensen-Shannon divergences of all tokens in the lth layer;

[0028] The two parts of the significance score of each token are added together and normalized to get the final score of each token;

[0029] Finally, the final score of each token is sorted in descending order, and a portion of the tokens with the highest significance scores are selected from each layer and aggregated with the classification tokens of the backbone layer through a Transformer encoding layer to obtain the classification token τ of the saliency feature screening engine. cls_a .

[0030] As a preferred technical solution, the visual structure perception of the selected tokens is performed by a visual structure perceiver to extract the overall structure of the object in the image to assist in classification, specifically:

[0031] Use the average score of each token attention weight vector factor The normalized information entropy H corresponding to the attention weight vector and Score2 are used as the score index. Based on the score index, the tokens encoded by the last layer in the network backbone are sorted, and then the token with the largest score is selected as the anchor point, and all tokens with scores greater than the average are input into a two-layer graph convolutional neural network to establish the structural relationship between these tokens, thereby obtaining the classification token τ containing the visual structural features of the object cls_b .

[0032] As a preferred technical solution, the training process is specifically as follows:

[0033] Two classification tokens τ are obtained through the salient feature screening engine and the visual structure perceptron cls_a and τ cls_b Afterwards, the two tokens are randomly partially exchanged within a batch to obtain the replaced classification tokens Y1 and Y2;

[0034] Then use the fully connected layer to calculate the two parts of the logistic regression vector l1 and l2 respectively;

[0035] In order to improve the feature interaction of these two modules, two learnable weight parameters w1 and w2 are defined, and the weighted regression vector l = w1l1 + w2l2 is calculated. This weighted vector is used as the final image feature to calculate the cross entropy loss.

[0036] In order to learn the subtle differences between subcategories, the contrast loss is also calculated for the salient feature vector Y1 By maximizing the intra-class similarity and minimizing the inter-class similarity, the discriminability of features is improved, allowing the model to learn more fine-grained differences in the image;

[0037] The model is trained using a combination of contrastive loss and cross entropy loss.

[0038] In a second aspect, the present invention provides a classification system for different plant leaves and different pests in the agricultural field, which is applied to the classification method for different plant leaves and different pests in the agricultural field, including a data acquisition module, a data enhancement module, a backbone network construction module, a first classification module, a second classification module and a feature fusion module;

[0039] The data acquisition module is used to acquire plant leaf datasets and agricultural pest datasets and perform preprocessing;

[0040] The data enhancement module is used to perform data enhancement on the data set;

[0041] The backbone network building module is used to build a Vision Transformer backbone network based on the self-attention mechanism, and input the enhanced data set into the Vision Transformer backbone network to extract image features. The Vision Transformer backbone network includes a Patch Embedding layer and multiple Transformer Encoder layers. Each Transformer Encoder layer includes a multi-head self-attention layer (MHA) and a multi-layer perceptron (MLP). The MHA uses multiple attention heads to process input in parallel to capture information in different subspaces. The MLP consists of linear layers and nonlinear activation functions to further process the output of the attention mechanism. In the Patch Embedding layer, each image block is encoded into a D-dimensional token. In all subsequent Transformer Encoder layers, each token maintains this dimension. After the Encoder encoding is completed, the additional D-dimensional classification token is used as a vector representing the features of the entire image for classification.

[0042] The first classification module is configured to select tokens corresponding to the most discriminative regions of the backbone network coding features for classification through a significant feature screening engine;

[0043] The second classification module is configured to perform visual structure perception on the selected tokens through a visual structure perceiver to extract the overall structure of the object in the image to assist in classification;

[0044] The feature fusion module is used to randomly exchange parts of the features generated by the significant feature screening engine and the visual structure perception to enrich the feature combination pattern, and use learnable parameters to fuse the features of the two parts for network training, and use the trained network to classify different plant leaves and different pests.

[0045] In a third aspect, the present invention provides an electronic device, comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the classification method for different plant leaves and different pests in the agricultural field.

[0049] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the classification method for different plant leaves and different pests in the agricultural field.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] (1) The present invention is based on the ViT-B_16 model and can achieve a high classification accuracy without using all the feature extraction layers of the benchmark model. The number of parameters and the amount of calculation required by the network are relatively small, and the demand for computing resources is not high.

[0052] (2) The present invention proposes a saliency feature screening engine, which comprehensively considers the saliency of tokens by the local correlation and global distribution of each token's attention weight vector, and accurately screens out the tokens that are most helpful for the classification process, thereby improving the classification accuracy.

[0053] (3) The present invention proposes a visual structure sensor, which selects tokens with higher information content by measuring the information entropy of tokens to perceive their structural information, thereby promoting the model's learning of the global structure of the image.

[0054] (4) The significant feature screening engine proposed in the present invention can be embedded into most neural networks as a plug-and-play module and has very good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 This is a flow chart of a method for classifying different plant leaves and different pests in the agricultural field according to an embodiment of the present invention;

[0057] Figure 2This is a diagram of the overall network structure of an embodiment of the present invention;

[0058] Figure 3 This is a structural diagram of the Vision Transformer-B_16 used in an embodiment of the present invention;

[0059] Figure 4 This is a diagram showing the internal structure of each encoder block in Vision Transformer-B_16 according to an embodiment of the present invention.

[0060] Figure 5 This is a diagram of the internal structure of a significant feature screening engine according to an embodiment of the present invention;

[0061] Figure 6 This is a diagram showing the internal structure of a visual structure sensor according to an embodiment of the present invention;

[0062] Figure 7 is a comparison chart of the results of the method of this embodiment and other models;

[0063] Figure 8 1 is a schematic diagram of a classification system for different plant leaves and different pests in the agricultural field according to an embodiment of the present invention;

[0064] Figure 9 It is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0067] like Figure 1 、 Figure 2 As shown, a classification method for different plant leaves and different pests in the agricultural field according to this embodiment includes the following steps:

[0068] S1. Obtain appropriate plant leaf datasets and agricultural pest datasets and perform preprocessing to convert the images into appropriate formats;

[0069] In a specific example, to verify the effectiveness of the present invention, two datasets of significantly different sizes were prepared through literature review and web searches for training and validation. Cotton80 is a publicly available, ultra-fine-grained image dataset containing 80 categories of cotton leaf images. Each variety has only six samples, divided into three training images and three test images. This dataset is annotated with only image-level labels. The images in the dataset are highly similar, and the images in this dataset are uniformly resized to 384×384×3, where 3 represents the number of channels in the image. IP102 is a large-scale pest identification dataset that includes over 75,000 images from 102 categories. These images exhibit a natural long-tail distribution and have a hierarchical classification structure, with pests that primarily affect specific agricultural products being grouped into the same category at the previous level. The images in this dataset are uniformly resized to 224×224×3.

[0070] S2, perform data enhancement on the dataset;

[0071] In a specific embodiment, data augmentation involves random cropping, horizontal flipping, vertical inversion, and random adjustments to brightness and contrast, all with a probability of 0.5. The image is then normalized using the variance and mean calculated from the ImageNet dataset: variance σ = [0.229, 0.224, 0.225], and mean μ = [0.485, 0.456, 0.406]. Finally, random erasing is performed using Random Erasing, as proposed in the paper "Random Erasing Data Augmentation." The image is erased with a probability of 0.5, a scale range of 0.02 to 0.33, an aspect ratio range of 0.3 to 3.3, and a fill value of 0 in the erased area. No modifications are made to the original image.

[0072] S3. Build a Vision Transformer backbone network based on the self-attention mechanism and input the enhanced dataset into the Vision Transformer backbone network to extract image features;

[0073] In this embodiment, ViT-B_16 proposed in the article "An Image is Worth 16×16Words: Transformer for Image Recognition at Scale" by Alexey Dosovitskiy et al. is used as the feature extractor. The network structure of ViT-B_16 is shown in the figure below. Figure 3 As shown, the core consists of a Patch Embedding layer and 12 Transformer Encoder layers. The specific structure of these Transformer Encoder layers is as follows Figure 4 As shown, each encoder layer primarily consists of a Multi-Head Attention (MHA) layer and a Multi-Layer Perceptron (MLP). The former utilizes multiple attention heads to process inputs in parallel, capturing information from different subspaces. The latter, consisting of a linear layer and a nonlinear activation function (GELU), further processes the output of the attention mechanism. In the Patch Embedding layer, each image patch is encoded into a D-dimensional token. In all subsequent encoder layers, each token maintains this dimension. After the encoder encoding is complete, the attached D-dimensional classification token can be used as a vector representing the features of the entire image for classification.

[0074] Furthermore, step S3 is specifically as follows:

[0075] S31. First, use a linear projection layer to project the K blocks divided into the image into a D-dimensional potential embedding τ k , and add position encoding to the generated tokens, and then add an additional classification token for classification. The process can be expressed as:

[0076] z0=[τ cls ;τ1E;τ2E;…;τ K E]+E pos (1)

[0077] Among them E pos represents the position encoding, τ cls Represents an additional classification token.

[0078] S32, then used the ViT-B_16 backbone pre-trained on the ImageNet-21K large dataset as the feature extractor. The process can be expressed as follows:

[0079] z′ l =MHA(LN(z l-1 ))+z l-1 (2)

[0080] z l =MLP(LN(z′ l ))+z′ l (3)

[0081] where LN(·) is the layer normalization operation, z l It is the image representation after encoding. The number of backbone layers is uniformly defined as L. MHA is a multi-head self-attention layer, which is used to learn multiple feature representations of input data in parallel. MLP is a multi-layer perceptron, which enhances the nonlinear processing capability of features through fully connected layers. The first token of the feature encoded by the last layer of the backbone is the final classification token τ cls It can be used for classification. It can be expressed by the following formula:

[0082] logits=FC(τ cls ) (4)

[0083] Among them, FC represents a fully connected layer, and logits represents the probability vector output by the model, that is, the prediction result of the classification.

[0084] S4. The salient feature screening engine is used to select the tokens corresponding to the most discriminative regions in the backbone network coding features for classification.

[0085] It is understandable that in fine-grained image classification tasks, since the overall differences between different categories are relatively small, it is necessary to focus on some detailed parts of the object area in the image to guide classification. VisionTransformer mainly focuses on the global features of the image, which may cause the network to pay insufficient attention to some subtle differences in the image. To solve this problem, this embodiment proposes a significant feature screening engine to select the feature subset that is most likely to be related to the detailed discrimination area of ​​the classified object in each layer from the multi-layer feature representation of ViT, and aggregate them to obtain a more detailed and rich feature expression.

[0086] The specific structure of the significant feature screening engine is as follows: Figure 5 As shown, the average score of the attention weight vector of each token is obtained factor and Jensen-Shannon divergence Score between the attention weight vector and the uniform distribution U JSD The normalized sum of the tokens is taken as the significance score Score1 of the token, thereby achieving a ranking of all tokens in each layer based on significance. Then, a specific number of tokens are selected from each layer for information aggregation, thereby obtaining a classification token τ containing rich and detailed features. cls_a .

[0087] Furthermore, step S4 is specifically as follows:

[0088] S41. In step S3, the attention weight between any two tokens in each layer of the backbone network layer is obtained, which can be expressed as:

[0089]

[0090] in is the attention weight between token i and token j in layer l, and K is the total number of tokens.

[0091] S42. At layer l, perform cumulative dot multiplication on the attention weights from layer 1 to layer l to obtain the cumulative attention weights of all tokens:

[0092]

[0093] in Represents the attention weight of token i to token j accumulated from layer 1 to layer l. First, the first part of the significance score of the i-th token is calculated directly using the size of this attention weight:

[0094]

[0095] This gives the first part of the summary of all token scores at layer l:

[0096] Score factor =[Sf0,Sf1,…,Sf i ,…,Sf K ] (9)

[0097] S43. The second part of calculating the significance score of the i-th token: Initialize a sum A uniformly distributed vector U with the same dimension is then measured using the Jensen-Shannon divergence Similarity to U data distribution:

[0098]

[0099] Combining the corresponding Jensen-Shannon divergences of all tokens in layer l yields:

[0100] Score JSD =[Sj0,Sj1,…,Sj i ,…,Sj K ] (11)

[0101] The two parts of the saliency score for each token are added together and normalized to get the final score for each token:

[0102] Score1=Score factor_norm +ScoreJSD_norm (12)

[0103] By sorting the scores in descending order, we can select a portion of tokens with the highest significance scores from each layer and aggregate them with the classification tokens of the backbone layer through a Transformer encoding layer to obtain the classification token τ of this module. cls_a .

[0104] S5, perform visual structure perception on the selected tokens to extract the overall structure of the object in the image to assist classification,

[0105] In order to reasonably extract the visual structure information of the object in the image and thus supplement and guide the detailed features of the object extracted by the salient feature screening engine, this embodiment designs a visual structure sensor. The specific structure of this visual structure sensor is as follows: Figure 6 As shown. Since the lower-level attention heads in Vision Transformer may pay more attention to local information, and the higher-level attention heads pay more attention to global and semantic information, this embodiment only uses this module in the last layer of the backbone. This embodiment establishes structural information in the image by considering the correlation between specific tokens in the image, so the goal of this embodiment is to select tokens with strong deterministic relationships. The information entropy between the attention weight vector of each token and the uniform distribution can reflect the certainty of the attention distribution between it and other tokens to a certain extent. The smaller the entropy value, the more concentrated the attention weight between the token and the other tokens. To a certain extent, it can be reflected that the semantic information correlation between them is relatively large, which also means that it is more appropriate to perceive structural information based on these tokens. Therefore, this embodiment uses the average value of the attention weight vector of each token Score factor The normalized information entropy H corresponding to the attention weight vector and Score2 are used as the score indicator. This embodiment sorts the tokens encoded by the last layer in the network backbone based on this score indicator, and then selects the token with the largest score as the anchor point, and all tokens with scores greater than the average are input into a two-layer graph convolutional neural network to establish the structural relationship between these tokens, thereby obtaining a classification token τ containing the visual structural features of the object. cls_b .

[0106] Furthermore, step S5 is specifically as follows:

[0107] S51. First, calculate the cumulative attention weight of each token in the last layer of the backbone network according to the cumulative attention weight calculation process described in step S4. Then use the information entropy of the attention weight vector of each token to measure the certainty of the relationship between the token and other tokens:

[0108]

[0109] in It represents the cumulative attention weight of token i to token j in the last layer of the backbone network. The cumulative attention weight and information entropy value of each token are summed and normalized to obtain the score vector Score2.

[0110] S52, filter out the token with the largest score in step S51 as the anchor point, and all tokens with scores greater than the average value to model the inter-token structural information. First, reshape the score vector Score2 into a Score matrix of size 2_new . Use polar coordinates to represent the spatial relationship between the anchor token and the image blocks corresponding to other selected tokens to mine the structural information between tokens. Given the image block corresponding to the anchor token and the horizontal reference direction, where (x0,y0) is the image The corresponding block position index in the block plane of size. Then the block P corresponding to the selected token i i =P (x,y) The polar coordinates of are defined as:

[0111]

[0112] θ x,y =(atan2(y-y0,x-x0)+π) / 2π (15)

[0113] where 0<Γ x,y ≤1 is used to measure the anchor point P0 and the corresponding block P of token i i The relative distance between x,y It's P i Normalized polar angle relative to the horizontal.

[0114] Use a two-layer graph convolutional neural network to establish the structural relationship between the selected tokens. First, use the previously generated score matrix to generate the adjacency matrix Adj = Score 2_new ·(Score 2_new ) T The image block features X and the adjacency matrix generated by the last layer of the backbone network are processed through two layers of graph convolution to obtain the visual structure features:

[0115] G=σ(Adj×σ(Adj×X×W 1 )×W 2 ) (16)

[0116] Where W 1 and W 2are two learnable parameters, and σ(·) is the activation function. The generated anchor block node feature G contains the object structure information in the image. This embodiment adds it to the classification token of the backbone network to obtain the classification token τ containing the structure information. cls_b .

[0117] S6. Randomly exchange parts of the features generated by the salient feature screening engine and the visual structure perception to enrich the feature combination pattern, and use learnable parameters to fuse the features of the two parts for network training. The trained network is used to classify different plant leaves and different pests.

[0118] like Figure 2 As shown, the entire network of this application includes two classifiers, one for obtaining the salient feature classification vector l1 and the other for obtaining the visual structure perception classification vector l2. These two classifiers are structurally identical, both consisting of a fully connected layer with an input node count equal to the feature embedding dimension D in the Vision Transformer and an output node count equal to the sample types in the dataset.

[0119] Two classification tokens τ are obtained through the salient feature screening engine and the visual structure perceptron cls_a and τ cls_b Afterwards, in order to enrich the pattern combination of features and improve the representativeness of the two classification tokens, it is necessary to perform a random partial exchange process on these two tokens within a batch and obtain the permuted classification tokens Y1 and Y2. Then, a fully connected layer is used to calculate the two parts of the logistic regression vectors l1 and l2 respectively. To improve the feature interaction of these two modules, two learnable weight parameters w1 and w2 are defined, and the weighted regression vector l = w1l1 + w2l2 is calculated. This weighted vector is used as the final image feature to calculate the cross entropy loss:

[0120]

[0121] Where N is the number of samples in a batch, C is the total number of categories in the dataset, and l i,k is the logit output of the i-th sample in category k, y i is the true category label index of the i-th sample, is the logit value of the true label of the i-th sample. In order to learn the subtle differences between subcategories, the contrast loss is also calculated for the significant feature vector Y1, which improves the discriminability of features by maximizing the intra-class similarity and minimizing the inter-class similarity, allowing the model to learn more fine-grained differences in the image:

[0122]

[0123] Where B is the number of samples in the batch, Pij and N ij are indicator variables for positive sample pairs and negative sample pairs, respectively, m is the preset margin, represents the L2-normalized feature vector.

[0124] To summarize, the model is trained using a combination of contrast loss and cross entropy loss, expressed as:

[0125]

[0126] Furthermore, unlike the training process, only data normalization is used in the data augmentation part during validation. The process of randomly partially exchanging the two classification token features obtained in steps S4 and S5 is as follows: Assume that the classification token before transformation is C, and its dimension is [B, H], where B is the batch size and H is its dimension size. First, the random exchange probability is calculated. For each iteration, the exchange probability p is randomly generated by Beta distribution Beta(1,1), and then scaled to the specified interval [p min ,p max ]:

[0127] p=p min +Beta(1,1)×(p max -p min ) (20)

[0128] Next, a random permutation index is generated, generating a random permutation index RP to permute the rows of C, where RP is a random permutation of [0, B-1]. For each column j in C, a permutation is chosen with probability p. If permuted, C[i,j] is replaced by C[RP[i],j], where i is the row index. This can be expressed as:

[0129]

[0130] Here, rand() generates a random number in the range (0, 1). This embodiment iterates this process and applies the swap function to C n times.

[0131] Apply the random permutation process described above to obtain the classification token τ after the permutation of modules (4) and (5) cls_a and τ cls_b . Then use the fully connected layer to calculate τ respectively cls_a and τ cls_b The logistic regression scores of these two features are l1 and l2. Define two learnable weight parameters w1 and w2, and calculate the weighted score l = w1l1 + w2l2. This embodiment uses this weighted score to calculate the cross entropy loss. This feature interaction can improve the model's overall understanding of the image. In order to learn the subtle differences between sub-category images, the significant feature vector τ extracted in step (4) is also required. cls_a (Record it as Y) Calculate the contrast loss It improves the distinguishability of features by maximizing intra-class similarity and minimizing inter-class similarity, allowing the model to learn more fine-grained differences in the image. Therefore, this example uses a combination of contrast loss and cross entropy loss to train the model, expressed as:

[0132]

[0133] In a specific embodiment, the above model is built using the pytorch deep learning framework. This embodiment loads the intermediate weights of the network backbone from the official ViT-B_16 model pre-trained on ImageNet21k. The SGD optimizer is used with a momentum of 0.9 and cosine annealing as the optimizer scheduler. For the Cotton80 dataset, the learning rate is set to 0.02 and the number of training steps is set to 6000. For the IP102 dataset, the learning rate is set to 0.003 and the number of training steps is set to 30,000. Every 200 steps, the accuracy of the model is verified on the test set, and the model with the highest accuracy is saved.

[0134] To mitigate chance, this example uses five-fold cross-validation to evaluate model performance. First, the dataset is divided equally into five parts. One part is used as the test set for each training run, and the remaining part is used as the training set. After five training runs, five models are obtained. The mean of the evaluation metrics of these five models on their respective test sets is used as the performance criterion for this method. The method of this example achieves an average accuracy of 62.5% and 79.6% on the Cotton80 and IP102 datasets, respectively, which are 7.8% and 6.2% higher than the baseline model ViT-B_16 of this example, respectively. Figure 7 This is the comparison result of the attention maps of the method of this embodiment and other models on the IP102 dataset, where the first row is the original samples randomly selected from the dataset, the second to fourth rows are the attention maps of TransFG, FFVT, and SIM-Trans on these samples, and the last row is the attention distribution map of the model of this embodiment on these samples. It can be observed from the figure that although each model pays more or less attention to the object part in the image, compared with the other three models, the model of this embodiment pays more attention to the object part and pays relatively uniform attention to the overall contour of the object, which reflects the model's capture of the object's detailed local features and grasp of the overall structural information, proving the effectiveness and interpretability of the method of this embodiment.

[0135] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0136] Based on the same concept as the method for classifying different plant leaves and different pests in the agricultural field in the above-mentioned embodiment, the present invention also provides a classification system for different plant leaves and different pests in the agricultural field, which can be used to implement the above-mentioned classification method for different plant leaves and different pests in the agricultural field. For ease of explanation, the structural schematic diagram of the embodiment of the classification system for different plant leaves and different pests in the agricultural field only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it can include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0137] See also Figure 8 In another embodiment of the present application, a classification system 100 for different plant leaves and different pests in the agricultural field is provided, the system comprising a data acquisition module 101, a data enhancement module 102, a backbone network construction module 103, a first classification module 104, a second classification module 105 and a feature fusion module 106;

[0138] The data acquisition module 101 is used to acquire plant leaf datasets and agricultural pest datasets and perform preprocessing;

[0139] The data enhancement module 102 is used to perform data enhancement on the data set;

[0140] The backbone network construction module 103 is used to construct a Vision Transformer backbone network based on the self-attention mechanism, and input the enhanced data set into the Vision Transformer backbone network to extract image features. The Vision Transformer backbone network includes a Patch Embedding layer and multiple Transformer Encoder layers. Each Transformer Encoder layer includes a multi-head self-attention layer (MHA) and a multi-layer perceptron (MLP). The MHA uses multiple attention heads to process input in parallel to capture information in different subspaces. The MLP consists of a linear layer and a nonlinear activation function to further process the output of the attention mechanism. In the Patch Embedding layer, each image block is encoded into a D-dimensional token. In all subsequent Transformer Encoder layers, each token maintains this dimension. After the Encoder encoding is completed, the additional D-dimensional classification token is used as a vector representing the features of the entire image for classification.

[0141] The first classification module 104 is configured to select tokens corresponding to the most discriminative regions of the backbone network coding features for classification through a significant feature screening engine;

[0142] The second classification module 105 is configured to perform visual structure perception on the selected tokens through a visual structure perceiver to extract the overall structure of the object in the image to assist in classification;

[0143] The feature fusion module 106 is used to randomly exchange parts of the features generated by the significant feature screening engine and the visual structure perception to enrich the feature combination pattern, and use learnable parameters to fuse the features of the two parts for network training, and use the trained network to classify different plant leaves and different pests.

[0144] It should be noted that the classification system for different plant leaves and different pests in the agricultural field of the present invention corresponds one-to-one to the classification method for different plant leaves and different pests in the agricultural field of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the classification method for different plant leaves and different pests in the agricultural field are applicable to the embodiment of the classification of different plant leaves and different pests in the agricultural field. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0145] In addition, in the implementation of the classification system for different plant leaves and different pests in the agricultural field in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be distributed to different program modules as needed, for example, for the convenience of software implementation, that is, the internal structure of the classification system for different plant leaves and different pests in the agricultural field is divided into different program modules to complete all or part of the functions described above.

[0146] See also Figure 9 In one embodiment, an electronic device for implementing a classification method for different plant leaves and different pests in the agricultural field is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a classification program 203 for different plant leaves and different pests in the agricultural field.

[0147] The first memory 202 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software installed in the electronic device 200 and various types of data, such as the code of the classification program 203 for different plant leaves and different pests in the agricultural field, but can also be used to temporarily store data that has been output or is about to be output.

[0148] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0149] Figure 9 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 9 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0150] The classification program 203 for different plant leaves and different pests in the agricultural field stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve the following:

[0151] Obtain plant leaf datasets and agricultural pest datasets and perform preprocessing;

[0152] Perform data augmentation on the dataset;

[0153] A Vision Transformer backbone network based on the self-attention mechanism is constructed, and the enhanced dataset is input into the Vision Transformer backbone network to extract image features. The Vision Transformer backbone network includes a Patch Embedding layer and multiple Transformer Encoder layers. Each Transformer Encoder layer includes a multi-head self-attention layer (MHA) and a multi-layer perceptron (MLP). The MHA uses multiple attention heads to process the input in parallel to capture information from different subspaces. The MLP consists of linear layers and nonlinear activation functions to further process the output of the attention mechanism. In the Patch Embedding layer, each image patch is encoded into a D-dimensional token. In all subsequent Transformer Encoder layers, each token maintains this dimension. After the Encoder is encoded, the additional D-dimensional classification token is used as a vector representing the features of the entire image for classification.

[0154] The salient feature filtering engine is used to select the tokens corresponding to the most discriminative regions in the backbone network coding features for classification;

[0155] The visual structure perceptron performs visual structure perception on the selected tokens to extract the overall structure of the object in the image to assist classification;

[0156] The features generated by the salient feature screening engine and the visual structure perceptron are randomly partially exchanged to enrich the feature combination pattern, and the features of the two parts are fused with learnable parameters for network training. The trained network is used to classify different plant leaves and different pests.

[0157] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0158] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for classifying different plant leaves and different pests in the agricultural field, characterized in that: The steps include: Obtain plant leaf datasets and agricultural pest datasets and perform preprocessing; Perform data augmentation on the dataset; A Vision Transformer backbone network based on the self-attention mechanism is constructed, and the enhanced dataset is input into the Vision Transformer backbone network to extract image features. The Vision Transformer backbone network includes a Patch Embedding layer and multiple Transformer Encoder layers. Each Transformer Encoder layer includes a multi-head self-attention layer (MHA) and a multi-layer perceptron (MLP). The MHA uses multiple attention heads to process the input in parallel to capture information from different subspaces. The MLP consists of linear layers and nonlinear activation functions to further process the output of the attention mechanism. In the Patch Embedding layer, each image patch is encoded into a D-dimensional token. In all subsequent Transformer Encoder layers, each token maintains this dimension. After the Encoder is encoded, the additional D-dimensional classification token is used as a vector representing the features of the entire image for classification. The salient feature screening engine is used to select the tokens corresponding to the most discriminative regions in the backbone network coding features for classification, specifically: Get the attention weight between any two tokens in each layer of the backbone network; In the l Layer 1 to layer l The cumulative attention weights of all tokens are obtained by cumulative multiplication of the attention weights of the layer, and the size of the attention weight is directly used to calculate the first i The first part of the token saliency score, thus obtaining the l The first part of the score of all tokens in the layer is summarized; Calculate the i The second part of the token score: Initialize a sum Uniformly distributed vectors of the same dimension U , and then use Jensen-Shannon divergence to measure and U The similarity of data distribution; l The corresponding Jensen-Shannon divergences of all tokens in the layer are combined to get the second part of the significance score of each token; The two parts of the significance score of each token are added together and normalized to get the final score of each token; Finally, the final score of each token is sorted in descending order, and a portion of the tokens with the highest significance scores are selected from each layer and aggregated with the classification tokens of the backbone layer through a Transformer encoding layer to obtain the classification tokens of the saliency feature screening engine. ; The visual structure perceptron performs visual structure perception on the selected tokens to extract the overall structure of the object in the image to assist classification, specifically: Use the average of each token's attention weight vector And the information entropy corresponding to the attention weight vector H The normalized sum Score 2 as its scoring index, based on the scoring index, the tokens encoded by the last layer in the network backbone are sorted, and then the token with the largest score is selected as the anchor point, and all tokens with scores greater than the average are input into a two-layer graph convolutional neural network to establish the structural relationship between these tokens, thereby obtaining classification tokens containing the visual structural features of the object ; The features generated by the salient feature screening engine and the visual structure perceptron are randomly partially exchanged to enrich the feature combination pattern, and the features of the two parts are fused with learnable parameters for model training. The trained model is used to classify different plant leaves and different pests.

2. The method for classifying different plant leaves and different pests in the agricultural field according to claim 1, characterized in that: The plant leaf dataset and the agricultural pest dataset are obtained and preprocessed, specifically: Collect plant leaf datasets and agricultural pest datasets; Filter the dataset, remove poor quality images and corresponding labels in the dataset, and ensure that the number of images of each category in the dataset is uniform and the images are complete; The images in the filtered dataset are uniformly processed into the set size.

3. The method for classifying different plant leaves and different pests in the agricultural field according to claim 1, characterized in that: The data enhancement of the dataset is specifically as follows: The images in the dataset are randomly cropped, horizontally flipped, vertically flipped, brightness and contrast are randomly adjusted, and the images are randomly erased, and the data is normalized at the end.

4. The method for classifying different plant leaves and different pests in the agricultural field according to claim 1, characterized in that: The Vision Transformer backbone network based on the self-attention mechanism is constructed, and the enhanced dataset is input into the Vision Transformer backbone network to extract image features. Specifically: First, a linear projection layer is used to divide the images in the dataset into K Each block is projected into a D-dimensional latent embedding , and add positional encoding to the generated tokens, and add an additional classification token for classification; Then the pre-trained ViT-B_16 backbone network is used as the feature extractor to extract the features of the image; Take the first token of the feature encoded by the last layer of the backbone network, which is also the final classification token Used for classification.

5. The method for classifying different plant leaves and different pests in the agricultural field according to claim 1, characterized in that: The training process is specifically as follows: Two classification tokens are obtained through the salient feature screening engine and the visual structure perceptron and Afterwards, the two tokens are randomly partially exchanged in a batch and the replaced classification tokens are obtained. Y 1 and Y 2; Then use the fully connected layer to calculate the two parts of the logistic regression vector l 1 and l 2; In order to improve the feature interaction between the two modules, two learnable weight parameters are defined w 1 and w 2, and calculate the weighted regression vector , use this weighted vector as the final image feature to calculate the cross entropy loss : In order to learn the subtle differences between subcategories, the salient feature vectors Y 1 Calculate contrast loss , by maximizing the intra-class similarity and minimizing the inter-class similarity, the discriminability of features is improved, allowing the model to learn more fine-grained differences in the image; The model is trained using a combination of contrastive loss and cross entropy loss.

6. A classification system for different plant leaves and different pests in the agricultural field, characterized in that: A classification method for different plant leaves and different pests in the agricultural field, applied to any one of claims 1-5, comprising a data acquisition module, a data enhancement module, a backbone network construction module, a first classification module, a second classification module, and a feature fusion module; The data acquisition module is used to acquire plant leaf datasets and agricultural pest datasets and perform preprocessing; The data enhancement module is used to perform data enhancement on the data set; The backbone network building module is used to build a Vision Transformer backbone network based on the self-attention mechanism, and input the enhanced data set into the Vision Transformer backbone network to extract image features. The Vision Transformer backbone network includes a Patch Embedding layer and multiple Transformer Encoder layers. Each Transformer Encoder layer includes a multi-head self-attention layer (MHA) and a multi-layer perceptron (MLP). The MHA uses multiple attention heads to process input in parallel to capture information in different subspaces. The MLP consists of linear layers and nonlinear activation functions to further process the output of the attention mechanism. In the Patch Embedding layer, each image block is encoded into a D-dimensional token. In all subsequent Transformer Encoder layers, each token maintains this dimension. After the Encoder encoding is completed, the additional D-dimensional classification token is used as a vector representing the features of the entire image for classification. The first classification module is configured to select tokens corresponding to the most discriminative regions of the backbone network coding features for classification through a significant feature screening engine; The second classification module is configured to perform visual structure perception on the selected tokens through a visual structure perceiver to extract the overall structure of the object in the image to assist in classification; The feature fusion module is used to randomly exchange parts of the features generated by the significant feature screening engine and the visual structure perception to enrich the feature combination pattern, and use learnable parameters to fuse the features of the two parts for network training, and use the trained network to classify different plant leaves and different pests.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the classification method for different plant leaves and different pests in the agricultural field as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the classification method for different plant leaves and different pests in the agricultural field according to any one of claims 1 to 5 is implemented.

Images