A crop pest optimal view angle recognition method based on multi-image attention feature fusion

Abstract

This invention discloses a method for optimal viewpoint recognition of crop pests based on multi-image attention feature fusion, comprising the following steps: feature extraction is performed on multiple pest images from different viewpoints using a pre-trained recognition model to obtain feature vectors for each pest image; the feature vectors of all pest images are adaptively selected through multi-head self-attention and a gating mechanism to obtain the optimal viewpoint features for each pest image; the optimal viewpoint features of all pest images are classified, and the pest categories in the multiple pest images from different viewpoints are selected. This invention's method achieves automatic data cleaning and optimal viewpoint recognition through feature fusion using attention and gating mechanisms, thereby making the trained classifier more robust and having higher recognition accuracy. The model is simple and efficient, and in use, farmers only need to take a few more photos to obtain more reliable recognition results than traditional methods, which has significant economic value.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of crop pest identification, and particularly relates to a crop pest optimal visual angle identification method based on multi-image attention feature fusion. BACKGROUND

[0002] Crops play an important role in human survival, green energy and industrial raw materials. In recent years, with the problems of population growth, climate change, species invasion, and decline of agricultural labor, coupled with people's increasing emphasis on green and pesticide-free food, crop production is facing new challenges. As one of the biggest hazards affecting crop safety, pests have the characteristics of suddenness, irregularity, diversity, wide spread, and rapid evolution. If they cannot be scientifically and effectively prevented and controlled as soon as possible, they can easily cause crop damage and reduce yield, and even hinder the sustainable development of the ecological environment. The premise of effective prevention and control is to quickly and accurately detect the pest species, so as to treat the disease accordingly.

[0003] With the development of modern agricultural intelligence and digitization, people seek to convert pest identification from traditional inefficient manual observation to precise and intelligent real-time identification, so that the pest type and control method can be quickly and accurately judged on a mobile phone through the use of a browser or a small program. However, such technology has not been popularized, one of the reasons being that the accuracy of pest identification is not trustworthy enough.

[0004] There are many factors that affect the accuracy of pest identification, among which the problems of pests themselves and image quality are particularly serious. As we all know, most pests have many growth stages in their life cycle, for example, one of the rice pests, the rice leaf roller, has egg, larva, pupa, and adult stages, which leads to large intra-class differences, especially for small target pests, which are difficult to accurately identify. At the same time, the morphological differences between pests in the same growth stage are very small, which leads to small inter-class differences. In addition, the colors of some pests are very similar to the colors of crops or soil, making it difficult to detect the presence of pests. On the other hand, there are few open pest data sets available for research at present, and the image quality is uneven, with problems such as low image resolution, labeling errors, absence of corresponding pests, inclusion of irrelevant elements, complex image backgrounds, and unbalanced data distribution in the data set. In addition, the quality of the images to be identified in actual agricultural scenes is also complex and unstable, and a single image from a single visual angle cannot extract accurate features, resulting in incorrect results. Therefore, how to make full use of existing data, intelligently clean and filter images, extract optimal pest features, and train a classifier is a difficult problem that needs to be solved.

[0005] To solve the above problems in pest identification, the most fundamental and effective solution is to design a fast and accurate identification algorithm. The traditional pest identification method mainly relies on manual observation, which is time-consuming and labor-intensive, and requires a high level of experience for agricultural practitioners, resulting in a high probability of false positives. With the gradual development of computer vision technology, people began to apply image processing technology and machine learning methods to pest identification, and the identification process generally followed the acquisition of pest data, data preprocessing, feature extraction, model training and identification. However, early work mainly relied on classic image processing programs and statistical methods, which were carefully designed and combined by humans to extract shallow features, and then trained shallow machine learning classifiers for identification. It can be seen that this method largely depends on the continuous combination of methods by system designers for tedious trial and error, and feature extraction is difficult, with unsatisfactory classification accuracy. In recent years, with the rapid development of deep learning technology, convolutional neural networks have shown outstanding performance as feature extractors and classifiers in image recognition, and have rapidly expanded to pest identification applications. Compared to early image processing methods, convolutional neural networks have simplified image preprocessing and modeling processes, can extract deep abstract features, and significantly improve recognition accuracy.

[0006] However, training a convolutional neural network requires as large an image dataset as possible, and due to the current size of the pest identification dataset, a widely used method that can significantly improve recognition accuracy is to use transfer learning to train a more robust convolutional neural network. Transfer learning uses pre-trained convolutional neural network model parameters on ImageNet, and then re-trains them on the pest dataset, either shallowly (fine-tuning the last linear layer) or deeply (fine-tuning all network layers). Studies have shown that deep transfer learning has higher recognition accuracy than shallow transfer learning and training from scratch.

[0007] However, deep transfer learning still performs poorly in complex pest image recognition. For example, on the IP102 dataset, Densenet169 can only achieve 72.8% test set accuracy, and Resnet50 can only reach 72.0%. Therefore, to improve recognition performance, many scholars have conducted extensive research, such as designing a coarse-to-fine convolutional neural network for multi-scale learning, designing a CNN ensemble for ensemble learning, using new data augmentation, and manually cleaning data, but none of these methods have achieved significant accuracy improvements, and the model design is complex, time-consuming and labor-intensive. SUMMARY

[0008] To solve the problems in the prior art, the purpose of the present application is to provide a crop pest optimal view recognition method based on multi-image attention feature fusion, which can improve the accuracy and efficiency of pest image recognition.

[0009] The technical solutions adopted by the present application are as follows:

[0010] A crop pest optimal view angle recognition method based on multi-image attention feature fusion includes the following processes:

[0011] Feature extraction is performed on the acquired multiple pest images of different view angles by using the pre-trained recognition model to obtain the feature vector of each pest image; the feature vectors of all pest images are adaptively selected by using multi-head self-attention and a gate mechanism to obtain the optimal view angle features of each pest image; the optimal view angle features of all pest images are classified, and the category of the pest in the multiple pest images of different view angles is screened out.

[0012] Preferably, the recognition model uses a convolutional neural network, the convolutional neural network is loaded with pre-training parameters of the convolutional neural network on ImageNet, and an attention feature fusion network is added after the last feature extraction layer and before the last classification layer of the convolutional neural network.

[0013] Preferably, the process of adaptively selecting the feature vectors of all pest images by using multi-head self-attention and a gate mechanism to obtain the optimal view angle features of each pest image includes:

[0014] The feature vectors of all pest images are combined into an input matrix;

[0015] The input matrix, the query parameter matrix, the key parameter matrix, and the value parameter matrix are used to obtain the query matrix, the key matrix, and the value matrix;

[0016] The attention score matrix is calculated by using the query matrix and the key matrix, the attention score matrix is normalized by using the softmax, and the value matrix is multiplied by the attention score matrix to obtain the attention output matrix;

[0017] The query parameter matrix, the key parameter matrix, and the value parameter matrix of each head are set in parallel, and the self-attention calculation is independently performed in each head to obtain the corresponding head output;

[0018] The gate mechanism is applied to the query matrix and the key matrix of each head output to obtain the improved head output;

[0019] Each improved head output is spliced together, the dimension conversion matrix is used to convert the spliced result into a vector with the same dimension as the original output to obtain the feature vector of each pest image.

[0020] Preferably, the process of performing self-attention calculation in each head to obtain the corresponding head output includes:

[0021] For each head, the query matrix and the key matrix of the head are linearly mapped to a joint space, and then a fusion operation of element-wise multiplication is performed to obtain a fusion result;

[0022] Then the fusion result is mapped, and the mapped result is processed through a sigmoid activation function to obtain a query mask and a key mask;

[0023] The query mask is multiplied element-wise with the query matrix of the head, and the key mask is multiplied element-wise with the key matrix K i Element-wise multiplication to obtain the improved output of the head.

[0024] Preferably, for the i-th head, the fusion result of the head is As follows:

[0025]

[0026] The query mask And the key mask As follows:

[0027]

[0028] The output of the improved head As follows:

[0029]

[0030] Wherein, n is the number of input images, d k is the dimension of the feature vector, Q i is the query matrix of the i-th head, K i is the key matrix of the i-th head, and are learnable mapping matrices, and are bias vectors, sigma() is an activation function, and Attention() is an attention calculation function.

[0031] Preferably, the optimal view angle features of all pest images are classified, and the process of screening the category of pests in the plurality of different view angle pest images includes:

[0032] The optimal view angle features of all pest images are classified through a linear classification layer to obtain the corresponding prediction results, and a majority voting strategy is used, and the class that has the most votes is taken as the final prediction result of these pest images, and the category of pests in the pest image is determined.

[0033] The number of the plurality of different view angle pest images is not less than two.

[0034] The application also provides a crop pest optimal view angle recognition system based on multi-image attention feature fusion.

[0035] The computing unit is used for extracting features of the acquired multiple pest images of different view angles by the pre-trained recognition model to obtain a feature vector of each pest image; the feature vectors of all pest images are adaptively selected by multi-head self-attention and gate mechanism to obtain optimal view angle features of each pest image; and the optimal view angle features of all pest images are classified, and the category of the pest in the multiple pest images of different view angles is screened out.

[0036] Preferably, the crop pest optimal view angle recognition system based on multi-image attention feature fusion further comprises

[0037] The image acquisition unit is used for acquiring the multiple pest images of different view angles and transmitting the pest images to the computing unit.

[0038] The application also provides an electronic device comprising:

[0039] One or more processors;

[0040] A storage device having one or more programs stored thereon;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the crop pest optimal view angle recognition method based on multi-image attention feature fusion as described above.

[0042] The application also provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the crop pest optimal view angle recognition method based on multi-image attention feature fusion as described above.

[0043] The application has the following beneficial effects:

[0044] The method of the application realizes automatic data cleaning and optimal view angle recognition through attention and gate mechanism, so that the trained classifier is more robust and has high recognition accuracy, and the model is simple and efficient. In practical application, agricultural practitioners can easily obtain multiple images of pests of multiple view angles and multiple stages, and after optimal view angle recognition by a local model or uploading to a backend model, the high-credibility pest category can be quickly obtained, so that corresponding prevention measures can be implemented in time to prevent the spread of disasters and protect the healthy growth of crops. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The crop pest optimal view angle recognition method based on multi-image attention feature fusion adopted by the application is shown in the flowchart. DETAILED DESCRIPTION

[0046] The present application will be further described in conjunction with the specific embodiments and the accompanying drawings.

[0047] The present application aims at the problems of pests and image quality in the prior art. Based on the existing public pest data set, multiple pest images with the same label are randomly selected, and the deep transfer learning convolutional neural network is used to extract the features of each image. Then, the multi-head self-attention and the gate mechanism are used to adaptively select the optimal features and filter the useless features. Finally, each fused image feature is classified through a linear layer, and then the majority voting strategy is used to determine the category to which the images belong. Referring to Figure 1 The specific scheme of the present application is as follows:

[0048] Step one: Obtain data

[0049] As many pest data as possible with different perspectives and different stages are obtained from network public data or targeted shooting, and these data are divided into training set, validation set and test set according to the proportion.

[0050] Step two: Build a model

[0051] A convolutional neural network is selected and the pre-training parameters of the convolutional neural network on ImageNet are loaded. An attention feature fusion network is added after the last feature extraction layer and before the last classification layer to obtain the optimal perspective features of each image, and then a linear classification layer for identification is connected.

[0052] Step three: Train the model

[0053] A group of multiple images with the same label is randomly selected each time (because there are generally 4-5 growth stages of pests, it is recommended to set the input number to 5 when training the model. Experiments show that the more the better the recognition effect, but the accuracy improvement is not much more than 5, and the calculation resources are wasted). According to the size of the training data set and the data fitting result, the hyperparameters are set and adjusted to make the trained model perform best on the validation set. Generally, the cross-entropy loss function, the stochastic gradient descent optimizer with momentum factor 0.9, the initial learning rate 0.001, and the strategy of learning rate decay 0.1 for half of the training period can be selected. After the image is input into the model after basic image enhancement, the recognition result of each image is obtained. The loss is calculated by comparing the real label and the result, and the result is optimized by back propagation. The class with the most votes in the result is selected as the unified recognition result of the group of images, and then the proportion of correct recognition is calculated to obtain the change of the training set recognition loss value and accuracy. The same operation is applied to the validation set, but the validation set does not perform back propagation and gradient update. The model parameters that make the validation set recognition accuracy the highest are saved as the best model parameters.

[0054] Step four: test model:

[0055] The model loads the best model parameters, inputs the test set data, obtains the recognition result, and uses the majority voting strategy to determine the final recognition result. The test set recognition accuracy is calculated by comparing with the true label. If the accuracy meets the expectation, it enters step five for practical application. If it does not meet the expectation, measures such as increasing the quality and quantity of training set images and adjusting the hyperparameters are taken to continue to improve the model recognition accuracy until the expectation is met. The process of adjusting the hyperparameters includes: adjusting the number of iterations, adjusting the batch sample size, improving the optimizer, modifying the learning rate and decay strategy.

[0056] Step five: practical application:

[0057] In actual use, the application mode can be single machine operation or "client-server" operation. After the front-end interface design is completed, the back-end model loads the best model parameters, and then continuously recognizes multiple images input by each user. Since it is relatively inexpensive for farmers to obtain multi-angle and multi-stage pest images, inputting as many pest images as possible is beneficial to accurate recognition results.

[0058] The present application realizes data filtering and optimal view recognition through attention and gate mechanism for multi-image feature fusion, so that the trained classifier is more robust and has high recognition ability. The model is simple and efficient, and in use, farmers only need to take a few more photos to obtain more reliable recognition results than traditional methods, which has great economic value.

[0059] In the above scheme of the present application, the model main body is a multi-head self-attention network. The input multiple images are respectively extracted into feature vectors by a convolutional neural network based on deep transfer learning (without the last classification layer), and then combined into an input matrix X∈R n×d , where n is the number of input images, and d is the dimension of the feature vector. Then, the query parameter matrix , the key parameter matrix and the value parameter matrix are obtained, as shown in the following formula:

[0060]

[0061] After calculating the attention score matrix and normalizing it through the softmax layer, the value matrix V is multiplied to obtain the attention output matrix, i.e.:

[0062]

[0063] In order to enhance the feature representation, the above single-head self-attention is improved to the following multi-head self-attention, the i-th head query parameter matrix is set as the i-th head key parameter matrix is and the i-th head value parameter matrix is h is the number of heads, and each head independently performs the above single-head self-attention calculation to obtain the corresponding i-th head output:

[0064] h i =Attention(Q i ,K i ,V i )

[0065] Finally, each head is spliced together and converted into a vector with the same dimension as the original single-head output through a dimension conversion matrix W O

[0066] MultiHead(Q,K,V)=Concat(h1,...,h h )W O

[0067] Self-attention obtains attention scores by simply calculating the dot product similarity between queries and keys, which may contain noisy or meaningless information. Therefore, in order to adaptively filter out useless information, the present application applies a gating mechanism to the query and key, and uses the nonlinear mapping function of the sigmoid function to adjust the information transmission of the query matrix and the key matrix, thereby enhancing the feature screening ability of attention, making the attention output more optimal identification features. The specific steps are as follows:

[0068] For the i-th head, first linearly map the i-th head query matrix Q i and the key matrix K i to a joint space, then perform element-wise multiplication fusion operation to obtain the fusion result

[0069]

[0070] wherein, and are learnable mapping matrices, and are bias vectors.

[0071] Then, the fusion result G i is further mapped through a fully connected layer followed by a sigmoid activation function (i.e., G i is further mapped and processed through a sigmoid activation function), to obtain the query mask and the key mask

[0072]

[0073] ​Finally, the two gating masks are applied to the i-th original query matrix Q i and the key matrix K i above (i.e., the query mask is multiplied element-wise with the original query matrix Q i and the key mask is multiplied element-wise with the original key matrix K i to obtain the improved i-th output), i.e., the i-th self-attention computation is improved to

[0074]

[0075] In the technical solution of the present application, the above-mentioned gating mechanism is used to suppress the transmission of useless information in the image, thereby improving the quality of the features. The multi-head self-attention network further filters and fuses the features to extract the optimal view angle features that are beneficial for the recognition of each image. Next, the processed n image features are input into the final linear classification layer to obtain the corresponding n prediction results. The majority voting strategy is used to determine the final prediction result of the n images as the one that has the most votes for a certain class.

[0076] To prove the effectiveness of the optimal view angle feature selection, the ordinary voting result without using self-attention and gating mechanism is compared. IP102 is used as the experimental data set, Densenet169 is used as the feature extractor, and n=5 is used to train the classifier. The test set recognition results are shown in Table 1:

[0077] Table 1

[0078]

[0079]

[0080] As can be seen from the results in Table 1, the more images are input during testing, the more conducive it is to pest identification. However, the effect is better than that of the ordinary voting method, which proves that the optimal view angle feature is helpful for pest identification.

[0081] In the conventional single-input image training test, Densenet169 based on deep transfer learning only achieves an accuracy of 72.8%, which proves the necessity of multi-image fusion recognition.

Claims

1. A method for optimal viewpoint recognition of crop pests based on multi-image attention feature fusion, characterized in that, The process includes the following: The pre-trained recognition model extracts features from multiple pest images from different perspectives to obtain the feature vector of each pest image. The feature vectors of all pest images are adaptively selected through multi-head self-attention and gating mechanism to obtain the optimal perspective features of each pest image. The optimal perspective features of all pest images are classified and the pest categories in the multiple pest images from different perspectives are selected. The process of adaptively selecting the feature vectors of all pest images through multi-head self-attention and a gating mechanism to obtain the optimal viewpoint features for each pest image includes: Combine the feature vectors of all pest images into an input matrix; The query matrix, key matrix, and value matrix are obtained using the input matrix, query parameter matrix, key parameter matrix, and value parameter matrix. The attention score matrix is ​​calculated by using the query matrix and the key matrix. After the attention score matrix is ​​normalized by softmax, it is multiplied by the value matrix to obtain the attention output matrix. By setting the query parameter matrix, key parameter matrix, and value parameter matrix of each head in parallel, the self-attention calculation is performed independently within each head to obtain the corresponding head output; By applying the gate mechanism to the query matrix and key matrix of each header output, an improved header output is obtained; The improved head outputs are concatenated together, and the concatenated result is transformed into a vector with the same dimension as the original output through a dimension transformation matrix, thus obtaining the feature vector of each pest image.

2. The optimal viewpoint identification method for crop pests based on multi-image attention feature fusion according to claim 1, characterized in that, The recognition model employs a convolutional neural network, which is loaded with pre-trained parameters on ImageNet. An attention feature fusion network is added to the convolutional neural network after the last feature extraction layer and before the last classification layer.

3. The optimal viewpoint identification method for crop pests based on multi-image attention feature fusion according to claim 1, characterized in that, The process of independently calculating self-attention within each head to obtain the corresponding head output includes: For each head, the query matrix and key matrix of that head are linearly mapped to a joint space, and then an element-wise multiplication fusion operation is performed to obtain the fusion result; The fusion result is then mapped, and the mapping result is further processed by the sigmoid activation function to obtain the query mask and key mask. Multiply the query mask element-wise with the query matrix of the header, and multiply the key mask element-wise with the key matrix of the header. Multiply each element to obtain the improved output of the head.

4. The optimal viewpoint identification method for crop pests based on multi-image attention feature fusion according to claim 3, characterized in that, For the i The size, the result of merging the head. as follows: query mask and key mask as follows: Improved head output as follows: in, n The number of input images, d k The dimension of the feature vector. For the first i Query matrix of size, For the first i The key matrix of the size, and A learnable mapping matrix, and For bias vectors, For activation function, This is the attention calculation function.

5. The optimal viewpoint identification method for crop pests based on multi-image attention feature fusion according to claim 1, characterized in that, The process of classifying the optimal viewpoint features of all pest images and filtering out the pest categories from the multiple pest images from different viewpoints includes: The optimal viewpoint features of all pest images are classified through a linear classification layer to obtain the corresponding prediction results. Then, a majority voting strategy is used to take the category with the most votes as the final prediction result for these pest images, thus determining the category of pests in the pest images. The number of the multiple images of pests from different perspectives shall not be less than two.

6. A crop pest identification system based on multi-image attention feature fusion, characterized in that, The method for implementing the optimal viewpoint recognition method for crop pests based on multi-image attention feature fusion as described in any one of claims 1-5 includes: The computing unit is used to extract features from multiple pest images from different perspectives using a pre-trained recognition model, obtaining a feature vector for each pest image; adaptively selecting the feature vectors of all pest images through multi-head self-attention and a gating mechanism to obtain the optimal perspective features for each pest image; classifying the optimal perspective features of all pest images and filtering out the pest categories from the multiple pest images from different perspectives.

7. The optimal viewpoint recognition system for crop pests based on multi-image attention feature fusion according to claim 6, characterized in that, Also includes: Image acquisition unit: Used to acquire multiple images of pests from different perspectives and transmit the images to the computing unit.

8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the optimal viewpoint recognition method for crop pests based on multi-image attention feature fusion as described in any one of claims 1 to 5.

9. A storage medium, characterized in that, It stores a computer program, wherein when the computer program is executed by a processor, it implements the optimal viewpoint recognition method for crop pests based on multi-image attention feature fusion as described in any one of claims 1 to 5.

Images