A pollen image classification method based on attention mechanism and convolutional neural network

Abstract

A pollen image classification method based on an attention mechanism and a convolutional neural network includes: constructing a pollen image dataset; building a pollen image classification model based on a DenseNet network structure, comprising four identical Dense Blocks and three identical Transition Layers, with an attention module added between the third Dense Block and the third Transition Layer; training the pollen image classification model using the pollen image dataset; and classifying new pollen images using the trained pollen image classification model. This method uses DenseNet as the underlying network structure and uses the attention module to learn the relationships between features, thereby improving classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to image classification, and in particular to a pollen image classification method based on an attention mechanism and a convolutional neural network. Background Art

[0002] Real-time monitoring of pollen grain types is of great significance in fields such as biology and epidemiology. Current pollen classification technologies primarily include manual classification, machine learning, and deep learning. Manual classification, the most traditional method, requires strong professional expertise and is time-consuming, labor-intensive, and has low accuracy. Machine learning classification methods primarily rely on artificial feature design and utilize machine learning algorithms for classification and identification. These methods require not only strong knowledge of plant morphology and practical experience, but also often require complex feature engineering. Their accuracy lags behind that of deep learning methods. Deep learning classification methods offer superior accuracy to these two methods due to their robust feature learning capabilities and good generalization. However, these methods are complex to design and computationally demanding, placing high demands on hardware. Furthermore, pollen images obtained using existing image processing methods often contain significant noise, which impacts classification accuracy. Summary of the Invention

[0003] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a pollen image classification method based on the attention mechanism and convolutional neural network, using DenseNet as the basic network structure, and learning the relationship between features through the attention module to improve the classification accuracy.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A pollen image classification method based on attention mechanism and convolutional neural network, including:

[0006] Step 1: Construct a pollen image dataset;

[0007] Step 2: Construct a pollen image classification model. The pollen image classification model uses Dense Net as the basic network structure, which includes four Dense Blocks with the same structure and three Transition Layers with the same structure, and adds an attention module between the third Dense Block and the third Transition Layer;

[0008] Step 3, using the pollen image dataset to train the pollen image classification model;

[0009] Step 4: Use the trained pollen image classification model to classify new pollen images.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. This paper is based on a convolutional neural network and draws on the self-attention module in the Transformer model. It innovatively adds cosine similarity to calculate attention based on the original inner product to further improve the classification accuracy.

[0012] 2. This invention addresses the lack of globality when calculating attention in the Transformer model. It innovatively adds attention calculations in both spatial and channel dimensions, making the features learned by the model more comprehensive.

[0013] 3. This invention addresses the problem of increased training time when the attention module is used multiple times in the model. Without reducing the classification accuracy, better classification results can be achieved by using the attention module only once and increasing the training time by a small amount.

[0014] 4. One of the advantages of the present invention is that the attention module can be "plug and play". It can be nested in any position in any basic model while ensuring that the input and output dimensions and sizes are the same, which is convenient for use. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A pollen image classification model based on attention mechanism.

[0016] Figure 2 Schematic diagram of the attention module.

[0017] Figure 3 Schematic diagram of the first path of the position attention module.

[0018] Figure 4 Schematic diagram of the second path of the position attention module.

[0019] Figure 5 Schematic diagram of the first path of the channel attention module.

[0020] Figure 6 Schematic diagram of the second path of the channel attention module. DETAILED DESCRIPTION

[0021] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0022] As mentioned above, the large amount of noise in pollen images leads to low model classification accuracy. To solve this problem, the present invention provides a convolutional neural network model combined with an attention module, which can enable the model to focus more attention on the pollen grains themselves during the learning process, thereby improving the classification accuracy of pollen.

[0023] Specifically, the pollen image classification method based on the attention mechanism and convolutional neural network of the present invention mainly includes the following steps:

[0024] Step 1: Build a pollen image dataset.

[0025] In this step, you can collect pollen images yourself, such as using a BAA500 device. Try to avoid mixing pollen samples from other types during the collection process to ensure sample purity. Alternatively, you can use an existing dataset, such as POLEN23E.

[0026] For ease of processing, the format needs to be unified. In one embodiment of the present invention, pollen images are captured using a SU 8220 field emission scanning electron microscope, and the following requirements are met:

[0027] Each pollen image is adjusted to a grayscale image in BMP format with a uniform size, for example, 224x224 pixels. Each pollen image contains only one complete pollen grain, and some pollen images contain background noise or other noise, such as incomplete pollen grains.

[0028] Step 2: Build a pollen image classification model.

[0029] like Figure 1 As shown, the pollen image classification model of the present invention is based on the Dense Net network structure, which is mainly composed of four Dense Blocks with the same structure and three Transition Layers with the same structure. The present invention adds an attention module between the third Dense Block and the third Transition Layer. In the order of input and output, the complete structure of the model of the present invention includes:

[0030] Convolutional layer, Dense Block1, Transition Layer1, Dense Block2, Transition Layer2, Dense Block3, Transition Layer3, Dense Block4, Transition Layer1, Linear Layer, and Softmax layer.

[0031] Step 3: Using the pollen image dataset to train the pollen image classification model until the accuracy is not significantly improved.

[0032] Step 4: Use the trained pollen image classification model to classify new pollen images, and the classification is based on the features learned during the training of the pollen image classification model.

[0033] In the pollen image classification model of the present invention, the main function of the convolutional layer is to extract preliminary features from the pollen image, wherein the features extracted by the lower convolutional layer are mainly low-dimensional features such as color features, texture features, and contour features, and the features extracted by the higher convolutional layer are high-dimensional features including category semantic information.

[0034] Each Dense Block further extracts features. Specifically, the features extracted by the underlying Dense Block are mainly color features, texture features, and contour features. Through gradual feature extraction, the features extracted by the high-level Dense Block have higher semantic information, which is close to human understanding of the image content.

[0035] Each Transition Layer compresses the model, reduces the channel dimension in the features, reduces the number of parameters, and prevents parameter explosion.

[0036] The Linear Layer classifies pollen images based on the learned features.

[0037] Specifically, each Dense Block contains a different number of Dense Layers, each consisting of a normalization layer, an activation function layer, and a convolutional layer. Each Transition Layer sequentially includes a normalization layer, an activation function layer, a convolutional layer, and a pooling layer. In each Dense Block, the input of the first Dense Layer is the output of the previous layer, and the input of the Dth Dense Layer is the output of the first to D-1th Dense Layers in the Dense Block, where D>1. For example: Dense Block 1 contains Dense Layer 11, Dense Layer 12, and Dense Layer 13; Dense Block 2 contains Dense Layer 21, Dense Layer 22, and Dense Layer 23. Therefore: the input of Dense Layer 11 is the output of the convolutional layer in the model, the input of Dense Layer 12 is the output of Dense Layer 11, and the input of Dense Layer 13 is the output of Dense Layer 11 and Dense Layer 12. The input of Dense Layer 21 is the output of Transition Layer 1, the input of Dense Layer 22 is the output of Dense Layer 21, the input of Dense Layer 23 is the output of DenseLayer 21 and Dense Layer 22, and so on. The purpose of using Dense Block is to better utilize the feature information learned by the network. The function of Transition Layer is to change the number of channels in the feature map and control the model size and number of parameters.

[0038] In existing technologies, the use of attention modules in computer vision tasks is mostly nested in the Transformer structure and used multiple times in the model, which increases the model training time. The information captured by the traditional attention mechanism lacks globality.

[0039] In the present invention, the attention module is located between the third Dense Block and the third Transition Layer. The main reason is that after passing through multiple Dense Blocks, the extracted features have more high-level semantics. At this time, performing attention calculation operations on the features can minimize the model training time and improve the classification effect.

[0040] Specifically, refer to Figure 2 The attention module of the present invention mainly consists of two parallel modules: the position attention module and the channel attention module.

[0041] The Position Attention module introduces a two-way attention mechanism to capture the spatial correlation between two positions in the feature vector. For each position's feature, an attention weight is added to the original feature and updated. The weight is determined by the feature similarity between the two corresponding positions. In other words, any two positions with similar features can improve each other, regardless of their spatial distance.

[0042] The channel attention module also uses a two-way attention mechanism to capture the channel dependency between any two channels and uses the weighted sum of all channels to update each channel. Since the pollen features extracted by each channel are different, the channel attention module is used to capture the mapping relationship between different channels to enhance the feature representation ability.

[0043] Finally, the outputs of the attention module and the channel attention module are added and fused to further enhance the feature representation and obtain more accurate classification results.

[0044] Furthermore, in the position attention module, attention is calculated in two ways, with the input feature vectors fed into each path. Here, the input feature vector refers to the feature information extracted by the neural network stored in vector form for subsequent calculations. In the input feature vector, there is a mutual influence relationship between different pollen pixels.

[0045] In the first path, a convolution operation is first performed on the input eigenvector to obtain three eigenvectors of the same size. Two of the eigenvectors are multiplied to obtain the attention matrix. The attention matrix is ​​then multiplied and summed with the third eigenvector to obtain the final output eigenvector of the path.

[0046] In the second path, the cosine similarity of the input feature vector is first calculated to obtain the similarity attention matrix. The similarity attention matrix is ​​then multiplied and summed with the input feature vector to obtain the final output feature vector of the path.

[0047] The final output result of the position attention module is obtained by adding the final output vector obtained from the two calculations and the initial input feature vector element by element. By combining the spatial attention mechanism, the neural network can focus on pixels that have a more significant impact on the pollen classification results, thereby improving the feature expression ability of key pollen areas.

[0048] refer to Figure 3 , the position attention module adopts a dual-path parallel attention mechanism. For the first path, given a feature vector A∈R C×H×W, where C represents the number of feature vector channels, H, W represent the length and width of the feature map respectively, and R represents the feature set. First, the feature vector is input into the convolution layer to obtain three feature vectors of the same size, namely Q, K, V, and {Q, K, V}∈R C×H×W Then transform the shape of Q and K into R C×N , where N = H × W, represents the number of pixels in the feature map under each channel. Then, matrix multiplication is performed on the transpose of Q and K, and the calculation result is softmax normalized to obtain the specific spatial attention weight S∈R N×N , the specific formula is shown in (1):

[0049]

[0050] Among them S ji Indicates that i in the feature vector th Position pair j th The influence of position, the more similar the feature representations between two positions are, the greater the correlation between them. i ,K j Represents position i in the query feature vector and position j in the queried feature vector.

[0051] At the same time, the shape of the feature vector V is changed to R C×N , C, N have the same meaning as above. Then do matrix multiplication between S and V, and convert the final calculated eigenvector shape into R C×H×W Finally, the calculated eigenvector is added to the original eigenvector A element by element to prevent the gradient from disappearing. The specific formula is shown in (2):

[0052]

[0053] E j Represents the position j in the final result feature vector, V i Represents position i in the content feature vector, A j Represents the position j in the original eigenvector.

[0054] From formula (2), it can be inferred that the feature of each position in E is the sum of all position features and the original features. Therefore, it has a global context view and selectively aggregates context based on the spatial attention map. Similar semantic features achieve reciprocity, thereby improving intra-class compactness and semantic consistency.

[0055] refer to Figure 4 , the second path in the position attention module, given a feature vector A∈R C×H×W, where C represents the number of feature vector channels, H and W represent the length and width of the feature map, and R represents the feature set. The difference from the first method is that when calculating the similarity between feature maps, simple matrix multiplication is no longer used, but the cosine similarity calculation formula is used. The specific formula is shown in (3):

[0056]

[0057] Where A and B represent eigenvectors, and || || represents the two-norm operation.

[0058] Copy the eigenvector A three times to get Q, K, V, and {Q, K, V}∈R C×H×W , and then transform the shape of Q, K, V into R C×N , where N = H × W, represents the number of feature map pixels under each channel. The cosine similarity Sim between the transposes of Q and K is calculated according to formula (3) i,j ∈R N×N , Sim i,j Indicates that i in the feature vector th Position and j th The similarity between the two pixels. i,j Perform matrix multiplication between V and the final calculated feature vector, activate it through the ReLU function and transform it into R C×H×W Then, an element-by-element addition operation is performed between the original feature vector A to prevent the gradient disappearance problem. The specific formula is shown in (4):

[0059] E i =f(Sim i ×V i )+A i (4)

[0060] Where f(·) function represents the ReLU activation function. According to formula (4), the feature of each position in E is the sum of all similar features and the original features. Therefore, we can pay more attention to the similar parts of the feature vector. i Represents the position i in the final result feature vector, V i Represents position i in the content feature vector, A i Represents the position i in the original eigenvector.

[0061] After the initial feature vector A is calculated by the dual-path position attention module, the final calculation results of the two paths are added element by element in the channel dimension to obtain the final feature vector containing position information.

[0062] In the channel attention module, attention is calculated in two ways, and the input feature vector is input into the two ways respectively. In the input feature vector, each channel includes the pollen color, texture, contour features, etc. extracted by the neural network.

[0063] In the first path, the input eigenvector is first multiplied by a matrix multiplication operation to obtain the attention matrix, and then the attention matrix and the input eigenvector are multiplied and summed to obtain the final output eigenvector of the path.

[0064] In the second path, the input feature vector is first divided into two parts. The first part uses maximum pooling and the second part uses average pooling. The cosine similarity of the two pooling results is calculated to obtain the similarity attention weight matrix. The similarity attention matrix is ​​used to perform matrix multiplication with the input feature vector after the previous bisection. Finally, the calculation results are spliced ​​together to obtain the final output feature vector of this path.

[0065] The final output vector obtained from the two calculations and the initial input feature vector are added element by element to obtain the final output result of the channel attention module. By combining the channel attention mechanism, the neural network can learn the features that have a more significant impact on the pollen classification results.

[0066] refer to Figure 5 , the channel attention module also adopts a two-way parallel attention mechanism. For the first way, the channel attention is calculated directly through the given feature vector A, where A∈R C×H×W , only perform the transformation operation on A to make it R C ×N , where C, H, W, and N represent the number of channels of the feature vector, the length and width of the feature map, and the number of pixels in each feature map. Perform matrix multiplication on the transpose of the feature vectors A and A, and finally perform softmax normalization on the calculated results to obtain X∈R C×C The specific formula is shown in (5):

[0067]

[0068] x ji Indicates that i in the feature vector th Channel pair j th The more similar the feature representations of two channels are, the greater the correlation between them. In addition, the calculated channel attention matrix X is multiplied by A, and the final result is transformed into R C×H×W , and finally perform element-by-element summation with the undeformed eigenvector A to obtain the final result E∈R C×H×W , the specific calculation is shown in formula (6):

[0069]

[0070] Formula (6) shows that the final feature of each channel is the sum of all channel features and the original features. It models the original reservoir semantic dependency between feature maps, which helps to improve the feature discrimination ability.

[0071] refer to Figure 6 , the second path in the channel attention module, for a given feature vector X∈R C×H×W , where C, H, and W represent the number of channels connected to the features and the length and width of the feature map respectively. First, the feature vector is divided into G groups along the channel dimension, X = [X1, X2, ..., X G ],X k ∈R C / G×H×W , and then each set of eigenvectors X k Divided into two groups along the channel direction, X k1 ,X k2 ∈R C / 2G×H×W For the first group, first X k1 Adaptive average pooling is used, and trainable parameters are added to control scaling and shifting. After the sigmoid activation function, matrix dot multiplication is performed with the original feature vector. The specific calculation is shown in formula (7):

[0072] X' k1 =σ(F(s))·X k1 =σ(W1s+b1)·X k1 (7)

[0073] where W1∈R C / 2G×1×1 and b1∈R C / 2G×1×1 is a trainable parameter, and s represents the result after adaptive average pooling. k1 represents the first set of result feature vectors, σ represents the sigmoid activation function, W1 represents the learnable scaling parameter, s represents the feature vector after adaptive average pooling, b1 represents the learnable displacement parameter, X k1 represents the first set of original eigenvectors.

[0074] For the second group, first k2 Adaptive max pooling is used, and trainable parameters are added to control scaling and shifting. After the sigmoid activation function, matrix dot multiplication is performed with the original feature vector. The specific calculation is shown in formula (8):

[0075] X' k2 =σ(H(s))·X k2 =σ(W2s+b2)·X k2 (8)

[0076] where W2∈R C / 2G×1×1and b2∈R C / 2G×1×1 is a trainable parameter, and s represents the result after adaptive maximum pooling. k2 represents the second set of result feature vectors, σ represents the sigmoid activation function, W2 represents the learnable scaling parameter, s represents the feature vector after adaptive maximum pooling, b2 represents the learnable displacement parameter, X k2 represents the second set of original eigenvectors.

[0077] After performing the pooling operation on the two groups, k1 ,X' k2 Perform the transformation operation to make it R C / 2G×N , and then use formula (3) to calculate the eigenvector X' k1 ,X' k2 The cosine similarity between the two is calculated by using the ReLU activation function and transforming it into R C / 2G×H×W , get the final channel attention weight S i,j , which means i in the feature vector th Channel and j th The similarity between channels. Finally, the attention weights are used to compare with the initial feature vectors X in the two branches. k1 ,X k2 Perform matrix multiplication and concatenate the results along the channel dimension, expressed as: X' k =[X' k1 ,X' k2 ]∈R C / G×H×W After the G groups perform the same operation, all groups are concatenated to obtain the final feature vector X' combined with the attention mechanism k ∈R C ×H×W .

[0078] After the initial feature vector A is calculated by the dual-path channel attention module, the final calculation results of the two paths are added element by element in the channel dimension to obtain the final feature vector containing channel information.

[0079] The principle of the present invention can be described as follows:

[0080] This paper adds an attention module to the original DenseNet basic network structure. This module reconstructs the features learned by the neural network to accurately classify pollen images. The main principle of the attention module is to calculate the attention weights between feature vectors and then perform a weighted sum of the attention weights with the original feature vectors, so that the model can focus more on the pollen body.

Claims

1. A pollen image classification method based on attention mechanism and convolutional neural network, characterized in that: include: Step 1: Construct a pollen image dataset; Step 2: Construct a pollen image classification model. The pollen image classification model uses Dense Net as the basic network structure, which includes four Dense Blocks with the same structure and three Transition Layers with the same structure. An attention module is added between the third Dense Block and the third Transition Layer; the attention module consists of a position attention module and a channel attention module; The position attention module uses a two-way attention mechanism to capture the spatial correlation between two positions in the feature vector. For the feature of a certain position, it updates it by adding an attention weight to the original feature and adding it to the original feature, where the attention weight is determined by the feature similarity between the two corresponding positions. The channel attention module uses a two-way attention mechanism to capture the channel dependency between any two channels and updates each channel using the weighted sum of all channels; Finally, the outputs of the attention module and the channel attention module are added and fused; The position attention module calculates attention in two ways, and inputs the input feature vector into the two ways respectively: in the first way, first perform convolution operation on the input feature vector to obtain three feature vectors of the same size, use matrix multiplication to calculate two of the feature vectors to obtain the attention matrix, and then use the attention matrix and the third feature vector to perform multiplication and summation operation to obtain the final output feature vector of the path; in the second way, first perform cosine similarity calculation on the input feature vector to obtain the similarity attention matrix, use the similarity attention matrix and the input feature vector to perform multiplication and summation operation to obtain the final output feature vector of the path; The final output feature vector calculated from the two paths and the initial input feature vector are added element by element to obtain the final output result of the position attention module; The channel attention module calculates attention in two ways, and inputs the input feature vector into the two ways respectively: in the first way, the input feature vector is first matrix multiplied to obtain the attention matrix, and then the attention matrix and the input feature vector are multiplied and summed to obtain the final output feature vector of the path; Second Road In [15], the input feature vector is first divided into two parts. The first part uses maximum pooling and the second part uses average pooling. The cosine similarity of the two pooling results is calculated to obtain the similarity attention matrix. The similarity attention matrix is ​​used to perform matrix multiplication with the input feature vector after the division. Finally, the calculation results are spliced ​​together to obtain the final output feature vector of the path. The final output vector calculated from the two paths and the initial input feature vector are added element by element to obtain the final output result of the channel attention module; Step 3, using the pollen image dataset to train the pollen image classification model; Step 4: Use the trained pollen image classification model to classify new pollen images.

2. The pollen image classification method based on attention mechanism and convolutional neural network according to claim 1 is characterized in that: In step 1, pollen images are captured using a SU 8220 field emission scanning electron microscope. Each pollen image is a grayscale image in a BMP format with the same size. Each pollen image contains only one complete pollen grain, and some pollen images contain noise.

3. The pollen image classification method based on attention mechanism and convolutional neural network according to claim 1 is characterized in that: The pollen image classification model includes: Convolutional layer, Dense Block1, Transition Layer1, Dense Block2, Transition Layer2, Dense Block3, Transition Layer3, Dense Block4, Transition Layer1, Linear Layer, and Softmax layer; The convolution layer extracts preliminary features from the pollen image; each dense block further extracts features; each transition layer compresses the model, reduces the channel dimension in the features, reduces the number of parameters, and prevents parameter explosion; the linear layer classifies the pollen image according to the learned features.

4. The pollen image classification method based on attention mechanism and convolutional neural network according to claim 3 is characterized in that: The features extracted by the convolutional layer are: Low-dimensional features including color features, texture features, and contour features; and high-dimensional features including category semantic information.

5. The pollen image classification method based on attention mechanism and convolutional neural network according to claim 1, characterized in that: Each of the Dense Blocks contains a different number of Dense Layers, each of which is composed of a normalization layer, an activation function layer, and a convolutional layer. In each Dense Block, the input of the first Dense Layer is the output of the previous layer, and the input of the Dth Dense Layer is the output of the first to D-1th Dense Layers in the Dense Block, where D>1; each Transition Layer sequentially includes a normalization layer, an activation function layer, a convolutional layer, and a pooling layer.

Images