Pollen classification method based on Swin Transform

Abstract

The invention discloses a pollen classification method based on Swin Transform, and the method comprises the steps: obtaining a pollen image, and carrying out the marking of the pollen image; constructing a classification model, wherein the classification model comprises a bar convolution scaling module, a backbone network, a full connection layer and a classification head; the bar convolution scaling module is used for scaling the input pollen image; the backbone network is divided into four cascaded stages based on a Swin Transformer encoder, and is used for collecting pollen features in a pollen image, and in the first stage and the second stage, the input features of the stages respectively pass through a window multi-head self-attention module and a large-view space bilateral attention module, are fused and then pass through a shift window multi-head self-attention module, and the pollen features in the pollen image are acquired through the shift window multi-head self-attention module. Obtaining output features of the stages; in the fourth stage, final pollen characteristics are output; the full connection layer and the classification head perform classification based on the final pollen features; and training the classification model, and performing pollen classification by using the trained classification model.

Description

Technical Field

[0001] The invention belongs to the technical field of image classification, and in particular relates to a pollen classification method based on Swin Transformer. Background Art

[0002] At present, pollen identification mainly relies on manual operation by experienced scientific researchers. The morphological similarities between pollen particles are easy to confuse. When comparing pollen classification data from different researchers, differences in standards and protocols may cause inconsistent classifications. At the same time, the experience and subjective judgment of pollen analysts affect the classification results. Traditional automated image analysis algorithms are difficult to identify complex or subtle pollen morphological features. This creates certain difficulties for pollen image recognition, such as SIFT (Scale Invariant Feature Transform) and HOG (Histogram of Oriented Gradients). When these methods process pollen images with complex shapes and subtle textures, they have problems such as limited feature extraction capabilities, sensitivity to noise, and poor generalization capabilities. Summary of the invention

[0003] In order to overcome the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a pollen classification method based on SwinTransformer to identify pollen types more accurately and efficiently.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A pollen classification method based on Swin Transformer comprises the following steps:

[0006] Step 1, obtain pollen images and mark them;

[0007] Step 2, constructing a classification model, the classification model includes a strip convolution scaling module, a backbone network, a fully connected layer and a classification head; the strip convolution scaling module is used to scale the input pollen image; the backbone network is based on a Swin Transformer encoder and is divided into four cascaded stages for collecting pollen features in the pollen image, wherein in the first stage and the second stage, the input features of the stage are respectively passed through a window multi-head self-attention module and a large field of view spatial bilateral attention module, and then fused and passed through a shifted window multi-head self-attention module to obtain the output features of the stage; the fourth stage outputs the final pollen features; the fully connected layer and the classification head perform classification based on the final pollen features;

[0008] Step 3: training the classification model and using the trained classification model to classify pollen.

[0009] In one embodiment, in step 1, the pollen image is obtained by:

[0010] Pollen grain samples were collected and photographed using a field emission scanning electron microscope to obtain image data, which were then manually annotated to construct a pollen dataset.

[0011] In one embodiment, the strip convolution scaling module is obtained by combining 1D convolution and 1D depth-separable convolution. For the input pollen image, the strip convolution scaling module sequentially performs the following processes:

[0012] Change the dimension through a 1×1 convolution block, then pass through a 1×14 depthwise separable convolution block and a 14×1 depthwise separable convolution block in sequence to capture long-range contextual dependencies from the horizontal and vertical axes of the image, respectively, and then restore the dimension through a 1×1 convolution block;

[0013] The features after dimensionality reduction and the original input pollen image are fused, and a 4×4 convolution block with a stride of 4 is passed to obtain a pollen image with reduced size but increased channels as the input of the backbone network.

[0014] In one embodiment, the large-view spatial bilateral attention module includes two branches, the first branch is a dual-scale channel attention branch, which uses two receptive fields of different sizes to sample input features, and concatenates the sampled features and performs maximum pooling and average pooling respectively, and then concatenates them again, calculates the channel score, multiplies the features sampled from the two receptive fields, adds the multiplication results, and finally multiplies the addition result with the input feature to obtain the output feature of the first branch;

[0015] The second branch is the spatial attention branch, which takes the features sampled from the first receptive field as input, first performs global average pooling, then calculates the spatial score, multiplies it with the input of the second branch, and obtains the output features of the second branch;

[0016] The output features of the first branch and the second branch are added together to obtain the output features of the large-viewing space bilateral attention module.

[0017] In one embodiment, the dual-scale channel attention branch, the input features are first scaled by group convolution to complete the feature extraction of the first receptive field, and then the second receptive field is used for feature extraction. The feature maps of two scales obtained by the two receptive fields are respectively subjected to 1×1 convolution calculation, and then spliced ​​in the channel dimension. The spliced ​​features are subjected to maximum pooling and average pooling respectively, and then spliced ​​in the channel dimension again, and then two channel scores are calculated, respectively multiplied with the features sampled from the two receptive fields, the multiplication results are added and then subjected to 1×1 convolution to achieve dimensional matching, and finally the added result is multiplied by the input feature.

[0018] In one embodiment, the size of the grouped convolution is 5×5, so as to realize feature extraction of a 5×5 receptive field; the features obtained by the grouped convolution are passed through a convolution block with a convolution kernel of 7×7 and a void rate of 3, so as to realize feature extraction of a 19×19 receptive field; the two channel scores are calculated, which is realized by a 7×7 convolution block and a sigmoid function;

[0019] The spatial attention branch, after global average pooling, undergoes a 1×1 convolution to change its dimension, is activated by a ReLu function, and then restores its dimension by a 1×1 convolution, and calculates the spatial score by a sigmoid function.

[0020] In one embodiment, in the second, third and fourth stages of the backbone network, the input features of the stage are first passed through a triple attention merging and downsampling module before feature extraction of the corresponding stage; the triple attention merging and downsampling module has three branches, the first branch is used to calculate the channel attention weight of the spatial dimension W, the second branch is used to calculate the channel attention weight of the spatial dimension H, and the third branch is used to calculate the spatial attention weight of the channel dimension C, and attention is given to features of three dimensions through the three branches.

[0021] In one embodiment, the triple attention merging and downsampling module extracts spatial pixels at equal intervals from the input features, recombines them to obtain multiple features and splices them in the channel dimension to generate a feature vector with a dimension of C×W×H, rotates the feature vector at corresponding angles and sends it to three branches to calculate the attention weights from the three angles of spatial dimension, spatial dimension and channel dimension; then, multiply the features of different rotation angles with the attention weights correspondingly to extract effective features of three different dimensions; finally, fine-tune the three effective features through 1×1 convolution, add the fine-tuned features, and then reduce the dimension through standardization and 1×1 convolution to obtain the output features.

[0022] In one embodiment, the window multi-head self-attention module performs the following steps:

[0023] The input image is divided into multiple small windows, and a new window is obtained in the next layer by shifting the patch position as a whole by 1 / 2 block size.

[0024] In one embodiment, the shifted window multi-head self-attention module performs the following steps:

[0025] In the first layer, the image is split into windows in a standard way and multi-head self-attention is applied within each window;

[0026] Starting from the second layer, the windows of each layer are rotated or offset relative to the window positions of the previous layer.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] In order to solve the problem of noise in most pollen backgrounds in the data set, the present invention integrates the good sparse spatial sampling capability of deep separable convolution and the global modeling capability of Swin Transformer to construct a Swin Transformer network structure based on spatial and channel attention. Through deep separable convolution, deep convolution is first performed on the key sampling points around each pixel, and then point-by-point convolution is performed to integrate information, thus achieving effective and sparse attention to local areas. This method not only enhances the model's ability to model local features of pollen, but also significantly reduces the computational complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the classification model of the present invention.

[0030] Figure 2 Schematic diagram of the self-attention principle.

[0031] Figure 3 Schematic diagram of the Transformer model principle.

[0032] Figure 4 Schematic diagram of the Swin Transformer model structure.

[0033] Figure 5 This is the Patch Partition structure diagram.

[0034] Figure 6 This is the Patch Merging structure diagram.

[0035] Figure 7 This is the process of outputting a feature map using a convolutional layer with a 3×3 convolution kernel containing 4 filters.

[0036] Figure 8 It is the process of outputting feature maps using Depthwise convolution.

[0037] Fig. 9 Schematic diagram of the strip convolution scaling module of the present invention.

[0038] Fig.10 Schematic diagram of the large-field-of-view spatial bilateral attention module of the present invention.

[0039] Fig.11 Schematic diagram of the window multi-head self-attention module of the present invention.

[0040] Fig.12 Schematic diagram of the shift window multi-head self-attention module of the present invention.

[0041] Fig.13 Schematic diagram of the triple attention mechanism.

[0042] Fig.14 Schematic diagram of the triple attention merging and downsampling module. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0044] In order to better realize pollen classification, the present invention provides a pollen classification method based on Swin Transformer, constructs a high-quality pollen data set of field emission scanning electron microscope, uses improved Swin Transformer as the backbone network of the classification model, extracts semantic features of pollen images from low-level to high-level and from concrete to abstract through a four-stage hierarchical design, and uses channel-related loss and distillation loss for optimization, so as to realize high-precision classification of pollen images under electron microscope. The main steps of the present invention can be described as follows:

[0045] Step 1: Obtain pollen images and annotate them.

[0046] In this step, pollen particle samples are collected and photographed using a field emission scanning electron microscope to obtain image data, which are then manually annotated to construct a pollen dataset. Specifically, this embodiment uses an ultra-high resolution field emission scanning electron microscope SU8220 to capture 80 types of original pollen images. Use a median filter to filter out the salt and pepper noise in the image, and then use a Gaussian pyramid and a Laplacian pyramid to extract the main information in the image. The formula is as follows:

[0047]

[0048] Among them, g(x,y) is the central element of the window, f(x+i,y+j) is the pixel of the noisy original image, w(i,j) is the weighted value, and its window size is (2k+1)×(2k+1).

[0049] The central element of the sliding window is traversed in the entire image. If the sliding window exceeds the image, the exceeding part is padded with zero or padded with the average value of the sum of the elements in the frame except the central element. Then the median of the pixels of the entire sliding window elements is taken as the pixel of the central element, as follows:

[0050] g(x,y)=med{(f(xk,yl),(k,l∈W))}

[0051] Among them, the function of the med function is to obtain the median of a set of numbers.

[0052] After processing, the image is randomly rotated. In an image, all pixels rotate around a pixel. When the angle of rotation of the cell image is within 180 degrees, it is recorded as clockwise rotation, and when the rotation angle is between 180 degrees and 360 degrees, it is recorded as counterclockwise rotation. As shown in the following formula:

[0053]

[0054] Then, erosion / expansion is performed in the cell image. The expansion operation can thicken the critical line between the cytoplasm and the nucleus, while the erosion operation thins the critical line. The convolution kernel size used in this embodiment is 3. The expansion operation is to convolve the convolution kernel with the entire feature map in sequence. After the convolution calculation, if the result of the calculation is less than 224, the value of the central pixel is set to 0. After each erosion or expansion operation, a new cell image will be generated.

[0055] At the same time, in order to prevent the model from overfitting, the diseased cell image was stretched and compressed after the rotation, corrosion / expansion operations. The stretching coefficient was 1.2 and the compression coefficient was 0.8. After each stretching or compression operation, a new cell image was also generated.

[0056] Step 2: Build a classification model.

[0057] The classification model architecture of the present invention is as follows Figure 1 As shown in the figure, it mainly includes a strip convolution scaling module, a backbone network, a fully connected layer, and a classification head. Among them, the strip convolution scaling module is used to scale the input pollen image. The backbone network is based on the Swin Transformer encoder and combines the convolution operation to improve the pipeline structure of the model. The backbone network is used to collect pollen features in the pollen image. It is divided into four cascade stages to achieve hierarchical modeling of the input image and learn the rich semantic information of the pollen image from low-level to high-level, specific to abstract.

[0058] Transformer is a model for processing sequence information. A large number of self-attention mechanisms are used in the model. The principle of self-attention is as follows Figure 2 As shown, the model principle is as follows Figure 3 As shown, the input of the model is a sequence from x1 to x3, and each vector is multiplied by a matrix W to obtain four new vectors. Then multiply the obtained vectors by W q , W k and W v , and obtain the results of query q, key k and value v of different vectors. Next, each query q of the α vector must perform attention operation with the key k of other vectors. The specific calculation formula is as follows:

[0059]

[0060] After performing all the attention operations, all the obtained α vectors need to be normalized by Softmax(). Then each α vector after normalization is multiplied by each value v. The specific calculation formula is as follows:

[0061]

[0062] Calculate all output vectors b according to the above formula.

[0063] The model constructed by the present invention uses Swin Transformer as the backbone network. The model draws on the hierarchical feature mapping method of convolutional neural networks and constructs a network structure suitable for advanced visual tasks such as target detection and instance segmentation through 4-fold, 8-fold and 16-fold downsampling operations. Swin Transformer adopts a windowed multi-head self-attention mechanism, namely a window multi-head self-attention module, which can divide the downsampled image feature map into multiple non-overlapping small windows and independently perform multi-head self-attention operations in each window. Compared with the Vision Transformer method of applying self-attention across the entire image, this method aims to reduce the amount of calculation required to process large shallow feature maps. Although this reduces the computational burden, it also limits the information exchange between windows. To solve this problem, Swin Transformer introduces a shifted window multi-head self-attention (SW-MSA) mechanism, which realizes information exchange between adjacent windows in this way, thereby enhancing the information flow of the model and the effectiveness of the global context. Overall, Swin Transformer is a Transformer-based framework designed for visual tasks. It optimizes computational efficiency and model performance through a series of innovations, adopting a hierarchical Transformer structure, deformable window strategy, and cross-window connections to maintain the effectiveness of global context.

[0064] The model structure of Swin Transformer is as follows Figure 4 As shown in the figure. The entire model performs 4 downsamplings in total. During each downsampling process, the width and height of the feature map will be reduced by 1 times, while the number of channels will be doubled. Self-attention modules appear in pairs, and the data needs to be normalized before performing the operation. The previous module undergoes a window multi-head self-attention operation and then passes through the fully connected layer.

[0065] The superposition effect of the Patch Partition operation and the Linear Embedding operation is similar to the Patch Merging operation. Taking a feature map with 4×4 pixels as an example, Patch Partition first divides the image into 16 windows, and then flattens each window in the channel direction, with the number of channels per pixel being 3. Finally, Linear Embedding is used to adjust the number of windows, such as Figure 5 shown.

[0066] The structure of Patch Merging is as follows Figure 6 As shown in the figure, Patch Merging is essentially a downsampling operation. After the operation, the length and width of the image will be halved, and the number of channels will be doubled. Taking an image with a pixel value of 16 as an example, the image is divided into 4 parts, and the pixels at the same position in each part are extracted and spliced ​​to obtain a 2×2 feature matrix of 4 pixels. Next, normalization is performed in the channel direction. Then, a fully connected layer is passed to linearly map each pixel in the depth direction, and the feature map output by Patch Merging is obtained.

[0067] Compared with the ordinary self-attention formula, the relative displacement self-attention formula needs to add the offset matrix B after the query q and key k are calculated. The specific calculation formula of attention is as follows:

[0068]

[0069] The calculation process of the offset matrix is ​​introduced below. In computer vision tasks processed by Transformer, using position offset will improve the performance of the model. Although the absolute position offset can slightly improve the performance of the model on the ImageNet dataset, the performance of the model has declined in target detection and image segmentation tasks. The relative position offset can significantly improve the model in all three visual directions.

[0070] Taking the first pixel α(0,0) of the image as an example, assuming that the expression for the self-attention calculation of this pixel and all the pixels in the window is represented by α, after the Softmax function is calculated, the formula can be obtained: α (0,0) ,α (0,1) …α (0,45) ,α (0,46) …α (0,90) ,α (0,91) …α (0,14397 α (0,14398) ....... The specific calculation formula of Softmax is as follows:

[0071]

[0072] Because the value after Softmax calculation is in the range of 0 and 1, the method adopted by the present invention is to perform Softmax calculation on the results of self-attention calculation of pixels in different areas and then subtract 1000, so that the calculation results of different areas have no effect on the overall calculation.

[0073] In the traditional multi-head self-attention mechanism, the model calculates the attention scores of each element in the input sequence to all other elements, and then adjusts the representation of the elements according to these scores. This allows the model to capture the dependencies between any two elements in the input sequence, no matter how far apart they are.

[0074] In traditional Transformers, the self-attention mechanism focuses on the all-to-all relationship between all elements in the feature sequence, which results in a computational complexity that grows with the square of the sequence length. This is particularly inefficient for images, which can be viewed as a high-resolution sequence of elements. To improve computational efficiency, this means that each element only calculates the attention score with other elements in its window. This reduces the amount of computation, but also limits the model's ability to capture global dependencies across windows.

[0075] Swin Transformer constructs a hierarchical feature representation by repeating the Swin Transformer Block and Patch Merging steps to perform hierarchical feature extraction. Each Patch Merging reduces the spatial resolution of the feature map while increasing the feature dimension. The Swin Transformer Block structure is as follows:

[0076]

[0077] In the formula Indicates the signed message output by the SW-MSA module or the W-MSA module, h l It represents the signature message output by the MLP module, and LN represents the layer paradigm.

[0078] The complexity level of computing self-attention within a single window is significantly lower than that of the global self-attention mechanism. Assuming that each window consists of M×M patches, for an image consisting of h×w patches, the global MSA and W-MSA networks have the following computational complexity:

[0079] Ω(MSA)=4hwC 2 +2(hw) 2 C

[0080] Ω(W-MSA)=4hwC2 +2M 2 HkDJ

[0081] The complexity of the MSA algorithm is quadratically related to the number of image patches. As the number of patches increases, the complexity of MSA increases significantly. When the value of M is constant, the complexity of W-MSA is linear. Therefore, when the number of patches is large, the increase in complexity is relatively small. This feature enables the W-MSA network to process massive images more efficiently.

[0082] For a 128×128 pixel, three-channel color input image (size is 128×128×3), after passing through a convolution layer with 4 filters and 3×3 convolution kernels (the number of filters corresponds to the number of output channels, and the convolution kernel size is 3×3×3×4), 4 feature maps are finally output. The whole process is as follows Figure 7 As shown. At this time, the convolution layer has 4 filters, each filter contains 3 kernels, and the size of each kernel is 3×3.

[0083] For a 128×128 pixel, three-channel color input image (size 128×128×3), the depthwise convolution first undergoes the first convolution operation, and the number of convolution kernels is the same as the number of channels in the previous layer, that is, the channels and convolution kernels correspond one to one. After the operation, a three-channel image generates 3 feature maps, such as Figure 8 As shown in the figure, the size of the convolution kernel of the Pointwise convolution operation is 1×1×M, where M is the number of channels in the previous layer. The convolution operation here performs a weighted combination of the feature map of the previous step in the channel direction to generate a new feature map.

[0084] In the classification model of the present invention, in the first and second stages of the backbone network, the input features of the stage are respectively passed through the window multi-head self-attention module and the large-viewing space bilateral attention module, and then fused and passed through the shifted window multi-head self-attention module to obtain the output features of the stage; the fourth stage outputs the final pollen features; the fully connected layer and the classification head perform classification based on the final pollen features.

[0085] This step is the core part of the present invention, and each module involved is further introduced in detail as follows.

[0086] 1. Strip convolution scaling module

[0087] When extracting features from an image, in order to reduce the amount of network calculations, the image is usually reduced in size and the number of channels is changed before entering the feature extraction network. The Patch Partition in Swin Transformer uses a common convolution with a convolution kernel of 4 and a step size of 4 to achieve image scale scaling. For electron microscope pollen images, pollen occupies a large area of ​​the image. When scaling in this way, the neighborhood information of the target will be weakened, and some detail information will be lost inadvertently. Therefore, in order to fully capture the target contextual semantic relationship at the shallow level of the network, the present invention designs a strip convolution scaling module (SCSM) that combines 1D convolution and 1D depth-separable convolution to achieve image scale scaling.

[0088] 1D convolution is mainly used to extract features from time series data with only one dimension. Assume that the input shape of 1D convolution is (n, m), where n means there are n continuous time series data and each time series data has m features. The convolution kernel of 1D convolution only needs to specify one dimension, and the depth of the convolution kernel is automatically consistent with the depth of the input layer. The convolution kernel size of 1D convolution is (k, m), which means that the convolution kernel covers k continuous time series data. Assuming the number of convolution kernels is f, the padding type is "same" (padding is only for the time series dimension), and the step size is 1, then each convolution kernel can obtain a vector of length n after 1D convolution of the input layer, and f convolution kernels can obtain f vectors of length n, so the final output layer shape is (n, f).

[0089] The essence of convolution is multiplication and addition operation. Assume that the weight of each position of the convolution kernel is w ij , the corresponding convolution area has a value of x at each position ij , the output value of each convolution area after convolution is y, and b is the offset. The calculation formula of 1D convolution is as follows:

[0090]

[0091] The convolution kernel of 2D convolution needs to specify two dimensions, and the depth of the convolution kernel is automatically consistent with the depth of the input layer. Assuming the 2D convolution input shape is (w, h, d), the 2D convolution kernel size is (k1, k2, d), the number of convolution kernels is f, the padding type is "same", and the step size is 1, the output feature map shape is (w, h, f).

[0092] In order to fully capture the contextual semantic relationship of shallow network targets, reference Fig. 9 As shown, for the input pollen image, the strip convolution scaling module of the present invention sequentially performs the following processes:

[0093] First, the feature map x∈R H×W×3 Input to 1×1 convolution for dimension enhancement to generate an intermediate attention map a∈R H ×W×24 . This graph then undergoes a 1D strip convolution of 1×14 on the horizontal axis, followed by a 1D strip convolution of 14×1 on the vertical axis. Finally, it undergoes a 1×1 convolution for dimensionality recovery, producing a finer feature map a∈R H×W×24 , and then connected to the input feature X through a residual connection to obtain the final output Y. This enhances the generalization ability of the network and prevents the gradient from disappearing. The whole process is followed by a 4×4 convolution to achieve dimensionality scaling. The whole process can be expressed as:

[0094] y = Conv2D4(Conv2D1(Conv1D h (Conv1D w (Conv2D1(x))))+x)

[0095] Where x∈R H×W×3 represents the input image features, Represents the output image features after dimensionality reduction, Conv2D1 represents a 2D ordinary convolution with a convolution kernel of 1×1, and Conv1D w Represents a 1D strip convolution with a horizontal calculation of the convolution kernel of 1×14, Conv1D h It represents a 1D strip convolution with a convolution kernel of 14×1 and vertical calculation, and Conv2D4 represents a 2D ordinary convolution with a convolution kernel of 4×4 and a stride of 4.

[0096] According to the above process, the strip convolution scaling module of the present invention has two advantages: first, the use of 1×14 and 14×1 1D convolutions can effectively capture long-range context dependencies from the horizontal and vertical axes of the image respectively; second, the entire process is implemented using small convolution kernels and 1D depthwise separable convolutions. Compared with directly using 2D large convolution kernels, this method reduces the parameters of 14×14 depthwise classifiable convolutions by half.

[0097] 2. Large-field spatial bilateral attention module.

[0098] The present invention introduces a strip convolution scaling module in the Patch Partition of Swin Transformer, and combines the unique window attention mechanism and cross-window connection of Swin Transformer to reduce the computational complexity of the model while taking into account good local attention. Since the model of the present invention is mainly used for pollen classification, the process of Swin Transformer extracting features adopts the window self-attention of the divided window. Although there is an operation of shifting the window, for electron microscope pollen images, the rough dimension reduction in the early stage (4 times downsampling to reduce the feature map) will lose the detailed information to distinguish the pollen, and the later window self-attention does not have a global view, and pollen with similar local features are easily confused. Therefore, a large-view spatial bilateral attention module (Large-View Spatial Bilateral Attention Module, LSA) is designed.

[0099] refer to Fig.10 The large-view spatial bilateral attention module includes two branches. The first branch is a dual-scale channel attention branch, which provides large-view spatial channel attention. It uses two receptive fields of different sizes to sample the input features, and concatenates the sampled features and performs maximum pooling and average pooling respectively, and then concatenates them again to calculate the channel score, which is multiplied by the features sampled from the two receptive fields, and the multiplication results are added. Finally, the added result is multiplied by the input feature to obtain the output feature of the first branch.

[0100] Specifically, in this branch, the features sent to Swin Transformer are respectively extracted from the small receptive field feature map and the large receptive field feature map of the original information through 5×5 convolution and 19×19 receptive field dilated convolution, and the large field of view spatial attention calculation is performed to make up for the deficiency of the original Swin Transformer global field of view. In addition, the small receptive field feature map after 5×5 convolution is also subjected to channel attention to obtain channel features to make up for the deficiency of Swin Transformer in pollen channel feature extraction.

[0101] More specifically, the input image feature x1′∈R C×H×W First, the scale is changed by 5×5 group convolution, and then the receptive field of 19×19 formed by the convolution kernel of 7×7 and the void rate of 3 is used for feature extraction to obtain the feature map x2′∈R C ×H×W , expressed as:

[0102] x1′=GroupConv5(x′)

[0103] x2′=cat dim＝(Conv2D1(x1′)+Conv2D1(DWConv(x1′)))

[0104] At this time, two scale feature maps are formed after two receptive fields of different sizes are extracted, and 1×1 convolution calculation is performed respectively. Then the two features are spliced ​​in the channel dimension. After maximum pooling and average pooling, the two features are spliced ​​in the channel dimension. After 7×7 convolution and sigmoid function, two channel scores are calculated, and multiplied with the previous features respectively to obtain two feature maps x3′∈R after channel attention calculation. 2×H×W , and then add them together, and then perform 1×1 convolution to achieve dimension matching. The above formula can be described as:

[0105] x3′=sigmoid(Conv2D7(cat dim (GAP(x2′)+GMP(x2′))))

[0106] x4′,x5′=split dim＝1 (x3′)

[0107] y1′=(Conv2D1(x4′·Conv2D1(x1′)+x5′·Conv2D1(DWConv(x1′))))x′

[0108] Where x1′∈R C×H×W Represents the input features of the LSA module, y1′∈R C×H×W Represents the features output after the attention of the upper channel, GroupConv5 represents the grouped convolution with a convolution kernel of 5, x1' represents the features after the grouped convolution; Conv2D1 represents the 2D convolution with a convolution kernel of 1, DWConv represents the deep separation dilated convolution with a convolution kernel of 7 and a dilation rate of 3, x2'∈R C×H×W It represents the features after concatenation in the first dimension after the formula operation; Conv2D7 is a common convolution with a convolution kernel of 7, GAP is global average pooling, GMP is global maximum pooling, Represents the features after the formula operation and concatenation in the first dimension; Depend on Obtained after separation in the first dimension; y1∈R C×H×W Indicates that the feature is calculated after the formula operation.

[0109] The second branch is the spatial attention branch, which takes the features sampled from the first receptive field as input, first performs global average pooling, then calculates the spatial score, multiplies it with the input of the second branch, and obtains the output features of the second branch.

[0110] Specifically, in the second branch, global average pooling is first performed, then the dimension is changed through 1×1 convolution, the ReLu function is activated, and then the dimension is changed through 1×1 convolution again. The sigmoid function is used to calculate the score and multiply it with the previous feature to obtain a feature map y2'∈R after spatial attention. C×H×W .

[0111] y2′=(sigmoid(Conv2D1(ReLu(Conv2D1(GAP(x1′))))))x1′

[0112] The output features of the first branch and the second branch are added together, and the output includes enhanced channel information and spatial information, that is, the output features of the large-view spatial bilateral attention module. The final output feature formula is expressed as:

[0113] Feature=y1′+y2′

[0114] Among them, GAP is global average pooling, Conv2D1 represents 2D convolution with a convolution kernel of 1, ReLu represents the activation function, sigmoid is the probability calculation function, and the spatial attention feature y2 is obtained after the formula operation.

[0115] The present invention is based on a large-view spatial bilateral attention module. In the early stage, 5×5 convolution and 19×19 receptive field hole convolution are used to extract small receptive field feature maps and large receptive field feature maps respectively, and spatial attention is performed so that the extracted feature maps contain feature information of different receptive fields. Then, channel attention is performed on the small receptive field feature map after 5×5 convolution to obtain important channel information. The local and global information are combined in the multi-scale pixel space to enhance the feature expression ability of the model. This method strengthens the focus on global and local attention, allowing the model to focus on key areas in the image more quickly, thereby accelerating the training and reasoning process.

[0116] 3. Window Multi-Head Self-Attention Module

[0117] The window multi-head self-attention module performs the following steps:

[0118] The input image is divided into multiple small windows, such as Fig.11 As shown in the figure, the feature map is divided into four 2×2 windows, and the 8×8 feature map in Layer 1 is divided into 2×2 blocks, each of which is 4×4. By shifting the patch position by 1 / 2 of the block size, a new window is obtained in the next layer, including 3×3 non-overlapping blocks. The moving window division method introduces connections between adjacent non-overlapping windows in the previous layer, greatly increasing the receptive field.

[0119] 4. Shift Window Multi-Head Self-Attention Module

[0120] The shifted window multi-head self-attention module addresses the limitations of window attention by rotating the position of the windows in each layer, as follows: In the first layer, the image is split into windows in a standard way, and multi-head self-attention is applied within each window. In order to overcome the limitation that self-attention within a window cannot capture cross-window dependencies, SwinTransformer introduces a rotating window strategy. Specifically, starting from the second layer, the windows of each layer are rotated (or offset) relative to the window position of the previous layer. Typically, this rotation is achieved by offsetting the window position by half the window size in the horizontal and vertical directions. The rotation details are as follows Fig.12 As shown in Figure 1, standard window splitting is applied at the first level without rotation. In subsequent levels, the window positions are rotated, usually offset by half the window size. This offset means that part of the elements of one window will appear in the adjacent window at the next level.

[0121] 5. Triple Attention Merging Downsampling Module

[0122] In terms of model optimization, the present invention improves the patch merging module to be suitable for Swin Transformer. The effective implementation of the hierarchical series structure of the Swin Transformer network depends on the patch merging module. This module acts as a connector for each level, changes the dimension of the feature to achieve downsampling, and is convenient for subsequent level calls. In the process of implementing the downsampling operation, the patch merging module divides the input feature map into several patches with equal distances, while in the patch merging of Swin Transformer, each 2×2 area is usually merged into a new patch, and finally a 1×1 convolution is used for dimensional adjustment. This process not only reduces the spatial dimension of the feature map, but also increases the number of feature channels to twice the original. The patch merging module does not consider the importance of different spatial positions and four channels, and simply gives all spatial features and channel features the same weight for fusion, which cannot effectively enhance the target features and weaken the background information.

[0123] A typical triple attention mechanism is as follows Fig.13 As shown in Figure 1, this is a new method to calculate attention weights by capturing cross-dimensional interactions using a three-branch structure. For the input tensor, triplet attention operates through rotation and then uses residual transformation to establish dependencies between dimensions and encode inter-channel and spatial information with negligible computational overhead.

[0124] On this basis, the present invention proposes a triple-attention consolidated downsampling module (TACD), referring to Fig.14As shown in the figure, it has three branches. The first branch is used to calculate the channel attention weight of the spatial dimension W, the second branch is used to calculate the channel attention weight of the spatial dimension H, and the third branch is used to calculate the spatial attention weight of the channel dimension C. The three branches give attention to the features of the three dimensions, improve the network's multi-angle understanding of the features, and make the feature extraction more comprehensive and in-depth.

[0125] The specific process of the triple attention merging downsampling module can be described as:

[0126] Input feature map x'∈R 1×4H×4W After that, the spatial pixels are extracted at equal intervals and recombined into multiple features, such as four feature maps x i ″∈R 4×H×W , concatenate in the channel dimension to generate a feature vector of dimension 4×W×H; then rotate the feature vectors correspondingly and send them to three branches, such as x1″∈R H×4×W , x2″∈R W×H×4 , x3″∈R 4×H×W , the attention weight is calculated from three perspectives: spatial dimension, spatial dimension and channel dimension. The formula is as follows:

[0127] x″ i =f1(x″)

[0128] x1″=f p (x i ″),x2=f p (x i ″),x3=x i ″

[0129] Next, the different rotation angle features are multiplied by the attention weights to extract the effective features x4″∈R in three different dimensions: 4×H×W ,x5″∈R 4×H×W ,x6″∈R 4×H×W , the formula is as follows:

[0130] x4″=f Sigmoid (DWConv7(cat dim＝0 (GAP(x1″),GMP(x1″))))*x1″

[0131] x5″=f Sigmoid (DWConv7(cat dim＝0 (GAP(x2″),GMP(x2″))))*x2″

[0132] x6″=f Sigmoid (DWConv7(cat dim＝0 (GAP(x3″),GMP(x3″))))*x3″

[0133] Finally, the three effective features are fine-tuned through 1×1 convolution, the fine-tuned features are added, and then the dimension is reduced by standardization and 1×1 convolution to obtain the output feature y″∈R 2×H×W .

[0134] The whole process uses triple attention to extract features, ensuring that scale transformation and dimension conversion can achieve downsampling while enhancing target features and weakening background information. The whole process can be expressed as:

[0135] y″=Conv1(f Norm (Conv1(f p (x4))+Conv1(f p (x5))+Conv1(f p (x6))))

[0136] Among them, x represents the input feature of the module, y represents the output feature, and x i It means extracting spatial pixels at equal intervals and recombining them into four feature maps; f p represents the dimension rotation and reorganization, x1, x2, x3 represent the input features of the three branches; GAP and GMP represent global average pooling and global maximum pooling respectively, cat dim＝0 Indicates concatenation in the 0th dimension, DWConv7 indicates a depth-wise separable convolution with a convolution kernel of 7; f Sigmoid is the probability calculation function, f Norm is a standard normalization function, and Conv1 represents a 2D convolution with a convolution kernel of 1. After a series of sequential calculations of the above formulas, the output feature y is obtained from the input feature x.

[0137] Based on the above description, in an embodiment of the present invention, in the second stage, the third stage and the fourth stage of the backbone network, the input features of the stage are first passed through a triple attention merging and downsampling module before feature extraction of the corresponding stage is performed.

[0138] According to the introduction of the above modules, the process of building a classification model in this step can be described as follows: First, use SwinTransformer to build a basic model. Second, introduce 1D convolution and 1D depth-separable convolution into the basic model to build a strip convolution scaling module. Third, the CBAM module learns the channel and space of the feature map to build a large-view spatial channel bilateral attention module. Fourth, apply the triple attention mechanism and combine Patch Merging to build a triple attention merging downsampling module.

[0139] Step 3: Classification model training.

[0140] Set parameters such as loss value and perform model training.

[0141] Step 4: Input the pollen image into the trained classification model to achieve pollen classification.

[0142] Using the data set constructed in step 1, the efficiency and accuracy of various models are checked, and the results are shown in the following table.

[0143] TABLE I

[0144] THE VARIOUS INDICATORS OF THE MODEL.

[0145]

[0146] It can be seen that Efficient-NetV2 performs best, achieving 95.1% accuracy, 94.7% recall, and 95.0% F1 value. The proposed model surpasses the mainstream benchmark in all indicators and achieves 96.4% accuracy, indicating that the model achieves high accuracy while maintaining low computational complexity. This performance is achieved by introducing SCSM, LSA, and TACD modules, which use dilated convolution to expand the receptive field and highlight important features through the channel attention mechanism, providing an effective way to enhance the contextual understanding and feature extraction efficiency of the model.

Claims

1. A pollen classification method based on Swin Transformer, characterized in that: The steps include: Step 1, obtain pollen images and mark them; Step 2, constructing a classification model, the classification model includes a strip convolution scaling module, a backbone network, a fully connected layer and a classification head; the strip convolution scaling module is used to scale the input pollen image; the backbone network is based on a SwinTransformer encoder and is divided into four cascaded stages for collecting pollen features in the pollen image, wherein in the first stage and the second stage, the input features of the stage are respectively passed through a window multi-head self-attention module and a large field of view spatial bilateral attention module, and then fused and passed through a shifted window multi-head self-attention module to obtain the output features of the stage; the fourth stage outputs the final pollen features; the fully connected layer and the classification head perform classification based on the final pollen features; Step 3: training the classification model and using the trained classification model to classify pollen.

2. The pollen classification method based on Swin Transformer according to claim 1, characterized in that: In step 1, the pollen image is obtained by: Pollen grain samples were collected and photographed using a field emission scanning electron microscope to obtain image data, which were then manually annotated to construct a pollen dataset.

3. The pollen classification method based on Swin Transformer according to claim 1, characterized in that: The strip convolution scaling module is obtained by combining 1D convolution and 1D depth-separable convolution. For the input pollen image, the strip convolution scaling module performs the following processes in sequence: Change the dimension through a 1×1 convolution block, then pass through a 1×14 depthwise separable convolution block and a 14×1 depthwise separable convolution block in sequence to capture long-range contextual dependencies from the horizontal and vertical axes of the image, respectively, and then restore the dimension through a 1×1 convolution block; The features after dimensionality reduction and the original input pollen image are fused, and a 4×4 convolution block with a stride of 4 is passed to obtain a pollen image with reduced size but increased channels as the input of the backbone network.

4. The pollen classification method based on Swin Transformer according to claim 1, characterized in that: The large-view spatial bilateral attention module includes two branches. The first branch is a dual-scale channel attention branch, which uses two receptive fields of different sizes to sample input features, and concatenates the sampled features and performs maximum pooling and average pooling respectively, and then concatenates them again to calculate the channel score, which is multiplied by the features sampled by the two receptive fields, and the multiplication results are added. Finally, the added result is multiplied by the input feature to obtain the output feature of the first branch; The second branch is the spatial attention branch, which takes the features sampled from the first receptive field as input, first performs global average pooling, then calculates the spatial score, multiplies it with the input of the second branch, and obtains the output features of the second branch; The output features of the first branch and the second branch are added together to obtain the output features of the large-viewing space bilateral attention module.

5. The pollen classification method based on Swin Transformer according to claim 4, characterized in that: In the dual-scale channel attention branch, the input features are first scaled by group convolution to complete the feature extraction of the first receptive field, and then the second receptive field is used for feature extraction. The feature maps of two scales obtained by the two receptive fields are respectively subjected to 1×1 convolution calculation, and then spliced ​​in the channel dimension. The spliced ​​features are subjected to maximum pooling and average pooling respectively, and then spliced ​​in the channel dimension again. Subsequently, two channel scores are calculated and multiplied with the features sampled from the two receptive fields respectively. The multiplication results are added and then subjected to 1×1 convolution to achieve dimensional matching, and finally the added result is multiplied by the input feature.

6. The pollen classification method based on Swin Transformer according to claim 5, characterized in that: The size of the grouped convolution is 5×5, which realizes the feature extraction of 5×5 receptive field; the features obtained by the grouped convolution are passed through a convolution block with a convolution kernel of 7×7 and a void rate of 3, which realizes the feature extraction of 19×19 receptive field; the two channel scores are calculated by a 7×7 convolution block and a sigmoid function; The spatial attention branch, after global average pooling, undergoes a 1×1 convolution to change its dimension, is activated by a ReLu function, and then restores its dimension by a 1×1 convolution, and calculates the spatial score by a sigmoid function.

7. The pollen classification method based on Swin Transformer according to claim 1, characterized in that: In the second, third and fourth stages of the backbone network, the input features of the stage are first passed through a triple attention merging and downsampling module before feature extraction of the corresponding stage; the triple attention merging and downsampling module has three branches, the first branch is used to calculate the channel attention weight of the spatial dimension W, the second branch is used to calculate the channel attention weight of the spatial dimension H, and the third branch is used to calculate the spatial attention weight of the channel dimension C. Attention is given to features of three dimensions through the three branches.

8. The pollen classification method based on Swin Transformer according to claim 7, characterized in that: The triple attention merged downsampling module extracts spatial pixels at equal intervals from the input features, recombines them to obtain multiple features and splices them in the channel dimension to generate a feature vector with a dimension of C×W×H. The feature vector is rotated at corresponding angles and sent to three branches to calculate the attention weights from the three angles of spatial dimension, spatial dimension and channel dimension. Then, the features of different rotation angles are multiplied by the attention weights to extract effective features of three different dimensions. Finally, the three effective features are fine-tuned through 1×1 convolution, the fine-tuned features are added, and then the output features are obtained by dimensionality reduction through standardization and 1×1 convolution.

9. The pollen classification method based on Swin Transformer according to any one of claims 1 to 8, characterized in that: The window multi-head self-attention module performs the following steps: The input image is divided into multiple small windows, and a new window is obtained in the next layer by shifting the patch position as a whole by 1 / 2 block size.

10. The pollen classification method based on Swin Transformer according to any one of claims 1 to 8, characterized in that: The shift window multi-head self-attention module performs the following steps: In the first layer, the image is split into windows in a standard way and multi-head self-attention is applied within each window; Starting from the second layer, the windows of each layer are rotated or offset relative to the window positions of the previous layer.

Images