A Transformer-based object detection method for electron microscope pollen images

Abstract

A Transformer-based electron microscope pollen image target detection method constructs an end-to-end pollen detection model based on the Swin Transformer. The model includes an encoder, a decoder, and a detection head. The convolutional layers in the encoder are replaced with deformable convolutional layers. A CBAM-based object query initialization module and a GRU-based cyclic update operator are introduced into the model. The initialization module includes a spatial channel attention module, a position encoding module, and an object query module. A pollen detection dataset is collected and constructed, and the model is trained. The trained model is used to perform pollen image detection. When detecting pollen targets, the present invention improves local attention to the image. By adding a channel spatial attention initialization fast module, the model convergence is accelerated. The bounding box is refined using a GRU-based cyclic update operator, which greatly improves the detection model's out-of-frame accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to pollen target detection, in particular to a Transformer-based electron microscope pollen image target detection method. Background Art

[0002] With the advancement of urbanization, the number of people suffering from pollen allergies is increasing. Hay fever has become a seasonal epidemic. Accurate and timely pollen reporting can better ensure the normal lives of those suffering from pollen allergies. Pollen detection is a fundamental technology for pollen reporting. Its goal is to accurately identify pollen grains in collected pollen sample images. Initially, pollen detection required experienced researchers to manually label pollen grains. This time-consuming and labor-intensive approach was inadequate for pollen reporting. With the rapid development of object detection, the accuracy of mainstream object detection algorithms, such as the YOLO and R-CNN series, has significantly improved. However, these algorithms require extensive manual intervention, prior knowledge, and NMS, making the entire detection framework complex, difficult to adjust and optimize, and challenging to deploy. To meet the requirements of accurate and real-time pollen reporting, pollen detection algorithms must strike a good balance between accuracy and efficiency. Summary of the Invention

[0003] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Transformer-based electron microscope pollen image target detection method, so as to improve the pollen detection accuracy without affecting the efficiency, and meet the requirements of pollen broadcasting accuracy and real-time performance.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A Transformer-based method for detecting objects in electron microscope pollen images includes the following steps:

[0006] Step 1: Build an end-to-end pollen detection model based on Swin Transformer. The model includes an encoder, a decoder, and a detection head. The encoder includes several Swin Transformer blocks, each of which includes two submodules: local window attention and cross-window attention. The decoder consists of several decoding layers, each of which includes an upsampling layer, a convolutional layer, and an attention mechanism.

[0007] Step 2: Replace the convolutional layer in the encoder with a deformable convolutional layer, thereby introducing deformable convolution into the model;

[0008] Step 3: Introduce a CBAM-based Object Query initialization module into the model. The CBAM-based Object Query initialization module includes a spatial channel attention module, a position encoding module, and an Object Query module. The original input image is added with attention by the spatial channel attention module and position encoding by the position encoding module. The sum of these is then input as the initialization value of the Object Query module. The Object Query module acts on the decoding layer to guide the decoder to generate features related to the pollen target.

[0009] Step 4: Introduce a GRU-based cyclic update operator into the model; the GRU-based cyclic update operator performs gradient backpropagation between adjacent decoding layers, so that each decoding layer refines the bounding box based on the prediction of the previous decoding layer;

[0010] Step 5: Collect and construct a pollen detection dataset and train the model;

[0011] Step 6: Use the trained model to perform pollen image detection.

[0012] Compared with the existing technology, the present invention improves the local attention of the image when detecting pollen targets. By adding a channel space attention initial fast module, the convergence of the model is accelerated. The bounding box is refined by using a GRU-based cyclic update operator to greatly improve the detection model's out-of-frame accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic flow chart of the present invention.

[0014] Figure 2 It is a schematic diagram of the deformable convolution principle of the present invention.

[0015] Figure 3 It is a schematic diagram of the principle of the Object Query initialization module based on CBAM of the present invention.

[0016] Figure 4 It is the prediction result of the present invention for a pollen target detection. DETAILED DESCRIPTION

[0017] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0018] With the development of urbanization, the number of people suffering from pollen allergies is increasing. To meet the requirements of accurate and real-time pollen reporting, pollen detection algorithms need to strike a good balance between detection accuracy and efficiency. This paper proposes a Transformer-based method for object detection in electron microscopy pollen images. By integrating the global modeling capabilities of deformable convolution and SwinTransformer, this method improves the local modeling capabilities of pollen while reducing the computational complexity of the model. Furthermore, a CBAM-based Object Query initialization module is introduced to improve the initialization efficiency of the decoder. Furthermore, a GRU-based cyclic update operator is introduced to refine the bounding box between decoding layers, thereby improving pollen detection accuracy.

[0019] like Figure 1 As shown, the present invention specifically includes the following steps:

[0020] Step 1: Build an end-to-end pollen detection model based on Swin Transformer.

[0021] This model is the basic model constructed by the present invention, mainly including an encoder, a decoder, and a detection head. The encoder includes several Swin Transformer blocks, each of which includes two sub-modules: local window attention and cross-window attention. Through the hierarchical combination of these two attention mechanisms, the Swin Transformer can effectively model global and local information in the image. The decoder mainly consists of several decoding layers, each of which includes an upsampling layer, a convolutional layer, and an attention mechanism module.

[0022] Functionally, the encoder primarily uses convolutional layers to extract initial features from the input image. It then extracts high-level image features from the resulting feature map through hierarchical segmentation and depth and width variations. These high-level features contain both global and local contextual information. The decoder is primarily responsible for decoding and reconstructing the high-level image features output by the encoder.

[0023] The detection head performs pollen detection based on the decoder's output features. Specifically, it consists of a global pooling layer, a fully connected layer, an activation function, a bounding box regression layer, and a classification layer. When performing object detection, the global pooling layer first performs global pooling (such as global average pooling or global max pooling) on ​​the feature map output by the encoder, reducing the feature map's height and width dimensions to 1 while retaining the channel dimension. Global pooling integrates the spatial information in the feature map to obtain a holistic feature representation. Subsequently, the fully connected layer maps the feature vector obtained after global pooling to the output space required for object detection, such as the predicted object category and bounding box information. These fully connected layers typically contain a certain number of neurons to learn the complex feature representations required for object detection tasks. Activation functions, such as ReLU (rectified linear units), are often inserted between fully connected layers to introduce nonlinearity, thereby increasing the model's expressive power and enabling it to learn more complex features. The bounding box regression layer is used to output bounding box information such as the coordinates and size of the target, while the classification layer converts the final output into a probability value to obtain the category probability distribution of the predicted target. The softmax activation function is usually used to convert the output into a probability value.

[0024] Step 2: Replace the convolutional layer in the encoder with a deformable convolutional layer, thereby introducing deformable convolution into the model.

[0025] In order to improve local attention, the present invention introduces deformable convolution in Swin Transformer. Swin Transformer can realize attention calculation between windows through sliding windows, but this will cause the local attention within the window to not be paid more effective attention. The use of the deformable convolution module can pay attention to only a part of the key sampling points around each pixel in the window, and shift the attention through a learnable offset, so that the local attention can be sparse and effective. During initialization, the Swin Transformer attention module applies almost the same attention weights to all pixels in each window, which greatly increases the amount of calculation. The fusion of the good sparse spatial sampling capability of deformable convolution and the global modeling capability of Swin Transformer can reduce the amount of calculation while paying better attention to local information.

[0026] The deformable convolution layer of the present invention adds 2D displacements to the mesh sampling positions of the standard convolution rule of the convolution layer, allowing the sampling grid to deform freely. Here, the displacements need to be learned from the features of the previous layer through another convolution layer, so the deformation is conditioned on the input features in a local, dense, and adaptive manner. In addition, the deformable convolution is lightweight, introducing a small number of parameters for displacement learning. It can easily replace the corresponding convolution layer or standard convolution module in the existing CNN, and the network can be easily learned end-to-end through standard backpropagation. The resulting network has intrinsic transformation parameters, and these parameters are learned directly from the data.

[0027] like Figure 2 As shown in the figure, the deformable convolution implementation process of the present invention is: for the convolution of the m×m convolution kernel, the output y(p0), R is the m×m position of the sampling point of the convolution kernel relative to the input x(p0), so the convolution result of the standard convolution is the "weighted sum" operation between the convolution kernel and the input feature, that is:

[0028]

[0029] In order to extract features that are more consistent with the target shape through convolution operation, deformable convolution is used to learn the displacement Δp of the sampling points of the convolution kernel. n , the convolution process is as follows:

[0030]

[0031] where p n is an enumeration of the positions listed in R, w(p n ) is p n The learnable weight of Δp n It can be a decimal, so the sampling position is determined by bilinear interpolation during the implementation process.

[0032] The introduction of deformable convolution operation does not require attention to all pixels in the window, but only to a few sparse key points around the current pixel, and uses a learnable offset to derive the position of the key points that need more attention. It can focus on more useful information with very little computation, improve the local modeling ability of the model, and pay more attention to small targets.

[0033] At the same time, using deformable convolutions instead of standard convolution operations improves the encoder's feature extraction capabilities. In the encoder, the input image is passed through a convolutional layer to extract features, and spatially deformable operations are introduced in the deformable convolution layer. This allows the convolution kernel to slightly offset the feature map within the receptive field, thereby better adapting to changes in the shape and position of the target. Therefore, the introduction of deformable convolutions can enhance the encoder's feature extraction capabilities. By introducing deformable operations in the encoder layer, the model can better adapt to changes in the shape and position of the target, thereby improving the accuracy of target detection.

[0034] Step 3: Introduce the CBAM-based Object Query initialization module into the model.

[0035] To accelerate model convergence, a CBAM module is used to focus on both the channel and spatial dimensions of the feature map, and the output features are used to initialize the Object Query in the decoder layer. This attention mechanism module leverages the global information of the feature map after the convolutional layer to dynamically model channel and spatial dependencies, improving the network's feature learning capabilities. This module, along two independent dimensions, enables the network to learn important features while suppressing unnecessary ones, allowing the network to selectively optimize based on feature importance. This attention mechanism module can be passed to the ObjectQuery for more efficient parameter initialization, accelerating model convergence.

[0036] like Figure 3 As shown in the figure, the CBAM-based Object Query initialization module of the present invention includes a spatial channel attention module, a position encoding module, and an Object Query module. The original input image is addressed by the spatial channel attention module, and a position encoding is added by the position encoding module. These two sums are then input as the initialization value for the Object Query module. The Object Query module acts on the decoding layer, guiding the decoder to generate features related to the pollen target.

[0037] The spatial channel attention module (Convolutional Block Attention Module, CBAM) of the present invention is a lightweight convolutional attention module, which combines the channel and spatial attention mechanism modules, including two sub-modules: Channel Attention Module (CAM) and Spatial Attention Module (SAM), which perform channel and spatial attention respectively. This not only saves parameters and computing power, but also ensures that it can be integrated into the existing network architecture as a plug-and-play module. The input image undergoes one-dimensional convolution through the channel attention module, and the convolution result is multiplied with the input image. Then, it undergoes two-dimensional convolution through the spatial attention module, and the output result is multiplied with the input image. In Figure 3 In the figure, the left side shows local window attention, and the right side shows cross-window (i.e., moving window) attention. W-MSA is the original module for calculating local window attention, and SW-MSA is the original module for calculating cross-window (moving window) attention. Both modules require ordinary convolution operations to calculate attention. Using deformable convolution instead of the original ordinary convolution, they become W-DMSA and SW-DMSA.

[0038] Specifically, the output feature map obtained by the spatial channel attention module contains the spatial and channel information of the original input image and has a strong correlation with the original image.

[0039] Channel attention module: The channel dimension remains unchanged and the spatial dimension is compressed. This module focuses on the meaningful information in the input image (the classification task focuses on what features divide the image into different categories). The input feature map passes through two parallel MaxPool layers and AvgPool layers, and the feature map is changed from C*H*W to C*1*1 in size. It then passes through the Share MLP module, in which it first compresses the number of channels to 1 / r times the original number, and then expands it to the original number of channels. It then passes through the ReLU activation function to obtain two activated results. These two output results are added element by element, and then a sigmoid activation function is used to obtain the output result of Channel Attention. This output result is then multiplied by the original image to return to the size of C*H*W.

[0040] Spatial Attention Module: This module maintains the spatial dimension while compressing the channel dimension. This module focuses on the location of the target. The output of Channel Attention is processed through max pooling and average pooling to generate two 1xHxW feature maps. These two feature maps are then concatenated using a concat operation, converted to a 1-channel feature map using a 7x7 convolution, and then processed through a sigmoid operation to obtain the Spatial Attention feature map. Finally, the output is multiplied by the original image to return it to the size of CxHxW.

[0041] The Object Query module of the present invention:

[0042] Used to model the relative position relationship between objects and transfer information between decoding layers;

[0043] Added to the attention mechanism to take into account the relative position information between objects;

[0044] The initialization value is optimized by back propagation to adapt it to different target layouts and relative position relationships.

[0045] The Object Query module of the present invention is used for the following two tasks:

[0046] Attention Weight Query: In each decoding layer, the Object Query module is used to query attention weights, which are applied to the feature maps output by the encoder to focus on specific areas related to each Object Query module. This helps the model focus on information related to each query object, thereby better understanding the semantic and spatial information of each object.

[0047] Detection Prediction: Within each decoding layer, the Object Query module generates detection predictions. By combining the ObjectQuery module with the decoder's features, it predicts the category and bounding box information for each query object. This allows the model to detect each object and output the final detection result.

[0048] Attention query and detection prediction are performed simultaneously at each decoder layer and are interrelated. By performing these operations at different layers, we can gradually focus on the features of different objects and perform target detection and prediction.

[0049] This step introduces a spatial channel attention module to learn the feature map, obtaining the channel and spatial information of the original image. The learned feature map information is highly correlated with the predicted bounding box. Adding this information to the position encoding as the initialization value for the object query reduces redundant operations in the decoder and accelerates training convergence.

[0050] Step 4: Introduce a GRU-based cyclic update operator into the model.

[0051] To improve the detection model's accuracy, the present invention applies a GRU-based cyclic update operator to the decoding layer. The GRU is a model that can process sequence data and is a type of recurrent neural network. It is also a variant of the LSTM. The GRU is better able to capture dependencies with large time step distances in time series. It controls the flow of information through learnable gates. Transformers are well-suited for time series prediction, and the GRU can be naturally extended to each decoding layer. Each decoding layer refines the bounding box based on the prediction of the previous layer, resulting in more accurate bounding box predictions after multiple iterations.

[0052] The present invention uses a GRU-based cyclic update operator to perform gradient backpropagation between adjacent decoding layers, so that each decoding layer can refine the bounding box according to the prediction of the previous decoding layer.

[0053] Gated recurrent neural networks (GRNs) were proposed to better capture dependencies between time series with large timestep distances. They control the flow of information through learnable gates. The Gated Recurrent Unit (GRU) is a commonly used GRN. It introduces the concepts of a reset gate and an update gate, modifying how hidden states are calculated in RNNs.

[0054] The input and output structure of GRU is the same as that of ordinary RNN. In the present invention, the input of the cyclic update operator is: the input x at time t t and the hidden layer state H at time t-1 t-1 , the hidden layer state contains the relevant information of the previous node. The output of the cyclic update operator is: the output y of the hidden node at time t t and the hidden layer state H passed to the next node t .

[0055] Through the last transmitted state H t-1 and the input x of the current node t Get two gate states:

[0056] Reset gate: r t =σ(x t W xr +H t-1 W hr +b r )

[0057] Update gate: z t =σ(x t W xz +H t-1 W hz +b z )

[0058] σ is the sigmoid function, which converts the data into a value in the range of 0-1. W and b are learnable weight matrices. W xr is the weight matrix of the reset gate at time t, W hr is the weight matrix of the reset gate at time t-1, b r is the bias matrix of the reset gate, W xz is the weight matrix of the update gate at time t, W hz is the weight matrix of the update gate at time t-1, b z To update the bias matrix of the gate, both weights and biases can be automatically optimized by the back-propagation algorithm.

[0059] The present invention introduces a cyclic update operator into the decoding layer, and the method is as follows:

[0060] Initialize the bounding box: In the first decoding layer, the predicted bounding box is initialized by the fully connected layer and activation function. The bounding box is the normalized coordinate relative to the input image, expressed as (center_x, center_y, height, width).

[0061] GRU Loop Update: In subsequent decoding layers, the GRU Loop Update operator is used to progressively refine the bounding box predictions. Specifically, for each decoding layer, the following steps are performed: A new feature map is sampled from the feature map using a deformable convolution based on the currently predicted bounding box. This new feature map is fused with the feature map generated by the attention mechanism to obtain features that are more focused on the target. The fused feature map and the bounding box obtained in the current round are input to the GRU Loop Update operator. The GRU Loop Update operator outputs a new bounding box prediction based on the current features and bounding box.

[0062] Multiple iterations: The decoder uses multiple iterations to gradually optimize the bounding box prediction. In each iteration, the bounding box prediction is updated. At the same time, the attention mechanism and deformable convolution help the model better utilize contextual information and local features, improving the accuracy and stability of the bounding box prediction.

[0063] Output bounding box prediction: After multiple rounds of iteration, the final bounding box prediction is obtained. These predictions will serve as the input of the next layer of decoder or as the final detection result.

[0064] The GRU cyclic update of the present invention involves performing the following steps for each decoding layer: using deformable convolution to sample a new feature map from the feature map based on the currently predicted bounding box. This new feature map is then fused with the feature map generated by the attention mechanism to obtain features that are more focused on the target. The fused feature map and the bounding box obtained in the current round are input to the GRU cyclic update operator. The GRU cyclic update operator then outputs a new bounding box prediction based on the current features and bounding box.

[0065] Each decoding layer refines the prediction box based on the prediction from the previous layer. Assuming there are D decoding layers, given the normalized bounding box predicted by the D-1th decoding layer, the Dth decoding layer refines the bounding box as

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] where d∈{1,2,3…,D}, It is predicted at the dth decoding layer, and the prediction heads of different decoding layers do not share parameters. The initial box is set to: To stabilize training, the gradient is passed only through Back propagation, and In the loop update operator, for the d-th decoding layer, the box predicted from the d-1-th decoding layer For the crisscross attention module of the d-th decoding layer, is used as a new reference point. Ultimately, the sampling position is made relevant to the center and size of the previously predicted box.

[0072] Step 5: Collect and construct pollen detection dataset and train the model.

[0073] In an embodiment of the present invention, a pollen detection dataset is constructed using electron microscopy to collect pollen from some mainstream pollen in Hohhot to train the end-to-end pollen detection model. The Swin Transformer uses a hierarchical construction method and a W-MSA structure to calculate self-attention within each window. The SW-MSA structure obtains attention between windows by moving the window. The Swin Transformer's self-attention within each window is not an efficient algorithm. The present invention introduces deformable convolution to focus only on a few key points around each pixel, rather than calculating attention for all points in the entire window, thereby enhancing the model's local modeling capabilities.

[0074] Step 6: Use the trained model to perform pollen image detection.

[0075] According to the above steps, a specific experimental process of the present invention is as follows:

[0076] A trained Transformer-based object detection model is used to perform inference on preprocessed electron microscope pollen images. During inference, the model generates bounding boxes and corresponding class predictions for each detected pollen object. Non-maximum suppression is performed on the object detection results obtained by model inference to remove overlapping bounding boxes and obtain the final detection results.

[0077] Experimental conditions: During prediction, the pollen categories in the predicted image cannot exceed those in the training dataset. During model inference, the same GPU used during training is selected to ensure efficient inference calculations.

[0078] Experimental effect description: Display the detection results of the electron microscope pollen image after model inference. Visualize the detected pollen target bounding box and mark the predicted category and probability, such as Figure 4 As shown, it can be seen that its prediction accuracy and precision are extremely high.

[0079] The present invention also provides a corresponding device, including:

[0080] Input module, calculation module, output module, communication module.

[0081] In the present invention, the input module can generally be a camera, such as a computer camera, a smartphone camera, or a camera associated with a wearable device. The computing module can obviously be implemented in various types of processors, such as those found in smartphones, VR devices, and other wearable devices. The output module is a display for displaying images, such as a computer monitor, a smartphone display, or the display of some VR devices.

[0082] Specifically, when the input module is a computer camera, the computing module is mounted on the computer processor, and the output module is the computer display. When the input module is a smartphone camera, the computing module is mounted on the smartphone processor, and the output module is the smartphone display. When the input module is a wearable device camera, the computing module can be mounted on the processor of a computer, smartphone, or smart VR device, and the output module can be the display of the computer, smartphone, or VR device.

[0083] During the testing process of the present invention, a 13th Gen Intel(R) Core(TM) i9-13900HX 2.20GHz processor with 16GB of memory and a GPU: Nvidia Tesla P100*2 with 12GB of video memory were used to meet the computing requirements.

[0084] In summary, this paper combines the global modeling capabilities of deformable convolution and the Swin Transformer to improve local pollen modeling while reducing model computational complexity. Furthermore, the CBAM module is introduced as an object query initialization module to improve decoder initialization efficiency. Furthermore, a GRU-based cyclic update operator is introduced to refine bounding boxes between decoding layers, improving pollen detection accuracy.

Claims

1. A Transformer-based electron microscope pollen image target detection method, characterized in that: The steps include: Step 1: Build an end-to-end pollen detection model based on Swin Transformer. The model includes an encoder, a decoder, and a detection head. The encoder includes several Swin Transformer blocks, each of which includes two submodules: local window attention and cross-window attention. The decoder consists of several decoding layers, each of which includes an upsampling layer, a convolutional layer, and an attention mechanism. Step 2: Replace the convolutional layer in the encoder with a deformable convolutional layer, thereby introducing deformable convolution into the model; Step 3: Introduce a CBAM-based Object Query initialization module into the model. The CBAM-based Object Query initialization module includes a spatial channel attention module, a position encoding module, and an Object Query module. The original input image is added with attention by the spatial channel attention module and position encoding by the position encoding module. The sum of these inputs serves as the initialization value of the Object Query module. The Object Query module acts on the decoding layer to guide the decoder to generate features related to the pollen target. Step 4: Introduce a GRU-based cyclic update operator into the model; the GRU-based cyclic update operator performs gradient backpropagation between adjacent decoding layers, so that each decoding layer refines the bounding box based on the prediction of the previous decoding layer; Step 5: Collect and construct a pollen detection dataset and train the model; Step 6: Use the trained model to perform pollen image detection.

2. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 1, characterized in that: In step 1, in the end-to-end pollen detection model: The encoder uses the convolutional layer to extract the initial features of the input image and obtain the initial feature map. It then extracts high-level image features through hierarchical segmentation, depth and width changes. The high-level image features contain global and local context information. The decoder decodes and reconstructs the high-level image features output by the encoder; The detection head performs pollen detection based on the output features of the decoder.

3. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 1 or 2, characterized in that: The detection head includes a global pooling layer, a fully connected layer, an activation function, a bounding box regression layer and a classification layer; The global pooling layer, when performing target detection, first performs global pooling on the feature map output by the encoder, reduces the height and width dimensions of the feature map to 1, and retains the channel dimension; The fully connected layer maps the feature vector obtained after global pooling to the output space required for target detection; The activation function is inserted between the fully connected layers to introduce nonlinearity; The bounding box regression layer outputs the coordinates and size information of the target; The classification layer converts the output into a probability value.

4. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 1, characterized in that: In step 2, the deformable convolution layer adds a 2D displacement to the mesh sampling position of the standard convolution rule of the convolution layer, so that the sampling grid can be deformed freely. For the convolution of the m×m convolution kernel, the output is y(p0), R is the m×m position of the convolution kernel sampling point relative to the input x(p0), then: Deformable convolution is the learning displacement Δp of the sampling point of the convolution kernel n , the convolution process is as follows: where p n is an enumeration of the positions listed in R, w(p n ) is p n The learnable weights.

5. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 1, characterized in that: In step 3, the Object Query module is used for the following two tasks: Attention weight query: In each decoding layer, the Object Query module is used to query the attention weights, which are applied to the feature maps output by the encoder to focus on the specific areas related to each Object Query module; Detection prediction: In each decoding layer, the Object Query module is used to generate detection predictions; by combining the ObjectQuery module with the features of the decoder, the category and bounding box information of each query object are predicted.

6. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 1, characterized in that: The spatial channel attention module is a lightweight convolutional attention module, which includes a channel attention module and a spatial attention module. The input image is convolved in one dimension through the channel attention module, the convolution result is multiplied with the input image, and then convolved in two dimensions through the spatial attention module, and the output result is multiplied with the input image.

7. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 1, characterized in that: In step 4, the input of the cyclic update operator is: the input x at time t t and the hidden layer state H at time t-1 t-1 , the hidden layer state contains the relevant information of the previous node; the output of the cyclic update operator is: the output y of the hidden node at time t t and the hidden layer state H passed to the next node t ; Through the last transmitted state H t-1 and the input x of the current node t Get two gate states: Reset gate: r t =σ(x t W xr +H t-1 W hr +b r ) Update gate: z t =σ(x t W xz +H t-1 W hz +b z ) σ is the sigmoid function, which converts the data into a value in the range of 0-1. xr is the weight matrix of the reset gate at time t, W hr is the weight matrix of the reset gate at time t-1, b r is the bias matrix of the reset gate, W xz is the weight matrix of the update gate at time t, W hz is the weight matrix of the update gate at time t-1, b z is the bias matrix of the update gate.

8. The method for detecting pollen targets in electron microscope images based on Transformer according to claim 7, characterized in that: The cyclic update operator is introduced into the decoding layer as follows: Initialize the bounding box: In the first decoding layer, the predicted bounding box is initialized by the fully connected layer and activation function. The bounding box is the normalized coordinate relative to the input image, expressed as (center_x, center_y, height, width); GRU loop update: In subsequent decoding layers, the GRU loop update operator is used to gradually optimize the prediction of the bounding box; Multiple iterations: The decoder uses multiple iterations to gradually optimize bounding box predictions. In each iteration, the bounding box predictions are updated. At the same time, the attention mechanism and deformable convolution help the model better utilize contextual information and local features, improving the accuracy and stability of bounding box predictions. Output bounding box prediction: After multiple rounds of iterations, the final bounding box prediction is obtained.

9. The method for detecting targets in electron microscope pollen images based on Transformer according to claim 8, characterized in that: The GRU is updated cyclically. For each decoding layer, the following steps are performed: a new feature map is sampled from the feature map according to the currently predicted bounding box using deformable convolution; the new feature map is fused with the feature map generated by the attention mechanism to obtain features that focus more on the target; Input the fused feature map and the bounding box obtained in the current round into the GRU cycle update operator; The GRU loop update operator outputs a new bounding box prediction based on the current features and bounding box.

10. The method for detecting objects in electron microscope pollen images based on Transformer according to claim 9, characterized in that: Each decoding layer refines the prediction box based on the prediction from the previous layer. Assuming there are D decoding layers, given the normalized bounding box predicted by the D-1th decoding layer, the Dth decoding layer refines the bounding box as where d∈{1,2,3…,D}, It is predicted at the dth decoding layer. The prediction heads of different decoding layers do not share parameters. The initial box is set to: To stabilize training, the gradient is passed only through Back propagation, and is blocked; in the loop update operator, for the d-th decoding layer, the box predicted from the d-1-th decoding layer Sampling of key elements; For the criss-cross attention module of the d-th decoding layer, is used as a new reference point; ultimately, the sampled position is made relative to the center and size of the previously predicted box.

Images