Pollen particle identification methods, devices, electronic equipment, and storage media

Abstract

The application provides a pollen grain identification method and device, electronic equipment and storage medium, and belongs to the technical field of machine vision. The method comprises the following steps: obtaining a pollen grain image to be identified; inputting the pollen grain image to be identified into a pollen type identification joint model to obtain a pollen type corresponding to the pollen grain image to be identified output by the pollen type identification joint model; the pollen type identification joint model is used for image super-resolution reconstruction on the pollen grain image to be identified, and the reconstructed image is identified to determine the pollen type corresponding to the pollen grain image to be identified; and the pollen type identification joint model is trained based on pollen grain image samples and corresponding pollen type labels. The application can effectively identify the pollen type corresponding to the pollen grain image to be identified, without manual intervention, thereby improving the pollen grain identification efficiency and greatly improving the pollen type identification precision.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method, apparatus, electronic device, and storage medium for identifying pollen particles. Background Technology

[0002] To address the serious problem of pollen allergies that severely impact residents' health and lives, research is needed on how to accurately classify and identify the main allergenic pollens in the air, and how to conduct effective pollen monitoring and forecasting. This will encourage pollen allergy sufferers to take proactive measures, which is of great practical significance for improving urban livability and the quality of life for citizens.

[0003] For pollen identification, traditional manual classification and identification methods not only require personnel with relevant professional knowledge, but the process is also time-consuming, labor-intensive, inefficient, and difficult to guarantee accuracy. Summary of the Invention

[0004] This invention provides a pollen grain identification method, device, electronic device, and storage medium to address the shortcomings of traditional manual classification methods in the prior art, which not only require personnel with relevant professional knowledge, but also involve time-consuming, labor-intensive, inefficient, and difficult-to-guarantee accuracy issues.

[0005] This invention provides a method for identifying pollen grains, comprising:

[0006] Acquire images of pollen grains to be identified;

[0007] The image of the pollen grain to be identified is input into the pollen type identification joint model to obtain the pollen type corresponding to the image of the pollen grain to be identified output by the pollen type identification joint model.

[0008] The pollen type recognition joint model is used to perform super-resolution reconstruction of the pollen grain image to be identified, and to identify the pollen type corresponding to the pollen grain image to be identified; the pollen type recognition joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0009] According to a pollen grain identification method provided by the present invention, the pollen type identification joint model includes a Transformer-based super-resolution model and a Swing Transformer model; the step of inputting the image of the pollen grain to be identified into the pollen type identification joint model to obtain the pollen type corresponding to the image of the pollen grain to be identified output by the pollen type identification joint model includes:

[0010] The pollen grain image to be identified is input into the Transformer-based super-resolution model to obtain the super-resolution image corresponding to the pollen grain image to be identified, which is output by the Transformer-based super-resolution model.

[0011] The super-resolution image is input into the Swin Transformer model to obtain the pollen type corresponding to the pollen grain image to be identified output by the Swin Transformer model.

[0012] According to a pollen grain identification method provided by the present invention, the pollen type identification joint model further includes a high-frequency information extraction module; after inputting the image of the pollen grain to be identified into the Transformer-based super-resolution model to obtain the super-resolution image corresponding to the image of the pollen grain to be identified output by the Transformer-based super-resolution model, the method further includes:

[0013] The super-resolution image is input into the high-frequency information extraction module to extract high-frequency information from the image, thereby obtaining the high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module.

[0014] The high-frequency feature image corresponding to the super-resolution image is input into the Swin Transformer model to obtain the pollen type corresponding to the pollen grain image to be identified output by the Swin Transformer model.

[0015] According to a pollen grain identification method provided by the present invention, the high-frequency information extraction module includes an average pooling layer, an upsampling layer, a first stacking layer, a convolutional layer, and a second stacking layer connected in sequence; the step of inputting the super-resolution image into the high-frequency information extraction module for high-frequency information extraction, and obtaining a high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module, includes:

[0016] The super-resolution image is input into the average pooling layer to obtain the first feature image output by the average pooling layer;

[0017] The first feature map is input into the upsampling layer to obtain the second feature image output by the upsampling layer;

[0018] The super-resolution image and the second feature image are input into the first overlay layer and the pixel difference is calculated to obtain the third feature image output by the first overlay layer;

[0019] The third feature image is input into the convolutional layer to obtain the fourth feature image output by the convolutional layer;

[0020] The fourth feature image and the super-resolution image are input into the second overlay layer for pixel summation to obtain the high-frequency feature image corresponding to the super-resolution image output by the second overlay layer.

[0021] According to a pollen grain identification method provided by the present invention, before inputting the image of the pollen grain to be identified into a pollen type identification joint model, the method further includes:

[0022] The pollen grain image samples and their corresponding pollen type labels are used as a set of training samples to obtain multiple sets of training samples.

[0023] The pollen type identification joint model was trained using the multiple sets of training samples.

[0024] According to the pollen grain identification method provided by the present invention, the step of training a joint model for pollen type identification using the multiple sets of training samples includes:

[0025] For any set of training samples, the training samples are input into the pollen type recognition joint model, and the predicted probability corresponding to the training samples is output.

[0026] Using a preset loss function, the loss value is calculated based on the predicted probability corresponding to the training sample and the pollen type label in the training sample;

[0027] Based on the loss value, the model parameters of the pollen type identification joint model are adjusted until the model training times reach the preset number of times.

[0028] The model parameters obtained when the model has been trained to the preset number of times are used as the model parameters of the trained pollen type recognition joint model.

[0029] The present invention also provides a pollen grain identification device, comprising:

[0030] The acquisition module is used to acquire images of pollen grains to be identified;

[0031] The identification module is used to input the image of the pollen grain to be identified into the pollen type identification joint model to obtain the pollen type corresponding to the image of the pollen grain to be identified output by the pollen type identification joint model.

[0032] The pollen type recognition joint model is used to perform super-resolution reconstruction of the pollen grain image to be identified, and to identify the pollen type corresponding to the pollen grain image to be identified; the pollen type recognition joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pollen grain identification method as described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pollen grain identification method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the pollen grain identification method as described above.

[0036] The pollen grain identification method, device, electronic device, and storage medium provided by this invention improve identification accuracy by considering the mining of deeper feature information of pollen grain images. A joint model for pollen type identification is trained using pollen grain image sample data and corresponding pollen type labels. The joint model for pollen type identification performs image super-resolution reconstruction on the acquired pollen grain images to be identified, and performs feature extraction and classification on the reconstructed high-resolution images. This can effectively identify the pollen type corresponding to the pollen grain images to be identified without manual intervention, improving pollen grain identification efficiency and significantly improving pollen type identification accuracy. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the pollen grain identification method provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the ESRT model in the pollen grain recognition method provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the HPB module in the pollen grain identification method provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the ET module in the pollen grain identification method provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the EMHA module in the pollen grain identification method provided by the present invention;

[0043] Figure 6 This is a schematic diagram of the Swin Transformer model in the pollen grain recognition method provided by the present invention;

[0044] Figure 7 This is a schematic diagram of the Swin Transformer block in the pollen grain recognition method provided by the present invention;

[0045] Figure 8 This is a schematic diagram of the structure of the pollen type identification joint model in the pollen grain identification method provided by the present invention;

[0046] Figure 9 This is a schematic diagram of the HF extraction module in the pollen grain identification method provided by the present invention;

[0047] Figure 10 This is a schematic diagram of the pollen particle identification device provided by the present invention;

[0048] Figure 11 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] In the description of the invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0051] The following is combined with Figures 1-11 The present invention describes a pollen grain identification method, apparatus, electronic device, and storage medium.

[0052] Figure 1 This is a flowchart illustrating the pollen grain identification method provided by the present invention, as shown below. Figure 1 As shown, it includes steps 110 and 120.

[0053] Step 110: Obtain an image of the pollen grains to be identified;

[0054] Step 120: Input the image of the pollen grain to be identified into the pollen type recognition joint model to obtain the pollen type corresponding to the image of the pollen grain to be identified output by the pollen type recognition joint model.

[0055] The pollen type recognition joint model is used to perform super-resolution reconstruction of images of pollen grains to be identified, and to identify the pollen type corresponding to the reconstructed images. This pollen type recognition joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0056] Specifically, the pollen grain image to be identified described in the embodiments of the present invention refers to the image obtained by capturing the pollen grain to be identified through an image acquisition device.

[0057] The pollen type described in this embodiment of the invention refers to the type of pollen grains in the image determined by identifying the image of the pollen grains to be identified. It may include Artemisia, Cucurbitaceae, Cupressaceae, Poaceae, Moraceae, Pinaceae, Populus, and Salix psammophila, etc.

[0058] The pollen type recognition joint model described in this embodiment of the invention is obtained by training the model using pollen grain image samples and corresponding pollen type labels. It is used to perform image super-resolution reconstruction on the image of the pollen grain to be identified, extract deeper image features, improve the ability to capture global information of the image, and perform recognition and classification operations on the reconstructed image, thereby outputting a more accurate recognition result and obtaining the pollen type corresponding to the image of the pollen grain to be identified.

[0059] The training samples consist of multiple sets of pollen grain image data samples carrying real pollen type labels.

[0060] The pollen type labels described in this invention are predetermined based on pollen grain image samples and correspond one-to-one with each sample. In other words, each pollen grain image sample in the training samples is pre-set to carry a corresponding pollen type as its true label.

[0061] For each pollen grain image sample, manual annotation can be used to identify and label the type of pollen in the pollen grain image sample, and the result of manual annotation can be used as the label corresponding to the pollen grain image sample.

[0062] It should be noted that pollen images obtained through scanning with instruments such as microscopes are typically small in size, and usually need to be resized to a suitable size for network training when loading data. However, the images obtained by the conventional resize() operation are often blurry and distorted. Deep learning-based image super-resolution restoration methods can effectively address this problem. Image super-resolution restoration algorithms can perform deeper feature extraction and reconstruction of images, thereby obtaining higher-quality super-resolution results.

[0063] Therefore, the pollen type identification joint model in this embodiment of the invention can be constructed using multiple deep neural networks, which are used to perform depth estimation between pollen type and pollen grain image information to determine the pollen type of each pollen grain image to be identified.

[0064] In this embodiment, the multiple deep neural networks may include a first deep neural network and a second deep neural network. The first deep neural network may employ a super-resolution (SR) algorithm model based on a convolutional neural network, or a Transformer-type neural network, such as using an Effificient Super-Resolution Transformer (ESRT) model as the super-resolution image reconstruction network. The second deep neural network may employ a Swin Transformer model to recognize and classify the reconstructed image. Of course, other neural networks used for super-resolution image reconstruction and classification can also be used, and this invention does not specifically limit their application.

[0065] In an embodiment of the present invention, in step 110, an image of the pollen grain to be identified can be acquired by an image acquisition device such as an optical microscope, an electron microscope, or a camera. This embodiment does not specifically limit the method of acquiring the pollen grain image.

[0066] Furthermore, in an embodiment of the present invention, in step 120, the obtained pollen grain image to be identified is input into a pre-trained pollen type recognition joint model. The pollen type recognition joint model performs super-resolution reconstruction on the pollen grain image to be identified, identifies the reconstructed feature image, and outputs the pollen type corresponding to the pollen grain image to be identified.

[0067] The pollen grain identification method of this invention improves identification accuracy by considering mining deeper feature information of pollen grain images. It trains a pollen type identification joint model using pollen grain image sample data and corresponding pollen type labels. The pollen type identification joint model performs image super-resolution reconstruction on the acquired pollen grain image to be identified, and performs feature extraction and classification on the reconstructed high-resolution image. It can effectively identify the pollen type corresponding to the pollen grain image to be identified without manual intervention, which improves the pollen grain identification efficiency and significantly improves the pollen type identification accuracy.

[0068] Based on the above embodiments, as an optional embodiment, the pollen type recognition joint model includes a Transformer-based super-resolution model and a Swing Transformer model; the pollen grain image to be identified is input into the pollen type recognition joint model to obtain the pollen type corresponding to the pollen grain image to be identified, as output by the pollen type recognition joint model, including:

[0069] The pollen grain image to be identified is input into the Transformer-based super-resolution model to obtain the super-resolution image corresponding to the pollen grain image to be identified output by the Transformer-based super-resolution model.

[0070] The super-resolution image is input into the Swin Transformer model to obtain the pollen type corresponding to the pollen grain image to be identified output by the Swin Transformer model.

[0071] Specifically, in embodiments of the present invention, the pollen type identification joint model can be composed of an ESRT model and a SwingTransformer model.

[0072] Among them, the ESRT model is a hybrid model that combines Convolutional Neural Networks (CNN) and Transformer structure. This model achieves a good balance between execution efficiency and super-resolution reconstruction effect. It can perform deeper feature extraction and reconstruction on pollen images with low resolution and achieve better results than traditional super-resolution algorithms.

[0073] The Swin Transformer model not only retains the Transformer's ability to capture global image information, but also, through its hierarchical structure and sliding window design, reduces the computational complexity of the attention mechanism from O(n log n). 2 The computational efficiency of the pollen type identification joint model is improved by reducing the computational efficiency of *d) to O(logn*d).

[0074] In an embodiment of the present invention, the image of the pollen grain to be identified is first input into the ESRT model for super-resolution reconstruction, and the super-resolution image corresponding to the image of the pollen grain to be identified is obtained from the output of the ESRT model.

[0075] Figure 2 This is a schematic diagram of the ESRT model in the pollen grain recognition method provided by the present invention, as shown below. Figure 2 As shown, the ESRT model includes a shallow feature extraction layer, a lightweight CNN backbone (LCB), a lightweight Transformer backbone (LTB), and an image reconstruction layer.

[0076] The shallow feature extraction layer can be a 3×3 convolutional layer used to extract shallow features from the input pollen grain image to be identified, as shown below:

[0077] F0 = f s (I LR );

[0078] Among them, I LR This represents a low-resolution image of the pollen grains to be identified; f s F0 represents the convolution operation; F0 is the output of the shallow feature extraction layer.

[0079] The role of the LCB network is to extract potential super-resolution features, giving the model a preliminary super-resolution capability. (Refer to...) Figure 2 An LCB can be composed of n High Preserving Blocks (HPBs). The input data F0 is processed through n HPB structures to obtain F. n At this point, the model possesses initial super-resolution capability. This process can be represented as follows:

[0080] F n =ζ n (ζ n-1 (…(ζ 1 (F0))));

[0081] Among them, F n ζ represents the output of the nth HPB module. n This represents the nth HPB process.

[0082] Figure 3 This is a schematic diagram of the HPB module in the pollen grain recognition method provided by the present invention, as shown below. Figure 3As shown, F is first extracted using Adaptive Residual Feature Block (ARFB). n-1 This serves as the input feature for the High-Frequency Filtering Module (HFM). The HFM is then used to compute the high-frequency information of the feature (denoted as P). high ). In obtaining P high Next, the size of the feature map is reduced to decrease computational cost and feature redundancy. The downsampled feature map is represented as F. n ′ -1 For F n ′ -1 Here, five ARFBs are used to extract latent information from the SR image, where these five ARFBs share weights to reduce parameters. After feature extraction, the F-axis is then transformed using bilinear interpolation. n ′ -1 Upsample to the original size. Simultaneously, use ARFB to process P. high P was obtained h ′ igh , so that the feature space is consistent with F n ′ -1 Align. Then, set F... n ′ -1 With P h ′ igh Fusion is performed to preserve the initial details, resulting in feature F. n ′′ -1 This operation can be represented as:

[0083]

[0084] Where ↑ and ↓ represent upsampling and downsampling operations, respectively; f a This indicates the ARFB operation.

[0085] Furthermore, after feature extraction via the LCB backbone network, n outputs F1, F2, ... F can be obtained. n These outputs serve as inputs to the LTB backbone network, which consists of n Efficient Transformer (ET) modules. The specific operations are as follows:

[0086] F d =φ n (φ n-1 (…(φ 1 ([F1,F2,…,F n ]))));

[0087] Among them, F dφ represents the output of LTB, and φ represents the operation of the ET module.

[0088] Figure 4 This is a schematic diagram of the ET module in the pollen grain recognition method provided by the present invention, as shown below. Figure 4 As shown, the ET module, like ViT, only uses the Transformer encoder structure. The difference is that the multi-head self-attention mechanism (MHA) in ET is replaced by an improved efficient multi-head attention mechanism (EMHA). The specific operation can be represented as follows:

[0089] E m1 =EMHA(Norm(E i ))+E i ;

[0090] E0 = MLP(Norm(E m1 ))+E m1 ;

[0091] Where E0 is the output of the ET module, EMHA represents the improved, efficient multi-head attention mechanism operation, E i It is an embedded input.

[0092] Figure 5 This is a schematic diagram of the EMHA module in the pollen grain identification method provided by the present invention, as shown below. Figure 5 As shown, assume the input embedding E i The dimensions are B×C×N. First, a Reduction layer reduces the number of channels in the input data to half (i.e., B×C1×N, C1=C / 2). Then, a linear layer projects the feature map onto three elements: Q, K, and V. As used in the Transformer, linear projections Q, K, and V are performed m times to execute multi-head attention, where m is the number of heads. Next, the shapes of the three elements are reshaped and arranged as B×m×N×C1 / m. Unlike the original MHA, a Feature Split module is used here, with s as the splitting factor, to divide Q, K, and V into s parts. The split data can be represented as: Q1,...,Q S ,K1,...K S ,V1,...V S Subsequently, a Scaled Dot-Product Attention (SDPA) operation is performed on each triple of these segments. Then, all outputs of the SDPA (CQ1, Q2, ..., Q...) are processed. S These are connected together to generate the entire output feature O. Finally, an expansion layer is used to restore the number of channels.

[0093] Furthermore, continue to refer to Figure 2 In an embodiment of the present invention, super-resolution reconstruction is finally performed through an image reconstruction layer. This part can consist of a 3×3 convolutional layer and a pixel shuffle layer. The operation of this part can be represented as follows:

[0094] I SR =f(f p (f(F d )))+f(f p (F0));

[0095] Among them, I SR This represents the RGB image after super-resolution reconstruction, i.e., the super-resolution image corresponding to the pollen grain image to be identified; f represents the convolution operation; f p This refers to the Pixel Shuffle layer operation.

[0096] Furthermore, the super-resolution image obtained above is input into the Swin Transformer model for classification and recognition, which can output the pollen type corresponding to the pollen grain image to be identified.

[0097] Figure 6 This is a schematic diagram of the Swin Transformer model in the pollen grain recognition method provided by this invention, as shown below. Figure 6 As shown, in an embodiment of the present invention, the Swin Transformer model segments the input super-resolution image into non-overlapping image patches using a patch partitioning layer similar to ViT. Each image patch is treated as a token, and its features are set as the concatenation of the original pixel RGB values. In this embodiment, a 4×4 image patch size can be used, so the feature dimension of each image patch is 4×4×3=48. A linear embedding layer is then applied to this original value feature, projecting it onto an arbitrary dimension (denoted as C).

[0098] Two Swing Transformer Blocks are applied to these Tokens. The number of Tokens in these Swing Transformer Blocks is [number missing]. It is combined with the linear embedding layer to form "Stage 1".

[0099] As the network deepens, a patch merging layer is used to reduce the number of tokens. The first patch merging layer concatenates the features of each group of 2×2 adjacent image patches and applies a linear layer to the 4C-dimensional concatenated features. This reduces the number of tokens by a factor of 2×2=4 (2×downsampling factor) and sets the output dimension to 2C. Then, a Swin Transformer block is applied for feature transformation, maintaining the resolution. The first block of the image patch merging layer and feature transformation layer combination is denoted as "Stage 2". This process is repeated twice, denoted as "Stage 3" and "Stage 4", with output image resolutions of [missing information]. and The number of Swin Transformer block layers in each stage are 2, 2, 6 and 2, respectively.

[0100] Figure 7 This is a schematic diagram of the Swin Transformer block in the pollen grain recognition method provided by the present invention, as shown below. Figure 7 As shown in the embodiment of the present invention, the Swin-Transformer block structure is calculated from the continuous Swin-Transformer blocks shown in the figure, as follows:

[0101]

[0102]

[0103]

[0104]

[0105] Among them, z l-1 z l These are the input and output of the l-th layer W_MSA, respectively. LN represents the linear normalization layer, MLP represents the multilayer perceptron layer, and W_MSA and SW_MSA represent the window multi-head attention mechanism and the moving window multi-head attention mechanism, respectively.

[0106] It should be noted that, in the embodiments of the present invention, the structure of the Swin Transformer model also includes an output layer for outputting the recognition result. The output layer may include a normalization layer, a global pooling layer, and a fully connected layer connected in sequence. The final recognition result can be output through the output layer to obtain the pollen type corresponding to the pollen grain image to be identified.

[0107] The method of this invention leverages the significant advantages of the ESRT model in terms of execution efficiency and super-resolution reconstruction effect to perform deeper feature extraction and reconstruction on pollen images with low resolution. At the same time, it utilizes the ability of the SwinTransformer model to capture global image information, as well as its hierarchical structure and sliding window design, which not only greatly improves the computational efficiency of the pollen type recognition joint model, but also achieves high-precision pollen type recognition.

[0108] Based on the above embodiments, as an optional embodiment, the pollen type identification joint model further includes a high-frequency information extraction module; after inputting the image of the pollen grain to be identified into the Transformer-based super-resolution model to obtain the super-resolution image corresponding to the image of the pollen grain to be identified output by the Transformer-based super-resolution model, the method further includes:

[0109] The super-resolution image is input into the high-frequency information extraction module to extract high-frequency information from the image, and the high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module is obtained.

[0110] The high-frequency feature image corresponding to the super-resolution image is input into the Swin Transformer model to obtain the pollen type corresponding to the pollen grain image to be identified output by the Swin Transformer model.

[0111] Specifically, the high-frequency feature image described in this embodiment of the invention refers to an image obtained by capturing the texture information and high-frequency details of an image from a super-resolution image through a high-frequency information (HF) extraction module.

[0112] In embodiments of the present invention, a preset HF extraction module is used to capture the texture information and high-frequency details of the image, thereby providing richer feature information for the Swing Transformer.

[0113] In an embodiment of the present invention, after inputting the pollen grain image to be identified into the ESRT model to obtain the super-resolution image corresponding to the pollen grain image to be identified, the super-resolution image is further input into the HF extraction module to extract high-frequency information of the image, and the high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module can be obtained.

[0114] Furthermore, in an embodiment of the present invention, the high-frequency feature image corresponding to the super-resolution image is input into the Swin Transformer model. The Swin Transformer model classifies and identifies the high-frequency feature image with richer feature information, outputs a high-precision identification result, and obtains the pollen type corresponding to the pollen grain image to be identified.

[0115] Figure 8 This is a schematic diagram of the structure of the pollen type identification joint model in the pollen grain identification method provided by the present invention, as shown below. Figure 8 As shown in the embodiments of the present invention, the pollen type identification joint model may further include an ESRT model, an HF extraction module, and a Swin Transformer model. The ESRT model is used to perform super-resolution reconstruction of the pollen grain image to be identified, making it suitable for processing by the Swin Transformer. In between, the HF extraction module extracts high-frequency information from the reconstructed high-resolution image, thereby providing richer feature information for the subsequent Swin Transformer model. Finally, the Swin Transformer model is used to further extract feature information from the feature image output by the HF extraction module and perform classification operations to output the pollen type corresponding to the pollen grain image to be identified.

[0116] The method of this invention, by introducing an HF extraction module, can extract texture information and high-frequency details from the reconstructed pollen super-resolution image, thereby providing richer feature information for the Swing Transformer model to perform classification and recognition, which can significantly improve the accuracy of pollen classification and recognition results and increase recognition precision.

[0117] Based on the above embodiments, as an optional embodiment, the high-frequency information extraction module includes an average pooling layer, an upsampling layer, a first stacking layer, a convolutional layer, and a second stacking layer connected in sequence; the super-resolution image is input to the high-frequency information extraction module for high-frequency information extraction, resulting in a high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module, including:

[0118] The super-resolution image is input into the average pooling layer to obtain the first feature image output by the average pooling layer;

[0119] The first feature map is input into the upsampling layer to obtain the second feature image output by the upsampling layer;

[0120] The super-resolution image and the second feature image are input into the first overlay layer and the pixel difference is calculated to obtain the third feature image output by the first overlay layer.

[0121] The third feature image is input into the convolutional layer to obtain the fourth feature image output by the convolutional layer;

[0122] The fourth feature image and the super-resolution image are input into the second stacking layer for pixel summation, resulting in the high-frequency feature image corresponding to the super-resolution image output by the second stacking layer.

[0123] Specifically, the first feature image described in the embodiments of the present invention refers to the image obtained by inputting the super-resolution image into the average pooling layer in the HF extraction module.

[0124] The second feature image described in this embodiment of the invention refers to the image obtained by further inputting the first feature image into the upsampling layer of the HF extraction module for upsampling operation.

[0125] The third feature image described in this embodiment of the invention refers to the image obtained by inputting the super-resolution image and the second feature image into the first overlay layer in the HF extraction module and performing pixel subtraction.

[0126] The fourth feature image described in this embodiment of the invention refers to the image obtained by inputting the third feature image into the convolutional layer of the HF extraction module for convolution operation.

[0127] In an embodiment of the present invention, in order to better extract the texture details of the image, a high-frequency information extraction model based on a high-pass filter, namely the HF extraction module, is also designed.

[0128] In an embodiment of the present invention, after obtaining the super-resolution image corresponding to the pollen grain image to be identified, the image is input into the HF extraction module to extract high-frequency information from the super-resolution image. This process enables the model to capture the texture details of the image, thereby improving the accuracy of subsequent Swin Transformer model classification. To further enhance the local features in the image, a two-layer convolution method can be used. Therefore, the convolutional layer in the HF extraction module may include a 1×1 convolutional layer Conv-1 and a 3×3 convolutional layer Conv-3 connected in sequence.

[0129] Figure 9 This is a schematic diagram of the HF extraction module in the pollen grain identification method provided by the present invention, as shown below. Figure 9 As shown, in an embodiment of the present invention, firstly, assume T L The input data representing the model, i.e., the super-resolution image corresponding to the pollen grain image to be identified, can be used to obtain the first feature image T by employing an average pooling layer (Avg Pool) with a kernel size of 7. A Then, the first feature image T A Upsampling to the original image T LThe size of the second feature image T is obtained U Then, element-wise subtraction is performed through the first overlay layer to transform the original image T. L Subtract the second feature image T U Obtain the third feature image T HF The purpose of this step is to capture subtle changes and high-frequency information in the image. Subsequently, the third feature image T... HF The input is fed into a convolutional layer, consisting of a 1×1 convolutional layer Conv-1 and a 3×3 convolutional layer Conv-3 connected in sequence, for the third feature image T. HF Two convolutional layers are performed to extract features at multiple levels, including attributes such as edges and textures, resulting in T after the convolution operation. HF That is, the fourth feature image. Finally, by introducing skip connections, the original image T is transformed. L Added to the convolution output, element-wise summation is performed through a second stacking layer, transforming the original image T... L Add T after convolution HF This yields the high-frequency feature image corresponding to the super-resolution image.

[0130] The method of this invention, through the HF extraction module based on high-frequency filtering, can effectively obtain the texture details of the image and capture the high-frequency information in the image. At the same time, the integration of skip connections further enhances gradient propagation, thereby improving the learning performance of the model and helping to further improve the accuracy of the classification and recognition results of the Swin Transformer model.

[0131] Based on the above embodiments, as an optional embodiment, before inputting the image of the pollen grains to be identified into the pollen type identification joint model, the method further includes:

[0132] Multiple sets of training samples are obtained by using pollen grain image samples and their corresponding pollen type labels as a set of training samples.

[0133] A joint model for pollen type identification was trained using multiple sets of training samples.

[0134] Specifically, before inputting the image of the pollen grain to be identified into the pollen type recognition joint model, the pollen type recognition joint model also needs to be trained. The specific training process is as follows:

[0135] Pollen grain image samples labeled with pollen type and the corresponding pollen type label are used as a set of training samples. The pollen type recognition joint model is trained using multiple pollen grain image samples carrying different pollen type labels, thereby obtaining multiple sets of training samples.

[0136] In the embodiments of the present invention, each pollen grain image sample and its corresponding pollen type label are one-to-one.

[0137] Then, after obtaining multiple sets of training samples, the multiple sets of training samples are input into the pollen type recognition joint model in sequence. That is, the pollen grain image samples and their corresponding pollen type labels in each set of training samples are simultaneously input into the pollen type recognition joint model. Based on each output result of the pollen type recognition joint model, the loss value is calculated through a preset loss function to adjust the model parameters of the pollen type recognition joint model, and finally the training process of the pollen type recognition joint model is completed.

[0138] The method of this invention uses pollen grain image samples labeled with pollen type tags and the pollen type tags as a set of training samples, and uses multiple sets of training samples to train the pollen type recognition joint model, thereby ensuring the recognition accuracy of the pollen type recognition joint model.

[0139] Based on the above embodiments, as an optional embodiment, the pollen type identification joint model is trained using multiple sets of training samples, including:

[0140] For any set of training samples, input the training samples into the pollen type recognition joint model and output the predicted probability corresponding to the training samples;

[0141] Using a preset loss function, the loss value is calculated based on the predicted probability corresponding to the training sample and the pollen type label in the training sample;

[0142] Based on the loss value, the model parameters of the pollen type identification joint model are adjusted until the model is trained for a preset number of times.

[0143] The model parameters obtained when the model has been trained a preset number of times are used as the model parameters of the trained pollen type recognition joint model.

[0144] Specifically, the preset loss function described in the embodiments of the present invention refers to the loss function pre-set in the pollen type identification joint model for model evaluation.

[0145] After obtaining multiple sets of training samples, for any set of training samples, the pollen grain image samples and corresponding pollen type labels in the training samples are simultaneously input into the pollen type recognition joint model, and the predicted probability value corresponding to the recognition result of the training sample is output.

[0146] Based on this, a loss value is calculated using a preset loss function, based on the predicted probability value corresponding to the recognition result of the training sample and the pollen type label carried in the training sample.

[0147] In embodiments of the present invention, the preset pollen type label representation method can be set according to actual needs, and no specific limitation is made here.

[0148] After calculating the loss value, the current training process ends. The model parameters in the pollen type recognition joint model are updated, and then the next training iteration begins. During training, when the number of training iterations reaches the preset maximum or the loss value is less than a preset threshold, the model parameters obtained at this point are used as the model parameters of the trained pollen type recognition joint model. At this point, the training of the pollen type recognition joint model is complete.

[0149] The method of this invention improves the pollen type identification accuracy by iteratively training the pollen type identification joint model within a preset range.

[0150] The pollen grain identification device provided by the present invention is described below. The pollen grain identification device described below can be referred to in correspondence with the pollen grain identification method described above.

[0151] Figure 10 This is a schematic diagram of the pollen grain identification device provided by the present invention, as shown below. Figure 10 As shown, it includes:

[0152] The acquisition module 1010 is used to acquire images of pollen grains to be identified;

[0153] The identification module 1020 is used to input the image of the pollen grains to be identified into the pollen type identification joint model to obtain the pollen type corresponding to the image of the pollen grains to be identified output by the pollen type identification joint model.

[0154] The pollen type recognition joint model is used to perform super-resolution reconstruction of the pollen grain image to be identified, and to identify the pollen type corresponding to the pollen grain image to be identified; the pollen type recognition joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0155] The pollen grain identification device described in this embodiment can be used to execute the pollen grain identification method embodiment described above. Its principle and technical effect are similar, and will not be repeated here.

[0156] The pollen grain recognition device of this invention improves recognition accuracy by considering the mining of deeper feature information of pollen grain images. It trains a pollen type recognition joint model using pollen grain image sample data and corresponding pollen type labels. The pollen type recognition joint model performs image super-resolution reconstruction on the acquired pollen grain image to be identified, and performs feature extraction and classification on the reconstructed high-resolution image. It can effectively identify the pollen type corresponding to the pollen grain image to be identified without manual intervention, which improves the pollen grain recognition efficiency and also significantly improves the pollen type recognition accuracy.

[0157] Figure 11 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute the pollen grain identification method provided by the above methods. The method includes: acquiring an image of pollen grains to be identified; inputting the image of pollen grains to be identified into a pollen type identification joint model to obtain the pollen type corresponding to the image of pollen grains to be identified output by the pollen type identification joint model; the pollen type identification joint model is used to perform image super-resolution reconstruction on the image of pollen grains to be identified, and to identify the pollen type corresponding to the image of pollen grains to be identified; the pollen type identification joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0158] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pollen grain identification method provided by the above methods. The method includes: acquiring an image of a pollen grain to be identified; inputting the image of the pollen grain to be identified into a pollen type identification joint model to obtain the pollen type corresponding to the image of the pollen grain to be identified output by the pollen type identification joint model; the pollen type identification joint model is used to perform image super-resolution reconstruction on the image of the pollen grain to be identified, and to identify the reconstructed image to determine the pollen type corresponding to the image of the pollen grain to be identified; the pollen type identification joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0160] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the pollen grain identification method provided by the above methods. The method includes: acquiring an image of a pollen grain to be identified; inputting the image of the pollen grain to be identified into a pollen type identification joint model to obtain the pollen type corresponding to the image of the pollen grain to be identified, output by the pollen type identification joint model; the pollen type identification joint model is used to perform image super-resolution reconstruction on the image of the pollen grain to be identified, and to identify the reconstructed image to determine the pollen type corresponding to the image of the pollen grain to be identified; the pollen type identification joint model is trained based on pollen grain image samples and corresponding pollen type labels.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pollen grain identification method characterized by, The method comprises: acquiring a pollen grain image to be identified; The pollen type identification joint model comprises a Transformer-based super-resolution model and a Swin Transformer model; the pollen type identification joint model further comprises a high-frequency information extraction module; the high-frequency information extraction module comprises, in sequence, an average pooling layer, an up-sampling layer, a first superposition layer, a convolution layer, and a second superposition layer; inputting the pollen grain image to be identified into the pollen type identification joint model to obtain a pollen type corresponding to the pollen grain image to be identified output by the pollen type identification joint model, comprising: inputting the pollen grain image to be identified into the Transformer-based super-resolution model to obtain a super-resolution image corresponding to the pollen grain image to be identified output by the Transformer-based super-resolution model; after obtaining the super-resolution image, the method further comprises: inputting the super-resolution image into the high-frequency information extraction module for image high-frequency information extraction to obtain a high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module; obtaining the high-frequency feature image comprises: inputting the super-resolution image into the average pooling layer to obtain a first feature image output by the average pooling layer; inputting the first feature image into the up-sampling layer to obtain a second feature image output by the up-sampling layer; inputting the super-resolution image and the second feature image into the first superposition layer for pixel difference to obtain a third feature image output by the first superposition layer; inputting the third feature image into the convolution layer to obtain a fourth feature image output by the convolution layer; inputting the fourth feature image and the super-resolution image into the second superposition layer for pixel summation to obtain a high-frequency feature image corresponding to the super-resolution image output by the second superposition layer; inputting the high-frequency feature image corresponding to the super-resolution image into the Swin Transformer model to obtain a pollen type corresponding to the pollen grain image to be identified output by the Swin Transformer model; the pollen type identification joint model is used for image super-resolution reconstruction on the pollen grain image to be identified, and identification on the reconstructed image to determine the pollen type corresponding to the pollen grain image to be identified; the pollen type identification joint model is trained based on pollen grain image samples and corresponding pollen type labels.

2. The pollen grain identification method according to claim 1, characterized by, Before the pollen grain image to be identified is input into the pollen type identification joint model, the method further comprises: acquiring a plurality of groups of training samples by taking the pollen grain image samples and the corresponding pollen type labels as a group of training samples; training the pollen type identification joint model by using the plurality of groups of training samples.

3. The pollen grain identification method according to claim 2, characterized in that, The training of the pollen type identification joint model by using the plurality of groups of training samples comprises: For any set of training samples, input the training samples into the pollen type recognition joint model, and output the prediction probability corresponding to the training samples; Using a preset loss function, calculate the loss value according to the prediction probability corresponding to the training samples and the pollen type label in the training samples; Based on the loss value, adjust the model parameters of the pollen type recognition joint model until the model training times reach a preset number; The model parameters obtained when the model training times reach the preset number are used as the model parameters of the trained pollen type recognition joint model.

4. A pollen grain recognition apparatus, characterized by, Comprise: An acquisition module is configured to acquire a pollen grain image to be identified; An identification module is configured to input the pollen grain image to be identified into a pollen type recognition joint model to obtain a pollen type corresponding to the pollen grain image to be identified output by the pollen type recognition joint model; The pollen type recognition joint model is used for image super-resolution reconstruction of the pollen grain image to be identified, and identification of the reconstructed image to determine the pollen type corresponding to the pollen grain image to be identified; the pollen type recognition joint model is trained based on pollen grain image samples and corresponding pollen type labels; The pollen type recognition joint model comprises a Transformer-based super-resolution model and a Swin Transformer model; the pollen type recognition joint model further comprises a high-frequency information extraction module; the high-frequency information extraction module comprises an average pooling layer, an upsampling layer, a first superposition layer, a convolution layer and a second superposition layer connected in sequence; the identification module is specifically configured to: Input the pollen grain image to be identified into the Transformer-based super-resolution model to obtain a super-resolution image corresponding to the pollen grain image to be identified output by the Transformer-based super-resolution model; after obtaining the super-resolution image, the identification module is further configured to: Input the super-resolution image into the high-frequency information extraction module for image high-frequency information extraction to obtain a high-frequency feature image corresponding to the super-resolution image output by the high-frequency information extraction module; Obtaining the high-frequency feature image comprises: Inputting the super-resolution image into the average pooling layer to obtain a first feature image output by the average pooling layer; Inputting the first feature image into the upsampling layer to obtain a second feature image output by the upsampling layer; Inputting the super-resolution image and the second feature image into the first superposition layer to perform pixel difference to obtain a third feature image output by the first superposition layer; Inputting the third feature image into the convolution layer to obtain a fourth feature image output by the convolution layer; Inputting the fourth feature image and the super-resolution image into the second superposition layer to perform pixel summation to obtain a high-frequency feature image corresponding to the super-resolution image output by the second superposition layer; The high-frequency feature image corresponding to the super-resolution image is input into the Swin Transformer model, and a pollen type corresponding to the pollen grain image to be identified output by the Swin Transformer model is obtained.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the pollen grain identification method of any one of claims 1 to 3 when executing the program.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the pollen grain identification method of any one of claims 1 to 3.

7. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the pollen grain identification method of any one of claims 1 to 3. The computer program, when executed by the processor, implements the pollen grain identification method of any one of claims 1 to 3.

Images