Pollen classification method, device, electronic device and storage medium

Abstract

The present invention provides a pollen classification method, device, electronic device and storage medium, wherein the method includes: determining a pollen image to be classified; applying a quality classification network based on the pollen image to be classified to determine an image quality result; if the image quality result is fuzzy, applying an image enhancement network based on the pollen image to be classified to determine a clear pollen image to be classified; applying a classification model based on the clear pollen image to be classified to obtain a classification result; the classification network is trained based on pollen sample images and pollen category labels; a synthetic clear pollen image is obtained by inputting a fuzzy pollen image into an image enhancement network; a clear pollen image and a fuzzy pollen image are determined by inputting the pollen image into the quality classification network, thereby realizing automatic judgment of image quality and generating a corresponding synthetic clear pollen image from the fuzzy pollen image, reducing the time cost of manual screening and making full use of the out-of-focus fuzzy pollen image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a pollen classification method, device, electronic device and storage medium. Background Art

[0002] Airborne pollen can cause allergic diseases such as rhinitis and asthma. Pollen allergies are common worldwide, with incidence rates increasing year by year, severely impacting people's lives, work, and health. Avoiding pollen exposure is the most effective treatment and prevention. However, different types of pollen have varying allergenicity, and individual sensitivity is specific to each species. Therefore, accurate and real-time monitoring and classification of airborne pollen is crucial.

[0003] Currently, the deep learning methods used in the automatic classification of pollen images are mainly based on convolutional neural networks, which are trained and tested in an end-to-end manner, inputting pollen images and outputting pollen category information.

[0004] However, this end-to-end convolutional neural network uses manually selected high-quality clear pollen data, resulting in high labor costs. It also ignores out-of-focus blurred pollen images, causing a large amount of data to not be effectively utilized. Summary of the Invention

[0005] The present invention provides a pollen classification method, device, electronic device and storage medium, which are used to solve the defects in the prior art of high labor cost for model training and inability to effectively utilize a large amount of fuzzy pollen image data.

[0006] The present invention provides a pollen classification method, comprising:

[0007] Determine the pollen image to be classified;

[0008] Based on the pollen image to be classified, applying a quality classification network to determine an image quality result;

[0009] If the image quality result is fuzzy, applying an image enhancement network based on the pollen image to be classified to determine a clear pollen image to be classified; otherwise, using the pollen image to be classified as the clear pollen image to be classified;

[0010] Based on the clear pollen image to be classified, applying a classification model to obtain a classification result;

[0011] The classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; the clear pollen images and the blurred pollen images are determined by inputting pollen images into the quality classification network; the quality classification network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on the clear pollen images and the blurred pollen images.

[0012] According to a pollen classification method provided by the present invention, the training steps of the quality classification network are as follows:

[0013] Determine the pollen images with quality labels, clear pollen images, and initial quality classification networks;

[0014] Performing blurring processing of different degrees on the clear pollen image, and combining the obtained pollen images with known blurring degrees into image pairs, thereby obtaining pollen image pairs with known blurring degrees;

[0015] Inputting the first image of the pollen image pair into a first network branch of the initial mass classification network to obtain a scalar value of the first image output by the first network branch; simultaneously, inputting the second image of the pollen image pair into a second network branch of the initial mass classification network to obtain a scalar value of the second image output by the second network branch; the network structure of the first network branch and the network structure of the second network branch are the same;

[0016] Determining a contrast loss based on the scalar value of the first image and the scalar value of the second image, and iteratively training the initial quality classification network based on the contrast loss until convergence to obtain an intermediate first network branch and an intermediate second network branch;

[0017] Based on the pollen image with the quality label, the intermediate first network branch is trained and fine-tuned, and the trained intermediate first network branch is used as the quality classification network.

[0018] According to a pollen classification method provided by the present invention, the training steps of the image enhancement network are as follows:

[0019] Determining the unpaired clear pollen image and the fuzzy pollen image, and an initial image enhancement network; the initial image enhancement network includes a clear image generation network branch and a fuzzy image generation network branch;

[0020] Inputting the blurred pollen image into the clear image generation network branch to obtain a first synthesized clear image, and simultaneously inputting the clear image into the blurred image generation network branch to obtain a first synthesized blurred image;

[0021] Inputting the first synthesized clear image into the blurred image generation branch to obtain a second synthesized blurred image, and simultaneously inputting the first synthesized blurred image into the clear image generation branch to obtain a second synthesized clear image;

[0022] Based on the blurred pollen image and the first synthesized blurred image, applying a fuzzy domain discriminator to determine a fuzzy domain adversarial loss; and based on the clear pollen image and the first synthesized clear image, applying a clear domain discriminator to determine a clear domain adversarial loss;

[0023] determining a cycle consistency loss based on the blurred pollen image, the second synthesized blurred image, the sharp pollen image, and the second synthesized sharp image;

[0024] Determining a blur domain perceptual loss based on the semantic features of the blurry pollen image and the semantic features of the second synthesized blurry image; and determining a clear domain perceptual loss based on the semantic features of the clear pollen image and the semantic features of the second synthesized clear image;

[0025] Based on the blur domain adversarial loss, the clear domain adversarial loss, the cycle consistency loss, the blur domain perceptual loss and the clear domain perceptual loss, a joint loss is determined, and based on the joint loss, parameters of the initial image enhancement network are iterated until the joint loss converges, and the clear image generation network branch is used as the image enhancement network.

[0026] According to a pollen classification method provided by the present invention, the training steps of the classification model are as follows:

[0027] Determine pollen sample images, pollen category labels, and initial classification models;

[0028] Based on the pollen sample image, applying basic data augmentation and / or cutting and occlusion data augmentation to obtain a pollen augmented image;

[0029] The initial classification model is trained based on the pollen augmented image and the pollen category label to obtain the classification model.

[0030] According to a pollen classification method provided by the present invention, based on the pollen sample image, basic data augmentation and / or cut-out occlusion data augmentation are applied to obtain a pollen augmented image, including:

[0031] Based on the pollen sample image, applying an impurity removal unit to remove impurities from the pollen sample image to obtain a clean pollen sample image;

[0032] Based on the clean pollen sample image, applying the basic data augmentation and / or the cut-occluded data augmentation to obtain a pollen augmented image to be processed;

[0033] Normalizing the pollen augmented image to be processed to obtain the pollen augmented image.

[0034] According to a pollen classification method provided by the present invention, the impurity removal unit performs the following impurity removal steps:

[0035] Determine the pollen image to be removed;

[0036] After removing noise points and brightening the image of the pollen to be removed, the color space of the pollen to be removed is converted into HSV and a preset binarization threshold is applied to perform image binarization to obtain a binarized image;

[0037] After performing closing and opening operations on the binary image, performing contour detection on the binary image to obtain a contour set;

[0038] Based on the radius of the circumscribed circle of each contour in the contour set, contours having a circumscribed circle radius smaller than a preset radius threshold are removed to obtain a contour map; and based on the contours in the contour map, a mask image is determined;

[0039] A pure pollen image is determined based on the mask image and the image of the pollen to be removed from impurities.

[0040] According to a pollen classification method provided by the present invention, the classification model is applied based on the clear pollen image to be classified to obtain a classification result, including:

[0041] Based on the clear pollen image to be classified, applying the impurity removal unit to determine a pure and clear pollen image to be classified;

[0042] Based on the pure and clear pollen image to be classified, the classification model is applied to obtain a classification result.

[0043] The present invention also provides a pollen classification device, comprising:

[0044] A determination module, used for determining the pollen image to be classified;

[0045] A quality judgment module, configured to apply a quality classification network to the pollen image to be classified and determine an image quality result;

[0046] an enhancement module configured to, if the image quality result is blurred, apply an image enhancement network to determine a clear pollen image based on the pollen image to be classified; otherwise, use the pollen image to be classified as the clear pollen image to be classified;

[0047] A classification module, configured to apply a classification model based on the clear pollen image to obtain a classification result;

[0048] The classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; the clear pollen images and the blurred pollen images are determined by inputting pollen images into the quality classification network; the quality classification network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on the clear pollen images and the blurred pollen images.

[0049] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described pollen classification methods is implemented.

[0050] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned pollen classification methods when executed by a processor.

[0051] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned pollen classification methods.

[0052] The pollen classification method, device, electronic device and storage medium provided by the present invention enhance the blurred image in the pollen sample image by using a quality classification network and an image enhancement network to obtain a synthetic clear pollen image, and obtain a classification model by training the synthetic clear pollen image and the clear pollen image in the pollen sample image. The quality classification network, image enhancement network and classification network are combined to classify the pollen image to be classified, thereby realizing the application of the quality classification network to automatically judge the image quality of the pollen image, and the application of the image enhancement network to generate a corresponding synthetic clear pollen image from the blurred pollen image, reducing the time cost of manually screening high-quality pollen data, and making full use of the out-of-focus blurred pollen images to expand the training samples of the classification model, thereby improving the accuracy of the classification model in pollen recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 1 is a schematic diagram of the process of the pollen classification method provided by the present invention;

[0055] Figure 2 It is a flow chart of the quality classification network training method provided by the present invention;

[0056] Figure 3 Schematic diagram of the process of the image enhancement network training method provided by the present invention;

[0057] Figure 4 It is a structural diagram of the initial image enhancement network architecture provided by the present invention;

[0058] Figure 5 It is a flow chart of the classification model training method provided by the present invention;

[0059] Figure 6 It is a schematic diagram of the impurity removal process provided by the present invention;

[0060] Figure 7 This is a schematic structural diagram of the pollen classification device provided by the present invention;

[0061] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0063] Automatic pollen image classification using deep learning methods, primarily based on convolutional neural networks, has become increasingly mainstream. This approach uses an end-to-end training and testing process, taking pollen images as input and outputting pollen classification information. However, these methods rely solely on manually selected, high-quality, clear pollen data. However, manual selection is time-consuming and labor-intensive, making it difficult to define a unified high-quality classification metric. Furthermore, they ignore out-of-focus, blurry pollen images, resulting in a significant amount of data going unused.

[0064] Therefore, how to use fuzzy pollen images to expand the classification model training samples in a low-labor-cost manner to improve the accuracy of the classification model in pollen classification is a technical problem that needs to be solved urgently by those skilled in the art.

[0065] In order to solve the above technical problems, an embodiment of the present invention provides a pollen classification method. Figure 1 FIG. 1 is a flow chart of the pollen classification method provided by the present invention. Figure 1 As shown, the method includes:

[0066] Step 110, determining the pollen image to be classified;

[0067] It should be noted that the pollen image to be classified is a digital pathological section image of pollen, which can be a clear image or a blurred image, and the embodiment of the present invention does not limit this. The category of the pollen image to be classified is any one of the pollen category labels used when training the classification model.

[0068] Step 120 , applying a quality classification network based on the pollen image to be classified to determine an image quality result;

[0069] Specifically, the pollen image to be classified is input into the quality classification network, and the quality classification network outputs an image quality result, which is either clear or blurry.

[0070] It should be noted that the quality classification network is used to score the clarity of the image and compare it with the preset clarity threshold to obtain the image quality result.

[0071] Step 130: If the image quality result is fuzzy, then applying an image enhancement network to the pollen image to be classified to determine a clear pollen image to be classified; otherwise, the pollen image to be classified is used as a clear pollen image to be classified;

[0072] Specifically, when the image quality result is judged to be blurred, the pollen image to be classified is input into the image enhancement network to obtain a clear pollen image to be classified generated by the image enhancement network.

[0073] It should be noted that the image enhancement network is the generator in the generative adversarial network, which is used to generate a clear pollen image from a blurred pollen image. The generative adversarial network can be a common generative adversarial network or a cyclic generative adversarial network, which is not limited in this embodiment of the present invention.

[0074] In addition, when the image quality result is judged to be clear, the pollen image to be classified is used as a clear pollen image to be classified.

[0075] Step 140 , applying a classification model based on the clear pollen image to be classified to obtain a classification result;

[0076] The classification network is trained based on pollen sample images and pollen category labels; pollen sample images include clear pollen images and synthetic clear pollen images; synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; clear pollen images and blurred pollen images are determined by inputting pollen images into the quality classification network; the quality classification network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on clear pollen images and blurred pollen images.

[0077] Specifically, the collected pollen images are first fed into a quality classification network to obtain an image quality result, i.e., whether the pollen image is clear or blurred. The blurred pollen image is then fed into an image enhancement network, which generates a synthetic clear pollen image from the blurred pollen image. The clear pollen image and the synthetic clear pollen image are then combined to form a pollen sample image, which is labeled with pollen category labels. The initial classification network is trained using the pollen sample images and pollen category labels to obtain a classification model. The classification model then classifies the input clear pollen image to be classified to obtain the classification result.

[0078] It should be noted that before training the classification model, it is necessary to first train a quality classification network and an image enhancement network. When training the quality classification network, the initial quality classification network can be first trained with pollen images of known pollen blur levels, and then the initial quality classification network can be trained with pollen images with quality labels in the second stage to obtain a quality classification network. The embodiment of the present invention does not impose any restrictions on this. Among them, the pollen images of known pollen blur levels can be obtained by manual identification, or by first performing different degrees of blurring on the images. The embodiment of the present invention does not impose any restrictions on this. When training the image enhancement network, the initial image enhancement network can be trained with clear pollen images and blurred pollen images to obtain an image enhancement network. The clear pollen images and blurred pollen images are obtained by the quality classification network performing image quality classification on the pollen images.

[0079] In addition, sample augmentation may be performed on the pollen sample images before training the initial classification model, and this embodiment of the present invention does not impose any limitation on this.

[0080] On the other hand, before the clear pollen image to be classified is input into the classification model, operations such as impurity removal and / or normalization can be performed on it. Before the initial classification model is trained, operations such as impurity removal and / or normalization can also be performed on the sample image. This embodiment of the present invention does not limit this.

[0081] The pollen classification method provided by the embodiment of the present invention uses a quality classification network and an image enhancement network to enhance the blurred image in the pollen sample image to obtain a synthetic clear pollen image, and obtains a classification model by training the synthetic clear pollen image and the clear pollen image in the pollen sample image. The quality classification network, the image enhancement network and the classification network are combined to classify the pollen image to be classified, thereby realizing the application of the quality classification network to automatically judge the image quality of the pollen image, and the application of the image enhancement network to generate a corresponding synthetic clear pollen image from the blurred pollen image, reducing the time cost of manually screening high-quality pollen data, and making full use of the out-of-focus blurred pollen images to expand the training samples of the classification model, thereby improving the accuracy of the classification model in pollen recognition.

[0082] Based on the above embodiments, Figure 2 FIG. 1 is a flow chart of the quality classification network training method provided by the present invention. Figure 2 As shown, the training steps of the quality classification network are as follows:

[0083] Step 210, determining pollen images with quality labels, clear pollen images, and an initial quality classification network;

[0084] It should be noted that pollen images with quality labels include real pollen images and their corresponding quality labels. Quality labels can be calculated using traditional no-reference image quality metrics and represent the quality score of an image. The initial quality classification network is a Siamese network. Clear pollen images are real pollen images that have been manually judged to be clear.

[0085] Step 220 , performing blurring processing on the clear pollen image to different degrees, and combining the obtained pollen images with known blurring degrees into image pairs, thereby obtaining pollen image pairs with known blurring degrees;

[0086] Considering that pollen images with different blur levels are difficult to obtain, the embodiment of the present invention performs different degrees of blurring on the clear pollen images obtained through manual judgment, and can obtain a large number of synthetic pollen images with known blur levels, thereby reducing the labor cost of training the quality classification network.

[0087] It should be noted that, after blurring, two pollen images with different blurring degrees are combined into a pollen image pair.

[0088] Step 230: Input the first image in the pollen image pair to the first network branch of the initial mass classification network to obtain a scalar value of the first image output by the first network branch; simultaneously, input the second image in the pollen image pair to the second network branch of the initial mass classification network to obtain a scalar value of the second image output by the second network branch; the network structure of the first network branch and the network structure of the second network branch are the same;

[0089] Step 240: determining a contrast loss based on the scalar value of the first image and the scalar value of the second image, and iteratively training the initial quality classification network based on the contrast loss until convergence to obtain an intermediate first network branch and an intermediate second network branch;

[0090] Step 250: Based on the pollen images with quality labels, the intermediate first network branch is trained and fine-tuned, and the trained intermediate first network branch is used as a quality classification network.

[0091] Considering that if the initial quality classification network is trained using only one stage, it is impossible to output image quality scores within the specified numerical range, therefore, after the initial quality classification network is trained using synthetic pollen images, one branch of the initial quality classification network is fine-tuned using real original pollen images and original pollen image quality labels, i.e., the second stage of training, to ensure that the quality classification results of the quality classification network obtained by the final training can fall within the specified numerical range, thereby providing a unified standard for subsequent judgments.

[0092] Specifically, the initial quality classification network is a twin network, comprising a first network branch and a second network branch, wherein the network structure of the first network branch is identical to that of the second network branch. The first image of a pollen image pair is input into the first network branch to obtain a scalar value of the first image output by the first network branch. Simultaneously, the second image of the pollen image pair is input into the second network branch to obtain a scalar value of the second image output by the second network branch. A contrast loss is then calculated based on the scalar values ​​of the first and second images to obtain a contrast loss. The initial quality classification network is then iteratively trained based on the contrast loss until the contrast loss converges, thereby obtaining an intermediate first network branch and an intermediate second network branch.

[0093] It should be noted that Hinge loss can be used to calculate the contrast loss based on the scalar value of the first image and the scalar value of the second image. The formula of the loss function is as follows:

[0094]

[0095] Where x1 represents the first image, x2 represents the second image, A scalar value representing the first image output by the first network branch, represents the scalar value of the second image output by the second network branch, and α represents the preset spacing to increase and The distance between them.

[0096] After obtaining the intermediate first network branch and the intermediate second network branch, the intermediate first network branch is trained and fine-tuned using the original pollen image and the original pollen image quality label, and the trained intermediate first network branch is used as the quality classification network.

[0097] It should be noted that, according to the characteristics of the twin network, the parameters of the first and second network branches are the same, so the second network branch can also be trained and fine-tuned. The loss function formula of the fine-tuning stage is as follows:

[0098]

[0099] Among them, N represents the number of images input to the first network branch, y i represents the quality label of the original pollen image, The quality score representing the quality label predicted by the first network branch.

[0100] The quality classification network compares the quality score of the output quality label with a preset threshold, and finally obtains the image quality result. The preset threshold is set based on cluster analysis.

[0101] Based on the above embodiments, Figure 3 FIG. 1 is a flow chart of the image enhancement network training method provided by the present invention. Figure 3 As shown in Figure 2, the training steps of the image enhancement network are as follows:

[0102] Step 310: determining unpaired clear pollen images and blurred pollen images, and an initial image enhancement network; the initial image enhancement network includes a clear image generation network branch and a blurred image generation network branch;

[0103] If clear pollen images can be generated from blurred pollen images, a large number of these images can be used to train the classification model, expanding the training sample set for the classification model and thereby improving the classification model's pollen recognition accuracy. Therefore, this embodiment of the present invention trains a recurrent generative adversarial model and uses the clear image generation network within this model as the image enhancement network.

[0104] Specifically, the initial image enhancement network is a cyclic adversarial generation model, which includes a clear image generation network branch and a blurred image generation network branch.

[0105] It should be noted that the unpaired clear pollen images and fuzzy pollen images are obtained by inputting pollen images into the quality classification network to obtain a clear pollen image set and a fuzzy pollen image set, and then randomly selecting one from the clear pollen image set and the fuzzy pollen image set to obtain the unpaired clear pollen images and fuzzy pollen images.

[0106] Step 320: Input the blurred pollen image into the clear image generation network branch to obtain a first synthesized clear image, and simultaneously input the clear image into the blurred image generation network branch to obtain a first synthesized blurred image;

[0107] Step 330: Input the first synthesized clear image to the blurred image generation branch to obtain a second synthesized blurred image, and simultaneously input the first synthesized blurred image to the clear image generation branch to obtain a second synthesized clear image;

[0108] Step 340: Apply a fuzzy domain discriminator based on the fuzzy pollen image and the first synthesized fuzzy image to determine a fuzzy domain adversarial loss; and apply a clear domain discriminator based on the clear pollen image and the first synthesized clear image to determine a clear domain adversarial loss.

[0109] Step 350 , determining a cycle consistency loss based on the blurred pollen image, the second synthesized blurred image, the clear pollen image, and the second synthesized clear image;

[0110] Step 360: Determine the blur domain perceptual loss based on the semantic features of the blurry pollen image and the semantic features of the second synthesized blurry image; and determine the clear domain perceptual loss based on the semantic features of the clear pollen image and the semantic features of the second synthesized clear image.

[0111] In step 370, a joint loss is determined based on the fuzzy domain adversarial loss, the clear domain adversarial loss, the cycle consistency loss, the fuzzy domain perceptual loss, and the clear domain perceptual loss. Based on the joint loss, the parameters of the initial image enhancement network are iterated until the joint loss converges, and the clear image generation network branch is used as the image enhancement network.

[0112] Specifically, the fuzzy pollen image is input into the sharp image generation network branch, and the sharp pollen image is also input into the fuzzy image generation network branch. This produces a first synthesized sharp image output by the sharp image generation network branch, and a first synthesized fuzzy image output by the fuzzy image generation network branch. At this point, the image completes the conversion from the source domain to the target domain. The first synthesized sharp image is then input into the fuzzy image generation network branch, and the first synthesized fuzzy image is also input into the sharp image generation network branch. This produces a second synthesized sharp image output by the sharp image generation network branch, and a second synthesized fuzzy image output by the fuzzy image generation network branch. At this point, the image completes the conversion from the target domain to the source domain.

[0113] The fuzzy domain discriminator discriminates the fuzzy pollen image and the first synthesized fuzzy image, and calculates the fuzzy domain adversarial loss based on the discrimination results. The clear domain discriminator discriminates the clear pollen image and the first synthesized clear image, and calculates the clear domain adversarial loss based on the discrimination results. The formula of the loss function of the fuzzy domain adversarial loss is as follows:

[0114]

[0115] Where, represents the expected value of the distribution function, p(b) represents the distribution of fuzzy pollen images, p(s) represents the distribution of clear pollen images, and D b represents the fuzzy domain discriminator, G b Represents the blurred image generation network branch.

[0116] The formula of the loss function of the clear domain adversarial loss is as follows:

[0117]

[0118] Where, represents the expected value of the distribution function, p(b) represents the distribution of fuzzy pollen images, p(s) represents the distribution of clear pollen images, and D s represents the clear domain discriminator, G s Represents the clear image generation network branch.

[0119] It should be noted that the fuzzy domain adversarial loss is used to distinguish true and false fuzzy pollen images from fuzzy pollen images, and the clear domain adversarial loss is used to distinguish true and false clear pollen images from clear pollen images.

[0120] After calculating the clear domain adversarial loss and the blur domain adversarial loss, the cycle consistency loss is calculated based on the blurry pollen image, the second synthesized blurry image, the clear pollen image, and the second synthesized clear image. The formula of the cycle consistency loss function is as follows:

[0121]

[0122] Where, represents the expected value of the distribution function, p(b) represents the distribution of fuzzy pollen images, p(s) represents the distribution of clear pollen images, b represents fuzzy pollen images, s represents clear pollen images, G s represents the clear image generation network branch, G b Represents the blurred image generation network branch.

[0123] It should be noted that the cycle consistency loss is used to further limit the space of generated samples and retain the content of the original image so that the content information of the cycle image is consistent with the original image.

[0124] Considering that the features extracted from the pre-trained deep network contain rich semantic information, their distance can be used as a perceptual similarity judgment. Therefore, the embodiment of the present invention adds a perceptual loss to preserve the original image structure and restore all texture information.

[0125] Specifically, after calculating the cycle consistency loss, based on the pre-trained CNN model, the semantic features of the blurred pollen image, the semantic features of the second synthetic blurred image, the semantic features of the clear pollen image, and the semantic features of the second synthetic clear image are extracted using the lth layer of the CNN model. Then, based on the semantic features of the blurred pollen image and the semantic features of the second synthetic blurred image, the fuzzy domain perception loss is calculated, and the clear domain perception loss is calculated using the semantic features of the clear pollen image and the semantic features of the second synthetic clear image. The loss function formula of the fuzzy domain perception loss is as follows:

[0126]

[0127] Where, φ l (x) represents the semantic features of image x in the first layer of the pre-trained CNN, b represents the blurred pollen image, G s represents the clear image generation network branch, G b Represents the blurred image generation network branch.

[0128] The loss function formula of the clear domain perception loss is as follows:

[0129]

[0130] Where, φ l (x) represents the semantic features of image x in the first layer of the pre-trained CNN, s represents a clear pollen image, G s represents the clear image generation network branch, G b Represents the blurred image generation network branch.

[0131] After obtaining the fuzzy domain adversarial loss, clear domain adversarial loss, cycle consistency loss, fuzzy domain perceptual loss, and clear domain perceptual loss, the joint loss is calculated based on the above losses. The formula for the joint loss is as follows:

[0132] L=λ adv L adv +λ cyc L cyc +λ per L per

[0133] Where λ adv represents the preset weight of the adversarial loss, λ cyc represents the preset weight of cycle consistency loss, λ per Represents the preset weight of perceptual loss.

[0134] L adv =L s +L b

[0135] L per =L per_s +L per_b

[0136] After obtaining the joint loss, the parameters of the initial image enhancement network are iterated according to the joint loss until the joint loss converges, and the clear image generation network branch is used as the image enhancement network.

[0137] also, Figure 4 This is a schematic diagram of the initial image enhancement network architecture provided by the present invention. Figure 4 As shown, the architecture includes a clear image generation network branch G s , fuzzy image generation network branch G b , clear domain discriminator D s and fuzzy domain discriminator D b The fuzzy pollen image B passes through the clear image generation network branch G s Generate the first synthetic clear image B s , the first synthetic clear image B s After the blurred image generates the network branch G b Generate the second synthetic blurred image B', the clear pollen image S passes through the blurred image generation network branch G b Generate the first synthetic blurred image S b , the first synthetic blurred image S b After the clear image generation network branch G s Generate the second synthetic clear image S'. b Used to distinguish the blurred pollen image B and the first synthesized blurred image S b True or false, clear domain discriminator D s Used to distinguish the clear pollen image S and the first synthesized clear image B s of truth or falsehood.

[0138] Based on the above embodiments, Figure 5 This is a flow chart of the classification model training method provided by the present invention. Figure 5 As shown in Figure 2, the training steps of the classification model are as follows:

[0139] Step 510, determining a pollen sample image, a pollen category label, and an initial classification model;

[0140] Step 520 , applying basic data augmentation and / or cut-out occlusion data augmentation based on the pollen sample image to obtain a pollen augmented image;

[0141] Step 530 : Training the initial classification model based on the pollen augmented image and the pollen category label to obtain a classification model.

[0142] Considering that pollen sample images are digital pathological section images with a relatively single type, which easily causes the classification model to overfit, the present invention implements the method of augmenting the pollen sample images to obtain a large number of pollen augmented images, and uses the pollen augmented images as samples to train the classification model, which can increase the diversity of samples and prevent overfitting.

[0143] Specifically, the pollen sample image is augmented through basic data augmentation and / or cut occlusion data augmentation to obtain a pollen augmented image. Basic data augmentation includes geometric transformations and color space transformations. Geometric transformations include horizontal flipping, vertical flipping, and rotation operations; color space transformations include grayscale, brightness, contrast, saturation, and hue transformations. Cut occlusion data augmentation is mainly achieved by cutting the image into k image blocks of equal size around a center point and converting these image blocks into black occlusions during the training phase.

[0144] After obtaining the pollen augmented image, a sample set is formed and combined with the pollen category label to train the initial classification model to obtain a classification model.

[0145] Based on the above embodiment, step 420 includes:

[0146] Step 421: Based on the pollen sample image, an impurity removal unit is applied to remove impurities from the pollen sample image to obtain a clean pollen sample image.

[0147] Step 422: Based on the clean pollen sample image, apply the basic data augmentation and / or the cut-out occlusion data augmentation to obtain a pollen augmented image to be processed;

[0148] Step 423 : normalize the pollen augmented image to be processed to obtain the pollen augmented image.

[0149] Considering that there may be irrelevant information such as bubbles and impurities in the pollen image, which will affect the feature learning process of the classification model, and considering that pollen grains, bubbles and impurities have different color characteristics (pollen grains are purple, impurities are mostly brown, and bubbles are white). Therefore, the embodiment of the present invention uses an impurity removal unit to remove impurities in the pollen image based on color features. At the same time, considering that when training the classification model, if sample images of the same specifications are used, the average brightness value of the image can be removed to highlight individual differences, which helps to improve the training effect of the classification model and make the classification results obtained by the classification model more accurate.

[0150] Specifically, the pollen sample image is input into the impurity removal unit for impurity removal operation to obtain a clean pollen sample image, and then the clean pollen sample image is subjected to basic data augmentation and / or cut occlusion data augmentation to obtain a pollen augmented image to be processed. Finally, the pollen augmented image to be processed is adjusted to a preset fixed size, and the augmented image is normalized according to the mean and variance of the pollen grain dataset to remove the average brightness value of the image, highlight individual differences, and obtain a pollen augmented image.

[0151] It should be noted that the impurity removal unit can generate a mask corresponding to the image by performing denoising, brightening, color space conversion, binarization, opening operation, closing operation and contour detection operations on the image in sequence, and then use the generated mask to remove impurities. The embodiment of the present invention does not limit this.

[0152] Based on the above embodiment, the impurity removal steps of the impurity removal unit are as follows:

[0153] Step 610, determining the pollen image to be removed of impurities;

[0154] Step 620: After removing noise points and brightening the image of the pollen to be removed, the color space of the image of the pollen to be removed is converted into HSV and a preset binarization threshold is applied to the image to obtain a binarized image;

[0155] Step 630 , after performing closing and opening operations on the binary image, contour detection is performed on the binary image to obtain a contour set;

[0156] Step 640: Based on the radius of the circumscribed circle of each contour in the contour set, contours having a circumscribed circle radius smaller than a preset radius threshold are removed to obtain a contour map; and a mask image is determined based on the contours in the contour map.

[0157] Step 650: Determine a pure pollen image based on the mask image and the image of pollen with impurities to be removed.

[0158] It should be noted that Figure 6 Schematic diagram of the impurity removal process provided by the present invention. Figure 6 As shown, the image of the impurity pollen to be removed is smoothed using the mean shift algorithm and Gaussian filtering to remove noise points in the image, and the image multiplication operation is used to multiply the image by a constant greater than 1, preferably 1.5, to increase the brightness of the image. After completing the above operations, the color space of the image of the impurity pollen to be removed is converted from RGB to HSV, and the image of the impurity pollen to be removed is binarized using the HSV color space to obtain a binary image. Afterwards, a closing operation is performed on the binary image to fill small closed areas, and an opening operation is performed on the binary image to eliminate noise outside the object. After performing the closing and opening operations, contour detection is performed on the binary image to obtain a contour set, and the circumscribed circle of each contour in the contour set is determined. Contours with a circumscribed circle radius less than a preset radius threshold are removed to remove smaller impurities with colors similar to pollen, and a contour map is obtained. Then, a pollen image mask is made according to the contours in the contour map to obtain a mask image. Finally, the mask image is used to cover the pollen image with impurities to be removed, and a flood fill algorithm is used to make the background color of the pollen image more similar to the original color, and a pure pollen image is obtained.

[0159] Based on the above embodiment, step 140 includes:

[0160] Step 141 , applying an impurity removal unit based on the clear pollen image to be classified to determine a pure and clear pollen image to be classified;

[0161] Step 142: Apply a classification model based on the pure and clear pollen image to be classified to obtain a classification result.

[0162] Considering that impurities may also exist in the clear pollen image to be classified, which will have a negative impact on the classification results of the classification model and lead to incorrect classification results, the embodiment of the present invention uses an impurity removal unit to remove impurities from the clear pollen image to be classified to obtain a pure and clear pollen image to be classified, and then inputs the pure and clear pollen image to be classified into the classification model to obtain the classification result output by the classification model.

[0163] The pollen classification device provided by the present invention is described below. The pollen classification device described below and the pollen classification method described above can be referenced to each other.

[0164] Figure 7 Schematic diagram of the structure of the pollen classification device provided by the present invention. Figure 7 As shown, the pollen classification device provided by the embodiment of the present invention includes: a determination module 710 , a quality judgment module 720 , an enhancement module 730 and a classification module 740 .

[0165] A determination module 710 is used to determine the pollen image to be classified;

[0166] The quality judgment module 720 is used to apply a quality classification network based on the pollen image to be classified to determine the image quality result;

[0167] The enhancement module 730 is configured to apply an image enhancement network to the pollen image to be classified to determine a clear pollen image to be classified if the image quality result is fuzzy; otherwise, use the pollen image to be classified as the clear pollen image to be classified;

[0168] A classification module 740 is configured to apply a classification model to the clear pollen image to be classified to obtain a classification result;

[0169] The classification network is trained based on pollen sample images and pollen category labels; pollen sample images include clear pollen images and synthetic clear pollen images; synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; clear pollen images and blurred pollen images are determined by inputting pollen images into the quality classification network; the quality classification network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on clear pollen images and blurred pollen images.

[0170] The pollen classification device provided by the embodiment of the present invention can be used to determine the pollen image to be classified through a determination module; a quality judgment module is used to apply a quality classification network based on the pollen image to be classified to determine the image quality result; an enhancement module is used to apply an image enhancement network based on the pollen image to be classified to determine a clear pollen image to be classified if the image quality result is fuzzy; otherwise, the pollen image to be classified is used as the clear pollen image to be classified; a classification module is used to apply a classification model based on the clear pollen image to be classified to obtain a classification result; the classification network is obtained by training based on the pollen sample image and the pollen category label; the pollen sample image includes a clear pollen image and a synthetic clear pollen image; the synthetic clear pollen image is obtained by applying the model The blurred pollen image is input into the image enhancement network; the clear pollen image and the blurred pollen image are determined by inputting the pollen image into the quality classification network; the quality classification network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on clear pollen images and blurred pollen images, realizing the automatic judgment of the image quality of pollen images by the quality classification network, and generating corresponding synthetic clear pollen images from blurred pollen images by the image enhancement network, reducing the time cost of manual screening of high-quality pollen data, and making full use of the out-of-focus blurred pollen images to expand the training samples of the classification model, thereby improving the accuracy of the classification model in pollen recognition.

[0171] Based on any of the above embodiments, the pollen classification device further includes a quality evaluation network training module, which includes:

[0172] The first determination submodule is used to determine the pollen image with quality label, the clear pollen image and the initial quality evaluation network;

[0173] An image pair determination submodule is used to perform different degrees of blurring on clear pollen images, and to form image pairs of pollen images with known blurring degrees, thereby obtaining pollen image pairs with known blurring degrees;

[0174] a branch scalar determination submodule, configured to input the first image of the pollen image pair into the first network branch of the initial quality assessment network to obtain a scalar value of the first image output by the first network branch; and simultaneously input the second image of the pollen image pair into the second network branch of the initial quality assessment network to obtain a scalar value of the second image output by the second network branch; the network structure of the first network branch and the network structure of the second network branch are the same;

[0175] an intermediate training submodule, configured to determine a contrast loss based on a scalar value of the first image and a scalar value of the second image, and iteratively train the initial quality assessment network based on the contrast loss until convergence to obtain an intermediate first network branch and an intermediate second network branch;

[0176] The fine-tuning training submodule is used to train and fine-tune the first intermediate network branch based on pollen images with quality labels, and use the trained first intermediate network branch as the quality evaluation network.

[0177] Based on any of the above embodiments, the pollen classification device further includes an image enhancement network training module, which includes:

[0178] A second determination submodule is used to determine unpaired clear pollen images and blurred pollen images, and an initial image enhancement network; the initial image enhancement network includes a clear image generation network branch and a blurred image generation network branch;

[0179] a forward submodule, configured to input the blurred pollen image into a clear image generation network branch to obtain a first synthesized clear image, and simultaneously input the clear image into a blurred image generation network branch to obtain a first synthesized blurred image;

[0180] a reverse submodule, configured to input the first synthesized clear image into the blurred image generation branch to obtain a second synthesized blurred image, and simultaneously input the first synthesized blurred image into the clear image generation branch to obtain a second synthesized clear image;

[0181] an adversarial loss submodule for applying a fuzzy domain discriminator based on the fuzzy pollen image and the first synthetic fuzzy image to determine a fuzzy domain adversarial loss; and applying a clear domain discriminator based on the clear pollen image and the first synthetic clear image to determine a clear domain adversarial loss;

[0182] a cycle consistency submodule for determining a cycle consistency loss based on the blurred pollen image, the second synthesized blurred image, the clear pollen image, and the second synthesized clear image;

[0183] a perception submodule, configured to determine a fuzzy domain perception loss based on the semantic features of the fuzzy pollen image and the semantic features of the second synthesized fuzzy image; and to determine a clear domain perception loss based on the semantic features of the clear pollen image and the semantic features of the second synthesized clear image;

[0184] The training submodule is used to determine the joint loss based on the fuzzy domain adversarial loss, the clear domain adversarial loss, the cycle consistency loss, the fuzzy domain perceptual loss and the clear domain perceptual loss, and iterate the parameters of the initial image enhancement network based on the joint loss until the joint loss converges, and use the clear image generation network branch as the image enhancement network.

[0185] Based on any of the above embodiments, the pollen classification device further includes a classification model training module, which includes:

[0186] The third determination submodule is used to determine the pollen sample image, pollen category label and initial classification model;

[0187] An augmentation submodule, configured to apply basic data augmentation and / or cut-out occlusion data augmentation based on the pollen sample image to obtain a pollen augmented image;

[0188] The training submodule is used to train the initial classification model based on the pollen augmented image and the pollen category label to obtain a classification model.

[0189] Based on any of the above embodiments, the augmentation submodule includes:

[0190] an impurity removal submodule, configured to apply an impurity removal unit to the pollen sample image to remove impurities from the pollen sample image to obtain a clean pollen sample image;

[0191] A sample augmentation submodule, configured to apply the basic data augmentation and / or the cut-out occlusion data augmentation based on the clean pollen sample image to obtain a pollen augmented image to be processed;

[0192] The normalization submodule is used to normalize the pollen augmented image to be processed to obtain the pollen augmented image.

[0193] Based on any of the above embodiments, the pollen classification device further includes an impurity removal unit, which includes:

[0194] A fourth determining submodule is used to determine the pollen image to be removed of impurities;

[0195] The basic processing submodule is used to remove noise points and brighten the image of the pollen to be removed, convert the color space of the image of the pollen to be removed into HSV, and apply a preset binarization threshold to binarize the image to obtain a binarized image;

[0196] The contour detection submodule is used to perform contour detection on the binary image after performing closing and opening operations on the binary image to obtain a contour set;

[0197] The mask making submodule is used to remove the contours whose circumscribed circle radius is less than a preset radius threshold based on the radius of the circumscribed circle of each contour in the contour set, thereby obtaining a contour map; and determine a mask image based on the contours in the contour map;

[0198] The impurity processing submodule is used to determine a pure pollen image based on the mask image and the pollen image with impurities to be removed.

[0199] Based on any of the above embodiments, the classification module 740 includes:

[0200] The classification image impurity removal submodule is used to apply the impurity removal unit based on the clear pollen image to be classified to determine the pure and clear pollen image to be classified;

[0201] The classification image classification submodule is used to apply the classification model based on the pure and clear pollen image to be classified to obtain the classification result.

[0202] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the pollen classification method, which includes: determining a pollen image to be classified; applying a quality evaluation network based on the pollen image to be classified to determine an image quality result; if the image quality result is blurred, applying an image enhancement network based on the pollen image to be classified to determine a clear pollen image to be classified; otherwise, using the pollen image to be classified as a clear pollen image to be classified; applying a classification model based on the clear pollen image to be classified to obtain a classification result; the classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen image is obtained by inputting the blurred pollen image into the image enhancement network; the clear pollen image and the blurred pollen image are determined by inputting the pollen image into the quality evaluation network; the quality evaluation network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on clear pollen images and blurred pollen images.

[0203] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0204] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pollen classification method provided by the above methods, which includes: determining a pollen image to be classified; based on the pollen image to be classified, applying a quality evaluation network to determine an image quality result; if the image quality result is fuzzy, applying an image enhancement network to the pollen image to be classified to determine a clear pollen image to be classified; otherwise, the pollen image to be classified is used as a clear pollen image to be classified; based A classification model is applied to the clear pollen images to be classified to obtain the classification results; the classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; the clear pollen images and blurred pollen images are determined by inputting the pollen images into the quality evaluation network; the quality evaluation network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on clear pollen images and blurred pollen images.

[0205] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the pollen classification method provided by the above-mentioned methods, the method comprising: determining a pollen image to be classified; applying a quality evaluation network to determine an image quality result based on the pollen image to be classified; if the image quality result is blurred, applying an image enhancement network to determine a clear pollen image to be classified based on the pollen image to be classified; otherwise, using the pollen image to be classified as a clear pollen image to be classified; applying a classification model to obtain a classification result based on the clear pollen image to be classified; the classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen image is obtained by inputting the blurred pollen image into the image enhancement network; the clear pollen image and the blurred pollen image are determined by inputting the pollen image into the quality evaluation network; the quality evaluation network is obtained by two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; the image enhancement network is trained based on clear pollen images and blurred pollen images.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0207] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A pollen classification method, characterized in that: include: Determine the pollen image to be classified; Based on the pollen image to be classified, applying a quality classification network to determine an image quality result; If the image quality result is fuzzy, applying an image enhancement network based on the pollen image to be classified to determine a clear pollen image to be classified; otherwise, using the pollen image to be classified as the clear pollen image to be classified; Based on the clear pollen image to be classified, applying a classification model to obtain a classification result; The classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; the clear pollen images and the blurred pollen images are determined by inputting pollen images into the quality classification network; the quality classification network is obtained by performing two-stage training on pollen images with known blur levels and pollen images with quality labels; the image enhancement network is trained based on the clear pollen images and the blurred pollen images; The training steps of the image enhancement network are as follows: Determining the unpaired clear pollen image and the fuzzy pollen image, and an initial image enhancement network; the initial image enhancement network includes a clear image generation network branch and a fuzzy image generation network branch; Inputting the blurred pollen image into the clear image generation network branch to obtain a first synthesized clear image, and simultaneously inputting the clear image into the blurred image generation network branch to obtain a first synthesized blurred image; Inputting the first synthesized clear image into the blurred image generation branch to obtain a second synthesized blurred image, and simultaneously inputting the first synthesized blurred image into the clear image generation branch to obtain a second synthesized clear image; Based on the blurred pollen image and the first synthesized blurred image, applying a fuzzy domain discriminator to determine a fuzzy domain adversarial loss; and applying a clear domain discriminator based on the clear pollen image and the first synthesized clear image to determine a clear domain adversarial loss; determining a cycle consistency loss based on the blurred pollen image, the second synthesized blurred image, the sharp pollen image, and the second synthesized sharp image; determining a blur domain perceptual loss based on the semantic features of the blurred pollen image and the semantic features of the second synthesized blurred image; and determining a clear domain perceptual loss based on the semantic features of the clear pollen image and the semantic features of the second synthesized clear image; Based on the blur domain adversarial loss, the clear domain adversarial loss, the cycle consistency loss, the blur domain perceptual loss and the clear domain perceptual loss, a joint loss is determined, and based on the joint loss, parameters of the initial image enhancement network are iterated until the joint loss converges, and the clear image generation network branch is used as the image enhancement network.

2. The pollen classification method according to claim 1, characterized in that: The training steps of the quality classification network are as follows: Determine the pollen images with quality labels, clear pollen images, and initial quality classification networks; Performing blurring processing of different degrees on the clear pollen image, and combining the obtained pollen images with known blurring degrees into image pairs, thereby obtaining pollen image pairs with known blurring degrees; Inputting the first image of the pollen image pair into the first network branch of the initial mass classification network to obtain a scalar value of the first image output by the first network branch; and simultaneously inputting the second image of the pollen image pair into the second network branch of the initial mass classification network to obtain a scalar value of the second image output by the second network branch; The network structure of the first network branch is the same as the network structure of the second network branch; Determining a contrast loss based on the scalar value of the first image and the scalar value of the second image, and iteratively training the initial quality classification network based on the contrast loss until convergence to obtain an intermediate first network branch and an intermediate second network branch; Based on the pollen image with the quality label, the intermediate first network branch is trained and fine-tuned, and the trained intermediate first network branch is used as the quality classification network.

3. The pollen classification method according to claim 1, characterized in that: The training steps of the classification model are as follows: Determine pollen sample images, pollen category labels, and initial classification models; Based on the pollen sample image, applying basic data augmentation and / or cutting and occlusion data augmentation to obtain a pollen augmented image; The initial classification model is trained based on the pollen augmented image and the pollen category label to obtain the classification model.

4. The pollen classification method according to claim 3, characterized in that: The step of applying basic data augmentation and / or cutting and occluding data augmentation based on the pollen sample image to obtain a pollen augmented image includes: Based on the pollen sample image, applying an impurity removal unit to remove impurities from the pollen sample image to obtain a clean pollen sample image; Based on the clean pollen sample image, applying the basic data augmentation and / or the cut-occluded data augmentation to obtain a pollen augmented image to be processed; Normalizing the pollen augmented image to be processed to obtain the pollen augmented image.

5. The pollen classification method according to claim 4, characterized in that: The impurity removal steps of the impurity removal unit are as follows: Determine the pollen image to be removed; After removing noise points and brightening the image of the pollen to be removed, the color space of the pollen to be removed is converted into HSV and a preset binarization threshold is applied to perform image binarization to obtain a binarized image; After performing closing and opening operations on the binary image, performing contour detection on the binary image to obtain a contour set; Based on the radius of the circumscribed circle of each contour in the contour set, contours having a circumscribed circle radius smaller than a preset radius threshold are removed to obtain a contour map; and based on the contours in the contour map, a mask image is determined; A pure pollen image is determined based on the mask image and the image of the pollen to be removed from impurities.

6. The pollen classification method according to claim 5, characterized in that: The step of applying a classification model based on the clear pollen image to be classified to obtain a classification result includes: Based on the clear pollen image to be classified, applying the impurity removal unit to determine a pure and clear pollen image to be classified; Based on the pure and clear pollen image to be classified, the classification model is applied to obtain a classification result.

7. A pollen classification device, characterized in that: include: A determination module, used for determining the pollen image to be classified; A quality judgment module, configured to apply a quality classification network to the pollen image to be classified and determine an image quality result; an enhancement module configured to, if the image quality result is fuzzy, apply an image enhancement network to determine a clear pollen image to be classified based on the pollen image to be classified; otherwise, use the pollen image to be classified as the clear pollen image to be classified; A classification module, configured to apply a classification model based on the clear pollen image to be classified to obtain a classification result; The classification network is trained based on pollen sample images and pollen category labels; the pollen sample images include clear pollen images and synthetic clear pollen images; the synthetic clear pollen images are obtained by inputting blurred pollen images into the image enhancement network; the clear pollen images and the blurred pollen images are determined by inputting pollen images into the quality classification network; the quality classification network is obtained by performing two-stage training based on pollen image pairs with known blur levels and pollen images with quality labels; The image enhancement network is trained based on the clear pollen image and the blurred pollen image; Also included is an image enhancement network training module, which includes: A second determination submodule is used to determine unpaired clear pollen images and blurred pollen images, and an initial image enhancement network; the initial image enhancement network includes a clear image generation network branch and a blurred image generation network branch; a forward submodule, configured to input the blurred pollen image into a clear image generation network branch to obtain a first synthesized clear image, and simultaneously input the clear image into a blurred image generation network branch to obtain a first synthesized blurred image; a reverse submodule, configured to input the first synthesized clear image into the blurred image generation branch to obtain a second synthesized blurred image, and simultaneously input the first synthesized blurred image into the clear image generation branch to obtain a second synthesized clear image; an adversarial loss submodule for applying a fuzzy domain discriminator based on the fuzzy pollen image and the first synthetic fuzzy image to determine a fuzzy domain adversarial loss; and applying a clear domain discriminator based on the clear pollen image and the first synthetic clear image to determine a clear domain adversarial loss; a cycle consistency submodule for determining a cycle consistency loss based on the blurred pollen image, the second synthesized blurred image, the clear pollen image, and the second synthesized clear image; a perception submodule, configured to determine a fuzzy domain perception loss based on the semantic features of the fuzzy pollen image and the semantic features of the second synthesized fuzzy image; and to determine a clear domain perception loss based on the semantic features of the clear pollen image and the semantic features of the second synthesized clear image; The training submodule is used to determine the joint loss based on the fuzzy domain adversarial loss, the clear domain adversarial loss, the cycle consistency loss, the fuzzy domain perceptual loss and the clear domain perceptual loss, and iterate the parameters of the initial image enhancement network based on the joint loss until the joint loss converges, and use the clear image generation network branch as the image enhancement network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the pollen classification method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the pollen classification method according to any one of claims 1 to 6 is implemented.

Images