An image classification method based on multi-view information fusion

By adopting a multi-view information fusion method in pollen classification, combining global and local information flow, and using segmentation guidance network for pollen particle positioning and feature extraction, the problem of ignoring details and global structure in the existing technology is solved, and more accurate pollen classification.

CN114648664BActive Publication Date: 2024-11-22SHENZHEN XINGYIFAN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210290777.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-23
Publication Date
2024-11-22
Estimated Expiration
2042-03-23

AI Technical Summary

Technical Problem

Existing deep learning methods ignore image details in pollen classification, resulting in inaccurate identification of subtle texture changes of pollen particles. Only focusing on local information will ignore global structural information, affecting the classification results.

Method used

A method of image classification based on multi-view information fusion was designed. By using deep learning networks to classify the global image information and local pollen particle regional information flow, and introducing segmentation guidance networks for pollen particle positioning and feature extraction, the final The information of the two is weighted to generate the final classification result.

Benefits of technology

It improves the accuracy of pollen classification, comprehensively learns global and local information, enhances the ability to identify subtle texture changes of pollen particles, and improves the accuracy of classification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114648664B_ABST
    Figure CN114648664B_ABST
Patent Text Reader

Abstract

An image classification method based on multi-view information fusion belongs to the field of computer vision and has the effect of comprehensively learning global image information and local pollen grain area information, which is more accurate than the previous method of only considering global image information. On the basis of paying attention to global image information, the present invention pays more targeted attention to local pollen grain area information. The network used contains two deep learning streams, corresponding to different levels and structural levels of pollen images. The first is classification based on the global image information flow, and pollen is directly classified on the global image structure; the second is classification based on the local pollen grain area information flow, in which a segmentation guidance network is added to locate pollen grains from the entire pollen image, retain pollen grain area pictures, and detect pollen categories at the local area image level; finally, the global image classification results and the local pollen grain area classification results are fused to obtain the final classification results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an image classification method based on multi-view information fusion, and belongs to the field of computer vision. Background Art

[0002] Pollen allergy is a common disease in allergy department. Identifying pollen types is important for diagnosis, treatment and prevention of allergy. Traditional pollen classification is mainly done manually by biological research experts under a microscope, but this classification process consumes a lot of time and human resources. Therefore, we need to explore a method to automatically classify pollen to improve classification efficiency.

[0003] With the rapid development of deep learning, there are now many deep learning methods for image classification, but most existing deep learning methods directly focus on the global image and need to downsample the input image to a low resolution in advance to make the network size manageable. However, this downsampling will lead to the loss of image details, which has a great impact on the identification of subtle texture changes in pollen grains. In contrast, local pollen grain regions have good feature representation, but this approach of focusing only on local area information will ignore global structural information.

[0004] In response to the above problems, the present invention proposes an image classification method that integrates global image information and local pollen grain area information. In view of the different attention levels of global image information and local pollen grain area information, the two types of information are integrated during the image classification process to improve the image classification performance. Summary of the invention

[0005] The entire pollen picture is very large and the area occupied by the gray background information is huge, while the pollen grain area is relatively small, which is far from the proportion of the entire picture. Existing image classification methods directly focus on global image information, which ignores image details, which is very important for identifying subtle texture changes in pollen grains, thus greatly affecting the image classification results. In response to this problem, the present invention designs an image classification method based on multi-perspective information fusion, which fuses the image classification results under the global perspective and the pollen grain area image classification results under the local perspective to obtain the final pollen image classification results. The results show that the process and solution we proposed have the effect of comprehensively learning global image information and local pollen grain area information, which is more accurate than the previous method of only considering global image information.

[0006] The present invention adopts the following technical solution: on the basis of paying attention to the global image information, it pays more attention to the local pollen grain area information. We propose a new pollen classification integrated network, which contains two deep learning streams corresponding to different levels and structural levels of pollen images. The first is the classification based on the global image information flow, which directly classifies pollen on the global image structure; the second is the classification based on the local pollen grain area information flow, in which a segmentation guidance network is added to locate the pollen grains from the entire pollen image, retain the pollen grain area picture, and detect the pollen category at the local area image level; finally, the global image classification result and the local pollen grain area classification result are fused to obtain the final classification result.

[0007] The specific scheme of the present invention is as shown in the attached Figure 1 shown.

[0008] S1: Image preprocessing: The pollen image scanned by the machine is converted into a format and cropped to obtain an image that meets the model input.

[0009] S2: Design a classifier based on global pollen image information. Resize the image generated in step S1 and output it to the classification model for image feature extraction, and then output the classification results and probabilities at the global pollen image structure level.

[0010] S3: Design a classifier based on local pollen grain area information. Similarly, the image obtained in step S1 is trained to output the classification results and probabilities of the local pollen grain area at the image structure level.

[0011] S31: Design a local locator. Introduce a segmentation guidance network to locate the pollen grain area in the global image and generate a pollen region localization map. Then, based on this localization map, remove the image redundant information and obtain a fine-grained image of the local pollen grain area.

[0012] S32: Design a local area classifier. Construct a classification network based on the pollen grain area image generated in step S31, train the model, extract image features, and output the classification results and probabilities.

[0013] S4: Design a global-local pollen information fusion device. The two parts of information, namely, the pollen image classification results and probabilities based on global information and the pollen image classification results and probabilities based on local information, are fused by calculating weights to generate the final pollen image classification results.

[0014] S41: Calculate the global image information weight.

[0015] S42: Calculate the weight of local pollen grain area information.

[0016] S43: Calculate the final pollen classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The overall structure diagram of the method of the present invention. DETAILED DESCRIPTION

[0018] The following is a detailed description of an implementation example of the present invention in conjunction with the accompanying drawings in the specification. The overall process is divided into three stages. The first is classification based on global image information, and pollen is directly classified based on the global image structure; the second is classification based on local pollen grain area information. A segmentation guidance network is added to this module to locate pollen grains from the entire pollen image, retain pollen grain area images, and detect pollen categories at the local area image level; finally, the global image classification results and the local pollen grain area classification results are fused to obtain the final classification results. The overall process is as follows: Figure 1 As shown, the specific steps include:

[0019] S1: Image preprocessing

[0020] The pollen image preprocessing includes two steps: image format conversion and image cropping. Image format conversion converts the multi-layer image (ndpi) obtained by the digital slice scanner into a single-layer image (tiff). Image cropping refers to cropping the single-layer image obtained in the previous step into an image of a suitable size (jpg), and finally obtaining an image that meets the model input.

[0021] S2: Design a classifier based on global pollen image information

[0022] The purpose of this module is to calculate the image classification results based on global pollen image information. This module represents the global pollen image structure at the image level and directly classifies the pollen image. The input of the module is the pollen image obtained by S1, and the output is the image classification result and probability based on global information.

[0023] The present invention adopts a network architecture based on the classic CNN network (ResNet-50) as a global information classifier. We resize the image obtained in step S1 to a size of 224×224 as the input of the network model. This network consists of 5 downsampling layers, each of which uses a convolution layer with a convolution kernel of 3×3 and a stride of 2 to extract features from the pollen image; followed by a global maximum pooling layer to reduce the overfitting probability of the model and improve the robustness of pollen particle feature extraction; then a fully connected layer to fuse the deep features of pollen particles learned by the model; finally, the model's classifier performs classification. Finally, the pollen image classification results and probabilities based on global information are output.

[0024] S3: Design a classifier based on local pollen grain regional information

[0025] The purpose of this module is to calculate the classification results and probabilities based on the local pollen grain area information image. The input of the module is the result image obtained by S1. After the segmentation guidance network and the classification network extract the characteristic information of the pollen grains, the output is the classification results and probabilities based on the local pollen grain area information. This module consists of two major stages.

[0026] S31: Local Area Locator

[0027] First, the present invention uses a CNN network (U-Net) to locate the pollen grain area. The network is divided into an encoder and a decoder. Each encoder and a filter form a convolution layer to generate a set of encoder feature maps, and ReLU is used as the activation function. The decoder uses a deconvolution layer to output the decoder feature map. The jump connection transmits the corresponding feature map from the encoder path and connects them to the upsampled decoder feature map. Finally, the classifier uses a 1×1 convolution layer as a pixel classification to generate the probability distribution of pollen grains in the pollen image and obtain the localization map of the pollen grain area.

[0028] The coefficients for evaluating pollen grain segmentation are:

[0029]

[0030] where N is the number of pixels, pi∈[0,1] and ci∈{0,1} represent the predicted probability and binary true label of the pollen grain region, respectively.

[0031] Then, based on the pollen grain localization map generated in the previous process, we remove the redundant background information in the pollen image and obtain a 224×224 fine-grained image of the local pollen grain area, preserving more detailed pollen grain information at a higher resolution.

[0032] S32: Local Region Classifier

[0033] The present invention is based on the classic CNN network, and constructs a classifier network structure to train a local area information classifier, learn the features of pollen grain images, and generate pollen image classification results and probabilities. Here we take ResNet-50 as an example, the main function of which is to learn the features of pollen grain images and then classify them. This network uses a convolution layer with a convolution kernel of 3×3 and a stride of 2 to extract features from pollen grain images, followed by a global average pooling layer (Global Average Pool, GAP) to regularize the entire network structure to prevent overfitting, and finally a softmax classifier is used for classification, and ReLU is used as the activation function. The final model outputs the classification results and probabilities based on local pollen grain area information.

[0034] S4: Design of global-local pollen information fusion

[0035] The purpose of this module is to obtain the final pollen classification result of the image. The input of this module is the pollen image classification result and probability based on global information and the pollen image classification result and probability based on local information. These two parts of information are fused by calculating the weights to generate the final pollen image classification result.

[0036] S41: Calculate global image information weight

[0037] The weight of the global image information is calculated based on the pixels of the pollen grain area and the pixels of the entire image in the pollen grain positioning map obtained in step S3.

[0038] The formula for calculating the weight of global image information is:

[0039]

[0040] Where n represents the number of pixels in the pollen grain area, and N represents the number of pixels in the entire pollen image;

[0041] S42: Calculate the weight of local pollen grain area information

[0042] To ensure the mutual influence and restriction between global image information and local pollen grain area information, we give different control strengths according to different image levels, and make the sum of global information weight and local information weight equal to 1. Therefore, the local score weight is:

[0043] W 2 =1-W 1

[0044] S43: Calculate the final pollen classification results

[0045] P=W 1 ×p global +W 2 ×p local

[0046] where p global Represents the probability of classification result based on global image information, p local represents the probability of the classification result based on the local pollen grain area information, P represents the final probability of the classification result, and the final classification result of the pollen image is obtained according to the final probability P.

[0047] Under specific scenario requirements, we will make corresponding adjustments to the global and local weight values. When the target task does not need to pay attention to the global structural position information, we set the global weight W1 to 0. At this time, we pay more attention to local information, such as when encountering the application scenario of natural images; when the target task requires the global structure of the image, such as position information, we need the information between the two to complement each other, so we use the formulas of steps S41 and S42 to calculate the weights of the two, such as when encountering the application scenario of medical images.

Claims

1. An image classification method based on multi-view information fusion, characterized by: S1: Image preprocessing; The pollen images scanned by the machine are converted into formats and cropped to obtain images that meet the model input requirements; S2: Design a classifier based on global pollen image information; The image generated in step S1 is resized and output to the classification model for image feature extraction, and then the classification result and probability at the global pollen image structure level are output; S3: Design a classifier based on local pollen grain region information; The image obtained in step S1 is trained to output the classification results and probabilities at the level of the image structure of the local pollen grain area; S31: Design local locator; A segmentation guidance network is introduced to locate the pollen grain area in the global image and generate a pollen area positioning map. Then, based on this positioning map, redundant image information is removed to obtain a fine-grained image of the local pollen grain area. S32: Design local region classifier; Building a classification network based on the pollen grain region image generated in step S31, training the model, extracting image features, and outputting classification results and probabilities; S4: Design a global-local pollen information fusion device; fuse the two parts of information, namely, the pollen image classification results and probabilities based on global information and the pollen image classification results and probabilities based on local information, by calculating weights to generate the final pollen image classification results; S41: Calculate the global image information weight; S42: Calculate the weight of local pollen grain area information; S43: Calculate the final classification results of pollen; The details are as follows: S1: Image preprocessing The pollen image preprocessing includes two steps: image format conversion and image cropping. Image format conversion converts the multi-layer image obtained by the digital slice scanner into a single-layer image. Image cropping refers to cropping the single-layer image obtained in the previous step into an image of 224×224 size for model input. S2: Design a classifier based on global pollen image information Calculate the image classification results based on global pollen image information, represent the global pollen image structure at the image level, and directly classify the pollen image. The input is the pollen image obtained by S1, and the output is the image classification result and probability based on global information. The network architecture based on the classic CNN network is used as the global information classifier; The image obtained in step S1 is resized to 224×224 as the input of the network model; this network consists of 5 downsampling layers, each of which uses a convolution layer with a convolution kernel of 3×3 and a stride of 2 to extract features from the pollen image; followed by a global maximum pooling layer to reduce the overfitting probability of the model and improve the robustness of pollen grain feature extraction; Then there is a fully connected layer, which integrates the deep features of pollen grains learned by the model. Finally, the model's classifier performs classification. Finally, the pollen image classification results and probabilities based on global information are output. S3: Design a classifier based on local pollen grain regional information Calculate the classification results and probabilities based on the local pollen grain area information image. The input is the result image obtained by S1. The feature information of the pollen grains is extracted through the segmentation guidance network and the classification network. The output is the classification results and probabilities based on the local pollen grain area information. It includes two major stages. S31: Local Area Locator A CNN network is used to locate the pollen grain area. The network is divided into an encoder and a decoder. Each encoder and a filter form a convolutional layer to generate a set of encoder feature maps, and ReLU is used as the activation function. The decoder uses the deconvolution layer to output the decoder feature map. The skip connection transfers the corresponding feature map from the encoder path and connects them to the upsampled decoder feature map. Finally, the classifier uses the 1×1 convolution layer as pixel classification to generate the probability distribution of pollen grains in the pollen image and obtain the localization map of the pollen grain area. The coefficients for evaluating pollen grain segmentation are: Where N is the number of pixels, p i ∈[0,1] and c i ∈{0,1} denotes the predicted probability and binary true label of the pollen grain region, respectively; According to the pollen grain positioning map generated in the previous process, the redundant background information in the pollen image is removed to obtain a 224×224 fine-grained image of the local pollen grain area, which preserves more detailed pollen grain information at a higher resolution; S32: Local Region Classifier Based on the classic CNN network, a classifier network structure is constructed to train the local area information classifier, learn the features of pollen grain images, and generate pollen image classification results and probabilities; A convolutional layer with a convolution kernel of 3×3 and a stride of 2 is used to extract features from the pollen grain image, followed by a global average pooling layer to regularize the entire network structure to prevent overfitting. Finally, a softmax classifier is used for classification, and ReLU is used as the activation function. The final model outputs the classification results and probabilities based on the local pollen grain area information. S4: Design of global-local pollen information fusion The input is the pollen image classification result and probability based on global information and the pollen image classification result and probability based on local information. The two parts of information are fused by calculating the weights to generate the final pollen image classification result. S41: Calculate global image information weight Calculate the weight of the global image information according to the pixels of the pollen grain area and the pixels of the entire image in the pollen grain positioning map obtained in step S3; The formula for calculating the weight of global image information is: Where n represents the number of pixels in the pollen grain area, and N represents the number of pixels in the entire pollen image; S42: Calculate the weight of local pollen grain area information Different control strengths are given according to different picture levels, so that the sum of the global information weight and the local information weight is 1; therefore, the local score weight is: W2=1-W1 S43: Calculate the final pollen classification results P=W1×p global +W2×p local where p global Represents the probability of classification result based on global image information, p local represents the probability of the classification result based on the local pollen grain area information, P represents the final probability of the classification result, and the final classification result of the pollen image is obtained according to the final probability P.