A fine-grained biological image classification method based on Transformer model
By designing the patch selection module and feature selection module in the fine-grained biological image classification method, the problem of the number of patches changing when different Transformer models extract features is solved, achieving high-accuracy fine-grained biological image classification, and is applicable to any Transformer model. .
Patent Information
- Application Number
- CN202311265910.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2025-01-24
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When different Transformer models are used as backbone networks for feature extraction, the number of extracted patches is different, which makes it impossible to enter the same classification network, and the existing mainstream methods cannot effectively solve this problem, resulting in poor classification of fine-grained biological images.
A fine-grained biological image classification method based on the Transformer model is designed to solve the problem of changes in the number of patches by setting up a patch selection module. The method includes a feature extraction module, a feature selection module and a prediction module. The feature selection module uses a two-layer fully connected network and a softmax function to generate weights, selects an appropriate number of patches as new image features, and optimizes the model through mapping networks and metric learning.
After implementing different Transformer model extraction features, effective feature selection and fine -grained biological image classification can be performed, which improves the accuracy of classification, and is suitable for any Transformer model, which improves the feature extraction efficiency.
Smart Images
Figure CN117593557B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of computer vision, biometric identification, and the like, and in particular to a fine-grained biological image classification method based on a Transformer model. Background Art
[0002] Image classification is a classic task in the field of computer vision. The goal of common fine-grained biological image classification, such as the classification of dogs and birds, is to achieve subclass classification by processing detailed features. Since the biological features between subclasses are similar and the differences are small, accurate classification is difficult. Therefore, fine-grained biological image classification is still a key problem that needs to be solved urgently in the field of computer vision.
[0003] In the current research on fine-grained biological image classification, there is a mainstream method based on the Transformer model. In order to more effectively select key features in this method, some scholars select more expressive patches based on attention scores. Patches are obtained by dividing an image into pixels of a certain size. For example, if the input image is 224x224 pixels in size and is divided into 16x16 pixels in size, (224 / 16)x(224 / 16) slices will be generated. Each slice can extract a patch, called a patch or feature block. However, using attention scores to select key patches is only applicable to Transformer models with a fixed number of patches. For example, in the VIT network, if the number of initial patches generated is 256, the number of output patches after encoding by the Transformer module is still 256. That is, in the above mainstream methods, the number of initial input patches and the number of final output patches are the same, and then more expressive patches are selected from them based on the attention scores to perform fine-grained biological image classification.
[0004] When the number of patches changes, for example, in the swin-transformer network, the number of patches input to the transformer module is 3136. As the modules in different stages of the network process, the number of patches changes from 3136 to 224, then to 196, and finally to 49 outputs. In this case, the number of extracted features has changed relative to the number of input patches. At this time, it is impossible to calculate the attention scores of all features corresponding to the number of input patches. Therefore, the above mainstream methods are not applicable to this situation. Directly using the features of the final output to use the above mainstream methods for fine-grained biological image classification is not effective. Existing studies have shown that the patches extracted from images are a high-dimensional data representation that is not interpretable and difficult to understand even after visualization. Therefore, the selection of feature blocks will be a problem.
[0005] Therefore, when using different transformer models as the backbone network for feature extraction, a separate post-processing module needs to be designed to improve the classification effect of fine-grained biological images. Summary of the invention
[0006] In view of the problem that when different Transformers are used as backbone networks for feature extraction in the study of fine-grained biological image classification, it is difficult to input the same classification network due to the different numbers of extracted patches. The present invention provides a fine-grained biological image classification method based on a Transformer model, which solves the above problem by setting a Patch selection module and improves the efficiency of fine-grained biological image classification.
[0007] The present invention provides a fine-grained biological image classification method based on a Transformer model, comprising the following steps:
[0008] Step 1: Generate training samples using historical biological images, where the labels of the training samples are fine-grained biological categories;
[0009] Step 2, establish a classification model, the classification module includes a feature extraction module, a feature selection module and a prediction module;
[0010] The feature extraction module uses a Transformer-based network to extract features from the input image and output image features represented by feature blocks.
[0011] The feature selection module first uses a two-layer fully connected network to perform nonlinear mapping on the patch, and then generates weights through the softmax function. According to the generated weights, a pre-set number of patches are selected as new image features F. choose ;
[0012] The prediction module is based on the image feature F choose Predict fine-grained biological taxonomy;
[0013] Step 3, using the training samples to train the classification model;
[0014] During training, the projection network is used to choose Map and calculate the cosine distance between samples as the distance loss L distance , use cross entropy loss to calculate the classification loss L cls , based on L distance and L cls Optimize classification model parameters;
[0015] Step 4: Use the trained classification model to perform fine-grained classification of biological images.
[0016] Furthermore, in step 3, during training, for two training samples, the cosine distance is calculated as follows:
[0017] Assume that the feature selection module selects n patches for each training sample and stretches the n patches into a row vector;
[0018] Then calculate the cosine distance of the row vectors corresponding to the two training samples; if the classification labels of the two training samples are inconsistent, the calculated cosine distance is negative.
[0019] For the same batch of training samples, the cosine distances between different training samples are calculated and then summed to obtain the distance loss of the current batch of training samples.
[0020] The method of the present invention has the following advantages and positive effects:
[0021] (1) When some improved transformer models are used for fine-grained recognition, if the number of patches changes when these models extract features, the existing mainstream fine-grained classification scheme will no longer be applicable. Therefore, the present invention sets a feature selection module as a post-processing module to solve this problem, so that when different transformers are used as backbone networks for feature extraction, they can all be further used for fine-grained biological image classification after feature selection in this post-processing module. Experiments have shown that the present invention achieves fine-grained biological image classification with high classification accuracy.
[0022] (2) The method of the present invention is no longer limited to the feature extraction network. The backbone network for implementing feature extraction can be any Transformer model, such as the swin-transformer network. There is no need to calculate the attention score of each patch to select a more expressive patch. The appropriate Transformer-based model can be selected according to the situation to improve the feature extraction efficiency. The feature selection module generates a weight for each feature to represent the importance of the corresponding patch and selects a certain number of features, so that different Transformer-based backbone networks can be applied.
[0023] (3) The method of the present invention also uses a mapping network and metric learning to optimize the model. Specifically, the mapping network is a learnable nonlinear transformation network. Through this network, the features extracted by the backbone can be mapped to a higher dimensional space through a nonlinear mapping, so as to better discover the hidden information between the features; metric learning is used for optimization in the loss function, and the cosine distance is used to calculate the distance between different categories as the loss. During the model updating process, the distance between the same category is ensured to be as small as possible, and the distance between different categories is as large as possible, thereby improving the accuracy of feature selection and the accuracy of fine-grained biological image classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a flow chart of the fine-grained biological image classification method of the present invention in the training stage;
[0025] Figure 2 It is a schematic diagram of the implementation of the feature selection module in the method of the present invention;
[0026] Figure 3 is the feature F selected in the method of the present invention choose Schematic diagram of mapping and calculating distance loss;
[0027] Figure 4 It is a schematic diagram of the processing flow of the fine-grained biological image classification method of the present invention in the reasoning stage. DETAILED DESCRIPTION
[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0029] In fine-grained classification images, patches generated using backbones of different transformers cannot be input into the same fine-grained biological image classification detection network, which is not conducive to fine-grained classification. Therefore, the present invention proposes a fine-grained biological image classification method based on a Transformer model.
[0030] The fine-grained biological image classification method based on the Transformer model of the present invention uses a classification model including a feature extraction module, a feature selection module and a prediction module. The feature extraction module uses the Swin-Transformer model to extract features from the image; the feature selection module is used to calculate the weight coefficient of the patch and perform feature selection; the prediction module performs fine-grained biological image classification according to the selected features. The present invention pre-trains the classification model composed of the above three modules using training samples, and then inputs the collected biological images into the trained classification model for biological classification. The training and implementation of the three modules will be further described in the following method.
[0031] The Transformer model-based fine-grained biological image classification method of the present invention uses a classification model to perform feature extraction, feature selection and biological type prediction, and is overall divided into a training phase and a testing phase.
[0032] like Figure 1 As shown, the fine-grained biological image classification method of the present invention performs the following steps in the training phase:
[0033] S100, input biological images. Generate training samples based on historical biological images. The training samples include input images and labels. The sample labels are fine-grained different biological categories. For example, it is necessary to specifically distinguish the breed of a cat or a dog.
[0034] S101, extracting features from an input image using a feature extraction module.
[0035] The feature extraction module uses a Transformer-based network. The embodiment of the present invention directly uses the Swin-Transformer model to extract features from the input image. After the original image is input into the feature extraction module, the patch is generated after Swin-Transformer network encoding.
[0036] S102, performing feature selection using a feature selection module.
[0037] The patch generated by S101 encoding is used as a high-dimensional representation of the feature. The present invention inputs all patches into the feature selection module to generate weight coefficients, and then performs feature selection based on the generated weight coefficients.
[0038] like Figure 2 As shown in Figure 1, the feature selection module first uses a two-layer fully connected network linear to perform nonlinear mapping on the patch, and then generates weights through the softmax function. According to the generated weights, a pre-set number of patches are selected as new image features, marked as F. choose .
[0039] Select the top n features as F choose ,as follows:
[0040] F choose =max n (softmax(Linear(Linear(patch input ))))
[0041] Among them, patch input Represents the input image features.
[0042] S103, the image feature F output by the feature selection module chooseAs the input of the prediction module, the fine-grained biological classification is predicted and the classification loss is calculated. In the embodiment of the present invention, the prediction module uses a fully connected network and uses the cross entropy loss to calculate the classification loss L cls .
[0043] S104, in the training phase, uses the projection network PN to F choose The mapping is done and the distance is calculated as the distance loss.
[0044] like Figure 3 As shown, in the embodiment of the present invention, a fully connected network linear is used to choose Perform nonlinear mapping and then calculate the cosine distance as the distance loss L distance . Assume that during training, k training samples are input in the same batch. After the feature selection module, each training sample obtains an image feature containing n patches. Assume that patch is a 1*m vector, and a 1*nm feature vector can be generated for each training sample. Assume that the feature vectors generated for two training samples are [x1,x2,…x nm ] and [y1,y2,…y nm ], the cosine distance D between two training samples is calculated as follows.
[0045]
[0046] For the same batch of k training samples, the cosine distance D between different samples i and j is calculated respectively. ij Then sum it up to get the distance loss L of the current batch of training samples distance .
[0047] For training samples belonging to the same class, the smaller the cosine distance, the closer they are, and the higher the probability of being classified into the same class. For training samples belonging to different classes, the larger the cosine distance, the less likely they are to be classified into the same class. Therefore, during training, if two training samples belong to different classes, the calculated cosine distance can be negative, so as to achieve the purpose of making the distance loss smaller during training, indicating a more accurate classification.
[0048] S105, based on the distance loss L distance and the cross entropy loss L cls Optimize model parameters in feature extraction module, feature selection module and prediction module.
[0049] After the training is completed, the embodiment of the present invention is in the reasoning stage, such as Figure 4 As shown, perform the following steps:
[0050] S200, input image data;
[0051] S201, use the Swin-Transformer model to extract features from the input image, mark the extracted features as F, and the image features F are expressed in the form of patches.
[0052] S202, using the feature extraction module to select feature blocks and output the selected image features F inference_choose ;
[0053] S203, feature F choose Input the prediction module and get the prediction result P inference_predict .
[0054] The fine-grained biological image classification method based on the Transformer model provided by the present invention is experimentally verified below. The experimental data uses the CUB-200-2011 dataset (CUB) and the Stanford Dogs dataset (Stanford Dogs). The CUB dataset contains 200 bird categories, with a total of 11,788 images. The Dogs dataset contains 120 dog categories, with a total of 20,580 images. During the experiment, the dataset is divided as shown in Table 1.
[0055] Table 1 Experimental dataset
[0056] Dataset Number of training set images Number of validation set images Total quantity CUB-200-2011 8841 2947 11788 Standford Dog 15435 5154 20580
[0057] The embodiment of the present invention adopts four-fold cross validation to train the fine-grained biological image classification model of the present invention on the training set and verify the model performance on the validation set. The classification accuracy ACC of the method of the present invention on the two data sets is shown in Table 2.
[0058] Table 2 Experimental results
[0059] CUB Dataset Dogs Dataset Method of the present invention 0.88 0.88
[0060] As shown in Table 2, after four-fold verification, the average accuracy of fine-grained biological classification using the method of the present invention on two data sets is 0.88.
[0061] The above experiments prove that the present invention realizes a fine-grained biological image classification method based on the Transformer model, and the classification accuracy is relatively high.
[0062] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.
Claims
1. A fine-grained biological image classification method based on a Transformer model, characterized in that: include: Step 1: Generate training samples using historical biological images, where the labels of the training samples are fine-grained biological categories; Step 2, establish a classification model, the classification module includes a feature extraction module, a feature selection module and a prediction module; The feature extraction module is based on the Transformer network, uses the Swin-Transformer model to extract features from the input image, and outputs image features represented by feature blocks. The feature selection module first uses a two-layer fully connected network to perform nonlinear mapping on the feature block patch, and then generates weights through the softmax function. According to the generated weights, a preset number of feature block patches are selected as new image features F. choose ; Select the top n features as F choose ,as follows: F choose =max n (softmax(Linear(Linear(patch input )))) Among them, patch input Represents the input image features; The prediction module is based on the image feature F choose Predict fine-grained biological taxonomy; Step 3, using the training samples to train the classification model; During training, the projection network is used to choose Map and calculate the cosine distance between samples after mapping as the distance loss L distance , use cross entropy loss to calculate the classification loss L cls , based on L distance and L cls Optimize classification model parameters; The method for calculating the cosine distance between two training samples is as follows: Assume that the feature selection module selects n feature patches for each training sample, uses the projection network to map the feature patches, and stretches the mapped n feature patches into one row vector; Then calculate the cosine distance of the row vectors corresponding to the two training samples; if the classification labels of the two training samples are inconsistent, the calculated cosine distance is negative; Step 4: Use the trained classification model to perform fine-grained classification of biological images.
2. The method according to claim 1, characterized in that In step 3, for the same batch of training samples, the cosine distances between different training samples are calculated and then summed to obtain the distance loss of the current batch of training samples.