Image hierarchical classification method based on visual transformer

Abstract

The application discloses an image hierarchical classification method based on a visual transformer, belongs to the technical field of image hierarchical classification, and solves the technical problems of low precision and complicated processing procedures of existing image hierarchical classification methods, and the method comprises the following steps: step S1, image data is acquired, and preprocessing is performed to obtain a public data set with hierarchical labels; step S2, a visual transformer-based image hierarchical classification model is constructed; step S3, the image hierarchical classification model is trained through the public data set; and step S4, the trained image hierarchical classification model is used for image hierarchical classification of to-be-classified images. Compared with the prior art, the visual transformer with a strong global information interaction mechanism is used as a backbone network to extract image features in the application, and meanwhile, the hierarchical information of the image is fully utilized, so that the hierarchical classification precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image hierarchical classification, more particularly, it relates to an image hierarchical classification method based on visual transformer. BACKGROUND

[0002] With the rapid development of modern Internet information technology, various media data such as images have also increased explosively, and how to intelligently and efficiently manage and classify these complex structured images is a challenging task. The traditional classification method only uses a single label of the picture and performs corresponding single-label classification, however, the image library data in the real world often has various categories, and even after classification, the number of categories is still huge, and each image may contain multiple labels, and there is often a certain hierarchical structure between these labels. If the inherent hierarchical information of the image itself can be used for hierarchical classification, it can also make the image database more simple and efficient hierarchical classification management, and also facilitate subsequent administrators or users to find images from the image library according to the hierarchical relationship of the images, greatly improving the retrieval efficiency and reducing the retrieval difficulty.

[0003] Patent CN110390350B provides a hierarchical classification method based on a bilinear structure, which combines the prior knowledge of the hierarchical structure with a bilinear network containing five convolution modules to realize hierarchical classification prediction. The technical solution is based on a convolutional neural network as the backbone network, and the convolutional neural network cannot capture global features well and lacks the ability to learn long-distance information, so the prediction accuracy of the method is low. Patent application file CN113449814A provides a picture hierarchical classification method and system, which pre-trains a first-level picture classification recognition model, performs first-level picture classification, and when there is a model type corresponding to the recognition result output by the first-level picture classification recognition model in the second-level picture classification recognition model, the picture being identified is transferred to the corresponding second-level classifier for re-identification to obtain the final hierarchical classification. The technical solution of the method needs to train two levels of models, and the processing procedure is complicated. SUMMARY

[0004] The technical problem to be solved by the present application is to solve the above-mentioned deficiencies of the prior art, and the purpose of the present application is to provide an image hierarchical classification method based on visual transformer.

[0005] The technical solution of the present application is: an image hierarchical classification method based on visual transformer, comprising the following steps:

[0006] Step S1. Obtain image data and pre-process to obtain a public data set with hierarchical labels;

[0007] Step S2. Constructing an image hierarchical classification model based on a Vision Transformer;

[0008] Step S3. Training the image hierarchical classification model through the public dataset;

[0009] Step S4. Using the trained image hierarchical classification model to perform image hierarchical classification on images to be classified.

[0010] As a further improvement, in step S1, the public dataset is obtained by adding N-1 labels with hierarchical structure to the original labels of each picture in the original CIFAR-100 dataset or CIFAR-10 dataset.

[0011] Further, in step S2, among all the tokens output by the Transformer Encoder,

[0012] First, average pooling is performed on the image block tokens;

[0013] Then, the concatenated features are respectively passed through N different fully connected layers to obtain image features of different levels;

[0014] Then, the N image features of different levels are concatenated to fuse the different levels of features implied by the image to obtain aggregated features;

[0015] Finally, the aggregated features are input into N different classifiers for hierarchical classification, and the classifiers are MLP modules.

[0016] Further, in step S3, take num consecutive samples in the public dataset as a group of input to the image hierarchical classification model, a total of all_group groups of samples, the maximum number of iterations is E, the patch block size is patch_size, the embedding dimension size is embed_dim, and the initial iteration number is t = 1; the training includes the following steps:

[0017] Step S31. Input the first group of data;

[0018] Step S32. Learn the features of a group of pictures through the Transformer Encoder module;

[0019] Step S33. Perform average pooling on the image block tokens output by the Transformer Encoder, then concatenate the class tokens, then pass the concatenated features through N different fully connected layers to obtain image features of different levels, and finally concatenate the N image features of different levels to fuse the different levels of features implied by the image to obtain aggregated features;

[0020] Step S34. Use N classifiers to perform hierarchical classification of the aggregated features;

[0021] Step S35. Calculate the cross-entropy of each classifier result and the true label respectively, and add them together to obtain the total cross-entropy loss. Minimize the cross-entropy loss to update the hyperparameters of the image hierarchical classification model;

[0022] Step S36. Input the next set of data, and execute steps S32 to S35 until all data in the all_group group has been processed;

[0023] Step S37. Increment the iteration count t by 1, and execute step S31 to continue iterating until the maximum iteration count E is reached.

[0024] Furthermore, in step S33, a token matrix is ​​output after the data passes through the Transformer Encoder. Where z0 represents the category term, and the other terms are image patch terms.

[0025] First, analyze the image block symbol X. w Perform 2×2 convolutional average pooling, then concatenate with the category token z0. Pass the concatenated features through N different fully connected layers to obtain image features F at different levels. i ;

[0026] Image features F at different levels i The formula is:

[0027] F i =FC i (AvgPool(X w )+z0),i∈{1,2,…,N}, where FC is a fully connected layer and AvgPool is an average pooling layer; the aggregated feature is obtained as F=[F1,F2,…,F N ].

[0028] Further, in step S35, the classification result of each level of the image is used to calculate the cross-entropy with the true label, and the prediction loss of the l-th level is calculated. Where y lj This represents the output prediction for the j-th class in the l-th level. Let k represent the true label of class j in level l, and k represent the number of classes to be classified in level l; then the total loss l is obtained. all This is the sum of the predicted losses at each level.

[0029] Beneficial effects

[0030] Compared with the prior art, the present application has the advantages that:

[0031] Compared with the prior art, the visual transformer with strong global information interaction mechanism is used as the backbone network to extract the features of the image in the present application, and the hierarchical information of the image is fully utilized, thereby improving the hierarchical classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The algorithm flowchart of the present application;

[0033] Figure 2 The algorithm model diagram of the present application;

[0034] Figure 3 The hierarchical label diagram of the CIFAR-10 dataset;

[0035] Figure 4 The hierarchical label diagram of the CIFAR-100 dataset. DETAILED DESCRIPTION

[0036] The present application will be further described below in combination with the specific embodiments in the drawings.

[0037] Referring to Figures 1-2 An image hierarchical classification method based on a visual transformer, comprising the following steps:

[0038] Step S1. Obtain image data and perform preprocessing to obtain a public dataset with hierarchical labels;

[0039] Step S2. Construct an image hierarchical classification model based on a Vision Transformer;

[0040] Step S3. Train the image hierarchical classification model through the public dataset;

[0041] Step S4. Use the trained image hierarchical classification model to perform image hierarchical classification on the image to be classified.

[0042] In step S1, the public CIFAR-100 dataset or the CIFAR-10 dataset is adopted. The original dataset has only one label for each picture. When N-level classification is required, N-1 labels with hierarchical structure to the original label of the image are added to each picture of the original dataset to obtain a public dataset. Consecutive num samples are taken as a group of data as the input of the model.

[0043] In step S2, the image-level classification model based on the visual transformer is improved based on the VisionTransformer. Among all the token outputs of the Transformer Encoder,

[0044] First, average pooling is performed on the image block token;

[0045] Then, the concatenated features are concatenated with the class token, and the concatenated features are respectively passed through N different fully connected layers to obtain image features of different levels;

[0046] Then, the N image features of different levels are concatenated to fuse the different levels of features implied by the image to obtain aggregated features;

[0047] Finally, the aggregated features are input into N different classifiers for hierarchical classification, and the classifier is an MLP module.

[0048] In step S3, the public dataset is a CIFAR-100 dataset or a CIFAR-10 dataset containing N hierarchical information labels, num consecutive samples in the public dataset are taken as a group of input image-level classification model, there are all_group samples in total, the maximum number of iterations is E, the patch block size is patch_size, the embedding dimension size is embed_dim, and the initial iteration number is t = 1. The training includes the following steps:

[0049] Step S31. Input the first group of data;

[0050] Step S32. Learn the features of a group of pictures through the Transformer Encoder module;

[0051] Step S33. Perform average pooling on the image block token output by the Transformer Encoder, then concatenate the class token, then pass the concatenated features through N different fully connected layers to obtain image features of different levels, and finally concatenate the N image features of different levels to fuse the different levels of features implied by the image to obtain aggregated features;

[0052] Step S34. Hierarchical classification of aggregated features using N classifiers;

[0053] Step S35. Calculate the cross-entropy of each classifier result and the true label respectively, and add them to obtain the total cross-entropy loss, and update the hyperparameters of the image-level classification model by minimizing the cross-entropy loss;

[0054] Step S36. Input the next group of data, execute steps S32 to S35 until all_group data are processed.

[0055] Step S37. Increment the iteration count t by 1, and execute step S31 to continue iterating until the maximum iteration count E is reached.

[0056] In step S33, the token matrix is ​​output after the data passes through the Transformer Encoder. Where z0 represents the category term, and the other terms are image patch terms.

[0057] First, analyze the image block symbol X. w Perform 2×2 convolutional average pooling, then concatenate with the category token z0. Pass the concatenated features through N different fully connected layers to obtain image features F at different levels. i ;

[0058] Image features F at different levels i The formula is:

[0059] F i =FC i (AvgPool(X w )+z0),i∈{1,2,…,N}, where FC is a fully connected layer and AvgPool is an average pooling layer; the aggregated feature is obtained as F=[F1,F2,…,F N ].

[0060] In step S35, the classification result of each level of the image is used to calculate the cross-entropy with the true label, and the prediction loss of the l-th level is calculated. Where y lj This represents the output prediction for the j-th class in the l-th level. Let k represent the true label of class j in level l, and k represent the number of classes to be classified in level l; then the total loss l is obtained. all This is the sum of the predicted losses at each level.

[0061] Experiments of this invention

[0062] The experimental data used two datasets: CIFAR-10 and CIFAR-100.

[0063] The CIFAR-10 dataset consists of 32×32 RGB images in 10 classes, totaling 50,000 training images and 10,000 test images. (See below.) Figure 3 As shown, the hierarchical labels of the CIFRA-10 dataset were manually partitioned using the method proposed by Xinqi Zhu et al. in the B-CNN approach. The number of categories in levels 1, 2, and 3 were 2, 7, and 10, respectively.

[0064] based onFigure 3 We used a label tree to perform two two-level classification experiments and one three-level classification experiment on the CIFAR-10 dataset. In the first two-level classification experiment, the first level had 2 classes and the second level had 10 classes, referred to as the "2-10" case. In the second two-level classification experiment, the first level had 7 classes and the second level had 10 classes, referred to as the "7-10" case, and so on, with the three-level classification being the "2-7-10" case. The performance of this method on the hierarchical classification of the CIFAR-10 dataset is shown in Table 1.

[0065] Table 1. CIFAR-10 Hierarchical Classification Accuracy

[0066]

[0067] The CIFAR-100 dataset consists of 32×32 RGB images across 100 classes, with a total of 50,000 training and 10,000 test examples. (See below.) Figure 4 As shown, the hierarchical labels of the CIFRA-100 dataset, which were manually partitioned using the method proposed by Xinqi Zhu et al. in the B-CNN method, are also used. The first layer contains 8 classes, the second layer contains 20 classes, each of which contains 5 subclasses, and the third layer contains 100 classes.

[0068] based on Figure 4 Using a label tree, two-layer and three-layer classification experiments were conducted on the CIFAR-100 dataset. In the two-layer classification experiment, the first layer had 20 classes and the second layer had 100 classes, referred to as the "20-100" case. Similarly, the three-layer classification experiment used the "8-20-100" case. The experimental results are shown in Table 2.

[0069] Table 2 CIFAR-100 Hierarchical Classification Accuracy

[0070]

[0071] Compared with existing technologies, this invention uses a visual transformer with a strong global information interaction mechanism as the backbone network to extract image features, while making full use of the hierarchical information of the image to improve the hierarchical classification accuracy.

[0072] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. An image hierarchical classification method based on visual transformer, characterized in that, Includes the following steps: Step S1. Acquire image data and preprocess it to obtain a public dataset with hierarchical labels; Step S2. Construct an image hierarchical classification model based on Vision Transformer; Step S3. Train the image hierarchical classification model using the public dataset; Step S4. Use the trained image hierarchical classification model to perform image hierarchical classification on the image to be classified; In step S2, among all the tokens output by the Transformer Encoder, First, perform average pooling on the image block symbols; Then, it is concatenated with category symbols, and the concatenated features are passed through N different fully connected layers to obtain image features at different levels. Then, these N different levels of image features are stitched together, and the features implied by the image at different levels are fused to obtain aggregated features; Finally, the aggregated features are input into N different classifiers for hierarchical classification. The classifiers are MLP modules. In step S3, take num consecutive samples from the public dataset as a group to input the image hierarchical classification model, for a total of all_group samples, with a maximum number of iterations of E, a patch size of patch_size, an embedding dimension of embed_dim, and an initial number of iterations of t=1. The training includes the following steps: Step S31. Input the first set of data; Step S32. Learn the features of a set of images using the Transformer Encoder module; Step S33. Perform average pooling on the image block tokens output by Transformer Encoder, then concatenate them with category tokens, and then pass the concatenated features through N different fully connected layers to obtain image features at different levels. Finally, concatenate these N different levels of image features and fuse the features at different levels implied in the image to obtain aggregated features. Step S34. Use N classifiers to perform hierarchical classification of the aggregated features; Step S35. Calculate the cross-entropy of each classifier result and the true label respectively, and add them together to obtain the total cross-entropy loss. Minimize the cross-entropy loss to update the hyperparameters of the image hierarchical classification model; Step S36. Input the next set of data, and execute steps S32 to S35 until all data in the all_group group has been processed; Step S37. Increment the iteration count t by 1, and execute step S31 to continue iterating until the maximum iteration count E is reached.

2. The image hierarchical classification method based on visual transformer according to claim 1, characterized in that, In step S1, the public dataset is obtained by adding N-1 hierarchical labels to each image in the original dataset using the publicly available CIFAR-100 or CIFAR-10 dataset.

3. The image hierarchical classification method based on visual transformer according to claim 1, characterized in that, In step S33, the token matrix is ​​output after the data passes through the TransformerEncoder. Where z0 represents the category term, and the other terms are image patch terms. First, analyze the image block symbol X. w Perform 2×2 convolutional average pooling, then concatenate with the category token z0. Pass the concatenated features through N different fully connected layers to obtain image features F at different levels. i ; Image features F at different levels i The formula is: F i =FC i (AvgPool(X w )+z0),i∈{1,2,…,N}, where FC is a fully connected layer and AvgPool is an average pooling layer; the aggregated features are obtained as F=[F1,F2,…,F N ].

4. The image hierarchical classification method based on visual transformer according to claim 1, characterized in that, In step S35, the classification result of each level of the image is used to calculate the cross-entropy with the true label, and the prediction loss of the l-th level is calculated. Where y lj This represents the output prediction for the j-th class in the l-th level. Let k represent the true label of class j in level l, and k represent the number of classes to be classified in level l; then the total loss l is obtained. all This is the sum of the predicted losses at each level.

Images