Low-resolution image classification method and system based on cnn and transformer

Abstract

The application relates to a low-resolution image classification method based on CNN and transformer, which comprises the following steps: step S1: constructing an image data set and classifying images in the image data set, and dividing the images into a training set and a verification set; step S2: pre-processing the images in the training set and the verification set; step S3: based on CNN and transformer, constructing a double-branch parallel network, extracting features of low-resolution images through a CNN network branch and a transformer network branch, effectively fusing features of each layer of the two branches through an attention feature fusion network, and mining semantic information of the fused features through a multi-channel attention network; step S4: training the double-branch parallel network according to the training set and the verification set, and constraining the double-branch parallel network through cross-entropy loss, so as to obtain an image classification network; and step S5: inputting a pre-processed image to be measured into the image classification network, so as to obtain a classification result. The application can effectively improve the classification performance of low-resolution images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image classification in computer vision, and particularly relates to a low-resolution image classification method and system based on CNN and transformer. BACKGROUND

[0002] The task of image classification is to learn and judge whether a certain specific target content is contained in an image, and to classify according to the content information. Image classification is the most basic computer vision task, which has been widely used in human-computer interaction, biomedicine, aerospace and public security and justice fields. Although great progress has been made in image classification research, most of these researches are carried out in high-resolution images. However, in real scenes, low-resolution images can only be obtained in most cases, so the research on low-resolution image classification is very important. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a low-resolution image classification method based on CNN and transformer, which can effectively improve the classification performance of low-resolution images.

[0004] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0005] A low-resolution image classification method based on CNN and transformer includes the following steps:

[0006] Step S1: constructing an image dataset and labeling the images in the dataset by categories, and dividing the image dataset into a training set and a validation set according to a certain proportion;

[0007] Step S2: preprocessing the images in the training set and the validation set;

[0008] Step S3: based on CNN and transformer, constructing a double-branch parallel network, extracting features of low-resolution images through the CNN network branch and the transformer network branch, effectively fusing the features of each layer of the two branches through an attention feature fusion network, and mining semantic information of the fused features through a multi-channel attention network;

[0009] Step S4: training the double-branch parallel network according to the training set and the validation set, and constraining it by using cross-entropy loss, to obtain an image classification network;

[0010] Step S5: inputting the preprocessed image to be tested into the image classification network to obtain a classification result.

[0011] Further, the preprocessing includes size adjustment and data enhancement, the size adjustment is to convert the input image sample into a preset size, and the data enhancement mode adopts random horizontal flip, random vertical flip and random rotation.

[0012] Further, the transformer network branch is composed of four stages, each stage is stacked by two transformer encoders, and the transformer network branch outputs four one-dimensional feature maps with different sizes.

[0013] Further, the one-dimensional features generated by the transformer network branch are input into the conversion network for conversion, so that the feature size is the same as that generated by the CNN network branch, and the conversion formula in the conversion network is as follows:

[0014] X i =1×1Conv(Reshape(x i ))i∈(1,2,3,4)

[0015] Where x is the one-dimensional feature output by the four stages of the transformer network branch, and X is the two-dimensional feature of four different scales generated after the one-dimensional feature is converted by the conversion network.

[0016] Further, the CNN network branch is composed of 5 layers, each layer generates two-dimensional features of different sizes, and the features extracted by the first layer network are input into the transformer branch for extraction.

[0017] Further, the attention feature fusion network fuses the features of two different semantic information extracted by the CNN branch and the transformer branch, as follows:

[0018] The features generated by the CNN network branch and the transformer network branch are added to obtain a fusion feature;

[0019] The fusion feature is processed by two branches, the first branch uses global average pooling to obtain a feature vector of the fusion feature, uses convolution to reduce the dimension, uses a relu activation function for processing, and finally uses convolution to change the feature dimension to obtain the attention weight of the first branch;

[0020] The second branch directly convolves the fusion feature to change its feature size, then uses relu for activation processing, and finally uses convolution to restore its feature size to obtain the attention weight of the second branch,

[0021] The attention weights of the two branches are added to obtain a new attention weight,

[0022] Finally, the new attention weight feature map is multiplied and added to the features generated by the CNN and transformer network to obtain the final fused features.

[0023] Further, the multi-channel attention network utilizes multiple branches, each branch uses different convolution kernels to extract features, and then uses channel attention to enhance the representation ability of the features, as follows:

[0024] x1 = 1 x 1 Conv(x)

[0025] x 11 = x1 + x1 * Sigmoid(Conv6(Relu(Conv5(MaxPool(x1)))))

[0026] x3 = 3 x 3 Conv(x)

[0027] x 33 = x3 + x 31 * Sigmoid(Conv6(Relu(Conv5(MaxPool(x3)))))

[0028] x5 = 5 x 5 Conv(x)

[0029] x 55 = x5 + x5 * Sigmoid(Conv6(Relu(Conv5(MaxPool(x5)))))

[0030] x output = x 11 + x 33 + x 55

[0031] wherein Conv5 is an empty convolution for dimension reduction, Conv6 is an empty convolution for dimension increase, and x output is the feature output by the attention network.

[0032] Further, the cross-entropy loss formula is as follows:

[0033]

[0034] wherein q i represents a soft label, B represents the number of samples in a batch, N represents the total number of categories and the length of the prediction vector, and p i represents the prediction value of the sample image of the teacher network.

[0035] A low-resolution image classification system based on CNN and transformer comprises a processor, a memory and a computer program stored on the memory, and when the processor executes the computer program, the low-resolution image classification method based on CNN and transformer is specifically implemented.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] 1、The present application ingeniously controls the depth of the network through the design of the double-branch network architecture, thereby reducing the loss of image feature information, and simultaneously utilizes the attention feature fusion network to effectively fuse the advantages of CNN and transformer together, so that the features extracted by the network are more discriminative, and finally the attention network is utilized to fully mine the semantic information in the low-resolution image.

[0038] 2、The present application can effectively reduce the influence of noise data on the final classification result when facing low-resolution image samples with complex background noise, and has good classification effect. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a method flowchart of the present application;

[0040] Figure 2 is a model framework diagram in an embodiment of the present application;

[0041] Figure 3 is a transformer branch structure diagram in an embodiment of the present application;

[0042] Figure 4 is a transformer encoder structure diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application will be further described below in combination with the drawings and embodiments.

[0044] Please refer to Figure 1 , the present application provides a low-resolution image classification method based on CNN and transformer, and the flowchart is as shown in Figure 1 , specifically comprising:

[0045] Step S1: constructing an image dataset and classifying the images therein, and simultaneously dividing the image dataset into a training set and a validation set according to a certain proportion;

[0046] Step S2: preprocessing the training set and validation set images;

[0047] Step S3: Based on CNN and transformer, a double-branch parallel network is constructed, the low-resolution image is feature-extracted through the CNN network branch and the transformer network branch, the features of each layer of the two branches are effectively fused through the attention feature fusion network, and the fused features are subjected to semantic information mining through a multi-channel attention network;

[0048] Step S4: The double-branch parallel network is trained according to the training set and the validation set, and is constrained by cross-entropy loss, to obtain an image classification network.

[0049] Step S5: The preprocessed image to be measured is input into the image classification network to obtain a classification result.

[0050] In this embodiment, the image data set is preprocessed by using two methods of size adjustment and data enhancement. The size adjustment is to convert the input image sample into a size of 224x224, and the data enhancement is to use random horizontal flipping, random vertical flipping and random rotation.

[0051] In this embodiment, the CNN branch is composed of 5 layers, each layer generates two-dimensional features of different sizes, and the features extracted by the first layer network are input into the transformer branch for extraction. The transformer is composed of four stages, each stage generates one-dimensional features of different sizes.

[0052] The transformer network branch is composed of four stages, each stage is stacked by two transformer encoders, and the transformer network branch outputs four one-dimensional feature maps of different sizes.

[0053] The one-dimensional features generated by the transformer network branch are input into the conversion network for conversion, so as to make the feature sizes the same as those generated by the CNN network branch. The conversion formula in the conversion network is as follows:

[0054] X i =1×1Conv(Reshape(x i ))i∈(1,2,3,4)

[0055] Where x is the one-dimensional feature output by the four stages of the transformer network branch, and X is the two-dimensional feature of four different scales generated by the one-dimensional feature after the conversion network.

[0056] In this embodiment, the specific process of realizing attention feature fusion is as follows:

[0057] The features generated by the CNN and transformer networks are summed to obtain a fused feature. This fused feature is then processed in two branches. The first branch uses global average pooling to obtain the feature vector of the fused feature, while convolution is used to reduce the dimensionality. ReLU activation is then applied, and finally, convolution is used to change the feature dimension, resulting in the attention weights for the first branch.

[0058] The second branch directly convolves the fused features to change their feature size, then uses ReLU for activation, and finally uses convolution to restore their feature size, thus obtaining the attention weights for the second branch.

[0059] The attention weights of these two branches are added together to obtain a new attention weight.

[0060] Finally, the new attention weight feature map is multiplied and added with the features generated by the CNN and transformer networks to obtain the final fused features.

[0061] output1 = X i +Y i

[0062] output2=Conv2(Relu(Conv1(GAP(output1))))

[0063] output3=Conv4(Relu(Conv3(output1)))

[0064] output4 = output2 + output3

[0065] output = X i *sigmoid(output4) + Yi*sigmoid(output4) where output1 represents the initial fused features, and output2 represents the attention features in the fusion network.

[0066] The first branch generates the attention map, output4 is the attention map generated by the second-lowest branch of the attention feature fusion network, output4 is the total attention feature map, and output is the final fused feature, X. i Y is the two-dimensional feature generated by the transformation network from the one-dimensional feature produced by the transformer branch. i Two-dimensional features generated by a CNN branch.

[0067] In this embodiment, the multi-channel attention network is used to fully extract semantic information from low-resolution images, specifically as follows:

[0068] The multi-channel attention network utilizes multiple branches, each branch uses different convolution kernels to extract features, and then uses channel attention to enhance the representation ability of the features, so as to fully mine the semantic information in the low-resolution image through this series of processing, so that the features finally sent to the classifier are more discriminative, and the formula is as follows:

[0069] x1=1x1Conv(x)

[0070] x 11 = x1+x1*Sigmoid(Conv6(Relu(Conv5(MaxPool(x1)))))

[0071] x3=3x3Conv(x)

[0072] x 33 = x3+x 31 *Sigmoid(Conv6(Relu(Conv5(MaxPool(x3)))))

[0073] x5=5x5Conv(x)

[0074] x 55 = x5+x5*Sigmoid(Conv6(Relu(Conv5(MaxPool(x5)))))

[0075] x output = x 11 +x 33 +x 55

[0076] Wherein, Conv5 is a hollow convolution, used for dimension reduction. Conv6 is a hollow convolution, used for dimension increase. x output is the feature output by the attention network.

[0077] The loss used in the present example is the cross-entropy loss function to represent the error between the output of the network model and the true label, so as to improve the performance of the network model, wherein q i represents a soft label, B represents the number of samples in a batch, N represents the total number of categories and the length of the prediction vector, p i represents the prediction value of the network for the sample image.

[0078] The formula for calculating the prediction probability value of each category of the network for the sample image is as follows:

[0079]

[0080] Where p i represents the probability of the network that the current sample belongs to category i, zi represents the logit of the current sample corresponding to class i, and N is the total number of sample classes.

[0081] The soft label formula is as follows:

[0082]

[0083] where N represents the total number of classes and is also the length of the prediction vector.

[0084] According to the needs, repeat the above steps until the network model converges.

[0085] The network model is used for image classification with the preprocessed test set.

[0086] Corresponding to the above embodiment 1, the present embodiment proposes a transformer image classification system based on knowledge distillation, which comprises:

[0087] A training set and test set construction module is used to construct a training set and a test set, and to label the classes of images in the training set and the test set; the images in the training set and the test set are preprocessed;

[0088] A model training module is used to simultaneously train a plurality of student network models using the preprocessed training set, and to guide the training of the student network models using a pre-trained teacher network model during the training process, and to increase the similarity loss and the diversity loss in the target loss function, and to output a student network model with higher accuracy

[0089] A model testing module is used to test the output student network model using the preprocessed test set.

[0090] The above is only the preferred embodiment of the present application, and any equivalent changes and modifications made within the scope of the present application shall be covered by the present application.

Claims

1. A low-resolution image classification method based on CNN and Transformer, characterized in that, Includes the following steps: Step S1: Construct an image dataset and label the images in it with categories. At the same time, divide the image dataset into a training set and a validation set according to a certain ratio. Step S2: Preprocess the training and validation set images; Step S3: Based on CNN and transformer, construct a dual-branch parallel network. Use the CNN network branch and the transformer network branch to extract features from low-resolution images. Use an attention feature fusion network to effectively fuse the features of each layer of the two branches. Use a multi-channel attention network to mine semantic information from the fused features. Step S4: Train the dual-branch parallel network using the training and validation sets, and constrain it using cross-entropy loss to obtain the image classification network; Step S5: Input the preprocessed image to be tested into the image classification network to obtain the classification result; The attention feature fusion network fuses features with two different semantic information extracted from the CNN branch and the transformer branch, as follows: The features generated by the CNN network branch and the transformer network branch are added together to obtain a fused feature; The fused features are processed in two branches. The first branch uses global average pooling to obtain the feature vector of the fused features, while using convolution to reduce the dimensionality, then using the ReLU activation function for further processing, and finally using convolution to change the feature dimension to obtain the attention weights of the first branch. The second branch directly convolves the fused features to change their feature size, then uses ReLU for activation, and finally uses convolution to restore their feature size, thus obtaining the attention weights for the second branch. The attention weights of these two branches are added together to obtain a new attention weight. Finally, the new attention weight feature map is multiplied and added with the features generated by the CNN and transformer networks to obtain the final fused features.

2. The low-resolution image classification method based on CNN and Transformer according to claim 1, characterized in that, The preprocessing includes resizing and data augmentation. The resizing involves converting the input image sample to a preset size. The data augmentation method employs random horizontal flipping, random vertical flipping, and random rotation.

3. The low-resolution image classification method based on CNN and Transformer according to claim 1, characterized in that, The transformer network branch consists of four stages, each consisting of two stacked transformer encoders. The transformer network branch outputs four different sizes of one-dimensional feature maps.

4. The low-resolution image classification method based on CNN and Transformer according to claim 3, characterized in that, The one-dimensional features generated by the transformer network branch are input into the transformation network for transformation, so that they have the same size as the features generated by the CNN network branch. The transformation formula in the transformation network is as follows: X i =1×1Conv(Reshape(x i ))i∈(1,2,3,4) Where x is the one-dimensional feature output by the four stages of the transformer network branch, and X is the four two-dimensional features of different scales generated after the one-dimensional feature is transformed by the transformer network.

5. The low-resolution image classification method based on CNN and Transformer according to claim 1, characterized in that, The CNN network branch consists of 5 layers, each of which generates two-dimensional features of different sizes. The features extracted by the first layer are input into the transformer branch for further extraction.

6. The low-resolution image classification method based on CNN and Transformer according to claim 1, characterized in that, The multi-channel attention network utilizes multiple branches, each employing a different convolutional kernel to extract features. Then, channel attention is used to enhance the representational power of these features, as shown in the following formula: x1 = 1 × 1 Conv(x) x 11 =x1+x1*Sigmoid(Conv6(Relu(Conv5(MaxPool(x1))))) x³ = 3 × 3Conv(x) x 33 =3+x 31 *Sigmoid(Conv6(Relu(Conv5(MaxPool(x3))))) x5 = 5 × 5Conv(x) x 55 =x5+x5*Sigmoid(Conv6(Relu(Conv5(MaxPool(x5))))) x output =x 11 +x 33 +x 55 Where Conv5 is dilated convolution, used for dimensionality reduction, and Conv6 is dilated convolution, used for dimensionality increase, x output These are the features output by the attention network.

7. The low-resolution image classification method based on CNN and Transformer according to claim 1, characterized in that, The cross-entropy loss formula is as follows: Where, q i This represents the soft label, B represents the number of samples in a batch, N represents the total number of categories and is also the length of the prediction vector, and p i This represents the predicted value of the sample images from the teacher network.

8. A low-resolution image classification system based on CNN and Transformer, characterized in that, It includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the low-resolution image classification method based on CNN and Transformer as described in any one of claims 1-7.

Images