A semi-supervised image classification method based on distillation network

Abstract

The present invention discloses a semi-supervised image classification method based on a distillation network, comprising: (1) preparing a data set, and marking a small number of images in the data set with category labels; (2) constructing a neural network model, the structure of which includes five convolution blocks and a projection layer; (3) performing rotational self-supervised learning on the neural network model to obtain model 1; (4) performing comparative self-supervised learning on the neural network model to obtain model 2; (5) fusing model 1 and model 2 using a mutually learned network distillation strategy to exchange the learned knowledge; (6) inputting the image to be classified into the trained model 1 or model 2 to obtain a classification result. By using the present invention, the feature quantity expressed by the network can be greatly increased, thereby improving the ability to recognize objects and improving the classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a semi-supervised image classification method based on a distillation network. Background Art

[0002] Deep neural networks have made important contributions to image classification, and convolutional neural networks can even better recognize objects than human vision. However, our pursuit of improving the classification capabilities of neural networks is uninterrupted. While paying attention to the capabilities of neural networks, we must also pay attention to the resources required to make deep neural networks both cheap and efficient.

[0003] In order to achieve high efficiency and low cost, the applicant found that the existing technology has two main problems:

[0004] First, when training a neural network for image classification, images with labeled data are required. After the image is input into the neural network, it is told the name of the object to be identified in the image. In this process, it takes a huge amount of manpower to manually label the image, which is extremely long and tedious. Therefore, semi-supervised learning can be introduced, that is, only a small number of labeled images are needed to achieve a high classification ability.

[0005] For example, a Chinese patent document with publication number CN108416370A discloses an image classification method based on semi-supervised deep learning, including: obtaining labeled training image samples and unlabeled training image samples to obtain a labeled training set; combining deep learning and semi-supervised learning to perform convolutional neural network training on the labeled training set, and establishing a unified semi-supervised deep learning and unlabeled sample category estimation model; performing image recognition and classification based on the semi-supervised deep learning and unlabeled sample category estimation model.

[0006] Second, the simplest and most effective way to improve the capabilities of neural networks is to increase the number of network layers and the number of parameters, but a large number of parameters also means that more computing power and storage space are required. Moreover, not every convolutional layer in a deep neural network can provide effective feature knowledge recognition, which means that its effective feature knowledge information accounts for a very low proportion, that is, the knowledge saturation is low.

[0007] For the first point, we need to solve how to use a small number of labels to achieve higher performance of the neural network. In traditional supervised learning, only the loss function between the predicted label and the true label of the image is used, ignoring the inherent information of the data set itself.

[0008] For the second point, mutual distillation can be used for fusion. The distillation network can transform the information carried by negative labels into feature knowledge for training. However, in the mutual distillation of knowledge of multiple networks, it is difficult to ensure the complementarity of network knowledge. If two neural networks can only recognize the same feature part of an object, then it is meaningless to fuse the two networks.

[0009] Therefore, it is urgent to design a new image classification method to solve the above problems existing in the existing technology. Summary of the invention

[0010] The present invention provides a semi-supervised image classification method based on a distillation network, which can greatly increase the feature quantity expressed by the network, thereby improving the ability to recognize objects and improving classification accuracy.

[0011] A semi-supervised image classification method based on a distillation network comprises the following steps:

[0012] (1) Prepare the dataset, preprocess the dataset, label a small number of images in the dataset with category labels, and use the remaining images as unlabeled training data;

[0013] (2) constructing a neural network model for classification, wherein the neural network model adopts an improved vgg11 network, and the structure includes five convolution blocks and a projection layer, which are: a first convolution layer, a first maximum pooling layer, a first relu layer, a second convolution layer, a second maximum pooling layer, a second relu layer, a third convolution layer, a third maximum pooling layer, a third relu layer, a fourth convolution layer, a fourth maximum pooling layer, a fourth relu layer, a fifth convolution layer, a fifth maximum pooling layer, a fifth relu layer and a projection layer;

[0014] Among them, the convolution kernel sizes of the first to fifth convolutional layers are all 3×3, and the number is set to 1, 1, 2, 2, and 2 respectively; the input channels of the first to fifth relu layers are 3, 16, 32, 64, and 128 respectively, and the output channels are 16, 32, 64, 128, and 128 respectively; the structure of the projection layer is the first fully connected layer, batchNorm1d layer, the sixth relu layer, and the second fully connected layer. The number of input nodes and output nodes of the first fully connected layer is 6272, and the number of output nodes is 100. The number of input nodes and output nodes of the second fully connected layer is 100 and 4;

[0015] (3) For images in the dataset that do not have category labels, they are rotated to different angles and automatically labeled with pseudo labels of corresponding angles. The images with pseudo labels are used to perform self-supervised learning on the initial neural network model.

[0016] Then, the projection layer of the neural network model after self-supervised learning is replaced with a classifier, and the adjusted neural network model is trained with a small number of images with category labels in the dataset until the loss function converges to obtain model 1;

[0017] (4) Each image in the data set is preprocessed twice, and each image obtains two similar images, img1 and img2; the projection layer of the initial neural network model is modified as follows: the number of input nodes of the first fully connected layer is 6272, the number of output nodes is 2048, and the number of input nodes of the second fully connected layer is 2048, and the number of output nodes is 128;

[0018] Use similar images img1 and img2 to perform self-supervised training on the neural network model. During training, the feature vector output by the projection layer is used to calculate the similarity of the two images and the similarity loss function.

[0019] Then, the projection layer of the neural network model after self-supervised training is replaced with a classifier, and the adjusted neural network model is trained with a small number of images with category labels in the dataset until the loss function converges to obtain model 2;

[0020] (5) Model 1 and Model 2 are fused using a mutually learned network distillation strategy to exchange the learned knowledge. During the fusion process, images with classification labels in the dataset are used to train Model 1 and Model 2 respectively. After obtaining two classification probabilities, the total loss function is calculated until the total loss function converges to obtain the trained Model 1 and Model 2.

[0021] (6) For the image to be classified, input the trained model 1 or model 2 to obtain the classification result of the image.

[0022] Furthermore, in step (1), there are two data sets, each of which contains images of different categories.

[0023] The pre-processing is specifically as follows:

[0024] Each image is randomly cropped and then scaled to 224×224; each image is horizontally flipped with a probability of 0.5; the brightness, contrast, saturation and hue of each image are adjusted; and each image is converted to grayscale with a probability of 0.2.

[0025] In step (3), for images without category labels in the dataset, the pseudo labels of the corresponding angles are automatically annotated after rotating them at different angles:

[0026] For each image, three rotations are performed respectively, and four images rotated by 0, 90, 180 and 270 degrees are obtained together with the original image, and the corresponding annotation labels are 1, 2, 3 and 4 respectively.

[0027] In step (3) and step (4), the classifier includes three fully connected layers, wherein the input and output nodes of the three fully connected layers are (6272, 2048), (2048, 512), and (512, 10), respectively, and the first two fully connected layers are connected to a relu layer and a dropout layer.

[0028] In step (4), the similarity formula of the two images is calculated using the feature vector output by the projection layer:

[0029]

[0030] Among them, z i , z j is the feature vector output by the projection layer; τ is an adjustable temperature parameter, which is taken as 0.5 here by scaling the input and expanding the range of cosine similarity, ‖z i ‖ is the norm of the vector.

[0031] The calculation formula of the similarity loss function is as follows:

[0032]

[0033] Where n represents the number of sample images.

[0034] In step (5), the formula for calculating the total loss function is:

[0035]

[0036] In the formula, α is used to adjust the proportion of ceLoss and klLoss, where α is 0.2; ceLoss is the cross loss function, klLoss is the kl divergence, and the calculation formula is as follows:

[0037]

[0038] out1 and out2 are obtained by inputting the labeled image img into model 1 and model 2 respectively. i ) and p2(out j ) are used to calculate the probability that the input image belongs to category c using two models respectively. The calculation formula is as follows:

[0039]

[0040] Among them, τ represents the adjustable temperature parameter, τ is taken as 20 here, and C represents the total number of categories.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention performs different self-supervised pre-training tasks on the same network model through self-supervised learning. Different pre-training tasks can mine different feature knowledge of the image, thereby achieving multi-view expression of the same object; at the same time, the networks of these different views are further fused, so that the feature quantity expressed by the network can be greatly increased, thereby improving the recognition ability of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flow chart of a semi-supervised image classification method based on a distillation network in an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of rotational self-supervised learning in an embodiment of the present invention;

[0045] Figure 3 This is a diagram of two trained neural networks learning from each other in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0047] like Figure 1 As shown, a semi-supervised image classification method based on a distillation network includes the following steps:

[0048] 1) Data processing

[0049] (1.1) Prepare datasets: Two datasets are used. The first dataset is Cifar-10, which contains 10 categories of images, 6,000 images per category, and each image is 32×32 in size, of which 5,000 images are designated as training samples and 1,000 images are designated as test samples. The second dataset is caltech-256, whose 256 image sets cover 257 categories. There are a total of 30,607 images in this dataset, with more than 80 images in each category.

[0050] (1.2) Dataset preprocessing: Make some appropriate changes to each image to adapt to model training.

[0051] ① Use torchvision.transforms.RandomResizedCrop(224) to randomly crop the given image, and then scale the cropped image to 224×224;

[0052] ②Flip each image horizontally with a probability of 0.5;

[0053] ③ Use torchvision.transforms.ColorJitter(brightness=0.4,contrast=0.4,saturation=0.4,hue=0.1) to adjust the brightness, contrast, saturation and hue of each image;

[0054] ④Convert each image to grayscale with a probability of 0.2.

[0055] (1.3) Dataset division: When training the self-supervised model, we randomly select 300 labeled images from each class of the Cifar-10 training set as labeled data, a total of 3,000 images, and the remaining 47,000 training images as unlabeled training data. We use 10,000 test images to evaluate the performance of the model. In the caltech-256 dataset, for each class, we take 60 images as unlabeled data, randomly select 20 images from them as labeled training data, and select 20 images from the remaining images as test images to evaluate the performance of the model.

[0056] 2) Rotational self-supervised learning

[0057] (2.1) Data loader: automatically annotate images with label information, perform three rotations on each image, and add the original image to obtain four images rotated by 0, 90, 180, and 270 degrees, as shown in the figure. Figure 2 As shown, they are labeled 1, 2, 3 and 4 respectively.

[0058] (2.2) Neural network model: Use a network similar to vgg11, as follows, CovBlock1: 3*3 convolution × 1 + maximum pooling + relu (input channel: 3, output channel: 16) CovBlock2: 3*3 convolution × 1 + maximum pooling + relu (input channel: 16, output channel: 32) CovBlock3: 3*3 convolution × 2 + maximum pooling + relu (input channel: 32, output channel: 64) CovBlock4: 3*3 convolution × 2 + maximum pooling + relu (input channel: 64, output channel: 128) CovBlock5: 3*3 convolution × 2 + maximum pooling + relu (input channel: 128, output channel: 128) Projection layer g: fully connected layer (6272, 100)

[0059] batchNorm1d+relu

[0060] Fully connected layer (100, 4)

[0061] (2.3) Self-supervised learning: supervised learning is performed on images with pseudo labels. The loss function is cross entropy. The Pytorch function F.cross_entropy(out,y) is used, where out is the output value of the neural network and y is the label corresponding to the image.

[0062] (2.4) Supervised training: Use a small number of labels to adjust the neural network model and replace the projection layer with a classifier, as follows:

[0063] FC1: fully connected layer (6272, 2048) + relu + dropout

[0064] FC2: fully connected layer (2048,512) + relu + dropout

[0065] FC3: fully connected layer (512,10)

[0066] Use cross entropy as the loss function until convergence to obtain model1.

[0067] 3) Contrastive self-supervised learning

[0068] (3.1) Data loader: Perform the above preprocessing changes twice on the same image to generate two similar images img1 and img2.

[0069] (3.2) Neural network model: Modify the projection layer g in step (2.2) as follows:

[0070] Projection layer g': fully connected layer (6272, 2048)

[0071] batchNorm1d+relu

[0072] Fully connected layer (2048, 128)

[0073] (3.3) Self-supervised training: Calculate the similarity between the two images img1 and img2 to make them closer and closer. The similarity calculation formula is as follows:

[0074]

[0075] Among them, z i , z j is the feature vector output by the projection layer, τ is an adjustable temperature parameter that can scale the input and expand the range of cosine similarity, here it is taken as 0.5, ‖z i ‖ is the norm of the vector;

[0076] The loss function calculation formula is as follows:

[0077]

[0078] The resulting similarity matrices are averaged.

[0079] (3.4) Supervised training refers to the process of (2.4) to obtain model2.

[0080] 4) Neural Network Fusion

[0081] Different self-supervised pre-trained deep neural networks are fused using a mutually learning network distillation strategy to exchange the knowledge they have learned, thereby increasing the amount of knowledge in the original structure of the neural network and improving the classification accuracy.

[0082] Use the labeled image img to put into the model model1 and model2 respectively, and get out1 and out2, such as Figure 3 , p1(out i ) and p2(out j ) is calculated as follows:

[0083]

[0084] This formula adds a τ to the softmax, and τ is taken as 20 here.

[0085] The total loss function is calculated using the following formula:

[0086]

[0087] Among them, α is used to adjust the proportion of ceLoss and klLoss, where α is 0.2; ceLoss is the cross loss function, calculated using the Pytorch function F.cross_entropy; klLoss is the kl divergence, and the calculation formula is as follows:

[0088]

[0089] After the total loss function converges, the trained model1 and model2 are obtained. For the image to be classified, the trained model1 or model2 is input to obtain the classification result of the image.

[0090] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A semi-supervised image classification method based on a distillation network, characterized in that: The following steps are involved: (1) Prepare the dataset, preprocess the dataset, label a small number of images in the dataset with category labels, and use the remaining images as unlabeled training data; (2) constructing a neural network model for classification, wherein the neural network model adopts an improved vgg11 network, and the structure is: a first convolutional layer, a first maximum pooling layer, a first relu layer, a second convolutional layer, a second maximum pooling layer, a second relu layer, a third convolutional layer, a third maximum pooling layer, a third relu layer, a fourth convolutional layer, a fourth maximum pooling layer, a fourth relu layer, a fifth convolutional layer, a fifth maximum pooling layer, a fifth relu layer and a projection layer; Among them, the convolution kernel sizes of the first to fifth convolutional layers are all 3×3, and the number is set to 1, 1, 2, 2, and 2 respectively; the input channels of the first to fifth relu layers are 3, 16, 32, 64, and 128 respectively, and the output channels are 16, 32, 64, 128, and 128 respectively; the structure of the projection layer is the first fully connected layer, batchNorm1d layer, the sixth relu layer, and the second fully connected layer. The number of input nodes and output nodes of the first fully connected layer is 6272, and the number of output nodes is 100. The number of input nodes and output nodes of the second fully connected layer is 100 and 4; (3) For images in the dataset that do not have category labels, they are rotated to different angles and automatically labeled with pseudo labels of corresponding angles. The images with pseudo labels are used to perform self-supervised learning on the initial neural network model. Then, the projection layer of the neural network model after self-supervised learning is replaced with a classifier, and the adjusted neural network model is trained with a small number of images with category labels in the dataset until the loss function converges to obtain model 1; (4) Each image in the data set is preprocessed twice, and each image obtains two similar images, img1 and img2; the projection layer of the initial neural network model is modified as follows: the number of input nodes of the first fully connected layer is 6272, the number of output nodes is 2048, and the number of input nodes of the second fully connected layer is 2048, and the number of output nodes is 128; Use similar images img1 and img2 to perform self-supervised training on the neural network model. During training, the feature vector output by the projection layer is used to calculate the similarity of the two images and the similarity loss function. Then, the projection layer of the neural network model after self-supervised training is replaced with a classifier, and the adjusted neural network model is trained with a small number of images with category labels in the dataset until the loss function converges to obtain model 2; (5) Model 1 and Model 2 are fused using a mutually learned network distillation strategy to exchange the learned knowledge. During the fusion process, images with classification labels in the dataset are used to train Model 1 and Model 2 respectively. After obtaining two classification probabilities, the total loss function is calculated until the total loss function converges to obtain the trained Model 1 and Model 2. (6) For the image to be classified, input the trained model 1 or model 2 to obtain the classification result of the image.

2. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (1), there are two data sets, each of which contains images of different categories.

3. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (1), the pretreatment is specifically: Each image is randomly cropped and then scaled to 224×224; each image is horizontally flipped with a probability of 0.5; the brightness, contrast, saturation and hue of each image are adjusted; and each image is converted to grayscale with a probability of 0.

2.

4. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (3), for images without category labels in the dataset, the pseudo labels of the corresponding angles are automatically annotated after rotating them at different angles: For each image, three rotations are performed respectively, and four images rotated by 0, 90, 180 and 270 degrees are obtained together with the original image, and the corresponding annotation labels are 1, 2, 3 and 4 respectively.

5. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (3) and step (4), the classifier includes three fully connected layers, wherein the numbers of input and output nodes of the three fully connected layers are (6272, 2048), (2048, 512), and (512, 10), respectively, and the first two fully connected layers are connected to a relu layer and a dropout layer.

6. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (4), the similarity formula of the two images is calculated using the feature vector output by the projection layer: Among them, z i , z j is the feature vector output by the projection layer; τ is an adjustable temperature parameter, which is taken as 0.5 here by scaling the input and expanding the range of cosine similarity, ‖z i ‖ is the norm of the vector.

7. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (4), the calculation formula of the similarity loss function is as follows: Where n represents the number of sample images.

8. The semi-supervised image classification method based on distillation network according to claim 1, characterized in that: In step (5), the formula for calculating the total loss function is: In the formula, α is used to adjust the proportion of ceLoss and klLoss, where α is 0.2; ceLoss is the cross loss function, klLoss is the kl divergence, and the calculation formula is as follows: out1 and out2 are obtained by inputting the labeled image img into model 1 and model 2 respectively. i ) and p2(out j ) are used to calculate the probability that the input image belongs to category c using two models respectively. The calculation formula is as follows: Among them, τ represents the adjustable temperature parameter, τ is taken as 20 here, and C represents the total number of categories.

Images