A semi-supervised image classification method for class imbalance problem
By combining the teacher-student network model and the meta-weight network, the problems of class imbalance and inconsistent distribution are solved, improving the classification accuracy of semi-supervised learning, especially the classification effect of minority classes.
Patent Information
- Application Number
- CN202310198574.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-03
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2043-03-03
AI Technical Summary
Existing technologies tend to favor the majority class when dealing with imbalanced semi-supervised datasets. Furthermore, when the distributions of labeled and unlabeled sample sets are inconsistent, biases in pseudo-label generation lead to poor classification results.
We employ a teacher-student network model and a meta-weight network. By adaptively adjusting weights and combining unsupervised and supervised losses, we optimize the pseudo-label generation and classification process. The meta-weight network assigns different weights to samples of different categories, alleviating the problems of class imbalance and inconsistent distribution.
It improves the classification accuracy of minority classes, reduces the dependence on hyperparameters, enhances the accuracy of pseudo-labels, and improves the accuracy of classification results.
Smart Images
Figure CN116229169B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision, and particularly relates to a semi-supervised image classification method for class imbalance problems. BACKGROUND
[0002] The data set collected in the real world is usually class imbalance. When the data is imbalanced, the model classification result will usually be biased towards the majority class due to the increase in the prior probability of the majority class. At present, the methods for solving the class imbalance problem can be mainly divided into data level method, algorithm level method and hybrid algorithm. However, most of these methods are based on supervised learning algorithm and need to be trained through a large number of labeled samples. In the real scene, obtaining labeled samples needs a large amount of human cost and time cost, while obtaining unlabeled samples is always very easy. This semi-supervised learning algorithm with labeled samples can learn additional distribution information from unlabeled samples on the basis of supervised learning algorithm.
[0003] The existing image classification method mainly has the following two problems:
[0004] 1. Supervised learning algorithm for class imbalance problem. When encountering semi-supervised data set with labeled samples and unlabeled samples, the classification result will have obvious deviation due to the lack of distribution information of the unlabeled samples.
[0005] 2. The traditional semi-supervised learning algorithm usually assumes that the data set is class balanced, so when processing the class imbalance data set, the classification result of the algorithm will be biased towards the majority class. For example, for the semi-supervised algorithm using pseudo label, class imbalance data will cause the pseudo label generated by the algorithm to be biased towards the majority class.
[0006] 3. The traditional semi-supervised learning algorithm usually assumes that the distribution of the labeled sample set and the unlabeled sample set is consistent, so in the case of inconsistent distribution, such as the labeled sample set is class balanced and the unlabeled sample set is class imbalance, the semi-supervised algorithm using pseudo label will have confirmation bias problem, resulting in poor classification effect.
[0007] Therefore, in view of the defects of the prior art, it is necessary to propose a technical scheme to solve the technical problems existing in the prior art. SUMMARY
[0008] The present application proposes a semi-supervised image classification method for class imbalance problems in view of the deficiencies of the prior art.
[0009] The present application comprises the following steps:
[0010] Step S1: The CIFAR-10 data set is made into a class imbalance semi-supervised data set.
[0011] Step S2: constructing a neural network model, including a teacher-student network model (F T , F S ) and a meta-weight network W for adding adaptive weights to the loss of input.
[0012] Step S3: inputting an unlabeled sample set X u into the teacher network F T , taking the prediction result of the teacher network as the pseudo label of the unlabeled sample
[0013] Input the same batch of unlabeled sample set X u into the student network F S , get the predicted value of the predicted student network F S (X u ; θ S );
[0014] Cross-entropy calculation of pseudo label and prediction F S (X u ; θ S ) to get unsupervised loss L Su ;
[0015] Finally, input the loss L Su into the meta-weight network W to get the student loss L WS containing weights, and update the parameters of the student network through the student loss L WS .
[0016] Step S4: inputting a labeled sample set X l into the updated student network F S , and performing cross-entropy calculation on the obtained prediction value and the label of the labeled sample to get the supervised loss L Sl of the student network; update the meta-weight network W through the supervised loss L Sl .
[0017] Step S5: inputting the labeled sample set X l in step S4 into the teacher network to get the supervised loss of the teacher network;
[0018] Then, the unlabeled sample set X u in step S3 is data enhanced, the enhanced sample set X u1 is input into the teacher network, and the obtained prediction and the pseudo label obtained in step S3 are cross-entropy calculated to get the consistency regularization loss of the teacher network;
[0019] Then, the supervised loss L SlUnsupervised loss converted into the teacher network;
[0020] Finally, the three losses obtained in this step are added to obtain the total teacher network loss; the teacher network is updated by gradient using the total teacher network loss.
[0021] Step S6: repeating steps S3-S5, each time extracting an unlabeled sample set and a labeled sample set from the data set without repetition, and after all extraction and training is completed, it is equivalent to one training cycle, and after each training cycle is completed, the verification sample is used to verify the classification effect of the student network model.
[0022] Step S7: collecting image samples that need to be classified and preprocessing, inputting into the student network model containing the optimal parameters, taking the obtained prediction result as the label of the image sample, and completing the classification task.
[0023] The present application also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the above method when executing the program.
[0024] Compared with the prior art, the present application has the following beneficial effects:
[0025] 1. The meta-weight network is added in the semi-supervised learning method, the classification accuracy of the minority class is improved by applying different weights to different samples, and the influence of the traditional semi-supervised learning algorithm on the classification result being biased towards the majority class is alleviated.
[0026] 2. The meta-weight network is used to replace the loss weight function, the meta-weight network parameters are adaptively adjusted by learning the supervised loss of the network, the setting of the hyperparameters in the loss weight function is reduced, and the number of experimental training times is reduced.
[0027] 3. The teacher network is improved by the supervised loss of the student network, the student network can guide the teacher network to generate more accurate pseudo-labels, thereby improving the classification accuracy of the student network for unlabeled samples. BRIEF DESCRIPTION OF DRAWINGS
[0028] Figure 1 The flowchart of the present application;
[0029] Figure 2 The network framework diagram of the present application;
[0030] Figure 3 The structure diagram of the network model F of the present application;
[0031] Figure 4 The structure diagram of the meta-weight network W of the present application. DETAILED DESCRIPTION
[0032] In order to more clearly illustrate the technical solutions of the present application, the specific embodiments of the present application will be described below with reference to the accompanying drawings.
[0033] As shown in Figure 1 and Figure 2 , the present embodiment includes the following steps:
[0034] Step S1: The CIFAR-10 dataset is made into a class-unbalanced semi-supervised dataset. The CIFAR-10 dataset contains 10 classes of samples: airplane, car, bird, cat, deer, dog, frog, horse, ship and truck, each containing 6000 32*32 color images.
[0035] The present embodiment first divides the CIFAR-10 dataset into a training set and a validation set, with a division ratio of 9:1. Then 4000 samples are randomly selected from each class of samples in the training set as a labeled sample set of the training set, and the remaining 50000 samples are used as an unlabeled sample set. Then the unlabeled sample set is processed for class imbalance, with the number of samples in each class adjusted to δ is the class imbalance rate and is set to 10, i={0, 1, 2, …, 9}. Finally, the dataset is scaled to convert it to a 64*64 image.
[0036] Step S2: Construct a neural network model, including a teacher-student network model (F T , F S ) and a meta-weight network model W. The teacher-student network model is composed of a teacher network F T and a student network F S , both of which use the same network model F, as shown in Figure 3 , but have different update methods.
[0037] The specific steps for constructing the network model F and the network model W in the present embodiment are as follows:
[0038] Step S21: constructing a network model F. The input of the network model F is a picture data set of 3*64*64, and the output is a predicted value of the picture. The first layer of the network model F is a convolution block with a convolution kernel size of 5*5, which includes a convolution layer, a batch normalization layer and an activation function, a step size of 2 and an edge padding of 2, and the output of the first layer is a feature map of 32*32*32. The second layer of the network model F is a convolution block with a convolution kernel size of 3*3, a step size of 2 and an edge padding of 1, and the output of the second layer is a feature map of 64*16*16. The third layer of the network model F is composed of three branches. The first branch is a 5*5 convolution block connected to a 5*5 convolution block, the second branch is a 7*7 convolution block connected to a 3*3 convolution block, and the third branch is a 3*3 convolution block connected to a 7*7 convolution block. The step size of the convolution kernel of the three branches is 1, and there is no edge padding. The outputs of the three branches are added to obtain a feature map of 128*8*8. The fourth layer of the network model F is a maximum pooling layer with a window size of 2*2 and a step size of 2, and the output is a feature map of 128*4*4. The fifth layer of the network model F is an average pooling layer with a window size of 4*4 and a step size of 1, and the output is a feature map of 128*1*1. The fifth layer of the network model F is composed of three fully connected layers, the first fully connected layer has 256 nodes, the second fully connected layer has 512 nodes, and the number of nodes of the third fully connected layer is the number of categories of the data set, which is 10.
[0039] Step S22: constructing a meta-weight network model W, see Figure 4 . The input of the network model W is the loss L Su , and the output is a one-dimensional loss L WS containing weights. The first layer of the network model W is a fully connected layer with 50 nodes, and the second layer is a fully connected layer with 1 node. The activation function of the first layer is a relu function, and the activation function of the second layer is a sigmoid function. The function of the network model W is to add adaptive weights to the loss. For a category with more samples, if its loss is smaller, it is given a smaller weight; for a category with fewer samples, if its loss is larger, it is given a larger weight.
[0040] Step S3: inputting an unlabeled sample set X u with a batch size of 64 into the teacher network F T , taking the prediction result of the teacher network as the pseudo label Y of the unlabeled sample, and then inputting the same batch of unlabeled sample set X u into the student network F S to obtain the prediction F S (X u ; θ S ). The pseudo label Y and the prediction F S (Xu ;θ S The unsupervised loss L is obtained by calculating the cross-entropy. Su Ultimately, the loss will be L. Su The input is fed into the meta-weighted network to obtain the student loss L with weights. WS Through loss L WS The parameters of the student network are updated using gradients.
[0041] In some embodiments:
[0042] Step S31: Transfer the unlabeled sample set X u Input to teacher network F T In the middle, the predicted value F of the teacher network is obtained. T (X u ;θ T ), where θ T These are the parameters for the teacher network.
[0043] Step S32: Use the argmax function to find the predicted value F T (X u ;θ T The index of the maximum value in ) will predict the value F. T (X u ;θ T Convert to pseudo tags
[0044]
[0045] Step S33: Combine the unlabeled sample set X from the same batch. u Input into student network F S In the middle, the predicted value F of the student network is obtained. S (X u ;θ S ), where θ S Parameters for the student network. Pseudo-labels for the teacher network. And the prediction F of student networks S (X u ;θ S The unsupervised loss is calculated using the cross-entropy function. The cross-entropy function is calculated in the following way:
[0046]
[0047] Step S34: Transfer the loss L Su The input is fed into the meta-weighted network W to obtain the weighted loss L. WS =W(L Su ;θ W The updated student network parameters were obtained using stochastic gradient descent. The specific formula is as follows:
[0048]
[0049] Where α is the learning rate of the student network, and T represents the matrix transpose.
[0050] Step S4: Take a labeled sample set X with a batch size of 64. l Input into the updated student network F S In this process, the predicted values and the labels of the marked samples are used to calculate the cross-entropy to obtain the supervised loss L of the student network. Sl Then, through supervised loss L Sl Gradient updates are performed on the meta-weight network. This involves the following steps:
[0051] Step S41: Label the sample set X l Input into the updated student network F S In the middle, the predicted value is obtained.
[0052] Step S42: Convert the predicted value and labeled sample X l The corresponding label Y l Perform cross-entropy calculation to obtain the supervised loss.
[0053] Step S43: By using chain rule differentiation, based on the supervised loss L Sl For the parameters θ of the meta-weighted network W Gradient updates are performed to obtain new network parameters. The specific formula is as follows:
[0054]
[0055] Where β is the learning rate of the meta-weight network.
[0056] Step S5: The labeled sample set X from step S4... l The supervised loss of the teacher network is obtained by inputting the data into the teacher network. Then, the unlabeled sample set X from step S3 is used... u Data augmentation is performed, resulting in an augmented sample set X. u1 Input the data into the teacher network and use the obtained prediction F T (X u1 ;θ T ) and the pseudo-labels obtained in step S3 The consistency regularization loss of the teacher network is obtained by calculating cross-entropy. The supervision loss L of the student network obtained in step S4 is then used. SlThis is transformed into an unsupervised loss for the teacher network. Finally, these three losses are summed to obtain the total teacher network loss, which is then used to update the gradient of the teacher network.
[0057] In some embodiments:
[0058] Step S51: Take the labeled sample set X from step S3. l Input to teacher network F T In the middle, the predicted value is obtained. Predicted values and labeled sample X l The corresponding label Y l Perform cross-entropy calculation to obtain the supervised loss.
[0059] Step S52: For the unlabeled sample set X from step S1 u Data augmentation is performed to obtain the augmented sample set X. u1 .
[0060] The data augmentation used in some of these embodiments primarily includes the following operations:
[0061] Step S521: Perform random horizontal and vertical flips on the image, with a random probability of 0.25.
[0062] Step S522: Randomly select a 59*59 area in the image for cropping, and then expand the image from 59*59 to 64*64.
[0063] Step S523: Randomly select three 2*2 regions A1, A2, and A3 in the image. For a color image, fill the red pixels in region A1 with 0, the green pixels in region A2 with 0, and the blue pixels in region A3 with 0. For a grayscale image, fill the grayscale values of all three regions with 0.
[0064] Step S53: Augment the unlabeled sample set X u1 Input to teacher network F T In the middle, the predicted value F is obtained. T (X u1 ;θ T ). Predicted values And the pseudo-label in step S12 Perform cross-entropy calculation to obtain the consistency regularization loss L. TC .
[0065]
[0066] Step S54: Expand the supervised loss L of the student network using chain rule differentiation. SlUnsupervised loss L of the teacher network Tu Since the supervised loss L Sl is obtained by the student network containing parameters and the parameters are obtained by the loss L WS between the teacher-student network, the supervised loss L Sl can be converted into the unsupervised loss L Tu of the teacher network.
[0067]
[0068] Step S55: Calculate the total teacher loss L T = L Tl + L TC + L Tu Update the parameters of the teacher network using the stochastic gradient descent method to obtain Y is the learning rate of the teacher network.
[0069] Step S6: Repeat steps S3-S5, each time extracting a batch of 64 unlabeled sample sets and a batch of 64 labeled sample sets from the data set without repetition. After all the extraction is completed and the training is completed, it is equivalent to one training cycle. After each training cycle is completed, the classification effect of the student network model is verified using the verification sample.
[0070] In the first 100 training cycles, set the learning rate of the student network α to 0.05, the learning rate of the meta-weight network β to 0, and the learning rate of the teacher network Y to 0.1. After 100 training cycles, α and Y remain unchanged, and β is set to 0.001. When the classification accuracy of the verification sample does not improve for 50 training cycles, it means that the parameters of the student network model have reached the optimal, and the training is terminated.
[0071] Step S7: Collect the image samples that need to be classified and pre-process them, and input them into the student network model containing the optimal parameters. The obtained prediction results are used as the labels of the image samples, and the classification task is completed.
[0072] The above embodiment introduces a meta-weight network and a consistency regularization loss on the basis of the meta pseudo-label algorithm, improves the generation proportion of the few-class pseudo-label, alleviates the influence of class imbalance and inconsistent distribution on the classification result, and improves the classification accuracy.
Claims
1. A semi-supervised image classification method for class imbalance problem, characterized in that The method comprises the following steps: Step S1: making the CIFAR-10 dataset into a class imbalance semi-supervised dataset; Step S2: constructing a neural network model, including a teacher-student network model (F T ,F S ) and a meta-weight network W for adding adaptive weights to the loss of input; Step S3: inputting the unlabeled sample set X u to the teacher network F T with the prediction result of the teacher network as the pseudo label of the unlabeled sample The same batch of unlabeled sample set X u Input to the student network F S , get the predicted value of the predicted student network F S (X u ; θ S ) pseudo-labels and predicted F S (X u ; θ S ) to get unsupervised loss L Su ; Ultimately, the loss will be L. Su The input is fed into the meta-weighted network W to obtain the student loss L containing the weights. WS Through student loss L WS The parameters of the student network are updated using gradients; Step S4: input the labeled sample set X l to the updated student network F S , and cross-entropy calculation is performed between the obtained prediction value and the label of the labeled sample to obtain the supervision loss L Sl of the student network; the meta-weight network W is updated by gradient based on the supervision loss L Sl ; Step S5: inputting the labeled sample set X in step S4 into the teacher network to obtain a supervision loss of the teacher network. l inputting to the teacher network to obtain a supervision loss of the teacher network; The unlabeled sample set X in step S3 is further labeled to obtain a labeled sample set X u Data augmentation is performed on the labeled sample set X to obtain an augmented sample set X u1 The augmented sample set X is input into the teacher network, cross-entropy calculation is performed on the obtained prediction and the pseudo-label obtained in step S3, and a consistency regularization loss of the teacher network is obtained. The supervised loss l of the student network obtained in step S4 is then converted into an unsupervised loss of the teacher network Sl The supervised loss l of the student network obtained in step S4 is then converted into an unsupervised loss of the teacher network Finally, the three losses obtained in this step are added to obtain the total teacher network loss; the total teacher network loss is used for gradient updating of the teacher network; Step S6: repeating steps S3-S5, each time extracting an unlabeled sample set and a labeled sample set from the dataset without repetition, and after all the extraction is completed and the training is completed, it is equivalent to one training cycle; after each training cycle is completed, the classification effect of the student network model is verified using the verification sample; Step S7: collecting image samples that need to be classified and preprocessing, inputting into the student network model with optimal parameters, taking the obtained prediction result as the label of the image sample, and completing the classification task.
2. The semi-supervised image classification method for class imbalance problem according to claim 1, characterized in that: The CIFAR-10 dataset in step S1 contains 10 classes of samples, each class containing 6000 32*32 color images; First, the CIFAR-10 dataset is divided into a training set and a verification set, and the division ratio is 9:1; Secondly, 4000 samples are randomly selected from each class of samples in the training set as the labeled sample set of the training set, and the remaining 50000 samples are used as the unlabeled sample set; Then the unmarked sample set is processed for class imbalance, and the number of samples of each class is adjusted to δ is the class imbalance rate and is set to 10, i = {0, 1, 2, …, 9}; Finally, the dataset is scaled to convert it into a 64*64 image.
3. The semi-supervised image classification method for class imbalance problem according to claim 1, wherein: The teacher-student network model in step S2 is composed of a teacher network F T and a student network F S , both of which use the same network model F but have different updating methods; The input of the network model F is a 3*64*64 picture dataset, and the output is the prediction value of the picture; The first layer of the network model F is a convolution kernel with a size of 5*5, which includes a convolution layer, a batch normalization layer and an activation function, and the step is 2 and the edge padding is 2; the output of the first layer is a 32*32*32 feature map; The second layer of the network model F is a convolution kernel with a size of 3*3, the step is 2 and the edge padding is 1, and the output of the second layer is a 64*16*16 feature map; The third layer of the network model F is composed of three branches: the first branch is a 5*5 convolution block connected to a 5*5 convolution block, the second branch is a 7*7 convolution block connected to a 3*3 convolution block, and the third branch is a 3*3 convolution block connected to a 7*7 convolution block; the convolution kernel step of the three branches is 1, and there is no edge padding; the outputs of the three branches are added to obtain a 128*8*8 feature map; The fourth layer of the network model F is a max pooling layer with a window size of 2*2 and a step of 2, and the output is a 128*4*4 feature map; The fifth layer of the network model F is an average pooling layer with a window size of 4*4 and a step of 1, and the output is a 128*1*1 feature map; The fifth layer of the network model F is composed of three fully connected layers, the first fully connected layer has 256 nodes, the second fully connected layer has 512 nodes, and the third fully connected layer has 10 nodes, which is the number of classes of the dataset.
4. The semi-supervised image classification method for class imbalance problem according to claim 3, characterized in that: The first layer of the meta-weight network W in step S2 is a fully connected layer with 50 nodes, and the second layer is a fully connected layer with 1 node.
5. The semi-supervised image classification method for class imbalance problem according to claim 1, characterized in that: Step S31: inputting the unlabeled sample set X u to the teacher network F T , obtaining the predicted value F T of the teacher network, where θ u is the parameter of the teacher network T . T Step S32: Find the index of the maximum value of the predicted value F T (X u ; θ T ) using the argmax function, convert the predicted value F T (X u ; θ T ) into pseudo-labels Step S33: Combine the unlabeled sample set X from the same batch. u Input into student network F S In the middle, the predicted value F of the student network is obtained. S (X u ;θ S ), where θ S For parameters of the student network; The loss L is calculated using the cross-entropy function Su ; Step S34: the loss L Su input into the meta-weight network W, obtaining the loss L WS containing weights using a random gradient descent method to obtain updated student network parameters 6. The semi-supervised image classification method for class imbalance problem according to claim 1, wherein: Step S5 is specifically: Step S51: Take the labeled sample set X from step S3. l Input to teacher network F T In the middle, the predicted value is obtained. The predicted value and the labeled sample X l The corresponding label Y l Cross-entropy calculation is performed to obtain the supervised loss L Tl ; Step S52: performing data augmentation on the unlabeled sample set X u to obtain an augmented sample set X u1 ; Step S53: obtaining the unlabeled sample set X u1 input into the teacher network F T , obtaining the predicted value F T (X u1 ; θ T ) The predicted value and pseudo labels are used to perform a cross-entropy calculation to obtain a consistency regularization loss L TC ; Step S54: The supervised loss L of the student network is converted into the unsupervised loss L of the teacher network by chain rule expansion. Sl ; and Tu ; and Step S55: Calculate the total teacher loss L T = L Tl + L TC + L Tu ; The parameters of the teacher network are updated using a stochastic gradient descent method to obtain γ is a learning rate of the teacher network.
7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the program to realize the method of claims 1-6.