Few-sample image classification method based on CGAN-MAML-ResNet

Abstract

The invention relates to the field of image classification, in particular to a few-sample image classification method based on CGAN-MAML-ResNet, and the method comprises the steps: obtaining an original image, constructing and training a CGAN model, and generating a sample image through the trained CGAN model; dividing a data set formed by the sample image and the original image into a training set and a test set, and constructing a plurality of classification meta-tasks; inputting the training set into the ResNet18 model for training to obtain feature parameters of each layer of the ResNet18 model after training is finished; an improved ResNet18 model is constructed and optimized, and optimal parameters of the model are obtained; and training the target classification task according to the optimal parameter to obtain a classification result. The method has the advantages of high accuracy and strong generalization ability.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a few-sample image classification method based on CGAN-MAML-ResNet. Background Art

[0002] In the field of image classification, traditional deep learning models usually rely on large-scale labeled data sets to train models in order to obtain good classification performance. This method is time-consuming and labor-intensive, and has high application costs. The Few-Shot Learning (FSL) technology is proposed to solve this problem by using a small number of labeled samples to train the model to achieve efficient recognition of new categories. This technology reduces the cost of data acquisition and annotation, and can effectively improve the application performance of the model in data-scarce or emerging fields. For example, in medical image analysis, FSL can help doctors accurately diagnose diseases when samples are insufficient. The FSL method usually includes a training phase and a testing phase: in the training phase, the model is trained using a small data set containing multiple tasks to learn how to extract effective features from a small number of samples; in the testing phase, the model is verified on new category data to evaluate the performance of the model under few sample conditions. In this process, a deep residual network structure is usually applied to enhance feature extraction capabilities and improve classification accuracy.

[0003] At present, the research direction of few-shot image classification mainly focuses on the optimization of model architecture, the improvement of feature representation and the improvement of learning strategy. One of the most cutting-edge research is the combination of Generative Adversarial Networks (GANs) and Meta-Learning.

[0004] Research on few-shot image classification technology has made some progress, but there are still areas that need improvement: in some specific tasks, the quality of generated samples has a greater impact on model performance, which requires more accurate generative adversarial network design; secondly, how to effectively combine different meta-learning strategies to adapt to diverse application scenarios is also a challenge. Therefore, there is an urgent need for a few-shot image classification method that can complete classification tasks with only a small number of sample images and high accuracy. Summary of the invention

[0005] The present invention aims to propose a small-sample image classification method which can complete the classification task with only a small number of sample images and has high accuracy, so as to solve the above problems.

[0006] The steps of the method of the present invention include:

[0007] S1. Obtain the dataset of the original image and target classification task; build and train the CGAN (Conditional Generative Adversarial Network) model, input the original image into the trained CGAN model, and generate sample images, including:

[0008] S11. Generate a random noise vector, flatten the category label image of the original sample image and concatenate it with the random noise vector, and the obtained composite tensor is used as the input information of the CGAN model generator;

[0009] S12. In the downsampling path of UNet, the encoder performs downsampling operation on the input information, and the input information is converted into a 4-level feature map after 4 layers of convolution-pooling;

[0010] S13. In the upsampling path of UNet, the feature maps of each level are fused to generate the sample image;

[0011] S14. Optimize the discriminator and generator, including:

[0012] S141. Initialize the discriminator weights and generator weights;

[0013] S142. Train the discriminator, and calculate the discrimination probability of the discriminator for the real sample and the generated sample according to the discriminator weight;

[0014] S143. Calculate the loss function gradient of the discriminator and use it to update the weight of the discriminator;

[0015] S144. Train the generator and calculate the probability of the discriminator distinguishing the samples generated by the generator according to the generator weight;

[0016] S145. Calculate the gradient of the generator's loss function and use it to update the generator's weight;

[0017] Loop S142 to S145 until the discrimination probability reaches 0.5, then exit the loop;

[0018] S2. Divide the dataset consisting of sample images and original images into training set E1 and test set T1, divide the dataset of the target classification task into training set E2 and test set T2, and construct the classification meta-task of E1 on T1;

[0019] S3. Input the training set E1 into the ResNet18 model for training, and obtain the characteristic parameters of each layer of the ResNet18 model after training, including:

[0020] S31. The generated sample and the original image are sequentially sent to the feature extraction layer of the ResNet18 model to obtain the feature map of the generated sample and the feature map of the original image;

[0021] S32. Multiplying the feature map of the generated sample and the feature map of the original image in the feature dimension to obtain a fused feature map;

[0022] S33. Input the fused feature map into the ResNet18 classification layer to obtain the classification result; calculate the loss function of the model according to the classification result, and when the loss function converges, complete the model training, and obtain the feature parameters of each layer of the ResNet18 model after the training;

[0023] S4. The ResNet18 model is improved according to the characteristic parameters of each layer of the ResNet18 model. The improved ResNet18 model is trained for the first time using the classification meta-task through the MAML algorithm to obtain the optimal parameters of the improved ResNet18 model, including:

[0024] S41. Freeze the parameters of the first convolutional layer and the first three residual blocks of the ResNet18 model to obtain an improved ResNet18 model, and use the characteristic parameters of the ResNet18 model as initial parameters of the improved ResNet18 model;

[0025] S42. Use the classification meta-task to update the inner layer of the improved ResNet18 model;

[0026] S43. Use the classification meta-task to perform outer layer updates on the improved ResNet18 model, and update the global parameters of the model based on the inner layer update results and the outer layer update results;

[0027] S44. Obtain the optimal parameters of the improved ResNet18 model after the inner layer update and the outer layer update;

[0028] S5. Use training set E2 to train the improved ResNet18 model for the second time, and use the optimal parameters of the first training as the initial parameters of the second training of the model. Use T2 to test the improved ResNet18 model after the second training to obtain the classification results of the target classification task.

[0029] The beneficial effects produced by the present invention are:

[0030] This patent uses the CGAN model to generate sample images, which can effectively solve the problems of insufficient data and weak model generalization ability in the task of few-sample image classification; by using the improved ResNet18 model to extract features from the data, the number of parameters for model training is reduced, the memory usage of the calculation process is reduced, and the extraction speed is improved; by using the MAML algorithm to train the improved ResNet18 model, the problem that the model is prone to fall into local optimality when sample data is insufficient can be effectively solved, and the model can be quickly adapted to different types of data, thereby improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A framework diagram of the method of the present invention;

[0032] Figure 2 It is a structural diagram of the CGAN model described in the present invention;

[0033] Figure 3 This is a structural diagram of the ResNet18 model described in the present invention;

[0034] Figure 4 This is a schematic diagram of the first training of the improved ResNet18 model using the classification meta-task described in the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution, characteristics and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] The framework diagram of the method of the present invention is as follows Figure 1 As shown, the specific steps are as follows:

[0037] S1. Obtain the dataset of original images and target classification tasks; build and train the CGAN model, input the original images into the trained CGAN model, and generate sample images:

[0038] Specifically, the generator of the conditional generative adversarial network adopts the UNet network architecture, and the discriminator adopts the PatchGAN network architecture. The structure diagram of the CGAN model is as follows: Figure 2 shown.

[0039] S11. Generate a random noise vector, flatten the category label image of the original sample image and concatenate it with the random noise vector, and the obtained composite tensor is used as the input information of the CGAN model generator;

[0040] Furthermore, the category label image of the sample image, that is, the two-dimensional matrix of pixel values, is first used as conditional information, and then random noise is generated. The label information is flattened and concatenated with the noise vector, and the composite tensor containing the conditional information and random noise is used as the input information of the generator.

[0041] S12. In the downsampling path of UNet, the encoder performs downsampling operation on the input information, and the input information is converted into a 4-level feature map after 4 layers of convolution-pooling;

[0042] Specifically, the encoder in the downsampling path of UNet consists of a convolutional layer, a normalization function, and a ReLU activation function.

[0043] S13. In the upsampling path of UNet, the feature maps of each level are fused to generate the sample image;

[0044] Specifically, in the upsampling path of UNet, the decoder consists of a transposed convolutional layer, a normalization function, and a ReLU activation function.

[0045] S14. Optimize the discriminator and generator, including:

[0046] S141. Initialize the discriminator weights and generator weights;

[0047] S142. Train the discriminator, and calculate the discrimination probability of the discriminator for the real sample and the generated sample according to the discriminator weight;

[0048] Specifically, [X i ,T i ] and [X i ,S i ] Two groups of images are input into the discriminator, and the discriminant probability of the discriminator is calculated to be close to 1; the generated false samples and their conditional information are input into the discriminator, and the discriminant probability of the discriminator is calculated to be close to 0; the expression of the discriminant probability is:

[0049]

[0050] Among them, P represents the discrimination probability, N pix represents the number of pixels in each image, X i , T i , S i They respectively represent the label image of the original sample image, the original sample image, and the image generated by the generator.

[0051] S143. Calculate the loss function gradient of the discriminator and use it to update the weight of the discriminator;

[0052] Specifically, calculate the loss function D of the discriminator loss, update the weight of the discriminator according to the gradient of the loss function, the loss function of the discriminator D loss The expression is:

[0053] D loss =Dr loss +D floss

[0054]

[0055] Among them, D rloss Indicates the use of labeled samples T i The true discriminator loss, D floss Represents the sample S generated by the generator i The false discriminator loss, D represents the discriminator, log(D[x]) represents the logarithmic value of the probability that the discriminator D discriminates x as a real image, log(1-D[X i ,S i ]) represents the logarithmic value of the probability that the discriminator D identifies x as a false image.

[0056] S144. Train the generator and calculate the probability of the discriminator distinguishing the samples generated by the generator according to the generator weight;

[0057] Specifically, in order to make the discriminator judge the fake samples as true, the image category labels of the generated samples and the original samples are input into the discriminator, and the discriminant probability P of the discriminator for the fake samples generated by these generators is calculated.

[0058] S145. Calculate the gradient of the generator's loss function and use it to update the generator's weight;

[0059] Specifically, calculate the loss function G of the generator loss , update the weight of the generator according to the gradient of the loss function, the loss function of the generator G loss The expression is:

[0060] G loss =L bce +λ·P loss

[0061]

[0062] Among them, L bce represents the binary cross entropy loss, and λ represents the weight factor, which controls the pixel-level loss relative to L bce The influence of P loss represents pixel-level loss, N S represents the number of samples in the training batch, D(T i ) and D(S i ) represent the discriminator D to T iand S i The prediction, r Represents the binary labels that distinguish real samples from fake samples.

[0063] Furthermore, r =1 represents the data sample is true, l r ×log(D(T i )) indicates that the discriminator is encouraged to output a high probability of a real image, log(D(T i The )) term penalizes the model when it predicts a low probability of a real image; r =0 represents the data sample is false, (1-l r )×log(1-D(S i )) indicates that the discriminator is encouraged to output a low probability of a fake image, log(1-D(S i The )) term penalizes the model when it predicts a high probability of a fake image.

[0064] Loop S142 to S145 until the discrimination probability reaches 0.5, and the discriminator can no longer judge whether the generated image is real or fake, and exit the loop;

[0065] Specifically, the value of the discriminant probability is used to determine whether the discriminator and the generator have reached Nash equilibrium. Reaching Nash equilibrium means that the discriminator can no longer distinguish the authenticity of the generated samples.

[0066] S2. Divide the dataset consisting of sample images and original images into training set E1 and test set T1, divide the dataset of the target classification task into training set E2 and test set T2, and construct the classification meta-task of E1 on T1;

[0067] Specifically, each meta-task has its own training set (support set) and test set (query set).

[0068] S3. Input the training set E1 into the ResNet18 model for training, and obtain the characteristic parameters of each layer of the ResNet18 model after the training.

[0069] The structure diagram of the ResNet18 model is as follows Figure 3 As shown, feature extraction of samples in the training set further includes:

[0070] S31. The generated sample and the original image are sequentially sent to the feature extraction layer of the ResNet18 model to obtain the feature map of the generated sample and the feature map of the original image;

[0071] Specifically, the feature extraction layer includes a 7×7 convolution layer and four residual blocks.

[0072] S32. Multiply the feature map of the generated sample and the feature map of the original image in the feature dimension to obtain a fused feature map.

[0073] S33. Input the fused feature map into the ResNet18 classification layer to obtain the classification result; calculate the loss function of the model according to the classification result, and when the loss function converges, complete the model training, and obtain the feature parameters of each layer of the ResNet18 model after the training;

[0074] Specifically, the classification layer includes a maximum pooling layer and three fully connected layers, and the number of neurons in the fully connected layers is 1024, 512 and 5 respectively.

[0075] S4. The ResNet18 model is improved according to the characteristic parameters of each layer of the ResNet18 model. The improved ResNet18 model is trained for the first time using the classification meta-task through the MAML algorithm to obtain the optimal parameters of the improved ResNet18 model. Further:

[0076] S41. Freeze the parameters of the first convolutional layer and the first three residual blocks of the ResNet18 model to obtain an improved ResNet18 model, and use the characteristic parameters of the ResNet18 model as the initial parameters of the improved ResNet18 model.

[0077] S42. Use the classification meta-task to update the inner layer of the improved ResNet18 model;

[0078] Specifically, obtain the support set E1 in the meta-task, input the samples in the support set into the classification model to obtain the classification result, and calculate the loss value of the model through the loss function. The inner layer updates the loss function The expression is:

[0079]

[0080] Where N represents the total number of samples in the support set, x i represents the i-th input image on the support set, θ represents the trainable parameters of the network, and y i Represents x i The corresponding classification label, f(θ,x i ) represents the output of the network, L MSE (y i ,f(θ,x i )) represents the mean square error between the model output and the sample labels in the support set. The mean square error calculation formula is:

[0081]

[0082] Calculate the descending gradient value of the parameters in the few-shot classification model, obtain the new parameters of the few-shot classification model, and complete the inner layer update; the expression for the inner loop parameter gradient update is:

[0083]

[0084] Among them, θ ′ represents the updated gradient, θ represents the gradient before the update, and α represents the inner layer update learning rate. Represented by the loss function The gradient obtained by differentiating the parameters θ.

[0085] S43. Use the classification meta-task to perform outer layer updates on the improved ResNet18 model, and update the global parameters of the model based on the inner layer update results and the outer layer update results;

[0086] Specifically, the model parameters after the inner layer update are used as the initial parameters, the query set T1 is obtained, the samples of the query set are input into the model, and the loss function is used to calculate the model. Calculate the loss value of the model. The formula for calculating the loss value is:

[0087]

[0088] in, represents the loss value of the internal update of the model on the query set in the jth task, x k represents the kth input image on the support set, C represents the total number of samples on the query set, and y k Represents x k The corresponding classification label, M represents the number of tasks used to perform outer layer updates.

[0089] Update the network parameters according to the gradient descent method. The expression of the gradient descent method is:

[0090]

[0091] Among them, θ ′ represents the updated gradient, β represents the learning rate when performing outer layer updates, Indicates that The gradient obtained by taking the derivative of θ.

[0092] Use the Adam optimizer to optimize the few-shot classification model and complete the outer layer update.

[0093] S44. Obtain the optimal parameters of the improved ResNet18 model after inner layer update and outer layer update.

[0094] S5. Use training set E2 to train the improved ResNet18 model for the second time, and use the optimal parameters of the first training as the initial parameters of the second training of the model. Use T2 to test the improved ResNet18 model after the second training to obtain the classification results of the target classification task.

[0095] Specifically, the optimal parameters are used as initial parameters and passed to the support set of the target classification task for rapid training, so that the model can achieve good performance on a small number of data sets of the target classification task with only one or two gradient updates, and obtain good classification results. The few-sample classification model of the target classification task is obtained, and then tested on the query set of the new task to obtain the classification result of the target task; the loss function L of the few-sample classification model of the new task is NEW The expression is:

[0096]

[0097] Where D represents the number of training samples in a batch, x W represents the Wth training sample, θ represents the parameters of the classification model, and y W Represents x W The corresponding label, f(θ,x W ) represents the output of the classification model, L MSE (f(θ,x W ),y W ) represents the mean squared error between the model output and the label.

[0098] Finally, it should be noted that the above only describes one embodiment of the present invention. For those skilled in the art, it is conceivable that various changes, modifications, substitutions and deformations may be made to these embodiments without departing from the principle and spirit of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents, and the above-mentioned actions should be covered within the scope of protection of the present invention.

Claims

1. A few-sample image classification method based on CGAN-MAML-ResNet, characterized in that: include: S1. Obtain the dataset of original images and target classification tasks; build and train the CGAN model, input the original images into the trained CGAN model, and generate sample images; S2. Divide the dataset consisting of sample images and original images into training set E1 and test set T1, divide the dataset of the target classification task into training set E2 and test set T2, and construct the classification meta-task of E1 and T1; S3. Input the training set E1 into the ResNet18 model for training, and obtain the characteristic parameters of each layer of the ResNet18 model after the training; S4. Improve the ResNet18 model according to the characteristic parameters of each layer of the ResNet18 model, and use the classification meta-task to perform the first training on the improved ResNet18 model through the MAML algorithm to obtain the optimal parameters of the improved ResNet18 model; S5. Use training set E2 to train the improved ResNet18 model for the second time, and use the optimal parameters of the first training as the initial parameters of the second training of the model. Use T2 to test the improved ResNet18 model after the second training to obtain the classification results of the target classification task.

2. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 1, characterized in that: The CGAN model consists of a generator and a discriminator, where the generator adopts the UNet network architecture and the discriminator adopts the PatchGAN network architecture.

3. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 2, characterized in that: The steps to build a CGAN model include: S11. Generate a random noise vector, flatten the category label image of the original sample image and concatenate it with the random noise vector, and the obtained composite tensor is used as the input information of the CGAN model generator; S12. In the downsampling path of UNet, the encoder downsamples the input information and converts it into a 4-level feature map after 4 layers of convolution-pooling. S13. Perform an upsampling operation on the 4-level feature maps, and fuse the upsampled feature maps of each level to generate a sample image.

4. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 1, characterized in that: When training the CGAN model, the discriminator and generator are optimized. The process of optimizing the discriminator and generator includes: Step 1. Initialize the discriminator weights and generator weights; Step 2. Train the discriminator and calculate the discriminant probability of the discriminator for real samples and generated samples according to the discriminator weights; Step 3. Calculate the gradient of the discriminator's loss function and use it to update the discriminator's weights; Step 4. Train the generator and calculate the probability of the discriminator distinguishing the samples generated by the generator based on the generator weight; Step 5. Calculate the gradient of the generator's loss function and use it to update the generator's weights; Step 6. Repeat steps 2 to 5 until the discrimination probability reaches 0.5, and then exit the loop.

5. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 1, characterized in that: The ResNet18 model includes: a 7*7 convolutional layer, residual block 0, residual block 1, residual block 2, residual block 3, a 3*3 maximum pooling layer, a 1024-size fully connected layer, a 512-size fully connected layer, and a connection layer of size 5. The modules are connected in sequence.

6. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 5, characterized in that: Feature extraction of samples in training set E1 based on the ResNet18 model includes: Step 1. Send the generated sample and the original image to the feature extraction layer of the ResNet18 model in sequence to obtain the feature map of the generated sample and the feature map of the original image; Step 2. Multiply the feature map of the generated sample and the feature map of the original image in the feature dimension to obtain a fused feature map; Step 3. Input the fused feature map into the ResNet18 classification layer to obtain the classification result; calculate the loss function of the model based on the classification result. When the loss function converges, the model training is completed, and the feature parameters of each layer of the ResNet18 model after the training are obtained.

7. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 5, characterized in that: The improvement of the ResNet18 model includes: freezing the 7*7 convolution layer, residual block 0, residual block 1, and residual block 2 of the ResNet18 model, connecting the remaining unfrozen layers to form a model as an improved ResNet18 model, and using the characteristic parameters of the ResNet18 model as the initial parameters of the improved ResNet18 model; the improved ResNet18 model includes: a residual block, a 3*3 maximum pooling layer, a 1024-size fully connected layer, a 512-size fully connected layer, and a 5-size fully connected layer, which are connected in sequence.

8. The method for few-sample image classification based on CGAN-MAML-ResNet according to claim 7, characterized in that: The first training and testing of the improved ResNet18 model using the classification meta-task includes: using the classification meta-task to perform an inner layer update on the improved ResNet18 model; using the classification meta-task to perform an outer layer update on the improved ResNet18 model; and updating the global parameters of the model according to the inner layer update results and the outer layer update results.

Images