Small sample image classification method based on self-supervised task aided training

Abstract

The invention provides a small sample image classification method based on self-supervised task aided training, and the method comprises the steps: carrying out the geometric rotation transformation of a support set image in a training stage, generating an extension set, and assisting a conventional scene training task through constructing an instance-level image rotation angle judgment task and a set-level classification consistency task. The measurement network model comprises a feature extraction module and a measurement module, and a three-branch loss optimization path is constructed: a top-layer branch executes traditional scene training to minimize classification loss; the intermediate branch constructs an instance-level proxy task by predicting an image rotation angle; and the bottom layer branch performs classification prediction on different extension sets to construct set-level classification consistency loss. Through joint optimization of a multi-branch loss function, model learning is guided to be independent of feature representation of category labels, and the generalization ability of the model is improved. According to the method, the deep structure features of the image can be effectively captured, and the classification precision and the model generalization ability are synchronously improved in a data scarce scene.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a small sample image classification method based on self-supervised task-assisted training. Background Art

[0002] In recent years, the development of deep learning has significantly fueled advances in artificial intelligence (AI), achieving capabilities comparable to or even exceeding those of humans in a variety of fields, including image classification, object detection, and natural language processing. Despite these remarkable advances, data scarcity and quality remain key challenges in training deep neural networks. In some areas, acquiring large amounts of balanced data is impractical due to limitations in data access, high confidentiality requirements, and the scarcity of rare event data.

[0003] In contrast, humans demonstrate the ability to draw inferences from one instance, enabling them to quickly grasp new concepts with only a handful of examples. This efficient learning is due in part to humans' powerful abilities for comparison, categorization, and association. Therefore, inspired by human cognition and learning, few-shot learning has become a proven technique for addressing data constraints. This technique involves finding task-relevant information within a very small dataset, even with only a few examples in certain categories, and using this information to complete the task.

[0004] The training strategies of most current small-sample image classification methods often employ scenario-based training. By simulating scenarios with scarce samples, the model is exposed to a variety of different tasks during the training phase, thereby enhancing its ability to quickly adapt to new tasks. However, this training strategy can also cause the model to overly focus on features from commonly seen categories during training during testing, weakening its ability to recognize key features in new tasks. To address this issue, it is necessary to conduct research on training strategies and design corresponding small-sample image classification methods. Summary of the Invention

[0005] This application provides a small sample image classification method based on self-supervised task-assisted training, which can be used to solve the problem that the model focuses too much on the features of common categories in the training phase during testing.

[0006] A small sample image classification method based on self-supervised task-assisted training, the method comprising:

[0007] Step A: Sample a small sample data set and divide the data set into training set, validation set and test set;

[0008] Step B: Build a metric network model based on ResNet-12 and Euclidean distance to extract image features and calculate the similarity between image features;

[0009] Step C: Construct a scenario training task, an instance-level self-supervision task, and a set-level self-supervision task, and use the total loss function to make the three tasks cooperate with each other to form a hybrid training task;

[0010] Step D: Use the constructed hybrid training task to train and validate the metric network model;

[0011] Step E: Test the trained metric network model to evaluate its classification accuracy in the small sample image classification task.

[0012] This application proposes an innovative method that combines self-supervised learning and small-sample learning. It constructs self-supervised training tasks such as image rotation angle judgment to assist traditional scenario training, and designs a closely related self-supervised loss function. This method not only enhances the model's learning ability under limited data conditions, but also introduces additional learning signals through self-supervised tasks to help the model better understand the essential features of the image, thereby extracting more generalized feature representations. These improvements have significantly improved the model's adaptability and classification accuracy on new categories. At the same time, the design of the self-supervised loss function effectively stabilizes the training process and reduces the risk of overfitting. This application has simple steps, reasonable design, and easy implementation. It can be effectively applied to small-sample image classification tasks with significant results and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Flowchart of the small sample image classification method based on self-supervised task-assisted training provided in this application.

[0014] Figure 2 Schematic diagram of the hybrid training task provided for this application.

[0015] Figure 3 This is a structural diagram of the feature extraction module provided in this application. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0017] Step A: Sample the small sample dataset and divide the dataset into training set, validation set and test set.

[0018] The small sample dataset is divided into three parts according to the ratio of 3:1:1, with three parts of the total number of images used as training sets, one part as validation set, and the last part as test set. The categories of the three datasets do not overlap with each other.

[0019] Step B: Build a metric network model based on ResNet-12 and Euclidean distance to extract image features and calculate the similarity between image features.

[0020] The constructed metric network model is divided into two parts, the first part is the feature extraction module, and the second part is the metric module.

[0021] The feature extraction module includes ResNet-12; Figure 3 As shown in the figure, ResNet-12 consists of 12 layers of convolution, which are constructed by grouped residual modules. Each residual module contains a skip connection to alleviate the gradient vanishing problem, ensuring the stability and efficient training of deep networks. ResNet-12 contains 4 residual blocks, each of which is composed of 3 layers of convolution. As the network depth increases, the number of channels gradually expands from 64 to 640.

[0022] The measurement module is used to calculate the similarity between the output features of the feature extraction module and use the similarity as the basis for classification. The measurement module mainly uses Euclidean distance as the measurement method. By calculating the Euclidean distance between feature vectors, it can distinguish the features of different categories of images and thus complete the classification task.

[0023] Step C: Construct a scenario training task, an instance-level self-supervision task, and a set-level self-supervision task, and use the total loss function to make the three tasks cooperate with each other to form a hybrid training task;

[0024] Step C includes:

[0025] Step C1: Construct scenario training tasks, and the loss function is recorded as L few ;

[0026] Randomly select N categories in the training set, and randomly select K labeled images from each category to form the support set And randomly select M unlabeled images from the remaining samples of the current category as the query set where x i ,y i Represents labeled images and labels, x j represents an unlabeled image; the support set and the query set together constitute a scenario training task. The task goal is to use the support set S to learn the data distribution of the current task and classify and predict the unlabeled images in the query set Q based on the current data distribution;

[0027] Calculate the prototype of each category in S; for category c, calculate all support samples x in the category i The mean of the feature vector, as the prototype h of the corresponding category c , as follows:

[0028]

[0029] Among them, S c represents the support set of category c, f φ (x i ) is the feature vector output by the feature extraction module;

[0030] For each sample x in the query set j , calculate its eigenvector f φ (x j ) and each category prototype h c The Euclidean distance d(f φ (x j ),h c ); Use the SoftMax function to convert the distance into a probability distribution, representing the query sample x j The probability of belonging to category c is as follows:

[0031]

[0032] Where c' represents the number of categories in the support set;

[0033] According to the true label y of the query sample true The class prototype h true Calculate the cross entropy loss, which is L few , as follows:

[0034]

[0035] Where Q is the number of samples in the query set;

[0036] Step C2: Construct instance-level self-supervision task, and the loss function is recorded as L inst The instance-level self-supervision task focuses on identifying image rotations. Images with different rotations are fed into the model, and the model is asked to predict the correct rotation angle. In theory, if the model can distinguish between different rotations, it must also be able to understand the standard pose of the object. This understanding of the fundamental structure of an object transcends categorization and constitutes a crucial component of transferable knowledge.

[0037] Define a set of two-dimensional rotation operations G = {g r |r=0,...,R-1}, where g r represents the operation of rotating the image by r*90 degrees and R is the total number of rotations (R=4 in the implementation of this application); given a set {S,Q} including a support set and a query set, all two-dimensional rotation operations in G are applied to the images in the set in sequence; an extended set E={{S}} including the original set is generated. r ,Q r}|r=0,...,R-1}, where S r ={(x i ,y i ,r)|y i ∈C e ,i=1,...,l k}, Q r ={(x j ,y j ,r)|y j ∈C e ,j=1,...,l q};{S 0 ,Q 0} is the original set {S,Q};

[0038] Now each set is represented as E r ={(x i ,y i ,r)|y i ∈C e ,i=1,...,l k ,l k +1,...,l k +l q}, where the first l k Samples from S r , the rest l q Samples from Q r ; Through the rotation transformation, each sample (x i ,y i ,r) not only carries a supervised learning class label y i (Supervised learning class labels come from the original categories), and also have self-supervised learning class labels r (from the rotation operation).

[0039] By introducing the rotation label r i , consider the following mapping relationship: in is a parameter θ that can be learned rot Rotation classifier; for each instance (x i ,y i ,r i )∈E, the task of the rotation classifier is to predict the correct rotation angle r i ;

[0040] Given an input pair (x i ,r i ), define the instance-level self-supervised task loss L instThe cross entropy loss is used to measure the difference between the predicted rotation category and the actual rotation label; this loss function encourages the model to learn representations that can be generalized to samples of different categories, because regardless of the specific category of the object, it needs to correctly identify the rotation transformation applied to it. This not only enhances the model's understanding of individual samples, but also promotes the development of its cross-category feature extraction capabilities, thereby improving the overall generalization performance of the model. The cross entropy loss is as follows:

[0041]

[0042] in, is the rotation score vector, f φ (x i ) is the sample x output by the feature extraction module i The eigenvector of Indicates taking the rth i elements;

[0043] Step C3: Construct a collection-level self-supervised task, and the loss function is recorded as L epi ; The extended set E = {{S r ,Q r}|r=0,...,R-1}, and use the classifier to make classification predictions for each extension set; although different extension sets contain images with different rotation transformations, these transformations do not change their class labels. Therefore, the classifier should produce a consistent probability distribution for each instance in different extension sets. This consistency can be measured by the Kullback-Leibler (KL) divergence. That is, using the classifier to make classification predictions for each extension set, by forcing the classification probability distributions of different extension sets to be consistent, the classification consistency between extension sets is maximized;

[0044] In the extended set E, each extended subset {S r ,Q r}, define the classifier on the query set Q r The probability distribution on in It's Q r The jth sample x in j The probability distribution of It represents the probability value of the cth category in the distribution, and the formula is as follows:

[0045]

[0046] Where d(·,·) represents the distance metric function, f φ (x j) is the sample x output by the feature extraction module j The eigenvector of is from the support set S r The prototype representation of the cth category;

[0047] In order to ensure the prediction consistency of the model under different rotation transformations, the average probability distribution at the set level is introduced The formula is as follows:

[0048]

[0049] The total set-level consistency regularization loss is calculated using the KL divergence loss, as follows:

[0050]

[0051] Among them, mean(·) is an average function; it is used to calculate each query sample x j The KL divergence between the predicted probability distribution and the mean probability distribution over all rotation expansion sets is averaged;

[0052] Step C4: Determine the total loss function L total , so that the three tasks cooperate with each other to form a mixed training task;

[0053] Total loss L total Including the self-supervised loss in the self-supervised training task and the supervised loss in the scenario training task, the formula is as follows:

[0054]

[0055] Among them, w1, w2 and w3 are the weights of the loss, which are used to balance the relative importance between the self-supervised loss and the supervised loss. In this application, through experimental research, they are finally set to w1 = 0.5, w2 = 0.5, and w3 = 0.75.

[0056] Step D: Use the constructed hybrid training task to train and validate the metric network model.

[0057] The metric network model is trained using the constructed hybrid training task, and finally a model that meets expectations on the training set is learned; during the training process, images are randomly selected from the validation set to form scenario tasks to verify the performance of the model.

[0058] Step E: Test the trained metric network model to evaluate its classification accuracy in the small sample image classification task.

[0059] Images are randomly selected from the test set to form the scenario task input model, and the query set is predicted through the support set; the average accuracy of multiple test processes is regarded as the performance evaluation result of the model on the test set.

[0060] This application proposes an innovative method that combines self-supervised learning and small-sample learning. It constructs self-supervised training tasks such as image rotation angle judgment to assist traditional scenario training, and designs a closely related self-supervised loss function. This method not only enhances the model's learning ability under limited data conditions, but also introduces additional learning signals through self-supervised tasks to help the model better understand the essential features of the image, thereby extracting more generalized feature representations. These improvements have significantly improved the model's adaptability and classification accuracy on new categories. At the same time, the design of the self-supervised loss function effectively stabilizes the training process and reduces the risk of overfitting. This application has simple steps, reasonable design, and easy implementation. It can be effectively applied to small-sample image classification tasks with significant results and broad application prospects.

[0061] The above-described embodiments of the present application do not constitute a limitation on the scope of protection of the present application.

Claims

1. A small sample image classification method based on self-supervised task-assisted training, characterized in that: The method comprises: Step A: Sample a small sample data set and divide the data set into training set, validation set and test set; Step B: Build a metric network model based on ResNet-12 and Euclidean distance to extract image features and calculate the similarity between image features; Step C: Construct a scenario training task, an instance-level self-supervision task, and a set-level self-supervision task, and use the total loss function to make the three tasks cooperate with each other to form a hybrid training task; Step D: Use the constructed hybrid training task to train and validate the metric network model; Step E: Test the trained metric network model to evaluate its classification accuracy in the small sample image classification task.

2. The method according to claim 1, characterized in that The small sample dataset is divided into three parts according to the ratio of 3:1:1, with three parts of the total number of images used as training sets, one part as validation set, and the last part as test set. The categories of the three datasets do not overlap with each other.

3. The method according to claim 1, characterized in that The constructed metric network model is divided into two parts, the first part is the feature extraction module, and the second part is the metric module. The feature extraction module consists of ResNet-12, which consists of 12 convolutional layers constructed using grouped residual modules. Each residual module contains skip connections to alleviate the vanishing gradient problem. ResNet-12 contains four residual blocks, each of which is composed of three layers of convolution. As the network depth increases, the number of channels gradually expands from 64 to 640. The measurement module is used to calculate the similarity between the output features of the feature extraction module and use the similarity as the basis for classification; The measurement module mainly uses Euclidean distance as the measurement method. By calculating the Euclidean distance between feature vectors, it can distinguish the features of different categories of images and thus complete the classification task.

4. The method according to claim 1, wherein Step C: Construct a scenario training task, an instance-level self-supervision task, and a set-level self-supervision task, and use the total loss function to make the three tasks cooperate with each other to form a hybrid training task; include: Step C1: Construct scenario training tasks, and the loss function is recorded as L few ; Randomly select N categories in the training set, and randomly select K labeled images from each category to form the support set And randomly select M unlabeled images from the remaining samples of the current category as the query set where x i ,y i Represents labeled images and labels, x j represents an unlabeled image; the support set and the query set together constitute a scenario training task. The task goal is to use the support set S to learn the data distribution of the current task and classify and predict the unlabeled images in the query set Q based on the current data distribution; Calculate the prototype of each category in S; for category c, calculate all support samples x in the category i The mean of the feature vector, as the prototype h of the corresponding category c , the method is as follows: Among them, S c represents the support set of category c, f φ (x i ) is the feature vector output by the feature extraction module; For each sample x in the query set j , calculate its eigenvector f φ (x j ) and each category prototype h c The Euclidean distance d(f φ (x j ),h c ); Use the SoftMax function to convert the distance into a probability distribution, representing the query sample x j The probability of belonging to category c is as follows: Where c' represents the number of categories in the support set; According to the true label y of the query sample true The class prototype h true Calculate the cross entropy loss, which is L few , the method is as follows: Where Q is the number of samples in the query set; Step C2: Construct instance-level self-supervision task, and the loss function is recorded as L inst The instance-level self-supervision task focuses on identifying the rotation transformation of an image. Images with different rotation angles are fed into the model, allowing the model to predict the correct rotation angle. Define a set of two-dimensional rotation operations G = {g r |r=0,...,R-1}, where g r represents the operation of rotating the image by r*90 degrees and R is the total number of rotations; given a set {S,Q} including a support set and a query set, all two-dimensional rotation operations in G are applied to the images in the set in turn; an extended set E = {{S r ,Q r }|r=0,...,R-1}, where S r ={(x i ,y i ,r)|y i ∈C e ,i=1,...,l k }, Q r ={(x j ,y j ,r)|y j ∈C e ,j=1,...,l q };{S 0 ,Q 0 } is the original set {S,Q}; Now each set is represented as E r ={(x i ,y i ,r)|y i ∈C e ,i=1,...,l k ,l k +1,...,l k +l q }, where the first l k Samples from S r , the rest l q Samples from Q r ; Through the rotation transformation, each sample (x i ,y i ,r) not only carries a supervised learning class label y i , also with self-supervised learning class label r; By introducing the rotation label r i , consider the following mapping relationship: in is a parameter θ that can be learned rot Rotation classifier; for each instance (x i ,y i ,r i )∈E, the task of the rotation classifier is to predict the correct rotation angle r i ; Given an input pair (x i ,r i ), define the instance-level self-supervised task loss L inst is the cross entropy loss, which is used to measure the difference between the predicted rotation category and the actual rotation label; the cross entropy loss is as follows: in, is the rotation score vector, f φ (x i ) is the sample x output by the feature extraction module i The eigenvector of Indicates taking the rth i elements; Step C3: Construct a collection-level self-supervised task, and the loss function is recorded as L epi ; The extended set E = {{S r ,Q r }|r=0,...,R-1}, use the classifier to make classification predictions for each extension set; use the classifier to make classification predictions for each extension set, and maximize the classification consistency between extension sets by forcing the classification probability distribution of different extension sets to be consistent; In the extended set E, each extended subset {S r ,Q r }, define the classifier on the query set Q r The probability distribution on in It's Q r The jth sample x in j The probability distribution of It represents the probability value of the cth category in the distribution, and the formula is as follows: Where d(·,·) represents the distance metric function, f φ (x j ) is the sample x output by the feature extraction module j The eigenvector of is from the support set S r The prototype representation of the cth category; In order to ensure the prediction consistency of the model under different rotation transformations, the average probability distribution at the set level is introduced The formula is as follows: The total set-level consistency regularization loss is calculated using the KL divergence loss, as follows: Among them, mean(·) is an average function; it is used to calculate each query sample x j The KL divergence between the predicted probability distribution and the mean probability distribution over all rotation expansion sets is averaged; Step C4: Determine the total loss function L total , so that the three tasks cooperate with each other to form a mixed training task; Total loss L total Including the self-supervised loss in the self-supervised training task and the supervised loss in the scenario training task, the formula is as follows: Among them, w1, w2 and w3 are the weights of the loss, which are used to balance the relative importance between self-supervised loss and supervised loss; they are set to w1 = 0.5, w2 = 0.5, w3 = 0.

75.

5. The method according to claim 1, wherein Step D: Use the constructed hybrid training task to train and validate the metric network model, including: The metric network model is trained using the constructed hybrid training task, and finally a model that meets expectations on the training set is learned; during the training process, images are randomly selected from the validation set to form scenario tasks to verify the performance of the model.

6. The method according to claim 1, characterized in that Step E: Test the trained metric network model to evaluate its classification accuracy in the small sample image classification task, including: Images are randomly selected from the test set to form the scenario task input model, and the query set is predicted through the support set; the average accuracy of multiple test processes is regarded as the performance evaluation result of the model on the test set.

Images