A small sample image classification method based on global and local contrastive learning
By combining local and global comparison learning in small sample image classification, the problem of insufficient local feature representation of the model under small sample conditions is solved, and the accuracy of image classification and the generalization ability of the model are improved.
Patent Information
- Application Number
- CN202311588879.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-11-27
AI Technical Summary
When the existing small sample image classification method uses global features, it leads to insufficient local feature representation ability of the model, affecting the classification level on new classes. Especially when the number of samples is limited, the model lacks local changes and foreground background. Sensitivity.
A small sample image classification method based on global and local contrast learning is proposed. Through local contrast learning, each local feature retains category information, and combined with global contrast learning, avoiding global information loss, so as to pay attention to the local area of the image in model training. , enhance local feature representation.
It effectively improves the generalization ability of the model and the accuracy of small sample image classification, ensuring that the model can fully learn object characteristics under limited sample conditions and improve classification performance.
Smart Images

Figure CN117611901B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a small sample image classification method based on global and local contrast learning, and belongs to the field of computer vision. Background Art
[0002] With the development of deep learning, research on small sample image classification technology has made great progress. Its mainstream methods include meta-learning-based methods, transfer learning-based methods, and generation-based methods. These methods have different focuses, but all rely on the effective extraction of image features. In the past two years, contrastive learning has become a new paradigm for image classification model training. Its core idea is to let the model distinguish which data points are similar or different to learn the generalization features of the data set. Many researchers have pointed out that contrastive learning can better extract image features and effectively improve the generalization performance of the model compared to the training method using cross entropy.
[0003] However, most existing methods for applying contrastive learning to small sample image classification are based on global features to calculate contrastive loss. In small sample image classification, using only global features will result in insufficient local feature representation of the model, affecting the classification level of new classes. The reason is that the use of global features is to hope that the feature extraction network can highly abstract image features and transform them into vectors that are insensitive to spatial positions, so that it can learn to distinguish foreground from background and eliminate intra-class differences during training. This idea is very effective when there are sufficient samples, but under small sample conditions, the limited samples make the model insufficiently trained and lack sensitivity to local changes, foreground and background. If the model can be guided to focus on local areas of the image and enhance local feature representation during training, the generalization ability of the model will be greatly improved. Summary of the invention
[0004] The purpose of the present invention is to enhance the local feature representation of the model and improve the accuracy of image classification under small sample conditions, and propose a small sample image classification method based on global and local contrast learning. The method of the present invention first proposes local contrast learning, so that each local feature retains category information and enhances local feature representation; then, global contrast learning is combined with local contrast learning to avoid global information loss. Through the above two methods, the model can fully learn the object features under the condition of limited sample number, thereby effectively improving the model performance.
[0005] In order to achieve the above object, the present invention is implemented by adopting the following technical solutions.
[0006] A small sample image classification method based on global and local contrast learning includes the following steps:
[0007] Step 1, build a small sample image classification model;
[0008] The image classification model uses ResNet-12 as the backbone network, does not contain a classification layer, calculates feature similarity during classification, and uses the nearest neighbor algorithm for classification;
[0009] Step 2: training a small sample image classification model;
[0010] The entire base class dataset is input into the model in batches to extract features. The input data is first enhanced, and then the model is optimized based on global contrastive learning and local contrastive learning.
[0011] Step 3, processing the meta-test dataset in the small sample image classification dataset;
[0012] During the test, the new class data is used to construct a small sample image classification meta-test dataset, and multiple N-way K-shot subtasks are sampled. Each subtask consists of a support set and a query set. The N-way K-shot means randomly extracting N types of (Way) samples from the new class dataset, and randomly extracting K+T (Shot) instances of each type of sample, where K and T represent the number of samples of each type in the support set and query set, respectively.
[0013] Step 4, test the small sample image classification model;
[0014] First, the feature prototype of the category is calculated, then the similarity between a single sample and the category prototype is calculated, and the nearest neighbor algorithm is used for classification.
[0015] As a preferred embodiment of the present invention, the detailed description of the steps described in step 1 is as follows:
[0016] ResNet-12 consists of four residual network structures with a depth of 3. Each residual network structure consists of three 3*3 convolutional layers and a BatchNorm layer. In each network structure, the number of convolution kernels in the three convolutional layers is the same, and there is a 2*2 maximum pooling layer at the end of the network. The number of convolution kernels in the four residual network structures is 64, 160, 320, and 640 respectively. The feature extraction network is represented by E(·).
[0017] As a preferred embodiment of the present invention, the detailed description of the steps described in step 2 is as follows:
[0018] S2-1. Randomly sample a batch of M images from the base dataset and input the images into the data enhancement module to generate a total of 2M images. Randomly select a sample from the enhanced data and record it as x m Represents the mth sample selected.
[0019] S2-2. Input the sample into the feature extraction network E(·), remove the average pooling layer at the last layer of the network, and obtain the local feature u of the image m =E(x m ),u m ∈R C×r , where C represents the feature vector dimension, r = h × w is the spatial size of the feature map, h and w represent the height and width respectively, R C×r The value range of the local feature is a real number set of size C×r; at this time, Represents the local feature vector at the i-th position. The number of local features is determined by the spatial size of the feature map. Each position on the feature map corresponds to a local feature.
[0020] Then, the local contrast learning method considers the local feature vector of each position separately and performs contrast learning between the local features at the same position in each image; represents a set of 2M image samples and their labels. The original image and the enhanced image have the same category label. According to the label information in A(m), we can get m The set of positive samples of the same category is denoted as P(m), where |P(m)| is the number of samples in the set; in the set A(m), m The set of negative samples of different categories is denoted as N(m); the loss form of local contrastive learning is as follows:
[0021]
[0022] Among them, τ∈i + represents the temperature hyperparameter, p represents the p-th sample in P(m), a represents the a-th sample in N(m), and S(·,·) is the cosine similarity between the feature vectors of two images;
[0023] and It is expressed as follows:
[0024]
[0025] in, Represents sample x m The local eigenvector at the i-th position The local feature vector of the i-th position of the p-th positive sample in its positive sample set The cosine similarity between ; Represents sample x m The local feature vector of the i-th position and the local feature vector of the i-th position of the a-th negative sample in its negative sample set The cosine similarity between ; therefore, for sample xm , the feature measurement between it and the positive sample and the negative sample is obtained by averaging the local feature measurement at each position.
[0026] The loss L local The single feature space of global features is expanded into multiple feature spaces of local features. In the feature space corresponding to each spatial position, L local The local features of samples of the same type are brought closer together, and the local features of samples of different types are pushed further away. Therefore, L local This ensures that each local feature of the image retains the identifiable information of its corresponding category as much as possible. After training, the backbone network will be more sensitive to local changes in the image.
[0027] S2-3, perform global contrastive learning, using 2M images randomly sampled and data-enhanced in local contrastive learning, Represents a collection of 2M image samples and their labels; same as local contrast learning, given an image x m , according to the label information in A(m), we can get m The set of positive samples of the same category is denoted as P(m), where |P(m)| is the number of samples in the set; in the set A(m), m The set of negative samples of different categories is denoted as N(m);
[0028] Unlike local contrastive learning, global contrastive learning retains the last average pooling layer of the feature extraction network E(·), so for an image x m , the global feature of the image is represented by z m =E(z m ), z m ∈R C ;
[0029] The form of global contrast loss is expressed as follows:
[0030]
[0031] Among them, τ∈i + represents the temperature hyperparameter, p represents the pth sample in P(m), a represents the ath sample in N(m); S(·,·) is the cosine similarity between the feature vectors of two images, which is expressed as follows:
[0032]
[0033]
[0034] As a preferred embodiment of the present invention, the detailed description of the steps described in step 3 is as follows:
[0035] The testing process is on the new class dataset D novel Multiple N-way K-shot classification task samplings are performed on each task; in each task, the new class dataset D novel It consists of a support set and a query set;
[0036] The number of samples in the support set is N*K, which means that N categories of data are randomly sampled, and each category contains K labeled samples; the support set is expressed by the formula: in Where s represents the sample number of the K samples of the nth category, x s represents the sth sample, y s is the corresponding sample label, S n Represents the set of K samples of the nth category in the new class;
[0037] The number of samples in the query set is N*T, which means that after the support set is sampled, T unlabeled samples are extracted from each category in the N categories of data that have not been extracted from the support set. The query set is expressed by the formula: q represents the sample number of the query set sample; during the test process, the image in the query set is classified into one of the N correct categories based on the labels and samples in the support set.
[0038] As a preferred embodiment of the present invention, the detailed description of the steps described in step 4 is as follows:
[0039] S4-1. During the test, the feature prototype of the category is first calculated;
[0040] Each N-way K-shot task contains a support set and a queryset For an image x in the support set s , after inputting it into the trained backbone network, the global feature z of the image is obtained s . Afterwards, the category prototype of each category in the new class is calculated:
[0041]
[0042] Among them, P n Represents the category prototype of the nth category in the new class;
[0043] S4-2. Use KNN for prediction and classification. For all samples in the query set, calculate the cosine distance between their features and each category prototype, and select the one with the closest distance as the final prediction category:
[0044]
[0045] where D(·,·) represents the cosine distance function, z qFor the query image x q characteristics.
[0046] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0047] The present invention proposes a small sample image classification method that combines local contrast learning and global contrast learning. The method includes two parts: local contrast learning and global contrast learning. Local contrast learning represents an image as multiple local feature vectors, and supervised contrast learning is performed between each local feature; global contrast learning represents an image as a whole, which ensures the category separability of the image as a whole. By combining the two learning methods, the model can pay more attention to the local information of the image and avoid the loss of global information, which can effectively improve the accuracy of small sample image classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a flow chart of a small sample image classification method based on global and local contrast learning of the present invention.
[0049] Figure 2 It is a schematic diagram of the structure of a small sample image classification method that combines global and local contrast learning. DETAILED DESCRIPTION
[0050] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0051] It is worth mentioning that the embodiments of the present invention are described in detail below with reference to the accompanying drawings, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0052] Under the condition of small samples, only focusing on the global features of the image may lead to the model's sensitivity to local changes such as illumination, occlusion, and posture, and the generalization performance may decrease. In order to effectively improve the shortcomings of the above methods, the present invention proposes a small sample image classification method that combines local contrast learning and global contrast learning. The method includes two parts: local contrast learning and global contrast learning. Local contrast learning represents the image as multiple local feature vectors, and supervised contrast learning is performed between each local feature; global contrast learning represents the image as a whole, which ensures the overall category separability of the image.
[0053] Figure 1 is a flow chart of a small sample image classification method based on global and local contrast learning of the present invention. Figure 1 As shown, the specific steps are as follows:
[0054] Step 1, build a small sample image classification model;
[0055] The overall structure of the model is as follows Figure 2 As shown in the figure, this method includes two stages: pre-training and testing. In both stages, ResNet-12 is used as the backbone network to build a small sample image classification model. First, in the pre-training stage, the model is trained through local contrast learning and global contrast learning. Then, in the testing stage, the trained model is used to extract image features, and finally the K nearest neighbor algorithm is used for classification.
[0056] In small sample classification tasks, the data set is usually small, and too many model parameters may lead to overfitting. ResNet-12 reduces the number of parameters that need to be trained while maintaining a certain performance, which helps to better process small sample data. At the same time, the residual connection introduced by the ResNet architecture can help the network to better learn and extract image features, which is particularly important for small sample classification tasks with relatively few labeled samples. Therefore, in the present invention, ResNet-12 is used as the backbone network of the small sample image classification model, and its main structure is 4 residual networks with a depth of 3, each residual network consists of 3 3*3 convolutional layers and BatchNorm layers. In each network structure, the number of convolution kernels of the 3 convolutional layers is the same, and there is a 2*2 maximum pooling layer at the end of the network. The number of convolution kernels of the 4 residual network structures is 64, 160, 320 and 640 respectively. Local contrast learning takes the features before the last pooling layer of ResNet-12, a total of 5*5*640 local features, and supervises contrast learning for each 640-dimensional local feature vector. Global contrastive learning directly uses the 640-dimensional global features output by ResNet-12 and uses supervised contrastive loss for pre-training. The feature extraction network is represented by E(·).
[0057] Step 2: Perform model training based on the classification model built in step 1. First, perform data enhancement on the base class data, and then optimize the model based on global contrastive learning and local contrastive learning. The specific process is as follows:
[0058] (1) Contrastive learning usually requires constructing positive and negative sample pairs, and learning more discriminative feature representations by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Therefore, in this method, the base class data is first expanded using the image enhancement module to provide enough positive and negative sample pairs for subsequent contrastive learning. The specific process is to randomly sample a batch of M images on the base class dataset, and input the images into the data enhancement module to generate a total of 2M images.
[0059] (2) Then randomly select samples from the enhanced sample data for local contrast learning, and record the randomly selected samples as After that, the sample is input into the feature extraction network E(·), and the average pooling layer of the last layer of the network is removed to obtain the local feature u of the image m =E(x m ),u m ∈R C×r , where C represents the feature vector dimension and r = h × w is the spatial size of the feature map. Then, the local contrast learning method considers the local feature vector of each position separately and performs contrast learning between the local features at the same position in each image; represents a set of 2M image samples and their labels. The original image and the enhanced image have the same category label. According to the label information in A(m), we can get m The set of positive samples of the same category is denoted as P(m), where |P(m)| is the number of samples in the set; in the set A(m), m The set of negative samples of different categories is denoted as N(m);
[0060] The loss form of local contrastive learning is as follows:
[0061]
[0062] Among them, τ∈i + represents the temperature hyperparameter, represents a set of 2M image samples and their labels, P(m) represents the m The positive sample set of the same category, N(m) represents the positive sample set with x m A set of negative samples of different categories, p represents the pth sample in P(m), a represents the ath sample in N(m), and S(·,·) is the cosine similarity between the feature vectors of two images;
[0063] and It is expressed as follows:
[0064]
[0065]
[0066] in, Represents sample x m The local eigenvector at the i-th position The local feature vector of the i-th position of the p-th positive sample in its positive sample set The cosine similarity between ; Represents sample x m The local feature vector of the i-th position and the local feature vector of the i-th position of the a-th negative sample in its negative sample set The cosine similarity between ; therefore, for sample xm , the feature measurement between it and the positive sample and the negative sample is obtained by averaging the local feature measurement at each position.
[0067] Loss L local The single feature space of global features is expanded into multiple feature spaces of local features. In the feature space corresponding to each spatial position, L local The local features of samples of the same type are brought closer together, and the local features of samples of different types are pushed further away. Therefore, L local This ensures that each local feature of the image retains the identifiable information of its corresponding category as much as possible. After training, the backbone network will be more sensitive to local changes in the image.
[0068] (3) Perform global contrastive learning
[0069] Global contrastive learning uses the same augmented dataset as local contrastive learning We can also use the given sample image x m The label of is used to obtain the positive sample set P(m) of the same category and the negative sample set N(m) of different categories.
[0070] Unlike local contrastive learning, global contrastive learning retains the last average pooling layer of the feature extraction network E(·), so for an image x m , the global feature of the image is represented by z m =E(z m ), z m ∈R C .
[0071] The form of global contrast loss is expressed as follows:
[0072]
[0073] Among them, τ∈i + represents the temperature hyperparameter, p represents the pth sample in P(m), a represents the ath sample in N(m), and S(·,·) is the cosine similarity between the feature vectors of two images, which is expressed as follows:
[0074]
[0075]
[0076] In global contrastive learning, the entire image is considered as a whole and similarities are learned by comparing global features in the image. The model tries to capture the structure and content of the entire image in this process so that global consistency is emphasized in learning.
[0077] Step 3: The global contrast loss and local contrast loss in step 2 are linearly summed as the final loss function to train the model, and then the new class dataset is used to test the model performance. Before testing, the new class dataset must be used to construct N-way K-shot meta-test data. The specific construction process is as follows:
[0078] The meta-test data contains independent N-way K-shot subtasks. Each subtask consists of a support set and a query set. The sample data of the support set and the query set come from the new class dataset D. novel The number of samples in the support set is N*K, which means that N categories of data are randomly sampled, and each category has K labeled samples. The support set can be expressed as: in It represents the set of K samples of the nth category in the new class. The number of samples in the query set is N*T, which means that after the support set is sampled, T unlabeled samples are extracted from each category in the N categories of data that have not been extracted from the support set. The query set can be expressed by the formula: During testing, the images in the query set need to be classified into one of the N correct categories based on the labels and samples in the support set.
[0079] Step 4: Test the performance of the small sample classification model based on the meta-test data set constructed in step 3. The K nearest neighbor algorithm is used for classification in the test phase. The specific test process is as follows:
[0080] (1) First, calculate the feature prototype of each category in the new class dataset. Each N-way K-shot task contains a support set and a queryset For an image x in the support set s , after inputting it into the trained backbone network, the global feature z of the image is obtained s . Afterwards, the category prototype of each category in the new class is calculated:
[0081]
[0082] Among them, P n Represents the category prototype of the nth category in the new class.
[0083] (2) Use KNN for prediction and classification. For all samples in the query set, calculate the cosine distance between their features and each category prototype, and select the one with the closest distance as the final prediction category:
[0084]
[0085] Where D(·,·) represents the cosine distance function, z q For the query image xq characteristics.
[0086] Under small sample conditions, the differences between categories tend to be concentrated in certain local areas of the image. Using local contrast learning can more effectively capture these local features and improve sensitivity to key details. In addition, due to the limited training data under small sample conditions, by focusing on the local information of the image, the limited data can be more fully utilized for training. However, focusing only on local information may lead to ignoring the global relationship of the image. Therefore, in this method, local contrast learning is combined with global contrast learning to ensure that the model can capture the local details of the image and pay attention to the overall structure of the image under small sample conditions, so that the model can understand the image content more comprehensively and effectively improve the classification accuracy.
[0087] The above embodiments are only for illustrating the technical idea of the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.
Claims
1. A small sample image classification method based on global and local contrast learning, characterized in that: The steps include: Step 1, build a small sample image classification model; The image classification model uses ResNet-12 as the backbone network, does not contain a classification layer, calculates feature similarity during classification, and uses the nearest neighbor algorithm for classification; the feature extraction network is represented by E(·); Step 2: training a small sample image classification model; The entire base class dataset is input into the model in batches to extract features. The input data is first enhanced, and then the model is optimized based on global contrastive learning and local contrastive learning. Step 3, processing the meta-test dataset in the small sample image classification dataset; During the test, we use new class data to construct a small sample image classification meta-test dataset, sampling multiple N-way K-shot subtasks, each of which consists of a support set and a query set. Step 4, test the small sample image classification model; First, the feature prototype of the category is calculated, then the similarity between a single sample and the category prototype is calculated, and the nearest neighbor algorithm is used for classification; The specific steps of step 2 are as follows: S2-1. Randomly sample a batch of M images from the base dataset and input the images into the data enhancement module to generate a total of 2M images. Randomly select a sample from the enhanced data and record it as x m Indicates the mth sample selected; S2-2. Input the sample into the feature extraction network E(·), remove the average pooling layer at the last layer of the network, and obtain the local feature u of the image m =E(x m ),u m ∈R C×r , where C represents the feature vector dimension, r = h × w is the spatial size of the feature map, h and w represent the height and width respectively, R C×r The value range of the local feature is a real number set of size C×r; at this time, Represents the local feature vector at the i-th position. The number of local features is determined by the spatial size of the feature map. Each position on the feature map corresponds to a local feature. Then, the local contrast learning method considers the local feature vector of each position separately and performs contrast learning between the local features at the same position in each image; represents a set of 2M image samples and their labels. The original image and the enhanced image have the same category label. According to the label information in A(m), we get m The set of positive samples of the same category is denoted as P(m), where |P(m)| is the number of samples in the set; in the set A(m), m The set of negative samples of different categories is denoted as N(m); the loss form of local contrastive learning is as follows: Among them, τ∈i + represents the temperature hyperparameter, p represents the p-th sample in P(m), a represents the a-th sample in N(m), and S(·,·) is the cosine similarity between the feature vectors of two images; and It is expressed as follows: in, Represents sample x m The local eigenvector at the i-th position The local feature vector of the i-th position of the p-th positive sample in its positive sample set The cosine similarity between ; Represents sample x m The local feature vector of the i-th position and the local feature vector of the i-th position of the a-th negative sample in its negative sample set The cosine similarity between m , the feature measurement between it and the positive sample and negative sample is obtained by averaging the local feature measurement at each position; S2-3, perform global contrastive learning, using 2M images randomly sampled and data-enhanced in local contrastive learning, Represents a collection of 2M image samples and their labels; same as local contrast learning, given an image x m , according to the label information in A(m), we can get m The set of positive samples of the same category is denoted as P(m), where |P(m)| is the number of samples in the set; in the set A(m), m The set of negative samples of different categories is denoted as N(m); Global contrastive learning retains the last average pooling layer of the feature extraction network E(·). For an image x m , the global feature of the image is represented by z m =E(z m ), z m ∈R C ; The form of global contrast loss is expressed as follows: Among them, τ∈i + represents the temperature hyperparameter, p represents the pth sample in P(m), a represents the ath sample in N(m); S(·,·) is the cosine similarity between the feature vectors of two images, which is expressed as follows:
2. The small sample image classification method based on global and local contrast learning according to claim 1, characterized in that: The specific description of step 1 is as follows: ResNet-12 consists of 4 residual network structures with a depth of 3. Each residual network structure consists of 3 3*3 convolutional layers and a BatchNorm layer. In each network structure, the number of convolution kernels in the three convolutional layers is the same, and there is a 2*2 maximum pooling layer at the end of the network. The number of convolution kernels in the four residual network structures are 64, 160, 320 and 640 respectively.
3. The small sample image classification method based on global and local contrast learning according to claim 2, characterized in that: The specific method of step 3 is as follows: The testing process is on the new class dataset D novel Perform multiple N-way K-shot classification task samplings on it; In each task, the new class dataset D novel It consists of a support set and a query set; The number of samples in the support set is N*K, which means that N categories of data are randomly sampled, and each category contains K labeled samples; the support set is expressed by the formula: in Where s represents the sample number of the K samples of the nth category, x s represents the sth sample, y s is the corresponding sample label, S n Represents the set of K samples of the nth category in the new class; The number of samples in the query set is N*T, which means that after the support set is sampled, T unlabeled samples are extracted from each category in the N categories of data that have not been extracted from the support set. The query set is expressed by the formula: q represents the sample number of the query set sample; During testing, images in the query set are classified into one of the N correct categories based on the labels and samples in the support set.
4. The small sample image classification method based on global and local contrast learning according to claim 3, characterized in that: The specific steps of step 4 are as follows: S4-1. During the test, the feature prototype of the category is first calculated; Each N-way K-shot task contains a support set and a queryset For an image x in the support set s , after inputting it into the trained backbone network, the global feature z of the image is obtained s ; Afterwards, calculate the category prototype for each category in the new class: Among them, P n Represents the category prototype of the nth category in the new class; S4-2. Use KNN for prediction and classification. For all samples in the query set, calculate the cosine distance between their features and each category prototype, and select the one with the closest distance as the final prediction category: where D(·,·) represents the cosine distance function, z q For the query image x q characteristics.
Citation Information
Cited By
Efficient data classification method and apparatus based on dictionary contrastive learning via adaptive label embedding
US20250336195A1
