A dual-branch small sample image classification method based on contrastive learning

Abstract

The present invention discloses a dual-branch small sample image classification method based on contrastive learning, and belongs to the technical field of image classification. This method designs a dual-branch model, in which one branch focuses on global feature extraction and the other branch focuses on local feature extraction, and realizes efficient fusion of the two features through contrastive learning. The proposed saliency mask mixing and confidence cropping selector module, combined with the global self-supervised contrast loss and cross-view scenario training mechanism, can effectively improve the robustness and generalization ability of the model. The specific technical scheme includes a data enhancement method based on random cropping and saliency fusion, an image mixing technology with saliency region priority, and a contrastive learning optimization strategy for images of different viewpoints. The present invention solves the potential performance bottleneck problem in the process of fusing global features and local features, and realizes the accurate classification of images under complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the field of image classification, in particular to an image classification method in a small sample learning technology, and specifically belongs to the field of a small sample image classification method based on contrastive learning, data enhancement and feature alignment. Background Art

[0002] Few-Shot Learning (FSL) is an important research direction in the field of machine learning and deep learning. It aims to solve the problem of how to effectively learn and generalize when the number of training samples is very limited. Traditional machine learning methods rely on a large amount of labeled data to train models, but in practical applications, obtaining large-scale labeled data is usually expensive and time-consuming. In this context, few-shot learning provides a method that can still achieve high performance when the number of samples is limited.

[0003] Current research in this field can be divided into three main categories: gradient-based methods, metric-based methods, and data augmentation-based methods. Gradient-based methods usually rely on the idea of ​​meta-learning, i.e., "learning how to learn". These methods optimize the initial parameters of the model in a multi-task environment so that the model can quickly adapt to new tasks through a small number of gradient updates. The advantage of gradient-based methods is that they can flexibly adapt to different task types and maintain consistency during training and inference. Metric-based methods mainly rely on constructing a feature space in which similar samples are closer and samples of different categories are farther apart. Data augmentation-based methods aim to improve the generalization ability of the model by expanding limited training data. This type of method includes traditional data augmentation techniques such as image rotation, cropping, flipping, etc., as well as more complex methods such as generative adversarial networks to generate new samples, or create new data by mixing different samples. Among these methods, the current feature extraction strategies can be divided into two main methods, namely global feature methods and local feature methods. Global feature methods simplify the subsequent measurement process by compressing the entire image into a unified feature vector. In order to improve the representation ability of the model, researchers mainly focus on two aspects of optimization: one is to design a more efficient network architecture to extract more accurate prototype features; the other is to develop a more accurate similarity evaluation method to ensure accurate measurement between different images. In contrast, the local feature method divides the image into multiple local regions and extracts corresponding features from each region. This method not only focuses on the accurate measurement of local features, but also strives to understand the semantic relationship between images. By capturing detailed information of the image and establishing richer inter-image associations, the local feature method significantly improves the robustness and accuracy of the measurement. However, relying solely on global features or local features for measurement is prone to introduce bias. Therefore, there is an urgent need for a method that can take into account both global and local features to further improve model performance. Summary of the invention

[0004] Purpose of the invention: Current feature extraction strategies can be divided into two main methods, namely global feature methods and local feature methods. The global feature method compresses the entire image into a unified feature vector, thereby simplifying the subsequent measurement process. In order to improve the representation ability of the model, researchers focus on two optimization schemes: designing a more effective network architecture to extract more accurate prototype features, or developing a more accurate similarity evaluation method to ensure accurate measurement results between different images. In contrast, the local feature method decomposes each image into multiple local regions and extracts corresponding features from each region. This method not only emphasizes the use of local features for accurate measurement, but also aims to understand the semantic associations between different images. By capturing detailed information in the image and establishing richer associations between different images, this method improves the robustness and accuracy of the measurement. However, this method has two problems:

[0005] (1) Global and local feature methods have been intensively studied and have shown excellent performance, but one key area has not received sufficient attention: combining global feature-based measurements with local feature-based measurements. Although some studies have attempted to combine the two types of features, they usually use global features to enhance local features rather than treating them as independent and equally important components.

[0006] (2) Global features are favored due to their inherent invariance, but are easily disturbed by image noise in complex backgrounds. In contrast, local features excel in capturing detailed image information and are robust to noise and minor changes, but face challenges in alignment. The complementary advantages of global and local features suggest that their combined use can achieve a synergistic effect and further improve model performance. However, simply fusing global and local features together for measurement cannot fully tap the potential and advantages of each feature.

[0007] In order to better integrate global features and local features for model training, in this patent, we proposed a dual-branch small sample image classification method based on contrastive learning to better integrate the two losses through contrastive learning. We designed a data enhancement method using cropping and splicing to force the model to learn local information and alleviate the problems caused by inductive bias. Global loss We generate images from different perspectives through two different data enhancement methods (such as rotation, brightness adjustment, etc.), and by aligning the prototypes, the model can recognize the target object from different perspectives, enhancing the model's global image recognition ability.

[0008] Technical solution: In order to achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0009] (1) Prepare datasets, including miniImageNet, tieredimageNet, and CUB datasets, and predefine the methods.

[0010] (2) The model performs a dual-branch processing, and the feature extractor shares parameters. One branch focuses on local features, and the other branch focuses on global features.

[0011] (3) In the first branch, the support set images are subjected to a data enhancement method based on saliency occlusion blending to expand the support set. Samples with the same labels as the support set are used as positive samples, and samples with different labels are used as negative samples.

[0012] (4) Randomly crop the query set images, sort them by confidence using the CCS method, remove the highest and lowest confidence images, and use the remaining images as the query set for comparative learning.

[0013] (5) Perform saliency mixing on the images, generate mixed labels for the images according to the degree of mixing, and calculate the classification loss.

[0014] (6) The category prototypes in the support set are processed through the multi-head attention mechanism (Attn), so that the prototypes of the same category are aligned in the support sets of different perspectives, thereby enhancing the generalization ability of the model in few-shot classification tasks.

[0015] (7) The global self-supervised contrastive loss increases the similarity between different views of the same image by aligning them, while reducing the similarity between different images.

[0016] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0017] To validate the method of this patent, experiments were conducted on several widely used datasets. miniImageNet is a sub-dataset from ImageNet. It has a total of 100 categories with 600 instances in each category. These categories are divided into training, validation, and test sets, which are 64, 16, and 20 categories, respectively. tiere-dImageNet is also sampled from ImageNet. It consists of 779,165 images from 608 categories. They are divided into training, validation, and test sets, which are 351 categories, 97 categories, and 160 categories, respectively. Fewshot-CIFAR100 (FC-100) data is a subset of CIFAR-100. A common division is 60 categories for training, 20 categories for validation, and testing. CUB is a fine-grained classification dataset covering 200 bird species, 100 for basic categories, 50 for evaluation, and the remaining 50 for novel categories.

[0018] In order to make a fair comparison with previous work, this patent selected ResNet12 as the basic architecture of the model in this study and implemented it strictly in accordance with the specifications of Meta-Baselin. The He-normal method was used to initialize the model parameters. In terms of optimization, this patent used stochastic gradient descent (SGD) and set the initial learning rate to 0.1. When experimenting on the miniimageNet dataset, this patent iteratively adjusted the learning rate at the 12,000th, 14,000th, and 16,000th episodes. At the same time, in the experiments on the tieredImageNet dataset, the learning rate was halved every 24,000 episodes. The model evaluation covered all experiments totaling 2,000 episodes. During the training phase, each batch included 4 episodes as training samples.

[0019] To evaluate the performance of our model, we compared it with several previous methods, including ProtoNet, infoP-atch, LSFSL, MML, TALDS, and RENet, etc. These methods either represent classic Few-Shot Learning (FSL) methods or have previously demonstrated the best reported results.

[0020] In Table 1, this patent compares the two datasets, miniImageNet and tie-redImageNet. The backbone used in the comparison methods is Resnet-12. Compared with DeepEMD, this patent has a smaller amount of calculation in calculating the saliency map, and does not require additional parameters compared with RENet and CVET, which also shows that the method of this patent is superior to other methods. In Table 2, the CUB dataset is compared. Even though the number of backbone network parameters of some methods is larger than that of this patent method, the effect of this patent method is still better than other methods.

[0021] The experimental results are shown in Table 1:

[0022] Table 1 Comparison of the accuracy of the model with other methods on the miniImageNet and tieredImageNet datasets

[0023]

[0024] Table 2 Comparison of the accuracy of the model with other methods on the CUB dataset

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1A flowchart of a dual-branch small sample image classification method based on contrastive learning provided in an embodiment of the present invention.

[0027] Figure 2 This is a specific flow chart of this branch of global loss output according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with the accompanying drawings.

[0029] This patent follows a two-stage training strategy and incorporates contrastive learning in both stages to learn more general representations. Our proposed framework Figure 1 In the pre-training stage, self-supervision and supervised contrastive loss are used to obtain good initial representations. In the meta-training stage, a new cross-view episode training (CVET) mechanism and distance-scale contrastive loss are proposed, which enables the model to overcome the bias between each episode viewpoint and generalize well in new tasks. We will introduce our framework in detail in the following subsections.

[0030] like Figure 1 As shown in FIG. 1 , the overall processing flow of the method mainly includes the following processing steps:

[0031] (1) In this section, we explore the field of small sample recognition in depth. train , X val and X test They represent the training set, validation set, and test set, respectively, and their corresponding labels are represented by Y train , Y val and Y test The whole set of these is denoted by D train =X train , Y train , D val =X val , Y val and D test =X test , Y test As for C train , C val and C test , which represent the categories in the training set, validation set, and test set. We adopt the scenario mechanism to simulate the evaluation setting. Specifically, each M-way K-shot set E consists of a support set and a query set. We first start from C base Randomly sample M classes for meta-training (or from C novelRandom sampling for meta-testing), we randomly sample K instances per class to get a support set S = {xi, yi}M*Ki = 1. We then sample Q instances in each of the selected classes to get a query set Q = {x i ,y i}M*Q i = 1. Please note that y i ∈{1, 2, ..., M} and The episodic training process classifies samples in Q into the categories corresponding to samples in S.

[0032] (2) This patent proposes a dual-branch network model based on contrastive learning, which includes a saliency mask mixing algorithm and a global distance scale algorithm. The overall process is as follows Figure 1 As shown in the figure, the algorithm of saliency mask mixing consists of three key modules. The first module is the visual feature hidden fusion module (M-Mix), which optimizes the infoNCE loss to adapt to small sample learning and uses novel data enhancement techniques to generate samples for contrastive learning. The second component is a module called confidence crop selector (CCS), which randomly crops the query image into multiple blocks and uses a confidence-based method to screen out high-quality query samples, increasing the number of query samples and improving the training quality. The third module uses saliency fusion to perform visual feature saliency fusion method (SF) to try to retain salient areas in the process of fusing images, mainly to provide better and richer information for the model.

[0033] (3) The visual feature hidden fusion module (M-Mix) improves contrastive learning by fusing two images into one sample, thereby constructing more informative positive and negative samples in small sample classification tasks. The image is divided into salient blocks and non-salient blocks, and a part of the blocks are randomly blocked before fusion, with salient areas being retained first. By generating contrast samples in this way, the model can capture key information more effectively. Finally, the inner product formula between the query and support and the infoNCE loss are used to train the model to enhance the robustness and generalization ability of the model. This patent considers support instances with the same label as its positive samples, and instances with different labels as negative samples. The same embedding network Φ is used for query instances and support instances. The loss function formula of this patent is:

[0034]

[0035] Where L i represents the contrastive learning loss of the i-th query sample, f i q refers to the feature representation of the i-th query sample, and and denote the feature representations of the support samples , , and k, respectively. Sample j shares the same label as query sample i Therefore, sample j is a positive sample. And k indicates that it has a different label from the query sample. Therefore, sample k is a negative sample.

[0036] (4) Improve contrastive learning by fusing two images into one sample, thereby constructing more informative positive and negative samples in small sample classification tasks. The image is divided into salient blocks and non-salient blocks, and a portion of the blocks are randomly masked before fusion, with salient areas being retained first. By generating contrastive samples in this way, the model can more effectively capture key information. Finally, the inner product formula between the query and the support and the infoNCE loss are used to train the model, enhancing the robustness and generalization ability of the model.

[0037] (5) The fusion formula is expressed as: Mix(x i ,y i )=λ·x i +(1-λ)·x j , where λ is a random weight sampled from a Beta distribution. Here, x i and i Denote the input image and label respectively. Let x1 and x2 be two input images, expressed as and in and Indicates an insignificant area. and represents a significant area. The mask can be expressed as D. The fusion formula can be expressed as:

[0038]

[0039] This operation generates contrasting samples, highlights the differences, and enables the model to prioritize key image regions. As a result, the model improves its ability to capture key information.

[0040] (6) The inner product between query and support is expressed as follows:

[0041] F=f i q ×M[f j (s1), f j (s2)],

[0042] Among them, s1 and s2 represent two images in the two supporting samples selected by this patent, and the inner product is calculated with the query set. The fused infoNCE can be written as:

[0043]

[0044] (7) In order to solve the problem of limited number of query samples, this patent adopts random cropping technology for data augmentation and proposes a confidence cropping selector (CCS) method. By generating multiple cropped images and extracting their feature vectors, these images are sorted by confidence, the images with the highest and lowest confidence are removed, and the images in the middle are retained for comparative learning, thereby improving the robustness and generalization of the model.

[0045] (8) Given the limited number of query samples, this patent uses random cropping technology to augment the data. The quality and diversity of query samples directly affect the robustness and generalization of the model. Using representative query samples helps the model adapt to different environments and background changes, but not all cropped images from the query can achieve optimal results. Some samples may have information value, while others may not be sufficient to provide sufficient information. Therefore, selecting high-quality samples is crucial to this patent. In order to make more effective use of query data, this patent proposes a confidence cropping selector (CCS) method for processing query images. This patent uses the original image to generate approximately six new images and randomly crops them. All cropped images are processed by the same feature extractor to generate corresponding feature vectors:

[0046]

[0047] The goal of this formula is to minimize the feature matrix by adjusting the parameter γ and predicted values The difference between is measured using the squared Frobenius norm. In this formula, represents the matrix of image features, γ represents the parameters of the linear model used to combine these features, and R(γ) represents the regularization term applied to control the complexity of the model. The ultimate goal is to rank images according to a certain criterion.

[0048] (9) As λ changes, γ gradually becomes sparser until all elements disappear. This patent determines γ i The vanishing λ value is used to rank the confidence of the pseudo-labeled data according to this vanishing point. Humans can recognize objects by part of them, and the model can learn better embedded features if it acquires this ability. In order to achieve semi-occluded scenes, images with high or low confidence are not needed. Therefore, this patent removes the images with the highest and lowest confidence, and uses the remaining images as separate queries for comparative learning.

[0049] (10) The visual feature saliency fusion module (SF) adopts a data enhancement strategy that does not require a mask. This patent optimizes the fusion of salient areas by dividing the image into multiple blocks and designating the most salient blocks for labeling. Different hybrid techniques are used to highlight the salient areas of the image, creating more distinctive and recognizable images, and improving the model's ability to recognize and understand key features. In addition, this patent proposes a formula for generating hybrid labels, which generates the final label based on the hybrid weight and the number of hybrid patches, thereby providing a variety of perspectives in model training.

[0050] (11) This patent will be the most significant N fore Blocks are designated as labeled blocks, represented by green and purple, and the fusion formula is:

[0051]

[0052] This method highlights the importance of salient regions in the image. Different mixing techniques lead to images with unique visual features. This mixing potential helps create more distinctive and recognizable images and provides a variety of perspectives for model training, which helps to keenly identify and understand key features. The formula for the mixed label is as follows:

[0053]

[0054] Where g(y0, y1) represents the mixed label, y0 and y1 correspond to the labels of the two original samples respectively. W×W represents the size of the image or the number of patches in the image. c(minter) represents the number of activated elements, that is, the value of the activated element in the binary mask m is 1. The parameter λ in the formula is the mixing weight, which determines the degree of mixing between the two labels. In addition, and denote the number of inactive patches in the two samples respectively. This formula aims to generate the final mixed label based on the mixed weight and the number of mixed patches.

[0055] (12) Cross-view episodic training. In order to capture the high-level conceptual information of the shared context, a common practice in contrastive learning is to maximize the mutual information between features extracted from multiple views of the shared context. Intuitively, given an episode E = {S, Q} in episodic meta-learning, we can obtain two episodes E1 = {S1, Q1} and E2 = {S2, Q2} by applying two different data augmentation strategies to E respectively. Here, we regard E as a shared context and the two enhanced episodes as its two views. Inspired by the above ideas, we propose a cross-view episodic training mechanism with the aim of forcing representation learning to play a key role in various few-shot classification tasks. Specifically, we i Apply the pre-trained feature extractor f in Section 3.3 φ And global average pooling (GAP), and derive the corresponding global vector Then, we calculate the prototype of category k in support sets S1 and S2 respectively The calculation is as follows:

[0056]

[0057] where r∈{1,2} and denotes the set of data points in the r-th support set belonging to category k∈{1, 2, ..., M}. We denote the task-specific module proposed in the baseline as Attn(·) and fix the number of heads to 1. Through the task-specific module, we obtain the aligned prototype set for each support set

[0058] (13) Based on the aligned prototypes, define the query set Q r Data point x i The probability distribution for M categories is as follows:

[0059]

[0060] Where d(·,·) represents the Euclidean distance. Therefore, the loss of the nearest centroid classifier on an episode can be calculated as:

[0061]

[0062] Where m, n∈{1, 2}. Obviously, different from the original episodic training, we classify Q1 on S1 and S2 respectively, and do the same for Q2. Formula (10) minimizes the difference between two views of the same category instance. The whole process is as follows Figure 2Therefore, we compute the classification loss across views.

[0063] (14) Global self-supervised contrast loss. This loss aims to enhance the similarity between different views of the same image, while reducing the similarity between different image views. Specifically, we start from the meta-training set D train Randomly select a batch of samples And apply two data augmentation methods to generate enhanced batches here, and Represents x i Two different perspectives of are considered as a positive sample pair. We define f φ is a feature extractor with learnable parameters φ, which is used to transform samples Convert to feature map And after the global average pooling (GAP) layer, the global features are obtained We implement the projection head proj(·) using an MLP with one hidden layer to generate the projection vector Then, the global self-supervised contrastive loss is calculated:

[0064]

[0065] where · represents the inner product after L2 regularization, τ1 is a scalar temperature parameter, is an indicator function. Here, the positive sample pair z′ i and z i is from the same sample x i Extracted from an enhanced version of .

Claims

1. A dual-branch small sample image classification method based on contrastive learning, characterized by: The following steps are involved: (1) Prepare small sample classification datasets, including miniImageNet, tieredImageNet, and CUB datasets. (2) Divide the support set and query set in the dataset to generate meta-training and meta-testing tasks. (3) The image is processed through a dual-branch model with shared parameters of the feature extractor, one branch is used to extract global features and the other branch is used to extract local features. (4) Apply the saliency occlusion hybrid data enhancement method to the support set images to expand the support set samples. (5) Random cropping technology is applied to the query set images, and high-quality samples for comparative learning are screened out through the confidence cropping selector. (6) Use saliency mixing technology to fuse images, generate mixed labels for the images, and calculate the classification loss. (7) The category prototypes in the support set are processed through a multi-head attention mechanism to achieve prototype alignment. (8) Enhanced versions of the same image from different views are aligned through a global self-supervised contrastive loss to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. (9) Combining global and local features, the training model’s generalization ability in few-shot classification tasks. (10) Verify and evaluate the classification effect of the model on the test set.

2. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (1) specifically includes: pre-defining miniImageNet, tieredImageNet and CUB datasets, and constructing an M-way K-shot meta-training task by dividing the support set and the query set. The support set is used for intra-class relationship learning of the model, and the query set is used for testing classification performance, and the division needs to ensure that the data categories are evenly distributed.

3. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (2) is specifically as follows: the support set and the query set are divided by random sampling technology, K instances are randomly sampled from M classes for each class as the support set, and Q instances are sampled from the remaining instances as the query set for simulating the evaluation task.

4. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (3) is specifically as follows: processing the image through a dual-branch model with shared parameters of the feature extractor, one branch focusing on global feature extraction to obtain the overall representation of the image, and the other branch focusing on local feature extraction to capture image detail information. Global features focus on the overall semantic consistency between images, while local features increase the sensitivity to key areas.

5. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (4) is specifically as follows: applying a saliency occlusion mixed data enhancement method to the support set images, dividing the images into salient areas and non-salient areas, randomly occluding some areas and fusing them to generate contrast samples to enhance the effect of contrast learning; the salient areas are preferentially retained to ensure that the core information is not lost, and at the same time, the randomness of the mixture increases the adaptability of the model to environmental changes.

6. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (5) specifically includes: applying random cropping technology to the query set images, and ranking the cropped samples by confidence using the confidence cropping selector (CCS) method, removing the samples with the highest and lowest confidence, and retaining only the samples in the middle for comparative learning.

7. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (6) is specifically as follows: the saliency mixing technique generates new training samples by weighted fusion of salient areas and non-salient areas of the image, and calculates mixed labels based on the mixing weights and the ratio of the mixed areas for classification loss calculation.

8. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (7) is specifically as follows: the multi-head attention mechanism aligns the category prototypes in the support set, so that the prototypes of the same category in different perspective support sets are more consistent, thereby improving the generalization ability of the model in the few-sample classification task.

9. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (8) is specifically as follows: the global self-supervised contrast loss aligns enhanced versions of the same image from different perspectives, maximizes the similarity between them, and reduces the similarity between different image perspectives, thereby enhancing the global recognition ability of the model.

10. The dual-branch small sample image classification method based on contrastive learning according to claim 1, characterized in that: The step (9) is specifically as follows: by combining global and local features, the robustness and generalization performance of the comprehensive training model in small sample learning tasks are evaluated, and the classification effect is verified and compared on the test set.

Images