A multi-label few-shot image classification method based on label embedding

Abstract

The present invention relates to a multi-label few-sample image classification method based on label embedding, which processes the mapping problem of multi-label few-sample class visual features to labels through a distillation operation similar to that in a chemical process. The method uses a prototype generation module to generate class semantic prototypes based on word embeddings and projects them into a visual-semantic joint feature space. In the joint feature space, the self-attention mechanism and gradient reflux mechanism in the feature distillation module are used to complete the visual-semantic aggregate feature distillation, and the label relationship provided by the word embedding is modeled to optimize the discriminability of the generated semantic prototype and its ability to guide visual features, thereby effectively utilizing label embedding as an information supplement in multi-label few-sample image classification tasks. Experiments have shown that the method has good classification performance and generalization ability for unseen classes, and can handle the two major problems of mapping between class features and labels and multi-label relationship modeling in multi-label few-sample image classification tasks.

Description

Technical Field

[0001] The present invention relates to an image classification method, and in particular to a multi-label and few-sample image classification method based on label embedding. Background Art

[0002] An important task in computer vision is multi-label image classification (MLIC). Unlike single-label image classification, where an image is associated with only one label, multi-label image classification requires assigning multiple labels to an image. This is a more challenging task, as it involves not only identifying multiple objects or attributes in an image but also understanding the relationships between them.

[0003] In real-world applications, multi-label classification tasks often face the severe challenge of long-tail label distributions, due to the varying difficulty of acquiring data for different categories in multi-label images and the co-occurrence dependencies between labels. For tail categories, using loss functions like ASL to suppress overfitting to simple negative classes does not fundamentally address the feature-label mapping problem in classifying tail positive labels. In this context, multi-label few-shot classification tasks have attracted increasing attention. This task aims to perform multi-label image classification with only a small number of positive training examples containing the target class.

[0004] Different from the traditional single-label few-shot image classification task, the multi-label few-shot task cannot simply extract the class prototype by dividing the support set, nor can it simply measure the distance from the query set image global features to the visual prototype. Figure 4 As shown in the first column of the figure, for single-label images, each image contains only one subject and is assigned a single label. Therefore, the features extracted by the network can be assumed to correspond to the visual features of that label. The average features of a small number of images can provide a general representation of the visual features of a particular class label. In contrast, for multi-label, few-shot image classification in the second column of the figure, the subjects in a single image may be diverse objects, scenes, or attributes, and the class-specific features cannot be described by averaging features from multiple images. This mixing of class features makes it difficult for multi-label models to construct class prototypes from the support set. The network must be able to locate and extract specific features of different target classes from a large number of features. Furthermore, query images are often not of the same class as those in the support set. Therefore, it is impossible to determine all labels in the query image by directly measuring visual feature distances with support set images. Furthermore, in the one-shot case, with only one sample image with multiple labels, these labels cannot be distinguished by the network without additional information, as the mapping from class-level features to labels cannot be directly obtained in the few-shot case.

[0005] Furthermore, learning representations in the few-shot case is extremely difficult. Networks often suffer from severe overfitting to a small number of training examples, losing their ability to generalize to the query set. This requires that the network not overemphasize mapping to the class space, but rather discriminate labels in the feature space. In fact, many metric learning methods in the few-shot domain adhere to this principle.

[0006] In addition to adhering to general paradigms in the few-shot domain, multi-label, few-shot image classification still requires good feature representation and the ability to model label relationships. In recent years, much of the research on multi-label, few-shot image classification has focused on these two aspects. Researchers have proposed work based on data augmentation, co-occurrence relationship modeling, prototype networks, and meta-learning. However, the feature extraction and label modeling methods used in these works lack a focus on both multi-label, few-shot feature extraction and label relationship modeling, resulting in suboptimal results.

[0007] Based on this, establishing the mapping relationship from class-level visual features to labels, modeling label relationships, and improving the expressiveness and generalization capabilities of visual networks for multi-label image classification in the case of few samples are the difficulties in solving the multi-label few-sample classification problem. Summary of the Invention

[0008] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to train a model with multiple labels and few samples to accurately perform image classification.

[0009] To solve the above technical problems, the present invention adopts the following technical solution: a multi-label and small-sample image classification method based on label embedding, comprising the following steps:

[0010] S1: Get a multi-label image dataset Its label space Will and Divide the image samples into disjoint and There are two parts, called Base Set and Novel Set. The categories of images in only include {c1,c2,…,c M}, this part of the data class is used as the visible class in pre-training, and the data with labels {(X B ,Y B )} is sufficient, for each The image x∈X in B , label y=[y(1), y(2),…,y(M)]∈Y B , for images that contain Label c in i , y(i)=1, otherwise it is 0.

[0011] The images in the category include N unseen class labels in, satisfying the total category space and Following the setting of few samples, several samplings will be performed. Each sampling is called an episode, which will be Sample a set of support sets with training annotations, symbolized as satisfy All tags in appears K times in .

[0012] S2: Construct a multi-label few-sample prototype distillation network RC-Trans, the RC-Trans model Including in order:

[0013] Used to calculate input image features Convolutional neural networks;

[0014] Used to obtain the semantic prototype of the class Class semantic prototype generation module;

[0015] Used to and Semantic feature splicing module for splicing, Represents the input of the feature distillation network;

[0016] Feature distillation module.

[0017] S3: Training RC-Trans

[0018] use The samples in the pre-training are used to pre-train RC-Trans, and the network model with parameters obtained after pre-training optimization is obtained.

[0019] described exist Fine-tune on the support set to obtain the fine-tuned network model

[0020] S4: Input a picture to be predicted into the fine-tuned network model In , each element of the output vector is the probability that the image to be predicted contains the corresponding label.

[0021] As an example, the class semantic prototype generation module in S2 obtains the class semantic prototype The process is:

[0022]

[0023] in, represents the label embedding of the input, W1, W2, b1 and b2 are the parameters of the two linear layers, and g represents the nonlinear activation function. m×t , W2∈R t×d , where t is set to t=τ×d in the present invention, and τ is an expansion coefficient that controls the hidden layer dimension of the prototype nonlinear extraction unit.

[0024] Preferably, the feature distillation module is composed of a Transformer encoder and a reflow mechanism during its training; the feature distillation module uses a self-attention mechanism to guide the aggregation of local visual features through class semantic prototypes while retaining rich label relationship information in the class semantic prototypes;

[0025]

[0026]

[0027]

[0028] in is an item in the input sequence. RC-Trans output feature matrix have The output of the feature distillation module is Part of attention As weights, we consider aggregating all input features including visual features and semantic features.

[0029] W q , W k , W v , W r , W o Both represent the weights of a single-layer linear network layer, p i represents the i-th semantic prototype generated by the semantic prototype deep layer module; represents the i-th semantic prototype after self-attention weighted calculation; p i ′ represents the i-th semantic prototype after the feature extraction and calculation of the feedforward layer.

[0030] The process of obtaining probability predictions for each class is:

[0031] p i ′∈R d It is p i ∈R dThe probability prediction for the i-th class obtained by guiding the aggregation of visual-label relationship features can be calculated by the following formula:

[0032]

[0033] Where i∈[1,k], ψ is the similarity measurement function.

[0034] The process of the reflux mechanism is:

[0035] In the Base Set pre-training phase and the Novel Set fine-tuning phase of RC-Trans, the loss function L is used to extract the nonlinear unit. The optimization of the parameter θ is carried out in two directions, namely, through f ω Or go directly to f θ :

[0036]

[0037] And through f ω Transformer needs to be completed on the path Optimization; directly reach f θ The path is directly from Therefore, the gradient can directly act on the θ parameter to optimize the discriminability of the class semantic prototype.

[0038] Preferably, the process of pre-training RC-Trans in S3 is as follows:

[0039] For the input image x i , assuming that the label probability set predicted by RC-Trans is Its true value label is Using ASL asymmetric loss function L ASL Perform pre-training:

[0040]

[0041] Among them, k is and Different, the two datasets are and is the confidence after ASL hard threshold processing, expressed as where γ + ≥0 and γ - ≥0 are the positive and negative sample attention hyperparameters defined in ASL, and μ is a threshold used to filter out negative samples with low confidence.

[0042] according to The value of RC-Trans is updated using the reflux mechanism to reach the number of training rounds N. T When pre-training ends:

[0043]

[0044] in The parameters in the network are The network model with parameters obtained after pre-training optimization.

[0045] As a preference, in said S3 exist The process of fine-tuning on the support set is as follows:

[0046] Give the network semantic consistency constraints:

[0047]

[0048] Where ρ is the normalization operation,

[0049] Loss function during fine-tuning as follows:

[0050]

[0051] Where α represents the hyperparameter that balances the two types of losses.

[0052] according to The value of the reflux mechanism is used to Update the parameters in the training round to reach N s When fine-tuning ends:

[0053]

[0054] Compared with the prior art, the present invention has at least the following advantages:

[0055] This invention has conducted an in-depth exploration of the feature mapping principle of multi-label and few samples. The proposed method based on label embedding and feature distillation can better complete the label set feature extraction and feature-label mapping of multi-label images, and performs better in classification than the existing technology. Experiments show that compared with the experimental results of the best method currently available, the mean average precision (mAP) index has achieved an absolute improvement of 3.7% and 2.5% in 1-shot and 5-shot settings, respectively. At the same time, the ablation experiment of the components can effectively verify the effectiveness of the proposed feature mapping principle of multi-label and few samples and the designed distillation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a simplified flow chart of the method of the present invention.

[0057] Figure 2 Schematic diagram of the multi-label few-sample feature mapping method.

[0058] Figure 3 Schematic diagram of the characteristic distillation module.

[0059] Figure 4 Single-label few-shot image classification vs multi-label few-shot image classification.

[0060] Figure 5 Schematic diagram of the self-attention mechanism. DETAILED DESCRIPTION

[0061] The present invention is described in further detail below.

[0062] The multi-label few-sample image classification task requires passing Learn a visual model Due to different categories, the visual model It should contain rich “meta-knowledge” to achieve rapid knowledge transfer The purpose of class migration. The model is obtained by iterative learning of a small amount of data with N categories and K samples per category (K = {1, 5}) Able to complete The overall generalization completes the episode Classification of query sets By measuring the classification evaluation indicators of multiple episodes, the overall classification effect of the few-shot model is evaluated. Different from the single-label few-shot task, since each sample has more than one label and there is a dependency relationship between the labels, the The number of images in a dataset is often smaller than N×K.

[0063] When completed After the training, The semantic prototype generated after the label embedding input can guide Since the label relationship in the corpus is not independent of the label relationship in the visual image, the guided feature can approximate the By fine-tuning on the support set, RC-Trans can quickly generalize to At the same time, the self-attention mechanism can capture the label relationship modeling contained in the class semantic prototype, providing label relationship support for multi-label and few-sample image classification, thereby achieving better classification results.

[0064] A multi-label few-sample image classification method based on label embedding includes the following steps:

[0065] S1: Get a multi-label image dataset Its label space Will and Divide the image samples into disjoint and There are two parts, called Base Set and Novel Set. The categories of images in only include There are M categories in this data, which are used as visible categories in pre-training. The data with labels {(X B ,Y B )} is sufficient, for each The image x∈X in B , label y=[y(1), y(2),…,y(M)]∈Y B , for images that contain Label c in i , y(i)=1, otherwise it is 0. The data in is further divided into training set and validation set

[0066] The images in the category include N unseen class labels in, satisfying the total category space and Following the setting of few samples, several samplings will be performed. Each sampling is called an episode, which will be Sample a set of support sets with training annotations, symbolized as satisfy All tags in Appears K times; at the same time, for each episode, sample enough additional data As the query set after training with the few-sample data of this support set, it is used for testing the few-sample training.

[0067] S2: Construct a multi-label few-sample prototype distillation network RC-Trans (ReCtification Transformer), the RC-Trans model Including in order:

[0068] Used to calculate input image features Convolutional neural network, In the pre-training stage, X = X T , in the fine-tuning stage X=X S ;

[0069] Used to obtain the semantic prototype of the class The semantic prototype generation module of the class is In the fine-tuning stage

[0070] Used to and Semantic feature splicing module for splicing, Represents the input of the feature distillation network.

[0071] Feature distillation module;

[0072] S3: Training RC-Trans

[0073] use The samples in the pre-training are used to pre-train RC-Trans, and the network model with parameters obtained after pre-training optimization is obtained.

[0074] described exist Fine-tune on the support set to obtain the fine-tuned network model

[0075] S4: Input a picture to be predicted into the fine-tuned network model In , each element of the output vector is the probability that the image to be predicted contains the corresponding label.

[0076] Specifically, the class semantic prototype generation module is used to model label embedding as a class semantic prototype, so that label relations and semantic information can be better used by RC-Trans;

[0077] Utilize a mapping from a semantic space to a joint visual-semantic space R k×m →R k×d , embed the m-dimensional label into Mapping to d-dimensional semantic prototype Where k∈{M,N}, R represents the real number space, k represents the number of categories, which may be or It depends on whether it is during pre-training or fine-tuning. m represents the dimension of the label embedding, which is related to the word embedding model. d represents the dimension of the joint space.

[0078] The class semantic prototype generation module in S2 obtains the class semantic prototype The process is:

[0079]

[0080] in, represents the label embedding of the input, W1, W2, b1 and b2 are the parameters of the two linear layers, and g represents the nonlinear activation function. m×t , W2∈R t×d , where t is set as t=τ×d in the present invention, and τ is an expansion coefficient that controls the hidden dimension of the prototype nonlinear extraction unit. By adopting a nonlinear semantic mapping module, the quality of the extracted semantic prototype can be effectively improved.

[0081] For the multi-label and few-sample prototype distillation network RC-Trans proposed in the present invention, since the class semantic prototype is both the key to guiding the Transformer encoder to complete the visual feature aggregation, and the classifier of the class-specific features after completing the visual feature aggregation. Therefore, the prototype generation network is required to have good nonlinear projection capabilities to establish a mapping relationship between label embedding and the visual-semantic joint space; at the same time, the module should be simple enough to ensure excellent generalization performance; in addition, the module must be calculated and classified in the feature space to maintain the ability to fit the sample feature space rather than the label space, so the module should be independent of the category. Graph neural networks that are strongly related to label categories, such as graph convolutional neural networks, require the network to perform calculations on a graph established by the relationship between labels. Its fitting process will be strongly correlated with the label space, which will reduce the generalization ability of the model.

[0082] The present invention designs a semantic prototype nonlinear extraction unit Inspired by the FeedForward module in the Transformer, the FeedForward module provides a simple yet powerful approach to modeling relationships between input sequence features. This design, with two fully connected layers and a nonlinear activation function, better characterizes the nonlinear interactions between semantic and visual feature vectors, mapping the semantic space to the joint visual-semantic space and extracting preliminary semantic-like prototypes without introducing bias into the input label space.

[0083] Specifically, the present invention designs a feature distillation module. The visual features extracted by the traditional Transformer-based image classification method are often projected into the label space through a linear layer. Different from this, for the case of multiple labels and few samples, the present invention uses the class semantic prototype as part of the input of the Transformer encoder, and then measures the similarity of the encoder output features with the input class semantic prototype to complete the classification in the feature space. The present invention calls this setting the aggregation feature distillation process, which is similar to the distillation operation in the chemical method. On the one hand, after the visual-semantic features are aggregated, the classification loss gradient can be directly passed back to the class prototype generation module to improve the quality of the generated class prototype; on the other hand, in the next round of iteration, the optimized class semantic prototype will better guide the aggregation of visual-semantic features, thereby forming a virtuous circle.

[0084] This process can be done through Figure 3 Indicates that the semantic prototype and the visual-label relationship aggregation features are the distillation products of the bottom and top layers of the feature distillation network and are cyclically optimized during the training of RC-Trans. Specifically, the local visual features of the Transformer encoder input are optimized by f cnn (x i ) obtain, x i Represents the i-th image in the current data. Following the hybrid network design in ViT, the last pooling layer of the convolutional neural network used for feature extraction is removed, and the output features are flattened in the width w and height h dimensions as the local features of the image input to the aggregation feature distillation module. This feature is passed through a projection module Projected into the joint visual-semantic space, it is represented as Where v = w × h. At the same time, k d-dimensional prototypes Semantic prototype nonlinear extraction unit Get its input is the label embedding matrix. and Connect as input to the feature distillation network Participate in the Transformer encoder Calculation. Where k∈{M,N} is the number of categories in the current BaseSet or Novel Set. In the propagation of self-attention, each class semantic prototype embedding will fully consider its own attention to all local visual features and other semantic prototypes. Therefore, in addition to the aggregation of visual features, the model can also implicitly model the label relationship. The multi-head self-attention mechanism is used in the specific calculation. Since only the semantic prototype propagation part is taken at the end, only the Some of the feature updates can be described as:

[0085] The feature distillation module consists of a Transformer encoder and a reflux mechanism during its training; the feature distillation module uses a self-attention mechanism to guide the aggregation of local visual features through class semantic prototypes while retaining the rich label relationship information in the class semantic prototypes;

[0086] where f ω (·) is a Transformer encoder model based on multi-head self-attention. Represents the local visual features of the input image, which are obtained by the convolutional neural network f cnn (x i ) obtain, x i represents the i-th image in the current data, Represents the encoder aggregate feature output, which is used as the feature to be classified;

[0087] Measure the similarity with the class semantic prototypes of k classes respectively to obtain the probability prediction for each class

[0088] The main feature of the feature distillation network proposed in this paper is to measure the similarity between the output of the Transformer encoder and its input. During training, the back propagation gradient of the classification error can be directly returned to

[0089]

[0090]

[0091]

[0092] in is an item in the input sequence. RC-Trans output feature matrix have The output of the characteristic distillation module is Part of attention To consider weights, all input features, including visual and semantic features, are aggregated. To reduce network complexity and minimize model overfitting, encoders are no longer stacked, and only one layer of Transformer encoder is used.

[0093] W q , W k , W v , W r , W o Both represent the weights of a single-layer linear network layer, which is a matrix used to change the dimension of the input feature or extract features. q , Wk , W v The three parts are used to calculate the Q (query), K (key), and V (value) matrices of the Transformer self-attention (this part is explained in detail below), W r , W o are the parameters of the two fully connected layers (linear layers) of the feed forward layer after the self-attention calculation.

[0094] p i is the i-th semantic prototype generated by the semantic prototype deep module.

[0095] is the i-th semantic prototype after self-attention weighted calculation.

[0096] p′ i is the i-th semantic prototype after the feature extraction and calculation of the feed forward layer (i.e., the output of the Transformer part). The process of obtaining the probability prediction of each class in S1 is:

[0097] p′ i ∈R d It is p i ∈R d The probability prediction for the i-th class obtained by guiding the aggregation of visual-label relationship features can be calculated by the following formula:

[0098]

[0099] Where i∈[1,k], ψ is the similarity measurement function;

[0100] The process of the reflux mechanism is:

[0101] In the Base Set pre-training phase and the Novel Set fine-tuning phase of RC-Trans, the loss function L is used to extract the nonlinear unit. The optimization of the parameter θ is carried out in two directions, namely, through f ω Or go directly to f θ :

[0102]

[0103] And through f ω Transformer needs to be completed on the path Optimize to improve feature aggregation capabilities; directly reach f θ The path is directly from The gradient is generated, so it can directly act on the θ parameter to optimize the discriminability of the class semantic prototype, decoupling the optimization process of the class semantic prototype as a classifier and the visual feature guidance. Because the classification probability is calculated using a metric, the optimized prototype will move closer to the visual features that produce high response and other class semantic prototypes. This allows the network to successfully aggregate visual and semantic features, and the distribution of the generated prototypes in the joint space is further optimized.

[0104] Specifically, the process of pre-training RC-Trans in S3 is as follows:

[0105] For the input image x i , assuming that the label probability set predicted by RC-Trans is Its true value label is Using ASL asymmetric loss function L ASL Perform pre-training:

[0106]

[0107] Among them, i for and Different, the two datasets are and is the ASL hard threshold, expressed as where γ + ≥0 and γ - ≥0 are the positive and negative sample attention hyperparameters defined in ASL, and μ is a threshold used to filter out negative samples with low confidence.

[0108] according to The value of RC-Trans is updated using the reflux mechanism to reach the number of training rounds N. T When pre-training ends:

[0109] Network in Pre-training is performed on Optimize the method of visual feature extraction in feature space and mapping from visual and semantic space to the joint space.

[0110]

[0111] in The parameters in the network are The network model with parameters obtained after pre-training optimization.

[0112] Specifically, in few-shot inference, insufficient sample size can lead to overfitting during model fine-tuning, resulting in significant discrepancies between learned features and the true distribution. This paper proposes a semantic consistency constraint based on the vector cosine distance metric to avoid overfitting when fine-tuning the network on a novel set.

[0113] The S3 exist The process of fine-tuning on the support set is as follows:

[0114] In order to not destroy the relevance of semantic prototypes in the case of few samples, the network semantic consistency constraint is given:

[0115]

[0116] Where ρ is the normalization operation, The L sem The generated label-class semantic prototype is restricted to maintain a cosine distance relationship close to the original embedding during training, so as to maintain the label relationship modeling of the class semantic prototype generation module during fine-tuning.

[0117] Loss function during fine-tuning as follows:

[0118]

[0119] Where α represents a hyperparameter that balances the two types of losses. In this invention, it is set to 1, and the α parameter controls the ratio of the two losses. The value of the reflux mechanism is used to Update the parameters in the training round to reach N s When fine-tuning ends:

[0120] Will further Fine-tuning is performed on the support set of , using a combination of classification loss and semantic consistency constraints

[0121]

[0122] Following the general method of training a few-sample classification model, the model proposed in this invention is Supports centralized continuous sampling of samples for feature extraction cnn (·), Transformer encoder f ω (·) and prototype generation module f θSince all training is based on the sample feature space, the model does not introduce direct projection and fitting to the label space, so it can learn the general sample visual-label relationship feature aggregation guided by the class prototype. Combined with the design of the multi-label and few-sample class-level feature mapping method, the method proposed in this paper can be well generalized to Specifically, the training process of RC-Trans proposed in the present invention is shown in Table 1:

[0123] Table 1 RC-Trans network training pseudocode

[0124]

[0125]

[0126] Unlike the metric-based methods in few-shot image classification, the present invention does not generate offline class representations or class prototypes through the support set, but only guides the self-attention mechanism through semantic prototypes to complete the online extraction of class features in the input image.

[0127] The attention mechanism generally has three related quantities: Q, K, and V. For each element involved in the attention calculation, Q represents a query. The attention weight is calculated by the correlation between the query and the key K. This weight matrix is ​​then multiplied by the value V corresponding to the key K to obtain the weighted output of the element's attention, which is organically integrated with the output of other elements.

[0128] The Transformer network describes a type of attention mechanism processing method that is different from the sequence-to-sequence task, called the self-attention mechanism. Figure 5 In the attention mechanism for sequence-to-sequence problems, queries and keys are often generated from different sequences. For example, the encoder sequence generates keys and values, while the decoder sequence generates queries. Self-attention, as the name suggests, generates queries, keys, and values ​​from a single sequence. Self-attention describes the attention relationship between elements within a feature sequence. By focusing on features at different positions in the sequence, the self-attention mechanism can learn to capture long-term dependencies between different elements. For example, the self-attention mechanism can learn to capture the spatial relationship between different regions in an image.

[0129] In the Transformer network, the query Q, key K, and value V exist in the form of a matrix, which is formed by combining the feature vector of each element through the attention vectors obtained by projecting them through three fully connected layers. The parameters of the fully connected layer are represented as W q , W k and W v , assuming that there is a feature sequence H = {h1,h2,…h n}, the attention weight is calculated by taking the query vector q of the sequence element i =W q h i With the key vector k j =W k h j Perform a scaled dot product operation and then normalize the result using the softmax function, i.e.:

[0130]

[0131] where h i ,h j ∈H, d represents h i ,h j The same feature dimension, α ij is the element in row i and column j of the attention matrix A, which represents the attention weight of the i-th position to the j-th position. By dividing by the square root of d, the variance of the weight can be effectively controlled. Thus, the content vector of the i-th position It is also obtained by the weighted sum of the values ​​of all positions by the attention weight A, and the formula is expressed as:

[0132]

[0133] The Transformer network also introduces a feedforward network to further process the extracted information. It consists of two linear layers and a nonlinear activation function. The output of the first linear layer passes through the activation function and then passes to the second linear layer. The input and output of the feedforward unit have the same size, with the aim of providing a simple and powerful method to model the nonlinear interactions between features in the input sequence. It allows the model to capture the complex dependencies between different parts of the input sequence, which is crucial for modeling the relationship between label-label and label-image in the multi-label image classification task using the Transformer network. It can be expressed using the formula:

[0134]

[0135] Where f is a nonlinear activation function, b1, b2 is the parameter of the corresponding two fully connected layers. i ′ is used as the input of the next layer.

[0136] At the same time, Transformer has designed a multi-head attention mechanism, in which the attention vector will be assigned to different attention heads after being amplified in the channel dimension. Each attention head will independently perform a self-attention process, and then project the merged channels back to the number of input channels to maintain consistency in the input and output dimensions.

[0137] In addition to multi-head attention and feedforward units, the Transformer also uses residual connections to prevent gradient loss or explosion, and uses layer normalization (LN) after the residual to ensure stability in the data feature distribution and accelerate model convergence. Unlike batch normalization (BN) commonly used in convolutional neural networks, which normalizes the same channel in each batch of data, layer normalization normalizes the different channels of each sample, using the formula:

[0138]

[0139] Among them, g (gain) and b (bias) are parameters used to ensure that the normalization operation does not destroy the original information; i is the value of each sequence data at the i-th position in the Batch; μ L and are the mean and variance of the channel direction at that position respectively.

[0140] The Transformer network employs an encoder-decoder architecture specifically designed for sequence-to-sequence problems in natural language. After the input sequence passes through multiple Transformer encoder iterations, the resulting hidden state vector is projected into key and value matrices through a linear layer, which serve as the input to the multi-head attention mechanism in all decoders. Because the key-value matrix comes from the encoder's hidden state, and the query matrix comes from another decoder input sequence, the attention mechanism in the decoder is also known as a cross-attention mechanism. After several decoding cycles, the final Transformer output sequence is obtained, with the same feature dimension and number as the input.

[0141] Experiment and analysis

[0142] 1. Dataset Setup

[0143] ①MS-COCO-FSL: The method proposed in this paper is mainly tested on the MS-COCO dataset.

[0144] In order to maintain the comparability of the test on the dataset, this paper uses the multi-label few-sample MS-COCO-FSL dataset partition provided by LaSO. In the specific setting, the 80-class label set is divided into 64 visible classes and 16 invisible classes, that is, and The training set and test set of the original MS-COCO dataset are divided into and Two parts. Among them, Training set and There is no overlap in the support sets used for fine-tuning. The support set of is composed of 10 samples. In each episode, each The classes in the support set appear only K∈{1,5} times. The LaSO setting relaxes this approach and tries to meet the requirement that each class appears only K times in the training set. Therefore, the number of images obtained is often less than N×K. and The query sets in both are all the data of the original MS-COCO test set, but the two only consider or This results in two settings: 16-way 1-shot and 16-way 5-shot. The labels of the 16 invisible classes are: bicycle, boat, stop sign, bird, backpack, frisbee, snowboard, surfboard, cup, fork, spoon, broccoli, chair, keyboard, microwave, vase.

[0145] ②NUS-WIDE-FSL: This paper proposes a new multi-label few-sample baseline data partitioning based on NUS-WIDE. In this dataset, 81 categories of data are divided into and The sampling follows the design method of LaSO small sample data set partitioning, using a histogram to maintain the sampling of small sample data. The classes in only appear K∈{1,5} times in the support set. Different from the LaSO setting, the NUS-WIDE-FSL partition proposed in this paper does not use all NUS-WIDE test sets as The test set and The query set, but excludes the sampling The test set and All labels in the query set are negative images, so that and There are no longer any overlapping images, making this more suitable for a few-shot dataset. For the NUS-WIDE-FSL dataset, two similar settings were used: 16-way 1-shot and 16-way 5-shot. The 16 randomly sampled unseen class labels are: boats, cityscape, dancing, frost, house, map, moon, police, reflection, rocks, sand, sun, sunset, swimmers, tattoo, and tower. Table 2 provides the specific number of images in the two dataset partitions.

[0146] Table 2 Comparison of data volume of multi-label and few-sample datasets

[0147]

[0148] 2. Experimental Setup

[0149] The present invention uses ResNet-101 as the backbone network for feature extraction on the two public datasets mentioned above. In the pre-training based on the Base Set dataset, the proposed method uses a 448×448 image resolution and the RandAugment data augmentation method. This method randomly samples from 15 basic data augmentations including translation, rotation, contrast, brightness, etc. to enhance the input image. In terms of optimization, the method uses the Adam (Adaptive Moment Estimation) optimizer with parameter weight decay, and the decay rate is 10 -4 . When the batch size is 64, the training -4 The learning rate is optimized, and the ASL hyperparameter γ + , γ - The loss weight hyperparameter α was set to 0, 4, and 1, respectively. 20 epochs of training were performed on the base set. For validation on the novel set, the backbone network parameters were fixed and fine-tuned for 80 epochs on the support set, followed by inference on the query set. All experimental results below are averages of 10 episodes. Early stopping was used during training to prevent overfitting.

[0150] In the RC-Trans model setup, unless otherwise specified, the label embeddings used are 300-dimensional word embeddings extracted using the GloVe model. The Transformer encoder's hidden layer dimension parameter d is set to 2048, and the depth is set to 1. Unless otherwise specified, the number of attention heads is 8, each with a dimension of 512, and the feedforward layer hidden dimension is set to 2048. The activation function used in class prototype generation and the Transformer in this paper is the Gaussian Error Linear Unit (GELU). The loss function g is a combination of the ReLU function and the hyperbolic tangent function (tanh), which has shown good results in Transformer-related networks. Unless otherwise specified, the hidden layer dimension expansion coefficient τ of the class semantic prototype module in the experiment is generally set to 0.5.

[0151] Each experiment was conducted using a single GeForce RTX 3090 graphics card in the Ubuntu 18.04.5 operating system environment and an Intel(R) Core(TM) i9-10900X CPU. Unless otherwise stated, the best method in the experimental results is indicated in bold; in some experiments, the second-best method is indicated with Underline At the same time, for the sake of convenience, the description of the 16-way 1 / 5-shot setting in the result table header is omitted and abbreviated as 1-shot or 5-shot setting.

[0152] Table 3 Comparison of the effects of different methods on the MS-COCO dataset using the data partitioning proposed by LaSO

[0153]

[0154] 3. Public Dataset Experiments

[0155] To verify the effectiveness of the proposed method, we use the publicly available multi-label few-shot baseline of the LaSO method on the MS-COCO dataset as a comparison benchmark. Table 3 shows the performance of several mainstream multi-label few-shot image classification methods on the MS-COCO and NUS-WIDE datasets. In the Meta method, Proto, Relation, and LPN represent three basic few-shot learning baselines: the Prototypical Network baseline, the RelationNetwork baseline, and the Label Propagation baseline, respectively. NLC stands for the Neural Label Count method, which predicts the number of labels to assist in feature extraction. The WGAP method is an abbreviation of the method of Yan et al.

[0156] The experiments used mAP, the most popular metric for multi-label image classification, as the evaluation criteria. These metrics were also tested using the same dataset-based baseline. To ensure comparability with other methods, we also provide experimental results using GoogleNet-v3 and ResNet-101 as the backbone networks. Most multi-label, few-shot research uses these two networks as feature extraction networks.

[0157] Observations show that the proposed method significantly improves upon all previous methods, achieving optimal performance among all methods. Furthermore, in a 1-shot experiment, the mAP indicator improved by 2.5% compared to the suboptimal meta-learning method. In a 5-shot experiment, the performance of GoogleNet-v3, which has relatively lower feature extraction capabilities, also surpassed all other methods in the literature. The 5-shot experiment using ResNet-101 as the backbone network demonstrated a 3.7% improvement in mAP compared to the best-performing prototype network method in previous work. This demonstrates the adaptability and effectiveness of the proposed method for multi-label, few-shot image classification tasks.

[0158] Table 4 Precision and recall performance of the methods under different backbone networks

[0159]

[0160] Table 4 reports the model's performance in terms of precision and recall for the multi-label, few-shot classification task. The threshold for calculating precision and recall was set at 0.8. It can be seen that as the amount of support set data increases, the improvement in recall metrics is most significant. While GoogleNet-v3 outperforms ResNet-101 in precision, its recall lags significantly behind ResNet-101. Since existing work does not provide results on these metrics, comparisons with other work are not reported in this table.

[0161] This paper also presents the performance of the proposed method in multi-label, few-shot image classification on the NUS-WIDE dataset, as shown in Table 5. The bolded columns in the table represent the optimal metrics for the 5-shot experiment, while the underlined columns represent the optimal metrics for the 1-shot experiment. In the Method section, G. denotes the abbreviation for the GoogleNet-v3 backbone network, while R. denotes the abbreviation for the method using ResNet-101 as the backbone network. This paper is the first method to perform partitioning and experimentation on this dataset. Due to the greater complexity and noisy annotations of the NUS-WIDE dataset, it is more challenging than the MS-COCO dataset. The results show that using the ResNet-101 backbone network outperforms GoogleNet-v3 in all metrics, with the largest improvement in the threshold-independent mAP metric, exceeding GoogleNet-v3 by 3.49% and 2.2% in the 1-shot and 5-shot experiments, respectively. Overall, ResNet-101 is more suitable for multi-label, few-shot image classification tasks. It performs better than the network using GooleNet-v3 in balancing precision and recall. At the same time, the experimental results show that the proposed model is robust to noise and can handle multi-label few-sample image classification problems on different datasets.

[0162] Table 5 Performance of methods on the NUS-WIDE-FSL dataset

[0163]

[0164] 4. Evaluation of Word Embedding Methods

[0165] The quality of label embedding affects the effect of visual-semantic feature fusion and classifier. Different language models have different modeling angles for word embedding, so their applicability in multi-label and few-sample image classification is also different. In order to verify the effect of commonly used word embeddings in visual tasks on the distillation network structure of the present invention, the present invention selects the following static word embedding model for verification, in which the word embedding dimension is fixed to 300 dimensions:

[0166] Word2vec: A local corpus-based language model developed by Google. This paper uses the official pre-trained GoogleNews-vectors-negative300 model, which was trained on the Google News dataset (approximately 100 billion words) using a bag-of-words (BOW) approach and skip-gram architecture. The model contains 3 million words and phrases, each represented as a 300-dimensional vector.

[0167] Fasttext: An open-source language model developed by Facebook for training word embeddings and text classification. It uses a CBOW (Continuous Boolean Words)-like training structure. The pre-trained model used is wiki-news-300d-1M, which was trained on the Wikipedia 2017, UMBC webbase corpus, and the statmt.org news dataset (approximately 16 billion words in total) with 1 million word embeddings, each with 300 dimensions.

[0168] GloVe: A neural network-based matrix factorization method using a global co-occurrence matrix. This uses the official pre-trained model glove.42B.300d. This model trains 1.9 million 300-dimensional word embeddings on 42 billion words from the Common Crawl dataset. Because it is based on word co-occurrence statistics, it is the most widely used word embedding for multi-label image classification.

[0169] To verify the effectiveness of word embeddings, we also extracted random embeddings (Randn) of size C × 300 based on a standard normal distribution, where C represents the number of labels in the dataset. This embedding, which does not represent any semantic relationship between the labels, was used to compare the performance of the model after the semantics were introduced, verifying the effectiveness of the distillation network's method of guiding class prototype distillation from label embeddings. The overall experimental results are shown in Table 6. The experimental settings used were the same for all methods, with only the word embeddings introduced being different.

[0170] It can be seen that the introduction of label embedding is crucial to the effectiveness of the method. Among them, the best method is to introduce GloVe word embedding. Compared with the Randn method that does not introduce effective label relationships, in the 1-shot and 5-shot experimental settings, label embedding brings up to 7.9% and 3.3% mAP index improvements, respectively. This is consistent with the idea of ​​the method of the present invention, that is, in the 1-shot experiment, more auxiliary semantic and label relationship information is used for modeling and classification. Therefore, a 15.4% improvement is obtained in 1-shot compared to the method that does not use auxiliary information, which is more than three times the 4.8% improvement of 5-shot. At the same time, since GloVe word embedding is based on the decomposition of the statistical co-occurrence matrix, the experiment verifies that label co-occurrence dependency is still the most important label relationship in multi-label images.

[0171] Table 6 Results of introducing different word embeddings

[0172]

[0173]

[0174] The table also shows that the proposed method surpasses all previous work results listed in Table 3 in 5-shot experiments without using auxiliary information. This demonstrates that feature distillation methods that utilize self-attention to guide visual features and measure them in feature space can significantly surpass other work in feature extraction and generalization capabilities. Because the mapping of class-level features to labels cannot be accomplished using Rand embedding, adding label embedding as auxiliary information significantly improves the 1-shot experimental results.

[0175] 5. Ablation Experiment

[0176] Comparison of attention mechanisms: Table 7 compares the self-attention mechanism and the cross-attention mechanism. It can be seen that the use of the self-attention mechanism is better than the use of the cross-attention mechanism, exceeding the cross-attention mechanism by 2.5% and 2.3% in the mAP indicators of 1-shot and 5-shot respectively. The present invention attributes this effect to the fact that the self-attention mechanism is a visual-semantic feature aggregation guided by the semantic prototype, which fully considers the input of the visual feature sequence and the prototype feature sequence. Therefore, in addition to extracting the visual features in the calculation, the label relationship modeling implicit in the prototype is also extracted. When using the cross-attention mechanism, only the semantic prototype is used to guide the aggregation of visual features, and the utilization rate of the label relationship is insufficient, resulting in suboptimal results.

[0177] Table 7 Comparison between self-attention mechanism and cross-attention mechanism

[0178]

[0179] 6. Characteristic distillation:

[0180] The ablation experiments using different settings for feature distillation are shown in Table 8. In the methods used in this table, linear projection classification refers to the setting that does not use feature space metrics, but directly uses This method uses a binary classifier implemented with a fully connected layer to perform classification. In the global feature metric, the CNN output is passed through a pooling layer to generate a global feature, and the dot product similarity between the aggregated feature and the global feature is measured, similar to Yan's method. The local feature metric is the benchmark setting for RC-Trans. Furthermore, the present invention blocked the dual-path gradient in the experiment, and the results are also presented in the table.

[0181] Table 8 Ablation experiments with different settings in feature distillation

[0182]

[0183]

[0184] Experiments have found that the linear projection method has caused serious overfitting of the label space, and the effect on the 1-shot setting is the worst among all settings; and the method based on the measurement of global features and local features uses the aggregated features as the global feature classifier, which is not strong in feature classification ability. Aggregated features cannot provide good classification effects for global features. The linear projection and global feature measurement methods cannot decouple the optimization gradient of the classifier from the optimization gradient of the visual features, that is, the idea of ​​aggregated feature distillation cannot be used. Next, the present invention introduces a feature distillation method, but blocks the backpropagation of the gradient to verify the effect of the feature distillation method.

[0185] In the experimental setting of the third row in the table, the present invention blocks the gradient of the backflow, so that the gradient can only be passed back to the class semantic prototype generation module through the Transformer encoder; in the experimental setting of the fourth row, the gradient of the Transformer encoder back to the class semantic prototype generation module is blocked, so that the gradient can only be optimized by directly reaching the class semantic generation module. It can be seen that both blocking methods have brought about a decrease in the classification accuracy of the model, especially in the 1-shot experiment, the decrease is more obvious. It can be proved that through the setting of dual-path gradient distillation in feature distillation, in the continuous benign cycle distillation of the class prototype, the quality of the class prototype generation can be improved, and the class prototype's ability to guide the aggregation of visual-semantic features can be increased.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-label few-sample image classification method based on label embedding, characterized in that: The steps include: S1: Get a multi-label image dataset Its label space Will and Divide the image samples into disjoint and Two parts, called Base Set and Novel Set; The categories of images in only include There are M categories in this data, which are used as visible categories in pre-training. The data with labels {(X B ,Y B )} is sufficient, for each The image x∈X in B , label y=[y(1), y(2),…,y(M)]∈Y B , for images that contain Label c in i , y(i)=1, otherwise 0; The images in the category include N unseen class labels in, satisfying the total category space and Following the setting of few samples, several samplings will be performed; each sampling is called an episode, which will be Sample a set of support sets SupportSet with training annotations, symbolized as satisfy All tags in appears K times in S2: Construct a multi-label few-sample prototype distillation network RC-Trans, the RC-Trans model Including in order: Used to calculate input image features Convolutional neural networks; Used to obtain the semantic prototype of the class Class semantic prototype generation module; Used to and Semantic feature splicing module for splicing, Represents the input of the feature distillation network; Feature distillation module; S3: Training RC-Trans use The samples in the pre-training are used to pre-train RC-Trans, and the network model with parameters obtained after pre-training optimization is obtained. described exist Fine-tune on the support set to obtain the fine-tuned network model S4: Input a picture to be predicted into the fine-tuned network model In , each element of the output vector is the probability that the image to be predicted contains the corresponding label.

2. The multi-label, few-sample image classification method based on label embedding according to claim 1, characterized in that: The class semantic prototype generation module in S2 obtains the class semantic prototype The process is: in, represents the label embedding of the input, W1, W2, b1 and b2 are the parameters of the two linear layers, g represents the nonlinear activation function; for W1∈R m×t , W2∈R t×d , where t is set to t=σ×d in the present invention, and τ is an expansion coefficient that controls the hidden layer dimension of the prototype nonlinear extraction unit.

3. The multi-label, few-sample image classification method based on label embedding according to claim 2, characterized in that: The feature distillation module consists of a Transformer encoder and a reflow mechanism during its training. The feature distillation module uses a self-attention mechanism to guide the aggregation of local visual features through class semantic prototypes while retaining the rich label relationship information in the class semantic prototypes. in is an item in the input sequence; RC-Trans outputs a feature matrix have The output of the feature distillation module is Part of attention For weights, consider aggregating all input features including visual features and semantic features; W q , W k , W v , W r , W o Both represent the weights of a single-layer linear network layer, p i represents the i-th semantic prototype generated by the semantic prototype deep layer module; represents the i-th semantic prototype after self-attention weighted calculation; p i ′ Represents the i-th semantic prototype after the feature extraction and calculation of the feed forward layer; The process of obtaining probability predictions for each class is: p i ′ ∈R d It is p i ∈R d The probability prediction for the i-th class obtained by guiding the aggregation of visual-label relationship features can be calculated by the following formula: Where i∈[1,k], ψ is the similarity measurement function; The process of the reflux mechanism is: In the Base Set pre-training phase and the Novel Set fine-tuning phase of RC-Trans, the loss function L is used to extract the nonlinear unit. The optimization of the parameter θ is carried out in two directions, namely, through f ω Or go directly to f θ : And through f ω The path needs to be completed Optimization; directly reach f θ The path is directly from Therefore, the gradient can directly act on the θ parameter to optimize the discriminability of the class semantic prototype.

4. The multi-label, few-sample image classification method based on label embedding according to claim 3, characterized in that: The process of pre-training RC-Trans in S3 is as follows: For the input image x i , assuming that the label probability set predicted by RC-Trans is Its true value label is Using ASL asymmetric loss function L ASL Perform pre-training: Among them, k is and Different, the two datasets are and is the confidence after ASL hard threshold processing, expressed as where γ + ≥0 and γ - ≥0 are the positive and negative sample attention hyperparameters defined in ASL, and μ is a threshold used to filter out negative samples with low confidence; according to The value of RC-Trans is updated using the reflux mechanism to reach the number of training rounds N. T When pre-training ends: in The parameters in the network are The network model with parameters obtained after pre-training optimization.

5. The multi-label, few-sample image classification method based on label embedding according to claim 4, characterized in that: The S3 exist The process of fine-tuning on the support set is as follows: Give the network semantic consistency constraints: Where ρ is the normalization operation, Loss function during fine-tuning as follows: Where α represents the hyperparameter that balances the two types of losses; according to The value of the reflux mechanism is used to Update the parameters in the training round to reach N s When fine-tuning ends:

Images