Long-tail multi-label image classification method based on vision-language pre-training model
By replacing the softmax layer with the sigmoid layer in the CLIP model and building a long-tail multi-marker image classification model, the prediction deviation and label correlation construction incompleteness of the long-tail multi-marker problem in the existing technology are solved, and the multi-marker is significantly improved Accuracy of classification.
Patent Information
- Application Number
- CN202411076021.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-29
AI Technical Summary
When dealing with the long-tail multi-marking problem, the softmax layer is difficult to deal with multi-marking scenarios. The acquisition of text description information is time-consuming and may contain noise, which limits the effectiveness of the model in long-tail multi-marking learning scenarios, resulting in limited prediction performance. .
Replace the softmax layer in the traditional CLIP model with the sigmoid layer, build a long-tail multi-marked image classification model, including image encoder, text encoder and graph convolutional neural network module, and train it through the sigmoid layer and reweighted long-tail loss function. , using text embedding and graph convolution neural network to build label correlation, and improve the multi-label classification capability of the model.
It effectively solves the prediction bias of the model under the long-tail category distribution, explicitly constructs label correlation, improves the accuracy of multi-label classification, and enhances the prediction performance of tail categories.
Smart Images

Figure CN119048810A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of machine learning, multi-label learning and long-tail learning, and specifically relates to a long-tail multi-label image classification method based on a vision-language pre-training model. Background Art
[0002] In the long-tail multi-label problem, a sample can be associated with multiple labels at the same time, and the distribution of these labels is in the form of a long tail, that is, a few labels appear more frequently, while most labels appear less frequently. This type of problem may appear in all aspects of production and life. For example, in medical diagnosis, rare diseases have a small number of samples; in species classification, endangered animals have a small number of samples.
[0003] At present, many methods for solving the long-tail multi-label problem only involve the consideration of the features of the image itself. Usually, this task uses a convolutional neural network pre-trained on the ImageNet dataset as a feature extractor. In order to better utilize the semantic information from the text, the researchers, inspired by the visual-language pre-training model, proposed a loss function based on category-aware margin, which uses additional text description information to enhance the semantic relationship between categories to obtain more fine-grained category text cues, thereby assisting the learning of the visual model. Although the existing methods have shown certain effectiveness in dealing with the long-tail multi-label problem, they still have some limitations. On the one hand, they follow the model structure of CLIP and use the softmax layer to obtain the prediction results. However, softmax cannot handle the situation where a sample may have multiple labels at the same time in the multi-label scenario. On the other hand, for the construction of label correlation, the construction of semantic relations in the existing methods is incomplete. First, the introduced text description information is usually generated by an open source image-caption model or manually annotated, which is time-consuming and inevitably produces some noise. Secondly, corresponding to the dataset, the text description still retains the long-tail property, so that the model may not be able to capture all the subtle connections between various categories, which may limit the effectiveness of the model in other long-tail multi-label learning scenarios. Therefore, the prediction performance of existing methods is limited. Summary of the invention
[0004] Purpose of the invention: The present invention proposes a long-tail multi-label image classification method based on a vision-language pre-training model, which is used to process multi-label classification problems in which training data has a long-tail category distribution.
[0005] Technical solution: The long-tail multi-label image classification method based on a visual-language pre-training model described in the present invention comprises the following steps:
[0006] (1) Obtain multi-label sample data with real labels
[0007] (2) Replace the softmax layer in the traditional CLIP pre-training model with a sigmoid layer to adapt the traditional CLIP to multi-label tasks;
[0008] (3) constructing a long-tail multi-label image classification model and training it; the model includes an image encoder, a text encoder, and a graph convolutional neural network module;
[0009] (4) After the data enhancement strategy, the image data to be tested is input into the trained model for prediction, and finally the label prediction result is obtained to realize long-tail multi-label image classification.
[0010] Furthermore, the implementation process of step (1) is as follows:
[0011] Obtain multi-label sample data with true labels Where (xk, Yk) represents a sample label pair, is the label set corresponding to sample xk, and N represents the number of multi-label samples collected; the dataset presents a long-tail category distribution, that is, a small number of categories have a large number of sample data, while other categories are only associated with a small number of samples.
[0012] Furthermore, the implementation process of step (2) is as follows:
[0013] The given template "a photo of a[CLS]" prompt word is passed through a text encoder to obtain the text embedding of the corresponding category, and the cosine similarity between the text embeddings of each category is calculated to construct a neighbor graph, in which [CLS] is replaced by the class name; as the training process, all node features in the graph tend to be consistent, hyperparameters are introduced to reweight the neighbor graph, so that when updating node features, the node itself will be assigned a fixed weight, and the weight of the related node will be determined by the neighborhood distribution; after the nodes are normalized by softmax, the adjacency matrix corresponding to the graph is taken as the correlation matrix containing rich category semantic relationships.
[0014] Furthermore, the implementation process of step (3) is as follows:
[0015] S1: The correlation matrix A* and the category features output by the learnable soft prompts through the text encoder are used as the input of the graph convolutional neural network module to obtain semantic fusion features; the training image is used as the input of the image encoder to obtain image features; the soft prompt seat is used as the input of the text encoder to obtain text features;
[0016] S2: The semantic fusion feature is residually connected with the text feature, the cosine similarity is calculated with the image feature, and the prediction probability is obtained through sigmoid.
[0017] S3: The model is supervised by the heavily weighted long-tail loss function, and efficient parameter fine-tuning is applied to avoid overfitting problems.
[0018] Furthermore, the S1 implementation process is as follows:
[0019] The training image xk is fed into E I (xk) obtains the image feature Fx, which is prompted by the category text:
[0020]
[0021] The composition t is fed into E T (t) Get text embedding F t ; where c∈{1,…,C}, m∈{1,…,M}, CLS]token is replaced by a specific class name; [V] m represents a learnable word embedding, M is a hyperparameter that specifies the number of context tokens;
[0022] The graph convolutional neural network module will receive A * and F t As input, the semantic correspondence between labels is encapsulated into the output H L Features.
[0023] Furthermore, the S2 implementation process is as follows:
[0024] Text Embedding F t With H L Improved text embedding by performing residual connection The predicted probability is calculated:
[0025]
[0026] Among them, τ is the temperature parameter, σ is the sigmoid function, and sim(·,·) represents the cosine similarity.
[0027] Furthermore, the S3 implementation process is as follows:
[0028] The reweighting weight r of the loss function is based on the class frequency n c / NCalculated:
[0029]
[0030] Among them, α, β, and θ are adjustable hyperparameters used to adjust the weight distribution of each category; the loss function is designed as:
[0031]
[0032] in, Represents the label of category c, and predicts the probability of positive samples For negative sample prediction probability, The corresponding model output loigt,v c is the class deviation value, which is defined as:
[0033]
[0034] Among them, κ and ζ are adjustable hyperparameters used to adjust the model's prediction bias for each category; during the training process, the parameters of the text encoder are fixed, and the text prompt is a learnable soft prompt; the image encoder applies efficient parameter fine-tuning technology to learn a small part of task-specific parameters; the parameters of the graph convolutional neural network module are updated as the training progresses.
[0035] Furthermore, the implementation process of step (4) is as follows:
[0036] The image x is resized to (224+e)×(224+e) to obtain x'; the 224×224 part of the center of x' is cropped, denoted as xc; the 224×224 part of the upper left of x' is cropped, denoted as xtl; the 224×224 part of the upper right of x' is cropped, denoted as xtr; the 224×224 part of the lower left of x' is cropped, denoted as xbl; the 224×224 part of the lower right of x' is cropped, denoted as xbr; {xc, xtl, xtr, xbl, xbr} is evenly divided into m patches After the image passes through the long-tail multi-label image classification model, logit{zc,ztl,ztr,zbl,zbr} is obtained; the logit result is z=Average(zc+ztl+ztr+zbl+zbr); the predicted probability of the test image x is p=σ(z).
[0037] Furthermore, the [V]m has the same dimension as the regular word embeddings in the vocabulary.
[0038] Furthermore, the m is 196, and the size of each patch is 16×16.
[0039] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention solves the problem that the real application data is distributed in long-tail categories, which causes the model to tend to predict the tail category as the head category; and by utilizing the characteristic that an instance is associated with multiple labels, the label correlation is explicitly constructed from the text semantic level. Based on label association modeling, the model strives to improve the prediction probability of semantically related labels during training, so as to pass the rich information learned from the head class to the tail class, thereby improving the accuracy of multi-label classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is a flow chart of the present invention;
[0041] Figure 2 It is a flow chart of the present invention for learning a long-tail multi-label image classification model. DETAILED DESCRIPTION
[0042] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0043] like Figure 1 As shown, the present invention proposes a long-tail multi-label image classification method based on a vision-language pre-training model, and the specific implementation process is as follows:
[0044] Step 1: Obtain multi-label sample data with true labels Where (xk, Yk) represents a sample label pair, is the label set corresponding to sample xk, and N represents the number of multi-label samples collected. The dataset presents a long-tail category distribution, that is, a small number of categories have a large number of sample data, while other categories are only associated with a small number of samples.
[0045] Step 2: Based on the CLIP pre-trained model structure, the softmax layer in the model is replaced with a sigmoid layer to make the traditional CLIP framework adaptable to multi-label tasks.
[0046] The given template "a photo of a[CLS]" prompt word is passed through the text encoder to obtain the text embedding of the corresponding category, and the cosine similarity between the text embeddings of each category is calculated to construct a neighbor graph, where [CLS] is replaced by the class name. In order to avoid the problem of over-smoothing, that is, as the features of all nodes in the graph tend to be consistent during the training process, a hyperparameter is introduced to reweight the neighbor graph, so that when the node features are updated, the node itself will be assigned a fixed weight, and the weight of the related node will be determined by the neighborhood distribution. Finally, after the nodes are normalized by softmax, the adjacency matrix corresponding to the graph is taken as the correlation matrix A* containing rich category semantic relationships.
[0047] Pass "a photo of a[CLS]" as input to the text encoder, generating a vector representation for each label represents the category. Subsequently, the cosine similarity between each representation is calculated to derive the correlation prior between the labels:
[0048] A=sim(ZT,Z)
[0049] Where A = (aij) C×C , and Z=[z1,z2,…,zc]C ×d .
[0050] To avoid the problem of over-smoothing of node features, A is modified as follows:
[0051]
[0052] Among them, s is a hyperparameter that determines the weights assigned to the node itself and its neighboring nodes. To this end, the label correlation graph Can be exported as:
[0053]
[0054] picture The adjacency matrix of
[0055] Step 3: Build and train a long-tail multi-label image classification model that can exploit label correlation. The specific implementation process is as follows Figure 2 shown.
[0056] The structure of the long-tail multi-label image classification model consists of an image encoder E I (·), a text encoder E T (·) and a graph convolutional neural network module.
[0057] S1: The correlation matrix A* and a soft prompt vector whose length can be set according to user needs are used as the category features output by the text encoder as the input of the graph convolutional neural network module to obtain semantic fusion features; the training image is used as the input of the image encoder to obtain image features; the soft prompt vector is used as the input of the text encoder to obtain text features.
[0058] S1: The training image xk is fed into E I (xk) obtains the image feature Fx, which is prompted by the category text:
[0059]
[0060] The composition t is fed into E T (t) Get text embedding F t . Where c∈{1,…,C}, m∈{1,…,M}, and [CLS] token are replaced by specific class names, such as "person", "horse", or "car". [V] m represents a learnable word embedding with the same dimension as regular word embeddings in the vocabulary, which is 512; M is a hyperparameter specifying the number of context tokens.
[0061] The graph convolutional neural network module will receive A * and F t As input, the semantic correspondence between labels is encapsulated into the output HL Features.
[0062] S2: Text Embedding F t With H L Improved text embedding by performing residual connection Finally, the predicted probability is calculated:
[0063]
[0064] Among them, τ is the temperature parameter, σ is the sigmoid function, and sim(·,·) represents the cosine similarity.
[0065] S3: Transformer is used for both image encoder and text encoder, and the pre-trained model CLIP is used for parameter initialization. The reweighted long-tail loss function is used as the loss function to supervise the training of the long-tail multi-label image classification model.
[0066] The reweighting weight r of the loss function is based on the class frequency n c / NCalculated:
[0067]
[0068] Among them, α, β, and θ are adjustable hyperparameters used to adjust the weight distribution of each category. The loss function is:
[0069]
[0070] in, Represents the label of category c, and predicts the probability of positive samples For negative sample prediction probability, The corresponding model output loigt,v c is the class deviation value, which is:
[0071]
[0072] Among them, κ and ζ are manually adjustable hyperparameters used to adjust the model's prediction bias for each category.
[0073] During the training process, the parameters of the text encoder are fixed, and the text prompts are learnable soft prompts; the image encoder applies efficient parameter fine-tuning technology to learn a small number of task-specific parameters; the parameters of the graph convolutional neural network module are updated as the training progresses, and finally a realigned model will be obtained.
[0074] Step 4: After the test data has been through five data augmentation strategies, it is passed through the long-tail multi-label image classification model to obtain the logit output. The average of all logit outputs is calculated to obtain the logit output corresponding to the test data, and the final prediction probability is obtained through sigmoid. The source image and the grayscale image to be matched are read, and the original local feature map of the image is extracted using a deep convolutional neural network with frozen parameters.
[0075] The size of the test image x is resized to (224+e)×(224+e) to obtain x'. The 224×224 part of the center of x' is cropped and recorded as x c ; Crop the 224×224 part of the upper left corner of x', record it as x tl ; Crop the 224×224 part of the upper right corner of x', record it as x tr ; Crop the 224×224 part of the lower left of x', record it as x bl ; Cut the 224×224 part of the lower right corner of x', record it as x br .
[0076] {x c ,x tl ,x tr ,x bl ,x br}Evenly split into m patches (Each The size of is 16×16, and m=196). After the image is passed through the long-tail multi-label image classification model, logit{z c ,z tl ,z tr ,z bl ,z br}. The final logit result is z = Average(z c +z tl +z tr +z bl +z br ). The predicted probability of the test image x is calculated as p = σ(z).
[0077] The present invention extracts the potential correlation between categories to assist the learning of image features and help the tail categories improve their classification performance.
[0078] The above content merely illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of this technical solution belong to the technical idea proposed by the present invention and fall within the protection scope of the claims of the present invention.
Claims
1. A long-tail multi-label image classification method based on a vision-language pre-training model, characterized in that: The following steps are involved: (1) Obtain multi-label sample data with real labels (2) Replace the softmax layer in the traditional CLIP pre-training model with a sigmoid layer to adapt the traditional CLIP to multi-label tasks; (3) constructing a long-tail multi-label image classification model and training it; the model includes an image encoder, a text encoder, and a graph convolutional neural network module; (4) After the data enhancement strategy, the image data to be tested is input into the trained model for prediction, and finally the label prediction result is obtained to realize long-tail multi-label image classification.
2. According to claim 1, a long-tail multi-label image classification method based on a vision-language pre-training model is characterized in that: The implementation process of step (1) is as follows: Obtain multi-label sample data with true labels Where (x k ,Y k ) represents a sample label pair, is the sample x k The corresponding label set, N represents the number of multi-label samples collected; the dataset presents a long-tail category distribution, that is, a small number of categories have a large amount of sample data, while other categories are only associated with a small number of samples.
3. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 1, characterized in that: The implementation process of step (2) is as follows: The given template "a photo of a[CLS]" prompt word is passed through a text encoder to obtain the text embedding of the corresponding category, and the cosine similarity between the text embeddings of each category is calculated to construct a neighbor graph, in which [CLS] is replaced by the class name; as the training process, all node features in the graph tend to be consistent, hyperparameters are introduced to reweight the neighbor graph, so that when updating node features, the node itself will be assigned a fixed weight, and the weight of the related node will be determined by the neighborhood distribution; after the nodes are normalized by softmax, the adjacency matrix corresponding to the graph is taken as the correlation matrix containing rich category semantic relationships.
4. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 1, characterized in that: The implementation process of step (3) is as follows: S1: The correlation matrix A * The category features output by the text encoder and the learnable soft prompts are used as the input of the graph convolutional neural network module to obtain semantically fused features; The training image is used as the input of the image encoder to obtain the image features; the soft prompt seat is used as the input of the text encoder to obtain the text features; S2: The semantic fusion feature is residually connected with the text feature, the cosine similarity is calculated with the image feature, and the prediction probability is obtained through sigmoid. S3: The model is supervised by the heavily weighted long-tail loss function, and efficient parameter fine-tuning is applied to avoid overfitting problems.
5. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 4, characterized in that: The S1 implementation process is as follows: Training image x k Sent to E I (x k ) to obtain the image feature F x , prompted by the text of the category: The components t are fed into E T (t) Get text embedding F t ; Where c∈{1,…,C}, m∈{1,…,M}, [CLS]token is replaced by a specific class name; [V] m represents a learnable word embedding, M is a hyperparameter that specifies the number of context tokens; The graph convolutional neural network module will receive A * and F t As input, the semantic correspondence between labels is encapsulated into the output H L Features.
6. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 4, characterized in that: The S2 implementation process is as follows: Text Embedding F t With H L Improved text embedding by performing residual connection The predicted probability is calculated: Among them, τ is the temperature parameter, σ is the sigmoid function, and sim(·,·) represents the cosine similarity.
7. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 4, characterized in that: The S3 implementation process is as follows: The reweighting weight r of the loss function is based on the class frequency n c / NCalculated: Among them, α, β, and θ are adjustable hyperparameters used to adjust the weight distribution of each category; the loss function is designed as: in, Represents the label of category c, and predicts the probability of positive samples For negative sample prediction probability, The corresponding model output loigt,v c is the class deviation value, which is defined as: Among them, κ and ζ are adjustable hyperparameters used to adjust the model's prediction bias for each category; during the training process, the parameters of the text encoder are fixed, and the text prompt is a learnable soft prompt; the image encoder applies efficient parameter fine-tuning technology to learn a small part of task-specific parameters; the parameters of the graph convolutional neural network module are updated as the training progresses.
8. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 1, characterized in that: The implementation process of step (4) is as follows: The image x is resized to (224+e)×(224+e), resulting in x'; the 224×224 portion of the center of x' is cropped and recorded as x c ; Crop the 224×224 part of the upper left corner of x', record it as x tl ; Crop the 224×224 part of the upper right corner of x', record it as x tr ; Crop the 224×224 part of the lower left of x', record it as x bl ; Cut the 224×224 part of the lower right corner of x', record it as x br ; Set {x c ,x tl ,x tr ,x bl ,x br }Evenly split into m patches The image is passed through the long-tail multi-label image classification model to obtain logit{z c ,z tl ,z tr ,z bl ,z br }; The logit result is z = Average(z c +z tl +z tr +z bl +z br ); the predicted probability of the test image x is p = σ(z).
9. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 5, characterized in that: Said [V] m Has the same dimensionality as regular word embeddings in the vocabulary.
10. The long-tail multi-label image classification method based on a vision-language pre-training model according to claim 8, characterized in that: The m is 196 and the size of each patch is 16×16.
Citation Information
Cited By
Long-tail multi-label image classification method based on nerve collapse
CN119580003A
Lens label extraction method and device based on full-modal understanding and medium
CN120635790A
Long-tail distribution image recognition method based on hierarchical guidance two-stage calibration
CN121305613A
A long-tail distribution image recognition method based on hierarchical guidance two-stage calibration
CN121305613B
