Text-region matching-based multi-label image recognition method in scene with missing labels

Abstract

The invention discloses a text-region matching-based multi-tag image recognition method in a missing tag scene, which comprises the following steps of: performing cross attention operation on visual features and query vectors by using a pre-trained vision-language model as a feature extractor to generate a region-level saliency map; utilizing a multi-layer perceptron to generate query-level features, and performing aggregation operation on the visual features through the region-level saliency map to generate region-level feature representation; then, a knowledge distillation process is carried out, knowledge represented by region-level features is migrated to query-level feature representation, and based on a multi-modal comparative learning method, storage and comparative learning are carried out by utilizing a memory block visual prototype and a text prototype, so that the features of the same specific category approach each other, and the features of different specific categories are properly pulled apart; therefore, a better intra-class and inter-class relationship is established by the original model, and the accuracy of multi-label image recognition in a scene with missing labels is effectively improved.

Description

Technical Field

[0001] The present invention relates to multi-label image recognition and deep learning technology, and specifically to a multi-label image recognition method in a scenario with missing labels based on text-region matching. Background Art

[0002] Multi-label image recognition is a basic task in the field of computer vision. It aims to simultaneously identify multiple targets in an image. It has a wide range of applications and has always been a research hotspot. However, building a dataset with complete labels is a very laborious task, which hinders the application and promotion of multi-label image learning. A new setting - multi-label image recognition with missing labels has received increasing attention recently. For the images in the dataset, only some labels are annotated. Although this reduces the annotation cost, it also causes the failure of traditional multi-label image recognition methods in this setting, which brings new challenges.

[0003] The existing methods of using known labels and treating unknown labels as negative labels and using graph learning for semantic transfer have solved certain problems but still cannot achieve satisfactory results. With the rise of visual language models, all methods based on prompt tuning have been used by researchers to improve recognition performance.

[0004] However, current methods for hinting at fine-tuning still have many limitations: 1) The “text-spatial point matching” method overemphasizes feature map details and ignores the rich structural information within visual features, which introduces irrelevant noise and affects final recognition; 2) The “text-image matching” method matches text with the entire image, which makes it difficult to distinguish different visual concepts in text. Multi-label image recognition involves multiple targets and diverse scenes, which can lead to poor matching; 3) The “text-text matching” method performs poorly on test images due to the differences between training data and test data.

[0005] Current hint tuning methods often discard valuable information in unknown annotations and block position labels when calculating losses, which is detrimental to the accurate recognition of the model. In addition, current hint tuning methods fail to effectively align visual and text representations in the joint embedding space. This difference between the modalities also hinders the matching of vision and text.

[0006] In general, existing multi-label image recognition methods in scenarios with missing labels have the following shortcomings: 1) Traditional recognition methods are difficult to obtain effective results; 2) Although the hint tuning method is effective, the methods based on "text-spatial point matching", "text-image matching" and "text-text matching" have their own shortcomings; 3) The current hint tuning method cannot make good use of unknown label information and cannot effectively align visual and text features. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a multi-label image recognition method in a scenario with missing labels based on text-region matching.

[0008] Technical solution: A multi-label image recognition method based on text-region matching in a scenario with missing labels includes the following steps:

[0009] Step 1: Build and train a visual-language model to extract features from images and texts to obtain visual features. Text feature G = {g1, g2, ..., g C}, and use the trained text features as text prototypes T = {t1, t2, …, t C};

[0010] Step 2: Set the query vector Q = {q1, q2, ..., q C} and visual features Perform cross-attention operations to generate a region-level saliency map R. The elements in the query vector Q contain all labels (both known and missing labels), and the elements in Q are automatically updated during training. Then, a multi-layer perceptron is used to generate a query-level feature representation. The original visual features are then Perform aggregation operations to generate region-level feature representations Use the region-level saliency map R to generate a visual prototype v c , all c-type visual prototypes are stored in V c ;

[0011] Step 3: Perform knowledge distillation, that is, first represent the query-level feature F q and region-level feature representation F r The predicted logical value is calculated by relative entropy divergence to obtain the distillation loss (The loss function is used to make the two probability distributions consistent), and then the knowledge of the region-level feature representation is transferred to the query-level feature representation to obtain a higher quality region-level saliency map;

[0012] Step 4: Use the memory block technology to modify the visual prototype V of step 1 c and the text prototype T of step 2, and use multimodal contrast loss in the projection space Perform contrastive learning to make the features of the same specific category closer to each other and the features of different specific categories appropriately farther away, so that the original vision-language model can establish better intra-class and inter-class relationships and improve recognition accuracy;

[0013] Step 5: Classification loss Distillation loss And multimodal contrast loss Constructing the joint loss function Used to optimize vision-language models;

[0014] Step 6: After eliminating the multimodal category prototype and loss function, use the text region matching score to obtain the recognition score as the final prediction score, and calculate multi-label recognition indicators such as mAP on the test set.

[0015] Furthermore, the detailed method of extracting features from images and texts in step 1 is as follows:

[0016] Use the visual encoder F(·) to extract visual features

[0017] Use the text encoder G(·) to extract features from the text and get the text features G = {g1, g2, …, g C};

[0018] Among them, the c-type prompt in the text is denoted by t c ={ω1,ω2,…,ω L ,CLS c},ω1,ω2,…,ω L is a cue to learn L learnable cue vectors, CLS c is a category-related prompt template (such as “a photo of person” is a prompt template for the “person” class), which converts the trained text features T = {t1, t2, …, t C} as a text prototype.

[0019] Furthermore, the detailed process of step 2 is as follows:

[0020] Step 2.1, generate a region-level saliency map R;

[0021] The query vector Q = {q1,q2,…,q C} and visual features Perform cross attention, calculated as follows:

[0022] R = Cross_Att(Q,f)

[0023] Where Q is a set of query vectors, f is the visual feature map extracted by the visual encoder, Cross_Att(·) represents a cross attention operation, and R is the generated region-level saliency map;

[0024] Step 2.2: Generate query-level feature representation F using multi-layer perceptron q , calculated as follows:

[0025] F q =Cross_Att(Q,f)+MLP(Cross_Att(Q,f))

[0026] Among them, Cross_Att(·) and MLP(·) represent the cross attention operation and multi-layer perceptron operation respectively;

[0027] Step 2.3: Aggregate the visual features f through the region-level saliency map R to generate region-level feature representation The calculation is as follows:

[0028]

[0029] Among them, e c is the position on the region-level salient feature map R, p c Represents the significant feature map R on e c The saliency of the eigenvector at the location;

[0030] Step 2.4: Use the region-level saliency map R to aggregate the visual features of the top K saliency positions for each category to obtain the visual prototype v c , the calculation formula is as follows:

[0031] v c =TopK(r c )·f

[0032] Among them, r c is the position on the regional-level salient feature map R, and TopK(·) represents the operation of taking the top K largest elements by value; here r c With e c Both represent the position on the regional level salient feature map, but r c Emphasize all positions on the region-level saliency feature map R, and then use TopK(·) to take the top K maximum responses, while e c It refers specifically to the area of ​​category perception.

[0033] Furthermore, the specific method of obtaining a high-quality region-level saliency map representation in step 3 is as follows:

[0034] First, the matching function is defined using cosine similarity as follows:

[0035]

[0036] where f c and g c Represent the visual features and text features of category c respectively, τ is a learnable parameter that plays an approximate normalization role, y c It represents the existence of class c;

[0037] Then, the matching scores between the query-level feature representation and the corresponding text and the matching scores between the region-level feature representation and the corresponding text are calculated as follows:

[0038]

[0039] in and They represent the query-level feature representation and region-level feature representation of class c, respectively. c Represents the text features of class c;

[0040] Next, the relative entropy divergence of the two matching scores obtained above is calculated to obtain the distillation loss

[0041]

[0042] in, and Represent query-level feature representations and region-level feature representation and the matching score (prediction probability) of the i-th category; finally, the resulting distillation loss (KL loss constraint), the knowledge of region-level feature representation can be transferred to query-level feature representation, thereby obtaining a better region-level saliency map.

[0043] Furthermore, the multimodal contrast loss in step 4 is The specific method is as follows:

[0044]

[0045] Among them, τ is the temperature coefficient used to adjust the output size, P and N represent the positive label set and negative label set respectively, and p a is the anchor point, indicating the text prototype t c ; Through contrast loss The constraints can establish better intra-class and inter-class relationships, thereby improving the recognition performance.

[0046] Furthermore, the step 5 combines the loss function The expression is as follows:

[0047]

[0048] Among them, β and γ are hyperparameters used to weigh the impact of each loss on the visual-language model; is the classification loss, and the calculation formula is as follows:

[0049]

[0050] Where p represents the probability output of the visual-language model, p m =max(pc,0), γ+, γ- and c are all adjustable hyperparameters.

[0051] Furthermore, during the training of the visual-language model, the pseudo-label generation method is as follows:

[0052] For image data in a batch First calculate a set of regional level feature representations i represents the i-th image, j represents the j-th category among C categories;

[0053] Secondly, we calculate the region-level feature representation F r The similarity with the visual prototype v is calculated as follows:

[0054]

[0055] Where cosine(·,·) represents cosine similarity; the cosine similarity operation here represents the calculation of the region-level feature representation and the corresponding visual prototype of all C categories;

[0056] Then, all region-level representations f of category c are calculated c And all corresponding visual prototypes v c The average cosine similarity between , the process is as follows:

[0057]

[0058] in, represents the average similarity associated with the positive label, represents the average similarity associated with negative labels, represents the average similarity associated with the location tags, and Represent the positive label set, negative label set and unknown label set respectively;

[0059] Next, compare to the positive and negative thresholds:

[0060]

[0061] in, and Represent the positive threshold and negative threshold respectively. The threshold is dynamically and adaptively updated for different categories using exponential moving average:

[0062]

[0063] Where η is the moving average parameter, and Represent the average similarity associated with positive labels and the average similarity associated with negative labels respectively;

[0064] In addition, a similar strategy is adopted for the estimation of unknown labels, and finally the pseudo labels estimated from the visual prototype and the text prototype are combined:

[0065]

[0066] in and Represents the result of matching with the visual prototype and the text prototype. =1 indicates that the matching result label exists, otherwise =-1 indicates that the label does not exist. When both matches exist, the existence of the pseudo-label is deduced; when neither exists, the non-existence of the pseudo-label is deduced; in other cases, it is deduced that the label is unknown.

[0067] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0068] (1) For the problem of multi-label recognition in scenarios with missing labels, this paper uses a unique "text-visual area matching" method to perform prompt tuning, which greatly improves the recognition effect. Compared with existing methods such as semantic transfer based on graph models, representation fusion, and treating unknown labels as negative samples, this model has better performance.

[0069] (2) Compared with the existing methods such as "text-spatial point matching", "text-image matching" and "text-text matching", the "text-visual area matching" method of the present invention can take into account the internal information between visual features and alleviate the problem of introduced noise. Secondly, the method of the present invention can make it easier to distinguish different visual concepts in text. In addition, the present invention will not cause poor performance due to the heterogeneity between training data and test data.

[0070] (3) The present invention can make good use of the information of unknown labels and can also effectively represent their visual and textual information in the joint embedding space, thereby achieving accurate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0072] Figure 2 It is a schematic diagram of the model network structure of the present invention;

[0073] Figure 3 It is a matching effect diagram of the embodiment. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0075] like Figure 1 As shown, the multi-label image recognition method based on text-region matching in a scenario with missing labels of the present invention comprises the following steps:

[0076] Step 1: Build and train a visual-language model to extract features from images and texts to obtain visual features. Text feature G = {g1, g2, ..., g C}, and use the trained text features as text prototypes T = {t1, t2, …, t C};

[0077] Step 2: Set the query vector Q = {q1, q2, ..., q C} and visual features Perform cross-attention operations to generate a region-level saliency map R. The elements in the query vector Q contain all labels (both known and missing labels), and the elements in Q are automatically updated during training. Then, a multi-layer perceptron is used to generate a query-level feature representation. The original visual features are then Perform aggregation operations to generate region-level feature representations Use the region-level saliency map R to generate a visual prototype v c , all c-type visual prototypes are stored in V c ;

[0078] Step 3: Perform knowledge distillation, that is, first represent the query-level feature F q and region-level feature representation F r The predicted logical value is calculated by relative entropy divergence to obtain the distillation loss Then the knowledge of region-level feature representation is transferred to query-level feature representation to obtain higher quality region-level saliency map.

[0079] Step 4: Use memory block technology to analyze the visual prototype V c and the text prototype T, and use multimodal contrast loss in the projection space Perform contrastive learning to make features of the same specific category closer to each other and features of different specific categories appropriately farther apart, so that the original vision-language model can establish better intra-class and inter-class relationships;

[0080] Step 5: Classification loss Distillation loss And multimodal contrast loss Constructing the joint loss function Used to optimize vision-language models;

[0081] Step 6: After eliminating the multimodal category prototype and loss function, use the text region matching score to obtain the recognition score as the final prediction score, and calculate multi-label recognition indicators such as mAP on the test set.

[0082] The detailed method of extracting features from images and texts in step 1 of this embodiment is as follows:

[0083] Use the visual encoder F(·) to extract visual features

[0084] Use the text encoder G(·) to extract features from the text and get the text features G = {g1, g2, …, g C};

[0085] Among them, the c-type prompt in the text is denoted by t c ={ω1,ω2,…,ω L ,CLS c},ω1,ω2,…,ω L is a cue to learn L learnable cue vectors, CLS c is a category-related prompt template (such as “a photo of person” is a prompt template for the “person” class), which converts the trained text features T = {t1, t2, …, t C} as a text prototype.

[0086] The detailed process of step 2 of this embodiment is as follows:

[0087] Step 2.1, generate a region-level saliency map R;

[0088] The query vector Q = {q1,q2,…,q C} and visual features Perform cross attention, calculated as follows:

[0089] R = Cross_Att(Q,f)

[0090] Where Q is a set of query vectors, f is the visual feature map extracted by the visual encoder, Cross_Att(·) represents a cross attention operation, and R is the generated region-level saliency map;

[0091] Step 2.2: Generate query-level feature representation F using multi-layer perceptron q , calculated as follows:

[0092] F q =Cross_Att(Q,f)+MLP(Cross_Att(Q,f))

[0093] Among them, Cross_Att(·) and MLP(·) represent the cross attention operation and multi-layer perceptron operation respectively;

[0094] Step 2.3: Aggregate the visual features f through the region-level saliency map R to generate region-level feature representation The calculation is as follows:

[0095]

[0096] Among them, e c is the position on the region-level salient feature map R, p c Represents the significant feature map R on e c The saliency of the eigenvector at the location;

[0097] Step 2.4: Use the region-level saliency map R to aggregate the visual features of the top K saliency positions for each category to obtain the visual prototype v c , the calculation formula is as follows:

[0098] v c =TopK(r c )·f

[0099] Among them, r c is the position on the regional-level salient feature map R, and TopK(·) represents the operation of taking the top K largest elements by value; here r c With e c Both represent the position on the regional level salient feature map, but r c Emphasize all positions on the region-level saliency feature map R, and then use TopK(·) to take the top K maximum responses, while e c It refers specifically to the area of ​​category perception.

[0100] In this embodiment, all formula symbols containing c are for a specific category c. More specifically, there are C categories in the image set, and c represents the cth category (i.e., it represents a certain category). For example: It represents the regional feature representation of class c, which is a representation vector; and f r The regional-level feature representation of all classes (C in total) is a set of C representation vectors, which is mathematically represented as a C×d feature matrix, where d is the length of the representation vector.

[0101] The specific method of obtaining a high-quality region-level saliency map representation in step 3 of this embodiment is as follows:

[0102] First, the matching function is defined using cosine similarity as follows:

[0103]

[0104] where f c and g c Represent the visual features and text features of category c respectively, τ is a learnable parameter that plays an approximate normalization role, y c It represents the existence of class c;

[0105] Then, the matching scores between the query-level feature representation and the corresponding text and the matching scores between the region-level feature representation and the corresponding text are calculated as follows:

[0106]

[0107] in and They represent the query-level feature representation and region-level feature representation of class c, respectively. c Represents the text features of class c;

[0108] Next, the relative entropy divergence of the two matching scores obtained above is calculated to obtain the distillation loss

[0109]

[0110] in, and Represent query-level feature representations and region-level feature representation and the matching score (prediction probability) of the i-th category; finally, the resulting distillation loss (The constraints on L loss are as follows Figure 2 As shown in Figure 3, the knowledge of region-level feature representation can be transferred to query-level feature representation, thereby obtaining a better region-level saliency map.

[0111] In step 4 of this embodiment, the multimodal contrast loss The specific method is as follows:

[0112]

[0113] Among them, τ is the temperature coefficient used to adjust the output size, P and N represent the positive label set and negative label set respectively, and p a is the anchor point, indicating the text prototype t c ; Through contrast loss The constraints can establish better intra-class and inter-class relationships, thereby improving the recognition performance.

[0114] Step 5 of this embodiment combines the loss function The expression is as follows:

[0115]

[0116] Among them, β and γ are hyperparameters used to weigh the impact of each loss on the visual-language model; is the classification loss, and the calculation formula is as follows:

[0117]

[0118] Where p represents the probability output of the visual-language model, p m =max(pc,0), γ+, γ- and c are all adjustable hyperparameters.

[0119] In the process of training the visual-language model in this embodiment, the pseudo-label generation method is as follows:

[0120] For image data in a batch First calculate a set of regional level feature representations i represents the i-th image, j represents the j-th category among C categories;

[0121] Secondly, we calculate the region-level feature representation F r The similarity with the visual prototype v is calculated as follows:

[0122]

[0123] Where cosine(·,·) represents cosine similarity; the cosine similarity operation here represents the calculation of the region-level feature representation and the corresponding visual prototype of all C categories;

[0124] Then, all region-level representations f of category c are calculated c And all corresponding visual prototypes v c The average cosine similarity between , the process is as follows:

[0125]

[0126] in, represents the average similarity associated with the positive label, represents the average similarity associated with negative labels, represents the average similarity associated with the location tags, and Represent the positive label set, negative label set and unknown label set respectively;

[0127] Next, compare to the positive and negative thresholds:

[0128]

[0129] in, and Represent the positive threshold and negative threshold respectively. The threshold is dynamically and adaptively updated for different categories using exponential moving average:

[0130]

[0131] Where η is the moving average parameter, and Represent the average similarity associated with positive labels and the average similarity associated with negative labels respectively;

[0132] In addition, a similar strategy is adopted for the estimation of unknown labels, and finally the pseudo labels estimated from the visual prototype and the text prototype are combined:

[0133]

[0134] in and Represents the result of matching with the visual prototype and the text prototype. =1 indicates that the matching result label exists, otherwise =-1 indicates that the label does not exist. When both matches exist, the existence of the pseudo-label is deduced; when neither exists, the non-existence of the pseudo-label is deduced; in other cases, it is deduced that the label is unknown.

[0135] The overall network structure of this embodiment is as follows Figure 2 As shown, a pre-trained visual language model (CLIP) is used as a feature extractor, which is mainly divided into four parts: a visual feature and text feature extraction module, a category-aware region learning module, a knowledge distillation module for optimized regions, and a multimodal contrastive learning module. Among them, the visual feature and text feature extraction module is implemented with the help of the existing pre-trained visual language model (CLIP), while the category-aware region learning module, the knowledge distillation module for optimized regions, and the multimodal contrastive learning module are proposed by the present invention to solve the problem of multi-label image recognition in label-missing scenarios.

[0136] Specifically, the first step is feature extraction, that is, the visual encoder and text encoder of CLIP are used to extract visual features and text features respectively, and the trained text features are input into the memory block as text prototypes; secondly, the query vector and visual features are used to perform cross-attention operations to generate a region-level saliency map, and the query-level feature representation and the region-level saliency map are used to generate the region-level feature representation. At the same time, the features of the top K high-response positions of the visual features are selected using the saliency map to aggregate and obtain the visual prototype and add it to the memory block. Then a knowledge distillation is performed, that is, the relative entropy divergence operation is performed on the predicted values ​​of the query-level feature representation and the region-level feature representation, and the model is optimized as a loss function, so that the knowledge of the region-level feature representation can be transferred to the query-level feature representation, thereby obtaining a better region-level saliency map. In addition, the visual prototype and the text prototype are contrastively learned, that is, the contrast loss is calculated in the projection space, so that the features of the same specific category are close to each other, and the features of different specific categories are appropriately pulled apart, so that the original model can establish better intra-class and inter-class relationships. Finally, the multi-label classification loss, distillation loss and contrast loss are combined as a joint loss function to optimize the model.

[0137] Figure 2 In the figure, R is the region-level saliency map, and F is the q and F r represent query-level feature representation and region-level feature representation respectively, and They represent distillation loss, contrast loss and classification loss respectively; G represents text features, but text features are a concept specific to an instance, such as matching a specific image with the text representation of a label; and the text prototype is a concept on a set. For example, in training, the text prototype represents the center of all label text features.

[0138] Example 1

[0139] To verify the effect of the present invention, Figure 3 As shown in (a), (b), (c) and (d), this embodiment takes four images as examples and gives different label texts to show the matching effect. It can be seen that the present invention finds the corresponding visual area according to the text well, so it is suitable for multi-label image recognition in the scenario of missing labels. In addition, Figure 3 Another popular multi-label image recognition method in the missing label scenario, DualCoOp, is also presented for comparison with the present invention to further demonstrate the advantages of the present invention.

[0140] Example 2

[0141] This implementation was experimentally verified on the MS-COCO, VOC 2007 and Visual Genome datasets. Table 1 shows the experimental results on the MS-COCO dataset. * indicates the results reproduced by the present invention and compared with the current advanced methods. Vanilla represents the weight of the backbone network from Imagenet, and CLIP represents the weight from CLIP. 10%, 20%, ... 90% represent the proportion of partial labels in the missing label scenario. From the experimental results, it can be seen that the effect of the present invention is better than the current advanced methods in all settings and overall.

[0142] Table 1

[0143]

[0144] Table 2

[0145]

[0146]

[0147] Table 3

[0148]

[0149] Tables 2 and 3 respectively show the results on the VOC 2007 and Visual Genome datasets. The descriptions of various parameters are the same as those in Table 1. From the experimental results, it can be seen that the effect of the present invention is the best in various settings and overall compared with the current advanced methods.

[0150] This implementation also conducts experiments on the most extreme setting in the missing label scenario - each image has only one positive label on the MS-COCO, VOC 2012, NUS-WIDE and CUB datasets, where LargeLoss setup and SPLCsetup are two mainstream settings in the field. From the experimental results in Table 4, it can be seen that under the extreme setting of only one positive label, the present invention is still ahead of the existing advanced methods.

[0151] Table 4

[0152]

[0153] Table 5

[0154] Method Total parm. Learnable parm. FLOPs mAP DualCoOp 91.17M 1.25M 36.97G 81.90％ SCPNet 96.02M 59.66M 36.60G 83.20％ The present invention 96.72M 6.80M 37.50G 85.40％

[0155] Table 5 shows the implementation efficiency of the present invention. Compared with DualCoOp, the technical solution of the present invention adds negligible learnable parameters, and the recognition accuracy is significantly ahead. Compared with SCPNet, the technical solution of the present invention has basically the same total parameter amount, but the number of learnable parameters is very small, and it also achieves a significant lead in recognition accuracy.

Claims

1. A multi-label image recognition method based on text-region matching in a scenario with missing labels, characterized in that: The following steps are involved: Step 1: Build and train a visual-language model to extract features from images and texts to obtain visual features. Text feature G = {g1, g2, ..., g C }, and use the trained text features as text prototypes T = {t1, t2, ..., t C }; Step 2: Set the query vector Q = {q1, q2, ..., q C } and visual features Perform cross-attention operation to generate region-level saliency map R, and then use multi-layer perceptron to generate query-level feature representation The original visual features are then Perform aggregation operations to generate region-level feature representations Use the region-level saliency map R to generate a visual prototype v c , all c-type visual prototypes are stored in V c ; Step 3: Perform knowledge distillation, that is, first represent the query-level feature F q and region-level feature representation F r The predicted logical value is calculated by relative entropy divergence to obtain the distillation loss Then the knowledge of region-level feature representation is transferred to query-level feature representation to obtain higher quality region-level saliency map. Step 4: Use the memory block technology to modify the visual prototype V in step 1 c and the text prototype T of step 2, and use multimodal contrast loss in the projection space Perform contrastive learning to make features of the same specific category closer to each other and features of different specific categories appropriately farther away, so that the original vision-language model can establish better intra-class and inter-class relationships; Step 5: Classification loss Distillation loss And multimodal contrast loss Constructing the joint loss function Used to optimize vision-language models; Step 6: After eliminating the multimodal category prototype and loss function, use the text region matching score to obtain the recognition score as the final prediction score, and calculate the mAP multi-label recognition index on the test set.

2. The multi-label image recognition method based on text-region matching in a scenario with missing labels according to claim 1, characterized in that: The detailed method of extracting features from images and texts in step 1 is as follows: Use the visual encoder F(·) to extract visual features Use the text encoder G(·) to extract text features and obtain text features G = {g1, g2, ..., g C }; Among them, the c-type prompt in the text is denoted by t c ={ω1, ω2, ..., ω L , CLS c }, ω1, ω2, ..., ω L is a cue to learn L learnable cue vectors, CLS c is a category-related prompt template that converts the trained text features T = {t1, t2, ..., t C } as a text prototype.

3. The multi-label image recognition method based on text-region matching in a scenario with missing labels according to claim 1, characterized in that: The detailed process of step 2 is as follows: Step 2.1, generate a region-level saliency map R; The query vector Q = {q1, q2, ..., q C } and visual features Perform cross attention, calculated as follows: R = Cross_Att(Q, f) Where Q is a set of query vectors, Cross_Att(·) represents a cross attention operation; Step 2.2: Generate query-level feature representation F using multi-layer perceptron q , calculated as follows: F q =Cross_Att(Q,f)+MLP(Cross_Att(Q,f)) Among them, Cross_Att(·) and MLP(·) represent the cross attention operation and multi-layer perceptron operation respectively; Step 2.3: Aggregate the visual features f through the region-level saliency map R to generate region-level feature representation The calculation is as follows: Among them, e c is the position on the region-level salient feature map R, p c Represents the significant feature map R on e c The saliency of the eigenvector at the location; Step 2.4: Use the region-level saliency map R to aggregate the visual features of the top K saliency positions for each category to obtain the visual prototype v c , the calculation formula is as follows: in c =TopK(r c )f Among them, r c is the position on the region-level saliency feature map R, and TopK(·) represents the operation of taking the top K largest elements by value.

4. The multi-label image recognition method based on text-region matching in a scenario with missing labels according to claim 1, characterized in that: The specific method of obtaining a high-quality region-level saliency map representation in step 3 is as follows: First, the matching function is defined using cosine similarity as follows: where f c and g c Represent the visual features and text features of category c respectively, τ is a learnable parameter, y c It represents the existence of class c; Then, the matching scores between the query-level feature representation and the corresponding text and the matching scores between the region-level feature representation and the corresponding text are calculated as follows: in and They represent the query-level feature representation and region-level feature representation of class c respectively, and gc represents the text feature of class c; Next, the relative entropy divergence of the two matching scores obtained above is calculated to obtain the distillation loss in, and Represent query-level feature representations and region-level feature representation and the matching score of the i-th category.

5. The multi-label image recognition method based on text-region matching in a scenario with missing labels according to claim 1, characterized in that: The multimodal contrast loss in step 4 The specific method is as follows: Among them, τ is the temperature coefficient used to adjust the output size, P and N represent the positive label set and negative label set respectively, and p a The anchor point represents the text prototype t c .

6. The multi-label image recognition method based on text-region matching in a scenario with missing labels according to claim 1, characterized in that: The step 5 joint loss function The expression is as follows: Among them, β and γ are hyperparameters used to weigh the impact of each loss on the visual-language model; is the classification loss, and the calculation formula is as follows: Where p represents the probability output of the visual-language model, p m =max(pc,0), γ+, γ- and c are all adjustable hyperparameters.

7. The multi-label image recognition method based on text-region matching in a scenario with missing labels according to claim 1, characterized in that: During the training of the visual-language model, the pseudo-label generation method is as follows: For image data in a batch First calculate a set of regional level feature representations i represents the i-th image, j represents the j-th category among C categories; Secondly, calculate the region-level feature representation F r The similarity with the visual prototype v is calculated as follows: Where cosine(·,·) represents cosine similarity; Then, all region-level representations f of category c are calculated c And all corresponding visual prototypes v c The average cosine similarity between , the process is as follows: in, represents the average similarity associated with the positive label, represents the average similarity associated with negative labels, represents the average similarity associated with the location tags, and Represent the positive label set, negative label set and unknown label set respectively; Next, compare to the positive and negative thresholds: in, and Represent the positive threshold and negative threshold respectively. The threshold is dynamically and adaptively updated for different categories using exponential moving average: Where η is the moving average parameter, and Represent the average similarity associated with the positive label and the average similarity associated with the negative label respectively; finally, the pseudo labels estimated from the visual prototype and the text prototype are combined: in and Represents the result of matching with the visual prototype and the text prototype. =1 indicates that the matching result label exists, otherwise =-1 indicates that the label does not exist. When both matches exist, the existence of the pseudo-label is deduced; when neither exists, the non-existence of the pseudo-label is deduced; in other cases, it is deduced that the label is unknown.

Images