A CLIP background knowledge-based image-text feature fusion method

Abstract

The application relates to a CLIP background knowledge-based image-text feature fusion method and belongs to the technical field of image-text fusion information processing and application. First, an OCR picture text recognition model is used to extract the text of each picture in a data set and add the text to the input. Different pre-training models are used to obtain vector representations of the input. The hidden vectors are concatenated, then pass through two Transformer encoders, and the relationship between the different modal vectors is calculated. Then, the CLIP model is used to extract the vectors of the picture and the text, and knowledge-rich multi-modal fusion features are obtained. The multi-modal fusion features are concatenated and input into a full connection layer, and a softmax function is used for normalization, so that the image-text fusion features are obtained in the CLIP context, and the multi-modal feature fusion based on the CLIP background knowledge is completed. The application enriches the relationship between the picture and the text, enhances the relationship of the picture mode on the text mode by using the OCR, and improves the representation degree of the multi-modal fusion features.

Description

Technical Field

[0001] This invention relates to a method for image-text feature fusion, specifically a method for image-text feature fusion based on CLIP (Contrastive Language-Image Pre-Training) pre-trained model as background knowledge, belonging to the field of image-text fusion information processing and application technology. Background Technology

[0002] In recent years, image-text feature fusion has been widely applied in scenarios such as classification and question answering. Common fusion methods can effectively represent the relationships between different modalities; however, their features still suffer from heterogeneity and poor interpretability. Effective multimodal feature fusion methods are mainly used to address these issues, aiming to further enhance the correlation between features from different modalities through feature fusion.

[0003] Currently, a common approach is to obtain feature vectors from different modalities using neural networks or pre-trained models, and then fuse them. Typical multimodal feature fusion methods can be divided into early fusion methods and late fusion methods. Early fusion methods consider the relationships between modalities during the modal feature extraction stage. Two common approaches are: one is to extract useful entities from the image and convert them into text; the other is to directly learn the associations between different modalities through attention mechanisms. Late fusion methods extract features from each modality separately and then fuse them. The most common method is to directly concatenate features from different modalities.

[0004] The aforementioned methods enhance the correlation between modal features through attention mechanisms, thereby strengthening the interpretability of the model on multimodal tasks. However, the models do not consider the wealth of available information and external knowledge present in samples from different modalities. Therefore, how to extract information and utilize external knowledge remains an important and challenging technical problem. In image modalities, whether the features can represent the intended meaning of the image is largely unaddressed by existing methods.

[0005] In recent years, how to eliminate heterogeneity between different images and text using image-text fusion methods has become a hot research topic. For example, in the paper "Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model" published at the 57th Annual Meeting of the Association for Computational Linguistics (ACL) in 2019 (pp. 2506-2515), a model for modality fusion at different stages was proposed. This model extracts the object text from the image and uses it as a new modality as task input, effectively reducing the heterogeneity between images and text. The fusion strategy at different stages also significantly improves the interpretability of modality fusion and the final result. However, this method does not consider whether the image features can well represent the image content, nor does it use external knowledge to support its model. Summary of the Invention

[0006] The purpose of this invention is to address the problem that existing technologies fail to fully acquire effective information from images and do not effectively utilize relevant background knowledge, resulting in poor image and text feature fusion effects. The invention creatively proposes an image and text feature fusion method based on CLIP background knowledge.

[0007] First, the relevant content of this invention will be explained.

[0008] CLIP (Contrastive Language-Image Pre-Training) is a pre-trained image-text model.

[0009] Background knowledge refers to the CLIP model parameters after pre-training with data, which contains a large amount of graphical and textual knowledge.

[0010] Multimodal refers to the image modality and text modality in a sample.

[0011] Modal fusion refers to a method of fusing image features with text features.

[0012] Attention mechanisms refer to the phenomenon where humans select specific parts of a visual region and focus their attention on them in order to make efficient use of limited visual information processing resources. Artificial intelligence leverages this phenomenon to propose enabling neural networks to select specific inputs. In this method, attention is manifested in assigning higher weights to entity pairs that support the focus and are more relevant to the query set.

[0013] OCR (Optical Character Recognition) is a text recognition model that can identify text in images.

[0014] Vision Transformer is an image feature extraction model that can segment an image into multiple parts and serialize them.

[0015] RoBERTa is a pre-trained language model that can capture the serialization features of text.

[0016] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language model that can capture the serialization features of text.

[0017] The Transformer encoder is a serialization model that focuses on the relationships between different features.

[0018] Accuracy, or accuracy, is the percentage of correct predictions out of all predictions.

[0019] The recall value refers to the probability of correctly identifying a positive sample out of all positive samples.

[0020] The Precision value refers to the probability of correctly detecting any target among all detected targets.

[0021] The F1 score is a statistical metric used to measure the precision of a binary classification model. It considers both precision and recall. A higher F1 score indicates better model performance.

[0022] The present invention is achieved using the following technical solution.

[0023] A method for fusing text and image features based on CLIP background knowledge includes the following steps:

[0024] Step 1: Use the OCR image text recognition model to extract text from each image in the dataset and add it to the input. The input consists of three parts: image, text, and OCR text.

[0025] If no text can be extracted, the original text will be used to fill the gap.

[0026] Step 2: Use different pre-trained models to obtain vector representations of the input from Step 1.

[0027] The images use the Vision Transformer model, or ViT model for short, while the text and OCR text use the RoBERTa model.

[0028] Specifically, step 2 includes the following steps:

[0029] Step 2.1: Given a sample from the task input , , This represents the dataset for the task. Represents an image. Represents text, This represents the text obtained through OCR.

[0030] Use BERT's vocabulary to clean all text in O, filtering out non-grammatical words.

[0031] Step 2.2: For image I, use the pre-trained ViT model to extract the latent vector representation of its last layer. The details are as follows:

[0032] (1)

[0033] in, This is the output of the pre-trained model, where R represents a vector matrix. This indicates the length of the image after serialization.

[0034] Step 2.3: For text T and text O, use two RoBERTa models respectively to extract the latent vectors of the last layer. and The details are as follows:

[0035] (2)

[0036] (3)

[0037] in, , This is the output of the pre-trained model, where R represents a vector matrix. , The length of the text;

[0038] Step 3: Concatenate the latent vectors obtained in Step 2, then pass them through two Transformer encoders to calculate the relationship between the different modal vectors, and take... (The representation of the first token in the sequence) is used as the final vector representation.

[0039] Specifically, step 3 includes the following steps:

[0040] Step 3.1: Series connection and This is then input into a Transformer encoder to obtain a vector representation of the relationship between the image and the text, as follows:

[0041] (4)

[0042] (5)

[0043] in, Indicates series connection. This represents the concatenation of image and text vectors. Indicates will The final result obtained after being input to the Transformer encoder.

[0044] Step 3.2: Series connection and This is then input into another Transformer encoder to obtain a vector representation of the relationship between the two texts, as follows:

[0045] (6)

[0046] (7)

[0047] in, Indicates series connection. This represents the concatenation of an image and a text vector. Indicates will The final result obtained after being input to the Transformer encoder.

[0048] Step 4: Using the CLIP model, extract vectors from the images and text respectively to obtain knowledge-rich multimodal fusion features, as follows:

[0049] (8)

[0050] in, These represent the vector representations of the image and text after passing through the CLIP model, respectively, both with a dimension of 512.

[0051] Step 5: The input is then concatenated and fed into a fully connected layer, where it is normalized using the softmax function.

[0052] (9)

[0053] (10)

[0054] Where h is the result of concatenating the vectors obtained in steps 2, 3, and 4. output is the final vector representation after multimodal fusion. The softmax function is a normalized exponential function.

[0055] Thus far, from step 1 to step 5, the image-text fusion features have been obtained in the CLIP context, completing the multimodal feature fusion based on CLIP background knowledge.

[0056] Beneficial effects

[0057] The method of the present invention has the following advantages compared with the prior art:

[0058] This method introduces a wealth of useful knowledge through the CLIP model, enriching the connection between images and text; and uses OCR to enhance the relationship between image modalities and text modalities, improving the representation of multimodal fusion features. Attached Figure Description

[0059] Figure 1 This is a schematic diagram illustrating the specific implementation process of the method of the present invention. Detailed Implementation

[0060] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] Example 1

[0062] like Figure 1 As shown, the present invention includes the following steps:

[0063] Step A: Extract text from the image using OCR;

[0064] In this specific embodiment, the procedure is the same as step 1;

[0065] Step B: The pre-trained model extracts features;

[0066] In this specific embodiment, it is the same as step 2;

[0067] Step C: Feature concatenation;

[0068] Specifically, in this embodiment, it is the same as steps 3.1 and 3.2;

[0069] Step D: Transformer encoder training;

[0070] Specifically, in this embodiment, it is the same as steps 3.1 and 3.2;

[0071] Step E: CLIP feature extraction;

[0072] In this specific embodiment, it is the same as step 4;

[0073] Step F: Fully connected layer;

[0074] In this specific embodiment, it is the same as step 5.

[0075] Example 2

[0076] Taking a sample (ID: 729970006449082369) from the multimodal image and text dataset in the paper "Multi-Modal Sarcasm Detection in Twitter with Hierarchical FusionModel" as an example, the specific operation steps of the method of the present invention will be described in detail.

[0077] like Figure 1 As shown, a multimodal fusion method based on CLIP background knowledge includes the following steps:

[0078] Step A1: Extract text from the image using OCR;

[0079] In this specific embodiment, the text in the image is extracted using an OCR model, and the text, along with the image itself and the text it carries, are used as three types of input.

[0080] Image ID: 729970006449082369

[0081] Corresponding text: today's todo list :)

[0082] OCR text: nice things to whisper when hugging someone

[0083] Step B1: The pre-trained model extracts features, specifically in the example ( Figure 1 The calculation process for the input representation is as follows:

[0084] The ViT pre-trained model was used to extract features from the images, and two different RoBERTa models were used to extract single-modal feature representations of the corresponding text and OCR text, as shown in Table 2:

[0085] Table 2. Single-modal feature representation in step B1 of Example 2

[0086] Modal Modal feature representation (first 5 dimensions) Dimension picture (0.922386,0.004932,-0.050307,-0.037010,-0.091474) 192*768 Corresponding text (-0.003813,-0.050697,1.017735,0.014269,-0.025330) 128*768 OCR text (0.025901,0.065208,-0.028254,0.069672,-0.046585) 128*768

[0087] Step C1: Feature concatenation.

[0088] Specifically, in the embodiments ( Figure 1 The three modal representations obtained in step B1 are concatenated pairwise. Specifically, image features are concatenated with corresponding text features, and corresponding text is concatenated with OCR text; the concatenated representations are shown in Table 3.

[0089] Table 3. Modal characteristics after cascading in step C1 of Example 2

[0090] Modal Multimodal feature representation (first 5 dimensions) Dimension Image + Corresponding Text (0.922386,0.004932,-0.050307,-0.037010,-0.091474) 320*768 Corresponding text + OCR text (-0.003813,-0.050697,1.017735,0.014269,-0.025330) 256*768

[0091] Step D1: Transformer encoder training;

[0092] Specifically, in this embodiment, the two multimodal representations obtained in step C1 are trained using two different Transformer encoders. The self-attention mechanism in the encoder is used to calculate the vectors of different modalities, and the first word of the final output layer is taken as the feature representation. The results are shown in Table 4.

[0093] Table 4 Results of step D1 in Example 2

[0094] Modal Multimodal feature representation (first 5 dimensions) Dimension Image + Corresponding Text (0.009471,-0.014569,-0.022956,-0.008901,-0.026102) 768 Corresponding text + OCR text (0.001853,-0.009944,-0.056892,0.003789,0.000534) 768

[0095] Step E1: CLIP feature extraction;

[0096] Specifically, in this embodiment, images and corresponding text are used as input to the CLIP model. The model outputs two vector representations, namely the feature representations of the image and the text, which contain a large amount of image-text multimodal knowledge. The results are shown in Table 5.

[0097] Table 5 Results of step E1 in Example 2

[0098] Modal Multimodal feature representation (first 5 dimensions) Dimension picture (0.003821,0.002937,0.045261,-0.049472,0.015706) 512 Corresponding text (0.007575,-0.003372,0.050863,-0.019459,0.029509) 512

[0099] Step F1, Fully Connected Layer;

[0100] The results obtained in steps D1 and E1 are concatenated, passed through a fully connected layer, and then subjected to softmax to obtain the final result, as shown in Table 6.

[0101] step Multimodal feature representation (first 5 dimensions) Dimension Series (0.003821,0.002937,0.045261,-0.049472,0.015706) 2560 Fully connected layer (0.0040152,-0.018595,0.0032516,-0.019360,0.016279) 512

[0102] Table 6 Results of step F1 in Example 2

[0103] Example 3

[0104] To further verify the effectiveness of the method of this invention, this example uses the multi-modal sarcasm dataset from the paper "Multi-Modal SarcasmDetection in Twitter with Hierarchical Fusion Model" for experimental comparison. The training set contains 19,816 image and text samples, including 8,642 positive samples and 11,174 negative samples; the validation set contains 2,409 samples, including 959 positive samples and 1,450 negative samples. The experimental data comparison is shown in Table 7.

[0105] Table 7. Results of the dataset comparison experiment

[0106] Model Precision Recall F1 Acc Hierarchical Fusion Model 76.57 84.15 80.18 83.44 Method of the present invention 82.18 86.55 84.31 87.17

[0107] As shown in Table 7, the method of this invention achieves good results in multimodal feature fusion, and improves upon the Hierarchical Fusion Model on the multimodal satire dataset. The definitions of each evaluation metric are provided above. Through improvements in the modal fusion method and the introduction of CLIP knowledge, the Acc score on the image-text satire dataset is improved by approximately 3.5%, and the F1 score by approximately 4.2%, demonstrating the effectiveness of this invention.

[0108] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications are within the scope of the invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for fusing image and text features based on CLIP background knowledge, characterized in that, Includes the following steps: Step 1: Using the OCR image text recognition model, extract the text from each image in the dataset and add it to the input. The input consists of three parts: image, text, and OCR text. If no text can be extracted, the original text will be used to fill the gaps. Step 2: Use different pre-trained models to obtain vector representations of the input from Step 1; The images use the Vision Transformer model, or ViT model for short, while the text and OCR text use the RoBERTa model. Specifically, step 2 includes the following steps: Step 2.1: Given a sample from the task input , , This represents the dataset for the task. Represents an image. Represents text, This represents the text obtained from OCR; Use BERT's vocabulary to clean all of O's text, filtering out non-textualized words; Step 2.2: For image I, use the pre-trained ViT model to extract the latent vector representation of its last layer. The details are as follows: (1) in, This is the output of the pre-trained model, where R represents a vector matrix. This indicates the length of the image after serialization; Step 2.3: For text T and text O, use two RoBERTa models respectively to extract the latent vectors of the last layer. and The details are as follows: (2) (3) in, , This is the output of the pre-trained model, where R represents a vector matrix. , The length of the text; Step 3: Concatenate the latent vectors obtained in Step 2, then pass them through two Transformer encoders to calculate the relationship between the different modal vectors, and take... As the final vector representation; Specifically, step 3 includes the following steps: Step 3.1: Series connection and This is then input into a Transformer encoder to obtain a vector representation of the relationship between the image and the text, as follows: (4) (5) in, Indicates series connection. This represents the concatenation of image and text vectors. Indicates will The final result obtained after being input to the Transformer encoder; Step 3.2: Series connection and This is then input into another Transformer encoder to obtain a vector representation of the relationship between the two texts, as follows: (6) (7) in, Indicates series connection. This represents the concatenation of the text and the OCR text vector. Indicates will The final result obtained after being input to the Transformer encoder; Step 4: Using the CLIP model, extract vectors from the images and text respectively to obtain knowledge-rich multimodal fusion features, as follows: (8) in, These represent the vector representations of the image and text after passing through the CLIP model, respectively, both with a dimension of 512; Step 5: The concatenated input is then fed into a fully connected layer, where it is normalized using the softmax function. (9) (10) Where h is the result of concatenating the vectors obtained in steps 3 and 4; output is the final vector representation after multimodal fusion; and softmax function is the normalized exponential function. The image-text fusion features were obtained in the CLIP context, and multimodal feature fusion based on CLIP background knowledge was completed.

Images