Training Methods for Multimodal Vectorized Models

Abstract

The present invention discloses a training method for a multimodal vectorization model, comprising the following steps: step one, data preparation, collecting image-text pair data, text pair data and image pair data; step two, building a unified multimodal vectorization model capable of processing text and image data simultaneously; step three, building a loss function; step four, using the image-text pair data, text pair data and image pair data simultaneously to perform two-stage training. The training method for a multimodal vectorization model of the present invention is a training method for synchronously optimizing image-text unimodal vectors and image-text cross-modal vectors during the training process.

Description

Technical Field

[0001] The present invention relates to the field of multimodal vector retrieval, and more specifically to a training method for a multimodal vectorization model. Background Art

[0002] At present, the deep learning method is used to solve the problem of cross-modal image and text retrieval. The general approach is to map the image text information to a one-dimensional feature space through a multimodal neural network, calculate the similarity of the image and text features to sort the results, and thus realize cross-modal image-text retrieval. Therefore, there is a publication number CN118364256A in the prior art, named a multimodal model training method, system and electronic device, which discloses a multimodal model training method. However, this method mainly relies on image-text pair data and contrastive loss for training during training. Although this method performs well in cross-modal retrieval (such as image-to-text search, text-to-image search), it is less effective in single-modal retrieval (such as text-to-text search, image-to-image search). Summary of the invention

[0003] In view of the deficiencies in the prior art, the object of the present invention is to provide a training method for synchronously optimizing image-text unimodal vectors and image-text cross-modal vectors during the training process.

[0004] To achieve the above object, the present invention provides the following technical solution: comprising the following steps:

[0005] Step 1: Prepare data and collect picture-text pair data, text-text pair data, and image-text pair data;

[0006] Step 2: Build a unified multimodal vectorization model that can process both text and image data;

[0007] Step 3: Construct loss function;

[0008] Step 4: Use the image-text pair data, text-pair data, and image-pair data to perform two-stage training.

[0009] As a further improvement of the present invention, the image-text pair data prepared in step 1 refers to each piece of data containing an image and a text description related thereto, and these data are used for cross-modal training to help the model learn how to align image and text representations.

[0010] As a further improvement of the present invention, the text pair data prepared in step one means that each piece of data contains two related text fragments, and these data are used for unimodal text training to help the model learn how to perform well in text matching tasks, and during the second stage of training in step four, hard negative examples are generated for the training data by synthesizing data with a large model, that is, data with similar text but different meanings are generated based on the training text.

[0011] As a further improvement of the present invention, the image pair data prepared in step 1 means that each data contains two related images, specifically generating image pair data from an image classification data set for single-modal image training, helping the model learn how to perform well in image classification and feature extraction tasks, and in the process of model training in step 4, enhancing training by selecting difficult negative examples, wherein the specific method of selecting difficult negative examples is to extract image feature vectors by training the model obtained in the first stage of step 4, and then calculate the Cosine similarity of the image feature vectors of two different classification category labels, retaining images with a similarity greater than 90% as difficult negative examples, and the specific calculation formula of Cosine cosine similarity is:

[0012]

[0013] Among them, A B represents the dot product of image feature vectors A and B, represents the Euclidean norm of vector A, Represents the Euclidean norm of vector B.

[0014] As a further improvement of the present invention, the multimodal vectorization model constructed in step 2 includes:

[0015] Image encoder, which is used to extract visual features of images. It uses convolutional neural networks or visual transformers, which are image processing technologies based on large-scale data pre-training.

[0016] The text encoder is used to extract the semantic features of the text. It is implemented using natural language processing technologies such as Transformer based on the attention mechanism and based on large-scale data pre-training.

[0017] As a further improvement of the present invention, the loss function constructed in the step three includes a picture-text pair loss function, a pure text pair contrast loss function or a pure image pair contrast loss function, and a pure text pair contrast loss function or a pure image pair contrast loss function based on difficult negative examples.

[0018] As a further improvement of the present invention, the image-text pair loss function is as follows:

[0019]

[0020] in: is the text semantic feature vector of the i-th text, is the image feature vector of the ith image, sim(t,v) is the cosine similarity between the text semantic feature vector and the image feature vector, τ is a learnable temperature parameter used to control the scaling ratio of the similarity value, and N is the data batch size during training. It is usually recommended that the larger the batch size, the better, which will be beneficial to the generalization of the model, but it also needs to be adjusted according to the training video memory.

[0021] As a further improvement of the present invention, the pure text pair contrast loss function or the pure image pair contrast loss function is as follows:

[0022]

[0023] in: and They represent the two vector representations of the i-th text pair or image pair respectively, and sim(a,b) represents the similarity between the two vectors.

[0024] As a further improvement of the present invention, the pure text pair contrast loss function or pure image pair contrast loss function based on hard negative examples is as follows:

[0025]

[0026] λ is the weighted coefficient for misjudgment of hard negative examples, and the recommended value is 0.5. is the original loss function mentioned above.

[0027] As a further improvement of the present invention, the specific steps of the training in step 4 are as follows:

[0028] Step 41: Mix image-text pairs, text pairs, and image pairs in each batch of data for training, optimize the vectorized model, preliminarily improve the vector of the image-text model, and align the cross-modal and unimodal vectors to complete the first stage of training;

[0029] Step 42: Add hard examples for further optimization, and introduce a new contrast loss function based on hard negative examples for the training of text pairs and image pairs to improve the model's ability to extract fine-grained features and complete the second stage of training;

[0030] In each training step, the cross-modal contrast loss and the pure modal loss function are calculated, and the two loss values ​​are summed up as the total loss value for gradient calculation and back propagation.

[0031] Beneficial effects of the present invention: The present invention proposes a new training method, which introduces text pair data and image pair data in the training process and adds corresponding text pair infoNCE loss and image pair infoNCE loss, thereby simultaneously optimizing the retrieval capabilities of single modality and cross-modality. The final model not only performs well in cross-modal retrieval, but also has strong capabilities in single modality retrieval, with the following significant effects:

[0032] 1. Improve single-modal retrieval capabilities:

[0033] By introducing text pair and image pair data, as well as the corresponding unimodal contrast loss, the performance of the model in text search and image search tasks is significantly improved.

[0034] 2. Enhance cross-modal retrieval capabilities:

[0035] The excellent performance of the original method in image-to-text and text-to-image tasks is maintained.

[0036] 3. Comprehensive performance improvement:

[0037] The model performs well in a variety of retrieval tasks, and can meet multiple retrieval needs with one model, saving computing power costs and resource consumption, and has a wider range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a training method for the multimodal vectorization model of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0040] Reference Figure 1 As shown, a training method for a multimodal vectorization model in this embodiment includes the following steps:

[0041] 1. Data preparation: The data types required for training in this invention mainly include the following three types:

[0042] Picture and text data:

[0043] Image-text pair data means that each piece of data contains an image and a text description associated with it. This data is used for cross-modal training to help the model learn how to align image and text representations. For example:

[0044] Image: A photo showing a city at sunset

[0045] Text description: "A panoramic view of the city at sunset, with buildings silhouetted against a colorful sky."

[0046] Text to Data:

[0047] Text pair data means that each piece of data contains two related text fragments. This data is used for single-modal text training to help the model learn how to perform well in text matching tasks. For example:

[0048] Text pairs: ("The quick brown fox jumped over the lazy dog.", "A quick brown fox jumped over a lazy dog.")

[0049] In addition, in order to further improve the distinguishing ability of the model, the present invention generates hard negative examples for the training data by synthesizing data with a large model in the second stage, that is, generating data with similar text but different meanings based on the training text. For example:

[0050] Positive text pairs: ("A black cat sits on the windowsill.", "A black cat is by the window.")

[0051] Difficult example text pairs: ("A black cat is sitting on the windowsill.", "A white cat is by the window.")

[0052] Image pair data:

[0053] Image pair data means that each piece of data contains two related images. The present invention generates image pair data from image classification datasets for single-modal image training, helping the model learn how to perform well in image classification and feature extraction tasks. For example:

[0054] Positive image pair: two photos of cats from different angles

[0055] In order to improve the model's ability to distinguish, we enhance training by selecting hard negative examples. For example:

[0056] Positive image pair: two photos of cats from different angles

[0057] Hard negative image pairs: a photo of a cat and a photo of a dog, these two photos have certain similarities in color and background. The selection method is to extract image feature vectors through the model obtained through the first stage of training, and then calculate the cosine similarity of the image feature vectors of two different classification category labels, and retain images with a similarity greater than 90% as hard negative examples.

[0058] The calculation formula of Cosine similarity is:

[0059]

[0060] Among them A B represents the dot product (inner product) of image feature vectors A and B, represents the Euclidean norm of vector A (i.e. the length of the vector), Represents the Euclidean norm of vector B (that is, the length of the vector).

[0061] These image pair data can be generated by pairing existing image classification data, ensuring that the image embedding model can effectively capture image features and perform classification tasks.

[0062] 2. Model Architecture:

[0063] A unified multimodal vectorization model is adopted to process text and image data simultaneously.

[0064] The model contains a text encoder and an image encoder, which are used to process text and image data respectively.

[0065] Image encoder: It is used to extract visual features of an image. It can use image processing techniques based on large-scale data pre-training, such as convolutional neural networks (CNN) or visual transformers (ViT).

[0066] Text encoder: It is used to extract the semantic features of text. It can use natural language processing technologies based on large-scale data pre-training, such as Transformer based on attention mechanism (such as BERT, GPT, etc.).

[0067] The specific model structure diagram is as follows Figure 1 shown.

[0068] 3. Loss function design:

[0069] Picture and text pair:

[0070] For multimodal data such as image-text pairs, a cross-modal contrastive learning loss function is used. The core idea of ​​this loss function is to maximize the similarity between matching image-text pairs and minimize the similarity between mismatching image-text pairs.

[0071] The specific formula is as follows:

[0072]

[0073] in:

[0074] is the text semantic feature vector of the ith text.

[0075] is the image feature vector of the ith image.

[0076] is the cosine similarity between the text semantic feature vector and the image feature vector.

[0077] τ is a learnable temperature parameter that controls the scaling of similarity values.

[0078] N is the data batch size during training. It is usually recommended that the larger the batch size, the better, which will be beneficial to the generalization of the model, but it also needs to be adjusted according to the training video memory.

[0079] Text / image pairs: For single-modal data such as pure text and pure image, a contrastive learning loss function is used to optimize the similarity of text pairs or image pairs.

[0080] The specific formula is as follows:

[0081]

[0082] in:

[0083] and Two vector representations representing the i-th text pair or image pair respectively

[0084] Represents the similarity between two vectors

[0085] The two parts of the formula calculate the fixed and all The similarity, and fixed and all The similarity.

[0086] Text / image pairs based on hard negative examples: In the second stage of training, hard negative examples are added to further improve the quality of vectors generated by the model. At the same time, the loss function is adjusted to weight hard negative examples. When the vector with the highest similarity in the positive sample match during training is a hard negative example, the loss is weighted and increased to enable the model to learn fine-grained information in a targeted manner.

[0087] When matching difficult negative examples, the specific loss formula is as follows:

[0088]

[0089] λ is the weighting factor for misjudgment of hard negative examples, and the recommended value is 0.5

[0090] is the original loss function mentioned above.

[0091] 4. Training process:

[0092] At the same time, we use image-text data, text-text data and image-text data for training. The overall training steps are divided into two steps:

[0093] The first step is to use a mixture of picture-text pairs, text pairs, and image pairs in each batch of data for training, optimize the vectorization model, preliminarily improve the vector of the picture-text model, and align cross-modal and unimodal vectors.

[0094] The second step is to add difficult examples (data that are similar to the correct answer but have different meanings) for further optimization, and introduce a new contrast loss function based on difficult negative examples for the training of text pairs and image pairs to improve the model's ability to extract fine-grained features.

[0095] In each training step, the cross-modal contrast loss and the pure modal loss function are calculated, and the two loss values ​​are summed as the total loss value for gradient calculation and back propagation.

[0096] Through the above methods, the model can simultaneously optimize the cross-modal and single-modal retrieval capabilities of images and texts, thereby performing well in various retrieval tasks.

[0097] In summary, the training method of the multimodal vectorization model of this embodiment, through the above method, the model can simultaneously optimize the cross-modal and single-modal retrieval capabilities of images and texts, thereby performing well in various retrieval tasks.

[0098] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for training a multimodal vectorization model, characterized in that: The steps include: Step 1: Prepare data and collect picture-text pair data, text-text pair data, and image-text pair data; Step 2: Build a unified multimodal vectorization model that can process both text and image data; Step 3: Construct loss function; Step 4: Use the image-text pair data, text-pair data, and image-pair data to perform two-stage training; The specific steps of the training in step 4 are as follows: Step 41: In each batch of data, mixed picture-text pairs, text pairs, and image pairs are used for training. The text semantic vector of the text pair is used to calculate the pure text contrast learning loss value, the text semantic vector of the picture-text pair and the image feature vector are used to calculate the multimodal contrast learning loss value, and the image feature vector of the image pair is used to calculate the pure image contrast learning loss value. The three loss values ​​are summed up as the total loss value for gradient calculation and back propagation, and the cross-modal and unimodal vectors are aligned to complete the first stage of training; Step 42: Add difficult example samples for further optimization, and introduce a new contrast loss function based on difficult negative examples for the training of text pairs and image pairs. Use the text semantic vector of the text pair with the introduction of difficult negative examples to calculate the pure text contrast learning loss value based on difficult negative examples. Use the text semantic vector and image feature vector of the image-text pair to calculate the multimodal contrast learning loss value. Use the image feature vector of the image pair with the introduction of difficult negative examples to calculate the pure image contrast learning loss value based on difficult negative examples. The three loss values ​​are summed up and used as the total loss value for gradient calculation and back propagation to improve the model's ability to extract fine-grained features to complete the second stage of training.

2. The method for training a multimodal vectorization model according to claim 1, characterized in that: The image-text pair data prepared in step 1 means that each piece of data contains an image and a text description related thereto. These data are used for cross-modal training to help the model learn how to align image and text representations.

3. The method for training a multimodal vectorization model according to claim 1 or 2, characterized in that: The text pair data prepared in the step 1 means that each piece of data contains two related text fragments. These data are used for unimodal text training to help the model learn how to perform well in text matching tasks. During the second stage of training in step 4, hard negative examples are generated for the training data by synthesizing data with a large model, that is, data with similar text but different meanings is generated based on the training text.

4. The method for training a multimodal vectorization model according to claim 1 or 2, characterized in that: The image pair data prepared in step 1 means that each piece of data contains two related images, specifically, image pair data is generated from an image classification data set for single-modal image training, to help the model learn how to perform well in image classification and feature extraction tasks, and in the process of model training in step 4, the training is enhanced by selecting hard negative examples, wherein the specific method of selecting hard negative examples is to extract image feature vectors through the model obtained by the first stage training in step 4, and then calculate the Cosine similarity of the image feature vectors of two different classification category labels, and retain images with a similarity greater than 90% as hard negative examples. The specific calculation formula of Cosine similarity is: ； Among them, A B represents the dot product of image feature vectors A and B, represents the Euclidean norm of vector A, Represents the Euclidean norm of vector B.

5. The method for training a multimodal vectorization model according to claim 1 or 2, characterized in that: The multimodal vectorization model constructed in step 2 includes: Image encoder, which is used to extract visual features of images. It uses convolutional neural networks or visual transformers, which are image processing technologies based on large-scale data pre-training. The text encoder is used to extract the semantic features of the text. It is implemented using natural language processing technologies such as Transformer based on the attention mechanism and based on large-scale data pre-training.

6. The method for training a multimodal vectorization model according to claim 5, characterized in that: The image-text pair loss function is as follows: ； in: is the text semantic feature vector of the ith text, is the image feature vector of the ith image, is the cosine similarity between the text semantic feature vector and the image feature vector, τ is a learnable temperature parameter used to control the scaling ratio of the similarity value, and N is the data batch size during training. It is usually recommended that the larger the batch size, the better, which will be beneficial to the generalization of the model, but it also needs to be adjusted according to the training video memory.

7. The method for training a multimodal vectorization model according to claim 6, characterized in that: The pure text pair contrast loss function or the pure image pair contrast loss function is as follows: ； in: and Represent the two vector representations of the i-th text pair or image pair, Represents the similarity between two vectors.

8. The method for training a multimodal vectorization model according to claim 6, characterized in that: The pure text pair contrast loss function or pure image pair contrast loss function based on hard negative examples is as follows: ； λ is the weighted coefficient for misjudgment of hard negative examples, and the recommended value is 0.

5. is the original loss function mentioned above.

Images