A multimodal visual cueing technology representation learning method based on pre-trained models

Abstract

A multimodal visual cueing technology representation learning method based on a pre-trained model. After the text encoder of the visual model encodes the label text for a prompt, the already prompted text is used to generate an entire prompt mask for the image, covering the entire image to obtain a better prompting effect. Secondly, the huge gap between the objective forms of model pre-training and fine-tuning leads to the need for a large amount of labeled data to stimulate the model's visual grounding ability in downstream tasks. The present invention reformulates visual grounding as a fill-in-the-blank problem, using pixel-based common reference tags in images and texts to minimize this gap.

Description

Technical Field

[0001] The present invention belongs to the field of visual prompts, and specifically relates to a multimodal visual prompt technology representation learning method based on a pre-trained model, and a prompt-based image determination method. Background Art

[0002] For natural language processing in the text domain, pre-trained language models are applied to downstream tasks through pre-training and fine-tuning, adjusting the pre-trained language models to meet the specific needs of downstream tasks. However, as the model size gradually increases, the cost of fine-tuning also increases. Faced with the diversity of downstream tasks, adjusting different downstream tasks also becomes more complex. Therefore, researchers have proposed a more convenient method with stronger generalization capabilities, namely prompt technology, which can help pre-trained models "recall" what they "learned" during pre-training.

[0003] Specifically, for the input text data x, there is a mapping function f prompt (x), convert x into prompt form x', that is:

[0004] x'=f prompt (x)

[0005] The mapping function generally operates in two steps:

[0006] 1. Choose a template, which is generally a natural language with two empty positions: one for the input data x position [X], and the other for the generated text answer z position [Z].

[0007] 2. Fill the input data x into the specified position [X].

[0008] Hint technology first flourished in the text field, inspiring the visual field to explore hints in this area. However, there is a significant gap between the objective forms of model pre-training and fine-tuning, resulting in the need for large amounts of labeled data to stimulate the visual grounding capabilities of pre-trained models in the visual field for downstream tasks. Fine-tuning and linear probing are standard adaptation methods in the visual field. Fine-tuning fixes the remaining layers of the model, leaving only one layer for parameter adjustment, and accesses the model parameters of the entire layer; linear probing accesses the model at output, typically activating the penultimate layer of the model. Compared to constructing hints using pixels, visual hints apply the input to the entire model. After receiving the visual hint, the model does not need to access the model during the testing phase. This means that the user controls the input space of the entire model, further approaching artificial intelligence.

[0009] The game-theoretic GAN algorithm consists of two modules: the generator and the discriminator. The generator is responsible for generating realistic and plausible sample data. Once generated, the data is fed into the discriminator, which determines whether it is original labeled data or generated by the generator. The model's principle is that throughout the training process, the generator generates realistic and plausible samples that cause the discriminator to err and lose its ability to distinguish. In contrast, the discriminator improves its ability to distinguish, enabling it to successfully distinguish data. Using text to describe image information to generate corresponding images is, in its literal sense, complex and diverse. For example, a change in a specific descriptive word in a text can cause a significant number of pixel changes in the generated image. For example, when a description changes from brick red to dark red, the correlation between the changed pixel information is difficult for the human eye to discern. In contrast, generating text from images does not present such a serious problem because text is modeled using a language model. Generating images from text can be solved using GANs. Summary of the Invention

[0010] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to propose a multimodal visual prompting technology representation learning method based on a pre-trained model. Different from the traditional visual prompting method, the prompting technology proposed in the present invention will use the already prompted text to generate an entire prompt mask for the image after the text encoder of the visual model encodes the label text for prompting, covering the entire image to obtain a better prompting effect. Secondly, the huge gap between the objective forms of model pre-training and fine-tuning leads to the need for a large amount of labeled data to stimulate the model's visual grounding ability in downstream tasks. The present invention restates the visual grounding as a fill-in-the-blank problem, and uses pixel-based common reference markers in images and texts to alleviate this gap to the greatest extent.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A multimodal visual cueing technology representation learning method based on a pre-trained model, characterized by comprising the following steps:

[0013] Step 1. Assume that the visual language model uses CLIP, which generates prompts from text and adds them to the image. The model uses different encoders to process text and image data respectively. The CLIP model has a text encoder and an image encoder. The text encoder uses Bert, and the image encoder uses the vision transformer (vit). The text and image are fed into the encoder respectively, and a text-image row-column matrix is ​​constructed by linear mapping to the multimodal embedding vector space. The similarity between the text and the image is calculated. The higher the similarity, the higher the degree of matching, and vice versa. Therefore, it is necessary to obtain the text-image pair with the highest similarity, that is, to take the diagonal elements. By feeding the public cultural cloud data into the CLIP model, the pre-trained text and image encoders are fine-tuned. The loss function is the cross-entropy loss constructed by maximizing the diagonal of the similarity matrix. The fine-tuning operation makes the two encoders loaded with the pre-trained model more focused on public cultural data.

[0014] The cosine of the angle is used to measure the difference between two vector directions. In machine learning, it can also be used to measure the difference between two samples. Therefore, the cosine of the angle formula is used to measure the similarity between text vector A and image vector B in two-dimensional space:

[0015]

[0016] Step 2: Perform prompt processing on the text label data. Convert the text label data in the text encoder Bert in CLIP into an embedding vector and add manually designed text prompt vectors related to the public cultural cloud. That is, use prompt technology to first train the labeled text: cut the text data using data, add manually given labels, and integrate them into text vectors of equal length. Add manually specified static prompts to the specified position of the text sentence vector, and use the prompts to predict the obscured text information. This is similar to the cloze test in reading comprehension. Although manually searching for the best text prompt given static prompts is very laborious, prompts in the visual field can compensate for low-quality text prompts.

[0017] That is, there are n objects x1, x2…x n ,After being prompted by the prompt template “a photo of a{object}”, it is fed into the encoder;

[0018] Step 3: Use GAN to generate an adversarial network and the image information described by the text to generate an image prompt mask. The text used is the text that has already been prompted. Then, a prompt is generated to prompt the image. The prompt is a pixel prompt in the visual field. As an image mask, it covers the entire image as a prompt and is compared with the original image without the image mask.

[0019] Furthermore, in the training phase, for a data sample captured in a training session, firstly, the features of all texts and images are obtained through the text encoder and the image encoder, and the inner products are calculated respectively to obtain a matrix. When observing from the perspective of the image, the row direction is regarded as a text classifier, and from the perspective of the text, the column direction is regarded as an image classifier. The matching relationship between the image and the text in the data sample captured in a training session is known in advance. For the same pair of successfully matched images and texts, maximizing the inner product of their features is the objective function, that is, the elements on the diagonal of the feature matrix, and minimizing the inner products of irrelevant features. By fine-tuning the pre-trained text and image encoders, the two encoders loaded with the pre-trained model are more focused on public cultural data; the labeled text data is sent to the created The prompt template is pre-trained to obtain text data that has been prompted once. This text data is then fed into the GAN generator to convert the text into an image, which is overlaid on the original image to obtain the prompt image. The CLIP operation is then performed again to pair the prompt text-related data with the prompt image data to form a text-image pair. This text-image pair is then repeated several times to form the number of data samples captured for one training session, which are fed into the CLIP model. The training data is the prompt image-text pair. Unlike the first time, the second time freezes the trained text and image encoders and only updates the GAN parameters. Finally, in a learning scenario for the test, the new image is superimposed with the image prompt generated by the GAN, and the category to which the new image belongs is found by maximizing the diagonal matrix.

[0020] The GAN generative adversarial network model has two modules: the generator G and the discriminator D. The generator is responsible for generating realistic and reasonable sample data. After the data is generated, it is sent to the discriminator, which determines whether the data is original labeled data or generated by the generator. The model principle is that during the entire training process, the generator generates very realistic and reasonable samples, causing the discriminator to make mistakes and lose its discrimination ability. In contrast, the discriminator needs to improve its discrimination ability so that it can successfully identify the data.

[0021] Modules D and G play the following game on V(D,G)

[0022]

[0023] Note: A bunch of x, is from p data The examples given in the above example are real images. The real images are fed into the generator to get a score and then logarithmize it. A bunch of y, is p g The images generated by the generator are sent to the discriminator to get a score and then take the logarithm;

[0024] Minimizing the game GAN will have a global optimal situation, that is, when p g =p data When , and when the generator and discriminator have sufficient capacity, p g It will converge to p data ;

[0025] In order to learn to obtain effective visual description information in text description information data, recursive text encoder and deep convolution are used to learn its corresponding image mapping function, and optimized structured loss is used to train the corresponding function f t Guided text classifiers;

[0026]

[0027] where v n is an image, t n is the corresponding text description, {(x n ,t n ,y n ):n=1,…,N} is the training dataset, Δ is the 0-1 loss, y n is the class label; the classifier f v and f t The parameters are as follows:

[0028]

[0029]

[0030] where τ(y) is the set of text descriptions of class y, ψ is the text encoder (e.g., a character-level CNN or LSTM), φ is the image encoder (e.g., a deep convolutional neural network), and similarly, υ(y) is the set of image descriptions.

[0031] Furthermore, before the data is fed into the model, it is pre-processed with data enhancement. For English text, synonyms and different forms between words are removed through a series of operations such as de-pluralization, noun removal, and form conversion to convert the words back to their original appearance. For Chinese text, third-party tools are needed to segment the text.

[0032] Furthermore, manually specified static prompts are added to the specified position of the text sentence vector, usually at the beginning or end of the sentence.

[0033] The beneficial effects of the present invention are:

[0034] Compared with existing visual cueing methods, the method described in the present invention can better achieve visual cueing effects, and does not require a large amount of labeled data to stimulate the visual grounding ability of the pre-trained model in downstream tasks, and is consistent with the standard downstream task adaptation method in the visual field, that is, introducing a separate task-specific head to adapt to model parameter activation. The present invention only needs to adapt the pre-trained model by modifying the input space, and uses a gradient-based learning scheme to learn a single, input-independent perturbation (i.e., a visual cue), which can be used to improve the performance of the task; by changing the visual cue, the model can perform different tasks. It has obvious performance effects in the public cultural cloud and can be better applied to downstream classification tasks. For image label classification involved in the public cultural cloud, the present invention can have better performance. At the same time, the present invention innovatively proposes to use text description information to generate an image cue mask added to the image, and then send it to the CLIP model, that is, the visual information has been prompted twice, and the effect is significantly improved compared to the image without the mask. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The core diagram of the CLIP network of the present invention, where a is the pre-training-fine-tuning stage diagram and b is the secondary training stage diagram;

[0036] Figure 2 This is the GAN network core diagram of the present invention.

[0037] Figure 3 It is a processing flow chart of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] like Figure 1 、 2 3, a multimodal visual cueing technology representation learning method based on a pre-trained model is characterized by comprising the following steps:

[0040] Step 1. Assume that the visual language model uses CLIP, which generates prompts from text and adds them to the image. The model uses different encoders to process text and image data respectively. The CLIP model has a text encoder and an image encoder. The text encoder uses Bert, and the image encoder uses the vision transformer (vit). The text and image are fed into the encoder respectively, and a text-image row-column matrix is ​​constructed by linear mapping to the multimodal embedding vector space. The similarity between the text and the image is calculated. The higher the similarity, the higher the degree of matching, and vice versa. Therefore, it is necessary to obtain the text-image pair with the highest similarity, that is, to take the diagonal elements. By feeding the public cultural cloud data into the CLIP model, the pre-trained text and image encoders are fine-tuned. The loss function is the cross-entropy loss constructed by maximizing the diagonal of the similarity matrix. The fine-tuning operation makes the two encoders loaded with the pre-trained model more focused on public cultural data.

[0041] The cosine of the angle is used to measure the difference between two vector directions. In machine learning, it can also be used to measure the difference between two samples. Therefore, the cosine of the angle formula is used to measure the similarity between text vector A and image vector B in two-dimensional space:

[0042]

[0043] Step 2: Perform prompt processing on the text label data. Convert the text label data in the text encoder Bert in CLIP into an embedding vector and add manually designed text prompt vectors related to the public cultural cloud. That is, use prompt technology to first train the labeled text: cut the text data using data, add manually given labels, and integrate them into text vectors of equal length. Add manually specified static prompts to the specified position of the text sentence vector, and use the prompts to predict the obscured text information. This is similar to the cloze test in reading comprehension. Although manually searching for the best text prompt given static prompts is very laborious, prompts in the visual field can compensate for low-quality text prompts.

[0044] That is, there are n objects x1, x2…x n ,After being prompted by the prompt template “a photo of a{object}”, it is fed into the encoder;

[0045] Step 3: Use GAN to generate an adversarial network and the image information described by the text to generate an image prompt mask. The text used is the text that has already been prompted. Then, a prompt is generated to prompt the image. The prompt is a pixel prompt in the visual field. As an image mask, it covers the entire image as a prompt and is compared with the original image without the image mask.

[0046] Furthermore, in the training phase, for a data sample captured in a training session, firstly, the features of all texts and images are obtained through the text encoder and the image encoder, and the inner products are calculated respectively to obtain a matrix. When observing from the perspective of the image, the row direction is regarded as a text classifier, and from the perspective of the text, the column direction is regarded as an image classifier. The matching relationship between the image and the text in the data sample captured in a training session is known in advance. For the same pair of successfully matched images and texts, maximizing the inner product of their features is the objective function, that is, the elements on the diagonal of the feature matrix, and minimizing the inner products of irrelevant features. By fine-tuning the pre-trained text and image encoders, the two encoders loaded with the pre-trained model are more focused on public cultural data; the labeled text data is sent to the created The prompt template is pre-trained to obtain text data that has been prompted once. This text data is then fed into the GAN generator to convert the text into an image, which is overlaid on the original image to obtain the prompt image. The CLIP operation is then performed again to pair the prompt text-related data with the prompt image data to form a text-image pair. This text-image pair is then repeated several times to form the number of data samples captured for one training session, which are fed into the CLIP model. The training data is the prompt image-text pair. Unlike the first time, the second time freezes the trained text and image encoders and only updates the GAN parameters. Finally, in a learning scenario for the test, the new image is superimposed with the image prompt generated by the GAN, and the category to which the new image belongs is found by maximizing the diagonal matrix.

[0047] The GAN generative adversarial network model has two modules: the generator G and the discriminator D. The generator is responsible for generating realistic and reasonable sample data. After the data is generated, it is sent to the discriminator, which determines whether the data is original labeled data or generated by the generator. The model principle is that during the entire training process, the generator generates very realistic and reasonable samples, causing the discriminator to make mistakes and lose its discrimination ability. In contrast, the discriminator needs to improve its discrimination ability so that it can successfully identify the data.

[0048] Modules D and G play the following game on V(D,G)

[0049]

[0050] Note: A bunch of x, is from p data The examples given in the above example are real images. The real images are fed into the generator to get a score and then logarithmize it. A bunch of y, is p g The images generated by the generator are sent to the discriminator to get a score and then take the logarithm;

[0051] Minimizing the game GAN will have a global optimal situation, that is, when p g =p data When , and when the generator and discriminator have sufficient capacity, p g It will converge to p data ;

[0052] In order to learn to obtain effective visual description information in text description information data, recursive text encoder and deep convolution are used to learn its corresponding image mapping function, and optimized structured loss is used to train the corresponding function f t Guided text classifiers;

[0053]

[0054] where v n is an image, t n is the corresponding text description, {(x n ,t n ,y n ):n=1,…,N} is the training dataset, Δ is the 0-1 loss, y n is the class label; the classifier f v and f t The parameters are as follows:

[0055]

[0056]

[0057] where τ(y) is the set of text descriptions of class y, ψ is the text encoder (e.g., a character-level CNN or LSTM), φ is the image encoder (e.g., a deep convolutional neural network), and similarly, υ(y) is the set of image descriptions.

[0058] Furthermore, before the data is fed into the model, it is pre-processed with data enhancement. For English text, synonyms and different forms between words are removed through a series of operations such as de-pluralization, noun removal, and form conversion to convert the words back to their original appearance. For Chinese text, third-party tools are needed to segment the text.

[0059] Furthermore, manually specified static prompts are added to the specified position of the text sentence vector, usually at the beginning or end of the sentence.

[0060] Example

[0061] While pre-training with the CLIP model, image prompts are generated from labeled text using a generative adversarial network in a single learning scenario. First, after preparing the Public Culture Cloud dataset, the data is fed into the CLIP model to fine-tune the pre-trained text and image encoders. The loss function is to maximize the cross-entropy loss constructed along the diagonal of the similarity matrix. This fine-tuning operation allows the two encoders loaded with the pre-trained model to focus more on the public culture data. Second, in a single learning scenario, a text prompt structure related to the public culture cloud is designed. The labels and prompt templates are combined to construct text prompt sentences, and the fine-tuned text encoder is used to generate text prompt features. Simultaneously, the constructed text prompt sentences are passed through a randomly initialized GAN network to generate image prompts. The image prompts are then superimposed on real training images and image features are generated using the fine-tuned image encoder. Subsequently, the image features and text features are multiplied in a second round to construct a similarity matrix. The loss function is to maximize the cross-entropy loss constructed along the diagonal of the similarity matrix. Unlike the first round, the trained text and image encoders are frozen in this second round, and only the GAN parameters are updated. Finally, in a learning scenario tested, a new image is superimposed with the image prompt generated by the GAN, and the category to which the new image belongs is determined by maximizing the diagonal matrix. To implement prompts in the visual field, this paper uses the visual model CLIP, written in Python, and developed using VSCode.

[0062] Datasets: We use multiple datasets to measure both in-distribution and out-of-distribution performance. The model learns visual cues on the training set and evaluates its performance on the test set. For in-distribution performance, we test on the CUB dataset for bird images and the Oxford-102 dataset for flower images.

[0063] The experimental environment is as follows:

[0064] ·CPU: AMD Ryzen 93900X 12-Core Processor 3.80GHz

[0065] Memory: 32GB

[0066] Hard Drive: 2TB

[0067] Operating system: Windows 10 (64-bit)

[0068] Network structure:

[0069] The generator network is represented as G:R Z ×R T →R D

[0070] The discriminator network is represented as D:R D×R T →{0,1}

[0071] Where T is the dimension of the text description embedding, D is the dimension of the image, and Z is the noise input dimension of G.

[0072] In the generator G, we first start from the noise prior z∈R Z ~ω(0,1), we then encode the text query t using the text encoder j. The description embedding j(t) is first compressed using a fully consistent encoding system.

[0073] Baseline Methods: We compare visual cues with textual cues (i.e., “zero-shot” CLIP) and linear detection. For text-cued CLIP, we use the cue “This is a photo of [LABEL]” as the default.

Claims

1. A multimodal visual cueing technology representation learning method based on a pre-trained model, characterized by , including the following steps: Step 1: Assume that the visual language model uses CLIP, which generates prompts from text and adds them to images. The model uses different encoders to process text and image data respectively. The text and image are fed into the encoder respectively, and a text-image row-column matrix is ​​constructed by linear mapping to the multimodal embedding vector space. The similarity between the text and the image is calculated. The higher the similarity, the higher the degree of matching, and vice versa. Therefore, it is necessary to obtain the text-image pair with the highest similarity, that is, to take the diagonal elements. By feeding the public cultural cloud data into the CLIP model, the pre-trained text and image encoders are fine-tuned. The loss function is the cross-entropy loss constructed by maximizing the diagonal of the similarity matrix. The cosine of the angle is used to measure the difference between two vector directions. In machine learning, it can also be used to measure the difference between two samples. Therefore, the cosine of the angle formula is used to measure the similarity between text vector A and image vector B in two-dimensional space: Step 2: Perform prompt processing on the text label data. Convert the text label data in the text encoder Bert in CLIP into an embedding vector and add manually designed text prompt vectors related to the public cultural cloud. That is, use prompt technology to first train the labeled text: cut the text data using data, add manually given labels, and integrate them into text vectors of equal length. Add manually specified static prompts to the specified position of the text sentence vector, and use the prompts to predict the obscured text information. This is similar to the cloze test in reading comprehension. Although manually searching for the best text prompt given static prompts is very laborious, prompts in the visual field can compensate for low-quality text prompts. That is, there are n objects x1, x2…x n ,After being prompted by the prompt template "a photo of a{object}", it is fed into the encoder; Step 3: Generate an adversarial network using GAN to generate an image prompt mask using the image information described by the text. The text used is the text that has already been prompted. The prompt is then generated to prompt the image. The prompt is a pixel prompt in the visual field. It serves as an image mask, covering the entire image as a prompt, and is compared with the original image without the image mask. Step 4. In the training phase, for a data sample captured in a training session, first obtain the features of all texts and images through the text encoder and the image encoder, calculate the inner product respectively, and obtain a matrix. From the perspective of the image, the row direction is regarded as a text classifier, and from the perspective of the text, the column direction is regarded as an image classifier. The matching relationship between the image and the text in the data sample captured in a training session is known in advance. For the same pair of successfully matched images and texts, maximizing the inner product of their features is the objective function, that is, the elements on the diagonal of the feature matrix, and minimizing the inner product of irrelevant features. By fine-tuning the pre-trained text and image encoders, the two encoders loaded with the pre-trained model are more focused on public cultural data; the labeled text data is sent to the created The prompt template is pre-trained to obtain text data that has been prompted once. This text data is then sent to the GAN generator to convert the text into an image, which is overlaid on the original image to obtain the prompt image. The CLIP operation is performed again to pair a prompt text-related data with the prompt image data to form a text-image pair. This text-image pair is then repeated several times to form the number of data samples captured for one training, which are sent to the CLIP model. The training data is the prompt image-text pair. Unlike the first time, the second time will freeze the two encoders of text and image that have been trained, and only update the parameters of GAN. Finally, in a learning scenario of the test, the new image will be superimposed with the image prompt generated by GAN, and the category to which the new image belongs will be found by maximizing the diagonal matrix.

2. A multimodal visual cueing technology representation learning method based on a pre-trained model according to claim 1, characterized in that Before the data is fed into the model, the data in the dataset is preprocessed with data enhancement. For English text in the text data, a series of operations such as removing plurals, removing nouns, and converting forms are performed to convert the words into their original appearance. For Chinese text, a third-party tool is needed to segment the text.

3. The method for learning a multimodal visual cueing technique representation based on a pre-trained model according to claim 1, characterized in that ,The manually specified static prompt is added to the specified position of the text sentence vector, which is usually the beginning or end of the sentence.

4. The method for learning a multimodal visual cueing technique representation based on a pre-trained model according to claim 1, characterized in that ,The text encoder is a character-level CNN or LSTM, and the image encoder is a deep convolutional neural network.

5. The method for learning multimodal visual cueing technology representation based on a pre-trained model according to claim 1, characterized in that ,CLIP model has a text encoder and an image encoder. The text encoder uses Bert and the image encoder uses vision transformer (vit).

6. The method for learning a multimodal visual cueing representation based on a pre-trained model according to claim 1, characterized in that ,GAN generative adversarial network model has two modules: generator G and discriminator D; The generator is responsible for generating realistic and reasonable sample data. After the data is generated, it is sent to the discriminator. The discriminator determines whether the data is original labeled data or generated by the generator. The principle of the model is that during the entire training process, the generator generates very realistic and reasonable samples to make the discriminator make mistakes and lose its discrimination ability. In contrast, the discriminator needs to improve its own discrimination ability so that it can successfully identify the data. Modules D and G play the following game on V(D,G): Note: A bunch of x, is from p data The examples given in the above example are real images. The real images are fed into the generator to get a score and then logarithmize it. A bunch of y, is p g The images generated by the generator are sent to the discriminator to get a score and then take the logarithm; Minimizing the game GAN will have a global optimal situation, that is, when p g =p data When , and when the generator and discriminator have sufficient capacity, p g It will converge to p data ; In order to learn to obtain effective visual description information in text description information data, recursive text encoder and deep convolution are used to learn its corresponding image mapping function, and optimized structured loss is used to train the corresponding function f t Guided text classifiers; where v n is an image, t n is the corresponding text description, {(x n ,t n ,y n ):n=1,…,N} is the training dataset, Δ is the 0-1 loss, y n is the class label; the classifier f v and f t The parameters are as follows: where τ(y) is the set of text descriptions of class y, ψ is the text encoder, φ is the image encoder, and similarly, υ(y) is the set of image descriptions.

Images