Cross-layer image text interaction prompt learning method based on attention mechanism

Abstract

The invention relates to the technical field of multi-mode prompt learning, and discloses a cross-layer image text interaction prompt learning method based on an attention mechanism, and the method comprises the steps: obtaining text data and image data; constructing a cross-layer image text interaction prompt learning network model; converting the text data into word vectors by utilizing a word embedding module; performing multi-level attention coding on the word vector and the text prompt vector by using a text encoder network to obtain a text code; a cross-layer interaction prompt learning module is used for generating an image prompt vector for the text prompt vector by adopting a multi-head attention mechanism; converting the image data into block feature maps by using a map segmentation module; performing multi-level attention coding on the block feature map and the image prompt vector by using an image encoder network to obtain an image code; and carrying out multi-modal contrast learning on the text codes and the image codes by using a multi-modal contrast learning module. According to the invention, the accuracy of multi-modal learning task execution can be further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal prompt learning, and in particular to a cross-layer image-text interactive prompt learning method based on an attention mechanism. Background Art

[0002] The combination of large models and multimodality gave birth to the CLIP (Contrastive Language-Image Pretraining) model, which was released by OpenAI in 2021 and was originally a pre-trained neural network model for matching images and text. The CLIP model consists of a text encoder and an image encoder, which encode text and images respectively. Then, by comparing the similarity between image encoding and text encoding, the similarity between related samples is maximized and the similarity between unrelated samples is minimized to obtain a pre-trained CLIP model.

[0003] Prompt learning is based on the CLIP model, which learns prompts for specific tasks in an end-to-end manner and adapts the CLIP model to downstream tasks. Existing prompt learning methods only perform prompt learning on the text side at the beginning. For example, CoOp uses learnable vectors to model the context words of text prompts, while the entire pre-trained parameters remain fixed, and the prompt vectors on the text side are trained to adapt to the task. Although CoOp is simple and effective, this will cause the model to overfit to the categories seen in training, and the generalization ability of unseen categories in the same field is poor. To overcome this problem, CoCoOp proposes conditional context optimization, which extends CoOp by learning a lightweight meta-neural network.

[0004] Subsequent studies adopted multimodal cue learning, such as MaPLe, which introduced learnable cue vectors on both the text and image sides, and learned separate cue vectors in the first few layers of the text encoder and image encoder, gradually modeling the stage-by-stage feature relationship to achieve rich contextual learning. A coupling function was also proposed to jointly learn the cue vectors of the two modalities. Specifically, the text cue vector is projected through a linear layer to obtain the image cue vector. However, the existing multimodal cue learning can only make the image and text branches of each layer interact, lacking coordinated interaction between layers. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a cross-layer image-text interaction prompt learning method based on the attention mechanism, which solves the problem that the image prompt vector and the text prompt vector of each layer in the prior art interact separately and lack of coordinated interaction between layers.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] A cross-layer image-text interaction prompt learning method based on attention mechanism includes the following steps:

[0008] Get text data and image data;

[0009] Constructing a cross-layer image-text interaction prompt learning network model; the cross-layer image-text interaction prompt learning network model includes a word embedding module, a text encoder network, a graph segmentation module, an image encoder network, a cross-layer interaction prompt learning module and a multimodal contrast learning module;

[0010] Use the word embedding module to convert text data into word vectors;

[0011] Use the text encoder network to perform multi-level attention encoding on word vectors and text prompt vectors to obtain text encoding;

[0012] The cross-layer interactive cue learning module is used to generate image cue vectors using a multi-head attention mechanism on the text cue vectors.

[0013] Use the graph segmentation module to convert image data into block feature maps;

[0014] Use the image encoder network to perform multi-level attention encoding on the block feature map and the image prompt vector to obtain the image encoding;

[0015] The multimodal contrastive learning module is used to perform multimodal contrastive learning on text encoding and image encoding.

[0016] Preferably, when converting text data into word vectors using a word embedding module, each word in the text data is mapped into a tensor according to set rules, and a word vector is generated based on all the obtained tensors.

[0017] Preferably, a text encoder network is used to perform multi-level attention encoding on word vectors and text prompt vectors. When obtaining the text encoding, multiple stacked encoding layers are used to concatenate the input word vector and the text prompt vector of the corresponding layer at each encoding layer, and then perform layer-by-layer encoding calculation to finally output the text encoding.

[0018] Preferably, when each encoding layer concatenates the input word vector and the text prompt vector of the corresponding layer and performs layer-by-layer encoding calculation, the concatenated vector of the input word vector and the text prompt vector of the corresponding layer is first normalized by a first normalization layer, and then the first output vector is calculated by a multi-head attention mechanism, and the concatenated vector is added to the first output vector through a first residual connection; then the addition result is normalized by a second normalization layer, and then the second output vector is calculated by a multi-layer perceptron, and the addition result is added to the second output vector through a second residual connection to obtain the final calculation result.

[0019] Preferably, when calculating the first output vector through the multi-head attention mechanism, the layer normalized vector is transformed by three linear layers to obtain the word relevance vector, the word weight vector and the word information vector; then the word relevance vector, the word weight vector and the word information vector are used to calculate the output vector through the scaled dot product attention mechanism; finally, the output vectors are concatenated and passed through a linear layer to generate the first output vector.

[0020] Preferably, when using a cross-layer interactive prompt learning module to generate an image prompt vector from a text prompt vector using a multi-head attention mechanism, the text prompt vectors corresponding to each encoding layer in the text encoder network are spliced ​​to obtain a text prompt sequence; then the text prompt sequence is transformed using three linear layers to obtain a word relevance vector, a word weight vector and a word information vector; then the word relevance vector, the word weight vector and the word information vector are used to calculate the output sequence through a multi-head attention mechanism; finally, the output sequences are spliced ​​and passed through a linear layer to generate an image prompt sequence, and the image prompt vector is obtained according to the image prompt sequence.

[0021] Preferably, when converting image data into a block feature map using a graph segmentation module, the image data is divided into blocks according to a set size, and each image block is projected and mapped into a one-dimensional vector to obtain a block feature map.

[0022] Preferably, an image encoder network is used to perform multi-level attention encoding on block feature maps and image prompt vectors. When obtaining the image code, multiple stacked coding layers are used to concatenate the input block feature map and the image prompt vector of the corresponding layer at each coding layer, and then perform layer-by-layer coding calculation to finally output the image code.

[0023] Preferably, when each coding layer concatenates the input block feature map and the image hint vector of the corresponding layer and performs layer-by-layer coding calculation, the concatenated vector of the input block feature map and the image hint vector of the corresponding layer is first normalized by the first image normalization layer, and then the third output vector is calculated by the multi-head attention mechanism, and the concatenated vector is added to the third output vector through the third residual connection; then the addition result is normalized by the fourth normalization layer, and then the fourth output vector is calculated by the multi-layer perceptron, and the addition result is added to the fourth output vector through the fourth residual connection to obtain the final calculation result.

[0024] Preferably, when calculating the third output vector through the multi-head attention mechanism, the layer normalized vector is transformed by three linear layers to obtain the word relevance vector, the word weight vector and the word information vector; then the word relevance vector, the word weight vector and the word information vector are used to calculate the output vector through the scaled dot product attention mechanism; finally, the output vectors are concatenated and passed through a linear layer to generate the third output vector.

[0025] The present invention has the following beneficial effects:

[0026] Aiming at the problem that prompts cannot interact between layers, the present invention proposes a cross-layer image-text interactive prompt learning method based on attention mechanism, which enables prompt vectors of different layers of text prompts to interact, thereby learning prompt vectors that are more effective for the current task. It can also generate image prompt vectors in more dimensions, further improving the accuracy of multimodal learning execution tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a cross-layer image-text interaction prompt learning method based on an attention mechanism;

[0028] Figure 2 Schematic diagram of the network model structure for cross-layer image-text interaction prompt learning. DETAILED DESCRIPTION

[0029] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0030] Aiming at the problem that prompts cannot interact between layers, the present invention proposes a cross-layer image-text interactive prompt learning method based on attention mechanism, which enables prompt vectors of different layers of text prompts to interact, thereby learning prompt vectors that are more effective for the current task. It can also generate image prompt vectors in more dimensions, further improving the accuracy of multimodal learning execution tasks.

[0031] Reference Figure 1 , this embodiment provides a cross-layer image-text interaction prompt learning method based on attention mechanism, comprising the following steps S1 to S8:

[0032] S1, obtaining text data and image data;

[0033] S2, constructing a cross-layer image-text interaction prompt learning network model; the cross-layer image-text interaction prompt learning network model includes a word embedding module, a text encoder network, a graph segmentation module, an image encoder network, a cross-layer interaction prompt learning module and a multimodal contrast learning module;

[0034] S3, use the word embedding module to convert text data into word vectors;

[0035] S4, using the text encoder network to perform multi-level attention encoding on the word vector and the text prompt vector to obtain the text encoding;

[0036] S5, using the cross-layer interactive prompt learning module to generate an image prompt vector using a multi-head attention mechanism for the text prompt vector;

[0037] S6, using a graph segmentation module to convert the image data into a block feature map;

[0038] S7, using the image encoder network to perform multi-level attention encoding on the block feature map and the image prompt vector to obtain the image encoding;

[0039] S8. Use the multimodal contrastive learning module to perform multimodal contrastive learning on text encoding and image encoding.

[0040] In an alternative embodiment of the present invention, referring to Figure 2 When the present embodiment uses the word embedding module to convert text data into word vectors, the pre-trained word embedding module is used to map each word in the text data into a tensor according to the set rules, and the word vectors are generated according to all the obtained tensors.

[0041] In an alternative embodiment of the present invention, referring to Figure 2 In this embodiment, the text encoder network is used to perform multi-level attention encoding on the word vector and the text prompt vector. When the text code is obtained, the text encoder network is constructed using multiple stacked coding layers, and the input word vector and the text prompt vector of the corresponding layer are concatenated at each coding layer, and then the coding calculation is performed layer by layer, and the text code is finally output. K-1 It is the text hint vector corresponding to each layer from layer 0 to layer K-1 on the text side, where K represents adding hint vectors to the first K layers. It is randomly initialized with a normal distribution vector with a mean of 0 and a standard deviation of 0.02, of size [K, text_dim]. Each row represents the hint vector of the corresponding layer, and the hint vector length is text_dim.

[0042] In this embodiment, when each encoding layer concatenates the input word vector and the text prompt vector of the corresponding layer and performs layer-by-layer encoding calculation, the concatenated vector of the input word vector and the text prompt vector of the corresponding layer is first normalized by the first normalization layer, and then the first output vector is calculated by the multi-head attention mechanism, and the concatenated vector is added to the first output vector through the first residual connection; then the addition result is normalized by the second normalization layer, and then the second output vector is calculated by the multi-layer perceptron, and the addition result is added to the second output vector through the second residual connection to obtain the final calculation result.

[0043] When this embodiment calculates the first output vector through the multi-head attention mechanism, the layer normalized vector is transformed using three linear layers to obtain a word relevance vector, a word weight vector, and a word information vector; the word relevance vector, the word weight vector, and the word information vector are then used to calculate the output vector through the scaled dot product attention mechanism; finally, the output vectors are concatenated and passed through a linear layer to generate the first output vector.

[0044] In an alternative embodiment of the present invention, referring to Figure 2 In this embodiment, when the cross-layer interactive prompt learning module is used to generate the image prompt vector by adopting the multi-head attention mechanism for the text prompt vector, the text prompt vector P0 to P0 corresponding to each encoding layer in the text encoder network is converted into the image prompt vector. K-1 Then three linear layers W are applied to the text prompt sequence P. k , W q , W v The word relevance vector K, word weight vector Q and word information vector V are converted; the word relevance vector K, word weight vector Q and word information vector V are then used to calculate the output sequence through a multi-head attention mechanism, and a mask mechanism is added to make it only pay attention to the text prompt vectors of the current layer and the previous layer when generating the image prompt vector; finally, the output sequence is concatenated and passed through a linear layer to generate an image prompt sequence According to the image prompt sequence Get image hint vector arrive in arrive Represents the image hint vector generated by the text hint vector through the multi-head attention mechanism.

[0045] In an alternative embodiment of the present invention, referring to Figure 2 In this embodiment, when the image data is converted into a block feature map using the graph segmentation module, the image data is divided into blocks according to a set size, and each image block is projected and mapped into a one-dimensional vector to obtain a block feature map. For example, the input image size is 224×224 and divided into blocks of 16×16 size. Among them, (224×224) / (16×16)=196, so 196 image blocks will be obtained, and the shape of each image block data is [16,16,3]. In order to be able to input into the image encoder network for feature extraction, each image block is projected and mapped into a one-dimensional vector, [16,16,3]->

[768] . In this way, the original [224,224,3] is converted to [196,768]. The above process can be achieved through convolution operation. By setting the convolution kernel size to 16x16 and the step size to 16x16, a tensor of size [14,14,768] can be obtained, which can be converted to a size of [196,768] by flattening.

[0046] This embodiment uses an image encoder network to perform multi-level attention encoding on block feature maps and image prompt vectors. When obtaining image codes, an image encoder network is constructed using multiple stacked encoding layers, and at each encoding layer, the input block feature map and the image prompt vector of the corresponding layer are concatenated and then layer-by-layer encoding calculations are performed to finally output the image code.

[0047] In this embodiment, when each coding layer concatenates the input block feature map and the image hint vector of the corresponding layer and performs layer-by-layer coding calculation, the concatenated vector of the input block feature map and the image hint vector of the corresponding layer is firstly subjected to layer normalization processing by the first image normalization layer, and then the third output vector is calculated by the multi-head attention mechanism, and the concatenated vector is added to the third output vector by the third residual connection; then the addition result is subjected to layer normalization processing by the fourth normalization layer, and then the fourth output vector is calculated by the multi-layer perceptron, and the addition result is added to the fourth output vector by the fourth residual connection to obtain the final calculation result.

[0048] When this embodiment calculates the third output vector through the multi-head attention mechanism, the layer normalized vector is transformed using three linear layers to obtain a word relevance vector, a word weight vector, and a word information vector; the word relevance vector, the word weight vector, and the word information vector are then used to calculate the output vector through the scaled dot product attention mechanism; finally, the output vectors are concatenated and passed through a linear layer to generate the third output vector.

[0049] In an alternative embodiment of the present invention, referring to Figure 2 In this embodiment, after obtaining the image encoding and text encoding respectively, multimodal contrastive learning is performed. During training, it is assumed that there are 64 text-image pairs in a batch, so that 64 image encodings and 64 text encodings can be obtained through the model. Such paired text-image encodings are natural positive samples, and they are paired. As for the text encoding of this sample, the other 63 image encodings are all negative samples, and they are unpaired. Similarly, as for the image encoding of this sample, the other 63 text encodings are all negative samples, and they are unpaired.

[0050] In this batch, there are 64 text-image pairs, and the sizes of the image encoding x and text encoding y that can be obtained are both [64, embedding_size], and the encodings of the same row of the two are paired. By cross-multiplying the two, we can get a [64,64] matrix, and the values ​​on the diagonal are the inner products of the paired features. The more similar the image encoding is to the corresponding text encoding, the larger its value will be. In this way, we can use cross entropy for training to maximize the values ​​on the diagonal and minimize other values. The cross entropy loss function is as follows:

[0051]

[0052] Among them, n represents the number of text image pairs, x i represents the i-th image code, y j represents the jth text encoding,

[0053] When testing, the cross product of the image code x and the text code y is calculated, and for each row, the text code corresponding to the maximum value is used as its predicted category. The prediction probability formula is as follows:

[0054]

[0055] in, Indicates that the label corresponding to the predicted image code x is The probability of the corresponding label, Indicates that the label is The corresponding text encoding.

[0056] The present invention enables text prompts at different layers to interact and generate image prompts through the attention mechanism, rather than only generating image prompts from text prompts at the corresponding layer, thus achieving richer contextual learning and greatly improving the accuracy of the model when performing multimodal learning tasks. A comparative experiment was conducted, and the average accuracy was improved by 0.8% compared with the case where this method was not used.

[0057] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0060] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0061] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A cross-layer image-text interaction prompt learning method based on attention mechanism, characterized in that: The following steps are involved: Get text data and image data; Constructing a cross-layer image-text interaction prompt learning network model; the cross-layer image-text interaction prompt learning network model includes a word embedding module, a text encoder network, a graph segmentation module, an image encoder network, a cross-layer interaction prompt learning module and a multimodal contrast learning module; Use the word embedding module to convert text data into word vectors; Use the text encoder network to perform multi-level attention encoding on word vectors and text prompt vectors to obtain text encoding; The cross-layer interactive cue learning module is used to generate image cue vectors using a multi-head attention mechanism on the text cue vectors. Use the graph segmentation module to convert image data into block feature maps; Use the image encoder network to perform multi-level attention encoding on the block feature map and the image prompt vector to obtain the image encoding; The multimodal contrastive learning module is used to perform multimodal contrastive learning on text encoding and image encoding.

2. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 1, it is characterized in that: When using the word embedding module to convert text data into word vectors, each word in the text data is mapped to a tensor according to the set rules, and the word vector is generated based on all the obtained tensors.

3. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 1, it is characterized in that: The text encoder network is used to perform multi-level attention encoding on word vectors and text prompt vectors. When obtaining the text encoding, multiple stacked encoding layers are used to concatenate the input word vector and the text prompt vector of the corresponding layer at each encoding layer, and then perform layer-by-layer encoding calculation to finally output the text encoding.

4. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 3, it is characterized in that: When each encoding layer concatenates the input word vector and the text prompt vector of the corresponding layer for layer-by-layer encoding calculation, the concatenated vector of the input word vector and the text prompt vector of the corresponding layer is first normalized by the first normalization layer, and then the first output vector is calculated by the multi-head attention mechanism, and the concatenated vector is added to the first output vector by the first residual connection; The addition result is then normalized by the second normalization layer, and the second output vector is calculated by the multilayer perceptron. At the same time, the addition result is added to the second output vector through the second residual connection to obtain the final calculation result.

5. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 4, it is characterized in that: When calculating the first output vector through the multi-head attention mechanism, the layer normalized vector is transformed by three linear layers to obtain the word relevance vector, word weight vector and word information vector; then the word relevance vector, word weight vector and word information vector are used to calculate the output vector through the scaled dot product attention mechanism; finally, the output vectors are concatenated and passed through a linear layer to generate the first output vector.

6. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 1, it is characterized in that: When the cross-layer interactive prompt learning module is used to generate the image prompt vector from the text prompt vector using the multi-head attention mechanism, the text prompt vectors corresponding to each encoding layer in the text encoder network are spliced ​​to obtain a text prompt sequence; then the text prompt sequence is transformed using three linear layers to obtain the word relevance vector, word weight vector and word information vector; then the word relevance vector, word weight vector and word information vector are used to calculate the output sequence through the multi-head attention mechanism; finally, the output sequence is spliced ​​and passed through a linear layer to generate an image prompt sequence, and the image prompt vector is obtained according to the image prompt sequence.

7. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 1, it is characterized in that: When the image data is converted into a block feature map using the graph segmentation module, the image data is divided into blocks according to a set size, and then each image block is projected and mapped into a one-dimensional vector to obtain a block feature map.

8. According to the attention mechanism-based cross-layer image-text interactive prompt learning method of claim 1, it is characterized in that: The image encoder network is used to perform multi-level attention encoding on the block feature map and the image prompt vector. When the image code is obtained, multiple stacked coding layers are used to concatenate the input block feature map and the image prompt vector of the corresponding layer at each coding layer, and then perform layer-by-layer coding calculation to finally output the image code.

9. The method for learning cross-layer image-text interaction prompts based on attention mechanism according to claim 8, characterized in that: When each encoding layer concatenates the input block feature map and the image hint vector of the corresponding layer and performs layer-by-layer encoding calculation, the concatenated vector of the input block feature map and the image hint vector of the corresponding layer is first normalized by the first image normalization layer, and then the third output vector is calculated by the multi-head attention mechanism, and the concatenated vector is added to the third output vector by the third residual connection; The addition result is then normalized by the fourth normalization layer, and the fourth output vector is calculated by the multilayer perceptron. At the same time, the addition result is added to the fourth output vector through the fourth residual connection to obtain the final calculation result.

10. The method for learning cross-layer image-text interaction prompts based on attention mechanism according to claim 9, characterized in that: When calculating the third output vector through the multi-head attention mechanism, the layer normalized vector is transformed by three linear layers to obtain the word relevance vector, word weight vector and word information vector; then the word relevance vector, word weight vector and word information vector are used to calculate the output vector through the scaled dot product attention mechanism; finally, the output vectors are concatenated and passed through a linear layer to generate the third output vector.

Images