An attention mechanism-based cross-layer image-text interaction prompt learning method

Abstract

The application relates to the technical field of multi-modal prompt learning, and discloses a cross-layer image-text interaction prompt learning method based on an attention mechanism, which comprises the following steps: acquiring text data and image data; constructing a cross-layer image-text interaction prompt learning network model; converting the text data into word vectors by using a word embedding module; performing multi-level attention coding on the word vectors and text prompt vectors by using a text encoder network to obtain text coding; generating an image prompt vector by using a multi-head attention mechanism on the text prompt vector by using a cross-layer interaction prompt learning module; converting the image data into block feature maps by using a graph splitting module; performing multi-level attention coding on the block feature maps and the image prompt vector by using an image encoder network to obtain image coding; and performing multi-modal contrast learning on the text coding and the image coding by using a multi-modal contrast learning module. The application can further improve the accuracy of multi-modal learning in executing tasks.

Description

Technical Field

[0001] This invention relates to the field of multimodal cue learning technology, specifically to a cross-layer image-text interaction cue learning method based on an attention mechanism. Background Technology

[0002] The combination of large-scale models and multimodal approaches has given rise to the CLIP (Contrastive Language-Image Pretraining) model. Released by OpenAI in 2021, this model was initially developed as a pre-trained neural network model for matching images and text. The CLIP model consists of a text encoder and an image encoder, which encode the text and image respectively. Then, by comparing the similarity between the image and text encodings, it maximizes the similarity between relevant samples and minimizes the similarity between irrelevant samples, thus obtaining the pre-trained CLIP model.

[0003] Cue learning, based on the CLIP model, learns task-specific cues in an end-to-end manner, adapting the CLIP model to downstream tasks. Existing cue learning methods initially only learn cues on the text side. For example, CoOp models the context words of text cues using learnable vectors, while keeping the entire pre-trained parameters fixed, adapting to the task by training the text-side cue vectors. Although CoOp is simple and effective, this leads to overfitting the model to categories it has seen during training, while exhibiting poor generalization ability to unseen categories in the same domain. To overcome this problem, CoCoOp proposes conditional context optimization, extending CoOp by learning a lightweight meta-neural network.

[0004] Subsequent research employed multimodal cue learning, such as MaPLe, which introduces learnable cue vectors on both the text and image sides, and learns individual cue vectors in the early layers of both the text and image encoders, progressively modeling the relationships between stage-specific features to achieve rich contextual learning. A coupling function was also proposed to jointly learn cue vectors from both modalities; specifically, the text cue vector is projected onto the image cue vector through a linear layer. However, existing multimodal cue learning methods only enable interaction between the image and text branches at each layer, lacking coordinated interaction between layers. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a cross-layer image-text interaction prompt learning method based on an attention mechanism. This method solves the problem that existing technologies involve separate interaction between image prompt vectors and text prompt vectors at each layer, lacking coordinated interaction between layers.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A cross-layer image-text interaction prompt learning method based on attention mechanism includes the following steps:

[0008] Acquire text and image data;

[0009] A cross-layer image-text interaction prompt learning network model is constructed; the cross-layer image-text interaction prompt learning network model includes a word embedding module, a text encoder network, a graph segmentation module, an image encoder network, a cross-layer interaction prompt learning module, and a multimodal contrast learning module;

[0010] The word embedding module is used to convert text data into word vectors;

[0011] The text encoding is obtained by using a text encoder network to perform multi-level attention encoding on word vectors and text prompt vectors;

[0012] The cross-layer interactive prompt learning module is used to generate image prompt vectors from text prompt vectors using a multi-head attention mechanism.

[0013] The image data is converted into block feature maps using the graph segmentation module;

[0014] The image encoder network is used to perform multi-level attention encoding on the block feature map and the image cue vector to obtain the image code;

[0015] A multimodal contrastive learning module is used to perform multimodal contrastive learning on text encoding and image encoding.

[0016] Preferably, when converting text data into word vectors using the word embedding module, each word in the text data is mapped to a tensor according to a set rule, and word vectors are generated based on all the obtained tensors.

[0017] Preferably, a text encoder network is used to perform multi-level attention encoding on word vectors and text prompt vectors. When obtaining the text encoding, multiple stacked encoding layers are used. In each encoding layer, the input word vectors and the corresponding text prompt vectors are concatenated and then encoded layer by layer to finally output the text encoding.

[0018] Preferably, when performing layer-by-layer encoding calculations after concatenating the input word vector and the corresponding layer's text prompt vector at each encoding layer, the concatenated vector is first processed by a first normalization layer for layer normalization, then the first output vector is calculated using a multi-head attention mechanism, and simultaneously the concatenated vector is added to the first output vector through a first residual connection; then the added result is processed by a second normalization layer for layer normalization, then the second output vector is calculated using a multilayer perceptron, and simultaneously the added result is added to the second output vector through a second residual connection to obtain the final calculation result.

[0019] Preferably, when calculating the first output vector through the multi-head attention mechanism, the layer normalization vector is transformed by three linear layers to obtain the word relevance vector, word weight vector, and word information vector; then the word relevance vector, word weight vector, and word information vector are used to calculate the output vector through the scaling dot product attention mechanism; finally, the output vectors are concatenated and then passed through a linear layer to generate the first output vector.

[0020] Preferably, when using the cross-layer interactive prompt learning module to generate image prompt vectors from text prompt vectors using a multi-head attention mechanism, the text prompt vectors corresponding to each encoding layer in the text encoder network are concatenated to obtain a text prompt sequence. Then, the text prompt sequence is transformed using three linear layers to obtain word relevance vectors, word weight vectors, and word information vectors. The word relevance vectors, word weight vectors, and word information vectors are then used to calculate the output sequence through a multi-head attention mechanism. Finally, the output sequences are concatenated and passed through linear layers to generate an image prompt sequence, and the image prompt vector is obtained based on the image prompt sequence.

[0021] Preferably, when converting image data into a block feature map using the graph segmentation module, the image data is divided into blocks of a set size, and each image block is then projected onto a one-dimensional vector to obtain the block feature map.

[0022] Preferably, an image encoder network is used to perform multi-level attention encoding on the block feature map and the image cue vector. When obtaining the image encoding, multiple stacked encoding layers are used. In each encoding layer, the input block feature map and the corresponding layer's image cue vector are concatenated and then encoded layer by layer to finally output the image encoding.

[0023] Preferably, when performing layer-by-layer encoding calculations after concatenating the input block feature map and the corresponding layer's image cue vector at each encoding layer, the concatenated vector of the input block feature map and the corresponding layer's image cue vector is first processed by the first image normalization layer for layer normalization, then the third output vector is calculated through a multi-head attention mechanism, and the concatenated vector is added to the third output vector through the third residual connection; then the addition result is processed by the fourth normalization layer for layer normalization, and the fourth output vector is calculated through a multilayer perceptron, and the addition result is added to the fourth output vector through the fourth residual connection to obtain the final calculation result.

[0024] Preferably, when calculating the third output vector through the multi-head attention mechanism, the layer normalization vector is transformed by three linear layers to obtain the word relevance vector, word weight vector and word information vector; then the word relevance vector, word weight vector and word information vector are used to calculate the output vector through the scaling dot product attention mechanism; finally, the output vectors are concatenated and then used to generate the third output vector through a linear layer.

[0025] The present invention has the following beneficial effects:

[0026] This invention addresses the problem of non-interactive prompts between layers by proposing a cross-layer image-text interaction prompt learning method based on an attention mechanism. This method enables prompt vectors from different layers to interact, thereby learning more effective prompt vectors for the current task. Furthermore, it can generate more dimensional image prompt vectors, further improving the accuracy of multimodal learning tasks. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a cross-layer image-text interaction prompt learning method based on an attention mechanism;

[0028] Figure 2 A schematic diagram of the learning network model structure for cross-layer image-text interaction prompts. Detailed Implementation

[0029] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0030] This invention addresses the problem of non-interactive prompts between layers by proposing a cross-layer image-text interaction prompt learning method based on an attention mechanism. This method enables prompt vectors from different layers to interact, thereby learning more effective prompt vectors for the current task. Furthermore, it can generate more dimensional image prompt vectors, further improving the accuracy of multimodal learning tasks.

[0031] Reference Figure 1 This embodiment provides a cross-layer image-text interaction prompt learning method based on an attention mechanism, including the following steps S1 to S8:

[0032] S1. Obtain text and image data;

[0033] S2. Construct a cross-layer image-text interaction prompt learning network model; the cross-layer image-text interaction prompt learning network model includes a word embedding module, a text encoder network, a graph segmentation module, an image encoder network, a cross-layer interaction prompt learning module, and a multimodal contrast learning module;

[0034] S3. Use the word embedding module to convert text data into word vectors;

[0035] S4. Use a text encoder network to perform multi-level attention encoding on word vectors and text prompt vectors to obtain text encoding;

[0036] S5. Use the cross-layer interactive prompt learning module to generate image prompt vectors from text prompt vectors using a multi-head attention mechanism;

[0037] S6. Use the graph segmentation module to convert the image data into block feature maps;

[0038] S7. Use an image encoder network to perform multi-level attention encoding on the block feature map and image cue vector to obtain the image code;

[0039] S8. Use the multimodal contrastive learning module to perform multimodal contrastive learning on text encoding and image encoding.

[0040] In an optional embodiment of the present invention, reference is made to... Figure 2 In this embodiment, when converting text data into word vectors using the word embedding module, a pre-trained word embedding module is used to map each word in the text data into a tensor according to a set rule, and word vectors are generated based on all the obtained tensors.

[0041] In an optional embodiment of the present invention, reference is made to... Figure 2 This embodiment utilizes a text encoder network to perform multi-level attention encoding on word vectors and text prompt vectors. To obtain the text encoding, the text encoder network is constructed using multiple stacked encoding layers. At each encoding layer, the input word vectors and the corresponding layer's text prompt vectors are concatenated and then encoded layer by layer, ultimately outputting the text encoding. Where P0 to P... K-1 This represents the text prompt vector for each layer from layer 0 to layer K-1, where K indicates that the prompt vector is added in the first K layers. It is initialized with a randomly selected vector of size [K, text_dim] that follows a normal distribution with a mean of 0 and a standard deviation of 0.02. Each row represents the prompt vector for the corresponding layer, and the length of the prompt vector is text_dim.

[0042] In this embodiment, when the input word vector and the corresponding text prompt vector of each layer are concatenated and then encoded layer by layer, the concatenated vector is first processed by a first normalization layer for layer normalization, and then the first output vector is calculated by a multi-head attention mechanism. At the same time, the concatenated vector is added to the first output vector through a first residual connection. Then, the addition result is processed by a second normalization layer for layer normalization, and then the second output vector is calculated by a multilayer perceptron. At the same time, the addition result is added to the second output vector through a second residual connection to obtain the final calculation result.

[0043] In this embodiment, when calculating the first output vector using a multi-head attention mechanism, the layer normalization vector is transformed by three linear layers to obtain the word relevance vector, word weight vector, and word information vector; then, the word relevance vector, word weight vector, and word information vector are used to calculate the output vector through a scaling dot product attention mechanism; finally, the output vectors are concatenated and then passed through a linear layer to generate the first output vector.

[0044] In an optional embodiment of the present invention, reference is made to... Figure 2 In this embodiment, when generating image cue vectors from text cue vectors using a multi-head attention mechanism via a cross-layer interactive cue learning module, the text cue vectors P0 to P0 corresponding to each encoding layer in the text encoder network are processed. K-1 The text prompt sequence P is obtained by concatenation; then, three linear layers W are applied to the text prompt sequence P. k W q W v The transformation yields a word relevance vector K, a word weight vector Q, and a word information vector V. These vectors are then used to calculate the output sequence via a multi-head attention mechanism. A masking mechanism is added to ensure that the generation of image cue vectors only considers the text cue vectors of the current and previous layers. Finally, the output sequences are concatenated and passed through a linear layer to generate the image cue sequence. Based on the image prompt sequence Obtain image cue vector arrive in arrive This represents the image cue vector generated by the text cue vector through a multi-head attention mechanism.

[0045] In an optional embodiment of the present invention, reference is made to... Figure 2 In this embodiment, when the image data is converted into a block feature map using the image segmentation module, the image data is divided into blocks of a set size, and then each image block is projected and mapped into a one-dimensional vector to obtain the block feature map. For example, the input image size of 224×224 is divided into blocks of size 16×16. Where (224×224) / (16×16)=196, this will result in 196 image blocks, and the shape of each image block data is [16,16,3]. In order to be input into the image encoder network for feature extraction, each image block is projected and mapped into a one-dimensional vector, changing [16,16,3]->

[768] . In this way, the original [224,224,3] is transformed into [196,768]. The above process can be achieved through convolution operations. By setting the convolution kernel size to 16x16 and the stride to 16x16, a tensor of size [14,14,768] can be obtained. Then, by tiling it, it can be transformed into a tensor of size [196,768].

[0046] This embodiment utilizes an image encoder network to perform multi-level attention encoding on the block feature map and image cue vector. When obtaining the image encoding, the image encoder network is constructed using multiple stacked encoding layers. In each encoding layer, the input block feature map and the corresponding layer's image cue vector are concatenated and then encoded layer by layer to finally output the image encoding.

[0047] In this embodiment, when the input block feature map and the corresponding layer's image cue vector are concatenated at each coding layer for layer-by-layer encoding calculation, the concatenated vector of the input block feature map and the corresponding layer's image cue vector is first processed by the first image normalization layer for layer normalization, and then the third output vector is calculated through a multi-head attention mechanism. At the same time, the concatenated vector is added to the third output vector through the third residual connection. Then, the addition result is processed by the fourth normalization layer for layer normalization, and then the fourth output vector is calculated through a multilayer perceptron. At the same time, the addition result is added to the fourth output vector through the fourth residual connection to obtain the final calculation result.

[0048] In this embodiment, when calculating the third output vector using a multi-head attention mechanism, the layer normalization vector is transformed using three linear layers to obtain the word relevance vector, word weight vector, and word information vector; then, the word relevance vector, word weight vector, and word information vector are used to calculate the output vector through a scaling dot product attention mechanism; finally, the output vectors are concatenated and then passed through a linear layer to generate the third output vector.

[0049] In an optional embodiment of the present invention, reference is made to... Figure 2 In this embodiment, after obtaining the image encoding and text encoding respectively, multimodal contrastive learning is performed. During training, it is assumed that there are 64 text-image pairs in a batch, so the model can obtain 64 image encodings and 64 text encodings. The paired text-image encodings are naturally positive samples; they are paired. For the text encoding of this sample, the other 63 image encodings are negative samples; they are not paired. Similarly, for the image encoding of this sample, the other 63 text encodings are negative samples; they are not paired.

[0050] In this batch of 64 text-image pairs, the available image encoding x and text encoding y are both [64, embedding_size], and the encodings in the same row are paired. Performing a cross product of the two yields a [64, 64] matrix. The values ​​on the diagonal are obtained from the pairwise feature inner product; the more similar the image encoding and its corresponding text encoding, the larger its value. This allows for training using cross-entropy to maximize the values ​​on the diagonal and minimize the other values. The cross-entropy loss function is as follows:

[0051]

[0052] Where n represents the number of text-image pairs, x i y represents the encoding of the i-th image. j This represents the encoding of the j-th text.

[0053] During testing, the cross product of the obtained image code x and text code y is calculated. For each row, the text code corresponding to the maximum value is taken as the predicted category. The prediction probability formula is as follows:

[0054]

[0055] in, This indicates that the label corresponding to the predicted image code x is The probability of the corresponding label. Indicates that the label is The corresponding text encoding.

[0056] This invention enables text prompts at different layers to interact and generate image prompts through an attention mechanism, rather than relying solely on text prompts at the corresponding layer to generate image prompts. This allows for richer contextual learning and significantly improves the model's accuracy when performing multimodal learning tasks. Comparative experiments show that compared to methods not using this approach, the average accuracy is improved by 0.8%.

[0057] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0061] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An attention mechanism-based cross-layer image-text interaction prompt learning method, characterized in that, The method comprises the following steps: Obtaining text data and image data; Building a cross-layer image-text interaction prompt learning network model; the cross-layer image-text interaction prompt learning network model comprises a word embedding module, a text encoder network, a graph cutting module, an image encoder network, a cross-layer interaction prompt learning module and a multi-modal contrast learning module; Converting the text data into a word vector by using the word embedding module; Performing multi-level attention coding on the word vector and the text prompt vector by using the text encoder network to obtain text coding; Generating an image prompt vector by using a multi-head attention mechanism on the text prompt vector by using the cross-layer interaction prompt learning module; Converting the image data into a block feature graph by using the graph cutting module; Performing multi-level attention coding on the block feature graph and the image prompt vector by using the image encoder network to obtain image coding; Performing multi-modal contrast learning on the text coding and the image coding by using the multi-modal contrast learning module; When generating the image prompt vector by using the multi-head attention mechanism on the text prompt vector by using the cross-layer interaction prompt learning module, the text prompt vector corresponding to each coding layer in the text encoder network is spliced to obtain a text prompt sequence; then, the text prompt sequence is converted into a word correlation vector, a word weight vector and a word information vector by using three linear layers; then, the word correlation vector, the word weight vector and the word information vector are calculated to output a sequence by using a multi-head attention mechanism; finally, the image prompt sequence is generated by splicing the output sequence through a linear layer, and the image prompt vector is obtained according to the image prompt sequence.

2. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 1, characterized in that, When converting the text data into a word vector by using the word embedding module, each word in the text data is mapped into a tensor according to a set rule, and the word vector is generated according to all the tensors obtained.

3. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 1, characterized in that, When performing multi-level attention coding on the word vector and the text prompt vector by using the text encoder network to obtain text coding, a plurality of stacked coding layers are used to splice the input word vector and the text prompt vector of the corresponding layer after each coding layer, and then perform layer-by-layer coding calculation, and finally output the text coding.

4. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 3, characterized in that, When splicing the input word vector and the text prompt vector of the corresponding layer after each coding layer and then performing layer-by-layer coding calculation, the spliced vector of the input word vector and the text prompt vector of the corresponding layer is first subjected to layer normalization processing through a first normalization layer, and then a first output vector is calculated through a multi-head attention mechanism, and at the same time, the spliced vector and the first output vector are added through a first residual connection; Then, the addition result is subjected to layer normalization processing through a second normalization layer, and then a second output vector is calculated through a multi-layer perceptron, and at the same time, the addition result and the second output vector are added through a second residual connection to obtain the final calculation result.

5. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 4, characterized in that, When calculating the first output vector through the multi-head attention mechanism, the layer normalization vector is converted into a word correlation vector, a word weight vector and a word information vector through three linear layers; then, the word correlation vector, the word weight vector and the word information vector are calculated to output a vector through a scaled dot-product attention mechanism; finally, the output vector is spliced to generate the first output vector through a linear layer.

6. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 1, characterized in that, When the image data is converted into the block feature map by the image segmentation module, the image data is divided into blocks according to a set size, and then each image block is projected and mapped into a one-dimensional vector to obtain the block feature map.

7. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 1, characterized in that, When the image encoder network is used to perform multi-level attention coding on the block feature map and the image prompt vector to obtain the image code, a plurality of stacked coding layers are used, and in each coding layer, the input block feature map and the image prompt vector of the corresponding layer are spliced and then coded layer by layer to finally output the image code.

8. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 7, characterized in that, When the input block feature map and the image prompt vector of the corresponding layer are spliced and then coded layer by layer in each coding layer, the spliced vector of the input block feature map and the image prompt vector of the corresponding layer is first subjected to layer normalization processing by a third normalization layer, and then a third output vector is calculated by a multi-head attention mechanism, and at the same time, the spliced vector and the third output vector are added through a third residual connection. Then, the addition result is subjected to layer normalization processing by a fourth normalization layer, and a fourth output vector is calculated by a multi-layer perception, and at the same time, the addition result and the fourth output vector are added through a fourth residual connection to obtain the final calculation result.

9. The cross-layer image-text interaction prompt learning method based on an attention mechanism according to claim 8, characterized in that, When the third output vector is calculated by the multi-head attention mechanism, the layer normalization vector is converted into a word relevance vector, a word weight vector and a word information vector by three linear layers; then the word relevance vector, the word weight vector and the word information vector are calculated by a scaled dot-product attention mechanism to obtain an output vector; finally, the output vectors are spliced and then passed through a linear layer to generate the third output vector.

Images