Method for generating text from image

By using ViTPatchEmbeddings and a shared K and V parameter caching method, an end-to-end image-to-text generation model is constructed, which solves the problem of low efficiency in existing technologies and achieves efficient image-to-text generation tasks.

CN119048644APending Publication Date: 2024-11-29GANSU WANWEI INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411244056.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing methods for generating text from images are inefficient and lose spatial location information of the image, making it difficult to achieve efficient text generation from images.

Method used

The ViTPatchEmbeddings module is directly concatenated into the text TokenEmbeddings to build an end-to-end image-to-text model. By improving the structure of the GPT model, a caching method for shared K and V parameters is implemented to reduce computational resource consumption.

Benefits of technology

It achieves efficient image-to-text generation, reduces computational resource consumption, and improves the efficiency and accuracy of image-to-text generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048644A_ABST
    Figure CN119048644A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of AI data processing, in particular to a method for generating a text from an image. An end-to-end image text generation task is provided, and end-to-end is mainly embodied in that an image does not need to pass through any pre-trained neural network module, such as CLIP, VGGNet and other neural networks, and only needs to be directly spliced with text TokenEmdings after PatchEmdings, and then the PatchEmdings and the text TokenEmdings are input into a language model. According to the method, the utilization of the PatchEmbedding on the image space is fully utilized, and the technical feasibility is verified through actual image-text data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of AI data processing technology, and in particular to a method for generating text from an image. Background Art

[0002] Image-to-text has a wide range of application scenarios in various industries. The general method of image-to-text is to obtain the vector representation of the image through the open source CLIP, and then splice the vector representation of the image and the token embedding representation of the text through full connection, and input it into the text generation model such as GPT for training, so as to realize the function of image-to-text. However, CLIP is a kind of training based on contrastive learning of massive image and text data. The CLIP model itself has a certain amount of parameters, and the model size is usually hundreds of megabytes to several G. In other words, in the image-to-text process, the image needs to pass through the CLIP model first. Although this utilizes the existing results of CLIP, the reasoning efficiency is low; there is another implementation scheme, that is, by inputting the image into the image-related neural network, such as VGGNet, and then obtaining the vector representation of the image, and then splicing it with the token vector representation of the text, the model of image-to-text generation is trained, but the spatial position information of the image is usually lost. This patent proposes a method of directly splicing PatchEmbedding in Vit (referred to as ViTPatchEmbeddings module in this patent) into the Embedding of text token, and inputting them into the GPT model together to realize an end-to-end image-to-text model. The ViTPatchEmbeddings module has a model parameter amount to be optimized (the parameter amount is much smaller than that of trained models such as CLIP and VGGNet). At the same time, the image information is established through patches one by one. The PatchEmbeddings and the TokenEmbeddings of the text are directly spliced ​​together to realize the unification of image and text modalities. In this way, the image is equivalent to a sequence of tokens with the sequence information of spatial image blocks. Summary of the invention

[0003] In the direction of ecological environment protection detection, it is often necessary to determine whether a certain place is suitable for building relevant chemical factories. Usually, on-site implementers need to take pictures and send them to experts in the ecological industry for specific suggestions. This operation process is usually extremely difficult and inefficient. If it is possible to use artificial intelligence technology to generate pictures, that is, to output specific implementation suggestions for building chemical factories, it will be of great significance. That is, through the content initially generated by the trained model for pre-judgment, this can improve the work efficiency of on-site personnel and industry experts. At the same time, the image-to-text language models in the industry usually require CLIP and VGGNet as image inputs to train the image-to-text language model, which is not necessary at the theoretical and technical level. In order to confirm the technical feasibility of deleting CLIP and VGGNet and directly training the image-to-text with ViTPatchEmbeddings and TokenEmbeddings, this patent is proposed. As the effects of large models are getting better and the amount of data is increasing, the optimization of the GPT model itself is becoming more and more important. Among them, the KV Cache technology is an important part of the GPT inference process. The core of the KV Cache technology is the idea of trading space for time, that is, reducing the calculation time by occupying more storage space. This patent optimizes the GPT model and realizes a caching method that shares K and V parameters by improving the structure of the GPT model, reducing the content of each layer of K and V caches in the KV Cache, and thus reducing the consumption of computing resources.

[0004] An image generation text method, comprising the following steps: S1. Collect image-text training set pairs and create a vocabulary Usually, training an image-to-text model requires a large number of image-text dataset pairs. If it is necessary to train a general-purpose and delicate image-to-text model, create a folder named pngs and place images in it. At the same time, place a labels.txt in this training set to store the image names in pngs and the corresponding model text outputs. At the same time, collect the tokens of open-source large language models, delete low-frequency tokens, and create a vocabulary for the model tokens, that is, tokens_Dict={0:‘[CLS]’, 1:”[PAD]”, 2:”[SEP]“, 3:”[NewLine]”, 4:”中”.......}. Take 30,000 tokens. Among them, [CLS] is the text start symbol, [PAD] means that when the text length does not reach the maximum length, it is padded with [PAD] to the max_len length, [SEP] is the separator in the text generation task, [NewLine] is the text line break symbol, and max_len=768 is the maximum length of the language model agreed upon by the input model; S2. Image and text modality adaptation Image and text are two modalities, that is, two different modal tensor forms. The CLIP multimodal alignment model or the encoder trained with related massive corpus is used. Even if a token without physical meaning is represented as a vector, the image is patched, that is, the image is cut into patches, each patch is represented by a 2048-dimensional vector, and these patches are equivalent to sequences. They are directly spliced ​​with the TokenEmbedding of the text and trained on the basis of the large language model. The vector representation of the image patch and the vector representation of the text token are directly spliced ​​together and input into the GPT model. S3. Constructing the graph-text network model structure A text is input into the language generation model as a tensor of size [16, 768, 2048] obtained through step S1, where 2048 is the dimension represented by each image patch or text token, 16 is the batch size, and 768 is the model training length; the causal attention module and the feedforward fully connected module are respectively repeated 16 times as a unit to form the neural network model to be trained, that is, first passing through 16 feedforward fully connected sublayers with different parameters and the same structure, and then passing through 16 causal attention sublayers with exactly the same structure but different parameters. The output of the feedforward fully connected module has three units, namely the key tensor, the value tensor, and the query tensor. The query tensor is transmitted in the subsequent causal attention module, and the value tensor and the key tensor share parameters in all causal attention modules and interact with the query tensor which is completely different in each attention sublayer; S4. Image-to-text model training According to the model structure set in step S3, the Adam optimizer and the cross entropy loss function are used for model training. The model input data obtained in step S2 is imported into the network structure obtained in step S3. The ultimate goal of the model is to minimize the loss function. For the image-to-text model, the loss function does not need to be calculated for the output of the language model at the corresponding position of the image PatchEmbeddings. It only needs to correspond to the cross entropy loss function from the 197th position to the end of the language model. That is, the model is trained to output the first text Token predicted from the image as much as possible from the 197th position, and the first real text token will predict the second text Token at the 198th position on the model input side. S5. Model prediction If the current user inputs an image, it is divided into patches through the PatchEmbeddings algorithm, and a vector is assigned to each patch. Then the CLS vector and the image PatchEmbeddings are concatenated and input into the GPT language model. According to the index of the maximum output at the 197th position, the corresponding Token is found through the index to complete the decoding. The Token is then spliced ​​to the model input side again, and the model output predicts the second Token. By analogy, the final image-to-text generation task can be completed. During the reasoning process, after each inference token passes through the embedding layer and multiple feed-forward layers, the key and value tensors are gradually cached, and the query tensor is transmitted in the causal attention sublayer and the shared cache tensors key tensor and value tensor are interacted with for attention.

[0005] The S2 step further includes: For images using ViTPatchEmbeddings and without training, the GPT model supports an input length of 768; for example, in a batch of batch_size=16 image-text pairs, the images are denoted as Pics and the texts are denoted as Txts. The images in this batch of Pics are first deformed to [16, 3, 224, 224], that is, they have three channels of RGB, and the width and height are both 224. Then, the convolution operation is nn.Conv2d(num_channels, hidden_size, kernel_size=patch_size, stride=patch_size), with patch_size=16, hidden_size=2048, num_channels=3. After the above convolution operation, the size is [16, 3, 224, 224] and the tensor is [16, 2048, 14, 14], that is, the image is divided into 14*14 small blocks, and the vector representation dimension of each block is 2048; the last two dimensions are flattened and exchanged, and the tensor representation of the batch of images is now [16, 196, 2048]. At this time, the dimension of a patch representation of an image is the same as the dimension of a token representation of a text; the maximum length of the text input to the model is agreed to be 768, and the tensor representation shape of the input model of a batch size is [16, 768, 2048], where the tensor representation size of the image mode is [16, 196, 2048] and the tensor representation size of the text mode is [16, 571, 2048]. When the text length is not enough, the [PAD] length supplement character is used; the tensor of the start character [CLS] The vector representation is [16, 1, 2048]; the start symbol [CLS] tensor representation, image patch tensor representation, and text token tensor representation are concatenated together in sequence, and the tensor size of the input GPT model is [16, 768, 2048]; the corresponding GPT model output tensor Y size is [16, 768, 30000], where 30000 is the number of all tokens; that is, the input image is divided into multiple patches, each patch is regarded as a text token, and the vector representation of the image patch and the vector representation of the text are both 2048, so that the GPT model accepts both image input and text input modes at the same time. It only needs the image to pass through the patch to train the image-to-text model end-to-end.

[0006] Beneficial effects: One of the innovations of this patent is reflected in the proposal of an end-to-end image generation text task. The end-to-end mainly means that the image does not need to go through any pre-trained neural network modules, such as neural networks like CLIP and VGGNet. After only PatchEmbeddings, it can be directly concatenated with text TokenEmbeddings and input into the language model. This patent makes full use of the utilization of the image space by PatchEmbedding and verifies the technical feasibility through actual image-text data pairs.

[0007] The main innovation of this patent lies in the proposal of a method for sharing key and value tensors in a multi-layer sub-attention network, that is, multiple feed-forward sub-layers with the same structure but different parameters form a unit, and multiple causal attention sub-layers with the same structure but different parameters form a unit. This is more conducive to the realization of the key tensor and value tensor sharing method, that is, the query tensor performs attention interaction with the shared key tensor and value tensor. Under the same model structure as GPT, Llama3, etc., if there are N layers, the technology proposed in this patent can reduce the KV Cache cache content by 1 / N, greatly reducing the waste of resources. For ultra-long text generation, the advantages of this patent are more obvious. Description of the Drawings

[0008] Figure 1 It is a schematic diagram of the Patch of the picture of the present invention; Figure 2 It is a schematic diagram of the model process of the present invention. Detailed Embodiments

[0009] S1. Collect image-text training set pairs and create a vocabulary: S1 described above is the prior art. Usually, training an image-to-text generation model requires a large number of image-text dataset pairs. If you need to train a general-purpose and delicate image-to-text generation model, this patent recommends that the data volume be more than 10 million. Create a folder pngs and place images in pngs; at the same time, place a labels.txt in this training set to store the image names in pngs and the corresponding model text outputs. At the same time, collect the tokens of the open-source large language model, delete the low-frequency tokens, and create a vocabulary for the model tokens, that is, tokens_Dict={0:‘[CLS]’, 1:”[PAD]”, 2:”[SEP]“, 3:”[NewLine]”, 4:”中”.......}, with a total of 30,000 tokens. Among them, [CLS] is the text start symbol, [PAD] means that when the text length does not reach the maximum length, [PAD] is used to fill it to the max_len length, [SEP] is the separator in the text generation task, [NewLine] is the text line break symbol, and max_len=768 is the maximum length of the language model agreed upon by the input model.

[0010] S2. Image and text modality adaptation: As two modalities, images and text are in the form of tensors of two different modalities. The industry generally adopts the CLIP multimodal alignment model or encoders trained with related massive corpora. However, the success of large language models tells us that even if a token without physical meaning is represented as a vector, it will have a good effect after a lot of training. Images naturally have physical meanings, so the patched PatchEmbedding ( Figure 1That is, the schematic diagram of the image patch, that is, cutting the image into patch blocks, each block is represented by a 2048-dimensional vector, and these blocks are equivalent to a sequence) and the TokenEmbedding of the text can be directly spliced ​​and trained on the basis of the large language model. The vector representation of the image patch and the text token vector representation of this patent are directly spliced ​​together and input into the GPT model. For images, only ViTPatchEmbeddings are used, and they have not been trained. The input length supported by the GPT model is 768. Suppose a batch (batch_size=16) of image-text pairs is denoted as Pics and the text is denoted as Txts. The images in this batch of Pics are first deformed and unified into [16, 3, 224, 224], that is, they have three channels of RGB, and the width and height are both 224. Then they are convolved with the operation nn.Conv2d(num_channels, hidden_size, kernel_size=patch_size, stride=patch_size) with patch_size=16, hidden_size=2048, num_channels=3. After the above convolution operation, the size is [16, 3, 224, 224] and the tensor is [16, 2048, 14, 14]. That is, the image is divided into small blocks of 14*14, and the vector representation dimension of each block is 2048. Flatten the last two dimensions and swap the last two dimensions. At this time, the tensor representation of the batch of images is [16, 196, 2048]. At this time, the dimension of a patch representation of the image is the same as the dimension of a token representation of the text. This patent stipulates that the maximum length of the text input to the model is 768. The tensor representation shape of the input model of a batch size is [16, 768, 2048]. The tensor representation size of the image modality is [16, 196, 2048], the tensor representation of the text modality is [16, 571, 2048] (When the text length is not enough, the [PAD] length supplement character is used.), and the tensor representation of the start character [CLS] is [16, 1, 2048]. The tensor representation of the start character [CLS], the image patch tensor representation, and the text token tensor representation are spliced ​​together in sequence, and the tensor size of the input GPT model is [16, 768, 2048]. The size of the corresponding GPT model output tensor Y is [16, 768, 30000], where 30000 is the number of all tokens. The input image is divided into multiple patches, each patch is regarded as a text token, and the vector representation of the image patch and the vector representation of the text are both 2048 in dimension, so that the GPT model can accept both image input and text input. This is completely different from the industry's image-to-text model. This work only needs the image to pass through the patch to train the image-to-text model end-to-end.

[0011] S3. Constructing a graph-to-text network model structure: S3 is the main innovation of this patent. The main innovation of this patent is to transform the model structure of GPT, aiming to be more conducive to the implementation of a KV Cache caching method. Specifically, for a text X in the corpus of the project, the size of the tensor obtained by step S1 is [16, 768, 2048] and input into the language generation model. 2048 is the dimension represented by each image patch or text token, 16 is the batch size, and 768 is the model training length. The model structure of this patent is different from that of GPT, Llama3 and other models in that this patent repeats the causal attention module and the feedforward fully connected module as a unit 16 times to form a neural network model to be trained, that is, it first passes through 16 feedforward fully connected sub-layers with different parameters and the same structure, and then passes through 16 causal attention sub-layers with exactly the same structure and different parameters. The output of the feedforward fully connected module has three units, namely the key tensor, the value tensor, and the query tensor. The query tensor is transmitted in the subsequent causal attention module. The value tensor and the key tensor share parameters in all causal attention modules and interact with the query tensor which is completely different in each attention sub-layer. The residual and variance normalization are the same as the language model structure of GPT, Llama3 and other language models. The above are the main innovations of this patent. The reasons for the improvement of this patent are as follows: In the KV Cache reasoning process, when the KV Cache technology has many layers, large embedding dimensions, and long reasoning length, the KV Cache will gradually increase the cache content. In order to reduce the occupation of computing resources by the KV Cache cache content, this patent adopts all causal attention sub-layers to share Key and Value tensors, and the query tensor is transmitted in the causal attention matrix. However, the shared causal attention sub-layer is extremely difficult to implement in the original GPT, Llama3 and other language model structures. Therefore, this patent uses multiple attention sub-layers stacked together and multiple feedforward layers stacked together. The advantage is that all feedforward layers and embedding layers are unified together to form a language token representation layer, and the causal attention sub-layer is combined with the subsequent layers. The output key, value, and query tensors of the feedforward layer and the embedding layer are conducive to parallel training and serial reasoning. At the same time, the parameter amount of the feedforward layer and the attention layer of this patent can be maintained with the parameter amount of the original model structure, and the KV Cache cache can be reduced without reducing the capacity of the model, that is, under the premise of ensuring the model effect, computing resources can be saved.

[0012] S4, graph-generated text model training: According to the model structure set in step S3, the Adam optimizer and cross-entropy loss function are used for model training. This is an existing technology. The specific training process is to feed the model input data obtained in step S2 into the network structure obtained in step S3. The ultimate goal of the model is to minimize the loss function. Because this patent belongs to a graph-generated text model, the loss function does not need to be calculated for the output of the language model at the corresponding position of the image PatchEmbeddings. It only needs to correspond to the cross-entropy loss function from the 197th position to the end of the language model. That is, the model is trained to output the first text token predicted from the image as much as possible from the 197th position, and the first real text token will predict the second text token at the 198th position on the model input side, and so on. The way S4 calculates the loss function is the same as the large language model technology.

[0013] S5. Model prediction: If the current user inputs an image, it is divided into patches through the PatchEmbeddings algorithm, and a vector is assigned to each patch. Then the CLS vector and the image PatchEmbeddings are concatenated and input into the GPT language model. According to the index of the maximum output at the 197th position, the corresponding Token is found through the index to complete the decoding. The Token is then concatenated to the model input side again, and the model output predicts the second Token. By analogy, the final image-to-text generation task can be completed. During the reasoning process, after each inference token passes through the embedding layer and multiple feedforward layers, the key and value tensors are gradually cached, and the query tensor is transmitted in the causal attention sublayer and the shared cache tensors key tensor and value tensor are interacted with for attention.

[0014] Glossary Image-to-text: Image-to-text refers to the use of computer vision and natural language processing technology to convert an image into a text description of the image content. This task combines the technologies in the fields of computer vision and natural language processing to enable computers to understand the content of an image and generate corresponding descriptions in natural language.

[0015] GPT language model: A large language model is a model with a large number of model parameters, usually more than one billion, and usually has high language understanding and analysis capabilities, but a slow reasoning speed; a small language model is a model with a small number of model parameters, usually has poor understanding and reasoning capabilities, but a fast reasoning speed and low computing resource usage.

[0016] Vision Transformer (ViT): Vision Transformer (ViT) is an image classification model based on the attention mechanism. Different from the traditional convolutional neural network (CNN), the ViT model uses a self-attention mechanism to capture information in the image to achieve image classification tasks.

[0017] ViTPatchEmbeddings: ViTPatchEmbeddings is a neural network structure for image processing. It is a key component in the Vision Transformer (ViT) model. The role of ViTPatchEmbeddings is to convert the input image into a series of image patches and represent each image patch as a vector. The purpose of this is to convert the image data into a form suitable for GPT model processing, so that the GPT language model can be used to process the image and train the image to generate text.

[0018] TokenEmbeddings: TokenEmbeddings is a neural network structure for text processing. The role of TokenEmbeddings is to convert the input text into a series of tokens and represent each token as a vector.

[0019] KV Cache: KV Cache is a technology that optimizes the inference performance of large models. It reduces repeated calculations by caching historical calculation results, thereby improving the inference speed. KV Cache mainly saves calculations in the Self-Attention layer. In the inference process of the generative model, the output of each token depends on the information of all previous tokens. KV Cache avoids repeated calculations by caching this information. KV Cache has no direct effect on the amount of calculation of the feedforward layer. The core of KVCache technology is the idea of ​​trading space for time, that is, reducing calculation time by occupying more storage space.

Claims

1. A method for generating text from an image, characterized in that It includes the following steps: S1. Collect image-text training set pairs and create a vocabulary Generally, training an image-to-text model requires a large number of image-text dataset pairs. If you need to train a general and delicate image-to-text model, create a folder named pngs and place images in it. At the same time, place a labels.txt in this training set to store the image names in pngs and the corresponding model text outputs. Also, collect the tokens of an open-source large language model, delete low-frequency tokens, and create a vocabulary of model tokens, that is, tokens_Dict={0:‘[CLS]’, 1:”[PAD]”, 2:”[SEP]", 3:”[NewLine]”, 4:”中”.......}. Select 30,000 tokens. Among them, [CLS] is the text start symbol, [PAD] means that when the text length does not reach the maximum length, it is padded with [PAD] to the max_len length, [SEP] is the separator in the text generation task, [NewLine] is the text line break symbol, and max_len=768 is the maximum length of the language model agreed upon for input to the model; S2. Adaptation of image and text modalities Images and texts are two modalities, that is, two different modal tensor forms. Use a CLIP multi-modal alignment model or an encoder trained with relevant large amounts of corpus to represent a token without physical meaning as a vector. Patchify the image, that is, cut the picture into individual Patch blocks, and each block is represented by a 2048-dimensional vector. These blocks are equivalently regarded as a sequence and directly concatenated with the TokenEmbedding of the text. Then train on the basis of the large language model. The vector representation of the image patch and the text token vector representation are directly concatenated and input into the GPT model; S3. Construct the structure of the image-to-text network model A tensor of size [16, 768, 2048] obtained through step S1 for a certain text is input into the language generation model. 2048 is the dimension of each image Patch or text Token representation, where 16 is the batch size and 768 is the model training length. Repeat the causal attention module and the feed-forward fully connected module 16 times each to form the neural network model to be trained, that is, first pass through 16 feed-forward fully connected sub-layers with different parameters but the same structure, and then pass through 16 causal attention sub-layers with exactly the same structure but different parameters. The output of the feed-forward fully connected module has three units, namely the key tensor, the value tensor, and the query tensor. Among them, the query tensor is transmitted in the subsequent causal attention module, and the value tensor and the key tensor share parameters in all causal attention modules and perform attention interaction with the query tensors that are completely different for each attention sub-layer; S4. Training of the image-to-text model According to the model structure set in step S3, the Adam optimizer and the cross entropy loss function are used for model training. The model input data obtained in step S2 is imported into the network structure obtained in step S3. The ultimate goal of the model is to minimize the loss function. For the image-to-text model, the loss function does not need to be calculated for the output of the language model at the corresponding position of the image PatchEmbeddings. It only needs to correspond to the cross entropy loss function from the 197th position to the end of the language model. That is, the model is trained to output the first text Token predicted from the image as much as possible from the 197th position, and the first real text token will predict the second text Token at the 198th position on the model input side. S5. Model prediction If the current user inputs an image, it is divided into patches through the PatchEmbeddings algorithm, and a vector is assigned to each patch. Then the CLS vector and the image PatchEmbeddings are concatenated and input into the GPT language model. According to the index of the maximum output at the 197th position, the corresponding Token is found through the index to complete the decoding. The Token is then spliced ​​to the model input side again, and the model output predicts the second Token. By analogy, the final image-to-text generation task can be completed. During the reasoning process, after each inference token passes through the embedding layer and multiple feed-forward layers, the key and value tensors are gradually cached, and the query tensor is transmitted in the causal attention sublayer and the shared cache tensors key tensor and value tensor are interacted with for attention.

2. According to claim 1, a method for generating text from an image is characterized in that The S2 step further includes: For images using ViTPatchEmbeddings and without training, the GPT model supports an input length of 768; for example, in a batch of batch_size=16 image-text pairs, the images are denoted as Pics and the texts are denoted as Txts. The images in this batch of Pics are first deformed to [16, 3, 224, 224], that is, they have three channels of RGB, and the width and height are both 224. Then, the convolution operation is nn.Conv2d(num_channels, hidden_size, kernel_size=patch_size, stride=patch_size), with patch_size=16, hidden_size=2048, num_channels=3. After the above convolution operation, the size is [16, 3, 224, 224] and the tensor is [16, 2048, 14, 14], that is, the image is divided into 14*14 small blocks, and the vector representation dimension of each block is 2048; the last two dimensions are flattened and exchanged, and the tensor representation of the batch of images is now [16, 196, 2048]. At this time, the dimension of a patch representation of an image is the same as the dimension of a token representation of a text; the maximum length of the text input to the model is agreed to be 768, and the tensor representation shape of the input model of a batch size is [16, 768, 2048], where the tensor representation size of the image mode is [16, 196, 2048] and the tensor representation size of the text mode is [16, 571, 2048]. When the text length is not enough, the [PAD] length supplement character is used; the tensor of the start character [CLS] The vector representation is [16, 1, 2048]; the start symbol [CLS] tensor representation, image patch tensor representation, and text token tensor representation are concatenated together in sequence, and the tensor size of the input GPT model is [16, 768, 2048]; the corresponding GPT model output tensor Y size is [16, 768, 30000], where 30000 is the number of all tokens; that is, the input image is divided into multiple patches, each patch is regarded as a text token, and the vector representation of the image patch and the vector representation of the text are both 2048, so that the GPT model accepts both image input and text input modes at the same time. It only needs the image to pass through the patch to train the image-to-text model end-to-end.

Citation Information

Cited By

  • Data caching method and host system

    TWI919859B