Visual language multi-modal large language model pre-training method based on image-text staggering

Abstract

The invention discloses a pre-training method for a visual language multi-modal large language model (VLM) based on image-text staggering, and the pre-training method is characterized in that the VLM is a multi-modal large language model. The method comprises the following steps: S1, constructing an image-text interlaced data set, an image description data set and a text data set; S2, constructing a multi-modal large-language model architecture, and carrying out preliminary fusion from a vector space of a visual model to a text vector space by utilizing image description; S3, fusing visual information and text information by utilizing image-text interlaced data, and the visual understanding capability of the multi-modal large model is optimized. Through the scheme, the visual understanding capability of the visual information and the text information in generating the large model can be fully fused under the condition that the text understanding capability and the logical reasoning capability of the large language model are kept. Meanwhile, due to the construction of the image-text interlaced data, the multi-modal large model trained by the method has the capability of processing a plurality of images. The scheme is suitable for the AIGC field.

Description

Technical Field

[0001] The present invention relates to a visual language multimodal language model, and more particularly to a pre-training method for a visual language multimodal large language model based on interlacing of images and texts. Background Art

[0002] With the rapid development of artificial intelligence and natural language processing, multimodal language models have become a hot area of ​​research and application. Traditional language models mainly focus on processing and generating text data, but information in the real world is not limited to text, but also includes other forms of non-text data such as images. The emergence of multimodal language models fills this gap, enabling the model to process and understand multimodal data, thereby better simulating human multimodal cognitive ability. Multimodal learning aims to fuse data of different modalities to make full use of complementary information between different modalities and improve the performance of the model. The current multimodal pre-trained image-text alignment is usually trained using image-text pairs of image descriptions. For example, the CLIP model proposed by OpenAI is trained with 400 million image-text pairs. However, the understanding of images in multimodal large language models is not limited to the description of images, but the understanding of the knowledge and reasoning contained in the images, as well as reasoning about images or focusing on a certain area of ​​the images according to the instructions of the text. In this context, the image description data type cannot fully meet the use requirements of multimodal large language models. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a pre-training method for a visual language multimodal large language model (Vision Large Language Model, VLM) based on image and text interlacing. To achieve the above purpose, the present invention provides the following technical solution: a pre-training method for a visual language multimodal large language model based on image and text interlacing, characterized in that it comprises the following steps:

[0004] Step 1: Build a dataset of interleaved images and texts, as well as a regular image description dataset and text dataset;

[0005] Step 2: After constructing the image-text interleaved dataset in step 1, as well as the conventional image description dataset and text dataset, a multimodal large language model architecture is constructed, and the image description is used to perform a preliminary mapping of the vector space of the visual model to the text vector space;

[0006] Step three: Use image and text interleaving data to integrate visual information and text information, train the parameters of the entire model, and optimize the multimodal model's visual understanding and reasoning capabilities.

[0007] As a further improvement of the present invention, the image description dataset in step one is a picture and text pair consisting of a picture and its corresponding picture text description; the image and text interlaced dataset refers to a dataset in which pictures and texts are mixed and spliced ​​in sequence, and its source can be online articles and news materials; the text dataset refers to text materials, including books, papers and online materials.

[0008] As a further improvement of the present invention, the image-text data set limits a sample to one image, and the text content is an interpretation of the image content, and the entire sample in the image-text interleaved data set is a logically complete paragraph, which is the corresponding position of the image in the text, but not an interpretation of the image.

[0009] As a further improvement of the present invention, the specific steps of mapping the vector space of the visual model to the text vector space using the image description data in step 2 are as follows:

[0010] Step 21: first encode the image information through the image encoder, and output the image vector V∈RN×dv, where N is the number of image-specific tokens and dv is the dimension of each token;

[0011] Step 22: For text, the text is converted into word-unit information through a text segmenter. Generally, it goes through two stages: word segmentation and symbolization. The word-unit is projected into a text vector through a text encoder. The text feature is represented as a matrix T∈RM×dt, where M is the number of text tokens and dt is the dimension of each word vector.

[0012] In step 23, the image vector in step 21 is fused with the text vector in step 22 by cross-attention between image and text, and the vector fused by cross-attention is input into the large language model for decoding to obtain image-based text prediction.

[0013] As a further improvement of the present invention, the steps of fusing the text and image by cross-attention in steps 2 and 3 are as follows:

[0014] Step 1: Calculate K and V of text vector through linear mapping

[0015] in, and Trainable;

[0016] Step 2: Calculate the Q of the visual vector through linear mapping

[0017] in, Trainable

[0018] Step 3: Calculate the attention matrix based on K and Q

[0019] Step 4: Apply V to the attention matrix H vt =A vt V t .

[0020] As a further improvement of the present invention, after the multimodal large language model architecture is constructed in step 2, the parameters of other modules of the model are frozen, and only the image description dataset is used to train multimodal fusion.

[0021] As a further improvement of the present invention, in the step three, when training the parameters of the entire model, the interleaved data set and the text data set are randomly shuffled so that the model has the ability to process multimodal and text data at the same time.

[0022] The beneficial effect of the present invention is that, by using an interleaved image-text dataset, the model can accept more flexible text information related to images, thereby fully integrating visual information and text information in generating the visual understanding and reasoning capabilities of the large model while maintaining the text understanding and logical reasoning capabilities of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the architecture diagram of the multimodal large language model constructed. DETAILED DESCRIPTION

[0024] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0025] Reference Figure 1 As shown, a method for pre-training a multimodal large language model of visual language based on interlaced images and texts in this embodiment mainly solves the problem that in a multimodal generative large language model, the language model has limited understanding of visual content and cannot understand the area of ​​image details, and provides a multimodal generative large language model algorithm based on multi-level visual representation vector alignment training. It includes the following steps:

[0026] S1. Construct image-text interleaved datasets, as well as conventional image description datasets and text datasets

[0027] S2. Build a multimodal large language model architecture and use image descriptions to perform a preliminary mapping of the vector space of the visual model to the text vector space.

[0028] S3. Utilize interleaved image and text data to integrate visual information with text information and optimize the multimodal model’s visual understanding and reasoning capabilities.

[0029] The above steps are described in detail as follows:

[0030] S1. Constructing a text-image interleaving dataset, as well as a conventional picture description dataset and a text dataset. This scheme proposes a multimodal large language model pre-training method based on text-image interleaving. The first step is to prepare a text-image interleaving dataset, etc. The purpose of this step is to prepare corpus materials for the pre-training of MLLM. The datasets that need to be prepared include three categories: text-image pair datasets, text-image interleaving datasets, and text-image datasets. Among them, the text-image pair dataset refers to a picture and text pair consisting of a picture and its corresponding picture text description. The text-image interleaving dataset refers to a dataset in which pictures and texts are mixed and spliced ​​in order, and its source can be online articles, news materials, etc. Pictures are supplementary materials for text materials and are placed in corresponding positions according to the language order. Text datasets refer to text materials, such as books, papers, online materials, etc., which are commonly used corpus materials for training large language models. Different from the existing common text-image data, the advantage of the interleaved dataset is that the number of pictures in a sample is not limited. In the current text-image dataset, it is generally limited to one picture in a sample, and the text content is an explanation of the picture content. In the interleaved dataset, the entire sample is a logically complete paragraph, which is the corresponding position of the picture in the text, but not the interpretation of the picture. In other words, during the training process, when the model is training the semantic matching of the picture and text, it is no longer an understanding of the picture content, but an inference of the picture logic or even the deep meaning contained in the picture. For example, through a calligraphy work by a celebrity, the information behind the calligraphy work, the calligraphy style, the history of the work, and other information are similar to its semantics; through a crowded picture, it is only the crowded people that are matched, but also contains more deep-level picture and text reasoning and related knowledge such as the relationship between man and nature, population governance, etc.

[0031] S2. Build a multimodal large language model architecture and use image description data to map the vector space of the visual model to the text vector space.

[0032] Based on the well-prepared image-text pairs, image-text interleaving, and text interleaving data, this step builds a multimodal large language model architecture, such as Figure 1As shown. In the figure, the image information is first encoded by an image encoder. The image encoder here is any image-based neural network model, such as convnext, visiontransformer, etc. The output image vector is V∈RN×dv, where N is the number of image feature tokens and dv is the dimension of each token. For text, the text is converted into word-unit information (token) by a text segmenter, generally through two stages of segmentation and symbolization. The word-unit is projected into a text vector by a text encoder. The text feature is represented as a matrix T∈RM×dt, where M is the number of text tokens and dt is the dimension of each word vector. The image vector and the text vector are fused by cross-attention between the image and the text. Through the cross-attention fusion method, the visual and text information can be more deeply integrated. The specific steps are:

[0033] 1) Calculate K and V of text vector through linear mapping

[0034] in, and Trainable

[0035] 2) Calculate the Q of the visual vector through linear mapping

[0036] in, Trainable

[0037] 3) Calculate the attention matrix based on K and Q

[0038] 4) Apply V to the attention matrix H vt =A vt V t

[0039] Here, the position encoding vectors of vision and text are retained to facilitate the subsequent hybrid mode of interleaving images and text. The vectors after cross-attention fusion enter the large language model (generally based on the transformer architecture) for decoding to obtain text prediction based on the picture. The large language model here can be any large language model, such as Llama.

[0040] In the first step of pre-training, the parameters of other modules of the model are frozen, and only the image description dataset is used to train multimodal fusion (such as Figure 1 This step is a preliminary fusion of the vector space of the visual model into the text vector space.

[0041] S3. Use interleaved image and text data to integrate visual information and text information, and optimize the multimodal model's ability to understand and reason about vision. After the visual model features are mapped to text features, this step continues with training using the interleaved image and text dataset and the text dataset. This step trains the parameters of the entire model. In this step of training, the interleaved dataset and the text dataset are randomly shuffled so that the model has the ability to process multimodal and text data at the same time. After adding the interleaved image and text and text datasets, the model can learn the ability to extract important information from images based on contextual information, rather than simply describing the image. At the same time, the introduction of a pure text dataset allows the VLM model to maintain the text understanding and logical reasoning capabilities of a large language model. In addition, since the interleaved image and text data added during the model training process may contain multiple images, this step of the training process allows the model to learn to understand the logical relationship between multiple images.

[0042] The following examples are provided in this embodiment:

[0043] 1. Build training data sets: image and text interlaced data sets, image description data sets, text data sets

[0044] 2. Model structure: Figure 1 Shown

[0045] 3. Loss function: autoregressive prediction loss

[0046] Goal: Predict the next word in a sequence.

[0047] Loss function: Cross-Entropy Loss is usually used

[0048]

[0049] Where x_t is the tth word in the sequence, T is the length of the sample sequence, and P(x t |x1,x2,…,x t-1 ) is the probability of the model predicting the tth word given the previous t-1 words.

[0050] Model reasoning: After the model is trained, the input supports the combination of pictures and text in any form. After the input passes through the model, the output is a text response calculated according to the input instructions.

[0051] In summary, the present embodiment is a pre-training method for a visual language multimodal large language model (VisionLarge Language Model, VLM) based on interlaced images and texts. It includes the following steps: S1, constructing an image-text interlaced data set, an image description data set, and a text data set. S2, constructing a multimodal large language model architecture, and using image descriptions to perform a preliminary fusion of the vector space of the visual model into the text vector space. S3, using image-text interlaced data, integrating visual information and text information, and optimizing the multimodal large model's ability to understand vision. Through this solution, while maintaining the large language model's ability to understand text and logical reasoning, the visual understanding ability of the large model can be fully integrated with visual information and text information. At the same time, due to the construction of image-text interlaced data, the multimodal large model trained by this method has the ability to process multiple pictures. This solution is applicable to the field of AIGC.

[0052] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for pre-training a multimodal large language model of visual language based on interlacing of images and texts, characterized by: The steps include: Step 1: Build a dataset of interleaved images and texts, as well as a regular image description dataset and text dataset; Step 2: After constructing the image-text interleaved dataset in step 1, as well as the conventional image description dataset and text dataset, a multimodal large language model architecture is constructed, and the image description is used to perform a preliminary mapping of the vector space of the visual model to the text vector space; Step three: Use image and text interleaving data to integrate visual information and text information, train the parameters of the entire model, and optimize the multimodal model's visual understanding and reasoning capabilities.

2. The method for pre-training a multimodal large language model based on image-text interleaving according to claim 1, characterized in that: The image description dataset in step 1 is a picture and text pair consisting of a picture and its corresponding picture text description; the image and text interlaced dataset refers to a dataset in which pictures and texts are mixed and spliced ​​in sequence, and its source can be online articles and news materials; the text dataset refers to text materials, including books, papers and online materials.

3. The method for pre-training a multimodal large language model based on visual language and text interleaving according to claim 2 is characterized in that: The image-text dataset limits a sample to one image, and the text content is an explanation of the image content. The entire sample in the image-text interleaved dataset is a logically complete paragraph, which is the corresponding position of the image in the text, but not an explanation of the image.

4. The method for pre-training a multimodal large language model based on visual language interleaving based on images and texts according to claim 1, 2 or 3, characterized in that: The specific steps of mapping the vector space of the visual model to the text vector space using the image description data in step 2 are as follows: Step 21: first encode the image information through the image encoder, and output the image vector V∈RN×dv, where N is the number of image-specific tokens and dv is the dimension of each token; Step 22: For text, the text is converted into word-unit information through a text segmenter. Generally, it goes through two stages: word segmentation and symbolization. The word-unit is projected into a text vector through a text encoder. The text feature is represented as a matrix T∈RM×dt, where M is the number of text tokens and dt is the dimension of each word vector. In step 23, the image vector in step 21 is fused with the text vector in step 22 by cross-attention between image and text, and the vector fused by cross-attention is input into the large language model for decoding to obtain image-based text prediction.

5. The method for pre-training a visual language multimodal large language model based on interlacing of images and texts according to claim 4 is characterized in that: The steps of fusing the text and image by cross-attention in steps 2 and 3 are as follows: Step 1: Calculate K and V of text vector through linear mapping in, and Trainable; Step 2: Calculate the Q of the visual vector through linear mapping in, Trainable Step 3: Calculate the attention matrix based on K and Q Step 4: Apply V to the attention matrix H vt =A vt V t .

6. The method for pre-training a multimodal large language model based on image-text interleaving according to claim 5, characterized in that: After the multimodal large language model architecture is constructed in step 2, the parameters of other modules of the model are frozen, and only the image description dataset is used to train multimodal fusion.

7. The method for pre-training a visual language multimodal large language model based on interlacing of images and texts according to claim 1, 2 or 3, characterized in that: In the step three, when training the parameters of the entire model, the interleaved data set and the text data set are randomly shuffled so that the model has the ability to process multimodal and text data at the same time.

Images