Method and apparatus for parameter alignment between visual model and language model, storage medium, and electronic device

Abstract

The present disclosure relates to a method and apparatus for parameter alignment between a visual model and a language model, a storage medium, and an electronic device. The method comprises: connecting a visual encoder and a large language model in parallel to obtain a first training structure, connecting the comparatively pre-trained visual encoder and large language model in series to obtain a second training structure, then connecting the generatively pre-trained visual encoder and large language model in series to a decoder having fixed parameters to obtain a third training structure, and fine-tuning the visual encoder and the large language model in the third training structure.

Description

Vision language model parameter alignment method and device, storage medium and electronic device

[0001] Cross-reference to Related Applications

[0002] The present disclosure claims priority to a Chinese patent application No. 202410802819.3, filed on June 20, 2024, entitled "Vision language model parameter alignment method and device, storage medium and electronic device" with Tsinghua University. TECHNICAL FIELD

[0003] The present disclosure relates to the field of large language models, and in particular, to a vision language model parameter alignment method, device, storage medium and electronic device. BACKGROUND

[0004] With the rapid development of the field of artificial intelligence, large language models (LLMs) and visual large language models (VLLMs) have become key technologies to drive the development of artificial intelligence, especially artificial general intelligence (AGI) systems. LLMs have led the progress in this field through their outstanding performance in open-world language tasks, and their model size and performance are still growing rapidly. VLLMs build on the foundation of LLMs and have made significant breakthroughs in visual-language dialogue and interaction, including image classification, video classification, image-text retrieval, image description generation, and multi-modal dialogue. However, compared with the rapid progress of LLMs, the development of visual and visual-language base models has been relatively lagging. Currently, in order to bridge visual models with LLMs, existing VLLMs usually use lightweight "glue" layers, such as QFormer or linear projection, to align the features of visual and language models, but related technologies result in low performance of the model in cross-modal interaction due to problems such as non-alignment of parameter quantities, inconsistent representation, and low connection efficiency of visual and language models.

[0005] SUMMARY

[0006] In view of this, the present disclosure proposes a vision language model parameter alignment method, device, storage medium and electronic device, aiming to align the parameters of visual and semantic models and improve the performance of the model in the process of cross-modal interaction.

[0007] According to a first aspect of the present disclosure, a vision language model parameter alignment method is provided, the method comprising:

[0008] determining a large language model and a visual encoder processed by parameter expansion;

[0009] determining a training set comprising a sample image and a sample text corresponding to the sample image;

[0010] The visual encoder and the large language model are connected in parallel to obtain a first training structure, and the visual encoder and the large language model in the first training structure are pre-trained in a contrastive manner according to the training set;

[0011] The visual encoder and the large language model after the contrastive pre-training are connected in series to obtain a second training structure, and the visual encoder and the large language model in the second training structure are pre-trained in a generative manner according to the training set;

[0012] The visual encoder and the large language model after the generative pre-training are connected in series with a decoder with fixed parameters to obtain a third training structure, and the visual encoder and the large language model in the third training structure are fine-tuned according to the training set, thereby completing parameter alignment of the visual encoder and the large language model.

[0013] In a possible implementation, the large language model is an LLaMA-7B model, and the visual encoder after the parameter expansion processing is an InternViT-6B encoder.

[0014] In a possible implementation, the pre-training of the visual encoder and the large language model in the first training structure according to the training set comprises:

[0015] The sample images in the training set are input into the visual encoder to obtain image feature vectors, and the corresponding sample texts are input into the large language model to obtain text feature vectors;

[0016] A first loss of the first training structure is calculated according to the image feature vectors corresponding to each sample image and the text feature vectors corresponding to each sample text.

[0017] The parameters of the visual encoder and the large language model are adjusted according to the first loss, and the contrastive pre-training is completed.

[0018] In a possible implementation, the pre-training of the visual encoder and the large language model in the second training structure according to the training set comprises:

[0019] The sample images are input into the visual encoder to obtain corresponding image feature encodings;

[0020] The image feature encodings are input into the large language model after a cross-attention mechanism, to obtain text feature vectors and twice-encoded image feature vectors;

[0021] A second loss of the second training structure is calculated according to the image feature vectors and the text feature vectors.

[0022] The parameters of the visual encoder and the large language model are adjusted according to the second loss, and the generative pre-training is completed.

[0023] In a possible implementation, the second loss includes an image-text contrast loss, an image-text matching loss, and an image-guided text generation loss.

[0024] In a possible implementation, the fine-tuning of the visual encoder and the large language model in the third training structure according to the training set includes:

[0025] The sample image is input into the visual encoder to obtain corresponding image feature encoding;

[0026] The image feature encoding is input into the large language model after passing through a cross-attention mechanism to obtain an image feature vector;

[0027] The sample image, the image feature vector, and the corresponding sample text are input into the fixed-parameter decoder to obtain corresponding predicted text information;

[0028] A third loss of the third training structure is calculated according to the image feature vector and the predicted text information;

[0029] The parameters of the visual encoder and the large language model are fine-tuned according to the third loss.

[0030] In a possible implementation, the first loss and the third loss are both cross-entropy losses.

[0031] According to a second aspect of the present disclosure, a visual language model parameter alignment device is provided, and the device includes:

[0032] A model determination module is configured to determine a large language model and a visual encoder after parameter expansion processing;

[0033] A training set determination module is configured to determine a training set including sample images and sample text corresponding to the sample images;

[0034] A first training module is configured to obtain a first training structure by connecting the visual encoder and the large language model in parallel, and to perform contrastive pre-training on the visual encoder and the large language model in the first training structure according to the training set;

[0035] A second training module is configured to obtain a second training structure by connecting the visual encoder and the large language model in series after contrastive pre-training, and to perform generative pre-training on the visual encoder and the large language model in the second training structure according to the training set;

[0036] a third training module configured to connect the pre-trained visual encoder and the large language model with a fixed parameter decoder to obtain a third training structure, and fine-tune the visual encoder and the large language model in the third training structure according to the training set, so as to complete parameter alignment of the visual encoder and the large language model.

[0037] In a possible implementation, the large language model is an LLaMA-7B model, and the visual encoder after the parameter expansion processing is an InternViT-6B encoder.

[0038] In a possible implementation, the first training module is further configured to:

[0039] input a sample image in the training set into the visual encoder to obtain an image feature vector, and input corresponding sample text into the large language model to obtain a text feature vector;

[0040] calculate a first loss of the first training structure according to the image feature vector corresponding to each sample image and the text feature vector corresponding to each sample text;

[0041] adjust parameters of the visual encoder and the large language model according to the first loss, and complete contrastive pre-training.

[0042] In a possible implementation, the second training module is further configured to:

[0043] input the sample image into the visual encoder to obtain corresponding image feature encoding;

[0044] input the image feature encoding and the corresponding sample text into the large language model after cross-attention mechanism to obtain a text feature vector and a twice-encoded image feature vector;

[0045] calculate a second loss of the second training structure according to the image feature vector and the text feature vector;

[0046] adjust parameters of the visual encoder and the large language model according to the second loss, and complete generative pre-training.

[0047] In a possible implementation, the second loss includes an image-text contrastive loss, an image-text matching loss, and an image-guided text generation loss.

[0048] In a possible implementation, the third training module is further configured to:

[0049] input the sample image into the visual encoder to obtain corresponding image feature encoding;

[0050] inputting the image feature code into the large language model through a cross attention mechanism to obtain an image feature vector;

[0051] inputting the sample image, the image feature vector, and the corresponding sample text into the decoder with fixed parameters to obtain corresponding predicted text information;

[0052] calculating a third loss of the third training structure according to the image feature vector and the predicted text information;

[0053] fine-tuning the parameters of the visual encoder and the large language model according to the third loss.

[0054] In a possible implementation, the first loss and the third loss are both cross entropy losses.

[0055] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0056] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, which stores computer program instructions, wherein the computer program instructions are executed by a processor to implement the above method.

[0057] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0058] In the embodiments of the present disclosure, the large language model and the visual encoder after parameter expansion are determined, and a training set is determined. The visual encoder and the large language model are connected in parallel to obtain a first training structure, and the first training structure is pre-trained in a contrastive manner according to the training set. The visual encoder and the large language model after contrastive pre-training are connected in series to obtain a second training structure, and the second training structure is pre-trained in a generative manner according to the training set. The visual encoder and the large language model after generative pre-training are connected in series with a decoder with fixed parameters to obtain a third training structure, and the visual encoder and the large language model in the third training structure are fine-tuned according to the training set. The present disclosure realizes preliminary model parameter alignment by parameter expansion of the visual encoder, further realizes model training through a three-stage training method, improves training efficiency and accuracy of the training result, realizes parameter alignment of the visual encoder and the large language model, and thus improves the performance of the model in cross-modal interaction, so that it can be flexibly combined to realize a wide range of multi-modal tasks.

[0059] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure, and explain the principles of the present disclosure.

[0061] FIG. 1 shows a flowchart of a visual language model parameter alignment method according to an embodiment of the present disclosure;

[0062] FIG. 2 shows a schematic diagram of a three-step training process according to an embodiment of the present disclosure;

[0063] FIG. 3 shows a schematic diagram of a visual language model parameter alignment apparatus according to an embodiment of the present disclosure;

[0064] FIG. 4 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features, and aspects of the present disclosure will be explained hereinafter with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0066] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0067] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the present disclosure are omitted. It will be appreciated that the present disclosure can be practiced with the exact details as appropriate to the particular application with the addition of routine, structure, and circuitry that are known or available to those skilled in the art.

[0068] The visual language model parameter alignment method of the embodiments of the present disclosure can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can realize the visual language model parameter alignment method of the embodiments of the present disclosure by calling computer readable instructions stored in the memory through the processor.

[0069] FIG. 1 shows a flowchart of a visual language model parameter alignment method according to an embodiment of the present disclosure. As shown in FIG. 1, the visual language model parameter alignment method of the embodiments of the present disclosure can include the following steps S10-S50.

[0070] Step S10, determining a large language model and a visual encoder after parameter expansion processing.

[0071] In a possible implementation, the electronic device determines a large language model and a visual encoder that need to be parameter-aligned. The electronic device can preliminarily align the large language model and the visual encoder at the model parameter scale level by performing parameter expansion processing on the visual encoder in advance. Parameter expansion refers to increasing the number of parameters so that the number of parameters of the large language model and the visual encoder is the same. Optionally, the visual encoder obtained after parameter expansion of the present disclosure can be an InternViT-6B encoder, which is a Transformer-based visual encoder with 600 million parameters. It can be specially designed to achieve a good balance between performance and efficiency, and is suitable for various visual perception tasks. The model structure parameters of the visual encoder can be as shown in the following table:

[0072] Optionally, the large language model determined by the electronic device in the embodiments of the present disclosure can be a language middleware QLLaMA, which is a language middleware with 800 million parameters and can be initialized by a pre-trained multi-language LLaMA-7B model. The language middleware not only provides powerful multi-language representation capability, but also serves as a bridge connecting the visual encoder and the ready-made LLM decoder.

[0073] Step S20, determining a training set including sample images and sample texts corresponding to the sample images.

[0074] In a possible implementation, the electronic device determines a training set for training the large language model and the visual encoder for parameter alignment. The training set can include sample images for inputting the visual encoder and sample texts for inputting the large language model, and the sample images and the sample texts have a corresponding relationship. Optionally, each sample text can also have a corresponding query vector for information introduction.

[0075] In step S30, a first training structure is obtained by connecting the visual encoder and the large language model in parallel, and the visual encoder and the large language model in the first training structure are pre-trained in a contrastive manner according to the training set.

[0076] In a possible implementation, the electronic device can train the visual encoder and the large language model in steps, and the first step training process can be contrastive pre-training. In the contrastive pre-training process, the electronic device determines a parallel structure of the visual encoder and the large language model as a first training structure, and trains the visual encoder and the large language model in the first training structure according to the training set.

[0077] Optionally, the contrastive pre-training process can include inputting the sample images in the training set into the visual encoder to obtain image feature vectors, and inputting the corresponding sample texts into the large language model to obtain text feature vectors. Then, a first loss of the first training structure is calculated according to the image feature vectors corresponding to each sample image and the text feature vectors corresponding to each sample text, and the parameters of the visual encoder and the large language model are adjusted according to the first loss to complete the contrastive pre-training. For example, the embodiments of the present disclosure can use a large number of sample image-text pairs with noise to perform contrastive learning to align InternViT-6B (the visual encoder) and a pre-trained multi-language LLaMA-7B. Specifically, in the training process, the image is input into InternViT-6B to obtain a corresponding image feature encoding, and then the image feature vector I i is obtained through an attention pooling mechanism. Meanwhile, the sample text corresponding to the sample image is input into LLaMA, and the EOS corresponding vector in the text feature encoding is selected to obtain the text feature vector T j .

[0078] Then, the electronic device can calculate the cross-entropy loss after obtaining the image feature vector I i and the text feature vector T j corresponding to the plurality of sample images and sample texts, and obtain a first loss of the first training structure by calculating the cross-entropy loss, and further maximize the similarity between the image and the text by minimizing the first loss. The cross-entropy loss can be calculated by the following loss function:

[0079] where I i is the image feature vector, T j is the text feature vector, sim(I i , T j ) is the cosine similarity between the image feature vector and the text feature vector, P is a set of positive sample pairs, each positive sample pair consisting of a sample image and a corresponding sample text, and τ is a pre-set temperature parameter. This loss function encourages the model to assign higher similarity scores to correct (positive sample) image-text pairs (i.e., sample images and sample texts that have a corresponding relationship) and lower similarity scores to incorrect (negative sample) image-text pairs (i.e., sample images and sample texts that do not have a corresponding relationship). By minimizing the first loss in a contrastive pre-training manner, the first training structure is enabled to learn to effectively distinguish between matching and non-matching image-text sample pairs.

[0080] Step S40, concatenating the contrastively pre-trained visual encoder and the large language model to obtain a second training structure, and performing generative pre-training on the visual encoder and the large language model in the second training structure according to the training set.

[0081] In one possible implementation, after the first step of contrastive pre-training, the electronic device performs the second step of generative pre-training. This pre-training process concatenates the contrastively pre-trained visual encoder and the large language model to obtain a second training structure, and then performs generative pre-training on the visual encoder and the large language model in the second training structure according to the training set.

[0082] Optionally, the generative pre-training process can include inputting the sample image into the visual encoder to obtain a corresponding image feature encoding. The image feature encoding is input into the large language model after passing through the cross-attention mechanism together with the corresponding sample text to obtain a text feature vector and a second encoding image feature vector. A second loss of the second training structure is calculated according to the image feature vector and the text feature vector. The parameters of the visual encoder and the large language model are adjusted according to the second loss to complete the generative pre-training. The second loss can include an image-text contrastive loss, an image-text matching loss, and an image-guided text generation loss. That is, the visual encoder and the large language model in the embodiment of the present disclosure inherit the weights of the contrastive pre-training, and the visual encoder and the text encoder are changed from the original parallel placement to a serial connection to obtain the second training structure. Then the generative pre-training is performed. Optionally, the generative pre-training process connects InternViT-6B and QLLaMA and adopts a generative training strategy. QLLaMA increases additional parameters on the basis of LLaMA in the contrastive pre-training stage. This includes newly added trainable query vectors and cross-attention layers. These newly added layers are randomly initialized and trained in the training of the generative pre-training stage.

[0083] In the generative pre-training process, the electronic device can input the sample image into InternViT-6B to obtain the image feature encoding as the image feature encoding, and then input the query vector corresponding to the sample text input into QLLaMA. The query vector is used as the query, and the image feature encoding is used as the key / value pair to perform cross-attention mechanism calculation, so as to introduce the visual information in the image feature encoding into the query vector. Unlike the contrastive pre-training stage, in this stage, the image feature vector I i and the text feature vector T j are both generated by QLLaMA. The second loss generated in this stage is the sum of the image-text contrast (ITC) loss, the image-text matching (ITM) loss, and the image-guided text generation (ITG) loss.

[0084] Optionally, the image-text contrast loss can be calculated according to the formula for calculating the first loss in the contrastive pre-training process, that is:

[0085] where I i is the image feature vector, T j is the text feature vector of the sample text corresponding (matching) to the sample image, sim(I i , T jis the cosine similarity between the image feature vector and the text feature vector, P is a set of positive sample pairs, each positive sample pair consists of a sample image and a corresponding sample text, and τ is a preset temperature parameter. The image-text matching loss can be a loss that re-predicts whether an image and text pair matches for a binary classification task, and can be calculated according to the following formula, that is:

[0086] wherein, I i is the image feature vector, T j is the text feature vector of the sample text corresponding (matching) to the sample image. M is a set of matching image-text pairs, and σ is a sigmoid function, is the text feature vector of the negative sample (i.e., the sample text that does not match).

[0087] Next, in the process of image-guided text generation, the sample image obtains its corresponding image feature encoding through the InternViT-6B model. Subsequently, these image feature encodings interact with the query vector (Query) below the QLLaMA model. In this interaction process, the query vector serves as a query, while the image feature encoding participates in the calculation of the cross-attention mechanism as a key / value pair (Key / Value). The purpose of this step is to effectively introduce visual information in the sample image into the query vector, providing rich contextual information for subsequent text generation. The image-guided text generation loss in this process can be defined as a loss based on next prediction, which can be calculated according to the formula L ITG =-∑ i logp(T i+1 ∣T ＜i ), wherein T i+1 represents the next word generated, T ＜i represents all word sequences before this, and p(T i+1 ∣T ＜i ) is the probability of the model predicting the next word T ＜i given the previous sequence T i+1 . By minimizing L ITG , the model after generative pre-training can generate more relevant and accurate text related to the image content.

[0088] Step S50, concatenating the generative pre-trained visual encoder and large language model with the parameter-fixed decoder to obtain a third training structure, and fine-tuning the visual encoder and large language model in the third training structure according to the training set.

[0089] In a possible implementation, after further generating pre-training, the electronic device can obtain a third training structure by connecting the visually encoded and large language model after generating pre-training and the decoder with fixed parameters in series, and perform final fine-tuning on the visually encoded and large language model in the third training structure according to the training set. The process of the fine-tuning can include inputting a sample image into the visually encoded to obtain a corresponding image feature code. The image feature code is input into the large language model after passing through the cross-attention mechanism to obtain an image feature vector. Then, the sample image, the image feature vector, and the corresponding sample text are input into the decoder with fixed parameters to obtain corresponding predicted text information. A third loss of the third training structure is calculated according to the image feature vector and the predicted text information, and the parameters of the visually encoded and the large language model are fine-tuned according to the third loss.

[0090] Optionally, in the supervised fine-tuning stage, the visually encoded and large language model after generating pre-training and the LLM decoder (such as Vicuna or InternLM) with fixed parameters obtained after pre-training are connected through an MLP layer and are fine-tuned in a supervised manner. The purpose of the fine-tuning stage in the embodiment of the present disclosure is to further optimize the performance of the model on specific visual language tasks by using high-quality instruction data. The fine-tuning strategy is similar to the training of the pure text model, that is, the model is trained to predict the next token by minimizing the cross-entropy loss. That is, the third loss can be a cross-entropy loss. Optionally, the formula for calculating the third loss can be L ITG = -∑ i logp(T i+1 | T ＜i ) is calculated, where T i+1 represents the generated next word, T ＜i represents all word sequences before this, and p(T i+1 | T ＜i ) is the probability of the model predicting the next word T ＜i under the condition of the given previous sequence T i+1 . By minimizing L ITG , the model after generating pre-training can generate more relevant and accurate text related to the image content.

[0091] Then, in the process of fine-tuning the visually encoded and large language model, the embodiment of the present disclosure can select to train only the MLP layer or train the MLP layer and the QLLaMA simultaneously according to whether it is desired to maintain the original language ability of the LLM.

[0092] FIG. 2 shows a schematic diagram of a three-step training process according to an embodiment of the present disclosure. As shown in FIG. 2, the embodiment of the present disclosure can go through three training processes of contrastive pre-training, generative pre-training and parameter fine-tuning in the process of parameter alignment of the visual encoder and the large language model. The structure of the contrastive pre-training can be a parallel structure of the visual encoder and the large language model, the training process can determine a first loss by a cross-entropy loss, and the parameters are adjusted according to the first loss. The structure of the generative pre-training can be a series structure of the visual encoder and the large language model, the training process can determine a second loss by an image-text contrastive loss, an image-text matching loss and an image-guided text generation loss, and the parameters are adjusted according to the second loss. The structure of the supervised fine-tuning can be a series structure of the visual encoder, the large language model and the decoder, the training process can determine a third loss by a cross-entropy loss, and the parameters are adjusted according to the third loss.

[0093] Based on the above technical features, the embodiment of the present disclosure first realizes preliminary model parameter alignment by parameter expansion of the visual encoder, and then further improves the training efficiency and the accuracy of the training result through three training stages, so that the trained visual encoder and the large language model can effectively combine visual and language features to process various visual language tasks, such as image classification, image-text retrieval, image caption generation, visual question answering and multi-modal dialogue, etc.

[0094] FIG. 3 shows a schematic diagram of a visual language model parameter alignment device according to an embodiment of the present disclosure. As shown in FIG. 3, the visual language model parameter alignment device of the embodiment of the present disclosure can include:

[0095] The model determination module 30 is configured to determine a large language model and a visual encoder after parameter expansion processing;

[0096] The training set determination module 31 is configured to determine a training set including sample images and sample texts corresponding to the sample images;

[0097] The first training module 32 is configured to obtain a first training structure in parallel with the visual encoder and the large language model, and perform contrastive pre-training on the visual encoder and the large language model in the first training structure according to the training set;

[0098] The second training module 33 is configured to obtain a second training structure in series with the visual encoder and the large language model after the contrastive pre-training, and perform generative pre-training on the visual encoder and the large language model in the second training structure according to the training set;

[0099] The third training module 34 is configured to connect the pre-trained visual encoder and the large language model in series to obtain a third training structure, and fine-tune the visual encoder and the large language model in the third training structure according to the training set, so as to complete the parameter alignment of the visual encoder and the large language model.

[0100] In a possible implementation, the large language model is an LLaMA-7B model, and the visual encoder after the parameter expansion processing is an InternViT-6B encoder.

[0101] In a possible implementation, the first training module 32 is further configured to:

[0102] input the sample images in the training set into the visual encoder to obtain image feature vectors, and input corresponding sample texts into the large language model to obtain text feature vectors;

[0103] calculate a first loss of the first training structure according to the image feature vectors corresponding to each of the sample images and the text feature vectors corresponding to each of the sample texts;

[0104] adjust the parameters of the visual encoder and the large language model according to the first loss, and complete the contrastive pre-training.

[0105] In a possible implementation, the second training module 33 is further configured to:

[0106] input the sample images into the visual encoder to obtain corresponding image feature encodings;

[0107] input the image feature encodings and corresponding sample texts into the large language model through a cross-attention mechanism to obtain text feature vectors and secondary encoded image feature vectors;

[0108] calculate a second loss of the second training structure according to the image feature vectors and the text feature vectors;

[0109] adjust the parameters of the visual encoder and the large language model according to the second loss, and complete the generative pre-training.

[0110] In a possible implementation, the second loss includes an image-text contrastive loss, an image-text matching loss, and an image-guided text generation loss.

[0111] In a possible implementation, the third training module 34 is further configured to:

[0112] input the sample images into the visual encoder to obtain corresponding image feature encodings;

[0113] inputting the image feature code into the large language model through a cross attention mechanism to obtain an image feature vector;

[0114] inputting the sample image, the image feature vector and the corresponding sample text into the parameter-fixed decoder to obtain corresponding predicted text information;

[0115] calculating a third loss of the third training structure according to the image feature vector and the predicted text information;

[0116] fine-tuning parameters of the visual encoder and the large language model according to the third loss.

[0117] In a possible implementation, the first loss and the third loss are both cross entropy losses.

[0118] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be described here.

[0119] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0120] The embodiments of the present disclosure also propose an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0121] The embodiments of the present disclosure also provide a computer program product, comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0122] FIG. 4 shows a schematic diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to FIG. 4, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0123] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM , or the like.

[0124] In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, is also provided, which can be executed by the processing component 1922 of the electronic device 1900 to implement the above method.

[0125] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0126] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0127] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0128] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0129] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0130] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions which execute via the one or more processors of the computer or other programmable data processing apparatus create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. A computer readable storage medium can be, but is not limited to, a floppy disk, a hard disk, a solid state drive, a DVD, a CD, a tape, a magnetic

[0131] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable data processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0132] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may

[0133] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive of the disclosed embodiments. Many modifications and variations of the disclosed embodiments are possible in light of this disclosure without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, the practical application, or technical improvement over prior art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of visual language model parameter alignment, wherein, The method comprises: determining a large language model and a visual encoder after parameter expansion processing; determining a training set comprising sample images and sample texts corresponding to the sample images; obtaining a first training structure in parallel with the visual encoder and the large language model, and performing contrastive pre-training on the visual encoder and the large language model in the first training structure according to the training set; obtaining a second training structure in series with the visual encoder and the large language model after contrastive pre-training, and performing generative pre-training on the visual encoder and the large language model in the second training structure according to the training set; obtaining a third training structure in series with the visual encoder and the large language model after generative pre-training and a decoder with fixed parameters, and fine-tuning the visual encoder and the large language model in the third training structure according to the training set to complete parameter alignment of the visual encoder and the large language model.

2. The method of claim 1, wherein, The large language model is an LLaMA-7B model, and the visual encoder after parameter expansion processing is an InternViT-6B encoder.

3. The method of claim 1 or 2, wherein, The contrastive pre-training on the visual encoder and the large language model in the first training structure according to the training set comprises: inputting the sample images in the training set into the visual encoder to obtain image feature vectors, and inputting the corresponding sample texts into the large language model to obtain text feature vectors; calculating a first loss of the first training structure according to the image feature vectors corresponding to each sample image and the text feature vectors corresponding to each sample text; adjusting the parameters of the visual encoder and the large language model according to the first loss to complete contrastive pre-training.

4. The method of any of claims 1-3, wherein, The generative pre-training on the visual encoder and the large language model in the second training structure according to the training set comprises: inputting the sample images into the visual encoder to obtain corresponding image feature encodings; inputting the image feature encodings after cross-attention mechanism and the corresponding sample texts into the large language model to obtain text feature vectors and twice-encoded image feature vectors; calculating a second loss of the second training structure according to the image feature vectors and the text feature vectors; adjusting the parameters of the visual encoder and the large language model according to the second loss to complete generative pre-training.

5. The method of claim 4, wherein, The second loss comprises an image-text contrast loss, an image-text matching loss, and an image-guided text generation loss.

6. The method of any of claims 1-5, wherein, The fine-tuning on the visual encoder and the large language model in the third training structure according to the training set comprises: inputting the sample images into the visual encoder to obtain corresponding image feature encodings; inputting the image feature encodings after cross-attention mechanism into the large language model to obtain image feature vectors; inputting the sample images, the image feature vectors, and the corresponding sample texts into the decoder with fixed parameters to obtain corresponding predicted text information; calculating a third loss of the third training structure according to the image feature vectors and the predicted text information; The parameters of the visual encoder and the large language model are fine-tuned according to the third loss.

7. The method of claim 3 or 6, wherein, The first loss and the third loss are both cross-entropy losses.

8. A visual language model parameter alignment apparatus, wherein, The apparatus comprises: a model determination module configured to determine a large language model and a visual encoder after parameter expansion processing; a training set determination module configured to determine a training set comprising sample images and sample texts corresponding to the sample images; a first training module configured to obtain a first training structure by connecting the visual encoder and the large language model in parallel, and to perform contrastive pre-training on the visual encoder and the large language model in the first training structure according to the training set; a second training module configured to obtain a second training structure by connecting the visual encoder and the large language model after contrastive pre-training in series, and to perform generative pre-training on the visual encoder and the large language model in the second training structure according to the training set; a third training module configured to obtain a third training structure by connecting the visual encoder and the large language model after generative pre-training and a decoder with fixed parameters in series, and to fine-tune the visual encoder and the large language model in the third training structure according to the training set, thereby completing parameter alignment of the visual encoder and the large language model.

9. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 7.

Images