Multi-modal model training method and device and text generation method and device

Abstract

The invention discloses a multi-modal model training method and device and a text generation method and device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. According to the specific implementation scheme, the method comprises the following steps: training a first visual encoder in a first multi-modal model to obtain a trained first visual encoder; wherein the number of model parameters of a first large language model in the first multi-modal model is smaller than a preset threshold value; training the second large language model and the trained first visual encoder to obtain a target multi-modal model; wherein the second large language model is included in the second multi-modal model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large model, etc., and can be applied to scenarios such as AIGC content generation based on artificial intelligence, and specifically relates to a multi-modal model training method, a text generation method and device. BACKGROUND

[0002] With the development of artificial intelligence, large language models have brought great convenience to people's lives. For example, a question and answer system based on a large language model can provide answers to questions raised by users. With the increasing demand of people, a multi-modal model including a visual encoder and a large language model has appeared, so that the questions raised by users can be answered in combination with pictures. SUMMARY

[0003] The present application provides a multi-modal model training method, a text generation method and device. The specific solutions are as follows:

[0004] According to an aspect of the present application, a multi-modal model training method is provided, comprising:

[0005] training a first visual encoder in a first multi-modal model to obtain a trained first visual encoder; wherein the number of model parameters of a first large language model in the first multi-modal model is less than a preset threshold;

[0006] training a second large language model and the trained first visual encoder to obtain a target multi-modal model; wherein the second large language model is included in a second multi-modal model.

[0007] According to another aspect of the present application, a text generation method is provided, comprising:

[0008] obtaining a target image and text prompt information corresponding to the target image;

[0009] processing the target image and the text prompt information using a target multi-modal model to generate a text output result; wherein the multi-modal model is trained using the training method of the above embodiments.

[0010] According to another aspect of the present application, a multi-modal model training device is provided, comprising:

[0011] a first training module configured to train a first visual encoder in a first multi-modal model to obtain a trained first visual encoder; wherein the number of model parameters of a first large language model in the first multi-modal model is less than a preset threshold;

[0012] The second training module is configured to train a second large language model and the trained first visual encoder to obtain a target multi-modal model, wherein the second large language model is included in the second multi-modal model.

[0013] According to another aspect of the present application, a text generation apparatus is provided, comprising:

[0014] The acquisition module is configured to acquire a target image and text prompt information corresponding to the target image.

[0015] The generation module is configured to process the target image and the text prompt information using a target multi-modal model to generate a text output result, wherein the multi-modal model is trained using the training method described in the above embodiments.

[0016] According to another aspect of the present application, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein

[0019] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.

[0020] According to another aspect of the present application, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method described in the above embodiments.

[0021] According to another aspect of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are used to better understand the present application, and do not limit the present application. Among them:

[0024] Figure 1 A flowchart of a multi-modal model training method provided by an embodiment of the present application is shown in the figure;

[0025] Figure 2 A structural diagram of a multi-modal model provided by an embodiment of the present application is shown in the figure;

[0026] Figure 3 A flowchart of a training method of a multi-modal model according to another embodiment of the present application is provided.

[0027] Figure 4 A flowchart of a training method of a multi-modal model according to another embodiment of the present application is provided.

[0028] Figure 5 A flowchart of a text generation method according to an embodiment of the present application is provided.

[0029] Figure 6 A structural diagram of a training device of a multi-modal model according to an embodiment of the present application is provided.

[0030] Figure 7 A structural diagram of a text generation device according to an embodiment of the present application is provided.

[0031] Figure 8 A block diagram of an electronic device for implementing a training method of a multi-modal model according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to help in understanding, which should be considered in the context of the present application. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of national laws and regulations, and do not violate public order and good customs.

[0034] The training method of a multi-modal model, the text generation method, the device, the electronic device and the storage medium according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0035] Figure 1 A flowchart of a training method of a multi-modal model according to an embodiment of the present application is provided.

[0036] The training method of a multi-modal model according to the embodiments of the present application can be executed by a training device of a multi-modal model according to the embodiments of the present application, which can be configured in an electronic device.

[0037] The electronic device can be any device with computing capability, such as a personal computer, a mobile terminal, a server, and the like. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and the like, which are hardware devices with various operating systems, touch screens, and / or display screens.

[0038] As an example, the multi-modal model structure can be as shown in Figure 2 Figure 2 In the multi-modal model 200, the multi-modal model 200 includes a visual encoder 210, a feature alignment layer 220, and a large language model 230, and the visual encoder 210 is connected to the large language model 230 through the feature alignment layer 220.

[0039] The visual encoder 210 is configured to extract features from an input image. The feature distribution output by the visual encoder, such as a distribution in a high-dimensional vector space, can not be consistent with the input distribution of the large language model. Therefore, the feature alignment layer 220 can be used to convert the visual features to the same distribution space as the text features, so that the aligned features can be processed by the large language model 230. The large language model 230 can process the result output by the feature alignment layer and the text related to the input image to generate a text processing result.

[0040] The text related to the input image can be a text prompt, which is used to instruct the multi-modal model to perform a specified task based on the input image, such as answering a question based on the content in the image, or translating English in the picture into Chinese, and the like.

[0041] As shown in Figure 1 The training method of the multi-modal model includes:

[0042] In step 101, the first visual encoder in the first multi-modal model is trained to obtain a trained first visual encoder.

[0043] In this application, the first multi-modal model can include a first visual encoder, a first large language model, and the like. The number of model parameters of the first large language model is less than a preset threshold. The preset threshold can be determined according to actual needs, which is not limited in this application.

[0044] As an example, the first visual encoder can be trained using an image and text related to the image to obtain a trained first visual encoder. The text related to the image can be information used to instruct the first large language model to perform a specified task based on the input image.

[0045] ​As an example, the parameters of all networks in the first multimodal model except the first visual encoder can be fixed, and only the first visual encoder can be trained to obtain a trained first visual encoder. This not only improves the feature extraction capability of the visual encoder but also reduces the amount of model parameter adjustment, thereby improving the training efficiency of the first visual encoder.

[0046] As another example, the first multimodal model can be trained as a whole to obtain a trained first multimodal model, and the visual encoder in the trained first multimodal model is the trained first visual encoder.

[0047] Step 102: Train the second largest language model and the trained first visual encoder to obtain a target multimodal model.

[0048] The second largest language model is included in the second multimodal model, that is, the second largest language model is the largest language model in the second multimodal model.

[0049] In this application, the second largest language model and the trained first visual encoder can be directly aligned and trained to obtain the target multimodal model, or the second largest language model in the second multimodal model can be trained first, and then the trained first visual encoder and the trained second largest language model are aligned and trained to obtain the target multimodal model. There is no limitation on this.

[0050] Alignment training can be understood as training the entire multimodal model, including the visual encoder and the large language model. For example, the goals of alignment training can include feature alignment, semantic alignment, etc.

[0051] It should be noted that, in the present application, the second largest language model and the first largest language model may be the same or different, and there is no limitation on this.

[0052] In addition, the number of model parameters of the second largest language model may be greater than the number of model parameters of the first largest language model, or may be less than or equal to the number of model parameters of the first largest language model, and this is not limited.

[0053] For example, the multimodal model trained in this application can be applied to visual question answering scenarios.

[0054] In the embodiments of the present application, the first visual encoder is trained by using the first large language model with a number of model parameters less than a preset threshold. Since the large language model used has a small model parameter size, the training efficiency can be improved and the training cost can be reduced. Furthermore, the first visual encoder is trained first, and then the trained first visual encoder is trained with the second large language model in the second multi-modal model to obtain the target multi-modal model. This not only improves the processing capability of the visual encoder, but also reduces the coupling between the visual encoder and the large language model in the multi-modal model. In addition, the trained first visual encoder has independent image processing capability and can be adapted to different second large language models, and has strong portability.

[0055] Figure 3 The flowchart of the training method of the multi-modal model provided for another embodiment of the present application is shown.

[0056] As shown in Figure 3 , the training method of the multi-modal model comprises the following steps.

[0057] Step 301: training the first visual encoder in the first multi-modal model to obtain a trained first visual encoder.

[0058] In the present application, step 301 can adopt any implementation manner of the embodiments of the present application, and therefore will not be described here.

[0059] Step 302: obtaining a second multi-modal model; wherein the second multi-modal model comprises a second large language model and a second visual encoder.

[0060] The second visual encoder in the second multi-modal model can be the same as or different from the trained first visual encoder, and no limitation is made thereto.

[0061] Optionally, the second multi-modal model can further comprise a first feature alignment layer, and the second visual encoder can be connected to the second large language model through the first feature alignment layer.

[0062] The first feature alignment layer is used to convert the visual features output by the second visual encoder to the same distribution space as the text features input to the second large language model, and it can be seen that the first feature alignment layer matches the second large language model.

[0063] Step 303: training the second large language model while fixing the parameters of the second visual encoder to obtain a trained second large language model.

[0064] In the present application, the second large language model can be trained with the parameters of the second visual encoder in the fixed second multi-modal model. Since only the second large language model is trained, the number of parameters of the model training is reduced, the training efficiency is improved, and the text processing ability of the trained second large language model can also be improved.

[0065] Since the large language model mainly focuses on instruction following ability, reasoning ability, etc., based on this, in some embodiments, a first sample set can be obtained, the first sample set can include cognitive type samples, and the second large language model can be trained using the first sample set with the parameters of the second visual encoder fixed to obtain a trained second large language model.

[0066] Among them, the cognitive type sample is a sample for training the cognitive ability of the second large language model in the second multi-modal model.

[0067] Among them, the cognitive type sample can include a graphic text sample, a pure text sample, etc. For example, a graphic text sample includes a picture A and corresponding text "Please simplify the solution process of question q according to picture A", and a pure text sample is "What is a U2I model in the field of artificial intelligence?"

[0068] Illustratively, the cognitive type sample can include but is not limited to a text knowledge sample, a mathematical reasoning sample, a geometric problem solving sample, etc.

[0069] Illustratively, the cognitive type sample can be used to train the cognitive task of the second large language model in the second multi-modal model, such as a cognitive task that can include but is not limited to a reasoning task, a knowledge task, etc.

[0070] Therefore, based on the first sample set including the cognitive type sample, the second large language model in the second multi-modal model is trained, which can improve the knowledge and reasoning ability of the second large language model, so that the multi-modal model is trained based on the trained second large language model, which can improve the accuracy of the content generated by the multi-modal model.

[0071] In some embodiments, the second multi-modal model can include a first feature alignment layer, the second visual encoder is connected with the second large language model through the first feature alignment layer, and based on this structure, the trained second large language model can be trained by the following method:

[0072] The first image-text sample can be obtained, the first image-text sample comprising a first sample image and first sample text related to the first sample image, a second visual encoder is used to perform feature extraction on the first sample image to obtain a first image feature vector, a first feature alignment layer is used to perform alignment processing on the first image feature vector to obtain a second image feature vector, and the first sample text is vectorized to obtain a first text vector, the second image feature vector and the first text vector are processed by a second large language model to obtain a first text output result, and the second large language model is trained according to the first text output result under the condition that the parameters of the second visual encoder and the first feature alignment layer are fixed, to obtain a trained second large language model.

[0073] The first sample text can be text prompt information, and the first sample text can be used to instruct the first multi-modal model to perform a specified task. For example, the specified task can be image classification, image description, OCR recognition, reasoning task, knowledge task, etc.

[0074] The dimension of the first text vector matches the input dimension of the second large language model.

[0075] For example, the first image feature vector and the first text vector can be spliced to obtain a spliced vector, and then the spliced vector is input into the second large language model, and the second large language model processes the spliced vector to generate the first text output result.

[0076] In the process of visual question answering based on the multi-modal model, the resolution of the obtained image can be different, and the aspect ratio of the image can also be different. Therefore, in order to enable the multi-modal model to process images with different size attributes, the second visual encoder can use the following method to perform feature extraction to obtain the first image feature vector.

[0077] The first sample image can be segmented according to the size attribute of the first sample image to obtain a plurality of first sub-block images, and the plurality of first sub-block images can be respectively subjected to feature extraction to obtain a first vector sequence, and each first sub-vector in the first vector sequence can be subjected to cross-attention calculation to obtain the first image feature vector.

[0078] The size of the plurality of first sub-block images can be the same or different, and no limitation is made in this regard.

[0079] The first vector sequence can include a first sub-vector obtained by performing feature extraction on each first sub-block image, and the number of first sub-vectors in the first vector sequence is the same as the number of first sub-block images.

[0080] The size attribute can include the resolution of the image, the aspect ratio of the image, etc.

[0081] For example, the aspect ratio of the first sample image is 224x224, the second visual encoder can divide the image into 14x14 first patch images, each of which has a size of 16x16, each first patch image is encoded into a vector with a length of 768, and finally 14x14=196 first sub-vectors are obtained, which are arranged in the order of from left to right and from top to bottom as a first vector sequence: [v1, v2, …, v196].

[0082] For another example, the resolution of the first sample image is 4096x4096, which can be divided into 256x256 first patch images, and the length of the first vector sequence is 65536.

[0083] As an example, the number of patch images in the length direction and the number of patch images in the width direction can be determined according to the aspect ratio of the first sample image, and the first sample image can be divided according to the number of patch images in the length direction and the number of patch images in the width direction to obtain a plurality of first patch images.

[0084] For example, the aspect ratio of the first sample image is 1024x64, and since the length is much larger than the width, the number of patch images in the length direction can be greater than the number of patch images in the width direction, for example, it can be divided into 64x4 first patch images, each first patch image can have a size of 16x16, and a first vector sequence with a length of 256 can be obtained after feature extraction.

[0085] Therefore, the first sample image can be adaptively divided according to the aspect ratio of the first sample image, so that the second visual editor can process different aspect ratio images.

[0086] It can be understood that a similar method can be used to adaptively divide the first sample image according to the resolution of the first sample image, so that the second visual editor can process images with different resolutions.

[0087] For image inputs with different size attributes, the corresponding vector sequence can be obtained through the visual encoder, and the number of patch images corresponding to the vector sequence can also be recorded to avoid mutual influence of image information in the same batch. Therefore, by retaining the size attributes of the image as much as possible and recording the number of patch images of different images in the batch, the cross-attention mechanism can be effectively implemented to further improve the perception ability of the visual encoder in the model structure.

[0088] Therefore, based on the size attribute of the first sample image, the first sample image is adaptively segmented, so that the second visual encoder can process sample images with different size attributes. Moreover, the first image feature vector obtained by performing cross-attention calculation on each first sub-vector in the first vector sequence not only retains the local features and spatial position information of the image, but also extracts the relationship between the local features, thereby obtaining the first text output result based on the cross-attention calculation, thereby improving the accuracy of the result.

[0089] Exemplarily, when the parameters of the second visual encoder and the first feature alignment layer are fixed, the first generation loss can be determined based on the first text output result and the first annotation result corresponding to the first image text sample. Based on the first generation loss, the parameters of the second largest language model are adjusted, and the second largest language model with adjusted parameters is continued to be trained until the training result conditions are met.

[0090] Therefore, if the second multimodal model also includes a first feature alignment layer, the first feature alignment layer can be used to align the output results of the second visual encoder so that the second largest language model can process the visual features and learn the semantic mapping between vision and text. When the parameters of the second visual encoder and the first feature alignment layer are fixed, only the second largest language model is trained, which can further improve the model training efficiency.

[0091] In some embodiments, plain text samples can also be used to train the second largest language model. In this case, the second largest language model is used to process the plain text samples to obtain text output results, and the second largest language model is trained based on the text output results, thereby improving the text processing capabilities of the second largest language model.

[0092] Step 304 : Perform alignment training on the trained first visual encoder and the trained second language model to obtain a target multimodal model.

[0093] In some embodiments, the trained first visual encoder, the second feature alignment layer, and the trained large language model may be sequentially connected to obtain a third multimodal model. The third multimodal model may be trained in the following manner:

[0094] The second sample image and the second sample text related to the second sample image can be obtained, the first trained visual encoder is used to extract features of the second sample image to obtain a third image feature vector, the second feature alignment layer is used to perform alignment processing on the third image feature vector to obtain a fourth image feature vector, the second sample text is vectorized to obtain a second text vector, the second large language model is trained, and the fourth image feature vector and the second text vector are processed to obtain a second text output result. The third multi-modal model is trained according to the second text output result to obtain a target multi-modal model.

[0095] The second sample text can be text prompt information, and the second sample text can be used to instruct the third multi-modal model to perform a specified task. For example, the specified task can be image classification, image description, OCR recognition, reasoning task, knowledge task, etc.

[0096] The dimension of the second text vector matches the input dimension of the second large language model.

[0097] For example, the fourth image feature vector and the second text vector can be spliced to obtain a spliced vector, and then the spliced vector is input into the trained second large language model, and the trained second large language model processes the spliced vector to generate the second text output result.

[0098] In the process of visual question answering based on the multi-modal model, the resolution of the obtained image can be different, and the aspect ratio of the image can also be different. Therefore, in order to enable the multi-modal model to process images with different size attributes, the trained first visual encoder can use the following method to extract features to obtain the third image feature vector.

[0099] The second sample image can be segmented according to the size attribute of the second sample image to obtain a plurality of second sub-block images, and the plurality of second sub-block images are respectively extracted to obtain a second vector sequence, and each second sub-vector in the second vector sequence is calculated by cross attention to obtain the third image feature vector.

[0100] The size of the plurality of second sub-block images can be the same or different, and no limitation is made thereto.

[0101] The second vector sequence can include a second sub-vector extracted by performing feature extraction on each second sub-block image, and the number of second sub-vectors in the second vector sequence is the same as the number of second sub-block images.

[0102] The size attribute can include the resolution of the image, the aspect ratio of the image, etc.

[0103] The specific method of segmenting the second sample image according to the size attribute of the second sample image is similar to the method of segmenting the first sample image according to the size attribute of the first sample image in the above embodiment, and thus will not be described here.

[0104] Thus, adaptive segmentation of the second sample image based on the size attribute of the second sample image can realize processing of sample images with different size attributes by the trained first visual encoder, and the third image feature vector obtained by cross-attention calculation of each second sub-vector in the second vector sequence not only retains local features and spatial position information of the image, but also extracts relationships between local features, thereby obtaining a second text output result based on cross-attention calculation, and improving the accuracy of the result.

[0105] For example, a second generation loss can be determined according to the second text output result and a second label corresponding to the second image-text sample, and parameters of the third multi-modal model can be adjusted according to the second generation loss, and the third multi-modal model with adjusted parameters can be continuously trained until a training result condition is met.

[0106] For example, a contrast loss can also be calculated according to the similarity between the fourth image feature vector and a text feature vector output by any layer of the trained second large language model, and the parameters of the third multi-modal model can be adjusted according to the second generation loss and the contrast loss.

[0107] The goal of the contrast loss is to make the feature representations of positive image-text samples (i.e., image-text pairs with semantic correlation) similar and the feature representations of negative image-text samples (i.e., image-text pairs with no semantic correlation) different.

[0108] Since the contrast loss can realize narrowing the distance of positive image-text samples, widening the distance of negative image-text samples, and forcing the model to learn the semantic correspondence between vision and text, and the generation loss can realize minimizing the difference between generated text and real text, and making the model implicitly learn the semantic mapping between vision and text, therefore, training the multi-modal model by combining the two losses can realize semantic alignment between the visual encoder and the large language model.

[0109] Thus, the output result of the trained first visual encoder can be aligned by the second feature alignment layer, so as to facilitate the trained second large language model to process visual features, learn the semantic mapping between vision and text, and perform semantic alignment.

[0110] In the alignment training process, the trained first visual encoder and the trained second large language model are both open for training. Compared with other training methods, the application respectively obtains a visual encoder with strong processing capability and a large language model with strong processing capability, so that the alignment training does not need to use a large amount of data, greatly reducing the training cost and time cost, and using a small amount of data for model alignment can obtain a multi-modal model with good comprehensive effect.

[0111] It should be noted that in the present application, the first visual encoder can be trained first, and then the second large language model can be trained, or the second large language model can be trained first, and then the first visual encoder can be trained, or the first visual encoder and the second large language model can be trained in parallel. The training order of the first visual encoder and the second large language model is not limited in the present application.

[0112] In the embodiments of the present application, the parameters of the second visual encoder in the second multi-modal model are fixed, and the second large language model in the second multi-modal model is trained. Since the parameters of the second visual encoder are fixed, the second large language model is trained, so the parameter amount of model training is reduced to a certain extent. At the same time, training in this way can improve the processing capability of the language model. Moreover, by training the visual encoder and the large language model separately, not only can the training time be greatly shortened, but also the coupling between the visual encoder and the large language model in the target multi-modal model obtained by training is reduced.

[0113] Figure 4 The flowchart of the training method of the multi-modal model provided by another embodiment of the present application is shown.

[0114] As shown in Figure 4 , the training method of the multi-modal model comprises:

[0115] Step 401, training the first visual encoder in the first multi-modal model by using a third sample set comprising perception type samples to obtain a trained first visual encoder.

[0116] In order to improve the perception ability of the visual encoder, such as the ability to obtain visual perception features (such as color, size, position, etc.) from an image, in some embodiments, a third sample set can be obtained, the third sample set can comprise perception type samples, and the first visual encoder can be trained by using the third sample set to obtain a trained first visual encoder.

[0117] The perception type sample is a sample used for training the perception ability of the first visual encoder in the first multi-modal model.

[0118] The perception type sample can be a graphic text sample.

[0119] Exemplarily, the sample of the perception type can include but is not limited to an image description sample, an OCR (Optical Character Recognition) sample, an image classification sample, etc.

[0120] The text sample in the image description sample can be used to instruct the multi-modal model to perform a task of describing image content, the text sample in the OCR sample can be used to instruct the multi-modal model to perform an OCR recognition task, and the text sample in the image classification sample can be used to instruct the multi-modal model to perform an image classification task.

[0121] Exemplarily, the sample of the perception type can be used to train the first visual encoder in the first multi-modal model to perform a perception task, such as an image description task, an OCR task, an image classification task, etc.

[0122] Thus, training the first visual encoder in the first multi-modal model based on the third sample set including the sample of the perception type can improve the perception capability of the first visual encoder, so that the multi-modal model trained based on the trained first visual encoder can improve the accuracy of the content generated by the multi-modal model.

[0123] In some embodiments, the first multi-modal model can further include a third feature alignment layer, and the first visual encoder is connected to the first large language model through the third feature alignment layer. Based on this structure, the trained first visual encoder can be trained by the following method:

[0124] A third image-text sample is obtained, the third image-text sample includes a third sample image and a third sample text related to the third sample image, and the first visual encoder is used to extract features of the third sample image to obtain a fifth image feature vector. The third feature alignment layer is used to align the fifth image feature vector to obtain a sixth image feature vector, and the third sample text is vectorized to obtain a third text vector. The first large language model is used to process the sixth image feature vector and the third text vector to obtain a third text output result. The first visual encoder is trained according to the third text output result to obtain the trained first visual encoder.

[0125] The third sample text can be text prompt information, and the third sample text can be used to instruct the first multi-modal model to perform a specified task. Exemplarily, the specified task can be image classification, image description, OCR recognition, reasoning task, knowledge task, etc.

[0126] The dimension of the third text vector matches the input dimension of the first large language model.

[0127] Exemplarily, the sixth image feature vector and the third text vector can be spliced to obtain a spliced vector, and then the spliced vector is input into the first large language model, and the first large language model processes the spliced vector to generate a third text output result.

[0128] In the process of visual question answering based on the multi-modal model, the resolution of the obtained image can be different, and the aspect ratio of the picture can also be different. Based on this, in order to enable the multi-modal model to process images with different size attributes, exemplarily, the first visual encoder can adopt the following method for feature extraction to obtain the fifth image feature vector:

[0129] The third sample image can be segmented according to the size attribute of the third sample image to obtain a plurality of third sub-block images, and the plurality of third sub-block images are respectively subjected to feature extraction to obtain a third vector sequence, and each third sub-vector in the third vector sequence is subjected to cross-attention calculation to obtain the fifth image feature vector.

[0130] The size of the plurality of third sub-block images can be the same or different, and no limitation is made thereto.

[0131] The third vector sequence can include a third sub-vector obtained by performing feature extraction on each third sub-block image, and the number of third sub-vectors in the third vector sequence is the same as the number of third sub-block images.

[0132] The size attribute can include the resolution of the image, the aspect ratio of the image, etc.

[0133] The specific method of segmenting the third sample image according to the size attribute of the third sample image is similar to the method of segmenting the first sample image according to the size attribute of the first sample image in the above embodiment, and therefore will not be described here.

[0134] Thus, based on the size attribute of the third sample image, the third sample image is adaptively segmented, the processing of the first visual encoder on sample images with different size attributes can be realized, and the fifth image feature vector obtained by performing cross-attention calculation on each third sub-vector in the third vector sequence not only retains the local features and spatial position information of the image, but also extracts the relationship between the local features, thereby obtaining the third text output result based on the cross-attention calculation, and improving the accuracy of the result.

[0135] Exemplarily, the third generation loss can be determined according to the third text output result and the third annotation result corresponding to the third text sample, the parameters of the first visual encoder are adjusted according to the third generation loss, and the first visual encoder with the adjusted parameters is continuously trained until the training result condition is met.

[0136] Therefore, if the first multi-modal model further comprises a third feature alignment layer, the third feature alignment layer can be used to align the output result of the first visual encoder, so as to facilitate the first large language model to process the visual features, learn the semantic mapping between the vision and the text, and perform semantic alignment.

[0137] At step 402, the second large language model in the second multi-modal model is trained by using the first sample set comprising the samples of the perception type, to obtain a trained second large language model, while the parameters of the second visual encoder are fixed.

[0138] In this application, step 402 can refer to the relevant content described in the above embodiments, which will not be repeated here.

[0139] At step 403, the trained first visual encoder and the trained second large language model are aligned trained by using the second sample set comprising the samples of the perception type and the samples of the cognition type, to obtain a target multi-modal model.

[0140] When the trained first visual encoder and the trained second large language model are aligned trained, the second sample set comprising the samples of the perception type and the samples of the cognition type can be used for alignment training, so as to further improve the perception ability of the visual encoder and the cognition ability of the large language model, and further improve the accuracy of the generation result of the target multi-modal model.

[0141] Since the samples of the perception type and the samples of the cognition type in the second sample set can be image-text samples, the process of alignment training in this application can refer to the description in the above embodiments, which will not be repeated here.

[0142] In the embodiments of this application, the visual encoder and the large language model are trained in stages, and the decoupled training is divided into three stages: in the first stage, the first video encoder is trained by using the third sample set comprising the samples of the perception type, which can improve the perception ability of the video encoder; in the second stage, the second large language model is trained by using the first sample set comprising the samples of the cognition type, which can improve the cognition ability of the large language model; in the third stage, the video encoder and the large language model trained in the previous two stages are aligned trained by using the second sample set comprising the samples of the perception type and the samples of the cognition type, so as to further improve the perception ability of the video encoder and the cognition ability of the large language model, and further improve the accuracy of the generation result of the multi-modal model trained.

[0143] In addition, the visual encoder obtained through the first-stage training has independent perception capability, can be adapted to large language models of different architectures, and has strong portability. After the visual encoder is aligned, the visual encoder can be migrated to a new large language model, and only a small amount of data is required to achieve high generation accuracy. In addition, the large language model maintains the pure text training feature, and can be used as an independent large language model when there is no image input.

[0144] Figure 5 A flowchart of a text generation method provided by an embodiment of the present application is shown.

[0145] As shown in Figure 5 , the text generation method comprises the following steps:

[0146] In step 501, a target image and text prompt information corresponding to the target image are obtained.

[0147] The text prompt information can be used to instruct the model to perform a specified task based on the target image. For example, the specified task can be image classification, image description, OCR recognition, reasoning task, knowledge task, etc.

[0148] In step 502, a target multi-modal model is used to process the target image and the text prompt information to generate a text output result.

[0149] The target multi-modal model can be trained by using the training method of any of the above embodiments.

[0150] In the present application, the visual encoder in the target multi-modal model can be used to extract features of the target image to obtain an image feature vector, and the large language model in the target multi-modal model can be used to process the text vector of the text prompt information and the image feature vector to obtain the text output result.

[0151] For example, the target multi-modal model further includes a feature alignment layer, which can be used to align the image feature vector output by the visual encoder to obtain an aligned image feature vector, and then the large language model can be used to process the text vector of the text prompt information and the aligned image feature vector to obtain the text output result.

[0152] For example, the target multi-modal model can be used for target images with different sizes and attributes, and the specific process can be referred to the above embodiments, which will not be repeated here.

[0153] In the embodiment of the present application, the first visual encoder is trained first in the above-mentioned embodiment, and then alignment training is performed based on the trained first visual encoder and the second large language model, which improves the processing capability of the visual encoder and further improves the processing capability of the multi-modal target. Therefore, by using the target multi-modal model to process the target image and the text prompt information, the accuracy of the text output result can be improved.

[0154] To implement the above-mentioned embodiment, the embodiment of the present application further provides a multi-modal model training device. Figure 6 The structure diagram of the multi-modal model training device provided by an embodiment of the present application is shown in the figure.

[0155] As shown in the figure, the multi-modal model training device 600 comprises: Figure 6

[0156] The first training module 610 is configured to train the first visual encoder in the first multi-modal model to obtain a trained first visual encoder; wherein the number of model parameters of the first large language model in the first multi-modal model is less than a preset threshold.

[0157] The second training module 620 is configured to train the second large language model and the trained first visual encoder to obtain a target multi-modal model; wherein the second large language model is included in a second multi-modal model.

[0158] Optionally, the second training module 620 is configured to:

[0159] obtain a second multi-modal model; wherein the second multi-modal model comprises the second large language model and a second visual encoder;

[0160] train the second large language model to obtain the trained second large language model while fixing the parameters of the second visual encoder in the second multi-modal model;

[0161] perform alignment training on the trained first visual encoder and the trained second large language model to obtain the target multi-modal model.

[0162] Optionally, the second training module 620 is configured to:

[0163] obtain a first sample set; wherein the first sample set comprises samples of a cognitive type;

[0164] train the second large language model to obtain the trained second large language model while fixing the parameters of the second visual encoder using the first sample set.

[0165] ​Optionally, the second multi-modal model further comprises a first feature alignment layer, and the second training module 620 is configured to:

[0166] obtain a first image-text sample, wherein the first image-text sample comprises a first sample image and first sample text related to the first sample image;

[0167] perform feature extraction on the first sample image by using the second visual encoder to obtain a first image feature vector;

[0168] perform alignment processing on the first image feature vector by using the first feature alignment layer to obtain a second image feature vector;

[0169] perform processing on the second image feature vector and a first text vector corresponding to the first sample text by using the second large language model to obtain a first text output result;

[0170] In a case where parameters of the second visual encoder and the first feature alignment layer are fixed, the second large language model is trained according to the first text output result to obtain the trained second large language model.

[0171] Optionally, the second training module 620 is configured to:

[0172] segment the first sample image according to a size attribute of the first sample image to obtain a plurality of first sub-block images;

[0173] perform feature extraction on the plurality of first sub-block images respectively to obtain a first vector sequence;

[0174] perform cross-attention calculation on each first sub-vector in the first vector sequence to obtain the first image feature vector.

[0175] Optionally, the second training module 620 is configured to:

[0176] obtain a second sample set, wherein the second sample set comprises a perception type sample and a cognition type sample;

[0177] perform alignment training on the trained first visual encoder and the trained second large language model by using the second sample set to obtain the target multi-modal model.

[0178] Optionally, the second training module 620 is configured to:

[0179] obtain a second image-text sample comprising a second sample image and second sample text related to the second sample image;

[0180] The trained first visual encoder is used for feature extraction on the second sample image to obtain a third image feature vector.

[0181] The second feature alignment layer is used for alignment processing on the third image feature vector to obtain a fourth image feature vector.

[0182] The trained second large language model is used for processing on the fourth image feature vector and a second text vector corresponding to the second sample text to obtain a second text output result.

[0183] According to the second text output result, the trained first visual encoder, the second feature alignment layer and the trained second large language model are sequentially connected to obtain a third multi-modal model, and the third multi-modal model is trained to obtain the target multi-modal model.

[0184] Optionally, the second training module 620 is configured to:

[0185] According to the size attribute of the second sample image, the second sample image is segmented to obtain a plurality of second sub-block images.

[0186] The plurality of second sub-block images are respectively subjected to feature extraction to obtain a second vector sequence.

[0187] Each second sub-vector in the second vector sequence is subjected to cross-attention calculation to obtain the third image feature vector.

[0188] Optionally, the first training module 610 is configured to:

[0189] A third sample set is obtained; wherein the third sample set includes samples of a perception type.

[0190] The first visual encoder is trained by using the third sample set to obtain the trained first visual encoder.

[0191] Optionally, the first multi-modal model further includes a third feature alignment layer, and the first training module 610 is configured to:

[0192] A third image-text sample is obtained; wherein the third image-text sample includes a third sample image and a third sample text related to the third sample image.

[0193] The first visual encoder is used for feature extraction on the third sample image to obtain a fifth image feature vector.

[0194] The third feature alignment layer is used for alignment processing on the fifth image feature vector to obtain a sixth image feature vector.

[0195] The first large language model is used to process a third text vector corresponding to the sixth image feature vector and the third sample text, to obtain a third text output result.

[0196] According to the third text output result, the first visual encoder is trained to obtain the trained first visual encoder.

[0197] Optionally, the first training module 610 is configured to:

[0198] According to the size attribute of the third sample image, the third sample image is segmented to obtain a plurality of third sub-block images;

[0199] The plurality of third sub-block images are respectively subjected to feature extraction to obtain a third vector sequence;

[0200] Each third sub-vector in the third vector sequence is subjected to cross-attention calculation to obtain the fifth image feature vector.

[0201] It should be noted that the explanation and description of the foregoing training method embodiment of the multi-modal model also apply to the training device of the multi-modal model of this embodiment, and therefore will not be described here.

[0202] In the embodiments of the present application, the first visual encoder is trained by using the first large language model with a number of model parameters less than a preset threshold. Since the large language model used has a small model parameter size, the training efficiency can be improved and the training cost can be reduced. Furthermore, the first visual encoder is trained first, and then the trained first visual encoder is trained with the second large language model. This not only improves the processing capability of the visual encoder, but also reduces the coupling between the visual encoder and the large language model in the multi-modal model. In addition, the trained first visual encoder has independent image processing capability and can be adapted to different second large language models, and has strong portability.

[0203] In order to realize the above-mentioned embodiments, the embodiments of the present application also propose a text generation device. Figure 7 The structure diagram of the text generation device provided by an embodiment of the present application is shown.

[0204] As shown in Figure 7 The text generation device 700 includes:

[0205] The acquisition module 710 is configured to acquire a target image and text prompt information corresponding to the target image.

[0206] The generation module 720 is configured to adopt the target multi-modal model to process the target image and the text prompt information to generate a text output result. The multi-modal model is trained by using the training method in any of the preceding embodiments.

[0207] It should be noted that the explanation of the text generation method in the foregoing embodiments is also applicable to the text generation apparatus in this embodiment, and thus will not be described here again.

[0208] In the embodiments of the present application, the first visual encoder is trained first in the foregoing embodiments, and then the alignment training is performed based on the trained first visual encoder and the second large language model, which improves the processing capability of the visual encoder and further improves the processing capability of the multi-modal target. Therefore, the target image and the text prompt information are processed by using the target multi-modal model, which can improve the accuracy of the text output result.

[0209] According to the embodiments of the present application, the present application further provides an electronic device, a readable storage medium and a computer program product.

[0210] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0211] As shown in Figure 8 The device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 802 or a computer program loaded from a storage unit 808 into a RAM (Random Access Memory) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0212] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through computer networks, such as the Internet, and / or various telecommunication networks.

[0213] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the multi-modal training method. For example, in some embodiments, the multi-modal training method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the multi-modal training method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the multi-modal training method by other any appropriate means, such as by means of firmware.

[0214] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on a Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0215] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0216] In the context of this application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a linearly-programmed electronic storage, a portable computer diskette, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory), or a flash memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0217] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0218] The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0219] The computer system can include clients and servers. This relationship can be

[0220] It should be noted that the electronic device used to implement the text generation method of the embodiments of the present application is similar in structure to the electronic device shown in the above figure, and therefore will not be described here.

[0221] According to the embodiments of the present application, the present application also provides a computer program product, when the instruction processor in the computer program product executes, executes the multi-modal training method proposed in the above embodiments of the present application, or executes the text generation method proposed in the above embodiments of the present application.

[0222] It should be understood that the steps can be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present application can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions disclosed in the present application can be achieved, which is not limited herein.

[0223] The above specific embodiments do not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A multimodal model training method, comprising: Training a first visual encoder in a first multimodal model to obtain a trained first visual encoder; wherein the number of model parameters of a first large language model in the first multimodal model is less than a preset threshold; The second largest language model and the trained first visual encoder are trained to obtain a target multimodal model; wherein the second largest language model is included in the second multimodal model.

2. The method according to claim 1, wherein The training of the second largest language model and the trained first visual encoder to obtain a target multimodal model includes: training the second large language model while fixing parameters of the second visual encoder in the second multimodal model to obtain the trained second large language model; Alignment training is performed on the trained first visual encoder and the trained second language model to obtain the target multimodal model.

3. The method according to claim 2, wherein: The step of training the second large language model while fixing the parameters of the second visual encoder in the second multimodal model to obtain the trained second large language model includes: Acquire a first sample set; wherein the first sample set includes samples of cognitive type; With the parameters of the second visual encoder fixed, the second large language model is trained using the first sample set to obtain the trained second large language model.

4. The method according to claim 2, wherein: The second multimodal model further includes a first feature alignment layer. The second large language model is trained while the parameters of the second visual encoder are fixed to obtain the trained second large language model, including: Acquire a first image and text sample; wherein the first image and text sample includes a first sample image and a first sample text related to the first sample image; Using the second visual encoder to perform feature extraction on the first sample image to obtain a first image feature vector; Using the first feature alignment layer, aligning the first image feature vector to obtain a second image feature vector; Using the second largest language model, processing the second image feature vector and the first text vector corresponding to the first sample text to obtain a first text output result; While fixing the parameters of the second visual encoder and the first feature alignment layer, the second largest language model is trained according to the first text output result to obtain the trained second largest language model.

5. The method according to claim 4, wherein: The step of extracting features from the first sample image using the second visual encoder to obtain a first image feature vector includes: Segmenting the first sample image according to a size attribute of the first sample image to obtain a plurality of first block images; Performing feature extraction on each of the plurality of first block images to obtain a first vector sequence; A cross-attention calculation is performed on each first sub-vector in the first vector sequence to obtain the first image feature vector.

6. The method of claim 2, wherein: The aligning training of the trained first visual encoder and the trained second language model to obtain the target multimodal model includes: Acquire a second sample set; wherein the second sample set includes perceptual type samples and cognitive type samples; The second sample set is used to perform alignment training on the trained first visual encoder and the trained second large language model to obtain the target multimodal model.

7. The method of claim 2, wherein: The aligning training of the trained first visual encoder and the trained second language model to obtain the target multimodal model includes: Acquiring a second image-text sample including a second sample image and a second sample text related to the second sample image; Using the trained first visual encoder, perform feature extraction on the second sample image to obtain a third image feature vector; Using a second feature alignment layer, aligning the third image feature vector to obtain a fourth image feature vector; Using the trained second language model, the fourth image feature vector and the second text vector corresponding to the second sample text are processed to obtain a second text output result; According to the second text output result, the trained first visual encoder, the second feature alignment layer and the trained second large language model are sequentially connected to obtain a third multimodal model for training to obtain the target multimodal model.

8. The method of claim 7, wherein: The step of extracting features from the second sample image using the trained first visual encoder to obtain a third image feature vector includes: Segmenting the second sample image according to a size attribute of the second sample image to obtain a plurality of second block images; Performing feature extraction on each of the plurality of second block images to obtain a second vector sequence; A cross attention calculation is performed on each second sub-vector in the second vector sequence to obtain the third image feature vector.

9. The method of claim 1, wherein: The step of training the first visual encoder in the first multimodal model to obtain a trained first visual encoder includes: Acquire a third sample set; wherein the third sample set includes samples of a perceptual type; The first visual encoder is trained using the third sample set to obtain the trained first visual encoder.

10. The method of claim 1, wherein: The first multimodal model further includes a third feature alignment layer, and the training of the first visual encoder in the first multimodal model to obtain a trained first visual encoder includes: Acquire a third image and text sample; wherein the third image and text sample includes a third sample image and a third sample text related to the third sample image; performing feature extraction on the third sample image using the first visual encoder to obtain a fifth image feature vector; Using the third feature alignment layer, aligning the fifth image feature vector to obtain a sixth image feature vector; Using the first language model, processing the sixth image feature vector and a third text vector corresponding to the third sample text to obtain a third text output result; The first visual encoder is trained according to the third text output result to obtain the trained first visual encoder.

11. The method according to claim 10, wherein: The using the first visual encoder to extract features from the third sample image to obtain a fifth image feature vector includes: Segmenting the third sample image according to a size attribute of the third sample image to obtain a plurality of third block images; performing feature extraction on each of the plurality of third block images to obtain a third vector sequence; A cross-attention calculation is performed on each third sub-vector in the third vector sequence to obtain the fifth image feature vector.

12. A text generation method comprising: Obtaining a target image and text prompt information corresponding to the target image; A target multimodal model is used to process the target image and the text prompt information to generate a text output result; wherein the multimodal model is trained using the training method described in any one of claims 1-11.

13. A multimodal model training device, comprising: a first training module, configured to train a first visual encoder in a first multimodal model to obtain a trained first visual encoder; wherein the number of model parameters of a first large language model in the first multimodal model is less than a preset threshold; A second training module is used to train a second large language model and the trained first visual encoder to obtain a target multimodal model; wherein the second large language model is included in the second multimodal model.

14. A text generation device, comprising: An acquisition module, used to acquire a target image and text prompt information corresponding to the target image; A generation module is used to use a target multimodal model to process the target image and the text prompt information to generate a text output result; wherein, the multimodal model is trained using the training method described in any one of claims 1-11.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images