Multi-modal large model training method, image-text translation method and device

Abstract

The invention discloses a multi-modal large model training method and an image-text translation method and device, and the method comprises the steps: determining a training sample comprising a training text, a translation text, a training image and inquiry information, training a to-be-trained multi-modal large model based on the training sample, and obtaining a trained multi-modal large model, the multi-modal large model comprises an image encoder, an adapter and a large language model, and the image encoder is used for extracting information with different granularities from pixel representation of an image and converting the information into image vector representation; the adapter is used for converting the dimension represented by the first image vector and the dimension represented by the second image vector into dimensions accepted by a large language model, and the large language model is used for outputting corresponding answers according to the input image vector representation and inquiry information. According to the invention, the fine-grained image encoder is added to the model framework, so that the defect that the multi-modal large model is insufficient in fine-grained image information encoding is overcome, and the visual capture capability of the multi-modal large model is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image-text translation, and more specifically, to a multimodal large model training method, an image-text translation method and a device. Background Art

[0002] Currently, the Multimodal Large Language Model (MLLM) has demonstrated outstanding capabilities in many multimodal tasks, such as image understanding, visual question answering, etc. Therefore, the MLLM is also used for image-to-text translation tasks. However, when using the MLLM for image-to-text translation, the visual capture effect is often poor, resulting in unsatisfactory image-to-text translation results. Summary of the invention

[0003] In view of the above problems, the present application proposes a multimodal large model training method, image-text translation method and device to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a method for training a multimodal large model, the method comprising:

[0005] Determine a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information; train a multimodal large model to be trained based on the training sample to obtain a trained multimodal large model, wherein the multimodal large model includes: an image encoder, wherein the image encoder includes a first image encoder and a second image encoder, wherein the first image encoder is used to extract first information from a pixel representation of an image and convert it into a first image vector representation, and the second image encoder is used to extract second information from a pixel representation of the image and convert it into a second image vector representation, wherein a granularity of the second information is finer than a granularity of the first information; an adapter, wherein the adapter is used to convert a dimension of the first image vector representation and a dimension of the second image vector representation into a dimension accepted by a large language model; and a large language model, wherein the large language model is used to output a corresponding answer according to the input image vector representation and query information.

[0006] In a second aspect, an embodiment of the present application provides a method for image-text translation, the method comprising: obtaining an image to be translated, wherein the image to be translated includes text to be translated; inputting the image to be translated into a trained multimodal large model, and obtaining a target text corresponding to the text to be translated output by the trained multimodal large model, wherein the trained multimodal large model is trained based on the above-mentioned multimodal large model training method.

[0007] In a third aspect, an embodiment of the present application provides a training device for a multimodal large model, the device comprising: a training sample determination module, used to determine a training sample, wherein the training sample comprises a training text, a translation text corresponding to the training text, a training image comprising the training text, and query information; a model training module, used to train a multimodal large model to be trained based on the training sample to obtain a trained multimodal large model, wherein the multimodal large model comprises: an image encoder, wherein the image encoder comprises a first image encoder and a second image encoder, wherein the first image encoder is used to extract first information from a pixel representation of an image and convert it into a first image vector representation, and the second image encoder is used to extract second information from a pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than that of the first information; an adapter, wherein the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by a large language model; a large language model, wherein the large language model is used to output a corresponding answer according to the input image vector representation and query information.

[0008] In a fourth aspect, an embodiment of the present application provides a graphic-text translation device, the device comprising: an image acquisition module, used to acquire an image to be translated, wherein the image to be translated includes a text to be translated; a graphic-text translation module, used to input the image to be translated into a trained multimodal large model, and obtain a target text corresponding to the text to be translated output by the trained multimodal large model, wherein the trained multimodal large model is trained based on the above-mentioned multimodal large model training method.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0010] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0011] The training method, image-text translation method and device of the multimodal large model provided in the embodiments of the present application determine a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, including a training image of the training text and query information, and trains the multimodal large model to be trained based on the training sample to obtain a trained multimodal large model, wherein the multimodal large model includes an image encoder, an adapter and a large language model, wherein the image encoder includes a first image encoder and a second image encoder, wherein the first image encoder is used to extract first information from a pixel representation of an image and convert it into a first image vector representation, and the second image encoder is used to extract second information from the pixel representation of the image, wherein the granularity of the second information is finer than that of the first information, the adapter is used to convert the dimension of the first image vector representation into the dimension accepted by the large language model, and the large language model is used to output a corresponding answer according to the input image vector representation and query information, thereby compensating for the disadvantage of insufficient encoding of fine-grained image information by the multimodal large model by adding a fine-grained image encoder to the model framework, so as to improve the visual capture capability of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of a process flow of a multimodal large model training method provided in an embodiment of the present application is shown;

[0014] Figure 2 A schematic diagram of the framework of a multimodal large model provided in an embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of a process flow of a multimodal large model training method provided in an embodiment of the present application is shown;

[0016] Figure 4 A schematic diagram of translation capability enhancement training provided by an embodiment of the present application is shown;

[0017] Figure 5 A schematic diagram of image encoder training provided by an embodiment of the present application is shown;

[0018] Figure 6 This application shows Figure 3 The flowchart of step S260 of the multimodal large model training method shown is as follows;

[0019] Figure 7 A schematic diagram of image-text translation training provided in an embodiment of the present application is shown;

[0020] Figure 8 This application shows Figure 6 A flowchart diagram of an embodiment of step S263 of a method for training a multimodal large model is shown;

[0021] Fig. 9 This application shows Figure 6 A flowchart diagram of an embodiment of step S263 of a method for training a multimodal large model is shown;

[0022] Fig.10 A schematic diagram of a process flow of a multimodal large model training method provided in an embodiment of the present application is shown;

[0023] Fig.11 A schematic diagram showing a flow chart of a method for image-text translation provided by an embodiment of the present application is shown;

[0024] Fig.12 A module block diagram of a multimodal large model training device provided in one embodiment of the present application is shown;

[0025] Fig.13 A module block diagram of a graphic-text translation device provided by an embodiment of the present application is shown;

[0026] Fig.14 A block diagram of an electronic device for executing a startup abnormality processing method according to an embodiment of the present application is shown;

[0027] Fig.15 A storage unit for storing or carrying program codes for implementing a method for handling power-on exceptions according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0029] End-to-end (E2E) image translation (IT), which directly translates images with text in one language into text in another language, has attracted more and more attention. Compared with traditional cascaded image translation, E2E image translation combines the automatic text recognition module (Optical Character Recognition, OCR) with the machine translation module (Machine Translation, MT), so that it does not need to recognize the text in the source language, thus having less error propagation and lower latency.

[0030] At present, there are two main challenges in end-to-end image-text translation. One is the data scarcity problem. Compared with OCR data and machine translation data, training the end-to-end image-text translation model requires triple data (source language image, source language text, target language text). Currently, there is no such parallel corpus data or only a small amount of data exists. The other is the modality gap problem. Because end-to-end image-text translation directly translates images containing source language text into target language text, the difference between image modality and text modality makes it difficult to train. If we want to better improve the quality of translation, the modality gap problem must be solved.

[0031] Based on the above two challenges, current end-to-end image-to-text translation methods are all based on modular training with small-parameter models. Specifically, they use a large amount of OCR data and machine translation data to train different modules of the image-to-text translation model separately, then design a mapping module to map the vector representation of the image modality to the vector representation of the text modality, and finally connect the entire model and use a small amount of image-to-text translation data for overall fine-tuning.

[0032] At present, multimodal large models have demonstrated outstanding capabilities in many multimodal tasks, such as image understanding, visual question answering, and other tasks. Its basic framework includes three modules: image encoder, adapter, and large language model (LLM). Among them, the image encoder aims to encode the pixel representation of the image into a continuous vector representation, the adapter aims to adjust the dimension of the vector representation of the image to the dimension accepted by the LLM, and the LLM aims to understand the vector representation of the image and the query and then output the corresponding answer. Such a framework allows the multimodal large model to directly output the corresponding answer based on the query when accepting image input, which coincides with the idea of ​​end-to-end image-text translation. Therefore, designing a multimodal large model that focuses on image-text translation tasks is in line with the design and idea of ​​end-to-end image-text translation.

[0033] However, after testing, it was found that the image-text translation performance of most open-source multimodal large models is poor, and even some closed-source multimodal large models such as GPT-4o have the phenomenon of "unable to translate" in image-text translation tasks. In addition, during the test process, it was also found that most of the current multimodal large models with a parameter size of 7B have poor capabilities in OCR tasks, and it was found that the main reason is that their image encoders directly use the open-source contrastive learning pre-trained visual language model (Contrastive Language-Image Pretrain, CLIP) model and remain frozen throughout the training process. Although some large-parameter multimodal large models (above 26B) have significantly improved OCR capabilities, the cost of expanding the size of the visual encoder to more than 6B (CLIP parameters are 300M) and retraining the visual encoder using hundreds of millions of image and text data is huge.

[0034] In addition, previous end-to-end image-text translation models were trained in a modular way, that is, some modules were trained with OCR data, and other modules were trained with machine translation data. As a "black box" model, the multimodal large model cannot be trained separately, which makes it difficult to migrate the previous end-to-end image-text translation methods to the multimodal large model training. In addition, due to the lack of data, the previous end-to-end image-text translation models were very limited in translation scenarios, and could only achieve good translation results in narrow scenarios such as subtitles, but could not have good performance in multiple scenarios at the same time.

[0035] In view of the above problems, the inventors have discovered after long-term research and proposed the training method, image-text translation method and device of the multimodal large model provided in the embodiments of the present application, which compensates for the shortcomings of the multimodal large model in insufficient encoding of fine-grained image information by adding a fine-grained image encoder to the model framework, so as to improve the visual capture ability of the multimodal large model. Among them, the specific training method of the multimodal large model and the image-text translation method are described in detail in the subsequent embodiments.

[0036] See also Figure 1 , Figure 1 The flowchart of the training method of the multimodal large model provided by an embodiment of the present application is shown. The method is used to improve the visual capture capability of the multimodal large model by adding a fine-grained image encoder to the model framework to compensate for the shortcomings of the multimodal large model in insufficient encoding of fine-grained image information. In a specific embodiment, the method is applied to Fig.12 The multimodal large model training device 200 and the electronic device 100 ( Fig.14). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a smart phone, a tablet computer, a desktop computer, a server, etc., which is not limited here. Figure 1 The process shown is described in detail, and the training method of the multimodal large model may specifically include the following steps:

[0037] Step S110: determining a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information.

[0038] It can be understood that the purpose of this embodiment is to train a multimodal large model that can be used for image-text translation. Therefore, training samples including training text, translation text corresponding to the training text, training images including training text, and query information can be collected to train the multimodal large model to be trained, and a trained multimodal large model that can be used for image-text translation of images can be obtained, and then used for image-text translation.

[0039] The training text may be original language data, for example, a sentence, a paragraph or a document. The selection of the training text may represent the diversity and complexity of the target language usage to ensure that the multimodal large model can generalize to unseen data. Optionally, the training text may be a text corresponding to the first language, wherein the first language may include Chinese, English, Russian, Japanese, Korean, etc., which is not limited here.

[0040] The translation text corresponding to the training text may be a translated version of the training text, and the translation text corresponding to the training text should be accurate and fluent to reflect the natural usage of the target language. Optionally, the translation text corresponding to the training text may be a text corresponding to a second language, wherein the second language is different from the first language, and the second language may include Chinese, English, Russian, Japanese, Korean, etc., which are not limited here. As an example, the first language may be English, and the second language may be Chinese.

[0041] The training image including the training text may refer to a training image formed by embedding the training text into an image. Optionally, the training image includes at least the training text, that is, the training image may include only the training text, or may include both the training text and the background image, etc., which is not limited here.

[0042] The query information may be related instructions for guiding the training of the multimodal large model, so that the multimodal large model updates parameters in the direction of improving the image-text translation effect.

[0043] As an implementable method, a large amount of OCR data including various scenes can be collected from the Internet, and training text and training images including the training text can be obtained based on the OCR data. After that, the training text is translated using translation software to obtain the translated text, and in the training process of the multimodal large model, the user can set the query information according to the training requirements. Based on the above process, the training text including the training text, the translated text corresponding to the training text, the training image including the training text, and the query information can be obtained.

[0044] In some embodiments, the training images in the training samples may include three main scenes: document, scene, and poster. Document data has very long text content, and is subdivided into three categories: scientific research paper, news, and novel according to its scope. The image scenes of street scene data are very complex, and are subdivided into three categories: title, sign, and introduction according to their semantic richness. The text shapes of poster data are abstract and the layout is diverse, and are subdivided into two categories: cover and leaflet according to their source.

[0045] Step S120: Training the multimodal large model to be trained based on the training samples to obtain a trained multimodal large model, wherein the multimodal large model includes: an image encoder, the image encoder includes a first image encoder and a second image encoder, the first image encoder is used to extract first information from the pixel representation of the image and convert it into a first image vector representation, the second image encoder is used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than the granularity of the first information; an adapter, the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by the large language model; a large language model, the large language model is used to output a corresponding answer according to the input image vector representation and query information.

[0046] In order to enhance the OCR capability of the multimodal large model in a resource-friendly manner, the embodiment of the present application has been improved under the framework of the original multimodal large model. The specific framework is as follows: Figure 2As shown. It consists of four main components: an image encoder consisting of a first image encoder and a second image encoder, an adapter, and a large language model. Among them, the first image encoder is used to extract part of the information from the pixel representation of the image and convert it into a vector representation; the second image encoder is intended to make up for the deficiency of the first image encoder in encoding fine-grained information such as text, and extract finer-grained visual information in the image; the adapter converts the vector representation dimension of the image into a dimension acceptable to the large language model, helping the large language model to accept the input of the image vector representation; the large language model is used to output the corresponding answer based on the input image vector representation and query information. That is, the embodiment of the present application adds a second image encoder under the framework of the original multimodal large model to extract finer-grained visual information in the image and make up for the deficiency of the original multimodal large model in encoding fine-grained information such as text.

[0047] Optionally, the first image encoder may adopt CLIP, ie, a currently open source pre-trained visual language model.

[0048] Optionally, the second image encoder may adopt a classic Transformer application in visual modality (Vision Transformer, ViT) architecture to make up for the deficiency of CLIP in encoding fine-grained information such as text, and extract finer-grained visual information in the image.

[0049] Optionally, the adapter may adopt a Multi Layer Perceptron (MLP) architecture.

[0050] Optionally, a large language model can be used for a 7B architecture, such as Qwen-7B, to focus on English and Chinese scenarios.

[0051] Based on this, in an embodiment of the present application, a multimodal large model is provided including an image encoder (a first image encoder and a second image encoder), an adapter, and a large language model. The first image encoder is used to extract the first information from the pixel representation of the image and convert it into a first image vector representation, and the second image encoder is used to extract the second information from the pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than that of the first information, so as to make up for the deficiency of the first image encoder in encoding fine-grained information such as text, the adapter is used to convert the dimension of the first image vector representation into the dimension accepted by the large language model, and the large language model is used to output the corresponding answer according to the input image vector representation (the first image vector representation and the second image vector representation) and the query information.

[0052] In this embodiment, when a training sample is obtained, a multimodal large model to be trained can be trained based on the training sample to obtain a trained multimodal large model.

[0053] In some embodiments, training conditions can be pre-prepared and stored. When training samples are obtained, the multimodal large model can be trained based on the training samples, and parameter information of the multimodal large model to be trained can be detected during the training process. If it is detected that the parameter information of the multimodal large model to be trained meets the training conditions, the training can be stopped and the trained multimodal large model can be obtained.

[0054] As an implementable method, the trained multimodal large model can be pre-trained and stored locally on the device. Based on this, when the image to be translated is obtained for text-image translation, the trained multimodal large model can be directly called locally. For example, a command can be directly sent to the trained multimodal large model to instruct the trained multimodal large model to read the image to be translated in the target storage area, or the device can directly input the image to be translated into the trained multimodal large model stored locally, thereby effectively avoiding the influence of network factors that reduce the speed of inputting the image to be translated into the trained multimodal large model, so as to increase the speed of the trained multimodal large model to obtain the image to be translated and improve the user experience.

[0055] As another feasible method, the trained multimodal large model can also be pre-trained and stored in a server that is connected to the device. Based on this, when the device obtains an image to be translated for text-image translation, it can send instructions to the trained multimodal large model stored in the server through the network to instruct the trained multimodal large model to read the image to be translated through the network, or the device can send the image to be translated to the trained multimodal large model stored in the server through the network, thereby reducing the storage space occupied by the device and reducing the impact on the normal operation of the device by storing the trained multimodal large model in the server.

[0056] A training method for a multimodal large model provided in an embodiment of the present application determines a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image of the training text, and query information, and trains the multimodal large model to be trained based on the training sample to obtain a trained multimodal large model, wherein the multimodal large model includes an image encoder, an adapter, and a large language model, wherein the image encoder includes a first image encoder and a second image encoder, wherein the first image encoder is used to extract first information from a pixel representation of an image and convert it into a first image vector representation, and the second image encoder is used to extract second information from the pixel representation of the image, wherein the granularity of the second information is finer than that of the first information, the adapter is used to convert the dimension of the first image vector representation into the dimension accepted by the large language model, and the large language model is used to output a corresponding answer according to the input image vector representation and the query information, thereby compensating for the shortcoming of the multimodal large model in insufficient encoding of fine-grained image information by adding a fine-grained image encoder to the model framework, so as to improve the visual capture capability of the multimodal large model.

[0057] See also Figure 3 , Figure 3 The flowchart of the training method of the multimodal large model provided by an embodiment of the present application is shown in FIG. Figure 3 The process is described in detail, and the training method of the multimodal large model may specifically include the following steps:

[0058] Step S210: determining a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information.

[0059] For the detailed description of step S210, please refer to step S110, which will not be repeated here.

[0060] Step S220: Obtain an initial large language model.

[0061] Among them, the currently open-source large language models are all pre-trained versions, which have good performance in various natural language processing (NLP) tasks. However, to have excellent image-text translation capabilities, the large language model needs to have strong translation capabilities and become a large model specialized in translation.

[0062] In this embodiment, an initial large language model may be obtained and fine-tuned to make the fine-tuned large language model have strong translation capabilities and become a large model specialized in translation. Optionally, the initial large language model may be a currently open source large language model.

[0063] Step S230: selecting the training text, the translation text and third query information from the query information from the training samples, wherein the third query information is used to instruct to translate the training text.

[0064] Among them, training samples suitable for fine-tuning the large language model to enhance the translation ability of the large language model can be selected from the training samples, and the obtained initial large language model can be fine-tuned through the selected training samples to obtain the large language model with enhanced translation ability.

[0065] In this embodiment, training text, translation text, and third query information for indicating the translation of training text can be selected from the training samples, that is, the translation task is indicated by the third query information, and the initial large language model is fine-tuned using large-scale training text and translation text, aiming to train the initial large language model to have a strong translation capability. As an example, Figure 4 As shown, the third query information can be "Translate the following English sentence into Chinese" to indicate the English-Chinese translation task, and use the selected large-scale English-Chinese parallel prediction pairs (training text (English)-translation text (Chinese)) to fine-tune the initial large language model, aiming to train the initial large language model to have strong English-Chinese translation capabilities. At this time, the selected training samples can be the triples (x, y, q) of (English, Chinese, Query).

[0066] Step S240: fine-tune the initial large language model based on the training text, the translated text and the third query information to obtain the large language model in the multimodal large model to be trained, wherein the multimodal large model includes: an image encoder, the image encoder includes a first image encoder and a second image encoder, the first image encoder is used to extract first information from the pixel representation of the image and convert it into a first image vector representation, the second image encoder is used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than that of the first information; an adapter, the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by the large language model; a large language model, the large language model is used to output a corresponding answer according to the input image vector representation and the query information.

[0067] In this embodiment, when an initial large language model is obtained and training text, translation text and third query information are selected from training samples, the initial large language model can be fine-tuned based on the training text, translation text and third query information to obtain a large language model with enhanced translation capability, and the large language model with enhanced translation capability can be used as the large language model in the multimodal large model for subsequent training.

[0068] In some implementations, assuming that the selected training sample is a triple (x, y, q) of (English, Chinese, Query), a cross entropy loss function may be used in this stage to fine-tune the initial large language model based on the selected training sample, which is defined as follows:

[0069]

[0070] Among them, (yiy<i,x,q) represents the probability that yi (a word or phrase in Chinese) is correctly translated given x (English) and q (Query, i.e., translation instruction). This probability is calculated by a large language model.

[0071] Step S250: selecting the training text, the training image, and the first query information in the query information from the training sample, wherein the first query information is used to instruct the recognition of the training text in the training image.

[0072] Since the embodiment of the present application adds a second image encoder (fine-grained image encoder) under the framework of the original multimodal large model, and generally only performs random initialization before training, the present embodiment also aims to train the second image encoder to enhance the OCR capability of the multimodal large model, so that the second image encoder can extract more fine-grained visual information from the image. Therefore, training samples suitable for training the second image encoder to enhance the OCR capability of the multimodal large model can be selected from the training samples, and the multimodal large model to be trained including the second image encoder can be trained through the selected training samples to obtain a trained multimodal large model.

[0073] In this embodiment, training text, training image and first query information for indicating the recognition of training text in training image can be selected from training samples, that is, the OCR task is indicated by the first query information, and the large-scale OCR data (training text and training image) collected is used to train the multimodal large model to be trained, focusing on training the second image encoder and enhancing the text understanding ability of the large language model. As an example, Figure 5As shown, the first query information can be "What is the text in the image" to indicate the OCCR task, and the large multimodal model to be trained is trained using the collected large-scale OCR data (training text and training image), focusing on training the second image encoder and enhancing the text understanding ability of the large language model. At this time, the selected training sample can be a triple (m, y, q) of (Image, English, Query).

[0074] Step S260: Freeze the parameters of the first image encoder of the multimodal large model to be trained, and train the second image encoder, adapter and large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain the trained multimodal large model.

[0075] In this embodiment, when the training text, training image and first query information are selected from the training sample, the parameters of the first image encoder of the multimodal large model to be trained can be frozen, and the second image encoder, adapter and large language model of the multimodal large model to be trained are trained based on the training text, training image and first query information to obtain the multimodal large model with enhanced OCR capability, that is, to obtain the trained multimodal large model. For example, the first image encoder can be CLIP, and the adapter can be MLP. Then, the CLIP parameters can be kept frozen during the training process, and the second image encoder, MLP and large language model participate in the training.

[0076] In some embodiments, assuming that the selected training sample is a triple (m, y, q) of (Image, English, Query), a cross entropy loss function can be used in this stage to train the multimodal large model to be trained based on the selected training sample (training the second image encoder, adapter and large language model in the multimodal large model to be trained) to evaluate the performance of the model by calculating the difference between the predicted probability distribution and the actual label distribution based on the cross loss function. It is defined as follows:

[0077]

[0078] See also Figure 6 , Figure 6 This application shows Figure 3 The flowchart of step S260 of the multimodal large model training method shown in FIG. Figure 6 The process shown is described in detail, and the method may specifically include the following steps:

[0079] Step S261: Freeze the parameters of the first image encoder of the multimodal large model to be trained, and train the second image encoder, adapter and large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain a first initially trained multimodal large model.

[0080] Among them, it can be understood that the multimodal large model obtained by training through the above steps already has good OCR and translation capabilities at the same time. However, in order to realize the multimodal large model to complete the image-text translation task, it is necessary to guide the multimodal large model to learn to combine OCR and translation knowledge. For this reason, this embodiment proposes to fine-tune the multimodal large model obtained by training in the above steps by combining the ideas of Chain Of Thought (COT) and prompt engineering.

[0081] In some embodiments, when training text, training images and first query information are selected from training samples, the parameters of the first image encoder of the multimodal large model to be trained can be frozen, and the second image encoder, adapter and large language model of the multimodal large model to be trained can be trained based on the training text, training images and the first query information to obtain the multimodal large model with enhanced OCR capability, which is used as the first initially trained multimodal large model for subsequent fine-tuning with the idea of ​​combining thinking chain and prompt engineering to learn to combine OCR and translation knowledge, so as to enable the final trained multimodal large model to complete the image-text translation task.

[0082] Step S262: selecting the translation text, the training image, and second query information in the query information from the training sample, wherein the second query information is used to instruct to translate the training text in the training image based on understanding the training text in the training image.

[0083] Among them, training samples suitable for multimodal large model training can be selected from the training samples to guide the multimodal large model to learn to combine OCR and translation knowledge, and the first initially trained multimodal large model obtained in the above steps is trained with the selected training samples to obtain a multimodal large model that has learned to combine OCR and translation knowledge, that is, to obtain a trained multimodal large model.

[0084] In this embodiment, translation text, training image and second query information for instructing to translate the training text in the training image based on understanding the training text in the training image can be selected from the training sample, that is, the image-text translation task is indicated by the second query information, and a thinking chain is added to the second query information to guide the multimodal large model to connect OCR and translation knowledge in series. As an example, Figure 7As shown, the second query information can be "Please translate the text in the image into Chinese" to indicate the image-text translation task, and the thought chain of "first understand the meaning of the text in the image and then translate it into Chinese" is added to the second query information to guide the multimodal large model to connect OCR and translation knowledge in series, and then use the translated text and training images to train the multimodal large model of the first initial training. At this time, the selected training sample can be a triple (m, y, q) of (Image, Chinese, Query).

[0085] Step S263: Freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image and the second query information to obtain the trained multimodal large model.

[0086] In this embodiment, when the first initially trained multimodal large model is obtained through the training of the aforementioned steps, and the translation text, training image and second query information are selected from the training samples, the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model can be frozen, and the adapter and the large language model in the first initially trained multimodal large model are trained based on the translation text, training image and second query information to obtain a multimodal large model that has learned to combine OCR and translation knowledge, that is, to obtain a trained multimodal large model. For example, the first image encoder can be CLIP, and the adapter can be MLP. Then, the parameters of CLIP and the second image encoder can be kept frozen during the training process, and the MLP and the large language model participate in the training.

[0087] In some embodiments, assuming that the selected training sample is a triple (m, y, q) of (Image, Chinese, Query), the cross entropy loss function can be used in this stage to train the first initially trained multimodal large model based on the selected training sample (train the adapter and the large language model in the first initially trained multimodal large model) to evaluate the performance of the model by calculating the difference between the predicted probability distribution and the actual label distribution based on the cross loss function. It is defined as follows:

[0088]

[0089] Among them, it is understandable that the multimodal large model trained by the above steps already has a good image-text translation capability, but after testing, it is found that the multimodal large model trained by the above steps still has two problems that hinder the translation performance: 1) Repetition problem. The multimodal large model trained by the above steps will repeat a certain "loop" when encountering complex sentences until the maximum output length of the model. 2) Omission problem. When processing images of complex scenes, the multimodal large model trained by the above steps will ignore part of the text and only translate another part of the text in the picture. In order to solve these two problems at the same time, we propose a multi-conditional direct preference optimization (MultiConditional DPO, mcDPO) method, which aims to combine the idea of ​​reinforcement learning and alleviate the two phenomena by improving the loss function of direct preference optimization (Direct Preference Optimization, DPO). mcDPO consists of two parts: repeated preference optimization (Stammer Preference Optimization, sPO) and omitted preference optimization (Visual Preference Optimization, vPO).

[0090] See also Figure 8 , Figure 8 This application shows Figure 6 The flowchart of step S263 of the multimodal large model training method is shown in FIG. Figure 8 The process shown is described in detail, and the method may specifically include the following steps:

[0091] Step S2631A: Freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image and the second query information to obtain a second initially trained multimodal large model.

[0092] In some embodiments, when the first initially trained multimodal large model is obtained through the training steps above and the translated text, training image and second query information are selected from the training samples, the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model can be frozen, and the adapter and the large language model in the first initially trained multimodal large model are trained based on the translated text, training image and the second query information to obtain a multimodal large model that has learned to combine OCR and translation knowledge as the second initially trained multimodal large model, so as to subsequently use the loss function of direct preference optimization to make the second initially trained multimodal large model stay away from bad outputs (outputs with repeated texts) and move closer to good outputs (outputs without repeated texts), so as to alleviate the repetition problem of the multimodal large model obtained through training.

[0093] Step S2632A: Obtain repeated text based on the translated text, wherein the repeated text includes loop content.

[0094] Among them, the second initially trained multimodal large model is trained using a loss function of direct preference optimization, so that the second initially trained multimodal large model is away from bad outputs (outputs with repeated texts) and close to good outputs (outputs without repeated texts). The required training samples are the four-tuple (ym, yl, X, q) of (Chosen_label, Rejected_label, Image, Query). Among them, chosen_label is the corresponding translation result of the image in the image-text translation data, that is, the translated text. rejected_label is the simulated repetition phenomenon, that is, the repeated text. Image is the corresponding source language image, that is, the training image. Query is consistent with the second query information, which is used to indicate that the training text in the training image is translated based on the understanding of the training text in the training image.

[0095] Therefore, the training samples required for training the second initial training multimodal large model include translation text, training pictures, second query information and repeated text. It can be understood that the translation text, training pictures and second query information can be directly selected from the determined training samples, while the repeated text needs to be re-acquired or generated.

[0096] In this embodiment, a repeated text may be obtained based on the translated text, wherein the repeated text includes a loop content.

[0097] In some implementations, a method for generating repeated texts may be pre-set and stored. When a translated text is obtained, the translated text may be processed based on the method for generating repeated texts to obtain the repeated texts.

[0098] As an implementable method, text content of the target character length can be selected from the translated text, and the selected text content can be looped to obtain loop content, and repeated text can be obtained based on the loop content and the translated text. As an example, a "loop section" with a length of 2-6 characters can be randomly selected from chosen_label, and the "loop section" can be repeated until the maximum output length of the multimodal large model is 4096, which is used as rejected_label.

[0099] Step S2633A: Using a direct preference optimization method, control the second initially trained multimodal large model to perform preference learning on the translated text, the repeated text, the training image and the second query information to obtain the trained multimodal large model.

[0100] In some embodiments, when the translated text, training images, second query information, and repeated text are obtained, a direct preference optimization method can be used to control the second initially trained multimodal large model to perform preference learning on the translated text, training images, second query information, and repeated text, and obtain a multimodal large model that has learned to move away from the output with repeated text and closer to the output without repeated text, as the trained multimodal large model.

[0101] As an implementable method, assuming that the selected training sample is a quadruple (ym, yl, X, q) of (Chosen_label, Rejected_label, Image, Query), a direct preference optimization loss function (such as a repeated preference optimization loss function) can be used to train Chosen_label, Rejected_label, Image, Query to alleviate the repetition problem of the multimodal large model based on the direct preference optimization loss function. Its definition is as follows:

[0102]

[0103] Among them, L spo : represents the direct preference optimization loss function used in this embodiment. θ (y w X,q): represents the probability of the model outputting translation y given the input image X and query q. ref (y wX,q): represents the probability of the model outputting translation y given the input image X and query q before fine-tuning. θ: represents the model parameters that need to be trained. ref: refers to the model parameters before fine-tuning. σ and β: are hyperparameters used to adjust the degree of difference between the model parameters before and after fine-tuning to balance the model's ability to adapt to new data and its retention of old knowledge. The direct preference optimization loss function aims to adjust the model parameters θ so that the model, given the input image X and query q, has a relatively large probability of outputting Chosen_label and a relatively small probability of outputting Rejected_label. At the same time, by introducing the reference model ref and hyperparameters σ and β, the model's ability to adapt to new data and its retention of old knowledge are further balanced.

[0104] See also Fig. 9 , Fig. 9 This application shows Figure 6 The flowchart of step S263 of the multimodal large model training method is shown in FIG. Fig. 9 The process shown is described in detail, and the method may specifically include the following steps:

[0105] Step S2631B: Freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image and the second query information to obtain a third initially trained multimodal large model.

[0106] In some embodiments, when the first initially trained multimodal large model is obtained through the training steps above and the translation text, training image and second query information are selected from the training samples, the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model can be frozen, and the adapter and the large language model in the first initially trained multimodal large model are trained based on the translation text, training image and the second query information to obtain a multimodal large model that has learned to combine OCR and translation knowledge as the third initially trained multimodal large model, so as to use the loss function of direct preference optimization to enable the third initially trained multimodal large model to select appropriate visual information for reasoning and solve redundant visual information.

[0107] Step S2632B: Obtain a missing image based on the training image, wherein the text content in the missing image is less than the text content in the training image.

[0108] Among them, the loss function of direct preference optimization is used to train the multimodal large model of the third initial training, so that the multimodal large model of the third initial training selects appropriate visual information for reasoning, and the required training sample is a four-tuple (Xm, Xl, y, q) of (Chosen_image, Rejected_image, Label, Query). Among them, Chosen_image is the source language image in the image-text translation data, that is, the training image. Rejected_image is an image of simulated missing text content, that is, the missing image. Label is the translation result corresponding to the source language image, that is, the translated text. Query is consistent with the second query information, which is used to indicate that the training text in the training image is translated based on the understanding of the training text in the training image.

[0109] Therefore, the training samples required for training the multimodal large model of the third initial training include training images, translation text, second query information and missing images. It can be understood that the training images, translation text and second query information can be directly selected from the determined training samples, while the missing images need to be re-acquired or generated.

[0110] In this embodiment, the missing image may be obtained based on the training image, wherein the text content in the missing image is less than the text content in the training image.

[0111] In some implementations, a method for generating a missing image may be pre-set and stored. When a training image is obtained, the training image may be processed based on the method for generating the missing image to obtain the missing image.

[0112] As an implementable method, a target ratio of training texts can be selected from the training texts included in the training image, and the target ratio of training texts can be covered to obtain a missed image. As an example, 20% of the texts can be randomly masked from the chosen_image to serve as the rejected_image.

[0113] Step S2633B: adopting a direct preference optimization method, controlling the third initially trained multimodal large model to perform preference learning on the training image, the missing image, the translated text and the second query information, and obtaining the trained multimodal large model.

[0114] In some embodiments, when training images, translated text, second query information, and missing images are obtained, a direct preference optimization method can be used to control the third initially trained multimodal large model to perform preference learning on the training images, translated text, second query information, and missing images, and obtain a multimodal large model that has learned to select appropriate visual information for reasoning, as a trained multimodal large model.

[0115] As an implementable method, assuming that the selected training sample is a quadruple (Xm, Xl, y, q) of (Chosen_image, Rejected_image, Label, Query), a direct preference optimization loss function (such as an omitted preference optimization loss function) can be used to train Chosen_image, Rejected_image, Label, Query to alleviate the omission problem of the multimodal large model based on the direct preference optimization loss function. Its definition is as follows:

[0116]

[0117] Among them, L vpo : represents the direct preference optimization loss function used in this embodiment. θ (yX w ,q): represents the model given the occluded image X w Output the probability of translation y under the condition of query q. ref (yX w ,q): indicates that the model before fine-tuning is given an occluded image X w The output of the translation y probability is given the query q. θ: represents the model parameters that need to be trained. ref: refers to the model parameters before fine-tuning. σ and β: are hyperparameters used to adjust the difference between the model parameters before and after fine-tuning to balance the model's ability to adapt to new data and retain old knowledge. The direct preference optimization loss function aims to adjust the model parameters θ so that the model can be trained on a given occluded image X. w Under the conditions of and query q, more accurate translation results can be given. At the same time, by introducing the reference model ref and hyperparameters σ and β, the model's ability to adapt to new data and its ability to retain old knowledge are further balanced.

[0118] Of course, in this embodiment, the duplication problem and the omission problem of the multimodal large model can also be alleviated at the same time. At this time, the direct preference optimization loss function used to solve the duplication problem of the multimodal large model and the direct preference optimization loss function used to solve the omission problem of the multimodal large model can be summed.

[0119] The training method of the multimodal large model provided in the first embodiment of the present application is compared with Figure 1The training method of the multimodal large model shown, first, this embodiment also obtains an initial large language model, selects training text, translation text and third query information from the training sample, wherein the third query information is used to indicate the translation of the training text, and fine-tunes the initial large language model based on the training text, translation text and third query information to obtain the large language model in the multimodal large model to be trained, and calmly improves the translation ability of the large language model and the multimodal large model. Secondly, this embodiment also selects training text, training image and first query information from the training sample, wherein the first query information is used to indicate the recognition of the training text in the training image, freezes the parameters of the first image encoder of the multimodal large model to be trained, and trains the second image encoder, adapter and large language model of the multimodal large model to be trained based on the training text, training image and first query information to obtain the trained multimodal large model, thereby improving the recognition and extraction ability of the multimodal large model for fine-grained images. Furthermore, the present embodiment also selects translation text, training images and second query information in the query information from the training samples, wherein the second query information is used to indicate that the training text in the training image is translated based on the understanding of the training text in the training image, freezes the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and trains the adapter and the large language model in the initially trained multimodal large model based on the translation text, training images and the second query information to obtain a trained multimodal large model to improve the image-text translation capability of the multi-model large model. In addition, the present embodiment also performs preference optimization on repeated text and missing text in image-text translation, which can improve the image-text translation effect.

[0120] See also Fig.10 , Fig.10 The flowchart of the training method of the multimodal large model provided by an embodiment of the present application is shown in FIG. Fig.10 The process shown is described in detail, and the training method of the multimodal large model may specifically include the following steps:

[0121] Step S310: Determine a background board, a first language text, and a second language text, wherein the first language and the second language form a language pair.

[0122] Among them, a large amount of OCR data including various scenes can be collected from the Internet. Based on the OCR data, training text and training images including the training text can be obtained. Afterwards, the training text is translated using translation software to obtain the translated text. However, the parallel prediction data existing in the network is limited after all. Therefore, this embodiment also produces large-scale image-text translation data to solve the problem of data shortage, and puts it into model training.

[0123] In this embodiment, a background board, a first language text and a second language text can be determined, wherein the first language and the second language are a language pair, for example, the first language is English and the second language is Chinese, then the first language text can be English text, the second language text can be Chinese text, and the English text and the Chinese text refer to the same text content.

[0124] In some implementations, a background board may be selected from a background library, and a corpus pair (including a first language text and a second language text) may be selected from a corpus, such as an English-Chinese corpus (En2Zh corpus).

[0125] Step S320: generating an image including the first language text as a training image based on the first language text and the background plate, and using the first language text as the training text and the second language text as the translation text.

[0126] In this embodiment, when a background plate, a first language text and a second language text are obtained, an image including the first language text can be generated as a training image based on the first language text and the background plate, and the first language text can be used as a training text and the second language text can be used as a translation text to produce image-text translation data.

[0127] As an implementable manner, the first language text and the background plate may be fused to obtain an image including the first language text, and the obtained image including the first language text may be used as a training image.

[0128] In some embodiments, generating an image including the first language text as a training image based on the first language text and the background plate may include: determining a paper style, generating an image including the first language text based on the paper style, the first language text, and the background plate, and using the obtained image including the first language text as the training image. It is understandable that adding the paper style to the generation of the training image can make the data of the produced training image closer to the image taken in the real scene, for example, a photographed newspaper, a photographed test paper, etc., so that the training image has more diversity and improves the generalization ability of the multimodal large model.

[0129] As an implementable manner, a paper style may be selected from a paper library (paper style).

[0130] In some embodiments, generating an image including text in the first language based on the text in the first language and the background plate as a training image may include: generating an image including text in the first language based on the text in the first language and the background plate as an initial image, performing a style transformation on the initial image to obtain a training image, wherein the style transformation includes one or a combination of several of text distribution transformation, rotation angle transformation, and brightness transformation, so as to perform noise processing on the training image to make the training image more diverse.

[0131] As an implementable manner, when an initial image including a first language text is generated based on the first language text and a background plate, the initial image may be subjected to a brightness transformation to obtain a plurality of images with different brightness, and the plurality of images with different brightness are all used as training images; when an initial image including a first language text is generated based on the first language text and a background plate, the initial image may be subjected to a text distribution transformation (such as text position, text layout, etc.) to obtain a plurality of images with different text distributions, and the plurality of images with different text distributions are all used as training images; and / or, when an initial image including a first language text is generated based on the first language text and a background plate, the initial image may be subjected to a rotation angle transformation (such as text arrangement parallel to a horizontal plane, text arrangement perpendicular to a horizontal plane, etc.) to obtain a plurality of images with different rotation angles, and the plurality of images with different rotation angles are all used as training images.

[0132] The generated training image may be added with noise by setting the initial image to different styles through random seeds. For example, the training image may be obtained by setting the initial image to different styles through random seeds, such as text distribution transformation, rotation angle transformation and / or brightness transformation.

[0133] Step S330: determining a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information.

[0134] Step S340: Training the multimodal large model to be trained based on the training samples to obtain a trained multimodal large model, wherein the multimodal large model includes: an image encoder, the image encoder includes a first image encoder and a second image encoder, the first image encoder is used to extract first information from the pixel representation of the image and convert it into a first image vector representation, the second image encoder is used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than the granularity of the first information; an adapter, the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by the large language model; a large language model, the large language model is used to output a corresponding answer according to the input image vector representation and query information.

[0135] For the detailed description of step S330 to step S340, please refer to step S110 to step S120, which will not be repeated here.

[0136] Compared with the multimodal large model training method provided in one embodiment of the present application, Figure 1 The training method of the multimodal large model shown in the present embodiment further determines a background plate, a first language text and a second language text, wherein the first language and the second language are a language pair, and based on the first language text and the background plate, an image including the first language text is generated as a training image, and the first language text is used as the training text, and the second language text is used as the translation text, thereby solving the problem of lack of training data.

[0137] See also Fig.11 , Fig.11 The flowchart of the image-text translation method provided by an embodiment of the present application is shown. The method is used to add a fine-grained image encoder to the model framework to make up for the shortcomings of the multimodal large model in insufficient encoding of fine-grained image information, so as to improve the visual capture ability of the multimodal large model and enhance the image-text translation ability. In a specific embodiment, the method is applied to Fig.13 The image-text translation device 300 and the electronic device 100 equipped with the image-text translation device 300 are shown in FIG. Fig.14 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a smart phone, a tablet computer, a desktop computer, a server, etc., which is not limited here. Fig.11 The process shown is described in detail, and the image-text translation method may specifically include the following steps:

[0138] Step S410: Acquire an image to be translated, wherein the image to be translated includes text to be translated.

[0139] In this embodiment, an image that needs to be translated into text can be obtained and used as an image to be translated. It can be understood that the image to be translated should include text to be translated, and the text to be translated can be used as text to be translated.

[0140] In some embodiments, the image to be translated may be obtained by photographing with a camera of the device, the image to be translated may be obtained by reading locally on the electronic device, the image to be translated may be obtained from a connected server via a network, or the image to be translated may be obtained from other connected devices via a network, etc., which are not limited here.

[0141] Step S420: Input the image to be translated into the trained multimodal large model to obtain the target text corresponding to the text to be translated output by the trained multimodal large model, wherein the multimodal large model comprises: an image encoder, the image encoder comprises a first image encoder and a second image encoder, the first image encoder is used to extract first information from the pixel representation of the image and convert it into a first image vector representation, the second image encoder is used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than that of the first information; an adapter, the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into the dimension accepted by the large language model; the large language model, the large language model is used to output the corresponding answer according to the input image vector representation and the query information.

[0142] In this embodiment, when an image to be translated is obtained, the image to be translated can be input into the trained multimodal large model obtained by training in the aforementioned embodiment to obtain a target text corresponding to the text to be translated output by the trained multimodal large model, thereby completing the image-text translation of the image to be translated through the multimodal large model.

[0143] An embodiment of the present application provides a method for image-to-text translation, which obtains an image to be translated, wherein the image to be translated includes a text to be translated, inputs the image to be translated into a trained multimodal large model, and obtains a target text corresponding to the text to be translated output by the trained multimodal large model, wherein the multimodal large model includes an image encoder, an adapter, and a large language model, the image encoder includes a first image encoder and a second image encoder, the first image encoder is used to extract first information from a pixel representation of an image and convert it into a first image vector representation, the second image encoder is used to extract second information from the pixel representation of the image, wherein the granularity of the second information is finer than that of the first information, the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by the large language model, and the large language model is used to output a corresponding answer according to the input image vector representation and query information, thereby adding a fine-grained image encoder to the model framework to compensate for the shortcoming of the multimodal large model in insufficient encoding of fine-grained image information, so as to improve the visual capture capability of the multimodal large model and the image-to-text translation effect.

[0144] In summary, the embodiment of the present application introduces a method for implementing a text-to-image translation task using a multimodal large model. In terms of training data, the embodiment of the present application collects and automatically produces large-scale text-to-image translation data to solve the problem of data scarcity, and puts it into model training; in terms of model framework, the embodiment of the present application adds a fine-grained visual encoder to make up for the shortcomings of the multimodal large model in insufficient encoding of fine-grained image information, so as to improve the model's visual capture ability; in terms of training strategy, the embodiment of the present application uses a four-stage training paradigm (translation capability enhancement, image encoder training, text-to-image translation training, and mcDPO optimization) to enable the multimodal large model to have excellent multi-scene text-to-image translation performance.

[0145] See also Fig.12 , Fig.12 The module block diagram of the training device of the multimodal large model provided by an embodiment of the present application is shown. Fig.12 The block diagram shown in FIG. 2 is used to illustrate that the multimodal large model training device 200 includes: a training sample determination module 210 and a model training module 220, wherein:

[0146] The training sample determination module 210 is used to determine a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information.

[0147] Furthermore, the training sample determination module 210 includes: a parameter determination submodule and a training sample generation submodule, wherein:

[0148] The parameter determination submodule is used to determine a background board, a first language text, and a second language text, wherein the first language and the second language are a language pair.

[0149] The training sample generation submodule is used to generate an image including the first language text as a training image based on the first language text and the background plate, and use the first language text as the training text and the second language text as the translation text.

[0150] Furthermore, the training sample generation submodule includes: a paper style determination unit and a first training sample generation unit, wherein:

[0151] The paper style determination unit is used to determine the paper style.

[0152] The first training sample generating unit is configured to generate an image including the first language text as a training image based on the paper style, the first language text and the background plate.

[0153] Furthermore, the training sample generation submodule includes: an initial image generation unit and a second training sample generation unit, wherein:

[0154] An initial image generating unit is used to generate an image including the first language text as an initial image based on the first language text and the background plate.

[0155] The second training sample generating unit is used to perform style transformation on the initial image to obtain the training image, wherein the style transformation includes one or a combination of text distribution transformation, rotation angle transformation and brightness transformation.

[0156] The model training module 220 is used to train the multimodal large model to be trained based on the training samples to obtain a trained multimodal large model, wherein the multimodal large model includes: an image encoder, the image encoder includes a first image encoder and a second image encoder, the first image encoder is used to extract first information from the pixel representation of the image and convert it into a first image vector representation, the second image encoder is used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein the granularity of the second information is finer than the granularity of the first information; an adapter, the adapter is used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by the large language model; the large language model, the large language model is used to output a corresponding answer according to the input image vector representation and query information.

[0157] Furthermore, the model training module 220 includes: a training sample extraction submodule and a model training submodule, wherein:

[0158] The training sample extraction submodule is used to select the training text, the training image and the first query information in the query information from the training sample, wherein the first query information is used to indicate the recognition of the training text in the training image.

[0159] The model training submodule is used to freeze the parameters of the first image encoder of the multimodal large model to be trained, and train the second image encoder, the adapter and the large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain the trained multimodal large model.

[0160] Furthermore, the model training submodule includes: a first model training unit, a training sample extraction unit and a second model training unit, wherein:

[0161] The first model training unit is used to freeze the parameters of the first image encoder of the multimodal large model to be trained, and train the second image encoder, the adapter and the large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain a first initially trained multimodal large model.

[0162] A training sample extraction unit is used to select the translation text, the training image and the second query information in the query information from the training sample, wherein the second query information is used to instruct to translate the training text in the training image based on understanding the training text in the training image.

[0163] The second model training unit is used to freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image and the second query information to obtain the trained multimodal large model.

[0164] Furthermore, the second model training unit includes: a first model training subunit, a repeated text obtaining subunit and a second model training subunit, wherein:

[0165] The first model training subunit is used to freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image and the second query information to obtain the second initially trained multimodal large model.

[0166] The repeated text obtaining subunit is used to obtain repeated text based on the translated text, wherein the repeated text includes loop content.

[0167] Furthermore, the repeated text obtaining sub-unit includes: a text content extraction sub-sub-unit and a repeated text obtaining sub-sub-unit, wherein:

[0168] The text content extraction sub-subunit is used to select text content of a target character length from the translation text, and perform cyclic processing on the text content to obtain cyclic content.

[0169] The repeated text obtaining sub-subunit is used to obtain the repeated text based on the loop content and the translated text.

[0170] The second model training subunit is used to control the second initially trained multimodal large model to perform preference learning on the translated text, the repeated text, the training image and the second query information by adopting a direct preference optimization method to obtain the trained multimodal large model.

[0171] Furthermore, the second model training unit includes: a third model training subunit, a missing image obtaining subunit and a fourth model training subunit, wherein:

[0172] The third model training subunit is used to freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image and the second query information to obtain a third initially trained multimodal large model.

[0173] The missing image obtaining subunit is used to obtain a missing image based on the training image, wherein the text content in the missing image is less than the text content in the training image.

[0174] Furthermore, the missing image obtaining sub-unit includes: a training text selection sub-sub-unit and a missing image obtaining sub-sub-unit, wherein:

[0175] The training text selection sub-sub-unit is used to select a target proportion of training text from the training text included in the training image.

[0176] The missing image obtaining sub-sub-unit is used to obtain the missing image by covering the training text of the target ratio.

[0177] The fourth model training subunit is used to control the third initially trained multimodal large model to perform preference learning on the training image, the missing image, the translated text and the second query information by adopting a direct preference optimization method to obtain the trained multimodal large model.

[0178] Furthermore, the multimodal large model training device 200 further includes: an initial large language model acquisition module, a training sample selection module and a large language model fine-tuning module, wherein:

[0179] The initial large language model acquisition module is used to obtain the initial large language model.

[0180] The training sample selection module is used to select the training text, the translation text and the third query information from the query information from the training sample, wherein the third query information is used to instruct the translation of the training text.

[0181] A large language model fine-tuning module is used to fine-tune the initial large language model based on the training text, the translation text and the third query information to obtain a large language model in the multimodal large model to be trained.

[0182] See also Fig.13 , Fig.13 The module block diagram of the image-text translation device provided by an embodiment of the present application is shown below. Fig.13 The block diagram shown in the figure is used to illustrate that the image-text translation device 300 includes: an image acquisition module 310 and an image-text translation module 320, wherein:

[0183] The image acquisition module 310 is used to acquire the image to be translated, wherein the image to be translated includes the text to be translated.

[0184] The image-text translation module 320 is used to input the image to be translated into a trained multimodal large model, and obtain a target text corresponding to the text to be translated output by the trained multimodal large model, wherein the trained multimodal large model is obtained by training using the training method of the multimodal large model.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0186] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0187] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0188] See also Fig.14 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0189] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0190] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0191] See also Fig.15 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 400 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0192] The computer readable storage medium 400 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer readable storage medium 400 has storage space for program code 410 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 410 can be compressed, for example, in an appropriate form.

[0193] In summary, the training method, image-text translation method and device of the multimodal large model provided in the embodiments of the present application determine a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, including a training image of the training text and query information, and trains the multimodal large model to be trained based on the training sample to obtain a trained multimodal large model, wherein the multimodal large model includes an image encoder, an adapter and a large language model, wherein the image encoder includes a first image encoder and a second image encoder, wherein the first image encoder is used to extract first information from the pixel representation of the image and convert it into a first image vector representation, and the second image encoder is used to extract second information from the pixel representation of the image, wherein the granularity of the second information is finer than that of the first information, and the adapter is used to convert the dimension of the first image vector representation into the dimension accepted by the large language model, and the large language model is used to output the corresponding answer according to the input image vector representation and the query information, thereby compensating for the disadvantage of insufficient encoding of fine-grained image information by the multimodal large model by adding a fine-grained image encoder to the model framework, so as to improve the visual capture capability of the multimodal large model.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a multimodal large model, characterized in that: The method comprises: Determining a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information; The multimodal large model to be trained is trained based on the training samples to obtain a trained multimodal large model, wherein the multimodal large model includes: An image encoder, the image encoder comprising a first image encoder and a second image encoder, the first image encoder being used to extract first information from a pixel representation of an image and convert it into a first image vector representation, the second image encoder being used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein a granularity of the second information is finer than a granularity of the first information; An adapter, the adapter being used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by a large language model; A large language model is used to output a corresponding answer based on the input image vector representation and query information.

2. The method according to claim 1, characterized in that The step of training the multimodal large model to be trained based on the training samples to obtain the trained multimodal large model includes: Selecting the training text, the training image, and first query information from the training sample, wherein the first query information is used to indicate recognition of the training text in the training image; Freeze the parameters of the first image encoder of the multimodal large model to be trained, and train the second image encoder, adapter and large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain the trained multimodal large model.

3. The method according to claim 2, characterized in that The freezing of the parameters of the first image encoder of the multimodal large model to be trained, and training the second image encoder, the adapter and the large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain the trained multimodal large model, includes: Freeze the parameters of the first image encoder of the multimodal large model to be trained, and train the second image encoder, the adapter and the large language model of the multimodal large model to be trained based on the training text, the training image and the first query information to obtain a first initially trained multimodal large model; Selecting the translation text, the training image, and second query information in the query information from the training sample, wherein the second query information is used to instruct to translate the training text in the training image based on understanding the training text in the training image; Freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image, and the second query information to obtain the trained multimodal large model.

4. The method according to claim 3, characterized in that The freezing of the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and training the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image, and the second query information to obtain the trained multimodal large model includes: Freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image, and the second query information to obtain a second initially trained multimodal large model; Obtaining a repeated text based on the translated text, wherein the repeated text includes a loop content; A direct preference optimization method is adopted to control the second initially trained multimodal large model to perform preference learning on the translated text, the repeated text, the training image and the second query information to obtain the trained multimodal large model.

5. The method according to claim 4, characterized in that The obtaining of repeated text based on the translated text comprises: Selecting text content of a target character length from the translation text, and looping the text content to obtain loop content; The repeated text is obtained based on the loop content and the translated text.

6. The method according to claim 3, characterized in that: The freezing of the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and training the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image, and the second query information to obtain the trained multimodal large model includes: Freeze the parameters of the first image encoder and the second image encoder in the first initially trained multimodal large model, and train the adapter and the large language model in the first initially trained multimodal large model based on the translated text, the training image, and the second query information to obtain a third initially trained multimodal large model; Obtaining a missing image based on the training image, wherein text content in the missing image is less than text content in the training image; A direct preference optimization method is adopted to control the third initially trained multimodal large model to perform preference learning on the training image, the missing image, the translated text and the second query information to obtain the trained multimodal large model.

7. The method according to claim 6, characterized in that The step of obtaining a missing image based on the training image comprises: Selecting a target proportion of training text from the training text included in the training image; The missing image is obtained by covering the training text with the target ratio.

8. The method according to any one of claims 1 to 7, characterized in that: Before training the multimodal large model to be trained based on the training samples to obtain the trained multimodal large model, the method further includes: Obtain an initial large language model; Selecting the training text, the translation text, and third query information from the query information from the training sample, wherein the third query information is used to instruct to translate the training text; The initial large language model is fine-tuned based on the training text, the translated text, and the third query information to obtain a large language model in the multimodal large model to be trained.

9. The method according to any one of claims 1 to 7, characterized in that: Determining the training sample comprises: Determining a background board, a first language text, and a second language text, wherein the first language and the second language form a language pair; An image including the first language text is generated as a training image based on the first language text and the background plate, and the first language text is used as the training text and the second language text is used as the translation text.

10. The method according to claim 9, characterized in that The step of generating an image including the first language text as a training image based on the first language text and the background plate includes: Determine the paper style; Based on the paper style, the first language text and the background plate, an image including the first language text is generated as a training image.

11. The method according to claim 9, characterized in that The step of generating an image including the first language text as a training image based on the first language text and the background plate includes: generating an image including the first language text as an initial image based on the first language text and the background plate; Performing style transformation on the initial image to obtain the training image, wherein the style transformation includes one or a combination of text distribution transformation, rotation angle transformation, and brightness transformation.

12. A method for translating text into images, characterized in that: The method comprises: Acquire an image to be translated, wherein the image to be translated includes text to be translated; The image to be translated is input into a trained multimodal large model to obtain a target text corresponding to the text to be translated which is output by the trained multimodal large model, wherein the trained multimodal large model is trained based on the training method of the multimodal large model according to any one of claims 1 to 11.

13. A multi-modal large model training device, characterized in that: The device comprises: A training sample determination module, used to determine a training sample, wherein the training sample includes a training text, a translation text corresponding to the training text, a training image including the training text, and query information; A model training module is used to train the multimodal large model to be trained based on the training samples to obtain a trained multimodal large model, wherein the multimodal large model includes: An image encoder, the image encoder comprising a first image encoder and a second image encoder, the first image encoder being used to extract first information from a pixel representation of an image and convert it into a first image vector representation, the second image encoder being used to extract second information from the pixel representation of the image and convert it into a second image vector representation, wherein a granularity of the second information is finer than a granularity of the first information; An adapter, the adapter being used to convert the dimension of the first image vector representation and the dimension of the second image vector representation into a dimension accepted by a large language model; A large language model is used to output a corresponding answer based on the input image vector representation and query information.

14. A picture-text translation device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be translated, wherein the image to be translated includes a text to be translated; The image-text translation module is used to input the image to be translated into a trained multimodal large model, and obtain a target text corresponding to the text to be translated output by the trained multimodal large model, wherein the trained multimodal large model is trained based on the training method of the multimodal large model according to any one of claims 1 to 11.

15. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 12.

Images